1 of 100

2024.3

Immuta Documentation - 2024.3

Your guide to discovering, securing, and monitoring your data with Immuta.

What is Immuta?

Immuta helps you achieve the following outcomes in your data platform:

Simplify Operations: Immuta’s dynamic access control and policy management require 93x fewer data policies to manage access control in your data platform according to the GigaOm study. It is simple and scalable, which improves change management and lowers the total cost of ownership of cloud data management.
Improve data security: Immuta helps prove compliance with rules and regulations, even when securing hundreds of thousands of tables. An Immuta customer, Swedbank, migrated all critical analytics workloads to the cloud in less than 12 months, including over 100 terabytes from more than 2,500 sources.
Unlock data’s value: Immuta helps organizations get access to more data 100x faster, which translates to improved productivity. An Immuta customer, Thomson Reuters enabled faster access to data, resulting in a 60x increase in data usage and greater productivity.

How does Immuta do it?

Immuta provides three modules to create a full data security platform suite.

Sensitive data discovery and classification: Immuta Discover

Discover sensitive data from millions of fields without manual effort. With over 60 pre-built and domain-specific identifiers, you can tailor data classification to your unique business needs based on your desired confidence level.

Continuous data security monitoring: Immuta Detect

Leverage timely insights into data access and user activity with anomaly indicators for faster analysis and proactive actions.

Data security and access control: Immuta Secure

Immuta’s attribute-based access control (ABAC) delivers scalable data access without role explosion, and dynamic data masking ensures the right users can access the right data.

Self-Managed Deployment

This section illustrates how to install Immuta on Kubernetes using the Immuta Enterprise Helm chart.

Getting started

This how-to guide includes instructions and links for installing Immuta in any Kubernetes environment.

Requirements

This reference guide provides an overview of the Immuta Enterprise Helm chart version requirements and infrastructure recommendations.

Install

The guides in this section illustrate how to install and deploy Immuta in your Kubernetes environment.

Configure

The guides in this section illustrate how to configure your Immuta Enterprise Helm chart for various scenarios, including optimizing your deployment for production environments.

Upgrade

The guides in this section illustrate how to upgrade Immuta.

Disaster recovery

This guide provides links to additional resources for disaster recovery strategies.

Troubleshooting

This page provides troubleshooting guidance and outlines frequently asked questions for the Immuta installation.

Getting Started

The instructions and how-to guides on this page illustrate how to install Immuta in your Kubernetes environment. If you are upgrading Immuta, navigate to the Upgrade section instead.

Prerequisites and requirements

Use a supported version of Kubernetes.
Use Helm 3.2.0 or newer (When using a Helm version older than 3.8.0, enable OCI experimental mode by exporting environment variable HELM_EXPERIMENTAL_OCI=1.)
Deploy the services listed on the Deployment requirements guide. See the recommendations table for guidance for specific cloud providers.
Grant RBAC permissions to create Kubernetes resources in the cluster.

Pull the Helm chart

Consult the upgrade overview if unsure which Helm chart to use.

ocir.immuta.com

Helm chart availability

The deprecated Immuta Helm chart (IHC) is not available from ocir.immuta.com.

Copy the snippet below and replace the placeholder text with the credentials provided to you by your customer success manager:

echo <token> | helm registry login --password-stdin --username <username> ocir.immuta.com

(Deprecated) archives.immuta.com

Obtain the Kubernetes Helm Installation Credentials to authenticate with Immuta's Helm registry from your Immuta support professional.
Copy the snippet below and replace the placeholder text with the credentials you obtained in the previous step to add the Helm repository:
```
echo <token> | helm repo add --username <username> --password-stdin immuta https://archives.immuta.com/charts
```
(Optional) Fetch the latest chart information from the repository:
```
helm repo update immuta
```

List all available versions of the chart.

Immuta Enterprise Helm chart (IEHC):

helm search repo --regexp 'immuta/immuta-enterprise\v' --versions

Immuta Helm chart (IHC):

helm search repo --regexp 'immuta/immuta\v' --versions

--pass-credentials flag

If you encounter an unauthorized error when adding the Immuta Enterprise Helm chart (IEHC), run helm repo add --pass-credentials.

Usernames and passwords are only passed to the URL location of the Helm repository by default. The username and password are scoped to the scheme, host, and port of the Helm repository. To pass the username and password to other domains Helm may encounter when it goes to retrieve a chart, the new --pass-credentials flag can be used. This flag restores the old behavior for a single repository as an opt-in behavior.

If you use a username and password for a Helm repository, you can audit the Helm repository in order to check for another domain that could have received the credentials. In the index.yaml file for that repository, look for another domain in the URL's list for the chart versions. If there is another domain found and that chart version is pulled or installed, the credentials will be passed on.

Install Immuta

Immuta can be installed on any Kubernetes cluster. Select a guide below that corresponds to your Kubernetes distribution to install Immuta. If your distribution is not listed below (such as K3s or RKE2), follow the generic installation instructions:

Managed public cloud: This guide includes instructions for
- Amazon Elastic Kubernetes Service (EKS)
- Google Kubernetes Engine (GKE)
- Microsoft Azure Kubernetes Service (AKS)
Red Hat OpenShift
K3s

Configure Ingress

To complete your installation and access the Immuta application, configure Ingress.

Additional recommendations

The configure section includes guidance for various scenarios you may encounter during and post-deployment. Below are several guides from that section that most customers follow to complete their deployment of Immuta, but none of these is a requirement for the Immuta installation to work.

TLS configuration: Secure your Ingress by specifying a Secret that contains a TLS private key and certificate.
Immuta in production: Follow these best practices for configuring your deployment for a production environment.
External cache configuration: The Immuta Enterprise Helm chart manages its own Memcached deployment inside the cluster. However, you can opt to externalize the key-value cache post-installation.

Deployment Requirements

Immuta comprises three core services: Secure, Discover, and Detect. These services rely on PostgreSQL and Elasticsearch to store their states, a caching layer, and Temporal for job execution. The illustration below shows the relationships among these services.

The Immuta Enterprise Helm chart (IEHC) does not include the deployment of PostgreSQL or Elasticsearch, so you must deploy them separately.

Although Immuta recommends using Elasticsearch because it supports several new Immuta features and services, you can deploy Immuta without Elasticsearch. The table below outlines the Immuta features supported with and without Elasticsearch and the dependencies you must deploy and manage yourself.

Immuta with Elasticsearch

Immuta without Elasticsearch

For more information about legacy features and services no longer enabled in the recommended deployment of Immuta, see the Legacy features and services section.

Version requirements

Kubernetes versions

Kubernetes distribution

Kubernetes versions

Metadata database (PostgreSQL)

PostgreSQL incompatibilities

Immuta is not compatible with PostgreSQL abstraction layers, such as Amazon Aurora.

PostgreSQL 15.0 or newer
The pgcrypto, btree_gin extensions must be enabled

Elasticsearch

Elasticsearch v7 API or newer
OpenSearch compatible with Elasticsearch v7 API or newer

Cache (Redis/Memcached)

Built-in cache

The IEHC manages its own Memcached deployment inside the cluster. The key-value cache can optionally be externalized post installation.

Redis 7.0 or newer
Memcached 1.6 or newer

Temporal

Built-in Temporal server

The IEHC deploys a Temporal server and its requisite components. However, you may choose to use your own Temporal instance.

Temporal 1.24.2 or newer

Infrastructure recommendations

Legacy features and services

Some legacy services and features are no longer enabled in the recommended configuration of the IEHC. The table below lists these features and provides links to documentation that outlines how to enable them in Immuta.

Next step

Follow the Getting started guide to install Immuta.

Install

The guides in this section illustrate how to install and deploy Immuta in your Kubernetes environment.

Managed public cloud: This guide includes instructions for
- Amazon Elastic Kubernetes Service (EKS)
- Google Kubernetes Engine (GKE)
- Microsoft Azure Kubernetes Service (AKS)
Red Hat OpenShift
SUSE Rancher Kubernetes Engine (RKE2)
K3s

Managed Public Cloud

This is a guide on how to deploy Immuta on Kubernetes in the following managed public cloud providers:

Amazon Web Services (AWS)
Microsoft Azure
Google Cloud Platform (GCP)

Prerequisites

The following cloud-managed services must be provisioned and running before proceeding:

(Optional)

(Optional)

(Optional)

Feature availability

If deployed without Elasticsearch/OpenSearch, several core services and features will be unavailable. See the for details.

Checklist

This checklist outlines the necessary prerequisites for successfully deploying Immuta.

Credentials

You have the credentials needed to access the ocir.immuta.com OCI registry. These can be viewed in your user profile at .

PostgreSQL

You have a PostgreSQL user account with the necessary privileges to create databases, extensions, and roles.

Elasticsearch

Setup

Helm

Authenticate with OCI registry

Kubernetes

Creating a dedicated namespace ensures a logically isolated environment for your Immuta deployment, preventing resource conflicts with other applications.

Create namespace

Create a Kubernetes namespace named immuta.
Switch to namespace immuta. All subsequent kubectl commands will default to this namespace.

Create registry secret

Create a container registry pull secret. Contact your Immuta representative to obtain credentials to authenticate with ocir.immuta.com.

PostgreSQL

Connecting a client

There are numerous ways to connect to a PostgreSQL database. This step demonstrates how to connect with psql by creating an ephemeral Kubernetes pod.

Connect to the database

Connect to the database as an admin (e.g., postgres) by creating an ephemeral container inside the Kubernetes cluster. A shell prompt will not be displayed after executing the kubectl run command outlined below. Wait 5 seconds, and then proceed by entering a password.

Create role

Temporal's upgrade mechanism utilizes SQL command CREATE EXTENSION when managing database schema changes. However, in cloud-managed PostgreSQL offerings, this command is typically restricted to roles with elevated privileges to protect the database and maintain the stability of the cloud environment.

To ensure Temporal can successfully manage its schema, an administrator role must be granted temporarily. The role name varies depending on the cloud-managed service:

Amazon RDS: rds_superuser
Azure Database: azure_pg_admin
Google Cloud SQL: cloudsqlsuperuser

Create the immuta role.
Grant administrator privileges to the immuta role. Upon successfully completing this installation guide, you can optionally revoke this role grant.
Grant the immuta role to the current user. Upon successfully completing this installation guide, you can optionally revoke this role grant.

Create databases

Create databases.
Configure the immuta database.
Configure the temporal database.
Configure the temporal_visibility database.
Exit the interactive prompt. Type \q, and then press Enter.

Install Immuta

This section demonstrates how to deploy Immuta using the Immuta Enterprise Helm chart once the prerequisite cloud-managed services are configured.

Feature availability

Deploy Immuta.
Wait for all pods to become ready.

Validation

Determine the name of the Secure service.
Listen on local port 8080, forwarding TCP traffic to the Secure service's port named http.
Press Control+C to stop port forwarding.

Next steps

SUSE Rancher Kubernetes Engine (RKE2)

This is a generic guide that demonstrates how to deploy Immuta into RKE2 (i.e., Rancher Government) without dependencies on any particular cloud provider. Advanced Kubernetes expertise is required; therefore, it is not suitable for beginners.

Considerations

For the purposes of this guide, the following state stores are deployed in Kubernetes using third-party Helm charts maintained by Bitnami:

Running production-grade stateful workloads (e.g., databases) in Kubernetes is difficult and heavily discouraged due to the following reasons.

Operational overhead: Managing PostgreSQL and Elasticsearch on Kubernetes requires expertise in deploying, maintaining, and scaling these databases and search engines effectively. This involves tasks like setting up monitoring, configuring backups, managing updates, and ensuring high availability. Cloud-managed services abstract much of this operational burden away, allowing teams to focus on application development rather than infrastructure management.
Resource allocation and scaling: Kubernetes requires careful resource allocation and scaling decisions to ensure that PostgreSQL and Elasticsearch have sufficient CPU, memory, and storage. Properly sizing these resources can be challenging and may require continuous adjustments as workload patterns change. Managed services typically handle this scaling transparently and can automatically adjust based on demand.
Data integrity and high availability: PostgreSQL and Elasticsearch deployments need robust strategies for data integrity and high availability. Kubernetes can facilitate high availability through pod replicas and distributed deployments, but ensuring data consistency and durability across database instances and search indexes requires careful consideration and often additional tooling.
Performance: Kubernetes networking and storage configurations can introduce performance overhead compared to native cloud services. For latency-sensitive applications or high-throughput workloads, these factors become critical in maintaining optimal performance.
Observability: Troubleshooting issues in a Kubernetes environment, especially related to database and search engine performance, can be complex. Managed services typically come with built-in monitoring, logging, and alerting capabilities tailored to the specific service, making it easier to identify and resolve issues.
Security and compliance: Kubernetes environments require careful attention to security best practices, including network policies, access controls, and encryption. Managed services often come pre-configured with security features and compliance certifications, reducing the burden on teams to implement and maintain these measures.

Prerequisites

Checklist

This checklist outlines the necessary prerequisites for successfully deploying Immuta.

Credentials

You have the credentials needed to access the ocir.immuta.com OCI registry. These can be viewed in your user profile at support.immuta.com.

Setup

Helm

Authenticate with OCI registry

echo <token> | helm registry login --password-stdin --username <username> ocir.immuta.com

Kubernetes

Creating a dedicated namespace ensures a logically isolated environment for your Immuta deployment, preventing resource conflicts with other applications.

Create namespace

Create a Kubernetes namespace named immuta.
```
kubectl create namespace immuta
```
Switch to namespace immuta. All subsequent kubectl commands will default to this namespace.
```
kubectl config set-context --current --namespace=immuta
```

Create registry secret

Create a container registry pull secret. Contact your Immuta representative to obtain credentials to authenticate with ocir.immuta.com.

kubectl create secret docker-registry immuta-oci-registry \
    --docker-server=https://ocir.immuta.com \
    --docker-username="<username>" \
    --docker-password="<token>" \
    --docker-email=support@immuta.com

Elasticsearch

Install Helm chart

Create a Helm values file named es-values.yaml with the following content:

es-values.yaml

```yaml
master:
    masterOnly: false
    replicaCount: 1

data:
    replicaCount: 0

coordinating:
    replicaCount: 0

ingest:
    replicaCount: 0
```

Deploy Elasticsearch.

helm install es-db oci://registry-1.docker.io/bitnamicharts/elasticsearch \
    --values es-values.yaml

Wait for all Elasticsearch pods to become ready.

kubectl wait --for=condition=Ready pods --selector "app.kubernetes.io/name=elasticsearch"

PostgreSQL

Install Helm chart

Create a Helm values file named pg-values.yaml.

pg-values.yaml

```yaml
primary:
  # The default 'nano' preset is insufficient and will lead to Out Of Memory (OOM) 
  # errors.
  #
  # Possible values: nano, micro, small, medium, large, xlarge, 2xlarge
  resourcesPreset: medium
  initdb:
    scripts:
      00_setup.sql: |
        CREATE ROLE immuta with LOGIN ENCRYPTED PASSWORD '<postgres-password>';
        ALTER ROLE immuta SET search_path TO bometadata,public;

        CREATE DATABASE temporal WITH OWNER immuta;
        CREATE DATABASE temporal_visibility WITH OWNER immuta;
        CREATE DATABASE immuta WITH OWNER immuta;

        GRANT ALL ON DATABASE temporal TO immuta;
        GRANT ALL ON DATABASE temporal_visibility TO immuta;
        GRANT ALL ON DATABASE immuta TO immuta;
```

Update all placeholder values in the pg-values.yaml file.

Deploy PostgreSQL.

helm install pg-db oci://registry-1.docker.io/bitnamicharts/postgresql \
    --values pg-values.yaml

Wait for all PostgreSQL pods to become ready.

kubectl wait --for=condition=Ready pods --selector "app.kubernetes.io/name=postgresql"

Configure databases

Determine the name of the PostgreSQL database pod. This will be referenced in a subsequent step.
```
kubectl get pod --selector "app.kubernetes.io/name=postgresql" --output name
```
Exec into the PostgreSQL database pod using psql.
```
kubectl exec --stdin --tty <database-pod> -- psql
```
Configure the immuta database.
```
\c immuta
CREATE EXTENSION pgcrypto;
```

Configure the temporal database.

\c temporal
GRANT CREATE ON SCHEMA public TO immuta;

Configure the temporal_visibility database.

\c temporal_visibility
GRANT CREATE ON SCHEMA public TO immuta;
CREATE EXTENSION btree_gin;

Exit the interactive prompt. Type \q, then press Enter.

Install Immuta

This section demonstrates how to deploy Immuta using the Immuta Enterprise Helm chart once the prerequisite cloud-managed services are configured.

immuta-values.yaml

global:
  imageRegistry: ocir.immuta.com
  imagePullSecrets:
    - name: immuta-oci-registry
  postgresql:
    # Each Kubernetes Service has a DNS record associated with it. See: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
    # The anatomy of a domain name is as follows:
    #   <service>.<namespace>.svc.<cluster-domain>
    #
    # Where the default cluster domain is: cluster.local
    host: pg-db-postgresql.immuta.svc.cluster.local
    port: 5432
    username: immuta
    password: <postgres-password>

audit:
  config:
    elasticsearchEndpoint: http://es-db-elasticsearch.immuta.svc.cluster.local:9200
    elasticsearchUsername: <elasticsearch-username>
    elasticsearchPassword: <elasticsearch-password>
  postgresql:
    database: immuta

secure:
  postgresql:
    database: immuta
    
temporal:
  enabled: true
  schema:
    createDatabase:
      enabled: false
  server:
    config:
      persistence:
        default:
          sql:
            database: temporal
        visibility:
          sql:
            database: temporal_visibility

Create a file named immuta-values.yaml with the above content, making sure to update all placeholder values.

Deploy Immuta.

helm install immuta oci://ocir.immuta.com/stable/immuta-enterprise \
    --values immuta-values.yaml \
    --version 2024.3.2

Wait for all pods to become ready.

kubectl wait --for=condition=Ready pods --all

Validation

This section helps you validate your Immuta installation by temporarily accessing the application locally. However, this access is limited to your own computer. To enable access for other devices, you must proceed with configuring Ingress outlined in the Next steps section.

Determine the name of the Secure service.

kubectl get service --selector "app.kubernetes.io/component=secure" --output name

Listen on local port 8080, forwarding TCP traffic to the Secure service's port named http.
```
kubectl port-forward <service-name> 8080:http
```
In a web browser, navigate to localhost:8080, to ensure the Immuta application loads.
Press Control+C to stop port forwarding.

Next steps

Configure Ingress for RKE2 (required).
Configure TLS certificates for RKE2.
Learn about best practices for running Immuta in production.

K3s

This is a generic guide that demonstrates how to deploy Immuta into K3s without dependencies on any particular cloud provider. Advanced Kubernetes expertise is required; therefore, it is not suitable for beginners.

Considerations

For the purposes of this guide, the following state stores are deployed in Kubernetes using third-party Helm charts maintained by :

Running production-grade stateful workloads (e.g., databases) in Kubernetes is difficult and heavily discouraged due to the following reasons.

Operational overhead: Managing PostgreSQL and Elasticsearch on Kubernetes requires expertise in deploying, maintaining, and scaling these databases and search engines effectively. This involves tasks like setting up monitoring, configuring backups, managing updates, and ensuring high availability. Cloud-managed services abstract much of this operational burden away, allowing teams to focus on application development rather than infrastructure management.
Resource allocation and scaling: Kubernetes requires careful resource allocation and scaling decisions to ensure that PostgreSQL and Elasticsearch have sufficient CPU, memory, and storage. Properly sizing these resources can be challenging and may require continuous adjustments as workload patterns change. Managed services typically handle this scaling transparently and can automatically adjust based on demand.
Data integrity and high availability: PostgreSQL and Elasticsearch deployments need robust strategies for data integrity and high availability. Kubernetes can facilitate high availability through pod replicas and distributed deployments, but ensuring data consistency and durability across database instances and search indexes requires careful consideration and often additional tooling.
Performance: Kubernetes networking and storage configurations can introduce performance overhead compared to native cloud services. For latency-sensitive applications or high-throughput workloads, these factors become critical in maintaining optimal performance.
Observability: Troubleshooting issues in a Kubernetes environment, especially related to database and search engine performance, can be complex. Managed services typically come with built-in monitoring, logging, and alerting capabilities tailored to the specific service, making it easier to identify and resolve issues.
Security and compliance: Kubernetes environments require careful attention to security best practices, including network policies, access controls, and encryption. Managed services often come pre-configured with security features and compliance certifications, reducing the burden on teams to implement and maintain these measures.

Prerequisites

Checklist

This checklist outlines the necessary prerequisites for successfully deploying Immuta.

Credentials

Setup

Helm

Authenticate with OCI registry

Kubernetes

Creating a dedicated namespace ensures a logically isolated environment for your Immuta deployment, preventing resource conflicts with other applications.

Create namespace

Create a Kubernetes namespace named immuta.
Switch to namespace immuta. All subsequent kubectl commands will default to this namespace.

Create registry secret

Create a container registry pull secret. Contact your Immuta representative to obtain credentials to authenticate with ocir.immuta.com.

Elasticsearch

Install Helm chart

Create a Helm values file named es-values.yaml with the following content:

Deploy Elasticsearch.
Wait for all Elasticsearch pods to become ready.

PostgreSQL

Install Helm chart

Create a Helm values file named pg-values.yaml.

Deploy PostgreSQL.
Wait for all PostgreSQL pods to become ready.

Configure databases

Determine the name of the PostgreSQL database pod. This will be referenced in a subsequent step.
Exec into the PostgreSQL database pod using psql.
Configure the immuta database.
Configure the temporal database.
Configure the temporal_visibility database.
Exit the interactive prompt. Type \q, then press Enter.

Install Immuta

This section demonstrates how to deploy Immuta using the Immuta Enterprise Helm chart once the prerequisite cloud-managed services are configured.

Deploy Immuta.
Wait for all pods to become ready.

Validation

Determine the name of the Secure service.
Listen on local port 8080, forwarding TCP traffic to the Secure service's port named http.
Press Control+C to stop port forwarding.

Next steps

Guides

Introduced in 2024.2, the Immuta Enterprise Helm chart (IEHC) is an entirely new Helm chart used to deploy Immuta. This section guides you through configuring the IEHC to finish and prepare your installation for a production environment.

Verify artifacts hosted on the ocir.immuta.com OCI registry.

Configure to complete your installation and access your Immuta application.

Configure TLS termination for an Ingress resource.

Tips when installing Immuta without internet access.

Configure pulling images from a private registry.

Follow these best practices when deploying Immuta in your production environment.

Configure an external key-value cache (such as Redis or Memcached) with the Immuta Enterprise Helm chart.

Update the credentials referenced in the Immuta Enterprise Helm chart.

Enable these legacy services for your deployment if they are required for your business use case:

If you are using any of the data platforms below, you must enable the query engine:
- Amazon Redshift
- Azure Synapse Analytics
- Google BigQuery
If you are using the legacy sensitive data discovery (SDD) feature, you must enable the query engine and fingerprint services.

TLS Configuration

This guide demonstrates how to configure TLS termination for an Ingress resource.

Kubernetes namespace

The following section(s) presume the Immuta Enterprise Helm chart was deployed into namespace immuta and that the current namespace is immuta.

Prerequisite

The Ingress configuration must be completed before proceeding.

Ingress-NGINX Controller

Edit immuta-values.yaml to include the following Helm values.

secure:
  ingress:
    hostname: <immuta-fqdn>
    annotations:
      nginx.ingress.kubernetes.io/auth-tls-secret: <namespace>/<secret-name>

Create a TLS secret from a given public/private PEM formatted key pair.

kubectl create secret tls <secret-name> --cert=path/to/tls.cert --key=path/to/tls.key

Perform a Helm upgrade to apply the changes made to immuta-values.yaml.

helm upgrade <release-name> oci://ocir.immuta.com/stable/immuta-enterprise --values immuta-values.yaml --version 2024.3.2

Refer to the Ingress-Nginx Controller documentation for further assistance.

GKE Ingress Controller

Edit immuta-values.yaml to include the following Helm values.

secure:
  ingress:
    hostname: <immuta-fqdn>
    annotations:
      ingress.gcp.kubernetes.io/pre-shared-cert: <certificate-name>

Perform a Helm upgrade to apply the changes made to immuta-values.yaml.

helm upgrade <release-name> oci://ocir.immuta.com/stable/immuta-enterprise --values immuta-values.yaml --version 2024.3.2

Refer to the GKE Ingress Controller documentation for further assistance.

AWS Load Balancer Controller

Edit immuta-values.yaml to include the following Helm values.

secure:
  ingress:
    hostname: <immuta-fqdn>
    annotations:
      alb.ingress.kubernetes.io/certificate-arn: <certificate-arn>

Perform a Helm upgrade to apply the changes made to immuta-values.yaml.

helm upgrade <release-name> oci://ocir.immuta.com/stable/immuta-enterprise --values immuta-values.yaml --version 2024.3.2

Refer to the AWS Load Balancer Controller documentation for further assistance.

AKS Application Gateway Ingress Controller

Edit immuta-values.yaml to include the following Helm values.

secure:
  ingress:
    hostname: <immuta-fqdn>
    annotations:
      appgw.ingress.kubernetes.io/appgw-ssl-certificate: <certificate-name>

Perform a Helm upgrade to apply the changes made to immuta-values.yaml.

helm upgrade <release-name> oci://ocir.immuta.com/stable/immuta-enterprise --values immuta-values.yaml --version 2024.3.2

Refer to the Application Gateway Ingress Controller documentation for further assistance.

Traefik

Edit immuta-values.yaml to include the following Helm values.

secure:
  ingress:
    annotations:
      traefik.ingress.kubernetes.io/router.tls: "true"
    hostname: <immuta-fqdn>
    tls: true
    # If left unset the TLS secret name defaults to <hostname>-tls
    secretName: <secret-name>

Create a TLS secret from a given public/private PEM formatted key pair.

kubectl create secret tls <secret-name> --cert=path/to/tls.cert --key=path/to/tls.key

Perform a Helm upgrade to apply the changes made to immuta-values.yaml.

helm upgrade <release-name> oci://ocir.immuta.com/stable/immuta-enterprise --values immuta-values.yaml --version 2024.3.2

Refer to the Traefik documentation for further assistance.

Cosign Verification

This guide demonstrates how to verify signed artifacts (i.e., container images, Helm charts) hosted on ocir.immuta.com using Cosign from Sigstore.

Cosign installation

This guide utilizes the cosign command to verify artifacts; ensure it's installed before proceeding. Refer to the Cosign documentation for further assistance.

Download public key

The provided key is used to sign the Helm chart and container images.

Identify container images

A DIGESTS.md markdown file comes bundled in the Helm chart and contains a comprehensive list of images and digests referenced. To view the file, follow these steps:

Download and extract the Helm chart into the working directory.

helm pull oci://ocir.immuta.com/stable/immuta-enterprise --destination . --untar --version 2024.3.2

Open file immuta-enterprise/DIGESTS.md

Verify signature

Verify an artifact's signature by referencing Immuta's public key.

cosign verify --key ./immuta-cosign.pub <image>

Production Best Practices

This guide highlights best practices when deploying Immuta in a production environment.

Kubernetes namespace

The following section(s) presume the Immuta Enterprise Helm chart was deployed into namespace immuta and that the current namespace is immuta.

Helm values

Back up or source control your immuta-values.yaml Helm values file.

Kubernetes resource requests and limits

Assign memory resource limits to pods.

Edit Helm values

Edit immuta-values.yaml to include the following recommended resource requests and limits for most Immuta deployments.

Increase replica count to 3 on web and backgroundWorker for large deployments.

audit:
  worker:
    replicaCount: 1
    resources:
      requests:
        cpu: 1000m
        memory: 1024Mi
      limits:
        cpu: 1000m
        memory: 2048Mi  
  deployment:
    replicaCount: 1
    resources:
      requests:
        cpu: 1000m
        memory: 4096Mi
      limits:
        cpu: 3000m
        memory: 8192Mi
secure:
  backgroundWorker:
    replicaCount: 2
    resources:
      requests:
        cpu: 1000m
        memory: 4096Mi
      limits:
        cpu: 4000m
        memory: 4096Mi  
  web:
    replicaCount: 2 
    resources:
      requests:
        cpu: 1000m
        memory: 4096Mi
      limits:
        cpu: 4000m
        memory: 4096Mi
discover:
  deployment:
    replicaCount: 1
    resources:
      requests:
        cpu: 500m
        memory: 4096Mi
      limits:
        cpu: 3000m
        memory: 4096Mi
cache:
  deployment:
    replicaCount: 1
    resources:
      requests:
        cpu: 500m
        memory: 512Mi
      limits:
        cpu: 1000m
        memory: 512Mi

Kubernetes secrets

Use Kubernetes secrets in the immuta-values.yaml file instead of passwords and tokens. The following section demonstrates how to create a secret and reference it in the Helm values file.

Create secret

Create a file named secret-data.env with the following content.

# audit
ELASTICSEARCH_USERNAME=<elasticsearch-username>
ELASTICSEARCH_PASSWORD=<elasticsearch-password>

# PostgreSQL connection string used by audit for the metadata database
#   postgresql://<user>:<password>@<postgres-fqdn>:5432/<database>?schema=audit
#
# More info
#   https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING
DATABASE_CONNECTION_STRING=postgresql://immuta:<postgres-password>@<postgres-fqdn>:5432/immuta?schema=audit

# secure
IMMUTA_DATABASES_IMMUTA_CONNECTIONS_IMMUTADB_PASSWORD=<postgres-password>

Create secret named immuta-secret from file secret-data.env.

kubectl create secret generic immuta-secret --from-env-file=secret-data.env

Delete file secret-data.env, as it's no longer needed.
```
rm -i secret-data.env
```

Edit Helm values

Edit immuta-values.yaml to include the following Helm values.

audit:
  deployment:
    existingSecret: immuta-secret
  export:
    cronJob:
      existingSecret: immuta-secret

secure:
  existingSecret:
    name: immuta-secret
    # Optional. Map expected keys with keys in existing secret
    # keyMapping: {}

Remove any sensitive key-value pairs from the immuta-values.yaml Helm values that were made redundant after the secret was created.

Apply Helm values

Perform a Helm upgrade to apply the changes made to immuta-values.yaml.

helm upgrade <release-name> oci://ocir.immuta.com/stable/immuta-enterprise --values immuta-values.yaml --version 2024.3.2

Rotating Credentials

This guide demonstrates how to update credentials referenced in the Immuta Enterprise Helm chart (IEHC).

Kubernetes namespace

The following section(s) presume the IEHC was deployed into namespace immuta and that the current namespace is immuta.

Kubernetes secrets

Edit secrets

Using an alternative editor

Set environment variable KUBE_EDITOR to specify an alternative text editor.

Validate that secret immuta-secret exists in the current namespace.
```
kubectl get secret/immuta-secret
```
Edit secret immuta-secret in place.
```
kubectl edit secret/immuta-secret
```
Edit secret immuta-legacy-secret in place. Skip this step if the legacy query engine and fingerprint services are disabled (the default).
```
kubectl edit secret/immuta-legacy-secret
```

Restart pods.

kubectl rollout restart deployment --all --selector "app.kubernetes.io/component=audit,app.kubernetes.io/component=secure"

Legacy query engine

Considerations when using the legacy query engine

The following section is only necessary if the legacy query engine and fingerprint services have been enabled.

Validate that secret immuta-legacy-secret exists in the current namespace.
```
kubectl get secret/immuta-legacy-secret
```

Get the query engine replica count, this value will be referenced in subsequent step(s).

kubectl get statefulset --all --selector "app.kubernetes.io/component=query-engine" --output template='{{ .status.replicas }}'

Scale the replica count down to 1.

kubectl scale statefulset --all --replicas 1 --selector "app.kubernetes.io/component=query-engine"

Get the query engine pod name, this value will be referenced in subsequent step(s).
```
kubectl get pod --selector "app.kubernetes.io/component=query-engine"
```

Update the placeholder value with a query engine superuser password.

kubectl exec pod/<query-engine-pod-name> -- \
    psql -d immuta -c \
    "ALTER USER postgres WITH ENCRYPTED PASSWORD '<new-patroni-superuser-password>'"

Update the placeholder value with a query engine replication password.

kubectl exec pod/<query-engine-pod-name> -- \
    psql -d immuta -c \
    "ALTER USER replicator WITH ENCRYPTED PASSWORD '<new-patroni-replication-password>'"

Update the placeholder value with a query engine feature password.

kubectl exec pod/<query-engine-pod-name> -- \
    psql -d immuta -c \
    "ALTER USER feature_service WITH ENCRYPTED PASSWORD '<new-immuta-feature-password>'"

Scale the replica count back up to the previous value by updating the placeholder value.

kubectl scale statefulset --all --replicas <query-engine-previous-replica-count> --selector "app.kubernetes.io/component=query-engine"

Apply Helm values

Update credentials in the immuta-values.yaml file.
Perform a Helm upgrade to apply the changes made to immuta-values.yaml. Update the placeholder value with your own release name.
```
helm upgrade <release-name> oci://ocir.immuta.com/stable/immuta-enterprise --values immuta-values.yaml --version 2024.3.2
```

External Cache Configuration

This guide demonstrates how to configure an external key-value cache (such as Redis or Memcached) with the Immuta Enterprise Helm chart (IEHC).

Kubernetes namespace

The following section(s) presume the IEHC was deployed into namespace immuta and that the current namespace is immuta.

Prerequisite

The Immuta in production guide must be completed before proceeding.

Redis

Edit secret immuta-secret that was created in the Immuta in production guide.
```
kubectl edit secret/immuta-secret
```
Add key-value IMMUTA_SERVER_CACHE_PROVIDER_OPTIONS_PASSWORD=<cache-password>.

Edit Helm values

Edit the immuta-values.yaml file to include the relevant Helm values listed below. Update all placeholder values with your own values.

Redis

TLS configuration

TLS must be configured both client-side and server-side. The following Helm values demonstrate connecting to Redis with TLS enabled.

cache:
  enabled: false

secure:
  extraConfig:
    server:
      cache:
        provider:
          constructor: catbox-redis
          options:
            host: <redis-fqdn>
            port: <port>
            # Setting options.tls to an empty dict enables TLS without configuring any other options.
            tls: {}

            # Dict representation of TLS config options json-object for package ioredis
            # https://github.com/redis/ioredis
            #
            # tls:
            #   ca:
            #   key:
            #   cert:

  extraEnvVars:
  - name: IMMUTA_SERVER_CACHE_PROVIDER_OPTIONS_PASSWORD
    valueFrom:
      secretKeyRef:
        key: IMMUTA_SERVER_CACHE_PROVIDER_OPTIONS_PASSWORD
        name: immuta-secret

Memcached

cache:
  enabled: false

secure:
  extraConfig:
    server:
      cache:
        provider:
          constructor: catbox-memcached
          options:
            host: <memcached-fqdn>
            port: <port>

Apply Helm values

Perform a Helm upgrade to apply the changes made to immuta-values.yaml.

helm upgrade <release-name> oci://ocir.immuta.com/stable/immuta-enterprise --values immuta-values.yaml --version 2024.3.2

Enabling Legacy Query Engine and Fingerprint

The query engine and fingerprint services are no longer installed by default. This guide demonstrates how to enable the query engine and fingerprint services using the Immuta Enterprise Helm chart (IEHC).

If you are using any of the data platforms below, you must enable the query engine:

Amazon Redshift
Azure Synapse Analytics
Google BigQuery
Any legacy database

If you are using the legacy sensitive data discovery (SDD) feature, you must enable the query engine and fingerprint services.

Kubernetes namespace

The following section(s) presume the IEHC was deployed into namespace immuta, and that the current namespace is immuta.

Prerequisites

The Immuta in production guide must be completed before proceeding.
Validate that secret immuta-secret exists in the current namespace.
```
kubectl get secret/immuta-secret
```

Create Kubernetes secret

Create a file named secret-data.env with the following content.

# query-engine
IMMUTA_FEATURE_PASSWORD=<immuta-feature-password>
PATRONI_SUPERUSER_PASSWORD=<patroni-superuser-password>
PATRONI_REPLICATION_PASSWORD=<patroni-replication-password>
PATRONI_RESTAPI_PASSWORD=<patroni-api-password>

Create secret named immuta-legacy-secret from file secret-data.env

kubectl create secret generic immuta-legacy-secret --from-env-file=secret-data.env

Delete file secret-data.env, as it's no longer needed.
```
rm -i secret-data.env
```

Edit Helm values

Edit the immuta-values.yaml file to include the following Helm values.

legacy:
  enabled: true

  queryEngine:
    statefulset:
      extraEnvVars:
      - name: IMMUTA_FEATURE_PASSWORD
        valueFrom:
          secretKeyRef:
            name: immuta-legacy-secret
            key: IMMUTA_FEATURE_PASSWORD
      - name: PATRONI_SUPERUSER_PASSWORD
        valueFrom:
          secretKeyRef:
            name: immuta-legacy-secret
            key: PATRONI_SUPERUSER_PASSWORD
      - name: PATRONI_REPLICATION_PASSWORD
        valueFrom:
          secretKeyRef:
            name: immuta-legacy-secret
            key: PATRONI_REPLICATION_PASSWORD
      - name: PATRONI_RESTAPI_PASSWORD
        valueFrom:
          secretKeyRef:
            name: immuta-legacy-secret
            key: PATRONI_RESTAPI_PASSWORD

    postgres:
      # Query Engine feature user
      # Instead use queryEngine.statefulset.extraEnvVars[].name[IMMUTA_FEATURE_PASSWORD]
      # password: <immuta-feature-password>

      # Query Engine superuser user
      # Instead use queryEngine.statefulset.extraEnvVars[].name[PATRONI_SUPERUSER_PASSWORD]
      # superuserPassword: <patroni-superuser-password>

      # Query Engine replication user
      # Instead use queryEngine.statefulset.extraEnvVars[].name[PATRONI_REPLICATION_PASSWORD]
      # replicationPassword: <patroni-replication-password>

      # Query Engine patroni api user
      # Instead use queryEngine.statefulset.extraEnvVars[].name[PATRONI_RESTAPI_PASSWORD]
      # patroniApiPassword: <patroni-api-password>
    immutaSecurity:
      # Each Kubernetes Service has a DNS record associated with it. See: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
      # The anatomy of a domain name is as followed:
      #   <service>.<namespace>.svc.<cluster-domain>
      #
      # Where the default cluster domain is: cluster.local
      authEndpoint: "http://immuta-secure.immuta.svc.cluster.local:8823"

secure:
  extraEnvVars:
  - name: IMMUTA_DATABASES_IMMUTA_CONNECTIONS_FEATURESTOREDB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: immuta-legacy-secret
        key: IMMUTA_FEATURE_PASSWORD

  extraConfig:
    databases:
      immuta:
        connections:
          featureStoreDb:
            # Each Kubernetes Service has a DNS record associated with it. See: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
            # The anatomy of a domain name is as followed:
            #   <service>.<namespace>.svc.<cluster-domain>
            #
            # Where the default cluster domain is: cluster.local
            host: "immuta-legacy-query-engine-service.immuta.svc.cluster.local"
            port: 5432
            ssl: false
            # Query Engine feature user
            # Instead use secure.extraEnvVars[].name[IMMUTA_DATABASES_IMMUTA_CONNECTIONS_FEATURESTOREDB_PASSWORD]
            # password: <immuta-feature-password>
    disableFeatureStore: false
    fingerprints:
      # Each Kubernetes Service has a DNS record associated with it. See: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
      # The anatomy of a domain name is as follows:
      #   <service>.<namespace>.svc.<cluster-domain>
      #
      # Where the default cluster domain is: cluster.local
      uri: "http://immuta-legacy-fingerprint-service.immuta.svc.cluster.local:5001/"
      queryEngineHost: "immuta-legacy-query-engine-service.immuta.svc.cluster.local"
      queryEnginePort: 5432

Update all placeholder values in the immuta-values.yaml file.

Apply Helm values

Perform a Helm upgrade to apply the changes made to immuta-values.yaml.

helm upgrade <release-name> oci://ocir.immuta.com/stable/immuta-enterprise --values immuta-values.yaml --version 2024.3.2

Private Container Registries

This guide demonstrates how to configure a private container registry with the Immuta Enterprise Helm chart (IEHC).

Image availability

This guide assumes that you have already copied all Immuta container images to your private registry. The process of copying images to a private registry can vary significantly depending on your specific environment and tools and is therefore outside the scope of this document.

Helm values

Image repository overrides

Each image.repository field defined in the default Helm values must be overridden. For the purposes of this guide, only the configuration for Secure is shown.

Examine the default Helm values in the chart; this will include all relevant values required to override the registry and images.
Edit the immuta-values.yaml to include the following Helm values. Update all with your own values.

Air-Gapped Environments

This guide demonstrates how to download and package the Immuta Enterprise Helm chart and its dependencies for consumption on a separate network with no internet access.

Prerequisite

Skopeo installation

This guide utilizes the skopeo command to copy container images; ensure it's installed before proceeding. Refer to the for further assistance.

Checklist

Skopeo

Skopeo is authenticated with Immuta's registry on the networked machine.
Skopeo is authenticated with the private registry on the air-gapped machine.

Helm

Helm is authenticated with Immuta's registry on the networked machine.

Download artifacts

This section demonstrates how to download the Helm chart and container images to your local machine. These artifacts will be packaged and transferred to the air-gapped environment later.

Upon completion of these steps, the saved artifacts can be found in local directory offline-kit.

Create a directory named offline-kit.
Download the Helm chart into directory offline-kit.
Extract file DIGESTS.md from the Helm chart archive.
Open file ./offline-kit/DIGESTS.md. This file includes the name and digest of every container image referenced by the Helm chart.
Download each image listed in file DIGESTS.md using . Each image will be saved to directory offline-kit with the filename<name>-<tag>.tar.

Transfer artifacts

This section demonstrates how to push the previously archived container images to a private registry that's accessible from within your air-gapped environment.

The exact process for transferring files into an air-gapped network can vary significantly depending on your specific security policies and infrastructure.

Transfer directory offline-kit (created in the previous section) onto a machine that's within your air-gapped environment.

Chart installation

A Helm chart can be referenced from a local file path, instead of remotely if desired. It is not necessary to reference it remotely. When referring to documentation, substitute any references to oci://ocir.immuta.com/stable/immuta-enterprise with the path to the unarchived (.tgz) chart file.

Upgrade

Introduced in 2024.2, the Immuta Enterprise Helm chart (IEHC) is an entirely new Helm chart used to deploy Immuta. Unlike the previous Immuta Helm chart (IHC), the IEHC shares the same version as the Immuta product. Each version of the chart supports a singular version of Immuta. Upgrading the Immuta version now entails upgrading the underlying Helm chart. Failure to do so will lead to an unsupported configuration.

Chart name

Common name

Immuta versions

Registry

Description

Helm chart deprecation notice

As of Immuta version 2024.2, the IHC has been deprecated in favor of the IEHC. The immuta-values.yaml Helm values files are not cross-compatible.

Preferred

Upgrading from Immuta 2024.1.x or older

If upgrading from 2024.1.x it is first necessary to , as these versions were all installed using the legacy IHC. This migration process will include upgrading Immuta.

): Upgrade from v2024.2 LTS to v2024.3 using the Immuta Enterprise Helm chart.

Legacy

: Upgrade from v2024.2 LTS to v2024.3 using the legacy Immuta Helm chart.

Migrating to the New Helm Chart

This guide demonstrates how to upgrade an existing Immuta deployment installed with the older Immuta Helm chart (IHC) to v2024.2 LTS using the Immuta Enterprise Helm chart (IEHC).

Helm chart deprecation notice

As of Immuta version 2024.2, the IHC has been deprecated in favor of the IEHC. Their respective immuta-values.yaml Helm values files are not compatible.

Prerequisites

Create a PostgreSQL database

The PostgreSQL instance has been provisioned and is actively running.
The PostgreSQL instance's hostname/FQDN is .
The PostgreSQL instance is .

For additional information, consult the Deployment requirements.

Validate the Helm release

Fetch the metadata for the Helm release associated with Immuta.
Review the output from the previous step and verify the following:
- The Immuta version (appVersion) is
  - The last LTS (2022.5.x) or 2024.1 or newer
  - Less than 2024.2
- The Immuta Helm chart (version) is greater than or equal to 4.13.5
- The Immuta Helm chart name (chart) is immuta
If any of the criteria is not met, it's first necessary to perform a .

Metadata database

The new IEHC no longer supports deploying a Metadata database (PostgreSQL) inside the Kubernetes cluster. Before transitioning to the new IEHC, it's first necessary to externalize the Metadata database.

Built-in

The following demonstrates how to take a database backup and import the data into each cloud provider's managed PostgreSQL service.

Create backup of old database

Get the metadata database pod name.
Spawn a shell inside the running metadata database pod.
Perform a database backup.
Type exit, and then press Enter to exit the shell prompt.
Copy file bometadata.dump from the pod to the host's working directory.

Setup new database

Create a pod named immuta-setup-db and spawn a shell.
Connect to the new PostgreSQL database as a superuser. Depending on the cloud provider, the default superuser name (postgres) might differ.
Create immuta, temporal, and temporal_visiblity databases and an immuta role.
Type \q, and then press Enter to exit the psql prompt.
Authenticate as the immuta user and create the pgcrypto extension.
Type \q, and then press Enter to exit the psql prompt.

Restore backup to new database

Create a pod named immuta-restore-db and spawn a shell.
Copy file bometadata.dump from the host's working directory to pod immuta-restore-db.
Spawn a shell inside pod immuta-restore-db.
Perform a database restore while authenticated as role immuta. Refer to the value substituted for <postgres-password> when prompted to enter a password.
Type exit, and then press Enter to exit the shell prompt.
Delete pod immuta-restore-db that was previously created.

External

No additional work is required. The existing database can be reused with the new IEHC.

Helm values

Helm values file compatibility

The immuta-values.yaml Helm values file used by the IHC is not compatible with the new IEHC.

Rename the existing immuta-values.yaml Helm values file used by the IHC.

Upgrading (IEHC)

This guide demonstrates how to upgrade an existing Immuta deployment installed with the Immuta Enterprise Helm chart (IEHC) to v2024.3.

Introduced in v2024.3

Temporal

Requirements

Starting in IEHC 2024.3, a Temporal server is included in the chart and requires two databases to store state. You can expand the existing PostgreSQL database in use for Immuta by creating Temporal databases like so:

These database names are the default values for the Temporal sub-chart but can be set (if necessary) with the following values:

Enabling Temporal

To enable the Temporal deployment, set the following values. Include the tls settings if using a Cloud database that requires TLS:

Helm values

To improve the experience using the IEHC, two Helm value changes have been introduced. Before deploying the IEHC 2024.3.x, you must perform the following Helm value changes:

PostgreSQL configuration

IEHC 2024.3.x adds support for global and component-level PostgreSQL connection details. This means you only need to specify the PostgreSQL connection information once in the global scope and apply overrides (if necessary) at a component level.

If you installed IEHC 2024.2 LTS using our install guides, your immuta-values.yaml file probably looks something like this to configure your PostgreSQL connection for multiple components:

Now, with PostgreSQL configuration in the global scope, your immuta-values.yaml file can look like this to specify the PostgreSQL connection:

Feature flags

Feature flags have moved from environment variables IEHC 2024.3.x as well. You may now set feature flags globally, and then the IEHC will properly configure all applications for you. Migrate all feature flags from secure.extraEnvVars to global.featureFlags.

If you fail to migrate the values from secure.extraEnvVars to global.featureFlags , then Helm will display warnings similar to below:

Upgrade Immuta

After updating your immuta-values.yaml file to include any of the changes for the updates above, you can upgrade Immuta with the following command:

Upgrading (IHC)

This guide demonstrates how to upgrade an existing Immuta deployment installed with the Immuta Helm chart (IHC) to v2024.3.

Helm chart deprecation notice

As of Immuta version 2024.2, the IHC has been deprecated in favor of the IEHC. Their respective immuta-values.yaml Helm values files are not compatible.

Prerequisites

Checklist

The Immuta Helm chart (IHC) has been updated to the latest version.

Upgrade Immuta

Edit immuta-values.yaml to include the following Helm values.
Perform a to apply the changes made to immuta-values.yaml.

How-to Guides

Integration Settings

Reference Guides

How-to Guides

Reference Guides

Configuration Settings

Databricks Spark Cluster Policies

How-to Guides

Reference Guides

Red Hat OpenShift

This is a guide on how to deploy Immuta on OpenShift.

Prerequisites

The following cloud-managed services must be provisioned and running before proceeding:

PostgreSQL
(Optional) Elasticsearch/OpenSearch

Feature availability

If deployed without Elasticsearch/OpenSearch, several core services and features will be unavailable. See the Deployment requirements page for details.

Checklist

This checklist outlines the necessary prerequisites for successfully deploying Immuta.

Credentials

You have the credentials needed to access the ocir.immuta.com OCI registry. These can be viewed in your user profile at support.immuta.com.

PostgreSQL

The PostgreSQL instance's hostname/FQDN is resolvable from within the Kubernetes cluster.
The PostgreSQL instance is accepting connections.
You have a PostgreSQL user account with the necessary privileges to create databases, extensions, and roles.

Elasticsearch

The Elasticsearch instance's hostname/FQDN is resolvable from within the Kubernetes cluster.
The Elasticsearch instance is accepting connections.

Setup

Helm

Authenticate with OCI registry

echo <token> | helm registry login --password-stdin --username <username> ocir.immuta.com

Kubernetes

Creating a dedicated namespace ensures a logically isolated environment for your Immuta deployment, preventing resource conflicts with other applications.

Create project

Create an OpenShift project named immuta.
```
oc new-project immuta
```
Get the UID range allocated to the project. Each running container's UID must fall within this range. This value will be referenced later on.
```
oc get project immuta --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
```
Get the GID range allocated to the project. Each running container's GID must fall within this range. This value will be referenced later on.
```
oc get project immuta --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
```
Switch to project immuta.
```
oc project immuta
```

Create registry secret

Create a container registry pull secret. Contact your Immuta representative to obtain credentials to authenticate with ocir.immuta.com.

oc create secret docker-registry immuta-oci-registry \
    --docker-server=https://ocir.immuta.com \
    --docker-username="<username>" \
    --docker-password="<token>" \
    --docker-email=support@immuta.com

PostgreSQL

Connecting a client

There are numerous ways to connect to a PostgreSQL database. This step demonstrates how to connect with psql by creating an ephemeral Kubernetes pod.

Connect to the database

oc run pgclient \
    --stdin \
    --tty \
    --rm \
    --image docker.io/bitnami/postgresql -- \
    psql --host <postgres-fqdn> --username <postgres-user> --port 5432 --password

Create Role

To ensure Temporal can successfully manage its schema, a pre-defined administrator role must be granted. The role name varies depending on the cloud-managed service:

Amazon RDS: rds_superuser
Azure Database: azure_pg_admin
Google Cloud SQL: cloudsqlsuperuser

Create the immuta role.

CREATE ROLE immuta with LOGIN ENCRYPTED PASSWORD '<postgres-password>';
ALTER ROLE immuta SET search_path TO bometadata,public;

Grant administrator privileges to the immuta role. Upon successfully completing this installation guide, you can optionally revoke this role grant.
```
GRANT <admin-role> TO immuta;
```

Create databases

Create databases.

CREATE DATABASE immuta OWNER immuta;
CREATE DATABASE temporal OWNER immuta;
CREATE DATABASE temporal_visibility OWNER immuta;

Grant role immuta additional privileges. Refer to the PostgreSQL documentation for further details on database roles and privileges.

GRANT ALL ON DATABASE immuta TO immuta;
GRANT ALL ON DATABASE temporal TO immuta;
GRANT ALL ON DATABASE temporal_visibility TO immuta;

Configure the immuta database.
```
\c immuta
CREATE EXTENSION pgcrypto;
```

Configure the temporal database.

\c temporal
GRANT CREATE ON SCHEMA public TO immuta;

Configure the temporal_visibility database.

\c temporal_visibility
GRANT CREATE ON SCHEMA public TO immuta;
CREATE EXTENSION btree_gin;

Exit the interactive prompt. Type \q, and then press Enter.

Install Immuta

This section demonstrates how to deploy Immuta using the Immuta Enterprise Helm chart once the prerequisite cloud-managed services are configured.

Why disable Ingress?

In OpenShift, Ingress resources are managed by OpenShift Routes. These routes provide a more integrated and streamlined way to handle external access to your applications. To avoid conflicts and ensure proper functionality, it's necessary to disable the pre-defined Ingress resource in the Helm chart.

immuta-values.yaml

global:
  imageRegistry: ocir.immuta.com
  imagePullSecrets:
    - name: immuta-oci-registry
  postgresql:
    host: <postgres-fqdn>
    port: 5432
    username: immuta
    password: <postgres-password>

audit:
  config:
    elasticsearchEndpoint: http://es-db-elasticsearch.immuta.svc.cluster.local:9200
    elasticsearchUsername: <elasticsearch-username>
    elasticsearchPassword: <elasticsearch-password>
  postgresql:
    database: immuta 

  deployment:
    podSecurityContext:
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
      runAsUser: <user-id>
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
      runAsGroup: <group-id>
      seccompProfile:
        type: RuntimeDefault
      
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
  worker:
    podSecurityContext:  
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
      runAsUser: <user-id>
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
      runAsGroup: <group-id>
      seccompProfile:
        type: RuntimeDefault
      
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL 

discover:
  deployment:
    podSecurityContext:
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
      runAsUser: <user-id>
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
      runAsGroup: <group-id>
      seccompProfile:
        type: RuntimeDefault
      
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL

secure:
  ingress:
    enabled: false

  postgresql:
    database: immuta
    ssl: false

  web:
    podSecurityContext:
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
      runAsUser: <user-id>
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
      runAsGroup: <group-id>
      seccompProfile:
        type: RuntimeDefault
  
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL

  backgroundWorker:
    podSecurityContext:
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
      runAsUser: <user-id>
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
      runAsGroup: <group-id>
      seccompProfile:
        type: RuntimeDefault
      
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
  
temporal:
  enabled: true
  schema:
    createDatabase:
      enabled: false
  server:
    podSecurityContext:
        # A number that is within the project range:
        #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
        runAsUser: <user-id>
        # A number that is within the project range:
        #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
        runAsGroup: <group-id>
        seccompProfile:
          type: RuntimeDefault
    config:
      persistence:
        default:
          sql:
            database: temporal
            tls:
              enabled: true
        visibility:
          sql:
            database: temporal_visibility
            tls:
              enabled: true
    frontend:
      containerSecurityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
    history:
      containerSecurityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
    matching:
      containerSecurityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
    worker:
      containerSecurityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
  schema:
    podSecurityContext:
        # A number that is within the project range:
        #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
        runAsUser: <user-id>
        # A number that is within the project range:
        #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
        runAsGroup: <group-id>
        seccompProfile:
          type: RuntimeDefault
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
  proxy:
    deployment:
      podSecurityContext:
        # A number that is within the project range:
        #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
        runAsUser: <user-id>
        # A number that is within the project range:
        #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
        runAsGroup: <group-id>
        seccompProfile:
          type: RuntimeDefault
      containerSecurityContext:
        enabled: true
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL

Feature availability

If deployed without Elasticsearch/OpenSearch, several core services and features will be unavailable. See the Deployment requirements page for details.

immuta-values.yaml

global:
  imageRegistry: ocir.immuta.com
  imagePullSecrets:
    - name: immuta-oci-registry
  featureFlags:
    AuditService: false
    detect: false
    auditLegacyViewHide: false
  postgresql:
    host: <postgres-fqdn>
    port: 5432
    username: immuta
    password: <postgres-password>

audit:
  enabled: false

discover:
  deployment:
    podSecurityContext:
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
      runAsUser: <user-id>
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
      runAsGroup: <group-id>
      seccompProfile:
        type: RuntimeDefault

    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL

secure:
  ingress:
    enabled: false

  postgresql:
    database: immuta
    ssl: true

  web:
    podSecurityContext:
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
      runAsUser: <user-id>
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
      runAsGroup: <group-id>
      seccompProfile:
        type: RuntimeDefault

    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL

  backgroundWorker:
    podSecurityContext:
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
      runAsUser: <user-id>
      # A number that is within the project range:
      #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
      runAsGroup: <group-id>
      seccompProfile:
        type: RuntimeDefault

    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL

temporal:
  enabled: true
  schema:
    createDatabase:
      enabled: false
  server:
    podSecurityContext:
        # A number that is within the project range:
        #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
        runAsUser: <user-id>
        # A number that is within the project range:
        #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
        runAsGroup: <group-id>
        seccompProfile:
          type: RuntimeDefault
    config:
      persistence:
        default:
          sql:
            database: temporal
            tls:
              enabled: true
        visibility:
          sql:
            database: temporal_visibility
            tls:
              enabled: true
    frontend:
      containerSecurityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
    history:
      containerSecurityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
    matching:
      containerSecurityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
    worker:
      containerSecurityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
  schema:
    podSecurityContext:
        # A number that is within the project range:
        #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
        runAsUser: <user-id>
        # A number that is within the project range:
        #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
        runAsGroup: <group-id>
        seccompProfile:
          type: RuntimeDefault
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
  proxy:
    deployment:
      podSecurityContext:
        # A number that is within the project range:
        #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
        runAsUser: <user-id>
        # A number that is within the project range:
        #   oc get project <project-name> --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
        runAsGroup: <group-id>
        seccompProfile:
          type: RuntimeDefault
      containerSecurityContext:
        enabled: true
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL

Create a file named immuta-values.yaml with the above content, making sure to update all placeholder values.

Deploy Immuta.

helm install immuta oci://ocir.immuta.com/stable/immuta-enterprise \
    --values immuta-values.yaml \
    --version 2024.3.2

Wait for all pods to become ready.

oc wait --for=condition=Ready pods --all

Validation

Determine the name of the Secure service.

oc get service --selector "app.kubernetes.io/component=secure" --output name

Listen on local port 8080, forwarding TCP traffic to the Secure service's port named http.
```
oc port-forward <service-name> 8080:http
```
In a web browser, navigate to localhost:8080, to ensure the Immuta application loads.
Press Control+C to stop port forwarding.

Next steps

Configure Ingress for OpenShift (required).
Learn about best practices for running Immuta in production.

SUSE Rancher Kubernetes Engine (RKE2)

Considerations

For the purposes of this guide, the following state stores are deployed in Kubernetes using third-party Helm charts maintained by Bitnami:

Running production-grade stateful workloads (e.g., databases) in Kubernetes is difficult and heavily discouraged due to the following reasons.

Operational overhead: Managing PostgreSQL and Elasticsearch on Kubernetes requires expertise in deploying, maintaining, and scaling these databases and search engines effectively. This involves tasks like setting up monitoring, configuring backups, managing updates, and ensuring high availability. Cloud-managed services abstract much of this operational burden away, allowing teams to focus on application development rather than infrastructure management.
Resource allocation and scaling: Kubernetes requires careful resource allocation and scaling decisions to ensure that PostgreSQL and Elasticsearch have sufficient CPU, memory, and storage. Properly sizing these resources can be challenging and may require continuous adjustments as workload patterns change. Managed services typically handle this scaling transparently and can automatically adjust based on demand.
Data integrity and high availability: PostgreSQL and Elasticsearch deployments need robust strategies for data integrity and high availability. Kubernetes can facilitate high availability through pod replicas and distributed deployments, but ensuring data consistency and durability across database instances and search indexes requires careful consideration and often additional tooling.
Performance: Kubernetes networking and storage configurations can introduce performance overhead compared to native cloud services. For latency-sensitive applications or high-throughput workloads, these factors become critical in maintaining optimal performance.
Observability: Troubleshooting issues in a Kubernetes environment, especially related to database and search engine performance, can be complex. Managed services typically come with built-in monitoring, logging, and alerting capabilities tailored to the specific service, making it easier to identify and resolve issues.
Security and compliance: Kubernetes environments require careful attention to security best practices, including network policies, access controls, and encryption. Managed services often come pre-configured with security features and compliance certifications, reducing the burden on teams to implement and maintain these measures.

Prerequisites

Checklist

This checklist outlines the necessary prerequisites for successfully deploying Immuta.

Credentials

You have the credentials needed to access the ocir.immuta.com OCI registry. These can be viewed in your user profile at support.immuta.com.

Setup

Helm

Authenticate with OCI registry

echo <token> | helm registry login --password-stdin --username <username> ocir.immuta.com

Kubernetes

Creating a dedicated namespace ensures a logically isolated environment for your Immuta deployment, preventing resource conflicts with other applications.

Create namespace

Create a Kubernetes namespace named immuta.
```
kubectl create namespace immuta
```
Switch to namespace immuta. All subsequent kubectl commands will default to this namespace.
```
kubectl config set-context --current --namespace=immuta
```

Create registry secret

Create a container registry pull secret. Contact your Immuta representative to obtain credentials to authenticate with ocir.immuta.com.

kubectl create secret docker-registry immuta-oci-registry \
    --docker-server=https://ocir.immuta.com \
    --docker-username="<username>" \
    --docker-password="<token>" \
    --docker-email=support@immuta.com

Elasticsearch

Install Helm chart

Create a Helm values file named es-values.yaml with the following content:

es-values.yaml

```yaml
master:
    masterOnly: false
    replicaCount: 1

data:
    replicaCount: 0

coordinating:
    replicaCount: 0

ingest:
    replicaCount: 0
```

Deploy Elasticsearch.

helm install es-db oci://registry-1.docker.io/bitnamicharts/elasticsearch \
    --values es-values.yaml

Wait for all Elasticsearch pods to become ready.

kubectl wait --for=condition=Ready pods --selector "app.kubernetes.io/name=elasticsearch"

PostgreSQL

Install Helm chart

Create a Helm values file named pg-values.yaml.

pg-values.yaml

```yaml
primary:
  # The default 'nano' preset is insufficient and will lead to Out Of Memory (OOM) 
  # errors.
  #
  # Possible values: nano, micro, small, medium, large, xlarge, 2xlarge
  resourcesPreset: medium
  initdb:
    scripts:
      00_setup.sql: |
        CREATE ROLE immuta with LOGIN ENCRYPTED PASSWORD '<postgres-password>';
        ALTER ROLE immuta SET search_path TO bometadata,public;

        CREATE DATABASE temporal WITH OWNER immuta;
        CREATE DATABASE temporal_visibility WITH OWNER immuta;
        CREATE DATABASE immuta WITH OWNER immuta;

        GRANT ALL ON DATABASE temporal TO immuta;
        GRANT ALL ON DATABASE temporal_visibility TO immuta;
        GRANT ALL ON DATABASE immuta TO immuta;
```

Update all placeholder values in the pg-values.yaml file.

Deploy PostgreSQL.

helm install pg-db oci://registry-1.docker.io/bitnamicharts/postgresql \
    --values pg-values.yaml

Wait for all PostgreSQL pods to become ready.

kubectl wait --for=condition=Ready pods --selector "app.kubernetes.io/name=postgresql"

Configure databases

Determine the name of the PostgreSQL database pod. This will be referenced in a subsequent step.
```
kubectl get pod --selector "app.kubernetes.io/name=postgresql" --output name
```
Exec into the PostgreSQL database pod using psql.
```
kubectl exec --stdin --tty <database-pod> -- psql
```
Configure the immuta database.
```
\c immuta
CREATE EXTENSION pgcrypto;
```

Configure the temporal database.

\c temporal
GRANT CREATE ON SCHEMA public TO immuta;

Configure the temporal_visibility database.

\c temporal_visibility
GRANT CREATE ON SCHEMA public TO immuta;
CREATE EXTENSION btree_gin;

Exit the interactive prompt. Type \q, then press Enter.

Install Immuta

This section demonstrates how to deploy Immuta using the Immuta Enterprise Helm chart once the prerequisite cloud-managed services are configured.

immuta-values.yaml

global:
  imageRegistry: ocir.immuta.com
  imagePullSecrets:
    - name: immuta-oci-registry
  postgresql:
    # Each Kubernetes Service has a DNS record associated with it. See: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
    # The anatomy of a domain name is as follows:
    #   <service>.<namespace>.svc.<cluster-domain>
    #
    # Where the default cluster domain is: cluster.local
    host: pg-db-postgresql.immuta.svc.cluster.local
    port: 5432
    username: immuta
    password: <postgres-password>

audit:
  config:
    elasticsearchEndpoint: http://es-db-elasticsearch.immuta.svc.cluster.local:9200
    elasticsearchUsername: <elasticsearch-username>
    elasticsearchPassword: <elasticsearch-password>
  postgresql:
    database: immuta

secure:
  postgresql:
    database: immuta
    
temporal:
  enabled: true
  schema:
    createDatabase:
      enabled: false
  server:
    config:
      persistence:
        default:
          sql:
            database: temporal
        visibility:
          sql:
            database: temporal_visibility

Create a file named immuta-values.yaml with the above content, making sure to update all placeholder values.

Deploy Immuta.

helm install immuta oci://ocir.immuta.com/stable/immuta-enterprise \
    --values immuta-values.yaml \
    --version 2024.3.2

Wait for all pods to become ready.

kubectl wait --for=condition=Ready pods --all

Validation

Determine the name of the Secure service.

kubectl get service --selector "app.kubernetes.io/component=secure" --output name

Listen on local port 8080, forwarding TCP traffic to the Secure service's port named http.
```
kubectl port-forward <service-name> 8080:http
```
In a web browser, navigate to localhost:8080, to ensure the Immuta application loads.
Press Control+C to stop port forwarding.

Next steps

Configure Ingress for RKE2 (required).
Configure TLS certificates for RKE2.
Learn about best practices for running Immuta in production.

Snowflake Integration

Snowflake Enterprise Edition required

In this integration, Immuta manages access to Snowflake tables by administering Snowflake row access policies and column masking policies on those tables, allowing users to query tables directly in Snowflake while dynamic policies are enforced.

Like with all Immuta integrations, Immuta can inject its ABAC model into policy building and administration to remove policy management burden and significantly reduce role explosion.

Architecture

When an administrator configures the Snowflake integration with Immuta, Immuta creates an IMMUTA database and schemas (immuta_procedures, immuta_policies, and immuta_functions) within Snowflake to contain policy definitions and user entitlements. Immuta then creates a system role and gives that system account the following privileges:

APPLY MASKING POLICY
APPLY ROW ACCESS POLICY
ALL PRIVILEGES ON DATABASE "IMMUTA" WITH GRANT OPTION
ALL PRIVILEGES ON ALL SCHEMAS IN DATABASE "IMMUTA" WITH GRANT OPTION
USAGE ON FUTURE PROCEDURES IN SCHEMA "IMMUTA".immuta_procedures WITH GRANT OPTION
USAGE ON WAREHOUSE
OWNERSHIP ON SCHEMA "IMMUTA".immuta_policies TO ROLE "IMMUTA_SYSTEM" COPY CURRENT GRANTS
OWNERSHIP ON SCHEMA "IMMUTA".immuta_procedures TO ROLE "IMMUTA_SYSTEM" COPY CURRENT GRANTS
OWNERSHIP ON SCHEMA "IMMUTA".immuta_functions TO ROLE "IMMUTA_SYSTEM" COPY CURRENT GRANTS
OWNERSHIP ON SCHEMA "IMMUTA".public TO ROLE "IMMUTA_SYSTEM" COPY CURRENT GRANTS

Optional features, like automatic object tagging, native query auditing, etc., require additional permissions to be granted to the Immuta system account, are listed in the supported features section.

Policy enforcement

Snowflake is a policy push integration with Immuta. When Immuta users create policies, they are then pushed into the Immuta database within Snowflake; there, the Immuta system account applies Snowflake row access policies and column masking policies directly onto Snowflake tables. Changes in Immuta policies, user attributes, or data sources trigger webhooks that keep the Snowflake policies up-to-date.

For a user to query Immuta-protected data, they must meet two qualifications:

They must be subscribed to the Immuta data source.
They must be granted SELECT access on the table by the Snowflake object owner or automatically via the Snowflake table grants feature.

After a user has met these qualifications they can query Snowflake tables directly.

See the integration support matrix on the Data policy types reference guide for a list of supported data policy types in Snowflake.

Comply with column length and precision requirements in a Snowflake masking policy

When a user applies a masking policy to a Snowflake data source, Immuta truncates masked values to align with Snowflake column length (VARCHAR(X) types) and precision (NUMBER (X,Y) types) requirements.

Consider these columns in a data source that have the following masking policies applied:

Column A (VARCHAR(6)): Mask using hashing for everyone
Column B (VARCHAR(5)): Mask using a constant REDACTED for everyone
Column C (VARCHAR(6)): Mask by making null for everyone
Column D (NUMBER(3, 0)): Mask by rounding to the nearest 10 for everyone

Querying this data source in Snowflake would return the following values:

Hashing collisions

Hashing collisions are more likely to occur across or within Snowflake columns restricted to short lengths, since Immuta truncates the hashed value to the limit of the column. (Hashed values truncated to 5 characters have a higher risk of collision than hashed values truncated to 20 characters.) Therefore, avoid applying hashing policies to Snowflake columns with such restrictions.

For more details about Snowflake column length and precision requirements, see the Snowflake behavior change release documentation.

Query performance

When a policy is applied to a column, Immuta uses Snowflake memoizable functions to cache the result of the called function. Then, when a user queries a column that has that policy applied to it, Immuta uses that cached result to dramatically improve query performance.

Registering data sources

Register Snowflake data sources using a dedicated Snowflake role. Avoid using individual user accounts for data source onboarding. Instead, create a service account (Snowflake user account TYPE=SERVICE) with SELECT access for onboarding data sources. No policies will apply to that role, ensuring that your integration works with the following use cases:

Snowflake project workspaces: Snowflake workspaces generate static views with the credentials used to register the table as an Immuta data source. Those tables must be registered in Immuta by an excepted role so that policies applied to the backing tables are not applied to the project workspace views.
Using views and tables within Immuta: Because this integration uses Snowflake governance policies, users can register tables and views as Immuta data sources. However, if you want to register views and apply different policies to them than their backing tables, the owner of the view must be an excepted role; otherwise, the backing table’s policies will be applied to that view.

Snowflake bulk data source creation

Private preview: This feature is only available to select accounts. Reach out to your Immuta representative to enable this feature.

Bulk data source creation is the more efficient process when loading more than 5000 data sources from Snowflake and allows for data sources to be registered in Immuta before running sensitive data discovery or applying policies.

To use this feature, see the Bulk create Snowflake data sources guide.

Resource allocations

Based on performance tests that create 100,000 data sources, the following minimum resource allocations need to be applied to the appropriate pods in your Kubernetes environment for successful bulk data source creation.

Limitations

Performance gains are limited when enabling sensitive data discovery at the time of data source creation.
External catalog integrations are not recognized during bulk data source creation. Users must manually trigger a catalog sync for tags to appear on the data source through the data source's health check.

Excepted roles/users

Excepted roles and users are assigned when the integration is installed, and no policies will apply to these users' queries, despite any Immuta policies enforced on the tables they are querying. Credentials used to register a data source in Immuta will be automatically added to this excepted list for that Snowflake table. Consequently, roles and users added to this list and used to register data sources in Immuta should be limited to service accounts.

Immuta excludes the listed roles and users from policies by wrapping all policies in a CASE statement that will check if a user is acting under one of the listed usernames or roles. If a user is, then the policy will not be acted on the queried table. If the user is not, then the policy will be executed like normal. Immuta does not distinguish between role and username, so if you have a role and user with the exact same name, both the user and any user acting under that role will have full access to the data sources and no policies will be enforced for them.

Data flow

An Immuta application administrator configures the Snowflake integration and registers Snowflake warehouse and databases with Immuta.
Immuta creates a database inside the configured Snowflake warehouse that contains Immuta policy definitions and user entitlements.
A data owner registers Snowflake tables in Immuta as data sources.
If Snowflake tag ingestion was enabled during the configuration, Immuta uses the host provided in the configuration and ingests internal tags on Snowflake tables registered as Immuta data sources.
A data owner, data governor, or administrator creates or changes a policy or a user's attributes change in Immuta.
The Immuta web service calls a stored procedure that modifies the user entitlements or policies.
Immuta manages and applies Snowflake governance column and row access policies to Snowflake tables that are registered as Immuta data sources.
If Snowflake table grants is not enabled, Snowflake object owner or user with the global MANAGE GRANTS privilege grants SELECT privilege on relevant Snowflake tables to users. Note: Although they are GRANTed access, if they are not subscribed to the table via Immuta-authored policies, they will not see data.
A Snowflake user who is subscribed to the data source in Immuta queries the corresponding table directly in Snowflake and sees policy-enforced data.

Authentication methods

The Snowflake integration supports the following authentication methods to configure the integration and create data sources:

Username and password: Users can authenticate with their Snowflake username and password.
Key pair: Users can authenticate with a Snowflake key pair authentication.
Snowflake External OAuth: Users can authenticate with Snowflake External OAuth.

Snowflake External OAuth

Immuta's OAuth authentication method uses the Client Credentials Flow to integrate with Snowflake External OAuth. When a user configures the Snowflake integration or connects a Snowflake data source, Immuta uses the token credentials (obtained using a certificate or passing a client secret) to craft an authenticated access token to connect with Snowflake. This allows organizations that already use Snowflake External OAuth to use that secure authentication with Immuta.

Workflow

An Immuta application administrator configures the Snowflake integration or creates a data source.
Immuta creates a custom token and sends it to the authorization server.
The authorization server confirms the information sent from Immuta and issues an access token to Immuta.
Immuta sends the access token it received from the authorization server to Snowflake.
Snowflake authenticates the token and grants access to the requested resources from Immuta.
The integration is connected and users can query data.

Supported Snowflake feature

The Immuta Snowflake integration supports Snowflake external tables. However, you cannot add a masking policy to an external table column while creating the external table in Snowflake because masking policies cannot be attached to virtual columns.

Supported Immuta features

The Snowflake integration supports the Immuta features outlined below. Click the links provided for more details.

Immuta project workspaces: Users can have additional write access in their integration using project workspaces.
Tag ingestion: Immuta automatically ingests Snowflake object tags from your Snowflake instance and adds them to the appropriate data sources.
User impersonation: Native impersonation allows users to natively query data as another Immuta user. To enable native user impersonation, see the Integration user impersonation page.
Native query audit: Immuta audits queries run natively in Snowflake against Snowflake data registered as Immuta data sources.
Multiple Snowflake instances
Snowflake low row access policy mode: The Snowflake low row access policy mode improves query performance in Immuta's Snowflake integration by decreasing the number of Snowflake row access policies Immuta creates.
Snowflake table grants: This feature allows Immuta to manage privileges on your Snowflake tables and views according to the subscription policies on the corresponding Immuta data sources.

Immuta project workspaces

Immuta system account required Snowflake privileges

CREATE [OR REPLACE] PROCEDURE
DROP ROLE
REVOKE ROLE

Users can have additional write access in their integration using project workspaces. For more details, see the Snowflake project workspaces page.

Caveat

To use project workspaces with the Snowflake integration, the default role of the account used to create data sources in the project must be added to the "Excepted Roles/Users List." If the role is not added, you will not be able to query the equalized view using the project role in Snowflake.

Tag ingestion

You can enable Snowflake tag ingestion so that Immuta will ingest Snowflake object tags from your Snowflake instance into Immuta and add them to the appropriate data sources.

The Snowflake tags' key and value pairs will be reflected in Immuta as two levels: the key will be the top level and the value the second. As Snowflake tags are hierarchical, Snowflake tags applied to a database will also be applied to all of the schemas in that database, all of the tables within those schemas, and all of the columns within those tables. For example: If a database is tagged PII, all of the tables and columns in that database will also be tagged PII.

To enable Snowflake tag ingestion, see the Configure a Snowflake integration page.

Caveats

Snowflake has some natural data latency. If you manually refresh the governance page to see all tags created globally, users can experience a delay of up to two hours. However, if you run schema detection or a health check to find where those tags are applied, the delay will not occur because Immuta will only refresh tags for those specific tables.

Native query audit

Immuta system account required Snowflake privilege

IMPORTED PRIVILEGES ON DATABASE snowflake

Once this feature has been enabled with the Snowflake integration, Immuta will query Snowflake to retrieve user query histories. These histories provide audit records for queries against Snowflake data sources that are queried natively in Snowflake.

This process will happen automatically every hour by default but can be configured to a different frequency when configuring or editing the integration. Additionally, audit ingestion can be manually requested at any time from the Immuta audit page. When manually requested, it will only search for new queries that were created since the last native query that had been audited. The job is run in the background, so the new queries will not be immediately available.

For details about prompting these logs and the contents of these audit logs, see the Snowflake query audit logs page.

Multiple Snowflake instances

A user can configure multiple integrations of Snowflake to a single Immuta tenant and use them dynamically or with workspaces.

Caveats

There can only be one integration connection with Immuta per host.
The host of the data source must match the host of the integration for the view to be created.
Projects can only be configured to use one Snowflake host.

Limitations

If there are errors in generating or applying policies natively in Snowflake, the data source will be locked and only users on the excepted roles/users list and the credentials used to create the data source will be able to access the data.
Once a Snowflake integration is disabled in Immuta, the user must remove the access that was granted in Snowflake. If that access is not revoked, users will be able to access the raw table in Snowflake.
Migration must be done using the credentials and credential method (automatic or bootstrap) used to configure the integration.
When configuring one Snowflake instance with multiple Immuta tenants, the user or system account that enables the integration on the app settings page must be unique for each Immuta tenant.
A Snowflake table can only have one set of policies enforced at a given time, so creating multiple data sources pointing to the same table is not supported. If this is a use case you need to support, create views in Snowflake and expose those instead.
You cannot add a masking policy to an external table column while creating the external table because a masking policy cannot be attached to a virtual column.
If you create an Immuta data source from a Snowflake view created using a select * from query, Immuta column detection will not work as expected because Snowflake views are not automatically updated based on backing table changes. To remedy this, you can create views that have the specific columns you want or you can CREATE AND REPLACE the view in Snowflake whenever the backing table is updated and manually run the column detection job on the data source page.
If a user is created in Snowflake after that user is already registered in Immuta, Immuta does not grant usage on the per-user role automatically - meaning Immuta does not govern this user's access without manual intervention. If a Snowflake user is created after that user is registered in Immuta, the user account must be disabled and re-enabled to trigger a sync of Immuta policies to govern that user. Whenever possible, Snowflake users should be created before registering those users in Immuta.
Snowflake tables from imported databases are not supported. Instead, create a view of the table and register that view as a data source.

Custom WHERE clause limitations

The Immuta Snowflake integration uses Snowflake governance features to let users query data natively in Snowflake. This means that Immuta also inherits some Snowflake limitations using correlated subqueries with row access policies and column-level security. These limitations appear when writing custom WHERE policies, but do not remove the utility of row-level policies.

Requirements for a custom WHERE policy

All column names must be fully qualified: Any column names that are unqualified (i.e., just the column name) will default to a column of the data source the policy is being applied to (if one matches the name).
The Immuta system account must have SELECT privileges on all tables/views referenced in a subquery: The Immuta system role name is specified by the user, and the role is created when the Snowflake instance is integrated.

Subquery limitations

Any subqueries that error in Snowflake will also error in Immuta.

Including one or more subqueries in the Immuta policy condition may cause errors in Snowflake. If an error occurs, it may happen during policy creation or at query-time. To avoid these errors, limit the number of subqueries, limit the number of JOIN operations, and simplify WHERE clause conditions.
For more information on the Snowflake subquery limitations see
- Understanding column-level security
- Understanding row access policies

Databricks Unity Catalog Integration Reference Guide

Immuta’s integration with Unity Catalog allows you to enforce fine-grained access controls on Unity Catalog securable objects with Immuta policies. Instead of manually creating UDFs or granting access to each table in Databricks, you can author your policies in Immuta and have Immuta manage and orchestrate Unity Catalog access-control policies on your data in Databricks clusters or SQL warehouses:

Subscription policies: Immuta subscription policies automatically grant and revoke access to specific Databricks securable objects.
: Immuta data policies enforce row- and column-level security.

Unity Catalog object model

Unity Catalog uses the following hierarchy of data objects:

Metastore: Created at the account level and is attached to one or more Databricks workspaces. The metastore contains metadata of all the catalogs, schemas, and tables available to query. All clusters on that workspace use the configured metastore and all workspaces that are configured to use a single metastore share those objects.
Catalog: Sits on top of schemas (also called databases) and tables to manage permissions across a set of schemas
Schema: Organizes tables and views
Table-etc: Table (managed or external tables), view, volume, model, and function

For details about the Unity Catalog object model, see the .

Feature support

The Databricks Unity Catalog integration supports

:
- applying column masks and row filters on specific securable objects
- applying subscription polices on tables and views
enforcing Unity Catalog access controls, even if Immuta becomes disconnected
allowing non-Immuta reads and writes
using Photon
using a proxy server

Architecture

Immuta uses this service principal to run queries that set up user-defined functions (UDFs) and other data necessary for policy enforcement. Upon enabling the native integration, Immuta will create a catalog that contains these schemas:

immuta_system: Contains internal Immuta data.
immuta_policies_n: Contains policy UDFs.

When policies require changes to be pushed to Unity Catalog, Immuta updates the internal tables in the immuta_system schema with the updated policy information. If necessary, new UDFs are pushed to replace any out-of-date policies in the immuta_policies_n schemas and any row filters or column masks are updated to point at the new policies. Many of these operations require compute on the configured Databricks cluster or SQL warehouse, so compute must be available for these policies to succeed.

Workspace-catalog binding

Immuta’s Databricks Unity Catalog integration manages a single metastore per integration. Prior to catalog-binding, a Unity Catalog metastore in Databricks was available unrestricted across workspaces in Databricks, which made integrating against a metastore independent of the workspace attached to that metastore possible. However, when catalog-binding is enabled on a Databricks workspace, you can assign catalogs available in the Unity Catalog metastore to that workspace. As a result, Immuta cannot see all catalogs in a metastore when integrated with that workspace. This behavior is problematic for customers who use catalog isolation to separate environments or business units, as Immuta cannot see all the data that may need to be governed.

To avoid this issue,

Set up a dedicated Databricks workspace for Immuta that has catalog-binding disabled so that Immuta can see all data in the metastore.
Configure Immuta’s Databricks Unity Catalog integration with that workspace to govern all data in the metastore.

Policy enforcement

Immuta’s Unity Catalog integration applies Databricks table-, row-, and column-level security controls that are enforced natively within Databricks. Immuta's management of these Databricks security controls is automated and ensures that they synchronize with Immuta policy or user entitlement changes.

Row-level security: Immuta applies SQL UDFs to restrict access to rows for querying users.
Column-level security: Immuta applies column-mask SQL UDFs to tables for querying users. These column-mask UDFs run for any column that requires masking.

The Unity Catalog integration supports the following policy types:

- Conditional masking
- Constant
- Custom masking
- Hashing
- Null
- Rounding (date and numeric rounding)
- Matching (only show rows where)
  - Custom WHERE
  - Never
  - Where user
  - Where value in column
- Minimization
- Time-based restrictions

Project-scoped purpose exceptions for Databricks Unity Catalog

Databricks Unity Catalog views

If you are using views in Databricks Unity Catalog, one of the following must be true for project-scoped purpose exceptions to apply to the views in Databricks:

The view and underlying table are registered as Immuta data sources and added to a project: If a view and its underlying table are both added as Immuta data sources, both of these assets must be added to the project for the project-scoped purpose exception to apply. If a view and underlying table are both added as data sources but the table is not added to an Immuta project, the purpose exception will not apply to the view because Databricks does not support fine-grained access controls on views.
Only the underlying table is registered as an Immuta data source and added to a project: If only the underlying table is registered as an Immuta data source but the view is not registered, the purpose exception will apply to both the table and corresponding view in Databricks. Views are the only Databricks object that will have Immuta policies applied to them even if they're not registered as Immuta data sources (as long as their underlying tables are registered).

Masked joins for Databricks Unity Catalog

Policy exemption groups

Some users may need to be exempt from masking and row-level policy enforcement. When you add user accounts to the configured exemption group in Databricks, Immuta will not enforce policies for those users. Exemption groups are created when the Unity Catalog integration is configured, and no policies will apply to these users' queries, despite any policies enforced on the tables they query.

The principal used to register data sources in Immuta will be automatically added to this exemption group for that Databricks table. Consequently, users added to this list and used to register data sources in Immuta should be limited to service accounts.

Policy support with `hive_metastore`

When enabling Unity Catalog support in Immuta, the catalog for all Databricks data sources will be updated to point at the default hive_metastore catalog. Internally, Databricks exposes this catalog as a proxy to the workspace-level Hive metastore that schemas and tables were kept in before Unity Catalog. Since this catalog is not a real Unity Catalog catalog, it does not support any Unity Catalog policies. Therefore, Immuta will ignore any data sources in the hive_metastore in any Databricks Unity Catalog integration, and policies will not be applied to tables there.

Authentication methods

The Databricks Unity Catalog integration supports the following authentication methods to configure the integration and create data sources:

Immuta data sources in Unity Catalog

External data connectors and query-federated tables

Native query audit

Access requirements

For Databricks Unity Catalog audit to work, Immuta must have, at minimum, the following access.

USE CATALOG on the system catalog
USE SCHEMA on the system.access schema
SELECT on the following system tables:
- system.access.audit
- system.access.table_lineage
- system.access.column_lineage

Tag ingestion

Design partner preview: This feature is available to select accounts. Reach out to your Immuta representative to enable this feature.

You can enable tag ingestion to allow Immuta to ingest Databricks Unity Catalog table and column tags so that you can use them in Immuta policies to enforce access controls. When you enable this feature, Immuta uses the credentials and connection information from the Databricks Unity Catalog integration to pull tags from Databricks and apply them to data sources as they are registered in Immuta. If Databricks data sources preexist the Databricks Unity Catalog tag ingestion enablement, those data sources will automatically sync to the catalog and tags will apply. Immuta checks for changes to tags in Databricks and syncs Immuta data sources to those changes every 24 hours.

Syncing tag changes

When syncing data sources to Databricks Unity Catalog tags, Immuta pulls the following information:

Table tags: These tags apply to the table and appear on the data source overview tab. Databricks tags' key and value pairs are reflected in Immuta as a hierarchy with each level separated by a . delimiter. For example, the Databricks Unity Catalog tag Location: US would be represented as Location.US in Immuta.
Column tags: These tags are applied to data source columns and appear on the columns listed in the data dictionary tab. Databricks tags' key and value pairs are reflected in Immuta as a hierarchy with each level separated by a . delimiter. For example, the Databricks Unity Catalog tag Location: US would be represented as Location.US in Immuta.
Table comments field: This content appears as the data source description on the data source overview tab.
Column comments field: This content appears as dictionary column descriptions on the data dictionary tab.

Limitations

Only tags that apply to Databricks data sources in Immuta are available to build policies in Immuta. Immuta will not pull tags in from Databricks Unity Catalog unless those tags apply to registered data sources.
Cost implications: Tag ingestion in Databricks Unity Catalog requires compute resources. Therefore, having many Databricks data sources or frequently manually syncing data sources to Databricks Unity Catalog may incur additional costs.
Databricks Unity Catalog tag ingestion only supports tenants with fewer than 2,500 data sources registered.

Configuration requirements

Supported Databricks cluster configurations

The table below outlines the integrations supported for various Databricks cluster configurations. For example, the only integration available to enforce policies on a cluster configured to run on Databricks Runtime 9.1 is the Databricks Spark integration.

Legend:

Unity Catalog caveats

Row access policies with more than 1023 columns are unsupported. This is an underlying limitation of UDFs in Databricks. Immuta will only create row access policies with the minimum number of referenced columns. This limit will therefore apply to the number of columns referenced in the policy and not the total number in the table.
If you disable table grants, Immuta revokes the grants. Therefore, if users had access to a table before enabling Immuta, they’ll lose access.
You must use the global regex flag (g) when creating a regex masking policy in this integration, and you cannot use the case insensitive regex flag (i) when creating a regex masking policy in this integration. See the examples below for guidance:
- regex with a global flag (supported): /^ssn|social ?security$/g
- regex without a global flag (unsupported): /^ssn|social ?security$/
- regex with a case insensitive flag (unsupported): /^ssn|social ?security$/gi
- regex without a case insensitive flag (supported): /^ssn|social ?security$/g

Azure Databricks Unity Catalog limitation

If a registered data source is owned by a Databricks group at the table level, then the Unity Catalog integration cannot apply data masking policies to that table in Unity Catalog.

Therefore, set all table-level ownership on your Unity Catalog data sources to an individual user or service principal instead of a Databricks group. Catalogs and schemas can still be owned by a Databricks group, as ownership at that level doesn't interfere with the integration.

Feature limitations

The following features are currently unsupported:

Databricks change data feed support
Immuta projects
Multiple IAMs on a single cluster
Column masking policies on views
Mixing masking policies on the same column
Row-redaction policies on views
R and Scala cluster support
Scratch paths
User impersonation
Policy enforcement on raw Spark reads
Python UDFs for advanced masking functions
Direct file-to-SQL reads
Data policies (except for masking with NULL) on ARRAY, MAP, or STRUCT type columns

Known issue

Snippets for Databricks data sources may be empty in the Immuta UI.

Manual Databricks Spark Configuration

This guide details the manual installation method for enabling native access to Databricks with Immuta policies enforced. Before proceeding, ensure your Databricks workspace, instance, and permissions meet the guidelines outlined in the Installation Introduction.

Databricks Unity Catalog: If Unity Catalog is enabled in a Databricks workspace, you must use an Immuta cluster policy when you set up the integration to create an Immuta-enabled cluster.

immuta_conf.xml is no longer required

The immuta_conf.xml file that was previously used to configure the native Databricks Spark integration is no longer required to install Immuta, so it is no longer staged as a deployment artifact. However, you can use these snippets if you wish to deploy an immuta_conf.xml file to set properties.

The required Immuta base URL and Immuta system API key properties, along with any other valid properties, can still be specified as Spark environment variables or in the optional immuta_conf.xml file. As before, if the same property is specified in both locations, the Spark environment variable takes precedence.

If you have an existing immuta_conf.xml file, you can continue using it. However, it's recommended that you delete any default properties from the file that you have not explicitly overridden, or remove the file completely and rely on Spark environment variables. Either method will ensure that any property defaults changed in upcoming Immuta releases are propagated to your environment.

1 - Download and Configure Immuta Artifacts

Spark Version

Use Spark 2 with Databricks Runtime prior to 7.x. Use Spark 3 with Databricks Runtime 7.x or later. Attempting to use an incompatible jar and Databricks Runtime will fail.

Navigate to the Immuta archives page. If you are prompted to log in and need basic authentication credentials, contact your Immuta support professional.
Navigate to the Databricks folder for your Immuta version. Ex: https://archives.immuta.com/hadoop/databricks/2024.3.2/.
Download the .jar file (Immuta plugin) as well as the other scripts listed below, which will load the plugin at cluster startup.
```
allowedCallingClasses.json
immuta-benchmark-suite.dbc
immuta-spark-hive-X.X.X_YYYYMMDD-hadoop-Z.Z.Z-public.jar
immuta_cluster_init_script.sh
obscuredCommands.yaml
```
The immuta-benchmark-suite.dbc is a collection of notebooks packaged as a .dbc file. After you have added cluster policies to your cluster, you can import this file into Databricks to run performance tests and compare a regular Databricks cluster to one protected by Immuta. Detailed instructions are available in the first notebook, which will require an Immuta and non-Immuta cluster to generate test data and perform queries. Note: Use Spark 2 with Databricks Runtime prior to 7.x. Use Spark 3 with Databricks Runtime 7.x or later. Attempting to use an incompatible jar and Databricks Runtime will fail.
Specify the following properties as Spark environment variables or in the optional immuta_conf.xml file. If the same property is specified in both locations, the Spark environment variable takes precedence. The variable names are the config names in all upper case with _ instead of .. For example, to set the value of immuta.base.url via an environment variable, you would set the following in the Environment Variables section of cluster configuration: IMMUTA_BASE_URL=https://immuta.mycompany.com
- immuta.system.api.key: Obtain this value from the Immuta Configuration UI under HDFS > System API Key. You will need to be a user with the APPLICATION_ADMIN role to complete this action.
- immuta.base.url: The full URL for the target Immuta tenant Ex: https://immuta.mycompany.com.
- immuta.user.mapping.iamid: If users authenticate to Immuta using an IAM different from Immuta's built-in IAM, you need to update the configuration file to reflect the ID of that IAM. The IAM ID is shown within the Immuta App Settings page within the Identity Management section. See Databricks to Immuta User Mapping for more details.

Environment variables with Google Cloud Platform

Do not use environment variables to set sensitive properties when using Google Cloud Platform. Set them directly in immuta_conf.xml.

Generating a key will destroy any previously generated HDFS keys. This will cause previously integrated HDFS systems to lose access to your Immuta console. The key will only be shown once when generated.

2 - Stage Immuta Artifacts

When configuring the Databricks cluster, a path will need to be provided to each of the artifacts downloaded/created in the previous step. To do this, those artifacts must be hosted somewhere that your Databricks instance can access. The following methods can be used for this step:

Host files in AWS/S3 and provide access by the cluster
Host files in Azure ADL Gen 1 or Gen 2 and provide access by the cluster
Host files on an HTTPS server accessible by the cluster
Host files in DBFS (Not recommended for production)

These artifacts will be downloaded to the required location within the clusters file-system by the init script downloaded in the previous step. In order for the init script to find these files, a URI will have to be provided through environment variables configured on the cluster. Each method's URI structure and setup is explained below.

AWS/S3

URI Structure: s3://[bucket]/[path]

Create an instance profile for clusters by following Databricks documentation.
Upload the configuration file, JSON file, and JAR file to an S3 bucket that the role from step 1 has access to.

Authenticating with Access Keys or Session Tokens (Optional)

If you wish to authenticate using access keys, add the following items to the cluster's environment variables:

IMMUTA_INIT_AWS_SECRET_ACCESS_KEY=<aws secret key>
IMMUTA_INIT_AWS_ACCESS_KEY_ID=<aws access key id>

If you've assumed a role and received a session token, that can be added here as well:

IMMUTA_INIT_AWS_SESSION_TOKEN=<aws session token>

Azure

ADL Gen 2

URI Structure: abfs(s)://[container]@[account].dfs.core.windows.net/[path]

Upload the configuration file, JSON file, and JAR file to an ADL gen 2 blob container.

Environment Variables:

If you want to authenticate using an account key, add the following to your cluster's environment variables:

IMMUTA_INIT_AZCOPY_CRED_TYPE=SharedKey
IMMUTA_INIT_ACCOUNT_NAME=<ADLg2 account name>
IMMUTA_INIT_ACCOUNT_KEY=<ADLg2 account key>

If you want to authenticate using an Azure SAS token, add the following to your cluster's environment variables:

IMMUTA_INIT_AZURE_SAS_TOKEN=<SAS token>

ADL Gen 1

URI Structure: adl://[account].azuredatalakestore.net/[path]

Upload the configuration file, JSON file, and JAR file to ADL gen 1.

Environment Variables:

If authenticating as a Microsoft Entra ID user,

IMMUTA_INIT_AZURE_AD_USER=<Microsoft Entra ID username>
IMMUTA_INIT_AZURE_PASSWORD=<Microsoft Entra ID password>

If authenticating using a service principal,

IMMUTA_INIT_AZURE_SERVICE_PRINCIPAL=<azure service principal>
IMMUTA_INIT_AZURE_PASSWORD=<azure service principal password>
IMMUTA_INIT_AZURE_TENANT=<tenant ID where principal was created>

HTTPS

URI Structure: http(s)://[host](:port)/[path]

Artifacts are available for download from Immuta using basic authentication. Your basic authentication credentials can be obtained from your Immuta support professional.

Environment Variables (Optional)

IMMUTA_INIT_HTTPS_USER=<basic auth username>
IMMUTA_INIT_HTTPS_PASSWORD=<basic auth password>

# Note: Credentials can also be included as part of the artifact URI. For example,
IMMUTA_INIT_JAR_URI=https://user:password@archives.immuta.com/path/to/file

DBFS

DBFS does not support access control

Any Databricks user can access DBFS via the Databricks command line utility. Files containing sensitive materials (such as Immuta API keys) should not be stored there in plain text. Use other methods described herein to properly secure such materials.

URI Structure: dbfs:/[path]

Upload the artifacts directly to DBFS using the Databricks CLI.

Since any user has access to everything in DBFS:

The artifacts can be stored anywhere in DBFS.
It's best to have a cluster-specific place for your artifacts in DBFS if you are testing to avoid overwriting or reusing someone else's artifacts accidentally.

3 - Protect Immuta Environment Variables with Databricks Secrets

It is important that non-administrator users on an Immuta-enabled Databricks cluster do not have access to view or modify Immuta configuration or the immuta-spark-hive.jar file, as this would potentially pose a security loophole around Immuta policy enforcement. Therefore, use Databricks secrets to apply environment variables to an Immuta-enabled cluster in a secure way.

Databricks secrets can be used in the Environment Variables configuration section for a cluster by referencing the secret path rather than the actual value of the environment variable. For example, if a user wanted to make the following value secret

MY_SECRET_ENV_VAR=super_secret_stuff

they could instead create a Databricks secret and reference it as the value of that variable. For instance, if the secret scope my_secrets was created, and the user added a secret with the key my_secret_env_var containing the desired sensitive environment variable, they would reference it in the Environment Variables section:

MY_SECRET_ENV_VAR={{secrets/my_secrets/my_secret_env_var}}

Then, at runtime, {{secrets/my_secrets/my_secret_env_var}} would be replaced with the actual value of the secret if the owner of the cluster has access to that secret.

Best practice: Replace sensitive variables with secrets

Immuta recommends that any sensitive environment variables listed below in the various artifact deployment instructions be replaced with secrets.

4 - Create and Configure the Cluster

Cluster creation in an Immuta-enabled organization or Databricks workspace should be limited to administrative users to avoid allowing users to create non-Immuta enabled clusters.

Create a cluster in Databricks by following the Databricks documentation.
Select the Custom Access mode.
Opt to adjust the Autopilot Options and Worker Type settings. The default values provided here may be more than what is necessary for non-production or smaller use-cases. To reduce resource usage you can enable/disable autoscaling, limit the size and number of workers, and set the inactivity timeout to a lower value.
In the Advanced Options section, click the Instances tab.
- IAM Role (AWS ONLY): Select the instance role you created for this cluster. (For access key authentication, you should instead use the environment variables listed in the AWS section.)

Click the Spark tab. In Spark Config field, add your configuration.

Cluster Configuration Requirements:

spark.executor.extraJavaOptions -Djava.security.manager=com.immuta.security.ImmutaSecurityManager /
    -Dimmuta.security.manager.classes.config=file:///databricks/immuta/allowedCallingClasses.json /
    -Dimmuta.spark.encryption.fpe.class=com.immuta.spark.encryption.ff1.ImmutaFF1Service
spark.driver.extraJavaOptions -Djava.security.manager=com.immuta.security.ImmutaSecurityManager /
    -Dimmuta.security.manager.classes.config=file:///databricks/immuta/allowedCallingClasses.json /
    -Dimmuta.spark.encryption.fpe.class=com.immuta.spark.encryption.ff1.ImmutaFF1Service
spark.databricks.repl.allowedLanguages python,sql
spark.databricks.pyspark.enableProcessIsolation true
spark.databricks.isv.product Immuta

In the Environment Variables section, add the environment variables necessary for your configuration. Remember that these variables should be protected with Databricks secrets as mentioned above.

# Specify the URI to the artifacts that were hosted in the previous steps
# The URI must adhere to the supported types for each service mentioned above
IMMUTA_INIT_JAR_URI=<Full URI to immuta-spark-hive.jar>
IMMUTA_INIT_CONF_URI=<Full URI to Immuta configuration file>
IMMUTA_INIT_ALLOWED_CALLING_CLASSES_URI=<full URI to allowedCallingClasses.json>
IMMUTA_INIT_OBSCURED_COMMANDS_URI=<full URI to obscuredCommands.yaml>

# (OPTIONAL)
# Specify an additional configuration file to be added to the spark.sparkContext.hadoopConfiguration.
# This file allows administrators to add sensitive configuration needed by the SparkSession that
# should not viewable by users.
# Further explanation of this variable as well as examples are provided below.
IMMUTA_INIT_ADDITIONAL_CONF_URI=<full URI to additional configuration file>

Click the Init Scripts tab and set the following configurations:
- Destination: Specify the service you used to host the Immuta artifacts.
- File Path: Specify the full URI to the immuta_cluster_init_script.sh.
- Add the new key/value to the configuration.
Click the Permissions tab and configure the following setting:
- Who has access: Users or groups will need to have the permission Can Attach To to execute queries against Immuta configured data sources.
(Re)start the cluster.

Additional Hadoop Configuration File (Optional)

As mentioned in the "Environment Variables" section of the cluster configuration, there may be some cases where it is necessary to add sensitive configuration to SparkSession.sparkContext.hadoopConfiguration in order to read the data composing Immuta data sources.

As an example, when accessing external tables stored in Azure Data Lake Gen 2, Spark must have credentials to access the target containers/filesystems in ADLg2, but users must not have access to those credentials. In this case, an additional configuration file may be provided with a storage account key that the cluster may use to access ADLg2.

To use an additional Hadoop configuration file, you will need to set the IMMUTA_INIT_ADDITIONAL_CONF_URI environment variable referenced in the Create and configure the cluster section to be the full URI to this file.

The additional configuration file looks very similar to the Immuta Configuration file referenced above. Some example configuration files for accessing different storage layers are below.

Amazon S3

IAM role for S3 access

S3 can also be accessed using an IAM role attached to the cluster. See the Databricks documentation for more details.

<configuration>
    <property>
        <name>fs.s3n.awsAccessKeyId</name>
        <value>[AWS access key ID]</value>
    </property>
    <property>
        <name>fs.s3n.awsSecretAccessKey</name>
        <value>[AWS secret key]</value>
    </property>
</configuration>

Azure Data Lake Gen 2

<configuration>
    <property>
        <name>fs.azure.account.key.[storage account name].dfs.core.windows.net</name>
        <value>[storage account key]</value>
    </property>
</configuration>

Azure Data Lake Gen 1

ADL prefix: Prior to Databricks Runtime version 6, the following configuration items should have a prefix of dfs.adls rather than fs.adl.

<configuration>
    <property>
        <name>fs.adl.oauth2.refresh.url</name>
        <value>https://login.microsoftonline.com/[directory ID]/oauth2/token</value>
    </property>
    <property>
        <name>fs.adl.oauth2.access.token.provider.type</name>
        <value>ClientCredential</value>
    </property>
    <property>
        <name>fs.adl.oauth2.credential</name>
        <value>[client secret from Azure]</value>
    </property>
    <property>
        <name>fs.adl.oauth2.client.id</name>
        <value>[client ID from Azure]</value>
    </property>
</configuration>

Azure Blob Storage

<configuration>
    <property>
        <name>fs.azure.account.key.[storage account name].blob.core.windows.net</name>
        <value>[storage account key]</value>
    </property>
</configuration>

5 - Register data

6 - Query Immuta Data

When the Immuta enabled Databricks cluster has been successfully started, users will see a new database labeled "immuta". This database is the virtual layer provided to access data sources configured within the connected Immuta instance.

Before users can query an Immuta data source, an administrator must give the user Can Attach To permissions on the cluster and GRANT the user access to the immuta database.

The following SQL query can be run as an administrator within a journal to give the user access to "Immuta":

%sql
GRANT SELECT,READ_METADATA ON DATABASE immuta TO `user@company.com`

Below are example queries that can be run to obtain data from an Immuta-configured data source. Because Immuta supports raw tables in Databricks, you do not have to use Immuta-qualified table names in your queries like the first example. Instead, you can run queries like the second example, which does not reference the immuta database.

%sql
select * from immuta.my_data_source limit 5;

%sql
select * from my_data_source limit 5;

Creating a Databricks Data Source

See the Databricks Data Source Creation guide for a detailed walkthrough.

Databricks to Immuta User Mapping

By default, the IAM used to map users between Databricks and Immuta is the BIM (Immuta's internal IAM). The Immuta Spark plugin will check the Databricks username against the username within the BIM to determine access. For a basic integration, this means the users email address in Databricks and the connected Immuta tenant must match.

It is possible within Immuta to have multiple users share the same username if they exist within different IAMs. In this case, the cluster can be configured to lookup users from a specified IAM. To do this, the value of immuta.user.mapping.iamid created and hosted in the previous steps must be updated to be the targeted IAM ID configured within the Immuta tenant. The IAM ID can be found on the App Settings page. Each Databricks cluster can only be mapped to one IAM.

Configure a Databricks Unity Catalog Integration

Databricks Unity Catalog allows you to manage and access data in your Databricks account across all of your workspaces. With Immuta’s Databricks Unity Catalog integration, you can write your policies in Immuta and have them enforced automatically by Databricks across data in your Unity Catalog metastore.

Permissions

Several different accounts are used to set up and maintain the Databricks Unity Catalog integration. The permissions required for each are outlined below.

Immuta account (required): This user configures the integration on the app settings page in Immuta. To access the app settings page, this user needs the following permission:
- APPLICATION_ADMIN Immuta permission
Databricks service principal (required): This service principal is used continuously by Immuta to orchestrate Unity Catalog policies and maintain state between Immuta and Databricks. In the automatic setup option, Immuta also uses this service principal to create the Immuta-managed catalog. This service principal needs the following Databricks privileges:
- CREATE CATALOG privilege on the Unity Catalog metastore. This is only required if you have Immuta automatically configure the integration in Databricks for you. If a separate user will run the Immuta script in Databricks to manually configure the integration, that Databricks user account needs this privilege instead.
- OWNER permission on the Immuta catalog you configure.
- OWNER privilege on one of the securables below so that Immuta can administer Unity Catalog row-level and column-level security controls.
  - on catalogs with schemas and tables registered as Immuta data sources. This permission could also be applied by granting OWNER on a catalog to a Databricks group that includes the Immuta service principal to allow for multiple owners.
  - on schemas with tables registered as Immuta data sources.
  - on all tables registered as Immuta data sources - if the OWNER permission cannot be applied at the catalog- or schema-level. In this case, each table registered as an Immuta data source must individually have the OWNER permission granted to the Immuta service principal.
- USE CATALOG and USE SCHEMA on parent catalogs and schemas of tables registered as Immuta data sources so that the Immuta service principal can SELECT and MODIFY securables within the parent catalog and schema.
- SELECT and MODIFY on all tables registered as Immuta data sources so that the Immuta service principal can grant and revoke access to tables and apply Unity Catalog row- and column-level security controls.
- For native query audit (optional)
  - USE CATALOG on the system catalog
  - USE SCHEMA on the system.access schema
  - SELECT on the following system tables:
    system.access.audit
    system.access.table_lineage
    system.access.column_lineage
Databricks account (recommended): This user account can manually configure the integration in Databricks to create the Immuta-managed catalog. To do so, this account requires the following Databricks privileges:
- CREATE CATALOG on the Unity Catalog metastore
- ACCOUNT ADMIN on the Unity Catalog metastore for native query audit (optional)

Requirements

Before you configure the Databricks Unity Catalog integration, ensure that you have fulfilled the following requirements:

Unity Catalog metastore created and attached to a Databricks workspace. Immuta supports configuring a single metastore for each configured integration, and that metastore may be attached to multiple Databricks workspaces.
Unity Catalog enabled on your Databricks cluster or SQL warehouse. All SQL warehouses have Unity Catalog enabled if your workspace is attached to a Unity Catalog metastore. Immuta recommends linking a SQL warehouse to your Immuta tenant rather than a cluster for both performance and availability reasons.

Unity Catalog best practices

Ensure your integration with Unity Catalog goes smoothly by following these guidelines:

Use a Databricks SQL warehouse to configure the integration. Databricks SQL warehouses are faster to start than traditional clusters, require less management, and can run all the SQL that Immuta requires for policy administration. A serverless warehouse provides nearly instant startup time and is the preferred option for connecting to Immuta.
Move all data into Unity Catalog before configuring Immuta with Unity Catalog. The default catalog used once Unity Catalog support is enabled in Immuta is the hive_metastore, which is not supported by the Unity Catalog native integration. Data sources in the Hive Metastore must be managed by the Databricks Spark integration. Existing data sources will need to be re-created after they are moved to Unity Catalog and the Unity Catalog integration is configured.

Migrate data to Unity Catalog

Ensure that all Databricks clusters that have Immuta installed are stopped and the Immuta configuration is removed from the cluster. Immuta-specific cluster configuration is no longer needed with the Databricks Unity Catalog integration.
Move all data into Unity Catalog before configuring Immuta with Unity Catalog. Existing data sources will need to be re-created after they are moved to Unity Catalog and the Unity Catalog integration is configured. If you don't move all data before configuring the integration, metastore magic will protect your existing data sources throughout the migration process.

Create the Databricks service principal

In Databricks, create a service principal with the privileges listed above.

Opt to enable native query audit for Unity Catalog

Enable a system schema where the <SCHEMA_NAME> is access.
Enable verbose audit logs in Unity Catalog.
If you will configure the integration using the manual setup option, the Immuta script you will use includes the SQL statements for granting required privileges to the service principal, so you can skip this step and continue to the manual setup section. Otherwise, manually grant the Immuta service principal access to the Databricks Unity Catalog system tables. For Databricks Unity Catalog audit to work, the service principal must have the following access at minimum:
- USE CATALOG on the system catalog
- USE SCHEMA on the system.access schema
- SELECT on the following system tables:
  - system.access.audit
  - system.access.table_lineage
  - system.access.column_lineage

Configure the Databricks Unity Catalog integration

Existing data source migration: If you have existing Databricks data sources, complete these migration steps before proceeding.

You have two options for configuring your Databricks Unity Catalog integration:

Automatic setup: Immuta creates the catalogs, schemas, tables, and functions using the integration's configured service principal.
Manual setup: Run the Immuta script in Databricks yourself to create the catalog. You can also modify the script to customize your storage location for tables, schemas, or catalogs. The user running the script must have the Databricks privileges listed above.

Automatic setup

Required permissions: When performing an automatic setup, the Immuta service principal must have the permissions listed above for the metastore associated with the specified Databricks workspace.

Click the App Settings icon in the left sidebar.
Scroll to the Global Integrations Settings section and check the Enable Databricks Unity Catalog support in Immuta checkbox.
Click the Integrations tab.
Click + Add Native Integration and select Databricks Unity Catalog from the dropdown menu.
Complete the following fields:
- Server Hostname is the hostname of your Databricks workspace.
- HTTP Path is the HTTP path of your Databricks cluster or SQL warehouse.
- Immuta Catalog is the name of the catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.
If using a proxy server with Databricks Unity Catalog, click the Enable Proxy Support checkbox and complete the Proxy Host and Proxy Port fields. The username and password fields are optional.
Opt to fill out the Exemption Group field with the name of a group in Databricks that will be excluded from having data policies applied and must not be changed from the default value. Create this account-level group for privileged users and service accounts that require an unmasked view of data before configuring the integration in Immuta.
Opt to scope the query audit ingestion by entering in Unity Catalog Workspace IDs. Enter a comma-separated list of the workspace IDs that you want Immuta to ingest audit records for. If left empty, Immuta will audit all tables and users in Unity Catalog.
Unity Catalog query audit is enabled by default; you can disable it by clicking the Enable Native Query Audit checkbox. Ensure you have enabled system tables in Unity Catalog and provided the required access to the Immuta service principal.
1. Configure the audit frequency by scrolling to Integrations Settings and find the Unity Catalog Audit Sync Schedule section.
2. Enter how often, in hours, you want Immuta to ingest audit events from Unity Catalog as an integer between 1 and 24.
3. Continue with your integration configuration.
Select your authentication method from the dropdown:
- Access Token: Enter a Databricks Personal Access Token. This is the access token for the Immuta service principal. This service principal must have the metastore privileges listed above for the metastore associated with the Databricks workspace. If this token is configured to expire, update this field regularly for the integration to continue to function.
- OAuth machine-to-machine (M2M):
  - AWS Databricks:
    Follow Databricks documentation to create a client secret for the Immuta service principal and assign this service principal the privileges listed above for the metastore associated with the Databricks workspace.
    Fill out the Token Endpoint with the full URL of the identity provider. This is where the generated token is sent. The default value is https://<your workspace name>.cloud.databricks.com/oidc/v1/token.
    Fill out the Client ID. This is a combination of letters, numbers, or symbols, used as a public identifier and is the client ID displayed in Databricks when creating the client secret for the service principal.
    Enter the Scope (string). The scope limits the operations and roles allowed in Databricks by the access token. See the OAuth 2.0 documentation for details about scopes.
    Enter the Client Secret you created above. Immuta uses this secret to authenticate with the authorization server when it requests a token.
  - Azure Databricks:
    Follow Databricks documentation to create a service principal within Azure and then populate to your Databricks account and workspace.
    Assign this service principal the privileges listed above for the metastore associated with the Databricks workspace.
    Within Databricks, create an OAuth client secret for the service principal. This completes your Databricks-based service principal setup.
    Within Immuta, fill out the Token Endpoint with the full URL of the identity provider. This is where the generated token is sent. The default value is https://<your workspace name>.azuredatabricks.net/oidc/v1/token.
    Fill out the Client ID. This is a combination of letters, numbers, or symbols, used as a public identifier and is the client ID displayed in Databricks when creating the client secret for the service principal (note that Azure Databricks uses the Azure SP Client ID; it will be identical).
    Enter the Scope (string). The scope limits the operations and roles allowed in Databricks by the access token. See the OAuth 2.0 documentation for details about scopes.
    Enter the Client Secret you created above. Immuta uses this secret to authenticate with the authorization server when it requests a token.
Click Save.

Manual setup

Required permissions: When performing a manual setup, the Databricks permissions listed above for the user running the script and the Immuta system account user are required.

Click the App Settings icon in the left sidebar.
Scroll to the Global Integrations Settings section and check the Enable Databricks Unity Catalog support in Immuta checkbox.
Click the Integrations tab.
Click + Add Native Integration and select Databricks Unity Catalog from the dropdown menu.
Complete the following fields:
- Server Hostname is the hostname of your Databricks workspace.
- HTTP Path is the HTTP path of your Databricks cluster or SQL warehouse.
- Immuta Catalog is the name of the catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.
If using a proxy server with Databricks Unity Catalog, click the Enable Proxy Support checkbox and complete the Proxy Host and Proxy Port fields. The username and password fields are optional.
Opt to fill out the Exemption Group field with the name of a group in Databricks that will be excluded from having data policies applied and must not be changed from the default value. Create this account-level group for privileged users and service accounts that require an unmasked view of data before configuring the integration in Immuta.
Opt to scope the query audit ingestion by entering in Unity Catalog Workspace IDs. Enter a comma-separated list of the workspace IDs that you want Immuta to ingest audit records for. If left empty, Immuta will audit all tables and users in Unity Catalog.
Unity Catalog query audit is enabled by default; you can disable it by clicking the Enable Native Query Audit checkbox. Ensure you have enabled system tables in Unity Catalog and provided the required access to the Immuta service principal.
1. Configure the audit frequency by scrolling to Integrations Settings and find the Unity Catalog Audit Sync Schedule section.
2. Enter how often, in hours, you want Immuta to ingest audit events from Unity Catalog as an integer between 1 and 24.
3. Continue with your integration configuration.
Select your authentication method from the dropdown:
- Access Token: Enter a Databricks Personal Access Token. This is the access token for the Immuta service principal. This service principal must have the metastore privileges listed above for the metastore associated with the Databricks workspace. If this token is configured to expire, update this field regularly for the integration to continue to function.
- OAuth machine-to-machine (M2M):
  - AWS Databricks:
    Follow Databricks documentation to create a client secret for the Immuta service principal and assign this service principal the privileges listed above for the metastore associated with the Databricks workspace.
    Fill out the Token Endpoint with the full URL of the identity provider. This is where the generated token is sent. The default value is https://<your workspace name>.cloud.databricks.com/oidc/v1/token.
    Fill out the Client ID. This is a combination of letters, numbers, or symbols, used as a public identifier and is the client ID displayed in Databricks when creating the client secret for the service principal.
    Enter the Scope (string). The scope limits the operations and roles allowed in Databricks by the access token. See the OAuth 2.0 documentation for details about scopes.
    Enter the Client Secret you created above. Immuta uses this secret to authenticate with the authorization server when it requests a token.
  - Azure Databricks:
    Follow Databricks documentation to create a service principal within Azure and then populate to your Databricks account and workspace.
    Assign this service principal the privileges listed above for the metastore associated with the Databricks workspace.
    Within Databricks, create an OAuth client secret for the service principal. This completes your Databricks-based service principal setup.
    Within Immuta, fill out the Token Endpoint with the full URL of the identity provider. This is where the generated token is sent. The default value is https://<your workspace name>.azuredatabricks.net/oidc/v1/token.
    Fill out the Client ID. This is a combination of letters, numbers, or symbols, used as a public identifier and is the client ID displayed in Databricks when creating the client secret for the service principal (note that Azure Databricks uses the Azure SP Client ID; it will be identical).
    Enter the Scope (string). The scope limits the operations and roles allowed in Databricks by the access token. See the OAuth 2.0 documentation for details about scopes.
    Enter the Client Secret you created above. Immuta uses this secret to authenticate with the authorization server when it requests a token.
Select the Manual toggle and copy or download the script. You can modify the script to customize your storage location for tables, schemas, or catalogs.
Run the script in Databricks.
Click Save.

Map Databricks users to Immuta

If the usernames in Immuta do not match usernames in Databricks, map each Databricks username to each Immuta user account to ensure Immuta properly enforces policies using one of the methods linked below:

If the Databricks user doesn't exist in Databricks when you configure the integration, manually link their Immuta username to Databricks after they are created in Databricks. Otherwise, policies will not be enforced correctly for them in Databricks. Databricks user identities for Immuta users are automatically marked as invalid when the user is not found during policy application, preventing them from being affected by Databricks policy until their Immuta user identity is manually mapped to their Databricks identity.

Opt to enable Databricks Unity Catalog tag ingestion

Design partner preview

This feature is only available to select accounts. Reach out to your Immuta representative to enable this feature.

Requirement: A Databricks Unity Catalog integration must be configured for tags to be ingested.

To allow Immuta to automatically import table and column tags from Databricks Unity Catalog, enable Databricks Unity Catalog tag ingestion in the external catalog section of the Immuta app settings page.

Navigate to the App Settings page.
Scroll to 2 External Catalogs, and click Add Catalog.
Enter a Display Name and select Databricks Unity Catalog from the dropdown menu.
Click Save and confirm your changes.

Register data

Environment Variables

This page outlines configuration details for Immuta-enabled Databricks clusters. Databricks Administrators should place the desired configuration in the Spark environment variables (recommended) or immuta_conf.xml (not recommended).

This page contains references to the term whitelist, which Immuta no longer uses. When the term is removed from the software, it will be removed from this page.

Environment variable overrides

Properties in the config file can be overridden during installation using environment variables. The variable names are the config names in all upper case with _ instead of .. For example, to set the value of immuta.base.url via an environment variable, you would set the following in the Environment Variables section of cluster configuration: IMMUTA_BASE_URL=https://immuta.mycompany.com

immuta.ephemeral.host.override
- Default: true
- Description: Set this to false if ephemeral overrides should not be enabled for Spark. When true, this will automatically override ephemeral data source httpPaths with the httpPath of the Databricks cluster running the user's Spark application.
immuta.ephemeral.host.override.httpPath
- Description: This configuration item can be used if automatic detection of the Databricks httpPath should be disabled in favor of a static path to use for ephemeral overrides.
immuta.ephemeral.table.path.check.enabled
- Default: true
- Description: When querying Immuta data sources in Spark, the metadata from the Metastore is compared to the metadata for the target source in Immuta to validate that the source being queried exists and is queryable on the current cluster. This check typically validates that the target (database, table) pair exists in the Metastore and that the table’s underlying location matches what is in Immuta. This configuration can be used to disable location checking if that location is dynamic or changes over time. Note: This may lead to undefined behavior if the same table names exist in multiple workspaces but do not correspond to the same underlying data.
immuta.spark.acl.enabled
- Default: true
- Description: Immuta Access Control List (ACL). Controls whether Databricks users are blocked from accessing non-Immuta tables. Ignored if Databricks Table ACLs are enabled (i.e., spark.databricks.acl.dfAclsEnabled=true).
immuta.spark.acl.whitelist
- Description: Comma-separated list of Databricks usernames who may access raw tables when the Immuta ACL is in use.
immuta.spark.acl.privileged.timeout.seconds
- Default: 3600
- Description: The number of seconds to cache privileged user status for the Immuta ACL. A privileged Databricks user is an admin or is whitelisted in immuta.spark.acl.whitelist.
immuta.spark.acl.assume.not.privileged
- Default: false
- Description: Session property that overrides privileged user status when the Immuta ACL is in use. This should only be used in R scripts associated with spark-submit jobs.
immuta.spark.audit.all.queries
- Default: false
- Description: Enables auditing all queries run on a Databricks cluster, regardless of whether users touch Immuta-protected data or not.
immuta.spark.databricks.allow.non.immuta.reads
- Default: false
- Description: Allows non-privileged users to SELECT from tables that are not protected by Immuta. See Limited Enforcement in Databricks Spark for details about this feature.
immuta.spark.databricks.allow.non.immuta.writes
- Default: false
- Description: Allows non-privileged users to run DDL commands and data-modifying commands against tables or spaces that are not protected by Immuta. See Limited Enforcement in Databricks Spark for details about this feature.
immuta.spark.databricks.allowed.impersonation.users
- Description: This configuration is a comma-separated list of Databricks users who are allowed to impersonate Immuta users.
immuta.spark.databricks.dbfs.mount.enabled
- Default: false
- Description: Exposes the DBFS FUSE mount located at /dbfs. Granular permissions are not possible, so all users will have read/write access to all objects therein. Note: Raw, unfiltered source data should never be stored in DBFS.
immuta.spark.databricks.disabled.udfs
- Description: Block one or more Immuta user-defined functions (UDFs) from being used on an Immuta cluster. This should be a Java regular expression that matches the set of UDFs to block by name (excluding the immuta database). For example to block all project UDFs, you may configure this to be ^.*_projects?$. For a list of functions, see the project UDFs page.
immuta.spark.databricks.filesystem.blacklist
- Default: hdfs
- Description: A list of filesystem protocols that this instance of Immuta will not support for workspaces. This is useful in cases where a filesystem is available to a cluster but should not be used on that cluster.
immuta.spark.databricks.jar.uri
- Default: file:///databricks/jars/immuta-spark-hive.jar
- Description: The location of immuta-spark-hive.jar on the filesystem for Databricks. This should not need to change unless a custom initialization script that places immuta-spark-hive in a non-standard location is necessary.
immuta.spark.databricks.local.scratch.dir.enabled
- Default: true
- Description: Creates a world-readable/writable scratch directory on local disk to facilitate the use of dbutils and 3rd party libraries that may write to local disk. Its location is non-configurable and is stored in the environment variable IMMUTA_LOCAL_SCRATCH_DIR. Note: Sensitive data should not be stored at this location.
immuta.spark.databricks.log.level
- Default Value: INFO
- Description: The SLF4J log level to apply to Immuta's Spark plugins.
immuta.spark.databricks.log.stdout.enabled
- Default: false
- Description: If true, writes logging output to stdout/the console as well as the log4j-active.txt file (default in Databricks).
immuta.spark.databricks.py4j.strict.enabled
- Default: true
- Description: Disable to allow the use of the dbutils API in Python. Note: This setting should only be disabled for customers who employ a homogeneous integration (i.e., all users have the same level of data access).
immuta.spark.databricks.scratch.database
- Description: This configuration is a comma-separated list of additional databases that will appear as scratch databases when running a SHOW DATABASE query. This configuration increases performance by circumventing the Metastore to get the metadata for all the databases to determine what to display for a SHOW DATABASE query; it won't affect access to the scratch databases. Instead, use immuta.spark.databricks.scratch.paths to control read and write access to the underlying database paths.
  Additionally, this configuration will only display the scratch databases that are configured and will not validate that the configured databases exist in the Metastore. Therefore, it is up to the Databricks administrator to properly set this value and keep it current.
immuta.spark.databricks.scratch.paths
- Description: Comma-separated list of remote paths that Databricks users are allowed to directly read/write. These paths amount to unprotected "scratch spaces." You can create a scratch database by configuring its specified location (or configure dbfs:/user/hive/warehouse/<db_name>.db for the default location).
  To create a scratch path to a location or a database stored at that location, configure
  <property> <name>immuta.spark.databricks.scratch.paths</name> <value>s3://path/to/the/dir</value> </property>
  To create a scratch path to a database created using the default location,
  <property> <name>immuta.spark.databricks.scratch.paths</name> <value>s3://path/to/the/dir, dbfs:/user/hive/warehouse/any_db_name.db</value> </property>
immuta.spark.databricks.scratch.paths.create.db.enabled
- Default: false
- Description: Enables non-privileged users to create or drop scratch databases.
immuta.spark.databricks.single.impersonation.user
- Default: false
- Description: When true, this configuration prevents users from changing their impersonation user once it has been set for a given Spark session. This configuration should be set when the BI tool or other service allows users to submit arbitrary SQL or issue SET commands.
immuta.spark.databricks.submit.tag.job
- Default: true
- Description: Denotes whether the Spark job will be run that "tags" a Databricks cluster as being associated with Immuta.
immuta.spark.databricks.trusted.lib.uris
- Description: Databricks Trusted Libraries
immuta.spark.non.immuta.table.cache.seconds
- Default: 3600
- Description: The number of seconds Immuta caches whether a table has been exposed as a source in Immuta. This setting only applies when immuta.spark.databricks.allow.non.immuta.writes or immuta.spark.databricks.allow.non.immuta.reads is enabled.
immuta.spark.require.equalization
- Default: false
- Description: Requires that users act through a single, equalized project. A cluster should be equalized if users need to run Scala jobs on it, and it should be limited to Scala jobs only via spark.databricks.repl.allowedLanguages.
immuta.spark.resolve.raw.tables.enabled
- Default: true
- Description: Enables use of the underlying database and table name in queries against a table-backed Immuta data source. Administrators or whitelisted users can set immuta.spark.session.resolve.raw.tables.enabled to false to bypass resolving raw databases or tables as Immuta data sources. This is useful if an admin wants to read raw data but is also an Immuta user. By default, data policies will be applied to a table even for an administrative user if that admin is also an Immuta user.
immuta.spark.session.resolve.raw.tables.enabled
- Default: true
- Description: Same as above, but a session property that allows users to toggle this functionality. If users run set immuta.spark.session.resolve.raw.tables.enabled=false, they will see raw data only (not Immuta data policy-enforced data). Note: This property is not set in immuta_conf.xml.
immuta.spark.show.immuta.database
- Default: true
- Description: This shows the immuta database in the configured Databricks cluster. When set to false Immuta will no longer show this database when a SHOW DATABASES query is performed. However, queries can still be performed against tables in the immuta database using the Immuta-qualified table name (e.g., immuta.my_schema_my_table) regardless of whether or not this feature is enabled.
immuta.spark.version.validate.enabled
- Default: true
- Description: Immuta checks the versions of its artifacts to verify that they are compatible with each other. When set to true, if versions are incompatible, that information will be logged to the Databricks driver logs and the cluster will not be usable. If a configuration file or the jar artifacts have been patched with a new version (and the artifacts are known to be compatible), this check can be set to false so that the versions don't get logged as incompatible and make the cluster unusable.
immuta.user.context.class
- Default: com.immuta.spark.OSUserContext
- Description: The class name of the UserContext that will be used to determine the current user in immuta-spark-hive. The default implementation gets the OS user running the JVM for the Spark application.
immuta.user.mapping.iamid
- Default: bim
- Description: Denotes which IAM in Immuta should be used when mapping the current Spark user's username to a userid in Immuta. This defaults to Immuta's internal IAM (bim) but should be updated to reflect an actual production IAM.

Customize Read and Write Access Policies for Starburst (Trino)

Private preview: Write policies are only available to select accounts. Contact your Immuta representative to enable this feature.

Requirements

Starburst (Trino) version 438 or newer
Write policies for Starburst (Trino) enabled. Contact your Immuta representative to get this feature enabled on your account.

Configuration options

In its default setting, the Starburst (Trino) integration's write access value controls the authorization of SQL operations that perform data modification (such as INSERT, UPDATE, DELETE, MERGE, and TRUNCATE). However, administrators can allow table modification operations (such as ALTER and DROP tables) to be authorized as write operations. Two locations allow administrators to specify how read and write access policies are applied to data in Starburst (Trino). Select one or both of the options below to customize these settings. If the access-control.properties file is used, it may override the policies configured in the Immuta web service.

Immuta web service: Configure write policies in the Immuta web service to allow all Starburst (Trino) clusters targeting that Immuta tenant to receive the same write policy configuration for data sources. This configuration will only affect tables or views registered as Immuta data sources.
Starburst (Trino) cluster: Configure write policies using the access-control.properties file in Starburst or Trino to broadly customize access for Immuta users on a specific cluster. This configuration file takes precedence over write policies passed from the Immuta web service. Use this option if all Immuta users should have the same level of access to tables regardless of the write policy setting in the Immuta web service.

Immuta web service configuration

Contact your Immuta representative to configure read and write access in the Immuta web service if all Starburst (Trino) data source operations should be affected identically across Starburst (Trino) clusters connected to your Immuta tenant. A configuration example is provided below.

Configuration example

The following example maps WRITE to READ, WRITE and OWN permissions and READ to just READ. Both READ and WRITE permissions should always include READ:

accessGrantMapping:
  WRITE: ['READ', 'WRITE', 'OWN']
  READ: ['READ']

Given the above configuration, when a user gets write access to a Starburst (Trino) data source, they will have both data and table modification permissions on that data source. See the Starburst (Trino) privileges section of the Subscription policy access types guide for details about these operations.

Starburst cluster configuration

Configure the integration to allow read and write policies to apply to any data source (registered or unregistered in Immuta) on a Starburst cluster.

Create the Immuta access control configuration file in the Starburst configuration directory (/etc/starburst/immuta-access-control.properties for Docker installations or <starburst_install_directory>/etc/immuta-access-control.properties for standalone installations).
Modify one or both properties below to customize the behavior of read or write access policies for all users:
- immuta.allowed.immuta.datasource.operations: This property governs objects (catalogs, schemas, tables, etc.) that are registered as data sources in Immuta. For these permissions to apply, the user must be subscribed to the data source in Immuta and not be an administrator (who gets all permissions).
  - READ: Grants SELECT on tables or views; grants SHOW on tables, views, or columns
  - WRITE: Grants INSERT, UPDATE, DELETE, MERGE, or TRUNCATE on tables; grants REFRESH on materialized views.
  - OWN: Grants ALTER and DROP on tables; grants SET on comments and properties
- immuta.allowed.non.immuta.datasource.operations: This property governs objects (catalogs, schemas, tables, etc.) that are not registered as data sources in Immuta. Use all or a combination of the following access values:
  - READ: Grants SELECT on tables or views; grants SHOW on tables, views, or columns
  - WRITE: Grants INSERT, UPDATE, DELETE, MERGE, or TRUNCATE on tables; grants REFRESH on materialized views.
  - OWN: Grants ALTER and DROP on tables; grants SET on comments and properties
  - CREATE: Grants CREATE on catalogs, schema, tables, and views. This is the only property that can allow CREATE permissions, since CREATE is enforced on new objects that do not exist in Starburst or Immuta yet (such as a new table being created with CREATE TABLE).
For example, the following configuration allows READ, WRITE, and OWN operations to be authorized on data sources registered in Immuta and all operations are permitted on data that is not registered in Immuta:
```
immuta.allowed.immuta.datasource.operations=READ,WRITE,OWN
immuta.allowed.non.immuta.datasource.operations=READ,WRITE,CREATE,OWN
```
Enable the Immuta access control plugin in the Starburst cluster's configuration file (/etc/starburst/config.properties for Docker installations or <starburst_install_directory>/etc/config.properties for standalone installations). For example,
```
access-control.config-files=/etc/starburst/immuta-access-control.properties
```

Trino cluster configuration

Create the Immuta access control configuration file in the Trino configuration directory (/etc/trino/config.properties for Docker installations or <trino_install_directory>/etc/config.properties for standalone installations).
Modify one or both properties below to customize the behavior of read or write access policies for all users:
- immuta.allowed.immuta.datasource.operations: This property governs objects (catalogs, schemas, tables, etc.) that are registered as data sources in Immuta. For these permissions to apply, the user must be subscribed to the data source in Immuta and not be an administrator (who gets all permissions).
  - READ: Grants SELECT on tables or views; grants SHOW on tables, views, or columns
  - WRITE: Grants INSERT, UPDATE, DELETE, MERGE, or TRUNCATE on tables; grants REFRESH on materialized views.
  - OWN: Grants ALTER and DROP on tables; grants SET on comments and properties
- immuta.allowed.non.immuta.datasource.operations: This property governs objects (catalogs, schemas, tables, etc.) that are not registered as data sources in Immuta. Use all or a combination of the following access values:
  - READ: Grants SELECT on tables or views; grants SHOW on tables, views, or columns
  - WRITE: Grants INSERT, UPDATE, DELETE, MERGE, or TRUNCATE on tables; grants REFRESH on materialized views.
  - OWN: Grants ALTER and DROP on tables; grants SET on comments and properties
  - CREATE: Grants CREATE on catalogs, schema, tables, and views. This is the only property that can allow CREATE permissions, since CREATE is enforced on new objects that do not exist in Starburst or Immuta yet (such as a new table being created with CREATE TABLE).
For example, the following configuration allows READ, WRITE, and OWN operations to be authorized on data sources registered in Immuta and all operations are permitted on data that is not registered in Immuta:
```
immuta.allowed.immuta.datasource.operations=READ,WRITE,OWN
immuta.allowed.non.immuta.datasource.operations=READ,WRITE,CREATE,OWN
```
Enable the Immuta access control plugin in Trino's configuration file (/etc/trino/config.properties for Docker installations or <trino_install_directory>/etc/config.properties for standalone installations). For example,
```
access-control.config-files=/etc/trino/immuta-access-control.properties
```