Policy Decision Data

Policy decision data is transmitted to ensure end users querying data are limited to the appropriate access as defined by the policies in Immuta.

1. A user runs a query against data in their environment.

2. The query is sent to the Immuta Web Service.

3. The Web Service queries the Metadata Database to obtain the policy definition, which includes data source metadata (tags, column names, etc.) and user entitlements (groups and attributes).

4. The policy information is transmitted to the remote data system for native policy enforcement.

5. Query results are displayed based on what policy definition was applied.

Sample Raw Data

Sample data is processed and aggregated or reduced during Immuta's fingerprinting process and specific policy processes. Note: Data Owners can see sample data when editing a data source. However, this action requires the database password, and the small sample of data visible is only displayed in the UI and is not stored in Immuta.

Fingerprinting Process

1. When enabled, statistical queries made during data source health checks are distilled into summary statistics, called fingerprints. The sample data processed for fingerprinting allows Immuta to track data source changes.

2. During this process, statistical query results and data samples (which may contain PII) are temporarily held in memory by the Fingerprint Service.

3. The fingerprinting process checks for new tables through schema monitoring (when enabled) and captures summary statistics of changes to data sources, including when policies were applied, external views were created, or sensitive data elements were added.

Policy Processes

1. Sample data is processed when k-anonymization or randomized response policies are applied to data sources.

2. Sample data exists temporarily in memory in the Fingerprint Service during the computation.

   • k-Anonymization Policies: At the time of its application, the columns of a k-anonymization policy are queried under a separate fingerprinting process that generates rules enforcing k-anonymity. The results of this query, which may contain PII, are temporarily held in memory by the Fingerprint Service. The final rules are stored in the Metadata Database as the policy definition for enforcement.

   • Randomized Response Policies: If the list of substitution values for a categorical column is not part of the policy specification (e.g., when specified via the API), a list is obtained via query and merged into the policy definition in the Metadata Database.

3. Raw data is processed for masking, producing either a distinct set of values or aggregated groups of values.

Encryption of Data at Rest

Immuta captures metadata and stores it in an internal PostgreSQL database. Customers can encrypt the volumes backing the database using an external Key Management Service to ensure that data is encrypted at rest.

Encryption of Data in Transit

To encrypt data in transit, Immuta uses TLS protocol, which is configured by the customer.

Encryption Key Management

Immuta encrypts values with data encryption keys, either those that are system-generated or managed using an external key management service (KMS). Immuta recommends a KMS to encrypt or decrypt data keys and supports the AWS Key Management Service; however, if no KMS is configured, Immuta will generate a data encryption key on a user-defined rollover schedule, using the most recent data key to encrypt new values while preserving old data keys to decrypt old values.

Masking

Immuta employs three families of functions in its masking policies:

• One-way Hashing: One-way (irreversible) hashing is performed via a salted SHA256 hash. A consistent salt is used for values throughout the data source, so users can count or track the specific values without revealing the true value. Since hashed values are different across data sources, users are unable to join on hashed values. Note: joining on masked values can be enabled in Immuta Projects.

• Reversible Masking: For reversible masking, values are encrypted using AES-256 CBC encryption. Encryption is performed using a cell-specific initialization vector. The resulting values can be unmasked by an authorized user. Note that this is dynamic encryption of individual fields as results are streamed to the querying system; Immuta is not modifying records in the data store.

• Reversible Format Preserving Masking: Format preserving masking maintains the format of the data while masking the value, and is achieved by initializing and applying the NIST standard method FF1 at the column level. The resulting values can be unmasked by an authorized user.

TCP Connection

Immuta communicates with remote databases over a TCP connection.

If you want to disable the metadata collection that requires sampling data, you must

1. Disable Immuta fingerprinting.

2. Stop all data source health checks.

3. Add the Skip Stats Job tag to all data sources.

These steps will ensure that Immuta queries no data, under any circumstances. Without this sample data, some Immuta features will be unavailable. Sensitive Data Discovery (SDD) cannot be used to automatically detect sensitive data in your data sources, and the following masking policies will not work:

• Masking with format preserving masking

• Masking with k-anonymization

• Masking using randomized response

Advanced Configuration

Fingerprinting

To stop Immuta from running fingerprints on all data sources,

1. Navigate to the App Settings page, and scroll to the Advanced Configuration section.

2. Enter the following YAML:

   Copy

   fingerprints:
     uri: null
     classification:
       enabled: false

3. Click Save.

Data Source Health Checks

To stop Immuta from running data source health checks on all data sources,

1. Navigate to the App Settings page, and scroll to the Advanced Configuration section.

2. Enter the following YAML:

   Copy

   plugins:
     snowflakeHandler:
       config:
         healthCheckQuery: null
     redshiftHandler:
        config:
         healthCheckQuery: null
     trinoHandler:
       config:
         healthCheckQuery: null
     databricksHandler:
       config:
         healthCheckQuery: null
     asaHandler:
       config:
         healthCheckQuery: null

3. Click Save.

Skip Stats Job Tag

Tag each data source with the seeded Skip Stats Job tag to stop Immuta from collecting a sample and running table stats on the sample. You can tag data sources as you create them in the UI or via the Immuta API.

Immuta does not require users to learn a new API or language to access data exposed there. Instead, Immuta integrates with existing tools and ongoing work while remaining invisible to downstream consumers. This page outlines those integrations.

The Snowflake integration differs based on your Snowflake Edition:

• Snowflake Integration Using Snowflake Governance Features: With this integration, policies administered in Immuta are pushed down into Snowflake as (row access policies and masking policies). This integration requires Snowflake Enterprise Edition or higher.

• Snowflake Integration Without Snowflake Governance Features: With this integration, policies administered by Immuta are pushed down into Snowflake as views with a 1-to-1 relationship to the original table and all policy logic is contained in that view.

Click a link below for details about each question:

This integration allows you to manage multiple Databricks workspaces through Unity Catalog while protecting your data with Immuta policies. Instead of manually creating UDFs or granting access to each table in Databricks, you can author your policies in Immuta and have Immuta manage and enforce Unity Catalog access-control policies on your data in Databricks clusters or SQL warehouse.

Immuta’s Databricks Spark integration with Unity Catalog support uses a custom Databricks plugin to enforce Immuta policies on a Databricks cluster with Unity Catalog enabled. This integration allows you to add your tables to the Unity Catalog metastore so that you can use the metastore from any workspace while protecting your data with Immuta policies.

This integration enforces policies on Databricks tables registered as data sources in Immuta, allowing users to query policy-enforced data on Databricks clusters (including job clusters). Immuta policies are applied to the plan that Spark builds for users' queries, all executed directly against Databricks tables.

The Starburst (Trino) integration enables Immuta to apply policies directly in Starburst and Trino clusters without going through a proxy. This means users can use their existing Starburst and Trino tooling (querying, reporting, etc.) and have per-user policies dynamically applied at query time.

With the Redshift integration, Immuta applies policies directly in Redshift. This allows data analysts to query their data directly in Redshift instead of going through a proxy.

The Azure Synapse Analytics integration allows Immuta to apply policies directly in Azure Synapse Analytics dedicated SQL pools without needing users to go through a proxy. Instead, users can work within their existing Synapse Studio and have per-user policies dynamically applied at query time.

The Amazon S3 integration allows users to apply subscription policies to data in S3 to restrict what prefixes, buckets, or objects users can access. To enforce access controls on this data, Immuta creates S3 grants that are administered by S3 Access Grants, an AWS feature that defines access permissions to data in S3.

In this integration, Immuta generates policy-enforced views in your configured Google BigQuery dataset for tables registered as Immuta data sources.

External Catalogs

If users have another catalog, or have customized their Collibra or Alation integrations, they can connection through the REST Catalog using the Immuta API.

Users can also connect a Snowflake account to allow Immuta to ingest Snowflake tags onto Snowflake data sources.

External IAMs

External identity managers configured in Immuta allow users to authenticate using an existing identity management system and can optionally be used to synchronize user groups and attributes into Immuta.

Feature, Audit, and Policy Support

Feature Support

The table below outlines the features supported by each of Immuta's integrations.

Audit Support Matrix

The table below outlines the audit support by each of Immuta's integrations and what information is included in the audit logs.

Legend:

• Limited support: There is limited support for audit for this integration.

Policy Support Matrix

Certain policies are unsupported or supported with caveats, depending on the integration:

*Supported with Caveats:

• On Databricks data sources, joins will not be allowed on data protected with replace with NULL/constant policies.

• On Trino data sources, the Immuta functions @iam and @interpolatedComparison for WHERE clause policies can block the creation of views.

The Immuta UI allows users to share, access, and analyze data from one secure location efficiently and easily. This section of documentation introduces all Immuta users to pages and basic features found in the Immuta console.

Immuta UI pages and features

• Data:

  • : Create, manage, and subscribe to data sources.

  • : Combine data sources, work under specified purposes, and collaborate with other users.

• : Manage user roles, groups, and attributes.

• : Manage global policies and view all policies and the data sources they apply to.

• : Configure purposes, run governance reports, and view notifications.

• : Analyze how data is being used across your organization.

• : Write, modify, and execute queries against data sources you're subscribed to in the Immuta UI.

• : Configure Immuta to meet your organization's needs.

• : View access requests and receive activity updates.

• : Manage username and password, access SQL credentials, and generate API keys.

The data sources page allows Immuta users to view, subscribe to, and create data sources in Immuta. On the main data source page is a list of data sources. Users can navigate between the All Data Sources tab and the My Data Sources tab to filter this list. Additionally, the Search bar can be used to filter search results by data source name, tag, project, connection strings, or columns.

To navigate to a specific data source, click on it from this list, and you will be taken to the data source overview page

Data source details page

In addition to the data source's health, this page provides detailed information about the data source and is organized by tabs across the top of the page: , , , , , , and . The visibility and appearance of the tabs will vary slightly depending on the type of user accessing the data source.

Data source health

This section includes detailed information regarding Data Source Health and Data Source Health Checks. The health status of a data source is visible in the top right corner of the data source details page.

If you click the health status text, a dropdown menu displays the status of specific data source checks.

• Health Check: When an Immuta data source is created, a background job is submitted to compute the row count and high cardinality column for the data source. This job uses the connection information provided at data source creation time. A data source initially has a health status of “healthy” because the initial health check performed is a simple SQL query against the source to make sure the source can be queried at all. After the background job for the row count/high cardinality column computation is complete, the health status is updated. If one or both of those jobs failed, the health status will change to “Unhealthy.”

• Fingerprint: Captures summary statistics of a data source when a data source is created, when a policy is applied or changed, or when a user manually updates the data source fingerprint.

• View: Depending on the integration, this records if a view has been created to represent the data source in an integration, when it was created, and gives a button to re-create the view if policies have been changed.

• Row Count: Calculates the number of rows in the data source.

• High Cardinality: Calculates the high cardinality columns, which contain unique values such as identification numbers, email addresses, or usernames. A high cardinality column is required to generate consistent random subsets of data for use in certain minimization techniques.

• Global Policies Applied: Verifies that relevant Global Policies are successfully applied.

• : Detects when a new table has been added in a remote database and automatically creates a new data source. Correspondingly, if a remote table is removed, that data source will be disabled in the console. Schema detection is set to run every night.

• : Detects when a column has been added or removed in a remote database and automatically updates the data source in Immuta. This detection is set to run every night, but users can manually trigger the job here.

Overview Tab

This tab includes detailed information about the data source, including its Description, Technology, Table Name, Remote Database, Remote Table, the Parent Server, and the Data Source ID.

Members Tab

This tab contains information about the users associated with the data source, their username, when their access expires, what their role is, how they are subscribed to the data source, and an Actions button that details the users' subscription history, including the reason users need access to the data and how they plan to use it.

Members can be filtered by Role or Subscription using the Filters button.

Policies Tab

This tab lists the policies associated with the data source and includes three components:

• Subscribers: Lists who may access the data source. If a Subscription Policy has already been set by a Global Policy, a notification and a Disable button appear at the bottom of this section. Data Owners can click the Disable button to make changes to the Subscription Policy.

• Activity Panel: Records all changes made to policies by Data Owners or Governors, including when the data source was created, the name and type of the policy, when the policy was applied or changed, and if the policy is in conflict on the data source. Global policy changes are identified by the Governance icon; all other updates are labeled by the Data Sources icon.

Data Dictionary tab

The Data Dictionary is a table that details information about each column in a data source. The information within the Data Dictionary is generated automatically when the data source is created if the remote data platform supports SQL. Otherwise, Data Owners or Experts can manually create Data Dictionaries. The Data Dictionary tab includes three sections:

• Name: The name of the column in the table.

• Type: The type of value, which may be text, integer, decimal, or timestamp.

• Actions: Users may use the buttons in this column to edit, comment, or tag items in the Data Dictionary.

Discussions tab

Users are able to comment on or ask questions about the Data Dictionary columns and definitions, public queries, and the data source in general. Resolved comments and questions are available for review to keep a complete history of all the knowledge sharing that has occurred on a data source.

Contacts tab

Contact information for Data Owners is provided for each data source, which allows users to ask questions about accessibility and attributes required for viewing the data.

Relationships tab

This tab lists all projects, derived data sources, or parent data sources associated with the data source and includes the reason the data source was added to a project, who added the data source to the project or created it, and when the data source was added to the project or created.

Tasks tab

When users submit an Unmask request in the UI, a Tasks tab appears beside the Relationships tab for the requesting user and the user receiving the request. This tab contains information about the request and allows users to view and manage the tasks listed.

For Immuta to enforce policies, it needs to catalog the resources policies are being applied to by performing metadata ingestion. Metadata ingestion is the process that occurs when you where Immuta gathers details about your tables. However, Immuta does not need access to the data within the tables in order to protect it, with the exception of a few specific and advanced masking policies detailed below.

Immuta Metadata Database

Immuta collects and stores the following kinds of information in Immuta's Metadata Database for policy enforcement. Further, policy information may be transmitted to data source host systems for enforcement purposes as part of a query or to enable the host system to perform native enforcement.

• Identity Management Information: Usernames, group information, and other kinds of personal identifiers may be stored and referenced for the purposes of performing authentication and access control and may be retained in audit logs. When such information is relevant for access determination under policy, it may be retained as part of the policy definition.

• Schema Information: Data source metadata such as schema, column data types, and information about the host.

Immuta's Metadata Database can also contain the following forms of metadata for policy enforcement. These forms contain sample data from your tables and if you do not want Immuta to have access to the data being protected.

• Fingerprints: When enabled, additional statistical queries made during the health check are distilled into summary statistics, called fingerprints. During this process, statistical query results and data samples (which may contain PII) are temporarily held in memory by the Fingerprint Service.

• k-Anonymization Policies: When a k-anonymization policy is applied, the columns under the k-anonymization policy are queried within a separate fingerprinting process which generates rules enforcing k-anonymity. The results of this query, which may contain PII, are temporarily held in memory by the Fingerprint Service. The final rules are stored for enforcement.

• Randomized Response Policies: If the list of substitution values for a categorical column is not part of the policy specification (e.g., when specified via the API), a list is obtained via query and merged into the policy definition.

Metadata Ingestion Workflow

If no metadata collection types have been disabled, data is processed in the following workflow to support data source creation, health checks, policy enforcement, and dictionary features.

1. A System Administrator configures the integration in Immuta.

2. A Data Owner registers data sources from their remote data platforms with Immuta. Note: Data Owners can see sample data when editing a data source. However, this action requires the database password, and the small sample of data visible is only displayed in the UI and is not stored in Immuta.

3. When a data source is created or updated, the Metadata Database pulls in and stores statistics about the data source, including row count and high cardinality calculations.

4. The data source health check runs daily to ensure existing tables are still valid.

5. If an external catalog is enabled, the daily health check will pull in data source attributes (e.g., tags and definitions) and store them in the Metadata Database.

Immuta Privileges

Immuta requires certain privileges to perform metadata ingestion. The user connecting a table to Immuta as a data source must have privileges specific to their data platform to perform metadata ingestion.

For example, a user registering a Snowflake table as an Immuta data source must have the REFERENCES privilege to view the structure of the table and allow Immuta access to that information as well. This does not require the user (or Immuta) to have access to view the data itself.

Purposes tab

Governors manage purposes for data use across Immuta. After creating a purpose, governors can customize acknowledgement statements that users must agree to before accessing a project or data source. Project owners also have the ability to create purposes that will populate on the purposes tab of the governance page.

Reports tab

Governors can build reports to analyze how data is being used and accessed across Immuta using this report builder. Reports can be based on users, groups, projects, data sources, tags, purposes, policies, and connections within Immuta.

For detailed information on how to run reports, see Immuta reports.

Notifications tab

This tab contains a list of all activity associated with the governor, data sources, and global and local policies.

Tags tab

This tab contains a list of all tags within the Immuta environment. This includes built-in Immuta tags, tags created by governors, and tags imported from an external catalog. These tags can then be applied to projects, data sources, and the data dictionary by governors, data owners, or data source experts.

Governors can click on the tags listed here to open up a tag details page. This details page has an overview tab with information about the tag's description, origin, and creation. It also includes a data sources tab that lists the data sources the tag has been applied to and information about its application. The tag details page also includes a columns tab with the columns the tag has been applied to and information about its application, like the other tags applied to that column.

For more information on tags, see the Tags in Immuta page.

The Immuta people page is visible only to user administrators; the following actions can be completed on the Immuta people page:

• Create, manage, and delete users.

• Add or delete permissions from users and groups.

• Add or delete attributes from users and groups.

• Create, manage, and delete groups.

Users tab

On this tab, administrators can add users, filter the list of users, or navigate to users' profiles by clicking on their name.

After clicking on an individual user from this list, the user's email, position, and last login and update appear. From here, admins can manage the user's permissions, attributes, and groups.

Groups tab

Similar to the Users tab, the Groups tab includes a list of groups. After clicking on a specific group, administrators can view the group details, add and remove group members, and manage attributes for the group.

All activity in Immuta is audited. This process provides rich audit logs that detail who subscribes to each data source, why they subscribe, when they access data, what SQL queries and blob fetches they run, and which files they access. Audit logs can be used for a number of intentions, including insider threat surveillance and data access monitoring for billing purposes. Audit logs can also be shipped to your enterprise auditing capability.

For more details about using audit logs, see the Audit Logs User Guide.

Audit Log Transport Details

Immuta's logging system is designed to easily connect with enterprise log collection and aggregation systems. Please see the Immuta System Audit Logs page for full details.

Audit UI Functionality

Immuta provides access to all of the audit logs via the Audit page.

Only users with the AUDIT permission can access this page. See the Administration section for more information.

Users can sort these logs by ascending (oldest entries first) or descending (latest entries first) order. By default, 50 log entries are displayed to a page, but that can be changed to 100 or 200. Additionally, users can filter the entries in a variety of ways, including by project purpose, blobId, remote query id, the entry timestamp, data source, project, record type, user, and SQL query. These query audit records detail the query run, the columns that were masked, and how the masking was enforced.

The user profile page contains personal information your user account, including contact information, API keys, and pending requests. To navigate to the user profile page or quick actions, click the profile icon in the header of the Immuta UI and select Profile..

User details

The following information about the user is displayed on their profile page. With the exception of the Databricks, Redshift, Snowflake, or Synapse username, this information may be edited by the user at any time.

• Name: The user's full name.

• Email: The user's email address.

• Position: The user's current position.

• Last Updated: The time of the user's last profile update.

• About: A short description about the user.

• Location: The user's work location.

• Organization: The organization that a user is associated with.

• Phone Number: The user's phone number.

• Databricks Username: The user's Databricks username. Only an admin may set this field.

• Redshift Username: The user's Redshift username. Only an admin may set this field.

• Snowflake Username: The user's Snowflake username. Only an admin may set this field.

• Synapse Username: The user's Synapse username. Only an admin may set this field.

• Receive System Notifications as Emails: The user can opt to receive email notifications.

SQL credentials tab

API key tab

API keys allow for a secure way to communicate with the Immuta REST API without requiring the username and password. Each key can be revoked at any time and new ones generated. Once a key is revoked it can no longer be used to access the REST API, and users will need to authenticate any tool that they were using with the revoked API key with a new key.

Once in the API keys tab, a user can generate API keys or revoke API keys.

An API key can be linked to a project. By linking an API key to a project, you will be limiting that API key's visibility to only data sources associated with that project.

Requests tab

The requests tab allows users to view and manage all pending access requests directly from their profile page.

Audience: All Immuta users

Content Summary: Notifications in the Immuta UI fall into two categories: Access Requests and Activity. This page illustrates these basic Notification features in the Immuta UI.

Access Request Notifications

Request notifications alert Data Owners that users wish to subscribe to their data sources.

• Users can view their request notifications by clicking on the cell phone icon in the top right corner of the Immuta Console.

• After clicking on the icon, Data Owners can grant or reject requests directly in the notifications drop-down.

• Users will see their pending access requests in the same dropdown.

Activity Notifications

Activity notifications are used to alert users to actions that other users have performed within Immuta. The activity requests that each user receives depend on their permissions and responsibilities.

• Data Users: Data Users receive activity notifications when Data Owners accept or deny their pending access requests.

• Data Owners: Data Owners receive notifications about activity in their data sources and projects and when users query their data sources that have policies enforced. These notifications are shown when the user selects the bell icon in the upper righthand corner.

• Governors: Governors receive notifications for all data source activity, including policy updates within Immuta. These notifications are shown when the user selects the bell icon in the upper right-hand corner.

• Administrators: Administrators receive notifications for user, group, and attribute activity, such as when a new user is created or when an attribute is added to a group. These notifications are shown when the user selects the bell icon in the upper right-hand corner.

For an extensive list of notifications, see the .

Email Notifications

Users can subscribe to email notifications by completing the following steps:

1. Navigate to the User Profile page, and select Edit from the dropdown menu in the top right corner of the user profile information panel.

2. Select the Receive System Notifications as Emails checkbox at the bottom of the window that appears.

3. Click Save.

Once this setting is enabled, Immuta will compile notifications and distribute these compilations via email at 8-hour intervals.

Run Queries in the Query Editor

The allows users to write, modify, and execute queries against data sources they are subscribed to.

Preview Sample Data

1. Click the Query Editor icon in the left sidebar.

2. Select a data source in the Tables list.

3. Click the dropdown menu icon next to the data source and select Preview Sample Data, or click Preview Sample Data in the Table Schema panel.

4. View data in the Results panel.

   • Filter results by clicking the overflow menu next to the column name.

   • Rearrange and resize columns by clicking and dragging.

   • Run and export full results or export current results to .csv by clicking one of the corresponding download buttons in the top right corner of the table.

Write and Execute Queries

1. Click the Query Editor icon in the left sidebar.

2. Write your query in the Query Editor panel.

3. Execute your query by clicking the Run Query button. Note: Clicking this button will only run the currently highlighted query. Queries (or portions of queries) can be executed by manually highlighting the query (or portion of the query) and clicking Run Query.

4. View data in the Results panel.

   • Filter results by clicking the overflow menu next to the column name.

   • Rearrange and resize columns by clicking and dragging.

   • Export results to .csv by clicking the download button in the top right corner of the table.

Audience: Data Owners, Data Users, and Data Governors

Content Summary: Projects allow users to collaborate in their data analysis by combining data sources and providing special access to data for project members. Projects are created, managed, and joined from the Projects page.

This page highlights the major features of the Projects page. For conceptual details or specific tutorials, click the links below or navigate to .

Project Details Page

This page lists all the public projects available to be joined by others in the All Projects tab and all projects users own or belong to are listed in the My Projects tab. Additionally, users with the CREATE_PROJECT permission can create a new project from this page.

To view details about a specific project, users click the project name.

Overview Tab

After navigating to a specific project from the Projects page, the following information about the project is visible to users on the Overview tab:

• Project Details: Information about the project appears in the sidebar on the left of the Overview tab. Details include when the project was created, the purposes associated with the project, a description of the project, the project ID, and credentials.

• Documentation: If Project Owners choose, they may add documentation about their project, which will appear in this section to viewers. If no additional documentation about the project is added, only the project name will appear here.

• Data Sources: The data sources associated with the project are listed here. Users can click on individual data sources to view the reason why it was added to the project and they can navigate to the data source itself. Project Owners can also manage their project data sources in this section.

• Tags: Tags associated with the data source are listed here. Project Owners can manage tags from this section.

• Activity Panel: All activity associated with with the project is listed in the sidebar on the right of the screen. Information recorded here includes who added data sources and tags to the project, members who have been added and removed from the project, and policy updates to the project.

Members Tab

This page includes a list of project members, their contact information and role, how they are subscribed, and when their membership expires. From this page, Project Owners can add and remove members from the project.

Members can be filtered by Role or Subscription using the Filters button.

Policies Tab

This tab allows Project Owners to choose who may request access to their project or whether or not their project is visible at all to users who are not project members.

The Project Equalization section enables Project Owners to level all members' access to data so that data appears the same to all project members, regardless of their individual attributes or groups.

The Subscribers section allows Project Owners to make their project open to anyone, to users who request and are granted access, to users with specified groups and attributes, or only to users the Project Owners manually add.

Discussions Tab

Project members can view, create, reply to, delete, and resolve discussion threads in this tab.

Data Sources Tab

A list of data sources within the project appears in this tab. Project members can view, comment on, and add data sources to the project here as well. Any project member can add data sources to the project, unless the Allow Masked Joins or Project Equalization features are enabled; in those instances, only Project Owners can add data sources to the project.

Audience: Application Administrators

Content Summary: The App Settings Page is visible only to Application Administrators and allows them to , to , and to .

Settings Tab

This tab is where the Administrator can add IAMs, external catalogs, and data providers. They can also adjust various Immuta settings to configure it better to their organization's needs.

For a tutorial on changing settings on this tab see .

Licenses Tab

This tab includes a list of licenses and details the universally unique identifier (UUID), the features associated with specific licenses, the expiration dates, the total number of seats, and the date the keys were added. Administrators can also .

System Status Tab

This tab allows Administrators to export a zip file called the Immuta status bundle. This bundle will include information helpful to assess and solve issues within an Immuta instance by providing a snapshot of Immuta, associated services, and information about the remote source backing any of the selected Data Sources. When generating the status bundle the Administrator may select the particular information that will help solve the issue at hand.

Audience: System Administrators

Content Summary: This page outlines how to configure an external metadata database for Immuta instead of using Immuta's built-in PostgreSQL Metadata Database that runs in Kubernetes.

Overview

The Metadata Database can optionally be configured to run outside of Kubernetes, which eliminates the variability introduced by the Kubernetes scheduler and/or scaler without compromising high-availability. This is the preferred configuration, as it offers infrastructure administrators a greater level of control in the event of disaster recovery.

Helm Configuration

1. Enable an external metadata database by setting database.enabled=false in the immuta-values.yaml file and passing the connection information for the PostgreSQL instance under the key externalDatabase.

2. Set queryEngine.rehydration.enabled=true. If set to false, then externalDatabase.superuser.username and externalDatabase.superuser.password must be provided.

External Database Object

Additionally, it is possible to use existingSecret instead of setting externalDatabase.password in the Helm values. These passwords map to the same keys that are used for the built-in database. For example,

First-Time PostgreSQL Setup

1. Log in to the external metadata database as a user with the superuser role attribute (such as the postgres user) using your preferred tool (e.g., psql, pgAdmin).

2. Connect to database postgres, and execute the following SQL.

1. Connect to database bometadata that was created in the previous step, and execute the following SQL. Azure Database for PostgreSQL: Extensions must be configured in the web portal.

Migrating from Immuta's Built-In PostgreSQL Database to External Metadata Database

For existing deployments, you can migrate from the built-in database to an external database. To migrate, backups must be configured. Reach out to your Immuta representative for instructions.

1. (Optional) Set default namespace:
2. Trigger manual backup:
3. Validate backup succeeded:
5. Edit immuta-values.yaml to enable the external metadata database and restore.
6. Apply the immuta-values.yaml changes made in the previous step:
7. Wait until the Kubernetes resources become ready.
8. Edit immuta-values.yaml to enable Query Engine rehydration and disable backup/restore.
9. Rerun the previous helm upgrade command to apply the latest immuta-values.yaml changes.

10. Connect to database postgres, and execute the following SQL. Azure Database for PostgreSQL: Delete the previously created role by running DROP ROLE postgres;.

Prerequisites

• 1.16 or greater

• 3.2 or greater

1 - Configure the Environment

1.1 - Check Helm Version

Immuta's Helm Chart requires Helm version 3+.

Run helm version to verify the version of Helm you are using:

1.2 - Configure Immuta's Helm Chart Repo

In order to deploy Immuta to your Kubernetes cluster, you must be able to access the Immuta Helm Chart Repository and the Immuta Docker Registry. You can obtain credentials from your Immuta support professional.

Run helm repo list to ensure Immuta's Helm Chart repository has been successfully added:

Example Output

Run kubectl get secrets to confirm your Kubernetes image pull secret is in place:

1.3 - Check/Update Your Local Immuta Helm Chart Version

Run helm search repo immuta to check the version of your local copy of Immuta's Helm Chart:

Update your local Chart by running helm repo update.

To perform an upgrade without upgrading to the latest version of the Chart, run helm list to determine the Chart version of the installed release, and then specify that version using the --version argument of helm repo update.

2 - Configure Immuta Helm Values

1. Once you have the Immuta Docker Registry and Helm Chart Repository configured, download the immuta-values.yaml file. This file is a recommended starting point for your installation.

2. Modify immuta-values.yaml based on the determined configuration for your Kubernetes cluster and the desired Immuta installation. You can change a number of settings in this file, such as
3. Replace the placeholder password value "<SPECIFY_PASSWORD_THAT_MEETS_YOUR_ORGANIZATIONS_POLICIES>" with a secure password that meets your organization's password policies.

Avoid these special characters in generated passwords

whitespace, $, &, : , \, /, '

3 - Configure Persistence

If you would like to disable persistence to disk for the database and query-engine components, you can do so by configuring database.persistence.enabled=false and/or queryEngine.persistence.enabled=false in immuta-values.yaml. Disabling persistence can be done for test environments. However, we strongly recommend against disabling persistence in production environments as this leaves your database in ephemeral storage.

By default, database.persistence.enabled and queryEngine.persistence.enabled are set to true and request 120Gi of storage for each component. Recommendations for the Immuta Metadata Database storage size for POV, Staging, and Production deployments are provided in the immuta-values.yaml as shown below. However, the actual size needed is a function of the number of data sources you intend to create and the amount of logging/auditing (and its retention) that will be used in your system.

While the Immuta query engine persistent storage size is configurable as well, the default size of 20Gi should be sufficient for operations in nearly all environments.

4 - Configure Backup and Restoration Values

At this point this procedure forks depending on whether you are installing with the intent of restoring from a backup or not. Use the bullets below to determine which step to follow.

• If this is a new install with no restoration needed, follow Step 4.1.

4.1 - Initial Immuta Kubernetes Installation -- No Backup Restoration

Immuta's Helm Chart has support for taking backups and storing them in a PersistentVolume or copying them directly to cloud provider blob storage including AWS S3, Azure Blob Storage, and Google Storage.

To configure backups with blob storage, reference the backup section in immuta-values.yaml and consult the subsections of this section of the documentation that are specific to your cloud provider for assistance in configuring a compatible resource. If your Kubernetes environment is not represented there, or a workable solution does not appear available, please contact your Immuta representative to discuss options.

If using volumes, the Kubernetes cluster Immuta is being installed into must support PersistentVolumes with an access mode of ReadWriteMany. If such a resource is available, Immuta's Helm Chart will set everything up for you if you enable backups and comment out the volume and claimName.

4.2 - Restore From Backup - Immuta Kubernetes Re-Installation

If using the volume backup type, an existing PersistentVolumeClaim name needs to be configured in your immuta-values.yaml because the persistentVolumeClaimSpec is only used to create a new, empty volume.

If you are unsure of the value for <YOUR ReadWriteMany PersistentVolumeClaim NAME>, the command kubectl get pvc will list it for you

5 - Set Replicas and Resource Limits

Adhering to the guidelines and best practices for replicas and resource limits outlined below is essential for optimizing performance, ensuring cluster stability, controlling costs, and maintaining a secure and manageable environment. These settings help strike a balance between providing sufficient resources to function optimally and making efficient use of the underlying infrastructure.

5.1 - Configure Replicas

Set the following replica parameters in your Helm Chart to the values listed below:

5.2 - Set Resource Limits

Add this YAML snippet to your Helm values file:

Database is the only component that needs a lot of resources, especially if you don’t use the query engine. For a small installation, you can set the database resources to 2Gi, and if you see slower performance over time, you can increase this number to improve performance.

Setting CPU resources and limits is optional. Resource contention over CPU is not a common occurrence for Immuta, so it won’t have a significant effect if the CPU resource and limit is set.

6 - Deploy Immuta

Run the following command to deploy Immuta:

Advanced Installations

Manage TLS

HTTP communication using TLS certificates is enabled by default in Immuta's Helm Chart for both internal (inside the Kubernetes cluster) and external (between the Kubernetes ingress and the outside world) communications. This is accomplished through the generation of a local certificate authority (CA) which signs certificates for each service - all handled automatically by the Immuta installation. While not recommended, if TLS must be disabled for some reason, this can be done by setting tls.enabled to false in the values file.

Use Your Own TLS Certificate(s)

Using your own certificates requires you to create a Kubernetes Secret containing the private key, certificate, and certificate authority certificate. This can be easily done using kubectl:

After creating the Kubernetes Secret, specify its use in the external ingress by setting tls.externalSecretName = immuta-external-tls in your immuta-values.yaml file:

Using Argo CD

For Argo CD versions older than 1.7.0 you must use the following Helm values in order for the TLS generation hook to run successfully.

Starting with Argo CD version 1.7.0 the default TLS generation hook values can be used.

tls.manageGeneratedSecret must be set to true when using Argo CD to deploy Immuta; otherwise, the generated TLS secret will be shown as OutOfSync (requires pruning) in Argo CD. Pruning the Secret would break TLS for the deployment, so it is important to set this value to prevent that from happening.

Troubleshoot

Software Versions

• 3.2 or greater

• : See the for a list of versions Immuta supports.

Helm and Immuta's Helm Chart

Storage

Database Backups

Database backups for the metadata database and Query Engine may be stored in either cloud-based blob storage or a Persistent Volume in Kubernetes.

Cloud Storage (Recommended)

Backups may be stored using one of the following cloud-based blob storage services:

• AWS S3

  • Supports authentication via AWS Access Key ID / Secret Key, IAM Roles via kube2iam or kiam, or IAM Roles in EKS.

• Azure Blob Storage

  • Supports authentication via Azure Storage Key, Azure SAS Token, or Azure Managed Identities.

• Google Cloud Storage

  • Supports authentication via Google Service Account Key

Database Persistence

When database persistence is enabled, Immuta requires access to PersistentVolumes through the use of a persistent volume claim template. These volumes should normally be provided by a block device, such as AWS EBS, Azure Disk, or GCE Persistent Disk.

Additionally, when database persistence is enabled, Immuta requires the ability to run an initContainer as root. When PodSecurityPolicies are in place, service accounts must be granted access to use a PodSecurityPolicy with the ability to RunAsAny user.

RBAC

The Immuta Helm Chart supports RBAC and will try to create all needed RBAC roles by default.

Ingress

Immuta needs Ingress for two services:

1. Immuta Web Service (HTTP)

2. Immuta Query Engine (TCP)

Worker Node Size

Immuta’s suggested minimum node size has 4 CPUs and 16GB RAM. The default Immuta Helm deployment requires at least 3 nodes.

TLS Certificates

Internal HTTPS communication refers to all communication between Immuta services. External HTTPS communication refers to communication between clients and the Immuta Query Engine and Web Service, which is configured using a Kubernetes Ingress resource.

Audience: System Administrators

Content Summary: This page outlines instructions for troubleshooting specific issues with Helm.

Pods Frozen with the Status "Pending" or "Init:0/1"

If you encounter Immuta Pods that have had the status Pending or Init:0/1 for an extended period of time, then there may be an issue mounting volumes to the Pods. You may find error messages by describing one of the pods that had the Pending or Init:0/1 status.

If an event with the message pod has unbound PersistentVolumeClaims is seen on the frozen pod, then there is most likely an issue with the database backup storage Persistent Volume Claims. Typically this is caused by the database backup PVC not binding because there are no Kubernetes Storage Classes configured to provide the correct storage type.

Helm Did Not Cleanup Immuta Kubernetes Resources

Occasionally Helm has bugs or loses track of Kubernetes resources that it has created. Immuta has created a Bash script that you may download and use to cleanup all resources that are tied to an Immuta deployment. This command should only be run after helm delete <YOUR RELEASE NAME>.

Solution

Download cleanup-immuta-deployment.sh.

Run the script:

Rolling-Restart for Database Pods

After a configuration change or cluster outage you may need to perform a rolling-restart to refresh the database pods without data loss. Use the command below to update a restart annotation on the database pods to instruct the database StatefulSet to roll the pods.

Rolling-Restart for Web Pods

After a configuration change or cluster outage you may need to perform a rolling-restart to refresh the web service pods. Use the command below to update a restart annotation on the web pods to Deployment to roll the pods.

Regenerating Internal TLS Certificates

Should the need arise that you need to regenerate internal TLS certificates follow the instructions below.

Rotating External TLS Certificates

Should you need to rotate external TLS certificates, follow the instructions below:

Helm Upgrade Process

1. Check Environment

2. Determine Upgrade Path:

   • Method A: Basic Helm Upgrade (Recommended)

   • Method B: Complete Backup and Restore

1 - Check Environment

1.1 - Check Helm Version

• Immuta's Helm Chart requires Helm version 3+.

• New installations of Immuta must use the latest version of Helm 3 and Immuta's latest Chart.

Run helm version to verify the version of Helm you are using:

version.BuildInfo{Version:"v3.0.2", GitCommit:"19e47ee3283ae98139d98460de796c1be1e3975f", GitTreeState:"clean", GoVersion:"go1.13.5"}

Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}

1.2 - Configure Immuta's Helm Chart Repo

In order to deploy Immuta to your Kubernetes cluster, you must be able to access the Immuta Helm Chart Repository and the Immuta Docker Registry. You can obtain credentials from your Immuta support professional.

Run helm repo list to ensure Immuta's Helm Chart repository has been successfully added:

NAME            URL
stable          https://kubernetes-charts.storage.googleapis.com
local           http://127.0.0.1:8879/charts
immuta          https://archives.immuta.com/charts

Run kubectl get secrets to confirm your Kubernetes image pull secret is in place:

NAME                  TYPE                                  DATA   AGE
immuta-registry       kubernetes.io/dockerconfigjson        1      5s

1.3 - Check/Update Your Local Immuta Helm Chart Version

Run helm search repo immuta to check the version of your local copy of Immuta's Helm Chart:

NAME          CHART VERSION APP VERSION DESCRIPTION
immuta/immuta 4.4.1         2.7.0       The Immuta

Update your local Chart by running helm repo update.

To perform an upgrade without upgrading to the latest version of the Chart, run helm list to determine the Chart version of the installed release, and then specify that version using the --version argument of helm repo update.

1.4 - Confirm Connectivity with Your Current Immuta Helm Installation

Run helm list to confirm Helm connectivity and verify the current Immuta installation:

NAME  REVISION  UPDATED                   STATUS    CHART         APP VERSION NAMESPACE
test  1         Tue Dec 17 01:04:36 2019  DEPLOYED  immuta-4.2.3  2.6.0       ns

Make note of:

• NAME - This is the '<YOUR RELEASE NAME>' that will be used in the remainder of these instructions.

• CHART - This is the version of Immuta's Helm Chart that your instance was deployed under.

1.5 - Confirm Access to the Helm Values Used in Your Current Immuta Installation

You will need the Helm values associated with your installation, which are typically stored in an immuta-values.yaml file. If you do not possess the original values file, these can be extracted from the existing installation using:

helm get values <YOUR RELEASE NAME> > immuta-values.yaml

2 - Determine Your Upgrade Path

Select your method:

• Method A - Basic Upgrade (Recommended)

• Method B - Backup and Restore: This method is intended primarily for recovery scenarios and is only to be used if you have been advised to by an Immuta representative. Reach out to your Immuta representative for instructions.

Basic Upgrade (Recommended)

After you make any desired changes in your immuta-values.yaml file, you can apply these changes by running helm upgrade:

helm upgrade <YOUR RELEASE NAME> immuta/immuta \
    --values immuta-values.yaml
    --version <YOUR DESIRED CHART VERSION>

Note: Errors in upgrades can result when upgrading Chart versions on the installation. These are typically easily resolved by making slight modifications of your values to accommodate the changes in the Chart. Downloading an updated copy of the immuta-values.yaml and comparing to your existing values is often a great way to debug such occurrences.

If you are on Kubernetes 1.22+, remove nginxIngress.controller.image.tag=v0.49.3 when upgrading; otherwise, your ingress service may not start after the upgrade.

Audience: System Administrators

Content Summary: Before installing Immuta, you will need to spin up your AKS or ACS cluster. This page outlines how to deploy Immuta cluster infrastructure on AKS and ACS.

If you would like to install Immuta on an existing AKS or ACS cluster, you can skip this section. However, we recommend deploying a dedicated resource group and cluster for Immuta if possible.

Once you have deployed your cluster infrastructure, please visit Helm Installation on Microsoft Azure Kubernetes Service to finish installing Immuta.

Support for AKS and ACS

Create Resource Group

You will need a resource group to deploy your AKS or ACS cluster in:

az group create --name immuta-resource-group --location <your desired region>

Note: There is no naming requirement for the Immuta resource group.

Create VM Cluster

Now it is time to spin up your cluster resources in Azure. This step will be different depending on whether you are deploying an AKS or ACS cluster.

After running the command, you will have to wait a few moments as the cluster resources are starting up.

Create AKS Cluster (Recommended):

az aks create \
--resource-group immuta-resource-group \
--name immuta-cluster \
--node-count 3 \
--node-vm-size Standard_DS3_v2 \
--generate-ssh-keys

Create ACS Cluster (Deprecated):

az acs create \
--orchestrator-type kubernetes \
--resource-group immuta-resource-group \
--name immuta-cluster \
--agent-count 3 \
--agent-vm-size Standard_DS3_v2 \
--generate-ssh-keys

Configure kubectl

You will need to configure the kubectl command line utility to use the Immuta cluster.

Install kubectl through Azure CLI

If you do not have kubectl installed, you can install it through the Azure CLI.

If you are using AKS, run

az aks install-cli

For ACS clusters, run

az acs kubernetes install-cli

Point kubectl to your Immuta cluster

If you are using AKS, run

az aks get-credentials --resource-group immuta-resource-group --name immuta-cluster

For ACS clusters, run

az acs kubernetes get-credentials --resource-group immuta-resource-group --name immuta-cluster

External References

• Microsoft Azure AKS Documentation

• Microsoft Azure ACS Documentation

• Microsoft Azure VM Sizing Documentation

• Microsoft Azure Network Security Group Documentation

Resetting Passwords

The following procedure walks through the process of changing passwords for the database users in the Immuta Database.

The commands outlined here will need to be altered depending on your Helm release name and chosen passwords. Depending on your environment, there may be other changes required for the commands to complete successfully, including, but not limited to, Kubernetes namespace, kubectl context, and Helm values file name.

This process results in downtime.

Helm Values

1. Scale database StatefulSet to 1 replica:

   Copy

   kubectl scale statefulset <release-name>-immuta-database --replicas 1

2. Change database.superuserPassword:

   1. Alter Postgres user password:

      Copy

      kubectl exec <release-name>-immuta-database-0 -- \
        psql -d bometadata -c \
        "ALTER USER postgres WITH ENCRYPTED PASSWORD '<new-password>'

   2. Update database.superuserPassword with <new-password> in immuta-values.yaml.

3. Change database.replicationPassword:

   1. Alter replicator user password:

      Copy

      kubectl exec <release-name>-immuta-database-0 -- \
        psql -d bometadata -c \
        "ALTER USER replicator WITH ENCRYPTED PASSWORD '<new-password>'"

   2. Update database.replicationPassword with <new-password> in immuta-values.yaml.

4. Change database.password:

   1. Alter bometa user password:

      Copy

      kubectl exec <release-name>-immuta-database-0 -- \
        psql -d bometadata -c \
        "ALTER USER bometa WITH ENCRYPTED PASSWORD '<new-password>'"

   2. Update database.password with <new-password> in immuta-values.yaml.

5. Update database.patroniApiPassword with <new-password> in immuta-values.yaml.

6. Run helm upgrade to persist the changes and scale the database StatefulSet up:

   Copy

   helm upgrade --reuse-values <release-name> immuta/immuta

7. Restart web pods:

   Copy

   kubectl rollout restart deployment/<release-name>-immuta-web

Kubernetes Secret

Users have the option to use an existing Kubernetes secret for Immuta database passwords used in Helm installations.

1. Update your existingSecret values in your Kubernetes environment.

2. Get the current replica counts:

   Copy

   DATABASE_REPLICA_COUNT=$(kubectl get statefulset --selector "app.kubernetes.io/component=database" --output=jsonpath='{.items[0].status.replicas}')

3. Scale database StatefulSet to 1 replica:

   Copy

   kubectl scale statefulset <release-name>-immuta-database --replicas 1

4. Change the value corresponding to database.superuserPassword in the existing Kubernetes Secret.

5. Alter Postgres user password:

   Copy

   kubectl exec <release-name>-immuta-database-0 -- \
     psql -d bometadata -c \
     "ALTER USER postgres WITH ENCRYPTED PASSWORD '<new-password>'"

6. Change the value corresponding to database.replicationPassword in the existing Kubernetes Secret.

7. Alter replicator user password:

   Copy

   kubectl exec <release-name>-immuta-database-0 -- \
     psql -d bometadata -c \
     "ALTER USER replicator WITH ENCRYPTED PASSWORD '<new-password>'"

8. Change the value corresponding to database.password in the existing Kubernetes Secret.

9. Alter bometa user password:

   Copy

   kubectl exec <release-name>-immuta-database-0 -- \
     psql -d bometadata -c \
     "ALTER USER bometa WITH ENCRYPTED PASSWORD '<new-password>'"

10. Scale the immuta-database StatefulSet up to the previous replica count determined in the previous step:

    Copy

    kubectl scale statefulset <release-name>-immuta-database --replicas $DATABASE_REPLICA_COUNT

11. Restart web pods:

    Copy

    kubectl rollout restart deployment/<release-name>-immuta-web