Immuta v2024.3 Release Notes

Immuta v2024.3.0

Immuta v2024.3.0 was released September 19, 2024.

New features and enhancements

Policy/entitlements engine

• Domains in general availability: Domains are containers of data sources that allow you to assign data ownership and access management to specific business units, subject matter experts, or teams at the nexus of cross-functional groups. Domains support organizations building a data mesh architecture and implementing a federated governance approach to data security, which can accelerate local, domain-specific decision making processes and reduce risk for the business.

• Masked joins for Snowflake and Databricks Unity Catalog integrations in general availability: This feature allows masked columns to be joined across data sources that belong to the same project, giving users additional capability for data analysis within a project while still securing sensitive data. Sensitive columns can be masked while still allowing users the ability to join on these within a project, helping organizations strike the correct balance between access and security.

• Subscription policies on views in Databricks Unity Catalog integration: Databricks Unity Catalog integration now supports subscription policies (grants) on views. This enhancement allows customers to apply and manage access controls more effectively through subscription policies, streamlining data governance and access management. While subscription policies are now supported, data policies are not possible at this time due to the lack of support for row filters and column masks on views by Databricks.

• Compliance with column length and precision in a Snowflake masking policy: Snowflake requires the outputs of masked columns to comply with the length, scale, and precision of what the Snowflake columns require. To comply with this Snowflake behavior change, Immuta truncates the output values in masked columns to match the Snowflake column requirements so that users' queries continue to complete successfully.

• Data policies on Snowflake Iceberg tables: Users can now apply fine-grained access controls to Snowflake Iceberg tables, making support for Immuta data policies and subscription policies consistent across standard Snowflake table types.

• Rotating the shared secret for Starburst (Trino): Users can rotate the shared secret used for API authentication between Starburst (Trino) and Immuta, which provides improved security management, compliance with organizational policies, and the following benefits:

  • Enhanced security: Regularly update your API credentials to mitigate potential security risks.

  • Compliance support: Meet security requirements that mandate periodic rotation of API keys.

  • Flexibility: Change the shared secret at any time after the initial integration setup.

  Existing integrations will continue to function normally. Downtime is required when rotating the shared secret, so follow the Starburst (Trino) integration API documentation to ensure continuous operation of your integration, and establish a regular schedule for rotating your shared secret as part of your security best practices.

Metadata registry

• Azure Purview data catalog tag ingestion: There is a standard (out-of-the-box) connector for tag ingestion from Azure Purview enterprise data catalog into Immuta. Previously, if tags resided in the Azure Purview data catalog, customers had to build the connector using Immuta's REST Catalog interface.

• Databricks Unity Catalog integration tag ingestion in preview: Customers who have tags defined and applied in Databricks Unity Catalog can seamlessly bring those tags into Immuta to leverage them for attribute based access control (ABAC), data classification, and data monitoring.

  This feature is currently in preview at the design partner level. To use this feature in preview, you must have no more than 2,500 Unity Catalog data sources registered in Immuta. See the design partner description for expectations and details, and then reach out to your Immuta representative to enable this feature.

• Adding a new external catalog integration automatically backfills tags for pre-existing data sources: Prior to this change, users had to manually link pre-existing data sources to the relevant external data catalog entry after a new external data catalog integration was set up, and only newly registered data sources were linked automatically. Now, Immuta triggers an auto-linking process for all unlinked data sources when a new external data catalog integration setup is saved. This change increases the level of automation, reduces cognitive and manual workload for data governors, and aligns external data catalog integration behavior with end user expectations.

• Running table statistics only if required (instead of by default): Table statistics consist of row counts, identification of high cardinality columns, and a sample data fingerprint. Immuta needs to collect this information to support the following data access policy types:

  • Column masking with randomized response

  • Column masking with format preserving masking

  • Column masking with k-anonymization

  • Column masking with rounding

  • Column masking with reversibility

  • Row minimization

  Prior to this change, table statistics were collected for every newly onboarded object by default, unless the object had a Skip_Stats tag applied. Post this change, table statistics are now only collected on a data object once they are required (i.e., if one of the above-mentioned policy types is applied). Even then, the Skip_Stats tag continues to be respected. This change results in performance improvements, as the number of standard operations during data object onboarding is significantly reduced.

• Alation custom fields integration: In addition to Alation standard tags, Immuta’s Alation integration now also supports pulling information from Alation custom fields as tags into Immuta.

  • Alation customers leverage custom fields because it allows them to explicitly control who can modify information associated with that field inside of Alation.

  • Alation standard tags are always modifiable by any user inside of Alation, which can pose a security risk if those were used to control access through Immuta policies.

  • Supporting both Alation tags and custom fields to be integrated into Immuta provides full flexibility to Immuta customers leveraging the Alation enterprise data catalog.

Data discovery and classification

• Simpler user experience for sensitive data discovery: Customizing sensitive data discovery is now easier and quicker with a single entry point for configuration. Instead of navigating to multiple pages in the Immuta application, use a single form to create an identifier for sensitive data and add tags and regex patterns.

• Schema monitoring for object type changes: Schema monitoring for Snowflake and Databricks Unity Catalog supports detecting and automatically reapplying policies on data sources that have changed their object type (for example, a VIEW that was changed into a TABLE or vice versa).

• Reduced the number of validation tasks for data owners from new data sources and columns found by schema monitoring: When schema monitoring is enabled, Immuta applies a New tag whenever a new data source is added or its columns change. This allows governors to create policies that automatically apply to all new data sources and columns (such as masking new data by default).

  • Previously, data owners were always asked to validate data source requests (which in turn removes the New tag) related to data source and column changes, even if there was no actual policy present targeting the New tag.

  • Now, data owners are only asked to validate data source requests if an actual policy is present that targets the New tag. Otherwise the validation request for data owners gets skipped.

  • As a result, in the absence of a relevant policy, data owners will now have fewer data source requests to validate which saves them time and increases efficiency.

• Schema monitoring enhancement for Databricks Unity Catalog: Schema monitoring for Databricks Unity Catalog now supports detecting and automatically reapplying policies on destructively recreated tables (from CREATE OR REPLACE statements), even if the table schema itself wasn’t changed.

• Governance permission required for Discover: The Discover UI for managing automated data identification and classification is only accessible to users with the GOVERNANCE permission in the Immuta application. Previously, Immuta users with permission to create data sources could also access the settings in the Discover UI.

• Removed the overview tab on identification frameworks: Under Discover, each identification framework now has two tabs: identifiers and data sources. Prior to this change, there was an overview tab that linked to the other two tabs. When clicking into an identification framework, you now land directly on the identifiers tab.

Unified audit

• New domain level permission - Audit Activity: For customers who use domains to define data products, the Audit Activity domain permission allows data product owners to review query activities of the data sources they manage using rich visualizations and dashboards. Without Immuta, customers would have to implement their own permission model if they wanted to allow specific users to only see query events related to certain selected tables (principle of least privilege). Immuta’s domain-scoped audit permission is an enabler to help open up access to query information to more users across the enterprise while staying compliant.

• Support role-based access for S3 audit export: Audit export supports AWS IAM authentication. Customers can use AWS assumed role-based authentication or access key authentication to secure access to S3 to export audit events.

• Released Immuta CLI v1.4.0: A new version of the CLI was released that includes new support for AWS IAM role authentication for audit export to S3 and some CLI breaking changes. See the CLI release note for more details.

• Trino universal audit model available with Trino 435 using the Immuta Trino plugin 435.1: For customers who are using EMR 7.1 with Trino 435.1 and have audit requirements, the Immuta Trino 435.1 plugin now supports audit in the universal audit model. The Immuta Trino 435.1 plugin audit information is on par with the Immuta Trino 443 plugin. The Immuta Trino 435.1 plugin is supported on SaaS and 2024.2 and newer.

• The Immuta Starburst (Trino) integration supports additional query audit metadata enrichment including the object accessed during the query event: Immuta query audit events for Starburst (Trino) will include the following information.

  • Object accessed: The tables and columns that were queried

  • Tags: The Immuta table and column tags, including data catalog tags synchronized to Immuta, for queried tables and columns

  • Sensitivity classification: The columns' sensitivity in context of other queried columns if an Immuta classification framework is enabled at the time of audit event processing

  • Query duration: The amount of time it took to execute the query in seconds

  • Database name: The name of the Starburst (Trino) catalog

• Data owners can now see audit events for the data sources that they own without having the AUDIT Immuta permission: Data owners can see query events for their data sources on the audit page, data overview page, data source pages, and the data source activity tab. They can also inspect Immuta audit events on the audit page and activity tab for the data sources they own. This enhancement gives data owners full visibility of activity in the data sources they own.

Deprecations and breaking changes

Deprecation announcements

Deprecated items remain in the product with minimal support until their end of life date.

Feature	Deprecation notice	End of life (EOL)
CREATE_FILTER permission	2024.3	2024.4
External policy handler	2024.3	2024.4
Policy exemptions	2024.3	2024.4
Quick create tab	2024.3	2024.4
Unmask requests	2024.3	2024.4
Conditional tags	2024.3	2024.4

Removed features

Feature	Deprecation notice	End of life (EOL)
Query text has been removed from all legacy audit records. Instead, use UAM events, which by default contain query text.	2023.3	2024.4

Breaking changes

Bug fix with breaking API change: Only users with the CREATE_DATA_SOURCE permission are authorized to use the POST api/v2/data endpoint. Users without that permission will be blocked and get a 403 status returned.

{
    "statusCode": 403,
    "error": "Forbidden",
    "message": "You must have the \"CREATE_DATA_SOURCE\" permission."
}

Immuta Enterprise Helm chart

• New job orchestrator - Temporal: Immuta is migrating all background job processing to Temporal. Temporal is a durable execution engine that will allow Immuta to secure data at scale with fewer disruptions. Temporal requires a connection to PostgreSQL and will be deployed in 2024.3.0 by default.

• New component - Detect Temporal Worker: The detect-temporal-worker is the first Temporal worker that Immuta is introducing to the Immuta Enterprise Helm chart. This component is primarily used to support Immuta's audit record export feature.

• New Feature Flag configuration: Feature flags are now managed under the global.featureFlags Helm value instead of using environment variables.

• Migration to ocir.immuta.com from registry.immuta.com: Container images for 2024.3.0 and beyond only exist in the ocir.immuta.com registry, and this is now the default registry for the Immuta Enterprise Helm chart. The legacy registry at registry.immuta.com will reach end of life when Immuta 2024.4.0 is released.

Bug fixes

• Fix for external tag ingestion related to Collibra Output Module API behavioral change: Incorrect filters were being passed to Collibra’s Output Module API when fetching column tag information. This resulted in a failed API request while linking or refreshing Collibra tags on a data source. Collibra’s Output Module API began performing additional request validation on approximately May 6, 2024, which indicated a problem. This fix ensures that the Collibra tag ingestion integration in Immuta is reflecting these changes. Without it, there was a residual risk that some incorrect column tags would get ingested.

• Snowflake External OAuth: The form field Client Secret stopped being displayed in the UI for Snowflake data source registration, which led customers to believe that Snowflake External OAuth using client secret was no longer a supported authentication mechanism. This fix reintroduced the client secret field in the UI. Customers who had already registered data sources with Snowflake External OAuth previously via the UI, API, or CLI while the bug existed were not affected, since the issue only affected the UI but not the backend or programmatic interfaces.

v2024.3 migration notes

• You must be on Immuta version 2024.2 or newer to migrate directly to 2024.3.

• You must migrate from registry.immuta.com to ocir.immuta.com to access the 2024.3 container images and Immuta Enterprise Helm chart.

  • The legacy registry at registry.immuta.com will reach end of life when Immuta 2024.4.0 is released.

• You must migrate feature flags set using secure.extraEnvVars to global.featureFlags or you will see warning messages from Helm. (Deployment will not be impacted if not updated.)

  • AuditService feature flag now defaults to true; it no longer needs to be set.

  • detect feature flag now defaults to true; it no longer needs to be set.

  • auditLegacyViewHide feature flag now defaults to true; it no longer needs to be set.

• 6 new Kubernetes deployments have been added to the Immuta Enterprise Helm chart:

  • 5 new pods for Temporal

  • 1 new pod for the Detect Temporal Worker