Immuta v2.4.0 Release Notes
Immuta version 2.4.0 was released March 29, 2019.
v2.4.0 New Features
- Complete refacing of the user interface to include changes to the user Administration page.
- Global Policy enhancements:
- All policies available as Local Policies are now available as Global Policies.
- Conflict resolution:
- If two Global Data Policies are applied to the same data source / column, the policy that is deeper in the tag hierarchy tree will be applied.
- If two Global Subscription Policies are applied to the same data source, they will be AND'ed together.
- Support for boolean logic on the tags to apply the policy to and where the policy is applied through ANDing or ORing tags.
- Policies can be applied to columns "with no tag," "any tag," or "all columns."
- A new variable is available when building custom SQL logic for policies: @columnTagged. This variable allows referencing columns in SQL by the tag name rather than the column name.
- Global rounding policies will leverage data source summary statistics to select the bucket size.
- Global Policies that can't be applied (for example, if the custom SQL assumes the wrong column type) will always fall back to a more restrictive policy.
- Derived data source policy inheritance allows users to expose data in an equalized project and inherit all policies from the parent sources appropriately. This feature allows users to expose data without needing to understand the policies to enforce.
- Data fingerprinting:
- Data summary statistics are captured from the data source. Those statistics are available through the Immuta API and drive policies such as differential privacy and format preserving masking.
- Versioning of fingerprints is available in equalized projects. Versions of fingerprints can be compared to view statistical and policy differences.
- Policy enhancements:
- Support for reversible masking as a masking policy.
- Support for format preserving masking as a masking policy.
- Subscriptions based on logic can now be optionally marked as discoverable. (Before only users that met the policy would be able to see the data source.)
- Support for complex boolean logic in Subscription Policies.
- Option to include rationale for why a policy is in place.
- Policies and Members tabs are always visible to all users in data sources and projects, but they are not editable.
- Enhanced time-based policies allow a new option for only showing rows "older than" x.
- Option to only show rows "never" at the end of "Otherwise" only show rows policies.
- When building a rounding masking policy, a bucket size is suggested based on the data summary statistics.
- High availability NameNode support for HDFS data sources.
- Spark enhancements:
- Support for view-based data sources that include multiple tables.
- Support for ORC files.
- "Smart" join support. When joining data from different database technologies in the Immuta PostgreSQL layer, the join will take into account if one side is smaller and poll the larger side with the data from the smaller side of the join.
- Upgraded Immuta to PostgreSQL 11.
- New Tasks tab in the data source screen:
- Requests for a detailed Query EXPLAIN to assist with any debugging issues.
- Requests for data to be unmasked (see next bullet).
- Workflow for requesting values to be unmasked. This feature is only relevant to reversible masking and format preserving masking column policies.
- Support for bulk request access to data sources and bulk approval to data sources.
- Support for requesting access to data sources on behalf of a group.
- Project Enhancements:
- Ability to auto-request access to all data sources in a project.
- Clearly view and understand why users are out of compliance in an equalized project.
- Equalization is allowed with a single user.
- Allowing joining on masked values within projects is turned off by default.
- Support for new databases:
- Saved search filters. This feature allows users with the appropriate permission to save searches for other users to leverage.
v2.4.0 Major Bug Fixes
- Groups in equalized projects are now appropriately handled for out- -of-compliance users in equalized projects.
- Spark and PostgreSQL are now consistent on when/how join on masked values works.
v2.4.0 Known Bugs
v2.4.0 Deprecation Notices
- Removed child data sources. This is covered through derived data source support or Otherwise policies.
- Immuta Spark --> JDBC access to data outside of the Hadoop cluster is disabled by default and must be configured in the Spark session.
v2.4.0 Breaking API Changes
- Replaced data source approve, data source deny, project approve, project deny
POST /subscription/denywhich allow you to send an arbitrary number of subscriptions in the payload.
- Endpoints for getting pending requests (either that you can approve
or that you submitted) were changed to
- Fields removed from
v2.4.0 Migration Notes
- Customers cannot upgrade directly from v2.1.0 to v2.4.0 if they have existing Hive or Impala data sources.
- This version relies on Postgres 11 for the Query Engine. (RDS/Postgres 10 can still be used for bometadata.)
- This version relies on
- The Partition Service needs to have
immuta.system.api.keyconfigured. In CDH that configuration should be added (for both the 1.6 and 2.x Partition Service) under
Immuta Spark 1.6 Partition Server Advanced Configuration Snippet (Safety Valve) for context/generator.xmland
Immuta Spark 2 Partition Server Advanced Configuration Snippet (Safety Valve) for session/generator.xml.
- Existing Spark jobs that have configuration overrides for
spark.driver.extraJavaOptionsneed to be updated to include the following configuration key/value pair for both
-Dimmuta.spark.encryption.fnr.class=com.immuta.spark.encryption.fnr.ImmutaFNRService. No changes are needed for any jobs that don't specifically modify those two configuration values.