Management of Global Policies, Tags, and Project Settings
Audience: Data Governors
Content Summary: This page describes Global Policy types, tags, purposes, acknowledgement statements, and project settings.
Global Policies can only be created by Data Governors and apply to all data sources across an organization. In contrast, Local Policies can be created by Data Owners or Data Governors and apply to a specific data source.
Staged Global Policies
Governors can create Staged Global Policies, which can then be safely reviewed and edited without affecting data sources. Once a policy is ready, Governors can activate it to immediately enforce the policy on relevant data sources.
Note: Policies that contain the circumstance When selected by data owners cannot be staged.
To access a data source, Immuta users must first be subscribed to that data source. A Subscription Policy determines who can request access and has one of four possible restriction levels:
- Anyone: Users will automatically be granted access (Least Restricted).
- Anyone Who Asks (and is Approved): Users will need to request access and be granted permission by the configured approvers (Moderately Restricted).
- Users with Specific Groups/Attributes: Only users with the specified groups/attributes will be able to see the data source and subscribe (Moderately Restricted).
- Individual Users You Select: The data source will not appear in search results; data owners must manually add/remove users (Most Restricted).
See Managing Users and Groups in a Data Source for details on managing Data Users.
Once a user is subscribed to a data source, the Data Policies that are applied to that data source determine what data the user sees. Data Policy types include masking, row redaction, minimization, time-based restrictions, differential privacy, and limiting to purpose.
You would use these to hide values in data. The masking policies have various levels of utility while still preserving data privacy. In order to create masking policies on object-backed data sources, you must create Data Dictionary entries and the data format must be either, csv, tsv, or json.
Row-Level Security Policies
These policies hide entire rows or objects of data based on the policy being enforced; some of these policies require the data to be tagged as well.
These policies restrict access to rows/objects/files that fall within the time restrictions set in the policy. If a
data source has time-based restriction policies, queries run against the data source by a user will only
return rows/blobs with a date in its
event-time column/attribute from within a certain range.
The time window is based on the event time you select when creating the data source. This value will come from a date/time column in relational sources. For S3 it can be retrieved by a metadata or tag on the S3 object, and for HDFS it is retrieved from the xattr on the file.
These policies restrict access to a limited percentage of the data, which is randomly sampled, but it is the same sample for all the users. For example, you could limit certain users to only 10% of the data. The data the user sees will always be the same, but new rows may be added as new data arrives in the system. This policy can only be applied to query-backed data sources.
Differential privacy provides mathematical guarantees that you cannot pinpoint an individual (row) in the data. This anonymization applies the appropriate noise (if any) to the response based on the sensitivity of the query. For example “average age” could be changed from 50.5 to 55 at query time. To do this the Immuta SQL layer restricts queries run on the data to only aggregate queries (AVG, SUM, COUNT, etc) and prevents very sensitive queries from running at all. This policy type can only be applied to query-backed data sources.
We encourage you to read our blog on this topic that dives into details of the theories behind this powerful anonymization technique.
In order to create this policy you must select a high cardinality column in the data. This is typically the primary key column, but could also be a column with many unique values. It is not recommended that you select a column with less than 90% unique values. Otherwise you could have invalid noise added to the responses.
It is also critical that you consider the latency tolerance on the data source when creating this policy. The latency tolerance drives how long differentially private query responses are cached. You should set this window to a length that allows sufficient time for the underlying data to change enough where the same query would get a statistically relevant dissimilar result. The caching is done to avoid the privacy budget problem, which is the problem of the user asking similar questions consecutively in order to determine the real response.
Limit to Purpose
Purposes help define the scope and use of data within a project and allow users to meet purpose restrictions on policies. Governors create and manage purposes and their sub-purposes, which project owners then add to their project(s) and use to drive Data Policies.
Purposes can be constructed as a hierarchy, meaning that purposes can contain nested sub-purposes, much like tags in Immuta. This design allows more flexibility in managing purpose-based restriction policies and transparency in the relationships among purposes.
For example, consider this organization of the sub-purposes of Research:
Instead of creating separate purposes, which must then each be added to policies as they evolve, a Governor could write the following Global Policy:
Limit usage to purpose(s) Research for everyone on data sources tagged PHI.
Now, any user acting under the purpose or sub-purpose of
Research - whether
Research.MedicalClaims - will meet the criteria of this policy. Consequently,
purpose hierarchies eliminate
the need for a Governor to re-write these Global Policies when sub-purposes are added or removed. Furthermore, if new
projects with new Research purposes are added, for example, the relevant Global Policy will automatically be enforced.
Data Policy Conflicts
In some cases, two conflicting Global Data Policies may apply to a single data source. When this happens, the policy containing a tag deeper in the hierarchy will apply to the data source to resolve the conflict.
Consider the following Global Data Policies created by a Data Governor:
Data Policy 1: Mask columns tagged
PIIusing a constant for everyone on data sources with columns tagged
Data Policy 2: Mask columns tagged
PII.SSNby making null for everyone on data sources with columns tagged
If a Data Owner creates a data source and applies
PII.Other tags, both of these Global Data Policies
will apply. Instead of having a conflict, the policy containing a deeper tag in the hierarchy will apply:
In this example, Data Policy 2 cannot be applied to the data source. If Data Owners wanted to use Data Policy 2 on the data source instead, they would need to disable Data Policy 1.
Once enabled on a data source, Global Data Policies can be edited and disabled by Data Owners. See the Managing Policies Tutorial for instructions.
Tags serve several functions: they can drive Local or Global Subscription and Data Policies, they can be used to generate Immuta Reports, and they can drive search results in the Immuta UI. Governors can create tags or import tags from external catalogs in the Governance UI. Data Owners and Governors can then apply these tags to or remove them from projects, data sources, and/or specific columns within the data sources.
Sensitive Data Detection
Sensitive Data Detection is a license-driven feature that must be added for you before it is available in your Immuta instance.
To help users identify sensitive data and to enhance the power of Global Policies, Immuta offers Sensitive Data Detection. When enabled on the App Setting page, this feature uses third party services to automatically identify and tag columns that contain sensitive data (PII, PHI, etc.) when the data source is created; this detection is based on an extremely small randomized sampling of underlying data, which is encrypted in transit, is used only for entity prediction, and remains confidential and managed by Immuta, subject to the same guarantees reviewed and agreed to in our license agreement.
During the fingerprint process Sensitive Data Detection divides the classification of the data into specific tags: Immuta “Discovered” tags.
The Immuta application is pre-configured with a set of these tags that the service can return so that they can be used to write Global Policies before data sources even exist. Consequently, sensitive data is tagged and appropriate policies are enforced immediately upon data source creation.
Only Application Admins have the option to enable Sensitive Data Detection on the App Settings page. However, users can disable auto-tagging on a data-source-by-data-source basis, and Governors can delete any unwanted “Discovered” tags in the Immuta application to prevent them from being used and auto-detected in the future.
Project Purposes, Acknowledgement Statements, and Settings
The Data Governor is responsible for configuring project purposes, acknowledgement statements, and settings.
Purposes: Purposes help define the scope and use of data within a project and allow users to meet purpose restrictions on policies. Governors can create purposes for project owners to use or owners can create their own purposes when they create their project (if the Governor allows them to). However, only Governors can delete purposes.
Acknowledgement Statements: Projects containing purposes require owners and subscribers to acknowledge that they will only use the data for those purposes by affirming or rejecting acknowledgement statements. If users accept the statement, they become a project member. If they reject the acknowledgement statement, they are denied access to the project. Once acknowledged, data accessed under the provision of a project will be audited and the purposes will be noted. Immuta provides default acknowledgement statements, but Data Governors can customize these statements in the Purposes or Settings tabs. Acknowledgement statements ensure that project members are aware of (and agree to) all purpose-based restrictions before accessing the project's content. Each purpose is associated with its own acknowledgement statement, meaning that a project with multiple purposes (if allowed) would require users to accept more than one acknowledgement statement. Immuta keeps a record of whether each project member has agreed to the acknowledgement statement(s), and if so, records the purpose associated to the acknowledgement, the time of the acknowledgement, and the text of the acknowledgement itself. All purposes are associated with the default acknowledgement statement unless their statement has been customized.
Settings: Governors can also determine if purposes are required to create a project, if purposes can be customized by project owners or must be chosen from purposes created by the data governor, or if a project can have more than one purpose. These settings are adjusted in the Settings tab of the Governance page and include the following options:
- A purpose must be included in projects: Selecting this option will require that every project contain a purpose. Utilizing data within a project outside of the stated purposes is prohibited. Projects without purposes, however, have no set restrictions.
- All data sources require a purpose restriction: Selecting this option will require every data source to have a purpose restriction.
- A project can have more than one purpose: Selecting this option allows projects to have more than one purpose.
- A project's purpose can change: Selecting this option will allow a project’s purpose to change at any time during the life of the project. Only users who created the project can change the purpose.
- Projects can have custom purposes: Selecting this option will allow project owners to describe the purpose of their project themselves, rather than choosing from a list of purposes created by a Governor.