Data policies determine what users see when they query data in a table they have access to.

How-to guides

• Author a masking data policy

• : Certify policies and view policy diffs on a data source.

Reference guides

• : This guide describes how data policies work in Immuta.

• : This guide describes the types of masking policies available and when to use each.

• : Row-level policies compare data values with user metadata at query-time to determine whether or not the querying user should have access to the individual rows of data.

1. Determine your policy scope:

   • Global policy: Click the Policies icon in the navigation menu and select the Data Policies tab. Click New data policy and complete the Policy name field.

   • Local policy: Navigate to a specific data source and click the Policies tab. Scroll to the Data Policies section and click New Policy.

2. Select Mask from the first dropdown menu.

3. Select columns tagged, columns with any tag, columns with no tags, all columns, or columns with names spelled like.

4. Select a masking type (some of these types will ):

5. Select everyone except, everyone, or everyone who to continue the condition.

   • everyone except: In the subsequent dropdown menus, choose is a member of group, possesses attribute, or is acting under purpose. Complete the condition with the subsequent dropdown menus. For a list of exceptions and an explanation of their behavior, see the .

6. Opt to complete the Enter Rationale for Policy (Optional) field, and then click Add.

7. For global policies: Click the dropdown menu beneath Where should this policy be applied and select When selected by data owners, On all data sources, or On data sources. If you selected On data sources, finish the condition in one of the following ways:

   • tagged: Select this option and then search for tags in the subsequent dropdown menu.

   • with columns tagged

8. Click Create Policy. If creating a global policy, you then need to click Activate Policy or Stage Policy.

Data owners who are not governors can write restricted subscription and data policies, which allow them to enforce policies on multiple data sources simultaneously, eliminating the need to write redundant local policies.

Unlike global policies, the application of these policies is restricted to the data sources owned by the users or groups specified in the policy and will change as users' ownerships change.

1. Click the Policies icon in the navigation menu and select Data Policies.

2. Click New data policy and complete the Policy name field.

3. Select how the policy should protect the data. Click a link below for instructions on building that specific data policy:

4. Opt to complete the Enter Rationale for Policy (Optional) field, and then click Add.

5. From the Where should this policy be applied dropdown menu, select When selected by data owners, On all data sources, or On data sources. If you selected On data sources, finish the condition in one of the following ways:

   • tagged: Select this option and then search for tags in the subsequent dropdown menu.

   • with columns tagged

6. Beneath Whose Data Sources should this policy be restricted to, add users or groups to the policy restriction by typing in the text fields and selecting from the dropdown menus that appear.

7. Click Create Policy, and then click Activate Policy or Stage Policy.

Required permissions

To manage and apply existing policies to data sources, a user must have either the CREATE_DATA_SOURCE Immuta permission or be manually assigned the owner role on a data source.

Limit to purpose policies use purposes and Immuta projects to govern access to data.

Purposes define the scope and use of data within a project, while projects allow users to meet . In general, here are how purposes and projects interact to enforce purpose-based access controls:

• Governors create purposes and include them in global data policies.

• Project owners then add purposes to their project(s).

1. Determine your :

   • Global policy: Click the Policies icon in the navigation menu and select the Data Policies tab. Click New data policy and complete the Policy name field.

1. Determine your policy scope:

   • Global policy: Click the Policies icon in the navigation menu and select the Data Policies tab. Click New data policy and complete the Policy name field.

   • Local policy: Navigate to a specific data source and click the Policies tab. Scroll to the Data Policies section and click New Policy.

2. Select the Only show rows action from the first dropdown.

3. Choose one of the following policy conditions:

   • Where user

     1. Choose the condition that will drive the policy from the next dropdown: is a member of a group or possesses an attribute.
4. Choose for everyone, everyone except, or for everyone who to drive the policy. If you choose for everyone except, use the subsequent dropdown to choose the group, purpose, or attribute for your condition. If you choose for everyone who as a condition, complete the Otherwise clause before continuing to the next step.

5. Opt to complete the Enter Rationale for Policy (Optional) field, and then click Add.

6. For global policies: Click the dropdown menu beneath Where should this policy be applied, and select On all data sources, On data sources, or When selected by data owners. If you select On data sources, finish the condition in one of the following ways:

   • tagged: Select this option and then search for tags in the subsequent dropdown menu.

   • with columns tagged

7. Click Create Policy. If creating a global policy, you then need to click Activate Policy or Stage Policy.

1. Determine your policy scope:

   • Global policy: Click the Policies icon in the navigation menu and select the Data Policies tab. Click New data policy and complete the Policy name field.

   • Local policy: Navigate to a specific data source and click the Policies tab. Scroll to the Data Policies section and click New Policy.

2. Select Minimize data source from the first dropdown.

3. Complete the enter percentage field to limit the amount of data returned at query-time.

4. Select for everyone except from the next dropdown menu to continue the condition. Additional options include for everyone and for everyone who.

5. Use the next field to choose the attribute, group, or purpose that you will match values against.

   Notes:

   • If you choose for everyone who as a condition, complete the Otherwise clause before continuing to the next step.

6. Opt to complete the Enter Rationale for Policy (Optional), and then click Add.

7. For global policies: Click the dropdown menu beneath Where should this policy be applied, and select On all data sources, On data sources, or When selected by data owners. If you select On data sources, finish the condition in one of the following ways:

   • tagged: Select this option and then search for tags in the subsequent dropdown menu.

   • with columns tagged

8. Click Create Policy. If creating a global policy, you then need to click Activate Policy or Stage Policy.

Immuta row-level policies compare data values with user metadata at query-time to determine whether or not the querying user should have access to the individual rows of data.

These policies hide entire rows or objects of data based on the policy being enforced; some of these policies require the data to be tagged as well.

How do row-level policies work?

The values contained in one or many columns in the table in question (or a ) need to be referenced by the policy for its logic to take effect.

Orchestrated masking policies (OMP) reduce conflicts between masking policies that apply to a single column, allowing policies to scale more effectively across your organization. Furthermore, OMP fosters distributed data stewardship, empowering policy authors who share responsibility of a data set to protect it while allowing data consumers acting under various roles or purposes to access the data.

When multiple masking policies apply to a column, Immuta combines the exception conditions of the masking policy so that data subscribers can access the data when they satisfy one of those exception conditions. Multiple masking policies will be enforced on a column if the following conditions are true:

• Policies use the same masking type.

• Policies use the for everyone except condition.

Requirements

Databricks Spark or Starburst (Trino) integration

Supported masking policy types

OMP supports the following masking types:

• Constant

• Hashing

• Format preserving masking

• Null

Global policy logic

Previous policy logic

Governors can apply policies to all columns in a data source or target specific columns with tags or a regular expression. Without orchestrated masking policies enabled, when multiple global policies apply to the same columns, Immuta could only apply one of those policies.

Consider the following example to examine how policies behaved when one tag is used in two different policies:

• Mask PII Global Policy 1: Mask using hashing the value in columns tagged email except when user is acting under the purpose Email Campaign.

• Mask PII Global Policy 2: Mask using hashing the value in columns tagged email except when user is acting under purpose Marketing.

For columns tagged email, only one of these policies is enforced. The Mask PII Global Policy 2 is not applied to the data source, so Immuta is not enforcing the masking policy properly for users who should be able to see emails because they are acting under the Marketing purpose.

Consider the following example where multiple masking policies apply to columns that have multiple tags, resulting in one policy applying:

• Global Policy 3: Mask using hashing the value in columns tagged Employee Data unless users are acting under the purpose Retention Analysis.

• Global Policy 4: Mask using hashing the value in columns tagged HR Data unless users are acting under the purpose Employee Satisfaction Survey.

If a column is tagged Employee Data and HR Data, Immuta will only apply one of the policies.

Orchestrated masking policy logic

With orchestrated masking policies, Immuta applies multiple global masking policies that apply to a single column by combining the policy exceptions with OR. For these policies to combine, the masking type must be identical and the policy must use the for everyone except condition.

Consider the following example, both of these policies will apply to the data source:

• Mask PII Global Policy 1: Mask using hashing the value in columns tagged email except when user is acting under the purpose Email Campaign.

• Mask PII Global Policy 2: Mask using hashing the value in columns tagged email except when user is acting under purpose Marketing.

Users acting under the purpose Marketing or Email Campaign will be able to see emails in the clear. However, in the following example, only one of these policies will apply to the data source because one masks using a constant and the other masks using hashing:

• Global Policy 5: Mask using the constant REDACTED the value in columns tagged Employee Data unless users are acting under the purpose Retention Analysis.

• Global Policy 6: Mask using hashing the value in columns tagged HR Data unless users are acting under the purpose Employee Satisfaction Survey.

Limitations

• No UI enhancements were made in this release. Multiple masking policies applied to the same column are visible on a data source, but there is no indication that the exceptions are combined with OR.

• Masking types must match exactly for the policies to be combined. For example, both policies must mask using rounding.

• Existing policies will not automatically migrate to the new policy logic when you enable the feature. To re-compute existing policies with the new logic, you must manually trigger global policy changes by staging and re-enabling each policy.

Data policies manage what users see when they query data in a table they are subscribed to.

There are three different ways to restrict data access with data policies:

• : Filter rows from certain users at query time.

• : Mask values in a column at query time.

Requirement and prerequisite:

• CREATE_DATA_SOURCE or GOVERNANCE Immuta permission

• A purpose has been created

Build the policy

1. Determine your :

   • Global policy: Click the Policies icon in the navigation menu and select the Data Policies tab. Click New data policy and complete the Policy name field.

   • Local policy: Navigate to a specific data source and click the Policies

Related guides

How-to guides

• : To restrict access to data and associate your data source with a purpose, create a project and add the purpose and relevant data sources to the project.

Reference guides

Conceptual guide

If user metadata is stored in a table in the same data platform where a policy is enforced, it is not necessary to move that user metadata in Immuta. Instead it can be referenced directly using custom WHERE functions in data policies.

Below is an example row-level policy that leverages a lookup table to dynamically drive access to rows in a table:

The final column in the table, ACCESS_LEVEL, defines who can see that row of data.

Now consider the following hierarchy:

In this diagram, there are 11 different access levels (AL) to data and the tree defines access. For example, if a user has Vegetables, they get access levels 2, 3, 4, 9, 10, and 11. If a user has Pear, they only get access level 8. In other words, a user with Vegetables would see the first row of the above table, a user with Pear would see the second row of the above table, and a user with Food would see both rows of the table.

Taking the example further, that hierarchy tree is represented as a table in the data platform that we wish to use to drive the row-level policy:

That hierarchy lookup table can be referenced in the row-level policy as user metadata like this:

@columnTagged('access_level') IN (SELECT ACCESS_LEVEL from [lookup table] where @attributeValuesContains('user_level', 'ROOT'))

Walking through the policy step-by-step:

• @columnTagged('access_level'): This allows us to target multiple tables with an ACCESS_LEVEL column that needs protecting with a single policy. Simply tag all the ACCESS_LEVEL columns with the access_level tag and this policy would apply to all of them.

• IN (SELECT ACCESS_LEVEL from [lookup table]: This is selecting the matching ACCESS_LEVEL from the lookup table to use as the IN clause for filtering the actual business table.

So, you can then add metadata to your users in Immuta, such as Vegetables or Pear and that will result in them seeing the appropriate rows in the business table in question.

The above example used a row-level policy, but it could instead do cell masking using the same technique:

Mask columns tagged Credit Card Number using hashing where @columnTagged('access_level') NOT IN (SELECT ACCESS_LEVEL from [lookup table] where @attributeValuesContains('user_level', 'ROOT'))

In this case, the credit card number will be masked if the access_level is not found for the user for that row.

Even if not using a lookup table, the power of the @columnTagged('tag name') function is apparent for applying your masking or row-level policies at scale.

The policy builder allows you to use custom functions that reference important Immuta metadata from within your where clause. These custom functions can be seen as utilities that help you create policies easier. Using the policy builder, you can include these functions in your masking or row-level policies by choosing where in the sub-action menu or using the custom function in the masking type dropdown menu.

The @attributeValuesContains() function

This function returns true for a given row if the provided column evaluates to an attribute value for which the querying user has a corresponding attribute value. This function requires two arguments and accepts no more than three arguments.

Parameters

The @columnReference() function

This function must be used in custom WHERE policies that reference a column name in the data source being protected. For example, to only show rows that have the value US in the Location column, you would create the following policy:

When using this function, match the casing of the column name you specify with the column name in the remote platform and escape the following characters with a backslash: \, ', ", ` .

This function should not be used to reference column names in external lookup tables. Instead, use the fully-qualified column name for external lookup tables.

Parameter

The @columnTagged() function

This function returns the column name with the specified tag.

If this function is used in a global policy and the tag doesn't exist on a data source, the policy will not be applied.

Parameters

The @groupsContains() function

This function returns true for a given row if the provided column evaluates to a group to which the querying user belongs. This function requires at least one argument.

Parameters

The @hasAttribute() function

This function returns a boolean indicating if the current user has the specified attribute name and value combination. If the specified attribute name or attribute value has a single quote, you will need to escape it using a \'\' expression within a custom WHERE policy.

Parameters

The @isInGroups() function

This function returns a boolean indicating if the current user is a member of all of the specified groups. If any of the specified groups has a single quote, you will need to escape it using a \'\' expression within a custom WHERE policy.

Parameter

The @isUsingPurpose() function

This function returns a boolean indicating if the current user is using the specified purpose. If the specified purpose has a single quote, you will need to escape it using a \'\' expression within a custom WHERE policy.

Parameter

The @purposesContains() function

This function returns true for a given row if the provided column evaluates to a purpose under which the querying user is currently acting. This function requires at least one argument and accepts no more than two arguments.

Parameters

The @username function

This function returns the current user's username.

Parameters

None.

Masking policies hide values in data, providing various levels of utility while still preserving privacy. Immuta offers column masking and cell-level masking.

As with all Immuta policy types, use global policies when authoring masking policies to manage policies at scale. When using global policies, tagging your data with metadata becomes critical and is described in detail in the Compliantly open more sensitive data for ML and analytics use case.

The masking options described on this page can be implemented in a variety of use cases, and there are several different approaches for masking data that allow you to make tradeoffs between privacy (how far you go with masking) and utility (how much you want the masked data to be useful to the data consumer). Use the table below to determine the circumstance under which a function should be used.

Masking types

Constant

Masking with a constant replaces any value in a column with a specified value. For example, you can replace the values in a column with the constant Redacted. The underlying data will appear to be a constant, removing any utility of that data.

Apply this policy to strings that require a specific repeated value.

Custom function

This option uses SQL functions native to the underlying database to transform the values in a column. This can be used in numerous use cases, but notional examples include top-coding to some upper limit, a custom hash function, and string manipulation.

Single quotes enclosing the regex and escaping special characters are required. The following example masks telephone numbers variably depending on the presence of a dash (implying a prefix), space, or only digits:

The image below illustrates authoring a global policy using this custom function:

Limitations

• The masking functions are executed against the remote database directly. A poorly written function could lead to poor quality results, data leaks, and performance hits.

• Using custom functions can result in changes to the original data type. In order to prevent query errors you must ensure that you cast this result back to the original type.

• The function must be valid for the data type of the selected column. If it is not

Format preserving masking

Format preserving masking uses a reversible function to mask the data in a way that the underlying structure of a value is preserved, so the length and type of a value are maintained. This is appropriate when the masked value should appear in the same format as the underlying value. Examples of this include social security numbers and credit card numbers where mask with format preserving masking would return masked values in a format consistent with credit cards or social security numbers, respectively.

There is larger overhead with this masking type, and it should really only be used when format is critically valuable, such as situations when an engineer is building an application where downstream systems validate content. In almost all analytical use cases, format should not matter.

Hashing

Hashing masks the values with an irreversible sha256 hash, which is consistent for the same value throughout the data source, so you can count or track the specific values, but not know the true raw value.

This policy type is appropriate for cases where the underlying value is sensitive, but there is a need to segment the population. Such attributes could be addresses, time segments, or countries. It is important to note that hashing is susceptible to inference attacks based on prior knowledge of the population distribution. For example, if state is hashed, and the dataset is a sample across the United States, then an adversary could assume that the most frequently occurring hash value is California. As such, it's most secure to use the hashing mask on attributes that are evenly distributed across a population.

Hashed values are different across data sources, so you cannot join on hashed values unless you . Immuta prevents joins on hashed values to protect against link attacks where two data owners may have exposed data with the same masked column (a quasi-identifier), but their data combined by that masked value could result in a sensitive data leak.

NULL

This masking type replaces the values in the column with NULL, removing any identifiability from the column and all utility of the data.

Apply this policy to numeric or text attributes that have a high re-identification risk, but little analytic value (names and personal identifiers).

Randomized response

Randomized response masks data by slightly randomizing the values in a column, preserving the utility of the data while preventing outsiders from inferring content of specific records.

This function randomizes the displayed value to make the true value uncertain, but maintains some analytic utility.

For example, if an analyst wanted to publish data from a health survey she conducted, she could remove direct identifiers to make it difficult to single out individuals. However, consider these survey participants, a cohort of male welders who share the same zip code:

All members of this cohort have indicated substance abuse, sensitive personal information that could have damaging consequences, and, even though direct identifiers have been removed, outsiders could infer substance abuse for an individual if they knew a male welder in this zip code.

In this scenario, using randomized response would change some of the Y's in substance_abuse to N's and vice versa; consequently, outsiders couldn't be sure of the displayed value of substance_abuse given in any individual row, as they wouldn't know which rows had changed.

The randomization is applied differently to both categorical and quantitative values. In both cases, the noise can be increased to enhance privacy or reduced to preserve more analytic value. Immuta requires that you .

• Categorical randomized response: Categorical values are randomized by replacing a value with some non-zero probability. Not all values are randomized, and the consumer of the data is not told which values are randomized and which ones remain unchanged. Values are replaced by selecting a different value uniformly at random from among all other values. If a randomized response policy were applied to a “state” column, a person’s residency could flip from Maryland to Virginia, which would provide ambiguity to the actual state of residency. This policy is appropriate when obscuring sensitive values such as medical diagnosis or survey responses.

• Datetime and numeric randomized response: Numeric and datetime randomized response apply a tunable, unbiased noise to the nominal value. This noise can obscure the underlying value, but the impact of the noise is reduced in aggregate. This masking type can be applied to sensitive numerical attributes, such as salary, age, or treatment dates.

How the randomization works

Immuta applies a random number generator (RNG) that is seeded with some fixed attributes of the data source, column, backing technology, and the value of the high cardinality column, an approach that simulates cached randomness without having to actually cache anything.

For string data, the random number generator essentially flips a biased coin. If the coin comes up as tails, which it does with the frequency of the replacement rate , then the value is changed to any other possible value in the column, selected uniformly at random from among those values. If the coin comes up as heads, the true value is released.

For numeric data, Immuta uses the RNG to add a random shift from a 0-centered Laplace distribution with the standard deviation specified in the policy configuration. For most purposes, knowing the distribution is not important, but the net effect is that on average the reported values should be the true value plus or minus the specified deviation value.

Preserving data utility

Using randomized response doesn't destroy the data because data is only randomized slightly; aggregate utility can be preserved because analysts know how and what proportion of the values will change. Through this technique, values can be interpreted as hints, signals, or suggestions of the truth, but it is much harder to reason about individual rows.

Additionally, randomized response gives deniability of record content not dataset participation, so individual rows can be displayed.

Regular expression (regex)

This masking option uses a regular expression to replace all or a portion of a column value.

This policy is similar to replacing with a constant, but it provides more utility because you can retain portions of the true value, and REGEX replacement allows for some groupings to be maintained, while providing greater ambiguity to the disclosed value. This masking technique is useful when the underlying data has some consistent structure, the remasked underlying data represents some re-identification risk, and a regular expression can be used to mask the underlying data to be less identifiable.

When authoring the policy in Immuta, the regex and the replacement value do not need to be in single or double quotes.

The image below illustrates authoring a regex global policy that will apply to Databricks Unity Catalog data sources:

Reversibility

This masking option masks the values using a token that is consistent for the same value throughout the data source, so you can count or track the specific values, but not know the true raw value.

This policy type is appropriate for cases where the underlying value is sensitive, but there is a need to segment the population. Such attributes could be addresses, time segments, or countries. Reversible hashing is susceptible to inference attacks based on prior knowledge of the population distribution. For example, if state is hashed, and the dataset is a sample across the United States, then an adversary could assume that the most frequently occurring hash value is California. As such, it's most secure to use the reversible hashing mask on attributes that are evenly distributed across a population.

Hashed values are different across data sources, so you cannot join on hashed values unless you . Immuta prevents joins on hashed values to protect against link attacks where two data owners may have exposed data with the same masked column (a quasi-identifier), but their data combined by that masked value could result in a sensitive data leak.

Reversibly masked fields can leak the length of their contents, so it is important to consider whether or not this may be an attack vector for applications involving its use.

Rounding

Rounding masking policies reduce, round, or truncate numeric or datetime values to a fixed precision.

This technique hides precision from numeric values while providing more utility than simply hashing. For example, you could remove precision from a geospatial coordinate. You can also use this type of policy to remove precision from dates and times by rounding to the nearest hour, day, month, or year.

• Datetime rounding: This policy truncates the precision of a datetime value to a user-defined precision. minute, hour, day, months, and year are the supported precisions.

• Numeric rounding: This policy maps the nominal value to the ceiling of some specified bandwidth. Immuta has a recommended bandwidth based on the Freedman-Diaconis rule.

Masking exceptions

Exceptions to masking policies allow users specified in the exception to see masked data in the clear. Expand the collapsible blocks below to see how the different masking exceptions work.

Advanced masking exceptions

The masking exceptions in the list above that match any attribute value or any group name to any data source or column tag allow you to scope the masking exceptions to a subset of the data that is being masked. This decoupling between the generic masking rule and its fine-grained masking exceptions means that even with complex requirements, you only need to author one policy in Immuta.

For example, if a compliance requirement states that users should only be able see restricted data in the clear if it is specific to their department, you could approach this by authoring the following masking policies in Immuta:

• Mask columns tagged restricted using hashing for everyone except when user is member of a group with name Marketing on columns tagged Marketing in domain Marketing

• Mask columns tagged restricted using hashing for everyone except when user is member of a group with name Finance on columns tagged Finance

However, the drawback of this approach is that each time a new department gets added, this also requires a new masking policy to be authored, tested, and deployed. This will lead to policy bloat, potentially resulting in dozens or thousands of masking policies.

With advanced masking exceptions you can achieve the same result by just writing a single policy:

Mask columns tagged restricted using hashing for everyone except when user is a member of a group with name that matches any column tag

Users will only see restricted columns if their group name matches a tag on the column (just like in the former example), but you don't have to specify every possible group name or column tag in separate policies. Furthermore, using this policy type helps ensure that access controls will continue to be enforced appropriately even if your user or data metadata changes. If additional groups are created or tags are added to columns, access will be automatically updated to reflect those user and tag changes, so you don't have to update or add new policies after each change.

Behavior of data source versus column tags

Advanced masking exceptions allow you to match any attribute value or any group name to any data source or column tag. But how can you decide whether you should use the data source tags or column tags option? To determine this, consider the granularity with which you want to manage masking exceptions:

• Do you need to exempt users on a column-by-column basis?

• Or do you need to exempt users on a data source-by-data-source basis?

The example below illustrates this decision point:

• The table Salaries contains the columns name, email and wage that are all tagged PII.

• User A is a member of a group named HR.

When preparing the masking exception policy, the policy author compares the results of the using data source tags and the using column tags options:

Evaluation logic for tag hierarchies

Masking exceptions respect tag hierarchies when evaluating user attribute key-value pairs against column and data source tags. The evaluation always goes from left to right: the attribute value gets evaluated against the tag starting at the tag's root level.

The table below illustrates this behavior by showing how the following masking policy enforces access controls for 4 different users:

Mask columns tagged Marketing.Analyst by making NULL for everyone except when user possesses an attribute with key Role and value that matches any column tag

In this example,

• User A can see data in the email column in the clear because their Marketing.Analyst attribute value exactly matches the Marketing.Analyst tag.

• User B can see data in the email column in the clear because their Marketing attribute value matches the root level in the Marketing.Analyst column tag. This example illustrates the benefits of using tag hierarchies, as you can match against tag parent values (instead of having to match against each individual child tag).

Cell-level masking

Use cell-level masking (sometimes also called conditional masking) to achieve granular, context-aware data protection that standard column-level security cannot provide alone. While regular masking policies are an all-or-nothing approach (a user can either see all values in a column or none at all), cell-level masking increases data utility by masking columns on a row-by-row basis. Cell-level masking is achieved by conditionally masking the content in one column based on the value in another column of the same row.

Building a cell-level masking policy is done in the same manner as . The primary difference is when selecting who the policy should apply to, a where clause is injected.

For example, a regular masking policy looks like this:

Mask columns tagged SSN using hashing for everyone except members of group admins

With this approach, users will either see all social security numbers or none at all. If only social security numbers for US-based subjects needed special protection, and all other can remain in the clear, you could insert a where clause into the policy condition:

Mask columns tagged SSN using hashing where @columnReference('country_of_residence') = 'US' for everyone except members of group admins

That policy will check the country_of_residence column in the table, and if the value is US the cell tagged SSN will be masked. For all rows where the column country_of_residence contains a different value than US, the data of the column tagged SSN will remain in the clear.

Furthermore, instead of using the physical column name as shown in the example above, you could use the function. Using this function would allow you to target the policy on any table with a column containing location information no matter the name of that column in the physical table:

Mask columns tagged SSN using hashing where @columnTagged('country') = 'US' for everyone except members of group admins

This example policy targets the column with the tag country in the policy logic dynamically instead of looking for the hard-coded column name of country_of_residence.

Note: When building conditional masking policies with custom SQL statements, avoid using a column that is masked using in the SQL statement, as this can lead to different behavior depending on your data platform and may produce results that are unexpected.

Mixing masking policies on the same column

In some cases, you may want several different masking policies applied to the same column through Otherwise policies. To build these policies, select everyone who instead of everyone or everyone except. After you specify who the masking policy applies to, select how it applies to everyone else in the Otherwise condition.

You can add and remove tags in Otherwise conditions for global policies (unlike local policy Otherwise conditions); however, all tags or regular expressions included in the initial everyone who rule must be included in an everyone or everyone except rule in the additional clauses.

Complex data types: masking fields within struct columns

Spark supports a class of data types called complex types, which can represent multiple data values in a single column. Immuta supports masking fields within array and struct columns:

• Array: an ordered collection of elements

• Struct: a collection of elements that are primitive or complex types

Without this feature enabled, the struct and array columns of a data source default to jsonb in the Data Dictionary, and the masking policies that users can apply to jsonb columns are limited. For example, if a user wanted to mask PII inside the column patient in the image below, they would have to apply null masking to the entire column or use a custom function instead of just masking name or address.

After Complex Data Types is enabled on the , the column type for struct columns for new data sources will display as struct in the data dictionary. (For data sources that are already in Immuta, users can edit the data source and change the column types for the appropriate columns from jsonb to struct.) Once struct fields are available, they can be searched, tagged, and used in masking policies. For example, a user could tag name, ssn, and street as PII instead of the entire patient column.

After a global or local policy masks the columns containing PII, users who do not meet the exception specified in the policy will see these values masked:

Note: Immuta uses the > delimiter to indicate that a field is nested instead of the . delimiter, since field and column names could include ..

Struct columns with many fields

To get column information about a data source, Immuta executes a DESCRIBE call for the table. In this call, Spark returns a simple string representation of the schema for each column in the table. For the patient column above, the simple string would look like this:

struct<name:string,ssn:string,age:int,address:struct<city:string,state:string,zipCode:string,street:text>>

Immuta then parses this string into the following format for the data source's dictionary:

However, if the struct contains more than 25 fields, Spark truncates the string, causing the parser to fail and fall back to jsonb. Immuta will attempt to avoid this failure by increasing the number of fields allowed in the server-side property setting, maxToStringFields.