This guide provides information and best practices for migrating from the deprecated legacy sensitive data discovery (SDD) option to the improved native SDD. This guide is for users who have already enabled SDD on their tenant and have Discovered tags on their data sources.

Before you begin

Native vs legacy SDD

Legacy SDD is deprecated. It will be removed and replaced by native SDD. Native SDD is significantly improved from legacy SDD for discovering and tagging your data with upgrades to the built-in identifiers. Additionally, the greatest benefit is the respect for data residency. Native SDD doesn't move any of your data when running. The discovery is done right in your data platform, and the platform only returns the matching identifiers and column names to Immuta.

See the Sensitive data discovery reference page for more information on native SDD.

Requirements

• Native SDD requires Snowflake, Databricks, Starburst (Trino), or Redshift data sources

• Legacy SDD enabled on your tenant

• Legacy SDD tags applied to your data sources: To find out if you have legacy SDD tags applied, create a governance report as described in the understand the context of you tags section.

Enable native SDD

Contact your Immuta representative to enable native SDD on your Immuta tenant. Many users already have native SDD enabled, so proceed to understand the context of your tags if you want to self-service check if native SDD is already running and tagging your data before you reach out to the representative.

This action will not change anything immediately on your tenant; however, anytime identification runs in the future, it will be native SDD instead of the legacy version.

To assess native SDD for your data, proceed with the steps below. If you do not review native SDD, the legacy SDD tags will all remain on your data source columns. However, when identification automatically runs on new data sources and columns, it will apply native SDD tags, and because of the improvements to SDD, it may tag different data than legacy SDD.

Understand the context of your tags

Requirement: Immuta permission GOVERNANCE

1. Manually run identification globally to run native identification on your data sources.

2. To check the tags on an individual data source, navigate to the data source data dictionary and select a Discovered tag. On the tag side sheet, you can determine the context of the tag. When identifiers match data, native SDD will apply tags, and their tag context will be Sensitive Data Discovery. Any tags with the context Legacy Sensitive Data Discovery were not matched by native SDD but will remain on the data source.

3. To check your tags globally, navigate to the governance reports page and build a report for sensitive data discovery. This report will present the legacy tags on your data sources' columns and native SDD tags that are also on those columns. Use this report to assess the context of the Discovered tags and understand if native SDD is matching the data you want it to.

These actions will allow you to understand the differences between how native SDD and legacy SDD tag your data and whether your data is recognized as expected by native SDD or if legacy SDD was over-tagging your data. This way you can better tune SDD to your data.

If there are any legacy SDD tags that you want native SDD to catch, you need to tune native SDD so that this type of data is discovered in future tables and columns; see guidance on that in the next section.

Tune SDD

Requirement: Immuta permission GOVERNANCE

Using the report you built above, complete these actions to tune SDD:

1. Focus on a legacy SDD tag properly applied to your data. Assess whether the native SDD tag on the column instead was applied more accurately than the legacy tag. If it is applied incorrectly, proceed to the next step.

2. Create a new regex or dictionary identifier in the framework to discover this data with the tag you want applied. Ensure it is specific and will match your data with at least 90% confidence (or match).

3. Complete the steps above for all legacy SDD tags.

4. Retest your updated identifiers by re-running identification on the select data sources and continue refining to the level of accuracy you want.

Completing the actions above will create parity between what legacy SDD was tagging your data and what native SDD will tag in the future.

Requirements:

• Immuta permission GOVERNANCE

• Registered Snowflake, Databricks, Redshift, or Starburst (Trino) data sources

This how-to guide is for enabling sensitive data discovery (SDD) for the first time. For additional information on sensitive data discovery, see the Data discovery page.

Turn on SDD

1. Navigate to the App Settings page and scroll to the Sensitive Data Discovery section.

2. Select the Enable Sensitive Data Discovery (SDD) checkbox to enable SDD.

3. Click Save and then click Confirm to apply your changes. Note that the Immuta tenant will have a system restart.

Create a new framework with identifiers

Once SDD is enabled on your tenant, SDD will automatically run when new data sources are added, but it must be manually run for all existing data sources. This allows you to test out SDD with a select few data sources without worrying that it will add tags throughout all your data sources.

For this step, you will pick the identifiers to match the data that matters to your organization. For example, for international data, you may want to enable many different identifiers for many countries, like the "Australia Passport" identifier and the "Finland National ID Number" identifier. However, if you are dealing with United States domestic financial data, those identifiers would be irrelevant. In that case, it would be better to identify the data likely to appear, like Bitcoin or US Bank Routing MICR.

First, create an empty framework,

1. Navigate to Discover and Identification.

2. Select Create New.

3. Enter a Name and Description for your new identification framework.

4. Select Create empty framework.

Then, add a new identifier to that framework,

1. Navigate to Discover and Identifiers.

2. Use the checkboxes to select all the identifiers relevant to your data. Tip: From the overview page you can see the name and the tags that will be applied by the identifier. To better understand the data it will match, click the name to read the description.

3. Once you have checked the identifiers you want in your framework, click Add to Framework.

4. Type the framework name in the text box.

5. Click Add to Framework.

Run identification on your data sources

Once you have created a framework relevant to your data, it is time to test it on your data and customize it. Run identification on a select number of data sources where you understand the data to assess and adjust the tags to reflect what you expect to see.

Add those select data sources to your new framework,

1. Navigate to Discover and Identification.

2. Click your new framework name.

3. Navigate to the Data Sources tab.

4. Click Add Data Sources.

5. Check the checkboxes for the select data sources you want to try SDD on.

6. Click Add Data Source(s).

Then, run identification on those data sources,

1. Navigate to Discover and Identification.

2. Click the action menu for your new framework.

3. Click Run Identification.

View the identification results

After identification runs, you will receive a notification that the job is complete. Then, you can view the results from the data source dictionary.

1. Navigate to the data source overview page of the data source you added to the framework.

2. Click the Data Dictionary tab.

3. Assess whether the Discovered tags are applied as expected.

4. If you are happy with the Discovered tags, follow the Assign data sources to frameworks guide to add the rest of your data sources to the framework and follow the Run identification guide to run identification on all your data sources.

5. If you want additional tags, follow the Create an identifier guide to create identifiers that matter to your data.

Requirements:

• Registered

• Immuta permission GOVERNANCE

Identification (or sensitive data discovery (SDD)) runs automatically. If you want to re-run identification when a new global framework is set or when new identifiers have been added to a framework, you can or from the UI by following a how-to below.

Run identification using a specific framework

1. Click the Discover icon and the Identification tab in the navigation menu.

2. Select the more actions icon.

3. Select Run Identification and then select it again in the modal.

Run identification on a data source

1. Navigate to the data source overview page.

2. Click the health status.

3. Select Re-run next to Sensitive Data Discovery (SDD).

Verify discovered tags

Disable Discovered tags from the data dictionary

If a governor, data owner, or data source expert disables a Discovered tag from the data dictionary, the column will not be re-tagged next time identification (or SDD) runs. When a Discovered tag is disabled, it will not completely disappear, and it can be manually enabled through the tag side sheet.

To disable a discovered tag,

1. Navigate to a data source and click the Data Dictionary tab.

2. Scroll to the column you want to remove the tag from and click the tag you want to remove.

3. Click Disable in the side sheet and then click Confirm.

Requirements:

• Sensitive data discovery (SDD) enabled

• Immuta permission GOVERNANCE

Create an identification framework

Create an identification framework with no identifiers

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Click Create New.

3. Enter a Name and Description for the identification framework.

4. Select the option to Create empty framework.

5. Click Create.

After you create the identification framework, you can create new identifiers.

Copy an existing identification framework and its identifiers

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Click Create New.

3. Enter a Name and Description for the identification framework.

4. Select the option to Create identifiers from an existing framework.

5. Select the checkbox for the framework you want to copy. You can only copy a single framework. For more information about a framework, click the framework name to open a new tab with details about the framework.

6. Click Create.

Manage an identification framework's identifiers

Add an identifier to a framework

To add an identifier to a framework,

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Select the framework name for the identification framework you want to edit.

3. Click Add Identifier.

4. Choose in the dropdown to add an identifier from those already in Immuta or create a new identifier for the framework.

   • For existing identifiers: Opt to edit the tags. Then click Add Identifier.

   • For new identifiers:

     1. Fill out a Name and Description.

     2. Enter criteria: Select the Type of criteria.

        1. For regex, enter a regex to be matched against column values. The default criteria encoding is case-sensitive. You can change this encoding using the regex criteria. The regex must use RE2.

        2. For column name regex, enter a regex to be matched against column names. The default criteria encoding is not case-sensitive. You can change this encoding using the regex criteria. The regex must use RE2 syntax.

        3. For a dictionary, enter the values in a comma-separated list to match against column values. Opt to toggle the Case insensitive switch to on if you want the dictionary to be case sensitive.

     3. Select the tags to apply: Use the text box to search for a tag under the "Discovered" hierarchy or type a tag name to create a new tag under the "Discovered" hierarchy to apply to columns that match your identifier.

     4. Click Next to review your new identifier and click Create Identifier to create it.

Edit an identifier in a framework

Only tags can be edited within a framework. Edits made to an identifier within a framework will only impact that specific identifier. To fully edit an identifier (including the name, description, or criteria) for all frameworks, use the Edit an identifier how-to guide.

To edit the tags applied by an identifier for a framework,

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Select the framework name for the identification framework you want to edit.

3. Click the more actions icon for an identifier and select Edit tags.

4. Remove the tags or type a tag name to add tags.

5. Click Save.

Delete an identifier from a framework

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Select the framework name for the identification framework you want to edit.

3. Click the more actions icon for an identifier and select Delete.

4. Click Delete again in the modal.

Manage an identification framework's data sources

Assign an identification framework to data sources

To assign a framework to run on specific data sources,

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Select the framework you want to assign and navigate to the Data Sources tab.

3. Click Add Data Sources.

4. Select the checkbox for the data source you want this framework to run on. You may select more than one.

5. Click Add Data Source(s).

Remove data sources from an identification framework

After a data source is removed from a framework, it will use the global framework for any SDD scans and the tags applied by the removed framework will be replaced. The global framework is signified by the globe icon.

To remove data sources from a framework,

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Select the framework you want to remove data sources from and navigate to the Data Sources tab.

3. Select the checkbox for the data source you want to remove from the framework. You may select more than one.

4. Select Remove and click Remove again in the modal.

Delete an identification framework

Requirement: No data sources assigned to the framework

To delete a framework,

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Click the more actions icon in the Action column for the framework you want to delete. The global framework cannot be deleted. If you want to delete it, configure a different framework as the global framework.

3. Select Delete and click Delete again in the modal.

Requirement: Immuta permission APPLICATION_ADMIN

Configure the global framework

1. Click the App Settings icon in the left sidebar.

2. Click Sensitive Data Discovery in the left panel to navigate to that section.

3. Enter the request-friendly name of your global identification framework in the Global SDD Template Name field. This name can be found in the URL when you navigate to the identification framework's page.

4. Click Save, and then Confirm your changes.

Requirements:

• Sensitive data discovery (SDD) enabled

• Immuta permission GOVERNANCE

Create an identifier

1. Click the Discover icon in the navigation menu and select the Identifiers tab.

2. Click Create New.

3. Enter a Name and Description for the new identifier.

4. Enter criteria: Select the Type of criteria.

   1. For regex, enter a regex to be matched against column values. The default criteria encoding is case-sensitive. You can change this encoding using the regex criteria. The regex must use RE2.

   2. For column name regex, enter a regex to be matched against column names. The default criteria encoding is case-insensitive. You can change this encoding using the regex criteria. The regex must use RE2 syntax.

   3. For a dictionary, enter the values in a comma-separated list to match against column values. Opt to toggle the Case insensitive switch to on if you want the dictionary to be case sensitive.

5. Select the tags to apply: Use the text box to search for a tag under the "Discovered" hierarchy or type a tag name to create a new tag under the "Discovered" hierarchy to apply to columns that match your identifier.

6. Click Next to review your new identifier and click Create Identifier to create it.

7. See the Manage identification frameworks page to add your new identifier to a framework.

Note that all user-created identifiers must be a 90% match or greater for the contents of the column to be tagged.

Edit an identifier

Editing the details or criteria of an identifier from the identifiers menu will affect any framework with that identifier throughout Immuta. Editing the tags will only affect new frameworks the identifier is added to.

To edit an identifier,

1. Click the Discover icon in the navigation menu and select the Identifiers tab.

2. Click the name of the identifier you want to edit.

3. Click Edit.

4. Edit the field you want to change.

5. Click Save.

Built-in identifiers cannot be edited.

Delete an identifier

Deleting an identifier will remove it from all the frameworks it is in throughout Immuta.

To delete an identifier,

1. Click the Discover icon in the navigation menu and select the Identifiers tab.

2. Click the more actions icon in the Action column for the identifier you want to delete.

3. Select Delete and click Delete again in the modal.

Built-in identifiers cannot be deleted.