Discover scans your data sources and applies relevant tags when data is recognized. This eliminates a manual tagging process for your data, saving you time and providing standard taxonomy across all your data sources.

Getting started

This guide illustrates how to implement sensitive data discovery and classification.

Introduction

This reference guide discusses the components and benefits of Immuta Discover.

Architecture

This reference guide describes the design of Immuta Discover.

Data discovery

Sensitive data discovery (SDD) is an Immuta feature that uses data patterns to determine what type of data your column represents. This saves the time of identifying your data manually and provides the benefit of a standard taxonomy across all your data sources in Immuta.

The guides in this section discuss the components of SDD and how to use it to tag your data.

Data classification

Classification is the process in which data is categorized by the content and the associated risk level based on context. The guides in this section illustrate how to configure and customize classification for your organization.

Requirements:

• Native SDD enabled and turned on

• Registered Snowflake, Databricks, Redshift, or Starburst (Trino) data sources

• Immuta permission GOVERNANCE

Create a framework

Create a framework with no rules

1. Click the Discover icon in the navigation menu and select the Frameworks tab.

2. Click Create New.

3. Enter a Name for the framework.

4. Enter a Description for the framework.

5. Select the option to Create empty framework.

6. Click Create.

After you create the framework, you can create new rules for it.

Copy an existing framework and its rules

1. Click the Discover icon in the navigation menu and select the Frameworks tab.

2. Click Create New.

3. Enter a Name for the framework.

4. Enter a Description for the framework.

5. Select the option to Create rules from an existing framework.

6. Select the checkbox for the framework you want to copy. You can only copy a single framework. For more information about a framework, click the framework name to open a new tab with details about the framework.

7. Click Create.

Assign a framework to data sources

To assign a framework to run on specific data sources,

1. Click the Discover icon in the navigation menu and select the Frameworks tab.

2. Select the framework you want to assign and navigate to the Data Sources tab.

3. Click Add Data Sources.

4. Select the checkbox for the data source you want this framework to run on. You may select more than one.

5. Click Add Data Source(s).

Remove data sources from a framework

After a data source is removed from a framework, it will use the global framework for any SDD scans and the tags applied by the removed framework will be replaced. To remove data sources from a framework,

1. Click the Discover icon in the navigation menu and select the Frameworks tab.

2. Select the framework you want to remove data sources from and navigate to the Data Sources tab.

3. Select the checkbox for the data source you want to remove from the framework. You may select more than one.

4. Select the Bulk Actions more options.

5. Select Remove Data Sources.

6. Click Confirm.

Delete a framework

Deleting a framework will remove it from any data sources. Those data sources will then use the global framework for any SDD scans and the tags applied by the deleted framework will be replaced. Governors can delete any framework, and users with the CREATE_DATA_SOURCE or CREATE_DATA_SOURCE_IN_PROJECT permissions can only delete frameworks they created. To delete a framework,

1. Click the Discover icon in the navigation menu and select the Frameworks tab.

2. Click the three dot menu in the Action column for the framework you want to delete. Note that the global framework cannot be deleted. If you want to delete it, configure a different framework as the global framework.

3. Select Remove.

4. Click Confirm.

Discover automates discovering and tagging data across your data platform. It encompasses the identification and classification of data using frameworks.

Requirements

• Native SDD enabled

• Frameworks enabled

• Registered Snowflake, Databricks, Redshift, or Starburst (Trino) data sources

Components

The Immuta UI has separate sections for identification frameworks and classification frameworks. Both frameworks are made of rules, criteria, and resulting tags, but the criteria types differ for each framework type. Identification frameworks use competitive pattern matching and column name matching to discover data types and tag them. Classification frameworks use tags on the column, neighboring columns, and data source for context and then tag the columns based on that context. Find more information about each framework type below.

Identification frameworks

Identification frameworks run with sensitive data discovery (SDD). They use data patterns to discover data and tag it based on what the data is.

Supported criteria and pattern types

• Competitive pattern analysis: This criteria is a process that will review all the regex and dictionary patterns within the rules of the framework and search for the pattern with the best fit. In this review, each competitive pattern analysis criteria in the framework competes against each other to find the best and most specific pattern that fits the data. The resulting tags for the best pattern's rule are then applied to the column.

  • Regex pattern: This pattern contains a case-insensitive regular expression that searches for matches against column values. Create a regex pattern in the UI or with the sdd/classifier endpoint.

  • Dictionary pattern: This pattern contains a list of words and phrases to match against column values. Create a dictionary pattern in the UI or with the sdd/classifier endpoint.

• Column name: This criteria matches a column name pattern to the column names in the data sources. The rule's resulting tags will be applied to the column where the name is found.

  • Column name pattern: This pattern includes a case-insensitive regular expression matched against column names, not against the values in the column. Create a column name pattern in the UI or with the sdd/classifier endpoint.

Related guides

• To start using identification frameworks in the UI, see the Getting started guide.

• To manage identification frameworks with the API, see the /sdd/template endpoint reference guide.

Classification frameworks

Classification frameworks run with the classify service. They determine rule match and criteria fit based on proximity tags and then tag data based on the context it is within.

Supported criteria

• Match column tag: This criteria applies resulting tags based on specific tags already on the column.

• Match neighboring column tag: This criteria applies resulting tags based on specific tags on neighboring columns.

Related guides

• To manage classification frameworks in the UI, see the Activate frameworks guide.

• To create a classification framework with the API, see the /frameworks endpoint reference guide.

Data inventory dashboard

The data inventory dashboard visualizes information about your organization's data. It presents your entire data corpus within the context of the frameworks you have actively tagging your data with details like when your data was scanned last or how much of the scanned data is relevant to your active frameworks.

In the data inventory dashboard you will see tiles for scanned coverage and the percent of data scanned within a specific time frame. These tiles are referencing data scanned by an identification framework with SDD. To increase the number of your data sources that have been scanned, run SDD.

The next section of the dashboard shows tiles for the compliance frameworks. Within each graph is the separation of columns found containing or not containing the data important to the compliance framework. These graphs update every time classification runs, which will happen from these events.

For information on the frameworks visualized in the dashboard, see the Immuta frameworks reference guide.

Workflow

The Discover workflow involves both identification with SDD and classification:

1. A user with the GOVERNANCE permission enables SDD and activates classification frameworks.

2. Users register data in Immuta.

3. SDD runs:

   1. Immuta generates a SQL query using the identification framework's rules.

   2. That query is executed in the native database.

   3. Immuta receives the query results containing the column name and the matching rules but no raw data values.

   4. SDD applies the resulting tags to the relevant columns.

4. Classification runs:

   1. The data source's current tags are checked against the framework's rules.

   2. When a matching rule is found, the resulting tags are applied to the relevant columns.

5. Users with the GOVERNANCE permission or data owners can view the data inventory dashboard with visualizations of their scanned data.

Frequency

This workflow will run when a new data source is manually registered in Immuta or found from schema monitoring. Additionally, SDD alone will run from the following events:

• A new data source is created.

• Schema monitoring is enabled, and a new data source is detected.

• Column detection is enabled, and new columns are detected. Here, SDD will only run on new columns, and no existing tags will be removed or changed.

• A user manually triggers it from the data source health check menu.

• A user manually triggers it from the identification frameworks page.

• A user manually triggers it through the API.

Classification will run from the following events:

• A framework gets created, updated, or deleted.

• A tag gets added to or removed from a column manually or by SDD.

• A tag gets added to a data source.

• A user manually triggers it from the data source health check menu.

• A user manually triggers it through the API.

Caveat

• Customizing classification frameworks currently requires users to use the Immuta API.

Discover section contents

Conceptual guides:

• Data classification

Getting started guide:

• Getting started with Discover

How-to guides:

• Identification guides:

  • Enable SDD

  • Create an identification framework

  • Create a pattern

  • Manage identification rules

  • Manage SDD and discovered Tags

  • Manage global SDD settings

• Classification guides:

  • Activate a classification framework

  • Adjust and accept entity and classification tags

Reference guides:

• Built-in pattern reference

• Discovered tag reference

• Built-in classification frameworks reference

Sensitive data discovery (SDD) is an Immuta feature that uses data patterns to determine what type of data your column represents. Using frameworks, rules, and patterns, Immuta evaluates your data and can assign the appropriate tags to your data dictionary based on what it finds. This saves the time of identifying your data manually and provides the benefit of a standard taxonomy across all your data sources in Immuta.

Supported technologies

Native SDD supports data discovery on data sources from the following technologies:

• Snowflake

• Databricks or Databricks Unity Catalog

• Starburst (Trino): Native SDD for Starburst (Trino) is currently in public preview and available to all accounts. Enable this feature on the Immuta app settings page.

• Redshift: Native SDD for Redshift is currently in private preview and available to all accounts. Please reach out to your Immuta representative to enable it on your tenant.

Architecture

To evaluate your data, SDD generates a SQL query using the identification framework's rules; the Immuta system account then executes that query in the native technology. Immuta receives the query result, containing the column name and the matching rules but no raw data values. These results are then used to apply the resulting tags to the appropriate columns.

This evaluating and tagging process occurs when SDD runs, which happens automatically from the following events:

• A new data source is created.

• Schema monitoring is enabled and a new data source is detected.

• Column detection is enabled and new columns are detected. Here, SDD will only run on new columns and no existing tags will be removed or changed.

Users can also manually trigger SDD to run from a data source's overview page or the identification frameworks page.

Components

Sensitive data discovery (SDD) runs frameworks to discover data. These frameworks are a collection of rules. These rules contain a single criteria and the resulting tags that will be applied when the criteria's conditions have been met. See the sections below for more information on each component.

Identification framework

An identification framework is a collection of rules that will look for a particular criteria and tag any columns where those conditions are met. While organizations can have multiple frameworks, only one may be applied to each data source. Immuta has the built-in Default Framework, which contains all the built-in patterns and assigns the built-in Discovered tags based on pattern matching.

For a how-to on the framework actions users can take, see the Manage frameworks page.

Global framework

Each organization has a single global framework that will apply to all the data sources in Immuta by default, unless they have a different framework assigned. It is labeled on the frameworks page with a globe icon. Users can bypass this global framework by applying a specific framework to a set of data sources.

Rule

A rule is a criteria and the resulting tags to apply to data that matches the criteria. When Immuta recognizes that criteria, it can tag the data to describe the type. Each rule is specific to its own framework, but all a framework's rules can be copied to create a new framework.

For a how-to on the rule actions users can take, see the Manage rules page.

Criteria

Criteria are the conditions that need to be met for resulting tags to be applied to data.

Supported criteria types

• Competitive pattern analysis: This criteria is a process that will review all the regex and dictionary patterns within the rules of the framework and search for the pattern with the best fit. If there are multiple rules in a framework using competitive pattern analysis, only one will be applied to any column. To learn more about the competitive nature, see the How competitive pattern analysis works guide.

• Column name: This criteria matches a column name pattern to the column names in the data sources. The rule's resulting tags will be applied to the column where the name is found.

Pattern

A pattern is the type of data Immuta will look for to meet the requirements to tag a column. They can be used in rules across multiple frameworks, but can only be used once within each framework. Immuta comes with built-in patterns to discover common categories of data. These patterns cannot be modified and are within preset rules with preset tags. Users can also create their own unique patterns to find their specific data. SDD only supports regex patterns written in RE2 syntax.

Supported pattern types

The three types of patterns are described below:

• Regex: This pattern contains a case-insensitive regular expression that searches for matches against column values.

• Column name: This pattern includes a case-insensitive regular expression that is only matched against column names, not against the values in the column.

• Dictionary: This pattern contains a list of words and phrases to match against column values.

Configuration

Only application admins can enable sensitive data discovery (SDD) globally on the Immuta app settings page. Then, data source creators can disable SDD on a data-source-by-data-source basis.

Tag mutability

When SDD is manually triggered by a data owner, all column tags that were previously applied by SDD are removed and the tags prescribed by the latest run are applied. However, if SDD is triggered because a new column is detected by schema monitoring, tags will only be applied to the new column, and no tags will be modified on existing columns. Additionally, governors, data source owners, and data source experts can disable any unwanted Discovered tags in the data dictionary to prevent them from being used and auto-tagged on that data source in the future.

Performance

The amount of time it takes to run identification on a data source depends on several factors:

• Columns: The time to run identification grows nearly linearly with the number of text columns in the data source.

• Identifiers: The number of identifiers being used weakly impacts the time to run identification.

• Row count: Performance of identification may vary depending on the sampling method used by each technology. For Snowflake, the number of rows has little impact on the time because data sampling has near-constant performance.

• Views: Performance on views is limited by the performance of the query that defines the view.

The time it takes to run SDD for all newly onboarded data sources in Immuta is not limited by SDD performance but by the execution of background jobs in Immuta. Consult your Immuta account manager when onboarding a large number of data sources to ensure the advanced settings are set appropriately for your organization.

Testing

For users interested in testing SDD, note that the built-in patterns by Immuta require a certain amount of confidence to be assigned to a column. This means that with synthetic data, there may be situations where the data is not real enough to fit the confidence needed to match patterns. To test SDD, use a dev environment, create copies of your tables, or use the API to run a dryRun and see the tags that would be applied to your data by SDD.

Considerations

Deleting the built-in Discovered tags is not recommended: If you do delete built-in Discovered tags and use the Default Framework, then when the pattern is matched the column will not be tagged. As an alternative, tags can be disabled on a column-by-column basis from the data dictionary, or SDD can be turned off on a data-source-by-data-source basis when creating a data source.

Supported data types and casing

Limitations with dictionary patterns

Immuta compiles dictionary patterns into a regex that is sent in the body of a query.

For Snowflake, the size of the dictionary is limited by the overall query text size limit in Snowflake of 1 MB.

Databricks limitations

• For Databricks, Immuta will start up a Databricks cluster to complete the SDD job if one is not already running. This can cause unnecessary cost if the cluster becomes idle. Follow Databricks best practices to automatically terminate inactive clusters after a set period of time.

• Native SDD for Databricks Unity Catalog will only work on data sources authenticated with a personal access token (PAT). OAuth machine-to-machine (M2M) is not supported with SDD.

Starburst (Trino) limitation

Native SDD will only work on Starburst (Trino) data sources authenticated with username and password. OAuth 2.0 is not supported with SDD.

Redshift limitations

• Redshift Spectrum is not supported with native SDD.

Redshift supported authentication methods

• Username and password is fully supported with native SDD.

• Okta is not supported with native SDD.

• AWS access key is supported with limitations with native SDD:

  • The AWS access key used to register the data source can do a minimum of the following redshift-data API actions:

    • redshift-data:BatchExecuteStatement

    • redshift-data:CancelStatement

    • redshift-data:DescribeStatement

    • redshift-data:ExecuteStatement

    • redshift-data:GetStatementResult

    • redshift-data:ListStatements

  • The AWS access key used to register the data source must have redshift:GetClusterCredentials for the cluster, user, and database that they onboard their data sources with.

  • If using a custom URL, then the data source registered with the AWS access key must have the region and clusterid included in the additional connection string options formatted like the following:

    Copy

      region=us-east-2;clusterid=12345

  • Redshift Serverless data sources are not supported for native SDD with the AWS access key authentication method.

Migrating from legacy to native SDD

If you had legacy SDD enabled, running native SDD can result in different tags being applied because native SDD is more accurate and has fewer false positives than legacy SDD. Running a new SDD scan against a table will change the context of the resulting tags, but no Discovered tags previously applied by legacy SDD will be removed.

See the Migrate from legacy to native SDD page for more information.

Immuta allows you to automate discovering and tagging data across your data platform. Tagging is critical for two reasons:

• It allows you to define data sensitivity, which in turn allows you to monitor where you have potential data security issues and gaps in your security posture.

• It allows you to abstract your physical structure from your access policy logic. For example, you can build access policies like mask all columns tagged PII (where PII was auto-tagged by Discover) rather than much less scalable policies that must be knowledgeable of your physical layers like mask column x in database y in data platform z.

Challenge and goals

Today’s sensitive data discovery tools give you a shallow overview of your data corpus across a long list of platforms. They give you pointers on where you have sensitive data without the granularity to drive your column- or row-level access controls. They help you understand what data you possess according to a regulatory framework, like HIPAA or PCI, but without the details needed to automate your audits or compliance reporting. Knowing that you need to drive east to west on a road map from New York to California is helpful but ultimately insufficient to get you from a specific location to another.

Existing tools promise a high degree of automation, yet their many false positives result in painful manual work that never stops. Although data gets scanned automatically, performance breaks down at scale, or you manually need to fine-tune the computing resources of the scanners. Last but not least, your security team objects to the agent-based processing that requires taking data out of your data platform, and the associated data residency concerns may give you pause.

At Immuta, we believe that data security should not be painful. We believe that you can innovate and move quickly, while at the same time protecting your data and adhering to your internal policies and external regulations. Technology and automation allow you to make the right trade-off decisions quickly. It all starts with highly accurate and actionable metadata. If you trust your metadata and if it’s actionable, you can leverage it to automatically grant access to data, mask sensitive information, and automate your audit reporting.

Immuta Discover was built to tackle those challenges and address them through a unique architecture that was designed in collaboration with the largest financial institutions, healthcare companies, and government agencies in the world. The cloud and AI paradigm requires a fundamentally different approach. You must assume that your data is dynamic, unique, and collected in a multitude of different geographies and legal jurisdictions. Immuta Discover is built for this new world and its specific demands.

How does it work?

Scalability through in-platform processing

Identifying and classifying data requires analyzing and looking at the data - there’s no way around it. Immuta Discover does all the analysis and processing inside the native technology. It takes advantage of those platforms’ inherent scalability to enable you to analyze large amounts of data quickly, efficiently, and without the need for separate resource optimization for containers or virtual machines.

Data residency compliance by design

By processing data directly inside the data platform, Immuta Discover automatically adheres to data residency and locality requirements. If you run your data warehouse or lake globally - across North America, the European Union, and Asia - Immuta processes the data in the region where your data is stored. No data ever leaves the data platform, and it will never move across different cloud regions.

Improved security and simplicity through agentless scanning

In-platform processing greatly reduces risk and improves your data security posture. Provisioning agents, whether they’re in a container, virtual machine, or Amazon Machine Image (AMI), create complexity and an unnecessary security risk. Not only can those agents become compromised, but their misconfiguration might lead to data leaks to other parts of your cloud infrastructure. An agentless approach can better leverage data platform optimizations to process data instead of transferring it out to re-optimize and analyze. This simplifies operations and increases efficiency for your infrastructure teams.

Cross-platform consistency

The advantages of in-platform processing are abundant, but implementing it across a multitude of platforms is challenging. Immuta helps bypass the obstacles by doing all the heavy lifting for you and building in specific implementations for each technology. Although all those implementations are ultimately different, Immuta abstracts the results to one standardized taxonomy, so you can have consistently accurate and granular metadata across all your data stores.

Granular query-level classification

Immuta Discover classifies data on a column level and instantaneously identifies schema changes. Only with that level of granularity and automation can you adhere to your audit requirements and understand what actions have been taken on your data. For example, if non-sensitive data is joined with sensitive data at query time, Immuta Discover will monitor and record that for your review. Continuous schema monitoring ensures schema changes never result in holes in your access controls and data security posture.

Highly accurate and actionable metadata

Trust in your metadata is critical for data security.

To unblock your data consumers, you need to automate your data access controls; this requires trusting that your classification and metadata are accurate and actionable. Immuta Discover provides you with highly accurate metadata and tags out-of-the-box and assists you in fine-tuning the classification mechanism to deal with false positives quickly. That enables you to build policies that dynamically grant or restrict access to protected data (like PHI or PII) depending on who is accessing it and what protections you want to apply.

Components of Discover

Immuta Discover works in three phases: identification, categorization, and classification.

1. Identification: In this first phase, data is identified by its kind – for example, a name or an age. This identification can be manually performed, externally provided by a catalog, or automatically determined by Immuta Discover through column-level analysis of patterns.

2. Categorization: In the second phase, data is categorized in the context of where it appears, subject to any active data compliance or security frameworks. For example, a record occurring in a clinical context containing both a name and individual health data is protected health information (PHI) under HIPAA.

   While every phase can and should be customized, for categorization Immuta provides a bundle of default frameworks. The generic Data Security Framework provides the base for the specific frameworks and gives fine-grained categorization of your data into a consistent set of security and compliance concepts. This categorization of data helps to understand the context it is in, including information like whether or not a record pertains to an individual, the composition and kinds of identifiers present, the data subject, whether the data belongs to any controlled data categories under certain legislation, etc.

   The categorization provided by the Immuta classification frameworks may be used out-of-the-box; however, they are best leveraged as a starting point for purpose-built compliance frameworks implementing organization-specific compliance categories.

3. Classification: In the third and final phase, data is classified according to its sensitivity level (e.g., Customer Financial Data is Highly Sensitive) and the risk associated to the data subject. Immuta supplies sensitivity level defaults in Detect and risk assessment default tags based on standard industry practice. However, customers are free to customize the assignments under their respective views.

Requirements:

• Native SDD enabled and turned on

• Immuta permission GOVERNANCE

Create a pattern

1. Click the Discover icon in the navigation menu and select the Patterns tab.

2. Click Create New.

3. In the modal, enter a name for the new pattern.

4. Write a Description for the type of data the pattern will find.

5. Select the Type of pattern.

   1. For regex and column name regex, enter the regex.

   2. For dictionary, enter the values you want the pattern to match and toggle the switch on if you want them to be case-sensitive.

6. Click Create Pattern.

7. See the Manage rules page to add your new pattern to a framework.

Note that all user-created patterns must be a 90% match or greater for the contents of the column to be tagged.

Edit a pattern

Editing a pattern will affect any rule built off the pattern throughout Immuta. To edit a pattern,

1. Click the Discover icon in the navigation menu and select the Patterns tab.

2. Click the name of the pattern you want to edit.

3. Click Edit.

4. Edit the field you want to change. Note any field shadowed is not editable, and the pattern must be deleted and re-created to change them.

5. Click Save.

Built-in patterns cannot be edited.

Delete a pattern

Deleting a pattern will remove it from Immuta and remove all the rules that relied on it in the frameworks throughout Immuta. To delete a pattern,

1. Click the Discover icon in the navigation menu and select the Patterns tab.

2. Click the three dot menu in the Action column for the pattern you want to delete.

3. Select Remove.

4. Click Confirm.

Built-in patterns cannot be deleted.

Requirement: Immuta permission GOVERNANCE

This how-to guide is for enabling sensitive data discovery (SDD). For additional information on sensitive data discovery and classification, see the Discover architecture page.

1. Navigate to the App Settings page and scroll to the Sensitive Data Discovery section.

2. Select the Enable Sensitive Data Discovery (SDD) checkbox to enable SDD.

3. Click Save and then click Confirm to apply your changes. Note that the Immuta tenant will have a system restart.

4. Run SDD for a select group of data sources; use one of the following options to run SDD on specific data sources:

   1. Run SDD on a data source in the UI.

   2. Run SDD using a specific framework in the UI.

   3. Make the following request specifying the data sources in the request using the Immuta API.

      Copy

      curl \
          --request 'POST' \
          'https://your-immuta-url.immuta.com/sdd/run' \
          --header 'Content-Type: application/json' \
          --header 'Authorization: 438a3096966c4a5188b3b468cedb213e' \
          --data '{"sources":["Example Data Source Name", "Example Data Source 2 Name"]}'

      A successful request will have the code 200 and a body with the number of jobs created from the request:

      Copy

      {
          "jobCount": 2
      }

5. Navigate to the data source overview page of the data source you listed in the payload.

6. Click the Data Dictionary tab.

7. Assess whether the Discovered and classification tags applied are accurate.

8. If they are, then repeat the steps above for more of your data sources. Once a majority of your data sources appear to have accurate tags, run SDD on all your data sources. If the tags are not accurate, you will need to tune SDD and classification frameworks. See the Adjust frameworks and tags guide for instructions.

Run SDD on all data sources

1. Click the Discover icon and the Identification tab in the navigation menu.

2. Select the more actions icon.

3. Select Run SDD and then select it again in the modal.

Run SDD on all data sources using the API

Requirement: Immuta permission GOVERNANCE

Make the following request using the Immuta API to run SDD for all data sources, specifying all as true:

curl \
    --request 'POST' \
    'https://your-immuta-url.immuta.com/sdd/run' \
    --header 'Content-Type: application/json' \
    --header 'Authorization: 438a3096966c4a5188b3b468cedb213e' \
    --data '{"all": true}'

A successful request will have the code 200 and a body with the number of jobs created from the request:

{
    "jobCount": 12
}

Discover scans your data sources and applies relevant tags when data is recognized. This eliminates a manual tagging process for your data, saving you time and providing standard taxonomy across all your data sources.

Requirements

• Native SDD enabled and turned on

• Frameworks enabled

• Registered Snowflake, Databricks, Redshift, or Starburst (Trino) data sources

• Immuta permission GOVERNANCE

Implement SDD

Sensitive data discovery (SDD) is an Immuta Discover feature that scans your data sources and applies relevant tags when data is recognized. This eliminates a manual tagging process for your data, saving you time and providing standard taxonomy across all your data sources.

To learn more, see the Data discovery page.

Enable sensitive data discovery

Enable sensitive data discovery to start using the default framework on all of your registered data sources. This out-of-the-box framework discovers common data types and tags them automatically when a new data source is registered.

1. Turn on SDD

2. Trigger SDD on a data source

Configure SDD for your data

For additional control, create your own patterns to recognize the data that matters to you. Add these patterns to new frameworks and specify the data sources that need this framework. This fine-level control creates automatic tagging that is relevant and accurate to your data, requiring fewer manual adjustments to the resulting tags.

1. Customize SDD for your data:

   1. Create patterns

   2. Create a framework

   3. Create rules

2. Assign your framework to specific data sources

Adjust Discovered tags

If you have any tags that are applied to your data sources by SDD that you don't want, you can easily disable these tags for each data source. This ensures that they will not be applied to the data source again if SDD is re-run.

1. Verify tags

2. Disable Discovered tags

Reference pages:

Immuta comes with a default framework containing built-in Discovered tags and built-in patterns. These patterns and tags can be used in your own frameworks.

1. Built-in Discovered tags

2. Built-in patterns

Implement classification

Classification is an Immuta Discover feature that categorizes your data based on the content and the associated risk the data poses. This increases your understanding of your data and allows you to make faster decisions about it.

Enable classification

Enable classification from the Immuta app settings page.

Activate a framework

Activate any of the following frameworks:

• To start seeing classification tags, enable the Data Security Framework.

• If you are using Snowflake and want to see information on your sensitive data in Detect, enable the Risk Assessment Framework.

• Opt to enable any of the other compliance frameworks.

Complete the following steps for each framework you want to activate:

1. Navigate to Discover and select the Classification tab.

2. Click the more actions icon in the Actions column for the framework you want to activate.

3. Select Activate.

Configure classification for your data

To configure or manage a framework using the Immuta API, see the Frameworks API reference page.

Adjust classification tags

If you have any tags that are applied to your data sources by classification that you don't want, you can easily disable these tags for each data source. This ensures that they will not be applied to the data source again when classification is re-run.

1. Assess your data source tags.

2. Tune your data dictionaries.

Requirement: Immuta permission APPLICATION_ADMIN

Configure the global framework

1. Click the App Settings icon in the left sidebar.

2. Click Sensitive Data Discovery in the left panel to navigate to that section.

3. Enter the request-friendly name of your global template in the Global SDD Template Name field. This name can be found in the tooltip on the framework's detail page.

4. Click Save, and then Confirm your changes.

Requirements:

• Native SDD enabled and

• Immuta permission GOVERNANCE

Create a rule

You can only have one rule per pattern in the framework. If you do not see the pattern for the rule you want to create, then it already has a rule built off of it.

1. Click the Discover icon in the navigation menu and select the Framework tab.

2. Select the framework you want to edit and navigate to the Discovery Rules tab.

3. Click Create New.

4. Select the Tags to apply from the dropdown. The tags you select are the tags applied when the pattern is matched. Note that resulting tags must be under the Discovered parent tag and cannot be parent tags themselves unless they have already been manually applied to a data source.

5. Select the Criteria type from the dropdown. See the .

   1. Competitive pattern analysis is for regex and dictionary patterns.

   2. Column name is for column name patterns.

6. Select the Pattern from the dropdown.

7. Click Create Rule.

Edit a rule

1. Click the Discover icon in the navigation menu and select the Frameworks tab.

2. Select the framework of the rule you want to edit and navigate to the Discovery Rules tab.

3. Select the rule you want to edit.

4. Click Edit.

5. Edit the field you want to change. Note any field shadowed is not editable, and the rule must be deleted and re-created to change them.

6. Click Save.

Delete a rule

Deleting a rule removes the tags once applied by that rule the next time SDD runs on a data source. To delete a rule,

1. Click the Discover icon in the navigation menu and select the Frameworks tab.

2. Select the framework you want to edit and navigate to the Discovery Rules tab.

3. Click the three dot menu in the Action column for the rule you want to delete.

4. Select Remove.

5. Click Confirm.

Immuta is pre-configured with a set of tags that can be used to write global policies before data sources even exist. See a list of the built-in Discovered tags below and the for information about where these tags will be applied by the built-in rules

Country tags

All the tags below belong to the Country parent. For example, the full tag name will appear as Discovered . Country . Argentina.

Entity tags

All the tags below belong to the Entity parent. For example, the full tag name will appear as Discovered . Entity . Aadhaar Individual.

Identifier tags

None of the tags below have an additional parent or child tag. For example, the full tag name will appear as Discovered . Identifier Direct.

Personal information tags

None of the tags below have an additional parent or child tag. For example, the full tag name will appear as Discovered . PCI.

Requirements:

• Native SDD enabled and

• Registered

• Immuta permission GOVERNANCE

Run SDD using a specific framework

SDD runs automatically, but if you want to re-run SDD when a new global framework is set or when new rules have been added, you can or for specific frameworks through the UI:

1. Click the Discover icon and the Identification tab in the navigation menu.

2. Select the more actions icon.

3. Select Run SDD and then select it again in the modal.

Run SDD on a data source

SDD runs automatically, but if you want to re-run SDD when a new global framework is set or when new rules have been added, you can or for specific data sources through the UI:

1. Navigate to the data source overview page.

2. Click the health status.

3. Select Re-run next to Sensitive Data Discovery (SDD).

Verify discovered tags

Disable Discovered tags from the data dictionary

If a governor, data owner, or data source expert disables a Discovered tag from the data dictionary, the column will not be re-tagged when that data source's fingerprint is recalculated or SDD is re-run. When a Discovered tag is disabled, the tag will not completely disappear, so it can be manually enabled through the tag side sheet.

To disable a discovered tag,

1. Navigate to a data source and click the Data Dictionary tab.

2. Scroll to the column you want to remove the tag from and click the tag you want to remove.

3. Click Disable in the side sheet and then click Confirm.

This guide provides information and best practices for migrating from the deprecated legacy sensitive data discovery (SDD) option to the improved native SDD. This guide is for users who have already enabled SDD on their tenant and have Discovered tags on their data sources.

Before you begin

Native vs legacy SDD

Legacy SDD is deprecated. It will be removed and replaced by native SDD. Native SDD is significantly improved from legacy SDD for discovering and tagging your data with upgrades to the built-in patterns. Additionally, the greatest benefit is the respect for data residency. Native SDD doesn't move any of your data when running. The discovery is done right in your data platform, and the platform only returns the matching patterns and column names to Immuta.

See the for more information on native SDD.

Requirements

• Native SDD requires Snowflake, Databricks, Redshift, or Starburst (Trino) data sources

• Legacy SDD enabled on your tenant

• Legacy SDD tags applied to your data sources: To find out if you have legacy SDD tags applied, create a governance report as described in the .

Enable native SDD

Contact your Immuta representative to enable native SDD on your Immuta tenant. Note that unless specifically disabled, all Immuta installations after the 2024.2 LTS have native SDD automatically enabled. Proceed to if you want to self-service check if native SDD is already running and tagging your data before you reach out to the representative.

This action will not change anything immediately on your tenant; however, anytime SDD runs in the future, it will be native SDD instead of the legacy version.

To assess native SDD for your data, proceed with the steps below. If you do not review native SDD, the legacy SDD tags will all remain on your data source columns. However, when on new data sources and columns, it will apply native SDD tags, and because of the improvements to SDD, it may tag different data than legacy SDD.

Understand the context of your tags

Requirement: Immuta permission GOVERNANCE

2. To check the tags on an individual data source, navigate to the data source data dictionary and select a Discovered tag. On the tag side sheet, you can determine the context of the tag. When patterns match data, native SDD will apply tags, and their tag context will be Sensitive Data Discovery. Any tags with the context Legacy Sensitive Data Discovery were not matched by native SDD but will remain on the data source.

3. To check your tags globally, navigate to the governance reports page and build a report for sensitive data discovery. This report will present the legacy tags on your data sources' columns and native SDD tags that are also on those columns. Use this report to assess the context of the Discovered tags and understand if native SDD is matching the data you want it to.

These actions will allow you to understand the differences between how native SDD and legacy SDD tag your data and whether your data is recognized as expected by native SDD or if legacy SDD was over-tagging your data. This way you can better tune SDD to your data.

If there are any legacy SDD tags that you want native SDD to catch, you need to tune native SDD so that this type of data is discovered in future tables and columns; see guidance on that in the next section.

Tune SDD

Requirement: Immuta permission GOVERNANCE

Using the report you built above, complete these actions to tune SDD:

1. Focus on a legacy SDD tag properly applied to your data. Assess whether the native SDD tag on the column instead was applied more accurately than the legacy tag. If it is applied incorrectly, proceed to the next step.

4. Complete the steps above for all legacy SDD tags.

Completing the actions above will create parity between what legacy SDD was tagging your data and what native SDD will tag in the future.

Immuta comes with a set of built-in patterns that look for common data types. These patterns were written by Immuta's research and development team and cannot be deleted or edited by users. However, users can build their own rules using these built-in patterns, which will customize the resulting tags based on the organization's needs.

When using SDD with , it is recommended to use the default resulting tags listed in the table below for these built-in patterns. This ensures that the framework rules apply sensitivity tags as intended.

Pattern descriptions and default resulting tags

Discover comes preconfigured with a bundle of classification frameworks for use out-of-the-box once endorsed by your organization's admins. These frameworks are designed by Immuta’s Legal Engineering and Research Engineering teams and informed by data privacy regulations and security standards: GDPR, CCPA, GLBA, HIPAA, PCI, and global best practices. They are a starting point for companies to customize to their own classification, security, and risk policies.

Data Security Framework

The Data Security Framework is the general classification framework. It provides the groundwork for categorizing data based on its context but is not specific to any regulatory framework and does not assign sensitivity or risk values to the data it tags. It provides a consistent taxonomy used throughout Immuta, from other built-in frameworks to customized frameworks that classify data valuable to your organization to Secure data and subscription policies.

The Data Security Framework is a supportive tool that accelerates data classification. Use the Data Security Framework in tandem with Discover identification frameworks out-of-the-box for the easy and quick onboarding of data sources and tags. Then, choose the compliance frameworks that matter to your industry or start building your own classification frameworks that assign sensitivity to the specific data of your organization. Your organization's compliance team should review the compliance frameworks as you would a template for a policy or contract and adapt them as needed to ensure a complete inventory and proper classification of your data.

You can view the Data Security Framework tags and their descriptions from the tags page in the UI or from the data dictionary when they are applied to a data source. Note the field and record tags. While they seem similar, the field and record tags are both necessary to convey the content of your data. Field tags describe the content of the columns, and record tags describe the content of the table.

Risk Assessment Framework

The Risk Assessment Framework provides the visible tags to your data's sensitivity based on the confidentiality risks it poses to your organization or the data subjects.

Use the Risk Assessment Framework out-of-the-box with the Data Security Framework and Discover identification frameworks to provide sensitivities to view in the Detect dashboards. Additionally, you can copy the framework using the API and create new rules to assign risk level and sensitivity to other data specific to your use case.

Risk assessment tags

The risk assessment tags have sensitivity level metadata assigned to them that will appear in the Detect dashboards as non-sensitive (when no risk assessment tag is applied), sensitive, and highly-sensitive. Additionally, use the risk assessment tags to build Secure policies to restrict access to highly-risky and confidential data.

Compliance frameworks

Immuta comes with four regulatory frameworks informed by the best practices of a specific regulation or standard. These are designed by Immuta’s Legal Engineering and Research Engineering teams as a general interpretation, but each organization should customize them based on their internal practices:

1. CCPA Framework: Classifies personal sensitive information controlled under the California Consumer Privacy Act (CCPA), as amended by the Consumer Privacy Rights Act (CCPA). This framework tags personal information, including communication content (like the body of a text message) and details about an individual's sexual orientation, religion, race, or biometric data.

2. GDPR Framework: Classifies personal data of specific categories protected under the EU General Data Protection Regulation (GDPR). This framework tags personal data, including details about an individual's health, sexual orientation, religion, race, or biometric data.

3. HIPAA Framework: Classifies protected health data controlled under the US Health Insurance Portability and Accountability Act (HIPAA). This framework tags health data connected to a specific individual.

4. PCI Framework: Classifies payment card information relevant to the Payment Card Industry (PCI) standard. This framework tags payment card information, including account, authentication, and cardholder data.

Some compliance frameworks are used to to add context and apply Data Security Framework tags. Use the data inventory dashboard to enable frameworks with information on the other frameworks they depend on.

About Immuta's frameworks

Organizations are responsible for making their own independent assessment of the framework rules. The framework rules are only templates and are not necessarily adapted to the specific context in which an organization operates. Framework rules do not constitute legal advice. They do not create any commitments or assurances from Immuta that users will necessarily comply with the statutes or standards that have informed these framework rules.

Requirements:

• Native SDD enabled and

• Registered

• Immuta permission GOVERNANCE

To activate a classification framework,

1. Navigate to Discover and select the Classification tab.

2. Click the more actions icon in the Actions column for the framework you want to activate.

3. Select Activate.

Repeat this process for all frameworks relevant to your data. See the for information on Immuta's built-in frameworks.

Deactivate a classification framework

1. Navigate to Discover and select the Classification tab.

2. Click the more actions icon in the Actions column for the framework you want to activate.

3. Select Deactivate.

Activate and manage classification frameworks using the API

Classification is the process in which data is categorized by the content and the associated risk level based on context. To classify your data, Discover evaluates your data in phases:

1. Sensitive data discovery (SDD) runs to identify your data by content type. The data is discovered and evaluated by the pattern it matches and is tagged.

2. The Data Security Framework scans those tags and any other tags applied to the data source and columns to categorize the data by context. This phase considers the data and the data surrounding it to understand the category of the data within the context of the data source.

3. Other regulatory-based frameworks scan and build off of the Data Security Framework tags. These frameworks are specific to regulations and standards and tag the data that matters to each framework.

4. The Risk Assessment Framework scans and builds off of the Data Security Framework. This framework tags data with specific risk assessment tags that describe the risk the data poses to your organization or the data subject. They also contain additional metadata used in the to describe the risk as sensitivity and visualize when that sensitive data is accessed.

Every phase of classification in Immuta can be customized to find and tag the data your organization cares about. Users can customize the Data Security Framework to find, match, and tag data they want categorized based on the organization's processes. Then, users can modify the by adjusting the sensitivity of classification tags to the organization’s policies or creating new tags and rules in customized frameworks. After data is classified, classification tags can be used to or .

Using Discover classification to assign risk and sensitivity levels to your data and Detect dashboards to visualize the risk levels offers these benefits:

• Increasing the semantic understanding of your data to better meet compliance requirements

• Reducing the time to make decisions about what data access is allowed under what purposes

• Reducing the effort and time to respond to auditors about data access in your company

• Reducing the labor of classifying data to enumerate what data is within the scope of security or regulatory compliance frameworks

What is the difference between entity tags and classification tags?

Both entity and classification tags describe the content of data on a per-column basis, and you can use them to and . However, there are key differences between the two:

• Entity tags are applied through identification and describe what the data is. SDD applies entity tags to columns based on the patterns of the data.

• Classification tags are applied through categorization and risk assessment and describe the context of the data and the risk it poses. Using classification frameworks, classification tags are applied to columns based on the entity tags previously applied by SDD. Additional classification tags can then be applied, providing even more context or expressing the property of the record rather than just the column.

Why isn’t entity tagging sufficient for classification?

Entity tags describe the contents of individual columns, in isolation. But you don't access individual columns in isolation, so why would you determine their sensitivity that way? Entity tags do not attempt to and cannot contextualize column contents with neighboring columns' contents. This means that connections between data are lost if they cannot be identified through a pattern within the column itself. Classification tags describe the contents of a table with the context of all its columns, providing a holistic view of the risk of the data for what it is, rather than the pattern it fits. Context is necessary to understand whether your data is public or private data, risky or safe to have ungoverned access, or sensitive and creating toxic joins when accessed with other tables.

Additionally, entity tagging does not indicate how sensitive the data is, but classification tags can carry a sensitivity level. For example, an entity tag may identify a column that contains telephone numbers, but the entity tag alone cannot say that the column is sensitive. A phone number associated with a person may be classified as sensitive, while the publicly listed phone number of a company might not be considered sensitive.

After you understand what entities your data contains using SDD, you need to adopt frameworks that determine what combinations of data constitute sensitive data and their level of sensitivity.

What is a framework?

Frameworks are a set of data categories and a set of classification rules to place data into those categories. In Immuta, the data categories are represented by tags, and when data fits a classification rule the tag is applied:

• Classification rules determine how each classification tag is applied. These rules can apply tags based on tags already on the column, tags applied to neighboring columns, and tags applied to the data source. This means that the complete data source is considered when classifying your data sources, and even tags applied to individual columns can affect the risk level of the entire data source.

What are the benefits of classification?

Data classification is a process, and with Immuta, much of it is automated. This means that you can reap the benefits of classified and tagged data quicker and easier than manually classifying and tagging it:

Requirements:

• Native SDD enabled and

• Registered

• Immuta permission GOVERNANCE

Immuta Discover provides identification frameworks out-of-the-box to recognize and tag data, and Discover also provides classification frameworks out-of-the-box to categorize and classify data. These frameworks are all generic to industry practices and should be customized to each organization's specific needs.

Tune SDD frameworks, rules, and patterns first to adjust where Discovered tags are applied. Because classification frameworks apply classification tags from the Discovered tags, tuning SDD should come first and will have trickle-down effects on classification. Customizing SDD requires some initial work but will automate data tagging for all data sources in the future.

Follow the steps below to tune SDD from the Default Framework:

1. : It is recommended to copy the Default Framework and adjust the rules from there.

2. .

3. .

4. : This will remove the tags from any previous identification frameworks and rerun SDD with your new framework. From here, either continue to edit patterns and rules to reconfigure the applied tags, or if you are happy with the results, proceed to the next step.

5. .

After SDD has applied entity tags, classification frameworks will automatically reapply their tags to account for any changes to Discovered tags. It may be necessary to adjust the classification tags based on your organization's data, security, and compliance needs.

Assess your queries with Detect

Requirements:

• Immuta permission AUDIT

Use the Detect dashboards to review queries at different sensitivity levels and review the tags that have been applied to your data source columns to understand the tags that Immuta applied there:

1. Have an Immuta user subscribed to a data source make multiple queries to a data source in Snowflake. The user should query both non-sensitive and sensitive data.

2. Navigate to the Audit page and click ↻Native Query Audit to pull in queries made in Snowflake.

3. Navigate to the Events (Beta) page. Note that Snowflake has a 15-minute data latency for all audit events.

4. Select the Event Id of one of the queries. Click the Columns tab.

5. The Column tab lists the columns in the query organized from highest to lowest sensitivity and the tags applied to each column. Check that the columns you know to be sensitive are here.

   For example, if the query has a column with last names, you should see a minimum of the following tags: Discovered.PII, DSF. Personal, DSF.Record.Subject.Type.Individual, DSF.Record.Identifiability.Identifiable, and DSF.Control.Personal.

6. Note any sensitive columns not labeled as sensitive.

7. Complete steps 2-5 for as many queries as you want.

Assess your data source tags

Requirement: Immuta permission GOVERNANCE or data owner

Target some data sources to manually review tags:

1. Navigate to the data dictionary for the data source by opening the Data Sources page and selecting a data source. Click the Data Dictionary tab to open the data dictionary.

2. The data dictionary lists the data source columns, with details about the name, data type, and a list of the tags on each column. Assess whether the tags are accurate to your data.

If you find that too many tags are applied

Tags may be unexpected but still accurate to your data. Additionally, they may have been applied because they were found to be the best match from the SDD patterns in the framework.

If you want to improve SDD and personalize it to your data,

1. Assess why the tag was applied to your data.

3. Is the pattern incorrectly matching this specific column, but correct in other places? It must have been the most correct match found by SDD. Create a better match by completing the following steps:

If you want to remove the unexpected tags, use one of the following how-to guides:

If you find that tags are missing

If you were expecting some sensitive data to be tagged and it is not, enable additional tags using one of the following how-to guides:

Tune your data dictionaries

Requirement: Immuta permissions GOVERNANCE and AUDIT

1. Navigate to the Data Sources page and select the data sources that you assessed and noted issues.

2. Click the Data Dictionary tab.

3. Delete unnecessary tags by clicking on the tag you want to remove from the column, and select Disable from the tag side sheet.

4. To add tags,

   1. Click Add Tags in the Actions column.

   2. Begin typing the name of the tag you want to add in the Search by Name field and select the tag from the dropdown list.

   3. Click Add.

The built-in in Immuta provide a quick way to leverage your own catalog or data platform tags to establish classifications tags. These classification tags can then be used in the Immuta Data Platform for query activity visualizations, monitors, reports, and policies. After you have configured a data catalog integration and registered data sources in Immuta, you can start automating data classification of a column based on its context by considering the combination of its associated tags, its neighboring columns' tags, or its table tag. Classification frameworks also provide . To use classification frameworks with your current tags from an external catalog, use one of the following options:

1. Follow the tutorial below: This starter framework is built to map a classification scale of restricted, confidential, internal, and public to Immuta's three level scale. It requires an , but all other steps are described below.

2. : This minimal framework allows you to map your own classification tags to Immuta classification tags. Then, your users' queries will have a sensitivity score on the Detect dashboard and in audit logs based on the classification tags on the data columns they queried. Use this option if you have already classified your organization’s data in an external catalog and want that metadata reflected in Immuta as Sensitive and Highly Sensitive.

3. : This option allows you to map your own tags describing your data to Immuta's predefined classification tags in the context of a specific compliance framework. Immuta provides built-in frameworks for GDPR, CCPA, and HIPAA. Map your tags to the most comparable Data Security Framework (DSF) tag, and Immuta will apply the classification tag based on the framework. Use this option if you have descriptive tags on your data and want that metadata mapped to a specific compliance framework.

Follow this guide to map your external catalog tags to the example framework, or consult the for more information about the framework schema.

Customize the framework

Using the example framework below, customize the framework for your organization's classification tags.

Example framework

Parameters

1. tags: These tags are automatically created in Immuta with the sensitivity you assign. All tags used in the classificationTag parameter should be defined here.

2. tags.sensitivities: This is metadata for the sensitivity of the new tag. Use confidentiality for dimension. Options for sensitivity are 1 (shown as sensitive in Detect dashboards) and 2 (shown as highly sensitive in Detect dashboards). For nonsensitive, leave this parameter empty.

3. rules: These are the rules for applying the tags defined above.

4. rules.classificationTag: This classification tag must be defined in tags. Add the name you want and the source is curated. This is the tag that will be applied if the rule requirement is met.

5. rules.columnTags: This object represents tags on a column. If the tag defined here is found on a column, then the rule's classificationTag will be applied to the same column.

6. rules.neighborColumnTags: This object represents tags on other columns in the data source. If the tag defined here is found on any column in the data source, then the rule's classificationTag will be applied to all the neighboring columns.

7. rules.tableTags: This object represents tags on the data source. If the tag defined here is found on the data source, then the rule's classificationTag will be applied to all the columns in that data source.

8. active: When true the framework is active and will apply tags when the rules are met.

How to edit rules

Follow the example below to map your external tags to the rules in the example framework.

The Immuta built-in framework, Risk Assessment Framework has a rule where columns tagged DSF.Interpretation.Credentials.Secret by sensitive data discovery will be tagged RAF.Confidentiality.High:

To translate this to your tags, replace the name and source value of the columnTags, neighborColumnTags, or tableTags with your own. This new example is for a Collibra tag that an organization uses for confidential data. This rule now states: Apply the classification tag RAF.Confidentiality.High to a column if it has the collibra tag Confidential. Repeat this for your organization's remaining classification levels.

Find the name and source for your tags

If you do not know the name or source for your tags, you can list your tags using the Immuta API:

This request will list all the tags in your Immuta environment, similar to this example response:

Activate your new framework

Requirement: Immuta permission GOVERNANCE

Once you have made all the customizations to the example framework, make the following request using the Immuta API, with your full customized framework as the payload.

Your new framework will now be visible in the Immuta UI by navigating the the Classification section under Discover.

Of sensitive data discovery's , regex and dictionary are competitive. This means that when assessing your data, if multiple patterns could match, only one of the competitive patterns will be chosen and tag the data. To better understand how Immuta executes this competition, read further.

Discover employs a three-phased competitive pattern analysis approach for sensitive data discovery (SDD):

1. : No data is moved, and Immuta checks the patterns against a sample of data from the table.

2. : Patterns that have less than a 90% match are filtered out.

3. : The remaining patterns are compared with one another to find the most specific pattern that qualifies and matches the sample.

In the end, competitive pattern analysis aims to find a single pattern for each column that best describes the data format.

Sampling

In the sampling process, no database contents are transmitted to Immuta; instead, Immuta receives only the column-wise hit rate (the number of times the pattern has matched a value in the column) information for each active pattern. To do this, Discover instructs a remote database to measure column-wise hit rate information for all active patterns over a row sample.

The sample size is decided based on the number of patterns and the data size, when available. In the most simplified case, the requested number of sampled rows depends only on the number of regex and dictionary patterns being run in the framework, not the data size. The sample size dependence on the number of patterns is weak and will not exceed 13,000 rows.

Sampling considerations

In practice, the number of sampled values for each column may be less than the requested number of rows. This happens when the target table has less than the requested number of rows, when many of the column values are null, or because of technology-specific limitations.

• Snowflake and Starburst (Trino): Discover implements native table sampling by row count.

• Databricks and Redshift: Due to technology limitations and the inability to predict the size of the table, Discover implements a best-effort sampling strategy comprising a flat 10% row sample capped at the first 10,000 sampled rows. In particular, under-sampling may occur on tables with less than 100,000 rows. Moreover, the resulting sample is biased towards earlier records.

• All platforms: Sampling from views can have significantly slower performance that varies by the performance of the query that defines the view.

Qualifying

Scoring

During the scoring phase, a machine inference is carried out among all qualified patterns, combining pattern-derived complexity information with hit rate information to determine which pattern best describes the sample data. This process prefers the more restrictive of two competing patterns since the ability to satisfy the more difficult-to-satisfy pattern itself serves as evidence that it is more likely. This phase ends by returning a single most likely pattern per the inference process.

Example

Here are a set of regex patterns and a sample of data:

Patterns:

1. [a-zA-Z0-9]{3} - This pattern will match 3 character strings with the characters a-z, lowercase or uppercase, or digits 0-9.

2. [a-c]{3} - This pattern will match 3 character strings with the characters a-c, lowercase.

3. (a|b|d){3} - This pattern will match 3 character strings with the characters a, b, or d, lowercase.

When qualifying the patterns, Pattern 1 and Pattern 3 both match 90% or more of the data. Pattern 2 does not, and is disqualified.

Then the qualified patterns are scored. Here, Pattern 1, despite matching 100% of the data, is unspecific and could match over 200,000 values. On the other hand, Pattern 3 matches just at 90% but is very specific with only 27 available values.

Therefore, with the specificity taken into account, Pattern 3 would be the match for this column, and its tags would be applied to the data source in Immuta.

Important notes

• Dictionaries are considered patterns by Immuta and are part of the competitive process, while column-name regex patterns are not.

• Scoring ties are rare but can occur if the same pattern is specified more than once (even in different forms). Scoring ties are inconclusive, and the scoring phase will not return a pattern in the case of a tie.

• Pattern complexity analysis is sensitive to the total number of strings a pattern accepts or, equivalently for dictionaries, the number of entries. Therefore, patterns that accept much more than is necessary to describe the intended column data format may perform more poorly in the competitive analysis because they are easier to satisfy.