Immuta allows you to automate discovering and tagging data across your data platform. Tagging is critical for two reasons:

• It allows you to define data sensitivity, which in turn allows you to monitor where you have potential data security issues and gaps in your security posture.

• It allows you to abstract your physical structure from your access policy logic. For example, you can build access policies like mask all columns tagged Person Name (where Person Name was auto-tagged by Immuta) rather than much less scalable policies that must be knowledgeable of your physical layers like mask column x in database y in data platform z.

Challenge and goals

Today’s sensitive data discovery tools give you a shallow overview of your data corpus across a long list of platforms. They give you pointers on where you have sensitive data without the granularity to drive your column- or row-level access controls. They help you understand what data you possess according to a regulatory framework, like HIPAA or PCI but without the details needed to automate your audit or compliance reporting. Knowing that you need to drive east to west on a road map from New York to California is helpful but ultimately insufficient to get you from a specific location to another.

Existing tools promise a high degree of automation, yet their many false positives result in painful manual work that never stops. Although data gets scanned automatically, performance breaks down at scale, or you manually need to fine-tune the computing resources of the scanners. Last but not least, your security team objects to the agent-based processing that requires taking data out of your data platform, and the associated data residency concerns may give you pause.

At Immuta, we believe that data security should not be painful. We believe that you can innovate and move quickly, while at the same time protecting your data and adhering to your internal policies and external regulations. Technology and automation allow you to make the right trade-off decisions quickly. It all starts with highly accurate and actionable metadata. If you trust your metadata and if it’s actionable, you can leverage it to automatically grant access to data, mask sensitive information, and automate your audit reporting.

Immuta was built to tackle those challenges and address them through a unique architecture that was designed in collaboration with the largest financial institutions, healthcare companies, and government agencies in the world. The cloud and AI paradigm requires a fundamentally different approach. You must assume that your data is dynamic, unique, and collected in a multitude of different geographies and legal jurisdictions. Immuta is built for this new world and its specific demands.

How does it work?

Scalability through in-platform processing

Identifying and classifying data requires analyzing and looking at the data - there’s no way around it. Immuta does all the analysis and processing inside the remote technology. It takes advantage of those platforms’ inherent scalability to enable you to analyze large amounts of data quickly, efficiently, and without the need for separate resource optimization for containers or virtual machines.

Data residency compliance by design

By processing data directly inside the data platform, Immuta automatically adheres to data residency and locality requirements. If you run your data warehouse or lake globally - across North America, the European Union, and Asia - Immuta processes the data in the region where your data is stored. No data ever leaves the data platform, and it will never move across different cloud regions.

Improved security and simplicity through agentless scanning

In-platform processing greatly reduces risk and improves your data security posture. Provisioning agents, whether they’re in a container, virtual machine, or Amazon Machine Image (AMI), create complexity and an unnecessary security risk. Not only can those agents become compromised, but their misconfiguration might lead to data leaks to other parts of your cloud infrastructure. An agentless approach can better leverage data platform optimizations to process data instead of transferring it out to re-optimize and analyze. This simplifies operations and increases efficiency for your infrastructure teams.

Cross-platform consistency

The advantages of in-platform processing are abundant, but implementing it across a multitude of platforms is challenging. Immuta helps bypass the obstacles by doing all the heavy lifting for you and building in specific implementations for each technology. Although all those implementations are ultimately different, Immuta abstracts the results to one standardized taxonomy, so you can have consistently accurate and granular metadata across all your data stores.

Granular query-level classification

Immuta classifies data on a column level and instantaneously identifies schema changes. Only with that level of granularity and automation can you adhere to your audit requirements and understand what actions have been taken on your data. For example, if non-sensitive data is joined with sensitive data at query time, Immuta will monitor and record that for your review. Continuous object sync ensures schema changes never result in holes in your access controls and data security posture.

Highly accurate and actionable metadata

Trust in your metadata is critical for data security.

To unblock your data consumers, you need to automate your data access controls; this requires trusting that your classification and metadata are accurate and actionable. Immuta's identification provides you with highly accurate metadata and tags out-of-the-box and assists you in fine-tuning the classification mechanism to deal with false positives quickly. That enables you to build policies that dynamically grant or restrict access to protected data (like PHI or PII) depending on who is accessing it and what protections you want to apply.

Components of data identification and classification

Immuta works in three phases to identify, categorize, and classify your data:

1. Identification: In this first phase, data is identified by its kind – for example, a name or an age. This identification can be manually performed, externally provided by a catalog, or automatically determined through column-level analysis of patterns.

2. Categorization: In the second phase, data is categorized in the context of where it appears, subject to your active frameworks. For example, a record occurring in a clinical context containing both a name and individual health data is protected health information (PHI) under HIPAA.

   This categorization of data helps to understand the context it is in, including information like whether or not a record pertains to an individual, the composition and kinds of identifiers present, the data subject, whether the data belongs to any controlled data categories under certain legislation, etc.

3. Classification: In the third and final phase, data is classified according to its sensitivity level (e.g., Customer Financial Data is Highly Sensitive) and the risk associated to the data subject. Audit dashboards support 3 sensitivity levels. However, organizations are free to customize the sensitivity names for the tags as needed.

Identification is an Immuta feature that uses data patterns to determine what type of data your column represents. Using identifiers within domains, Immuta evaluates your data and can assign the appropriate tags to your data dictionary based on what it finds. This saves the time of identifying your data manually and provides the benefit of a standard taxonomy across all your data sources in Immuta.

Architecture

To evaluate your data, Immuta generates a SQL query using a domain's identifiers. The Immuta system account then executes that query in the remote technology to match any regex and dictionary identifiers. Immuta receives the query result, containing the column name and the matching identifiers but no raw data values. Column name identifiers are all matched within Immuta and don't require any query to the remote technology. These results are then used to apply the resulting tags to the appropriate columns.

This evaluating and tagging process occurs when identification runs and happens automatically from the following event:

• A new data source is added to a domain with identifiers (either manually or automatically via tags)

The following actions will also trigger identification:

• Column detection is enabled, and new columns are detected on data sources within a domain with identifiers. Here, identification will only run on new columns, and no existing tags will be removed or changed.

• A user manually triggers it from the data source health check menu. Note, this will use the identifiers that already applied to the data source.

• A user manually triggers it from the domain page.

• A user manually triggers it through the API.

Identifiers

Identification runs identifiers to discover data. These identifiers are grouped into domains with data sources. Each identifier contains a single criteria and the tags that will be applied when the criteria's conditions have been met.

There are two types of identifiers in Immuta:

1. Reference identifiers: This is a library of the identifiers that can be added to domains. When added to a domain, a copy of the reference identifier is made as the domain-specific identifier.

   1. Immuta comes with built-in identifiers to discover common categories of data. These cannot be modified or deleted.

   2. Data governors can create their own reference identifiers for use within your organization.

2. Domain-specific identifiers: These identifiers only exist within a specific domain and are checked against the data sources in that domain when identification runs.

   1. Users with the Manage Identifiers permission can create these identifiers or add them to a domain from a reference identifier.

   2. If a domain-specific identifier was copied over from a reference identifier, there is no lineage and any edits to the reference identifier will not be reflected in the domain-specific copy.

Criteria

Criteria are the conditions in an identifier that need to be met for resulting tags to be applied to data.

• Competitive criteria analysis: This criteria is a process that will review all the regex and dictionary criteria within the identifiers of the domain and search for the identifier with the best fit. In this review, each competitive criteria analysis identifier in the domain competes against each other to find the best and most specific identifier that fits the data. The resulting tags for the best identifier are then applied to the column. Only one competitive criteria analysis identifier for each domain will apply per column. Competitive criteria identifiers, both built-in and custom, must match at least 90% of the data sampled. To learn more about the competitive nature, see the How competitive criteria analysis works guide.

  • Regex: This criteria contains a case-insensitive regular expression (regex) that searches for matches against column values. Immuta only supports regular expressions written in RE2 syntax.

  • Dictionary: This criteria contains a list of words and phrases to match against column values.

• Column name: This criteria includes a case-insensitive regular expression (regex) matched against column names, not against the values in the column. The identifier's tags will be applied to the column where the name is found. Multiple column name identifiers can match a column and be applied. Immuta only supports regular expressions written in RE2 syntax.

Create a new identifier in the Immuta UI or with the sdd/identifier endpoint.

Supported technologies

Identification has varied support for data sources from different technologies based on the identifier type.

What has changed with SDD when using this new feature?

If you used SDD prior to this feature release in January 2025, there are some differences:

1. There are now two types of identifiers:

   1. Reference identifiers

   2. Domain identifiers

   See information about these in the Identifiers section.

2. There is a new permission to manage identifiers within domains: Manage Identifiers. The permission allows you to do the following:

   1. Create an identifier within your domain

   2. View the reference identifiers in Immuta

   3. Add, edit, and delete identifiers within your domain

3. Previously, tags applied by SDD had to be from the parent Discovered tag. However, with identifiers in domains, any tag can be used in an identifier.

4. The following have been removed:

   1. Identification frameworks: Previously, all identifiers had to be contained within a framework and that framework had to be assigned to a data source to run. Now, identifiers are added to domains with data sources.

   2. Global framework: Previously, a global framework could be set to run SDD automatically on all new data sources. This behavior can be similarly obtained if you are using connections by creating a domain with dynamic assignment based on the Immuta Connections tag.

Tag mutability

When identification is manually triggered by a data owner, all column tags previously applied by identification are removed and the tags prescribed by the latest run are applied. However, if identification is triggered because a new column is detected by schema monitoring or object sync, tags will only be applied to the new column, and no tags will be modified on existing columns. Additionally, governors, data source owners, and data source experts can disable any unwanted tags in the data dictionary to prevent them from being used and auto-tagged on that data source in the future.

Performance

The amount of time it takes to run identification on a data source depends on several factors:

• Columns: The time to run identification grows nearly linearly with the number of text columns in the data source.

• Identifiers: The number of identifiers being used weakly impacts the time to run identification.

• Row count: Performance of identification may vary depending on the sampling method used by each technology. For Snowflake, the number of rows has little impact on the time because data sampling has near-constant performance.

• Views: Performance on views is limited by the performance of the query that defines the view. Running identification on complex views with large amounts of data is more likely to result in timeouts. Immuta recommends running identification on the underlying base tables.

The time it takes to run identification for all newly onboarded data sources in Immuta is not limited by identification performance but by the execution of background jobs in Immuta. Consult your Immuta account manager when onboarding a large number of data sources to ensure the advanced settings are set appropriately for your organization.

Testing

For users interested in testing identification, note that the built-in identifiers by Immuta require a 90% match to data to be assigned to a column. This means that with synthetic data, there may be situations where the data is not real enough to fit the confidence needed to match identifiers. To test identification, use a dev environment and create copies of your tables.

Audit

The following identification-related events are audited and can be found on the audit page in the UI:

• SDDClassifierCreated: An identifier is created.

• SDDClassifierDeleted: An identifier is deleted.

• SDDClassifierUpdated: An identifier's criteria, description, name, or tag is updated.

• ​TagApplied: A tag is applied to a data source or column. Tag events from identification will have actor.name.Immuta System Account and will include the related identifiers in the event as relatedResources.type.CLASSIFIERS.

• ​TagRemoved: A tag is removed from a data source or column. Tag events from identification will have actor.name.Immuta System Account and will include the related identifiers in the event as relatedResources.type.CLASSIFIERS.

Considerations

Deleting the built-in Discovered tags is not recommended: If you do delete built-in Discovered tags and use the built-in identifiers without editing the tags, then when the identifier is matched the column will not be tagged. As an alternative, tags can be disabled on a column-by-column basis from the data dictionary, or identification won't run if you do not add identifiers to domains.

Supported data types and casing

*Two built-in patterns support and match based on additional data types:

• DATE: Columns will match this identifier if they are string and the regex matches or if the data type is date, date+time, or timestamp.

• TIME: Columns will match this identifier if they are string and the regex matches or if the data type is time. Note that if the date is included in the data, it will not match this identifier.

Limitations with query size

The size of the identification query for dictionary patterns, which are compiled into a regex and regex patterns, is limited by the backing technology:

• For Snowflake, the overall query text size limit is 1 MB.

• For Starburst (Trino), the default query character limit is 1,000,000 characters. However, this limit can be increased if your identifiers require it.

Databricks limitations

• Immuta will start up a Databricks cluster to complete the identification job if one is not already running. This can cause unnecessary costs if the cluster becomes idle. Follow Databricks best practices to automatically terminate inactive clusters after a set period of time.

• The following Databricks Unity Catalog securable objects are supported with Immuta, but cannot be used with identification:

  • Volumes (external and managed)

  • Models

  • Functions

• Using a large number of files to store the data in a table with a large number of rows may result in the Databricks planner scanning the entire table, resulting in a slow performing query.

Redshift limitations

• The Redshift cluster must be up and running for identification to successfully run

AWS access key limitations

To use AWS access key authentication on a Redshift data source and have competitive criteria analysis identifiers supported,

• The AWS access key used to register the data source must be able to do a minimum of the following redshift-data API actions:

  • redshift-data:BatchExecuteStatement

  • redshift-data:CancelStatement

  • redshift-data:DescribeStatement

  • redshift-data:ExecuteStatement

  • redshift-data:GetStatementResult

  • redshift-data:ListStatements

• The AWS access key used to register the data source must have redshift:GetClusterCredentials for the cluster, user, and database that they onboard their data sources with.

• If using a custom URL, then the data source registered with the AWS access key must have the region and clusterid included in the additional connection string options formatted like the following:

  Copy

    region=us-east-2;clusterid=12345

• Redshift Serverless data sources are not supported for competitive criteria analysis identifiers with the AWS access key authentication method.

Legacy SDD

Legacy SDD was available before October 2023. It is no longer available, but some users may still see the term "legacy SDD" in the context of their data tags applied to specific data sources. These tags will be removed the next time identification runs.

This how-to guide is for enabling identification for the first time. For additional information on identification, see the Data identification page.

Requirement: Immuta permission GOVERNANCE

Prerequisites

Identifiers can be added to and identification can run in any of your current domains. However, if you are not already using domains, set up a domain specifically to run identification:

1. Register your data sources.

2. Create a domain.

3. Grant the appropriate users the Manage Identifiers permission for the domain.

Add identifiers to a domain

1. Navigate to the Identifiers tab of your domain.

2. Click Get Started.

3. Add reference identifiers to your domain that are relevant to your data by clicking the checkboxes. The identifier becomes a point-in-time copy of the reference identifier. It has the same name, criteria, and tags. Note you cannot add multiple identifiers with the same name to the same domain, so if you want to add an improved reference identifier, edit the name.

4. Click Add Identifiers.

Create a new identifier

This action can be done within a domain from the Identifiers tab to create a domain-specific identifier, or it can be done from the Identifiers page to create a reference identifier.

1. Click Create New.

2. Enter a name and description for your identifier.

3. Click Next.

4. Enter criteria: Select the Type of criteria.

   1. For regex, enter a regex to be matched against column values. The default criteria encoding is case-sensitive. You can change this encoding using the regex criteria. The regex must use RE2 syntax.

   2. For column name regex, enter a regex to be matched against column names. The default criteria encoding is case-insensitive. You can change this encoding using the regex criteria. The regex must use RE2 syntax.

   3. For a dictionary, enter the values in a comma-separated list to match against column values. Opt to toggle the Case insensitive switch to on if you want the dictionary to be case sensitive.

5. Click Next.

6. Select the tags to apply: Use the text box to search for a tag or type a tag name to create a new tag under the "Discovered . Entity" hierarchy to apply to columns that match your identifier.

7. Click Next to review your new identifier and click Create Identifier to create it.

Note that all user-created identifiers must be a 90% match or greater for the contents of the column to be tagged.

Run identification in your domain

Once you have created identifiers relevant to your data, it is time to run them on your data. You may choose to run identification on a select number of data sources where you understand the data to assess and adjust the tags to reflect what you expect to see.

1. Navigate to the Domains page and select your domain.

2. Open the More Actions icon.

3. Select Run Identification from the dropdown.

View the identification results

After identification runs, you will receive a notification that the job is complete. Then, you can view the results from the data source dictionary.

1. Navigate to the data source overview page of the data source you have in the domain.

2. Click the Data Dictionary tab.

3. Assess whether the tags are applied as expected.

4. If you are happy with the tags, follow the Assign data sources to domains guide to add the rest of your data sources to the domain and then run identification on the domain again.

5. If you want additional tags, follow the Create an identifier guide again to create additional identifiers that matter to your data.

Requirement: Immuta permission GOVERNANCE or domain-specific Manage Identifiers

Create an identifier

This action can be done within a domain from the Identifiers tab to create a domain-specific identifier, or it can be done from the Identifiers page to create a reference identifier.

1. Click Create New.

2. Enter a name and description for the new identifier.

3. Click Next.

4. Enter criteria: Select the Type of criteria.

   1. For regex, enter a regex to be matched against column values. The default criteria encoding is case-sensitive. You can change this encoding using the regex criteria. The regex must use RE2.

   2. For a dictionary, enter the values in a comma-separated list to match against column values. Opt to toggle the Case insensitive switch to on if you want the dictionary to be case sensitive.

   3. For column name regex, enter a regex to be matched against column names. The default criteria encoding is case-insensitive. You can change this encoding using the regex criteria. The regex must use RE2 syntax.

5. Click Next.

6. Select the tags to apply: Use the text box to search for a tag or type a tag name to create a new tag under the "Discovered . Entity" hierarchy to apply to columns that match your identifier.

7. Click Next to review your new identifier and click Create Identifier to create it.

Note that all user-created identifiers must be a 90% match or greater for the contents of the column to be tagged.

Edit an identifier

Editing the details of an identifier from the identifiers page will only affect that identifier; no copies of that identifier will be impacted.

Delete an identifier

Deleting a domain identifier from the identifiers page will remove it from the domain it is in.

Requirement: Immuta permission GOVERNANCE or domain-specific Manage Identifiers

Identification can be configured to run automatically or manually on a domain-by-domain basis. If you want to re-run identification when a data source or new identifiers have been added to a domain, you can or from the UI.

Configure a domain's autoscanning setting

1. Navigate to the Domains page and select your domain.

2. Click the Settings tab.

3. Select the autoscanning on the toggle:

   • On: Identification will automatically run when new data sources are added to the domain or when object sync detects new columns on data sources already in the domain.

   • Off: Identification will only run when manually started by a user.

Run identification in a domain

1. Navigate to the Domains page and select your domain.

2. Select the more actions icon.

3. Select Run Identification and then select it again in the modal.

Run identification on a data source

1. Navigate to the data source overview page.

2. Click the health status.

3. Select Re-run next to Sensitive Data Discovery (SDD).

Verify tags

Disable tags from the data dictionary

If a governor, data owner, or data source expert disables a tag from the data dictionary applied by identification, the column will not be re-tagged next time identification runs. When a tag is disabled, it will not completely disappear, and it can be manually enabled through the tag side sheet.

To disable a tag,

1. Navigate to a data source and click the Data Dictionary tab.

2. Scroll to the column you want to remove the tag from and click the tag you want to remove.

3. Click Disable in the side sheet and then click Confirm.

May 21, 2025

Identifiers in domains is released as GA and these identifier updates are coupled with that release.

Improvements

The following identifiers have been improved to better match their intended data patterns. These updates have only been made to the built-in reference identifiers. If these are already in your domains, they will remain there as domain-specific identifiers with the previous pattern. If you want to add these improved identifiers to your domain, edit the name because identifier names must be unique within each domain.

To see more about the specific changes made, see the annotations on the .

• AUSTRALIA_MEDICARE_NUMBER

• AUSTRALIA_PASSPORT

• BRAZIL_CPF_NUMBER

• CANADA_PASSPORT

• CREDIT_CARD_NUMBER

• DATE

• DOMAIN_NAME

• FDA_CODE

• FRANCE_NIR

• GENDER

• ICD10_CODE

• IMEI_HARDWARE_ID

• MAC_ADDRESS

• PERSON_NAME

• POSTAL_CODE

• SPAIN_NIF_NUMBER

• TIME

• UK_NATIONAL_INSURANCE_NUMBER

• URL

• US_HEALTHCARE_NPI

• US_SOCIAL_SECURITY_NUMBER

• US_STATE

Deprecations

The following identifiers are deprecated and no longer included in the reference identifiers. If these are already in your domains, they will remain there as domain-specific identifiers.

• AGE

• DENMARK_CPR_NUMBER

• FINLAND_NATIONAL_ID_NUMBER

• FRANCE_CNI

• GERMANY_IDENTITY_CARD_NUMBER

• SPAIN_NIE_NUMBER

• SWEDEN_NATIONAL_ID_NUMBER

• SWEDEN_PASSPORT

• THAILAND_NATIONAL_ID_NUMBER

• UK_TAXPAYER_REFERENCE

• US_BANK_ROUTING_MICR

• US_PASSPORT

• US_TOLLFREE_PHONE_NUMBER

New

The following identifiers are newly created to identify common data patterns. Copy these new reference identifiers to any of your domains.

• BELGIUM_NATIONAL_REGISTRATION_NUMBER: Detects numeric strings consistent with Belgium's National Registration Number. Requires 11 characters in the form YY.MM.DD-NNN-XX, where YY.MM.DD corresponds to birth date, NNN is a number, and XX is a checksum digit.

• COUNTRY: Detects strings consistent with the names of all countries in the world. This identifier is case-insensitive.

• FINANCIAL_INSTITUTIONS: Matches strings consistent with names of financial institutions based on lists provided by the FDIC and OCC, includes alternative names.

• GREAT_BRITAIN_DRIVERS_LICENSE: Previously named UK_DRIVERS_LICENSE_NUMBER. Now, renamed because it does not detect license numbers from Northern Ireland.

• ICD_10_PCS: Detects strings consistent with procedure codes from the International Statistical Classification of Diseases and Related Health Problems (ICD), as drawn from the Clinical Modification lexicon from 2020.

• NAICS_CODE: Detects strings consistent with North American Industry Classification System (NAICS). A two-digit number represents a basic sector and each preceding digit represents a more specific sub sector with a maximum of six digits.

• SEC_STOCK_TICKER: Matches strings consistent with the stock tickers recognized by the U.S. Securities and Exchange Commission (SEC).

• US_PERSON_FULL_NAME: Detects strings consistent with a person's {first name} space {last name}. Uses the same names from the PERSON_NAME identifier. This identifier must match at least 20% of the data sampled and is case-insensitive.

• US_STREET_ADDRESS: Previously named STREET_ADDRESS.

First identifier pack released

62 built-in identifiers are released for use with identification.

Of identification's three criteria options, regex and dictionary are competitive. This means that when assessing your data, if multiple identifiers could match, only one with competitive criteria will be chosen to tag the data. To better understand how Immuta executes this competition, read further.

Immuta employs a three-phased competitive criteria analysis approach for identification:

1. Sampling: No data is moved, and Immuta checks the identifiers against a sample of data from the table.

2. Qualifying: Identifiers with a criteria match of less than a 90% match are filtered out.

3. Scoring: The remaining identifiers are compared with one another to find the most specific criteria that qualifies and matches the sample.

In the end, competitive criteria analysis aims to find a single identifier for each column that best describes the data format.

Sampling

In the sampling process, no database contents are transmitted to Immuta; instead, Immuta receives only the column-wise hit rate (the number of times the criteria has matched a value in the column) information for each active identifier. To do this, Immuta instructs a remote database to measure column-wise hit rate information for all active identifiers over a row sample.

The sample size is decided based on the number of identifiers and the data size, when available. In the most simplified case, the requested number of sampled rows depends only on the number of regex and dictionary criteria being run in the domain, not the data size. The sample size dependence on the number of identifiers is weak and will not exceed 13,000 rows.

Sampling considerations

In practice, the number of sampled values for each column may be less than the requested number of rows because columns are not independently sampled but rather projected from a row-wise sample. This can impact the sample when the target table has less than the requested number of rows, when some of the column values are null, or because of technology-specific limitations.

• Snowflake and Starburst (Trino): Immuta implements table sampling by row count.

• Databricks and Redshift: Due to technology limitations and the inability to predict the size of the table, Immuta implements a best-effort sampling strategy comprising a flat 10% row sample capped at the first 10,000 sampled rows. In particular, under-sampling may occur on tables with less than 100,000 rows. Moreover, the resulting sample is biased towards earlier records.

• All platforms: Sampling from views can have significantly slower performance that varies by the performance of the query that defines the view.

• All platforms: Any null values included in the sample will not count towards the qualification or scoring when included in the sample. However, it will lower the number of available values to match against the patterns, as the sample size is not dynamic based on the ignored null values.

Qualifying

During the qualification phase, identifiers that do not agree with the data are disqualified. An identifier agrees with the data if the hit rate on the remote sample exceeds the predefined threshold. This threshold is 90% match for most built-in identifiers; however, a few built-in identifiers have lower threshold requirements. The 90% threshold is standard for all custom identifiers as well to ensure the criteria matches the data within the column and to avoid false positives. Note that threshold calculations are relative to the number of non-null entries for each column.

If no identifiers qualify, then no identifier is assessed for scoring and the column is not tagged.

Scoring

During the scoring phase, a machine inference is carried out among all qualified identifiers, combining criteria-derived complexity information with hit rate information to determine which identifier best describes the sample data. This process prefers the more restrictive of two competing identifiers since the ability to satisfy the more difficult-to-satisfy identifier itself serves as evidence that it is more likely. This phase ends by returning a single most likely identifier per the inference process.

Example

Here are a set of regex identifiers and a sample of data:

Identifiers:

1. [a-zA-Z0-9]{3} - This regex will match 3 character strings with the characters a-z, lowercase or uppercase, or digits 0-9.

2. [a-c]{3} - This regex will match 3 character strings with the characters a-c, lowercase.

3. (a|b|d){3} - This regex will match 3 character strings with the characters a, b, or d, lowercase.

When qualifying the identifiers, Identifier 1 and Identifier 3 both match 90% or more of the data. Identifier 2 does not, and is disqualified.

Then the qualified identifiers are scored. Here, Identifier 1, despite matching 100% of the data, is unspecific and could match over 200,000 values. On the other hand, Identifier 3 matches just at 90% but is very specific with only 27 available values.

Therefore, with the specificity taken into account, Identifier 3 would be the match for this column, and its tags would be applied to the data source in Immuta.

Important notes

• Dictionaries are part of the competitive process, while column-name regex are not.

• Scoring ties are rare but can occur if the same criteria (either dictionary or regex) is specified more than once (even in different forms). Scoring ties are inconclusive, and the scoring phase will not return an identifier in the case of a tie.

• Criteria complexity analysis is sensitive to the total number of strings an identifier accepts or, equivalently for dictionaries, the number of entries. Therefore, identifiers that accept much more than is necessary to describe the intended column data format may perform more poorly in the competitive analysis because they are easier to satisfy.

Immuta is pre-configured with a set of tags that can be used to write global policies before data sources even exist. See a list of the built-in Discovered tags below and the Built-in identifier reference page for information about where these tags will be applied by the built-in identifiers.

Country tags

All the tags below belong to the Country parent. For example, the full tag name will appear as Discovered . Country . Argentina.

Entity tags

All the tags below belong to the Entity parent. For example, the full tag name will appear as Discovered . Entity . Aadhaar Individual.

Immuta comes with a pack of built-in identifiers that look for common data types. These identifiers were written by Immuta's research and development team and cannot be deleted or edited by users. However, users can add these built-in identifiers to their own domains and edit the tags applied by them.

Identifiers must match at least 90% of the sampled data to be tagged, with three exceptions noted below. See the How competitive pattern analysis works guide for more information about sampling and thresholds.

Identifier descriptions and default resulting tags