> For the complete documentation index, see [llms.txt](https://documentation.immuta.com/SaaS/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.immuta.com/SaaS/configuration/tags/data-discovery/data-identification-reference-guide.md).
# Data Identification Reference Guide
Learn about how data identification discovers and tags your data
Identification uses data patterns to determine what type of data your column represents. Using identifiers within domains, Immuta evaluates your data and can assign the appropriate tags to your data source columns based on what it finds. This saves the time of identifying your data manually and provides the benefit of a standard taxonomy across all your data sources in Immuta.
## Identifiers
Identification runs identifiers to discover data. These identifiers are grouped into [domains](/SaaS/configuration/domains/domains.md) with data sources. Each identifier contains a single criteria and the tags that will be applied when the criteria's conditions have been met.
There are two types of identifiers in Immuta:
1. **Reference identifiers**: This is a library of the identifiers that can be added to domains. When added to a domain, a copy of the reference identifier is made as the domain-specific identifier.
1. Immuta comes with [built-in identifiers](/SaaS/configuration/tags/data-discovery/data-identification-reference-guide/improved-pack-built-in-identifier-reference.md) to discover common categories of data. *These cannot be modified or deleted*.
2. Data governors can create their own reference identifiers for use within your organization.
2. **Domain-specific identifiers**: These identifiers only exist within a specific domain and are checked against the data sources in that domain when identification runs.
1. Users with the `Manage Identifiers` permission can create these identifiers or add them to a domain from a reference identifier.
2. If a domain-specific identifier was copied over from a reference identifier, there is no lineage and any edits to the reference identifier will not be reflected in the domain-specific copy.
### **Criteria**
Criteria are the conditions in an identifier that need to be met for resulting tags to be applied to data.
* **Competitive criteria analysis**: This criteria is a process that will review all the regex and dictionary criteria within the identifiers of the domain and search for the identifier with the best fit. In this review, each competitive criteria analysis identifier in the domain competes against each other to find the best and most specific identifier that fits the data. The resulting tags for the best identifier are then applied to the column. Only one competitive criteria analysis identifier for each domain will apply per column. Competitive criteria identifiers, both built-in and custom, must match at least 90% of the data sampled. To learn more about the competitive nature, see the [competitive criteria analysis section](#competitive-criteria-analysis).
* **Regex**: This criteria contains a case-insensitive regular expression (regex) that searches for matches against column values. Immuta only supports regular expressions written in RE2 syntax.
* **Dictionary**: This criteria contains a list of words and phrases to match against column values.
* **Column name**: This criteria includes a case-insensitive regular expression (regex) matched against column names, not against the values in the column. The identifier's tags will be applied to the column where the name is found. Multiple column name identifiers can match a column and be applied. Immuta only supports regular expressions written in RE2 syntax.
Create a new identifier in the [Immuta UI](/SaaS/configuration/tags/data-discovery/enable-sdd/manage-rules.md#create-an-identifier) or with the [`sdd/identifier` endpoint](/SaaS/developer-guides/api-intro/immuta-v1-api/configure-your-instance-of-immuta/manage-identifiers-in-domains.md#post-sdd-identifier).
## What does Immuta do in my environment?
The way Immuta runs identification depends on your [criteria](#criteria):
* **Dictionary and regex identifiers**: To evaluate your data for matches to dictionary and regex identifiers, Immuta generates a SQL query using a domain's identifiers. The Immuta system account then executes that query in the remote technology (e.g., in Snowflake) to match any regex and dictionary identifiers. Immuta receives the query result which contains the column name and the matching identifiers but no raw data values.
* **Column name regex identifiers**: To evaluate your data for matches to column name regex identifiers, Immuta does not need to query your remote technology. Instead, column name identifiers are matched with the column metadata within Immuta (i.e., the column names of your tables).
The results of these processes are then used to apply the resulting tags to the appropriate columns.
### When does identification run?
This evaluating and tagging process occurs when identification runs and happens automatically from the following event:
* A new data source is added to a domain with identifiers (either manually or [automatically via tags](/SaaS/configuration/domains/domains.md#domain-data-sources))
The following actions will also trigger identification:
* Column detection is enabled, and new columns are detected on data sources within a domain with identifiers. Here, identification will only run on new columns, and no existing tags will be removed or changed.
* [A user manually triggers it from the data source health check menu](/SaaS/configuration/tags/data-discovery/enable-sdd/manage-sdd-tags.md#run-identification-on-a-data-source). *Note, this will use the identifiers that already applied to the data source.*
* [A user manually triggers it from the domain page](/SaaS/configuration/tags/data-discovery/enable-sdd/manage-sdd-tags.md#run-identification-in-a-domain).
* [A user manually triggers it through the API](/SaaS/developer-guides/api-intro/immuta-v1-api/configure-your-instance-of-immuta/manage-identifiers-in-domains.md#post-sdd-identification).
### Competitive criteria analysis
Of identification's [three criteria options](/SaaS/configuration/tags/data-discovery.md#identifiers), regex and dictionary are competitive. This means that when assessing your data, if multiple identifiers could match, only one with competitive criteria will be chosen to tag the data. To better understand how Immuta executes this competition, read further.
Immuta employs a three-phased competitive criteria analysis approach for identification:
1. [Sampling](#sampling): No data is moved, and Immuta checks the identifiers against a sample of data from the table.
2. [Qualifying](#qualification): Identifiers with a criteria match of less than a 90% match are filtered out.
3. [Scoring](#scoring): The remaining identifiers are compared with one another to find the most specific criteria that qualifies and matches the sample.
In the end, competitive criteria analysis aims to find a single identifier for each column that best describes the data format.
#### Sampling
In the sampling process, no database contents are transmitted to Immuta; instead, Immuta receives only the column-wise hit rate (the number of times the criteria has matched a value in the column) information for each active identifier. To do this, Immuta instructs a remote database to measure column-wise hit rate information for all active identifiers over a row sample.
The sample size is decided based on the number of identifiers and the data size, when available. In the most simplified case, the requested number of sampled rows depends only on the number of regex and dictionary criteria being run in the domain, not the data size. The sample size dependence on the number of identifiers is weak and will not exceed 13,000 rows.
| Number of identifiers | Sample size |
| --------------------- | ----------- |
| 5 | 7369 rows |
| 50 | 9211 rows |
| 500 | 11053 rows |
| 5000 | 12895 rows |
**Sampling considerations**
In practice, the number of sampled values for each column may be less than the requested number of rows because columns are not independently sampled but rather projected from a row-wise sample. This can impact the sample when the target table has less than the requested number of rows, when some of the column values are `null`, or because of technology-specific limitations.
* Snowflake and Starburst (Trino): Immuta implements table sampling by row count.
* Databricks and Redshift: Due to technology limitations and the inability to predict the size of the table, Immuta implements a best-effort sampling strategy comprising a flat 10% row sample capped at the first 10,000 sampled rows. In particular, under-sampling may occur on tables with less than 100,000 rows. Moreover, the resulting sample is biased towards earlier records.
* All platforms: Sampling from views can have significantly slower performance that varies by the performance of the query that defines the view.
* All platforms: Any `null` values included in the sample will not count towards the qualification or scoring when included in the sample. However, it will lower the number of available values to match against the patterns, as the sample size is not dynamic based on the ignored `null` values.
#### Qualifying
During the qualification phase, identifiers that do not agree with the data are disqualified. An identifier agrees with the data if the [hit rate](#user-content-fn-1)[^1] on the remote sample exceeds the predefined threshold. This threshold is 90% match for most built-in identifiers; however, a few built-in identifiers have lower threshold requirements[^2]. The 90% threshold is standard for all custom identifiers as well to ensure the criteria matches the data within the column and to avoid false positives. Note that threshold calculations are relative to the number of non-null entries for each column.
If no identifiers qualify, then no identifier is assessed for scoring and the column is not tagged.
#### Scoring
During the scoring phase, a machine inference is carried out among all qualified identifiers, combining criteria-derived complexity information with hit rate information to determine which identifier best describes the sample data. This process prefers the more restrictive of two competing identifiers since the ability to satisfy the more difficult-to-satisfy identifier itself serves as evidence that it is more likely. This phase ends by returning a single most likely identifier per the inference process.
#### Example
Here are a set of regex identifiers and a sample of data:
**Identifiers**:
1. `[a-zA-Z0-9]{3}` - This regex will match 3 character strings with the characters a-z, lowercase or uppercase, or digits 0-9.
2. `[a-c]{3}` - This regex will match 3 character strings with the characters a-c, lowercase.
3. `(a|b|d){3}` - This regex will match 3 character strings with the characters a, b, or d, lowercase.
| Sample data | Matches Identifier 1 | Matches Identifier 2 | Matches Identifier 3 |
| ----------- | -------------------- | -------------------- | -------------------- |
| dad | Yes | :x: | Yes |
| baa | Yes | :x: | Yes |
| add | Yes | :x: | Yes |
| add | Yes | :x: | Yes |
| cab | Yes | Yes | :x: |
| bad | Yes | :x: | Yes |
| aba | Yes | :x: | Yes |
| baa | Yes | :x: | Yes |
| dad | Yes | :x: | Yes |
| baa | Yes | :x: | Yes |
When **qualifying** the identifiers, Identifier 1 and Identifier 3 both match 90% or more of the data. Identifier 2 does not, and is disqualified.
Then the qualified identifiers are **scored**. Here, Identifier 1, despite matching 100% of the data, is unspecific and could match over 200,000 values. On the other hand, Identifier 3 matches just at 90% but is very specific with only 27 available values.
Therefore, with the specificity taken into account, Identifier 3 would be the match for this column, and its tags would be applied to the data source in Immuta.
#### Important notes
* Dictionaries are part of the competitive process, while column-name regex are not.
* Scoring ties are rare but can occur if the same criteria (either dictionary or regex) is specified more than once (even in different forms). Scoring ties are inconclusive, and the scoring phase will not return an identifier in the case of a tie.
* Criteria complexity analysis is sensitive to the total number of strings an identifier accepts or, equivalently for dictionaries, the number of entries. Therefore, identifiers that accept much more than is necessary to describe the intended column data format may perform more poorly in the competitive analysis because they are easier to satisfy.
## Supported technologies
Identification has varied support for [data sources](/SaaS/configuration/integrations/data-and-integrations/registering-metadata/data-source-overview.md) from different technologies based on the identifier type.
| Technology | Regex | Dictionary | Column name regex |
| -------------------------- | -------------------- | -------------------- | -------------------- |
| Amazon Redshift | :white\_check\_mark: | :white\_check\_mark: | :white\_check\_mark: |
| Amazon Redshift Spectrum | :white\_check\_mark: | :white\_check\_mark: | :white\_check\_mark: |
| Amazon S3 | :x: | :x: | :white\_check\_mark: |
| AWS Lake Formation | :x: | :x: | :white\_check\_mark: |
| Azure Synapse Analytics | :x: | :x: | :white\_check\_mark: |
| Databricks Lakebase | :x: | :x: | :white\_check\_mark: |
| Databricks Spark | :white\_check\_mark: | :white\_check\_mark: | :white\_check\_mark: |
| Databricks Unity Catalog | :white\_check\_mark: | :white\_check\_mark: | :white\_check\_mark: |
| Google BigQuery view-based | :x: | :x: | :white\_check\_mark: |
| Google BigQuery viewless | :x: | :x: | :x: |
| MariaDB | :x: | :x: | :white\_check\_mark: |
| MySQL | :x: | :x: | :white\_check\_mark: |
| Oracle | :x: | :x: | :white\_check\_mark: |
| PostgreSQL | :x: | :x: | :white\_check\_mark: |
| Snowflake | :white\_check\_mark: | :white\_check\_mark: | :white\_check\_mark: |
| SQL Server | :x: | :x: | :white\_check\_mark: |
| Starburst (Trino) | :white\_check\_mark: | :white\_check\_mark: | :white\_check\_mark: |
| Teradata | :x: | :x: | :white\_check\_mark: |
## Tag mutability
When identification is manually triggered by a data owner, all column tags previously applied by identification are removed and the tags prescribed by the latest run are applied. However, if identification is triggered because a new column is detected by schema monitoring or object sync, tags will only be applied to the new column, and no tags will be modified on existing columns. Additionally, governors, data source owners, and data source experts can [disable any unwanted tags in the data source](/SaaS/configuration/tags/data-discovery/enable-sdd/manage-sdd-tags.md#disable-tags-from-the-data-dictionary) to prevent them from being used and auto-tagged on that data source in the future.
## Performance
The amount of time it takes to run identification on a data source depends on several factors:
* **Columns**: The time to run identification grows nearly linearly with the number of text columns in the data source.
* **Identifiers**: The number of identifiers being used [weakly impacts](#competitive-criteria-analysis) the time to run identification.
* **Row count**: Performance of identification may vary depending on the sampling method used by each technology. For Snowflake, the number of rows has little impact on the time because data sampling has near-constant performance.
* **Views**: Performance on views is limited by the performance of the query that defines the view. Running identification on complex views with large amounts of data is more likely to result in timeouts. Immuta recommends running identification on the underlying base tables.
The time it takes to run identification for all newly onboarded data sources in Immuta is not limited by identification's performance but by the execution of background jobs in Immuta. [Consult your Immuta account manager](https://support.immuta.com/) when onboarding a large number of data sources to ensure the advanced settings are set appropriately for your organization.
{% hint style="warning" %}
**Default 15-minute timeout**
Identification queries will timeout after 15 minutes to avoid overconsumption of resources and reduce the cost of running identification. If your identification run was not completed because of this timeout, [submit a support ticket](https://support.immuta.com) to change the default setting.
{% endhint %}
## Testing
For users interested in testing identification, note that the built-in identifiers by Immuta require a 90% match to data to be assigned to a column. This means that with synthetic data, there may be situations where the data is not real enough to fit the confidence needed to match identifiers. To test identification, use a dev environment and create copies of your tables.
## Audit
The following identification-related events are [audited](/SaaS/govern/detect-your-data/audit/reference-guides/index/uam-schema.md) and can be found on the [audit page in the UI](/SaaS/govern/detect-your-data/audit/how-to-guides/use-immuta-audit.md):
* [SDDClassifierCreated](/SaaS/govern/detect-your-data/audit/reference-guides/index/uam-schema.md#sddclassifiercreated-event): An identifier is created.
* [SDDClassifierDeleted](/SaaS/govern/detect-your-data/audit/reference-guides/index/uam-schema.md#sddclassifierdeleted-event): An identifier is deleted.
* [SDDClassifierUpdated](/SaaS/govern/detect-your-data/audit/reference-guides/index/uam-schema.md#sddclassifierupdated-event): An identifier's criteria, description, name, or tag is updated.
* [TagApplied](/SaaS/govern/detect-your-data/audit/reference-guides/index/uam-schema.md#tagapplied-event): A tag is applied to a data source or column. Tag events from identification will have `actor.name.Immuta System Account` and will include the related identifiers in the event as `relatedResources.type.CLASSIFIERS`.
* [TagRemoved](/SaaS/govern/detect-your-data/audit/reference-guides/index/uam-schema.md#tagremoved-event): A tag is removed from a data source or column. Tag events from identification will have `actor.name.Immuta System Account` and will include the related identifiers in the event as `relatedResources.type.CLASSIFIERS`.
## Considerations
* **Deleting the built-in Discovered tags is not recommended**: If you do delete built-in Discovered tags and use the built-in identifiers without editing the tags, then when the identifier is matched the column will not be tagged. As an alternative, tags can be disabled on a [column-by-column basis from the data source](/SaaS/configuration/tags/data-discovery/enable-sdd/manage-sdd-tags.md#disable-tags-from-the-data-dictionary), or identification won't run if you do not add identifiers to domains.
* **Regex patterns with nested wildcards are not recommended**: When creating regexes for identifiers, it is best to avoid nested wildcards. They can be too complex and cause internal timeouts. As an alternative, break up the regexes into simpler patterns. Then after they are tagged, use a [classification framework](/SaaS/configuration/tags/data-classification/how-to-use-a-built-in-classification-framework-with-your-own-tags.md) to group them under a parent tag.
### Supported data types and casing
| Type of identifier | Supported data types | Case sensitivity |
| ------------------ | -------------------- | ------------------------------------------- |
| Data regex\* | Text string columns | Case-sensitive |
| Column name regex | Any column | Not case-sensitive |
| Dictionary | Text string columns | Can be toggled in the identifier definition |
\*Two built-in patterns support and match based on additional data types:
* `DATE`: Columns will match this identifier if they are string and the regex matches **or** if the data type is date, date+time, or timestamp.
* `TIME`: Columns will match this identifier if they are string and the regex matches **or** if the data type is time. Note that if the date is included in the data, it will not match this identifier.
### Limitations with query size
The size of the identification query for dictionary patterns, which are compiled into a regex and regex patterns, is limited by the backing technology:
* For Snowflake, the [overall query text size limit is 1 MB](https://docs.snowflake.com/en/user-guide/query-size-limits).
* For Starburst (Trino), the default query character limit is 1,000,000 characters. However, [this limit can be increased if your identifiers require it](https://trino.io/docs/current/admin/properties-query-management.html#query-max-length).
### Databricks limitations
* Immuta will start up a Databricks cluster to complete the identification job if one is not already running. This can cause unnecessary costs if the cluster becomes idle. Follow [Databricks best practices](https://docs.databricks.com/clusters/cluster-config-best-practices.html#automatic-termination) to automatically terminate inactive clusters after a set period of time.
* The following Databricks Unity Catalog securable objects are supported with Immuta, but cannot be used with identification:
* Volumes (external and managed)
* Models
* Functions
* Using a large number of files to store the data in a table with a large number of rows may result in the Databricks planner scanning the entire table, resulting in a slow performing query.
### Redshift Spectrum limitations
* The Redshift cluster must be up and running for identification to successfully run.
#### AWS access key limitations
To use AWS access key authentication on a Redshift data source and have competitive criteria analysis identifiers supported,
* The AWS access key used to register the data source must be able to do a minimum of the following [redshift-data API actions](https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonredshiftdataapi.html):
* `redshift-data:BatchExecuteStatement`
* `redshift-data:CancelStatement`
* `redshift-data:DescribeStatement`
* `redshift-data:ExecuteStatement`
* `redshift-data:GetStatementResult`
* `redshift-data:ListStatements`
* The AWS access key used to register the data source must have `redshift:GetClusterCredentials` for the cluster, user, and database that they onboard their data sources with.
* If using a custom URL, then the data source registered with the AWS access key must have the `region` and `clusterid` included in the [additional connection string options](/SaaS/configuration/integrations/data-and-integrations/registering-metadata/register-data-sources/redshift-tutorial.md) formatted like the following:
```
region=us-east-2;clusterid=12345
```
* Redshift Serverless data sources are not supported for competitive criteria analysis identifiers with the AWS access key authentication method.
[^1]: The number of values that match the identifier.
[^2]: US\_STREET\_ADDRESS has an 80% threshold, LOCATION has an 80% threshold, PERSON\_NAME has a 45% threshold, and US\_PERSON\_FULL\_NAME has a 20% threshold.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://documentation.immuta.com/SaaS/configuration/tags/data-discovery/data-identification-reference-guide.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.