Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Learn about connecting external catalogs
Connect an external catalog to use tagging capabilities outside of Immuta and pull tags from external table schemas. Once the catalog has been connected, Immuta ingests a data dictionary from the catalog and applies data source and column tags directly to the data source. These tags can then be used to create policies.
Configure an external catalog: Configure an external catalog to ingest tags into Immuta.
External catalog integrations: This reference guide describes the requirements of the external catalogs Immuta supports.
Custom REST catalog interface endpoints: This reference guide describes the endpoints for configuring a custom REST catalog.
Learn about how data classification categorizes the risk level associated with your data
Classification is the process in which data is categorized by the content and the associated risk level based on context. Classification complements identification, and the tags classification applies can give additional information in the audit dashboards for data sources.
Activate classification frameworks: Use the API to activate a classification framework.
: Create a classification framework using a provided template.
: This reference guide describes classification frameworks and how classification works in Immuta.
Learn about how to manage tags, external catalogs, data identification tags, or data classification tags
This section includes guidance for connecting and managing data metadata, which is used by Immuta to identify data targeted by policies or to generate Immuta reports.
Integrate your existing data catalog with Immuta.
Automatically identify and tag data based on its content or column names.
Automatically tag data based on its sensitivity and the associated risk level.
Create and manage tags in Immuta.
Learn about how tags are used and what actions associated with tags are audited in Immuta
Tags have several uses, mainly to drive policies, but they can be used for the following purposes:
Use tags for global subscription or data policies that will apply to all data sources in the organization. In doing this, company-wide data security restrictions can be controlled by the administrators and governors, while the users and data owners need only to worry about tagging the data correctly.
Generate Immuta reports from tags for anything from insider threat surveillance to data access monitoring.
Activate a classification framework to categorize the sensitivity of your data
Requirement: Immuta permission GOVERNANCE
To activate a classification framework,
Click Metadata in the navigation menu and select Classifications.
Select Activate.
Click Metadata in the navigation menu and select Classifications.
Click the more actions icon in the Actions column for the framework you want to activate.
Select Deactivate.
To activate a framework using the Immuta API, see the Frameworks API reference page.
Drive search results with tags in the Immuta UI.
Every user within Immuta can see tags, but they will all interact with them differently as their roles require. Governors create, manage, and delete tags or import tags from external catalogs. Data owners, data source experts, and governors apply these tags to or remove them from projects, data sources, and columns within the data sources. Data users view tags and tag metadata on data sources they have access to.
The following tag-related events are audited and can be found on the audit page in the UI:
TagApplied: A tag is applied to a data source or column.
TagCreated: A tag is created.
TagDeleted: A tag is deleted.
: A tag is removed from a data source or column.
: A tag name is updated.
Managing tags best practice: Use the minimum number of tags possible to achieve the data privacy needed.
Run identification in a domain or on a data source to automatically tag your data
Requirement: Immuta permission GOVERNANCE or domain-specific Manage Identifiers
Identification can be configured to run automatically or manually on a domain-by-domain basis. If you want to re-run identification when a data source or new identifiers have been added to a domain, you can manually run it for the domain using the API or from the UI.
Click Domains in the navigation menu and select your domain.
Click the Settings tab.
Select the autoscanning on the toggle:
On: Identification will automatically run when new data sources are added to the domain or when object sync detects new columns on data sources already in the domain.
Off: Identification will only run when manually started by a user.
Click Domains in the navigation menu and select your domain.
Select the more actions icon.
Select Run Identification and then select it again in the modal.
Navigate to the data source overview page.
Click the health status.
Select Re-run next to Sensitive Data Discovery (SDD).
If a governor, data owner, or data source expert disables a tag from the data source column applied by identification, the column will not be re-tagged next time identification runs. When a tag is disabled, it will not completely disappear, and it can be manually enabled through the tag side sheet.
To disable a tag,
Navigate to a data source and click the Columns tab.
Scroll to the column you want to remove the tag from and click the tag you want to remove.
Click Disable in the side sheet and then click Confirm.
Learn about the challenges of traditional data discovery tools and how the Immuta solution differs
Immuta allows you to automate discovering and tagging data across your data platform. Tagging is critical for two reasons:
It allows you to define data sensitivity, which in turn allows you to monitor where you have potential data security issues and gaps in your security posture.
It allows you to abstract your physical structure from your access policy logic. For example, you can build access policies like mask all columns tagged Person Name (where Person Name
Create, edit, or delete an identifier
Requirement: Immuta permission GOVERNANCE or domain-specific Manage Identifiers
This action can be done within a domain from the Identifiers tab to create a domain-specific identifier, or it can be done from the Identifiers page to create a reference identifier.
Click Create New.
Verify tags
When using identification, manually adding tags to columns in the data source will be unnecessary in most cases. The data owner will just need to verify that the applied tags are correct.
mask column x in database y in data platform zToday’s sensitive data discovery tools give you a shallow overview of your data corpus across a long list of platforms. They give you pointers on where you have sensitive data without the granularity to drive your column- or row-level access controls. They help you understand what data you possess according to a regulatory framework, like HIPAA or PCI but without the details needed to automate your audit or compliance reporting. Knowing that you need to drive east to west on a road map from New York to California is helpful but ultimately insufficient to get you from a specific location to another.
Existing tools promise a high degree of automation, yet their many false positives result in painful manual work that never stops. Although data gets scanned automatically, performance breaks down at scale, or you manually need to fine-tune the computing resources of the scanners. Last but not least, your security team objects to the agent-based processing that requires taking data out of your data platform, and the associated data residency concerns may give you pause.
At Immuta, we believe that data security should not be painful. We believe that you can innovate and move quickly, while at the same time protecting your data and adhering to your internal policies and external regulations. Technology and automation allow you to make the right trade-off decisions quickly. It all starts with highly accurate and actionable metadata. If you trust your metadata and if it’s actionable, you can leverage it to automatically grant access to data, mask sensitive information, and automate your audit reporting.
Immuta was built to tackle those challenges and address them through a unique architecture that was designed in collaboration with the largest financial institutions, healthcare companies, and government agencies in the world. The cloud and AI paradigm requires a fundamentally different approach. You must assume that your data is dynamic, unique, and collected in a multitude of different geographies and legal jurisdictions. Immuta is built for this new world and its specific demands.
Identifying and classifying data requires analyzing and looking at the data - there’s no way around it. Immuta does all the analysis and processing inside the remote technology. It takes advantage of those platforms’ inherent scalability to enable you to analyze large amounts of data quickly, efficiently, and without the need for separate resource optimization for containers or virtual machines.
By processing data directly inside the data platform, Immuta automatically adheres to data residency and locality requirements. If you run your data warehouse or lake globally - across North America, the European Union, and Asia - Immuta processes the data in the region where your data is stored. No data ever leaves the data platform, and it will never move across different cloud regions.
In-platform processing greatly reduces risk and improves your data security posture. Provisioning agents, whether they’re in a container, virtual machine, or Amazon Machine Image (AMI), create complexity and an unnecessary security risk. Not only can those agents become compromised, but their misconfiguration might lead to data leaks to other parts of your cloud infrastructure. An agentless approach can better leverage data platform optimizations to process data instead of transferring it out to re-optimize and analyze. This simplifies operations and increases efficiency for your infrastructure teams.
The advantages of in-platform processing are abundant, but implementing it across a multitude of platforms is challenging. Immuta helps bypass the obstacles by doing all the heavy lifting for you and building in specific implementations for each technology. Although all those implementations are ultimately different, Immuta abstracts the results to one standardized taxonomy, so you can have consistently accurate and granular metadata across all your data stores.
Immuta classifies data on a column level and instantaneously identifies schema changes. Only with that level of granularity and automation can you adhere to your audit requirements and understand what actions have been taken on your data. For example, if non-sensitive data is joined with sensitive data at query time, Immuta will monitor and record that for your review. Continuous object sync ensures schema changes never result in holes in your access controls and data security posture.
Trust in your metadata is critical for data security.
To unblock your data consumers, you need to automate your data access controls; this requires trusting that your classification and metadata are accurate and actionable. Immuta identification provides you with highly accurate metadata and tags out-of-the-box and assists you in fine-tuning the classification mechanism to deal with false positives quickly. That enables you to build policies that dynamically grant or restrict access to protected data (like PHI or PII) depending on who is accessing it and what protections you want to apply.
Immuta works in three phases to identify, categorize, and classify your data:
Identification: In this first phase, data is identified by its kind – for example, a name or an age. This identification can be manually performed, externally provided by a catalog, or automatically determined through column-level analysis of patterns.
Categorization: In the second phase, data is categorized in the context of where it appears, subject to your active frameworks. For example, a record occurring in a clinical context containing both a name and individual health data is protected health information (PHI) under HIPAA.
This categorization of data helps to understand the context it is in, including information like whether or not a record pertains to an individual, the composition and kinds of identifiers present, the data subject, whether the data belongs to any controlled data categories under certain legislation, etc.
Classification: In the third and final phase, data is classified according to its sensitivity level (e.g., Customer Financial Data is Highly Sensitive) and the risk associated to the data subject. support 3 sensitivity levels. However, organizations are free to customize the sensitivity names for the tags as needed.
Enter a name and description for the new identifier.
Click Next.
Enter criteria: Select the Type of criteria.
For regex, enter a regex to be matched against column values. The default criteria encoding is case-sensitive. You can change this encoding using the regex criteria. The regex must use RE2.
For a dictionary, enter the values in a comma-separated list to match against column values. Opt to toggle the Case insensitive switch to on if you want the dictionary to be case sensitive.
For column name regex, enter a regex to be matched against column names. The default criteria encoding is case-insensitive. You can change this encoding using the regex criteria. The regex must use RE2 syntax.
Click Next.
Select the tags to apply: Use the text box to search for a tag or type a tag name to create a new tag under the "Discovered . Entity" hierarchy to apply to columns that match your identifier.
Click Next to review your new identifier and click Create Identifier to create it.
Note that all user-created identifiers must be a 90% match or greater for the contents of the column to be tagged.
Editing the details of an identifier from the identifiers page will only affect that identifier; no copies of that identifier will be impacted.
To edit a reference identifier,
Click Metadata in the navigation menu and select Identifiers.
Click the more actions icon of the identifier you want to edit.
Select Edit.
Edit the field you want to change.
Click Save.
Built-in identifiers cannot be edited.
To edit a domain identifier,
Click Domains in the navigation menu and select your domain.
Select the Identifiers tab.
Click the more actions icon of the identifier you want to edit.
Deleting a domain identifier from the identifiers page will remove it from the domain it is in.
To delete a reference identifier,
Click Metadata in the navigation menu and select Identifiers.
Click the more actions icon of the identifier you want to delete.
Select Delete and click Save.
Built-in identifiers cannot be deleted.
To delete a domain identifier,
Click Domains in the navigation menu and select your domain.
Select the Identifiers tab.
Click the more actions icon of the identifier you want to delete.
Keep track of the updates made to the built-in identifiers
Identifiers in domains is released as GA and these identifier updates are coupled with that release.
The following identifiers have been improved to better match their intended data patterns. These updates have only been made to the built-in reference identifiers. If these are already in your domains, they will remain there as domain-specific identifiers with the previous pattern. If you want to add these improved identifiers to your domain, edit the name because identifier names must be unique within each domain.
To see more about the specific changes made, see the annotations on the Built-in identifier reference page.
AUSTRALIA_MEDICARE_NUMBER
AUSTRALIA_PASSPORT
BRAZIL_CPF_NUMBER
CANADA_PASSPORT
CREDIT_CARD_NUMBER
DATE
DOMAIN_NAME
FDA_CODE
FRANCE_NIR
GENDER
ICD10_CODE
IMEI_HARDWARE_ID
MAC_ADDRESS
PERSON_NAME
POSTAL_CODE
SPAIN_NIF_NUMBER
TIME
UK_NATIONAL_INSURANCE_NUMBER
URL
US_HEALTHCARE_NPI
US_SOCIAL_SECURITY_NUMBER
US_STATE
The following identifiers are deprecated and no longer included in the reference identifiers. If these are already in your domains, they will remain there as domain-specific identifiers.
AGE
DENMARK_CPR_NUMBER
FINLAND_NATIONAL_ID_NUMBER
FRANCE_CNI
The following identifiers are newly created to identify common data patterns. Copy these new reference identifiers to any of your domains.
BELGIUM_NATIONAL_REGISTRATION_NUMBER: Detects numeric strings consistent with Belgium's National Registration Number. Requires 11 characters in the form YY.MM.DD-NNN-XX, where YY.MM.DD corresponds to birth date, NNN is a number, and XX is a checksum digit.
COUNTRY: Detects strings consistent with the names of all countries in the world. This identifier is case-insensitive.
FINANCIAL_INSTITUTIONS: Matches strings consistent with names of financial institutions based on lists provided by the FDIC and OCC, includes alternative names.
62 built-in identifiers are released for use with identification.
Tune identification and classification frameworks to adjust where tags are applied based on your own security and compliance needs
Requirements:
Registered data sources; see the reference page for supported technologies
Immuta permission GOVERNANCE
Immuta provides identifiers out-of-the-box to recognize and tag data. Users can then utilize classification frameworks and build them to apply tags based off those identifier tags and their own catalog tags.
Tune identifiers first to adjust where the tags are applied. Because classification frameworks can apply classification tags from the identification applied tags, tuning identification should come first and will have trickle-down effects on classification. Customizing identification requires some initial work but will automate data tagging for all data sources in the future.
Follow the steps below to tune identification for your data:
After identification has applied entity tags, any active classification frameworks will automatically reapply their tags to account for any changes to tags. It may be necessary to adjust the classification tags based on your organization's data, security, and compliance needs.
After identification runs, you will receive a notification that the job is complete. Then, you can view the results from the data source columns.
Navigate to the data source overview page of the data source you added to the framework.
Click the Columns tab.
Assess whether the tags are applied as expected.
Requirement: Immuta permission GOVERNANCE or data owner
Target some data sources to manually review tags:
Navigate to the Data Sources page and click the Columns tab to open the data source columns.
You will see the data source columns, with details about the name, data type, and a list of the tags on each column. Assess whether the tags are accurate to your data.
Tags may be unexpected but still accurate to your data. Additionally, they may have been applied because they were found to be the best match from the identifiers in the framework.
If you want to improve identification and personalize it to your data, assess why the tag was applied to your data:
Is the identifier incorrectly matching your data and irrelevant to your organization? .
Is the identifier incorrectly matching this specific column, but correct in other places? It must have been the most correct match found by identification. Create a better match by completing the following steps:
.
If you want to remove the unexpected tags, use one of the following how-to guides:
.
Ensure the tags are applied properly by adjusting identification.
. Note that classification tags build off of other tags, so removing a single classification or identification tag can have trickle-down effects on the data source.
If you were expecting some sensitive data to be tagged and it is not, enable additional tags using one of the following how-to guides:
.
Ensure the tags are applied properly by adjusting identification.
. Note that classification tags build off of other tags, so adding a single tag can have trickle-down effects on the data source.
Requirement: Immuta permissions GOVERNANCE and AUDIT
Tags can be edited on an individual basis for each data source. If broad changes to the classification framework are necessary to re-tag your data, use the .
Navigate to the Data Sources page and select the data sources that you assessed and noted issues.
Click the Columns tab.
Delete unnecessary tags by clicking on the tag you want to remove from the column, and select Disable from the tag side sheet.
Create, view, or import tags
Click the Metadata icon in the navigation menu and select the Tags tab.
Click Add Tags.
Complete the Enter tag name field.
Additional nested tags are optional. These nested tags follow a tree structure. There are parent, sibling, and child tags. Click Remove Tag to remove a nested tag.
Click Save.
Click the Data icon in the navigation menu and select the Data Sources tab.
Select a data source.
Navigate to the Columns tab.
Click the Metadata icon in the navigation menu and select the Tags tab.
A list of all top-level tags will be displayed. Click the expand arrow to view nested tags.
Click the tag itself or the icon in the Actions column to edit tags, generate tag reports, or delete tags.
You can pull external tags that you had previously defined in the external catalog (e.g., Collibra, Snowflake, etc.).
Click the Metadata icon in the navigation menu and select the Tags tab.
Click Refresh External Tags.
External tags will be automatically detected when you create a new data source that originates in an external catalog, or they can be linked directly from the data source details page.
When using custom REST catalogs, the GET/dataSource/page/{id} endpoint returns a human-readable information page from the REST catalog for the data source associated with {id}. Immuta provides this as a mechanism for allowing the REST catalog to provide additional information about the data source that may not be directly ingested by or visible within Immuta. This link is accessed in the Immuta UI when a user clicks the catalog logo associated with the data source on the data source details page.
Add tags to data sources and projects so that data can be targeted by policies that use those tags
Click the Data icon in the navigation menu and select the Data Sources tab.
Select a data source.
Click the Add Tags button on the Details tab.
Begin typing a tag name in the Search by Name field and select the tag from the dropdown list.
Click Add. A list of the applied tags will populate on the Details tab.
Repeat as necessary for other data sources and tags.
Click the Data icon in the navigation menu and select the Data Sources tab.
Select a data source.
Scroll to the Tags section on the Details tab, and click on the tag you want to remove.
The columns tab of the data source lists the columns and the value type of the data within each column. From this page, governors can add tags to or remove them from specific columns in a data source.
Navigate to a data source and click the Columns tab.
Scroll to the column you want to add a tag to and click Add Tags.
Begin typing in the Search by Name field and select the tag from the dropdown list.
Navigate to a data source and click the Columns tab.
Scroll to the column you want to remove the tag from and click on the tag you want to delete.
Click Delete in the side sheet and then click Confirm.
Click the Data icon and select Projects in the navigation menu.
Select a project.
Click the Add Tags button on the Project Overview tab.
Click the Data icon and select Projects in the navigation menu.
Select a project.
Scroll to the Tags section on the Overview tab, and then click the tag you want to delete.
Enable identification to automatically tag your data
This how-to guide is for enabling identification for the first time. For additional information on identification, see the .
Requirement: Immuta permission GOVERNANCE
Prerequisites
Identifiers can be added to and identification can run in any of your current domains. However, if you are not already using domains, set up a domain specifically to run identification:
Click Edit.
Edit the field you want to change.
Click Save.
Select Delete and click Save.
GERMANY_IDENTITY_CARD_NUMBER
SPAIN_NIE_NUMBER
SWEDEN_NATIONAL_ID_NUMBER
SWEDEN_PASSPORT
THAILAND_NATIONAL_ID_NUMBER
UK_TAXPAYER_REFERENCE
US_BANK_ROUTING_MICR
US_PASSPORT
US_TOLLFREE_PHONE_NUMBER
GREAT_BRITAIN_DRIVERS_LICENSE: Previously named UK_DRIVERS_LICENSE_NUMBER. Now, renamed because it does not detect license numbers from Northern Ireland.
ICD_10_PCS: Detects strings consistent with procedure codes from the International Statistical Classification of Diseases and Related Health Problems (ICD), as drawn from the Clinical Modification lexicon from 2020.
NAICS_CODE: Detects strings consistent with North American Industry Classification System (NAICS). A two-digit number represents a basic sector and each preceding digit represents a more specific sub sector with a maximum of six digits.
SEC_STOCK_TICKER: Matches strings consistent with the stock tickers recognized by the U.S. Securities and Exchange Commission (SEC).
US_PERSON_FULL_NAME: Detects strings consistent with a person's {first name} space {last name}. Uses the same names from the PERSON_NAME identifier. This identifier must match at least 20% of the data sampled and is case-insensitive.
US_STREET_ADDRESS: Previously named STREET_ADDRESS.
Deleting tags from the governance page will not remove them from data sources
Deleting a tag from the governance page only means it cannot be used on data sources in the future. To remove a tag from a data source, delete it from the data source directly. This design prevents mass exposure of data from just the deletion of a tag.
Click Delete in the side sheet and then click Confirm.
Click Add. A list of the applied tags will populate on the project overview.
If you want additional tags, follow the Create an identifier guide to create identifiers that matter to your data.
Add the identifier to the domain so this column is correctly matched by identification.
Click Add Tags in the Actions column.
Begin typing the name of the tag you want to add in the Search by Name field and select the tag from the dropdown list.
Click Add.
Navigate to the Identifiers tab of your domain.
Click Get Started.
Add reference identifiers to your domain that are relevant to your data by clicking the checkboxes. The identifier becomes a point-in-time copy of the reference identifier. It has the same name, criteria, and tags. Note you cannot add multiple identifiers with the same name to the same domain, so if you want to add an improved reference identifier, edit the name.
Click Add Identifiers.
This action can be done within a domain from the Identifiers tab to create a domain-specific identifier, or it can be done from the Identifiers page to create a reference identifier.
Click Create New.
Enter a name and description for your identifier.
Click Next.
Enter criteria: Select the .
For regex, enter a regex to be matched against column values. The default criteria encoding is case-sensitive. You can change this encoding using the regex criteria. The regex must use RE2 syntax.
For column name regex, enter a regex to be matched against column names. The default criteria encoding is case-insensitive. You can change this encoding using the regex criteria. The regex must use RE2 syntax.
For a dictionary, enter the values in a comma-separated list to match against column values. Opt to toggle the Case insensitive switch to on if you want the dictionary to be case sensitive.
Click Next.
Select the tags to apply: Use the text box to search for a tag or type a tag name to create a new tag under the "Discovered . Entity" hierarchy to apply to columns that match your identifier.
Click Next to review your new identifier and click Create Identifier to create it.
Note that all user-created identifiers must be a 90% match or greater for the contents of the column to be tagged.
Once you have created identifiers relevant to your data, it is time to run them on your data. You may choose to run identification on a select number of data sources where you understand the data to assess and adjust the tags to reflect what you expect to see.
Click Domains in the navigation menu and select your domain.
Open the More Actions icon.
Select Run Identification from the dropdown.
After identification runs, you will receive a notification that the job is complete. Then, you can view the results from the data source columns tab.
Navigate to the data source overview page of the data source you have in the domain.
Click the Columns tab.
Assess whether the tags are applied as expected.
If you are happy with the tags, follow the to add the rest of your data sources to the domain and then on the domain again.
If you want additional tags, follow the to create additional identifiers that matter to your data.
Add only the identifiers that are relevant to your organization's data
For example, for international data, you may want to enable many different identifiers for many countries, like the built-in "Australia Passport" identifier and the "Finland National ID Number" identifier. However, if you are dealing with United States domestic financial data, those identifiers would be irrelevant. In that case, it would be better to identify the data more likely to appear, like "Bitcoin" or "US Bank Routing MICR".
Learn how competitive pattern analysis assesses your data to apply column tags
Of identification's , regex and dictionary are competitive. This means that when assessing your data, if multiple identifiers could match, only one with competitive criteria will be chosen to tag the data. To better understand how Immuta executes this competition, read further.
Immuta employs a three-phased competitive criteria analysis approach for identification:
: No data is moved, and Immuta checks the identifiers against a sample of data from the table.
Scoring: The remaining identifiers are compared with one another to find the most specific criteria that qualifies and matches the sample.
In the end, competitive criteria analysis aims to find a single identifier for each column that best describes the data format.
In the sampling process, no database contents are transmitted to Immuta; instead, Immuta receives only the column-wise hit rate (the number of times the criteria has matched a value in the column) information for each active identifier. To do this, Immuta instructs a remote database to measure column-wise hit rate information for all active identifiers over a row sample.
The sample size is decided based on the number of identifiers and the data size, when available. In the most simplified case, the requested number of sampled rows depends only on the number of regex and dictionary criteria being run in the domain, not the data size. The sample size dependence on the number of identifiers is weak and will not exceed 13,000 rows.
5
7369 rows
50
9211 rows
500
11053 rows
In practice, the number of sampled values for each column may be less than the requested number of rows because columns are not independently sampled but rather projected from a row-wise sample. This can impact the sample when the target table has less than the requested number of rows, when some of the column values are null, or because of technology-specific limitations.
Snowflake and Starburst (Trino): Immuta implements table sampling by row count.
Databricks and Redshift: Due to technology limitations and the inability to predict the size of the table, Immuta implements a best-effort sampling strategy comprising a flat 10% row sample capped at the first 10,000 sampled rows. In particular, under-sampling may occur on tables with less than 100,000 rows. Moreover, the resulting sample is biased towards earlier records.
All platforms: Sampling from views can have significantly slower performance that varies by the performance of the query that defines the view.
All platforms: Any null values included in the sample will not count towards the qualification or scoring when included in the sample. However, it will lower the number of available values to match against the patterns, as the sample size is not dynamic based on the ignored null values.
During the qualification phase, identifiers that do not agree with the data are disqualified. An identifier agrees with the data if the hit rate on the remote sample exceeds the predefined threshold. This threshold is 90% match for most built-in identifiers; however, a few built-in identifiers have lower threshold requirements. The 90% threshold is standard for all custom identifiers as well to ensure the criteria matches the data within the column and to avoid false positives. Note that threshold calculations are relative to the number of non-null entries for each column.
If no identifiers qualify, then no identifier is assessed for scoring and the column is not tagged.
During the scoring phase, a machine inference is carried out among all qualified identifiers, combining criteria-derived complexity information with hit rate information to determine which identifier best describes the sample data. This process prefers the more restrictive of two competing identifiers since the ability to satisfy the more difficult-to-satisfy identifier itself serves as evidence that it is more likely. This phase ends by returning a single most likely identifier per the inference process.
Here are a set of regex identifiers and a sample of data:
Identifiers:
[a-zA-Z0-9]{3} - This regex will match 3 character strings with the characters a-z, lowercase or uppercase, or digits 0-9.
[a-c]{3} - This regex will match 3 character strings with the characters a-c, lowercase.
(a|b|d){3} - This regex will match 3 character strings with the characters a, b, or d, lowercase.
dad
Yes
❌
Yes
When qualifying the identifiers, Identifier 1 and Identifier 3 both match 90% or more of the data. Identifier 2 does not, and is disqualified.
Then the qualified identifiers are scored. Here, Identifier 1, despite matching 100% of the data, is unspecific and could match over 200,000 values. On the other hand, Identifier 3 matches just at 90% but is very specific with only 27 available values.
Therefore, with the specificity taken into account, Identifier 3 would be the match for this column, and its tags would be applied to the data source in Immuta.
Dictionaries are part of the competitive process, while column-name regex are not.
Scoring ties are rare but can occur if the same criteria (either dictionary or regex) is specified more than once (even in different forms). Scoring ties are inconclusive, and the scoring phase will not return an identifier in the case of a tie.
Criteria complexity analysis is sensitive to the total number of strings an identifier accepts or, equivalently for dictionaries, the number of entries. Therefore, identifiers that accept much more than is necessary to describe the intended column data format may perform more poorly in the competitive analysis because they are easier to satisfy.
Learn about how classification frameworks categorize the risk level of your data
Classification is the process in which data is categorized by the content and the associated risk level based on context. To classify your data, Immuta evaluates your data in two phases:
Identification runs to identify your data by content type. The data is discovered and evaluated by the identifier it matches and is tagged.
Classification runs to classify your data by its context. The data is classified by the rules within a framework and the tags currently applied to the column and table. Once the data is classified, it's tagged with special tags with additional metadata used in the as sensitivity and visualize when that sensitive data is accessed.
5000
12895 rows
baa
Yes
❌
Yes
add
Yes
❌
Yes
add
Yes
❌
Yes
cab
Yes
Yes
❌
bad
Yes
❌
Yes
aba
Yes
❌
Yes
baa
Yes
❌
Yes
dad
Yes
❌
Yes
baa
Yes
❌
Yes
Using classification to assign risk and sensitivity levels to your data and audit dashboards to visualize the risk levels offers these benefits:
Increasing the semantic understanding of your data to better meet compliance requirements
Reducing the time to make decisions about what data access is allowed under what purposes
Reducing the effort and time to respond to auditors about data access in your company
Reducing the labor of classifying data to enumerate what data is within the scope of security or regulatory compliance frameworks
Both entity and classification tags describe the content of data on a per-column basis, and you can use them to monitor data access and build access policies. However, there are key differences between the two:
Entity tags are applied through identification and describe what the data is. Identification applies entity tags to columns based on the patterns of the data.
Classification tags are applied through categorization and risk assessment and describe the context of the data and the risk it poses. Using classification frameworks, classification tags are applied to columns based on the entity tags previously applied by identification. Additional classification tags can then be applied, providing even more context or expressing the property of the record rather than just the column.
Entity tags describe the contents of individual columns, in isolation. But you don't access individual columns in isolation, so why would you determine their sensitivity that way? Entity tags do not attempt to and cannot contextualize column contents with neighboring columns' contents. This means that connections between data are lost if they cannot be identified through a pattern within the column itself. Classification tags describe the contents of a table with the context of all its columns, providing a holistic view of the risk of the data for what it is, rather than the pattern it fits. Context is necessary to understand whether your data is public or private data, risky or safe to have ungoverned access, or sensitive and creating toxic joins when accessed with other tables.
For example, under HIPAA, a list of procedures a doctor performed is only considered protected health information (PHI) if it can be associated with the identity of patients. Since entity tagging operates on a single column-by-column basis, it cannot reason whether or not a column containing procedure codes merits classification as PHI. Therefore, entity tagging will not tag procedure codes as PHI. But classification tagging will tag it PHI if it detects patient identity information in the other columns of the table.
Additionally, entity tagging does not indicate how sensitive the data is, but classification tags can carry a sensitivity level. For example, an entity tag may identify a column that contains telephone numbers, but the entity tag alone cannot say that the column is sensitive. A phone number associated with a person may be classified as sensitive, while the publicly listed phone number of a company might not be considered sensitive.
After you understand what entities your data contains using identification, you need to adopt frameworks that determine what combinations of data constitute sensitive data and their level of sensitivity.
Frameworks are a set of data categories and a set of classification rules to place data into those categories. In Immuta, the data categories are represented by tags, and when data fits a classification rule the tag is applied:
Classification tags are applied based on the tags applied by identification or other tags on the data source. Classification tags contain additional metadata about each column, such as the source of the tag, the dimension, and the sensitivity level. This metadata is used in the framework rules and complex formulas that assign the sensitivity of queries visible in audit dashboards.
Classification rules determine how each classification tag is applied. These rules can apply tags based on tags already on the column, tags applied to neighboring columns, and tags applied to the data source. This means that the complete data source is considered when classifying your data sources, and even tags applied to individual columns can affect the risk level of the entire data source.
Frameworks are often built off of an interpretation of regulatory frameworks or standards, such as the US Health Insurance Portability and Accountability Act (HIPAA) and the PCI standard. However, organizations can also build frameworks that represent their internal business processes. When used in Immuta, they automate data tagging and provide information about what data you have immediately after it is registered in Immuta.
Data classification is a process, and with Immuta, much of it is automated. This means that you can reap the benefits of classified and tagged data quicker and easier than manually classifying and tagging it:
Quick data access control: Use classification to identify and classify your data immediately after registration in Immuta. Then, build governance policies off of those tags. This repeatable process will protect your data in its current state and whenever any new data sources are created. Automate the process further with schema monitoring; schema monitoring allows you to register data just once. Then, Immuta will monitor your data environment for changes and, when found, update the data source in Immuta, update the tags on that data source, and then update user access based on your governance policies when changes happen.
Scale your data monitoring: Use classification to identify and classify your data immediately after registration in Immuta. Then, view your data users' access to your sensitive and risky data through the audit dashboards.
Build data platform compliance: Create classification frameworks to identify and classify your data based on the industry practices and regulations your organization needs to abide by. Once the frameworks are built, they will automatically tag data as it's registered, ensuring your data sources are properly tagged to abide by the regulations you care about.
Customize a classification framework to use your own catalog tags
After you have registered data sources in Immuta, you can start automating data classification of a column based on its context, which is the combination of
associated tags already applied to the column
tags applied to the neighboring columns and
table tags on the data source.
The starter framework in this how-to is built to map a classification scale of restricted, confidential, internal, and public to Immuta's three-level scale of sensitivity. The sensitivity in the classification tags will then appear in .
Follow this guide to map your tags to the example framework, or consult the for more information about the framework schema.
Using the example framework below, customize the framework for your organization's classification tags:
For more information about these parameters see the .
tags: These tags are automatically created in Immuta with the sensitivity you assign. They must not already exist in Immuta. All tags used in the classificationTag parameter should be defined here.
tags.sensitivities: This is metadata for the sensitivity of the new tag. Use confidentiality for dimension. Options for sensitivity
Follow the example below to map your tags to the rules in the example framework.
This example framework has a rule where columns tagged DSF.Interpretation.Credentials.Secret by identification will be tagged RAF.Confidentiality.High:
To translate this to your tags, replace the name and source value of the columnTags, neighborColumnTags, or tableTags with your own. This new example is for a Collibra tag from the external catalog that an organization uses for confidential data. This rule now states: Apply the classification tag RAF.Confidentiality.High to a column if it has the collibra tag Confidential. Repeat this for your organization's remaining classification levels.
If you do not know the name or source for your tags, you can list your tags using the Immuta API:
This request will list all the tags in your Immuta environment, similar to this example response:
Requirement: Immuta permission GOVERNANCE
Once you have made all the customizations to the example framework, make the following request using the Immuta API, with your full customized framework as the payload.
Your new framework will now be visible in the Immuta UI by navigating the Classification section.
Connect your external catalog to pull data metadata into Immuta
This page outlines how to connect an external catalog to Immuta. For details on external catalogs in Immuta, see the .
Requirements:
APPLICATION_ADMIN Immuta permission
An Alation user with the permission
12rules: These are the rules for applying the tags defined above. Each rule contains the classification tag to apply if the requirements are met and the requirements: the column tags, neighboring column tags, and table tags that must be present. All requirements within each defined rule must be met for the classification tag to be applied.
rules.classificationTag: The name and source of the tag you want applied if the rule requirements are met. This classification tag must be defined in tags. The source is curated.
rules.columnTags: These are the required tags for a column. If the tags defined here are found on a column, and the other tag rules are met, then the rule's classificationTag will be applied to the same column.
rules.neighborColumnTags: These are the required tags on other columns in the data source (or in the query if dynamic query classification is enabled). If the tags defined here are found on any column in the data source, and the other tag rules are met, then the rule's classificationTag will be applied to all the neighboring columns.
rules.tableTags: These are the required tags on the data source. If the tags defined here are found on the data source, and the other tag rules are met, then the rule's classificationTag will be applied to all the columns in that data source.
active: When true the framework is active and will apply tags when the rules are met.
{
"shortName": "ECMC Framework",
"name": "External Catalog Mapping Classification Framework",
"description": "This framework maps the classification tags the organization has in Collibra to Immuta data sources.",
"tags": [
{
"name": "ECMC.Confidentiality.Highly Sensitive",
"source": "curated",
"sensitivities": [
{
"dimension": "confidentiality",
"sensitivity": 2
}
]
},
{
"name": "ECMC.Confidentiality.Sensitive",
"source": "curated",
"sensitivities": [
{
"dimension": "confidentiality",
"sensitivity": 1
}
]
},
{
"name": "ECMC.Confidentiality.Nonsensitive",
"source": "curated",
"sensitivities": []
}
],
"rules": [
{
"name": "ECMC 00001",
"classificationTag": {
"name": "ECMC.Confidentiality.Highly Sensitive",
"source": "curated"
},
"columnTags": [
{
"name": "Restricted",
"source": "collibra"
}
],
"neighborColumnTags": [],
"tableTags": []
},
{
"name": "ECMC 00002",
"classificationTag": {
"name": "ECMC.Confidentiality.Sensitive",
"source": "curated"
},
"columnTags": [
{
"name": "Confidential",
"source": "collibra"
}
],
"neighborColumnTags": [],
"tableTags": []
},
{
"name": "ECMC 00003",
"classificationTag": {
"name": "ECMC.Confidentiality.Sensitive",
"source": "curated"
},
"columnTags": [
{
"name": "Internal",
"source": "collibra"
}
],
"neighborColumnTags": [],
"tableTags": []
},
{
"name": "ECMC 00004",
"classificationTag": {
"name": "ECMC.Confidentiality.Nonsensitive",
"source": "curated"
},
"columnTags": [
{
"name": "Public",
"source": "curated"
}
],
"neighborColumnTags": [],
"tableTags": []
}
],
"active": true
}"rules": [
{
"name": "RAF 00004",
"classificationTag": {
"name": "RAF.Confidentiality.High",
"source": "curated"
},
"columnTags": [
{
"name": "DSF.Interpretation.Credentials.Secret",
"source": "curated"
}
],
"neighborColumnTags": [],
"tableTags": []
}
]"rules": [
{
"name": "RAF 00004",
"classificationTag": {
"name": "RAF.Confidentiality.High",
"source": "curated"
},
"columnTags": [
{
"name": "Confidential",
"source": "collibra"
}
],
"neighborColumnTags": [],
"tableTags": []
}
]curl \
--request GET \
--header "accept: application/json" \
--header "Authorization: Bearer <your-token." \
https://your-immuta-url.com/tag[
{
"id": 114,
"name": "DataProperties.Cross-Sectional",
"source": "curated",
"deleted": false,
"systemCreated": true
},
{
"id": 2,
"name": "Discovered.Country.Argentina",
"source": "curated",
"deleted": false,
"systemCreated": true
},
{
"id": 9,
"name": "Discovered.Country.Australia",
"source": "collibra",
"deleted": false,
"systemCreated": true
}
]curl \
--request POST \
--header "Content-Type: application/json" \
--header "Authorization: Bearer <your-token>" \
--data @example-payload.json \
https://your.immuta.com/frameworks/Navigate to the App Settings page.
Scroll to 2 External Catalogs, and click Add Catalog.
Enter a Display Name and select Alation from the dropdown menu.
Enter the HTTP endpoint of the catalog in the URL field.
Select an Authentication Method from the dropdown menu. Immuta will use the credentials provided to connect to the external catalog:
API key: The API key must be an for your Alation instance. To change the default expiration period for your Alation catalog's API tokens, see .
OAuth 2.0:
Configure whether or not Alation tags and custom fields are imported as Immuta tags:
Link Alation tags: When selected, Immuta imports Alation tags as Immuta tags.
Link Alation Custom Fields: When selected, Immuta imports Alation custom fields as Immuta tags. Follow the Alation documentation to , , and apply custom fields to tables and columns.
Opt to select Upload Certificates.
Upload the Certificate Authority, Certificate File, and Key File.
Opt to enable Strict SSL by selecting the checkbox.
Click the Test Connection button.
Once the connection is successful, click Save.
Requirements:
APPLICATION_ADMIN Immuta permission
An Atlan API key with permissions to read Atlan assets that correspond to Immuta data sources
Navigate to the App Settings page.
Scroll to 2 External Catalogs, and click Add Catalog.
Enter a Display Name and select Atlan from the dropdown menu.
Complete the URL and API key fields. The API key must be an API access token for your Atlan instance. Immuta will use this API key to connect to the external catalog.
Click the Test Connection button.
Once the connection is successful, click Save.
Requirements:
APPLICATION_ADMIN Immuta permission
A Collibra user with visibility on all assets relevant to Immuta data sources (Collibra global role Catalog or Catalog Author)
A Collibra physical data dictionary with assets that correspond to your Immuta data sources
Navigate to the App Settings page.
Scroll to 2 External Catalogs, and click Add Catalog.
Enter the Display Name and select Collibra from the dropdown menu.
Enter the HTTP endpoint of the catalog in the URL field.
Select an authentication method from the dropdown menu. Immuta will use the credentials provided to connect to the external catalog:
Username and password: Complete the Username and Password fields.
OAuth 2.0:
Complete the Asset Mappings modal to set which align to the Immuta data source and column. Immuta will only link data sources from the asset types you specify.
Complete the Attributes as Tags modal to specify which you want in Immuta. These attributes will come in as parent tags with their values as children tags.
Opt to select Upload Certificates.
Upload the Certificate Authority, Certificate File, and Key File.
Opt to enable Strict SSL by selecting the checkbox.
Click the Test Connection button.
Once the connection is successful, click Save.
Requirements:
APPLICATION_ADMIN Immuta permission
A Microsoft Purview catalog with assets that correspond to your Immuta data sources
The ability to create a registered app in the Azure portal. See the prerequisite.
Register an app in the Azure portal with the with the following settings:
Supported account type: "Accounts in this organizational directory only"
Microsoft-Graph: User.Read API permission
A client secret
Using that registered app, navigate to Immuta and complete the following:
Navigate to the App Settings page.
Scroll to 2 External Catalogs, and click Add Catalog.
Enter the Display Name and select Microsoft Purview from the dropdown menu.
Complete the following fields:
Enter the Microsoft Purview endpoint URL including the Azure Account Name, like https://<ACCOUNTNAME>.purview.azure.com, in the Purview Endpoint URL field.
Complete the Microsoft Entra Directory (tenant) ID and Microsoft Entra (client) ID fields.
Click the Test Connection button.
Once the test is successful, click Save.
Requirements:
APPLICATION_ADMIN Immuta permission
An external catalog with tags that correspond to your Immuta data sources
Integrating a custom REST catalog service with Immuta requires implementing a REST interface. For details about the necessary endpoints that must be serviced, see the Custom REST catalog interface endpoints page.
Navigate to the App Settings page.
Scroll to 2 External Catalogs, and click Add Catalog.
Enter the Display Name and select Rest from the dropdown menu.
Select the Internal Plugin checkbox if the catalog has been uploaded to Immuta as a custom server plugin.
Complete the following fields:
Enter the HTTP endpoint of the catalog in the URL field.
Complete the Username and Password fields.
Opt to enter the path to the information page for a column in the Column Link Template field.
Opt to upload a Catalog Image.
Opt to select Upload Certificates.
Upload the Certificate Authority, Certificate File, and Key File.
Opt to enable Strict SSL by selecting the checkbox.
Click the Test Connection button.
Click the Test Data Source Link.
Once both tests are successful, click Save.
You can manually link and remove external catalogs from data sources on the data source details tab.
Navigate to your data source.
In the connection information section, click the Link Catalog icon (or Unlink Catalog to remove an external catalog from a data source).
Select your external catalog from the dropdown menu and enter the appropriate ID:
Alation: Enter the from Alation into the Catalog Id field.
Atlan: Enter the from Atlan into the Catalog Id field.
Collibra: Enter the from Collibra into the Catalog Id field.
Microsoft Purview: Enter the GUID of the asset from Microsoft Purview into the Catalog Id field.
Click Link to confirm.
Navigate to your data source and click the data source Health dropdown menu.
Click Re-run in the External Catalog section.
Private preview: This feature is available to select accounts. Contact your Immuta representative to enable this feature.
Private preview: This feature is available to select accounts. Contact your Immuta representative to enable this feature.
Alation OAuth provider: Generate the client ID and client secret in Alation. Immuta will use these credentials to communicate with Alation. See the Alation documentation for more details.
Fill out the Client ID. This is a combination of letters, numbers, or symbols used as a public identifier.
Enter the Client Secret you created.
Leave the Token URL and Scope field blank.
External OAuth provider: Use your external OAuth provider client ID and client secret. Immuta will use these credentials to request an access token from Alation's token endpoint. Then, Immuta will use that returned access token as the bearer token in API calls with Alation.
In Alation, allow for external tokens to be accepted and validated in the JWT format. See the Alation documentation for more details.
In Immuta, fill out the Client ID from your external OAuth provider. This is a combination of letters, numbers, or symbols used as a public identifier.
Enter the Client Secret from your external OAuth provider. Immuta uses this secret to authenticate with the authorization server when it requests a token.
Add the token endpoint URL in the Token URL field.
Leave the Scope field blank.
Collibra OAuth provider: Generate the client ID and client secret in Collibra. Immuta will use these credentials to communicate with Collibra. See the Collibra documentation for more details.
Fill out the Client ID. This is a combination of letters, numbers, or symbols used as a public identifier.
Enter the Client Secret you created above.
Leave the Token URL field blank.
Opt to enter the Scope. The scope limits the operations and roles allowed in Collibra. See the for details about scopes.
External OAuth provider: Use your external OAuth provider client ID and client secret. Immuta will use these credentials to request an access token from Collibra's token endpoint. Then, Immuta will use that returned access token as the bearer token in API calls with Collibra.
Set up Collibra to accept and validate external tokens in the JWT format. See the Collibra documentation for more details.
In Immuta, fill out the Client ID from your external OAuth provider. This is a combination of letters, numbers, or symbols used as a public identifier.
Enter the Client Secret from your external OAuth provider. Immuta uses this secret to authenticate with the authorization server when it requests a token.
Add the token endpoint URL in the Token URL field.
Opt to enter the Scope. The scope limits the operations and roles allowed in Collibra. See the for details about scopes.
Enter the path of the Data Source Endpoint.
Enter the path to the information page for a data source in the Data Source Link Template field.
Learn about the custom REST catalog integration and its endpoints
The custom REST catalog integration allows Immuta to make a defined set of API calls to a custom REST service you develop to retrieve metadata. The custom REST service receives Immuta's calls, and then collects the relevant information and delivers it back to Immuta, allowing you to build and maintain your own solutions that provide metadata required to effectively use Immuta within your organization.
The custom-developed service must be built to receive and handle calls to the REST endpoints specified below. Immuta will call these endpoints when certain events occur and at various intervals. The required responses to complete the connection are detailed below.
Tags are attributes applied to data at the data-source-level or at the column-level.
Tags in Immuta take the form of a nested tree structure:
| Parent (root)
|\_ Child1
| \_ Grandchild1 (leaf)
\_ Child2
\_ Grandchild1 (leaf)The REST catalog interface interprets a tag's relationship mapping from a string based on a standard dot (.) notation:
Tags returned must meet the following constraints:
They must be no longer than 500 characters. Longer tags will not throw an error but will be truncated silently at 500 characters.
They must be composed of letters, digits, underscores, dashes, and whitespace characters. A dot (.) is used as a delimiter as described above. Other special characters are not supported.
A tag object has a single id property, which is used to uniquely identify the tag within the catalog. This id may be either a string or integer type, and its value is up to the designer of the REST catalog service. Common examples include a standard integer value, a UUID, or a hash of the tag's string value (if it is unique within the system).
For this example REST catalog interface, tags are represented in a JSON object. The object below specifies 3 different tags:
For more information on tags and how they are created, managed, and displayed within Immuta, see the .
Descriptions are strings that can be applied to either a data source or an individual column. These strings support UTF-8, including special and various language characters.
Immuta can make requests to your REST catalog service using any of the following authentication methods:
Username and password: Immuta can send requests with a username and a password in the authorization HTTP header. In this case, the custom REST service will need to be able to parse a basic authorization header and validate the credentials sent with it.
PKI certificate: Immuta can also send requests using a CA certificate, a certificate, and a key.
No authentication: Immuta can make unauthenticated requests to your REST catalog service. However, this should only be used if you have other security measures in place (e.g., if the service is in an isolated network that's reachable only by your Immuta environment).
The /tags endpoint is used to collect ALL the tags the catalog can provide. It is used by Immuta to populate Immuta's tags list on the tags page. These tags can then be used for policy creation ahead of actual data sources being created that make use of them. This enables policies to immediately apply when data sources are registered with Immuta.
As with all external catalogs, tags ingested by Immuta from the REST catalog interface cannot be modified locally within Immuta, since this catalog is the source of truth for them.
The custom REST service must respond with an object that maps all tag name strings to associated ids. The fully-qualifies the location of the tag in the tree structure, and the id is a globally unique identifier assigned by the REST catalog to that tag.
The /dataSource endpoint does the majority of the work. It receives a POST request from Immuta, and returns the mapping of a data source and its columns to the applied tags and descriptions.
Immuta will try to fetch metadata for a data source in the system at various times:
During data source creation. During data source creation, Immuta will send metadata to the REST catalog service, most notably the connection details of the data source, which includes the schema and table name. It is important that the custom REST service implemented can parse this information and search its records for an appropriate record to return with an ID unique to this data source in its catalogMetadata object.
. Data sources that either fail to auto-link or that were created prior to the custom REST catalog being configured can still be manually linked. To do so, a data source owner can provide the ID of the asset as defined by the custom REST catalog via the Immuta UI. In order for this to work, the custom REST catalog service must support matching data source assets by unique ID.
Immuta's POST requests to the /dataSource endpoint will consist of a payload containing many of the elements outlined below.
This object must be parsed by the custom REST catalog in order to determine the specific data source metadata being requested.
Immuta will provide the id of the data source as part of the catalogMetadata. This should be used as the primary metadata lookup value.
When a data source is being created, such an id will not yet be known to Immuta. Immuta will instead send handlerInfo information as part of the request.
When an id is not specified, the schema and table name elements should be parsed in an attempt to identify the desired catalog entry and provide an appropriate id. If such a lookup is successful and an id is returned to Immuta in the catalogMetadata section, Immuta will establish an automatic link between the new data source and the catalog entry, and future references will use that id.
The schema for the /dataSource response uses the same tag object structure from the , along with the following set of metadata keys for both data sources and columns.
Example
This endpoint returns a human-readable information page from the REST catalog for the data source associated with {id}. Immuta provides this as a mechanism for allowing the REST catalog to provide additional information about the data source that may not be directly ingested by or visible within Immuta. This link is accessed in the Immuta UI when a user clicks the catalog logo associated with the data source.
Immuta will send a GET request to the /dataSource/page/{id} endpoint.
Example
The custom REST catalog can either provide such a page directly, or can redirect the user to any resource where the appropriate page would be provided - for example a backing full service catalog such as Collibra, if this custom REST catalog is simply being used to support a custom data model.
This endpoint returns the catalog's human-readable information page for the column associated with {id}. Immuta provides this as a mechanism for allowing the REST catalog to provide additional information about the specific column that may not be directly ingested by or visible within Immuta.
Immuta will send a GET request to the /column/{id} endpoint.
Example
The custom REST catalog can either provide such a page directly, or it can redirect the user to any resource where the appropriate page would be provided. For example, it could redirect to a backing full service catalog such as Collibra, if the custom REST catalog is simply being used to support a custom data model.
Learn about supported external catalogs and how they interact with Immuta
Users who want to use tags from outside of Immuta can connect an external catalog to automatically pull and apply them to Immuta data sources. These tags can then be used to drive or .
Immuta supports automated tag ingestion from the following external catalogs:
/dataSource endpoint to ensure information is up to date.catalogMetadata.name
string
The name of the data source in the catalog.
handlerInfo
dictionary
Object holding the data source's connection details.
handlerInfo.database
string
The data source’s database name in the source system. In some platforms this is referred to as the catalog or project.
handlerInfo.schema
string
The data source’s schema name in the source system.
handlerInfo.table
string
The data source’s table name in the source system.
handlerInfo.hostname
string
The data source’s connection schema in the source storage system.
handlerInfo.port
integer
The data source’s connection port in the source storage system.
dataSource
dictionary
Object holding general data source information from Immuta. This can be viewed with debugging, but is not usually required for catalog purposes.
catalogMetadata.name
string
The name of the data source in the catalog.
description
string
A description of the data source.
tags
<tags object>
Object containing the data source-level tags.
dictionary
dictionary
Object containing the column names of the data source as its keys.
dictionary.<column>
dictionary
Object containing a single column's metadata.
dictionary.<column>.catalogMetadata.id
string or integer
The unique identifier of the column in the catalog.
dictionary.<column>.description
string
A description of the column.
dictionary.<column>.tags
<tags object>
Object containing the column-level tags as keys.
catalogMetadata
dictionary
Object holding the data source's catalog metadata.
catalogMetadata.id
string or integer
The unique identifier of the data source in the catalog.
catalogMetadata
dictionary
Object holding the data source's catalog metadata.
catalogMetadata.id
string or integer
The unique identifier of the data source in the catalog.
id
URL parameter, integer, or string
The unique identifier of the data source in the remote catalog system.
id
URL parameter, integer, or string
The unique identifier of the column in the remote catalog system.
Authentication and specific endpoints
When accessing the /dataSource and /tags endpoints, Immuta will use the configured username and password. If you choose to also protect the human-readable pages with authentication, users will be prompted to authenticate when they first visit those pages.
"Parent.Child1.Grandchild1""REST_Catalog_Root": {
"id": "id_is_set_by_catalog_and_can_be_int_or_string"
},
"REST_Catalog_Root.Child1": {
"id": "d3e859da-40e9-43d2-a302-294458e79a64"
},
"REST_Catalog_Root.Child2.Grandchild1": {
"id": 10
}curl http://<your_custom_rest_catalog>/tags \
--header 'Authorization: Basic <base64 of username:password>'{
"REST_Catalog_Root": {
"id": "id_is_set_by_catalog_and_can_be_int_or_string"
},
"REST_Catalog_Root.Child1": {
"id": "d3e859da-40e9-43d2-a302-294458e79a64"
},
"REST_Catalog_Root.Child2.Grandchild1": {
"id": 10
}
}{
"catalogMetadata": {
"id": <unique integer or string value>
}
}{
"handlerInfo": {
"schema": "schema_name",
"table": "table_name"
}
} "catalogMetadata": {
"id": <unique integer or string>
},
"description": <string>,
"tags": {
"Parent": {
"id": <tag_id1>
},
},
"dictionary": {
"some_column_name": {
"catalogMetadata": {
"id": <col_id1>
},
"description": "This column has example data in it",
"tags": {
"Parent.Child1": {
"id": <tag_id2>
},
"Parent.Child1.Grandchild1": {
"id": <tag_id3>
}
}
}
}
}curl http://<your_custom_rest_catalog>/dataSource/page/123<html>
<head>
<title>data source 123</title>
</head>
<body> data source 123 is a data source that was created just for documentation.
</body>
</html>curl http://<your_custom_rest_catalog>/column/10<html>
<head>
<title>data source 123 Column 10</title>
</head>
<body>
Column 10 is full of example data for documentation reasons.
</body>
</html>You can also ingest tags from the following data platforms:
To configure an external catalog, see the Configure an external catalog guide.
Once an external catalog has been configured on the Immuta app settings page, there are two recurring process steps:
Linking to data sources and columns: Immuta will attempt to automatically link data sources to their corresponding assets in an external catalog from two actions:
A new data source is created
An external catalog is set up
This is done by comparing the fully qualified name of a data source in Immuta with its corresponding asset name in the external catalog, so data sources must have the same database, schema, and object name in Immuta and the external catalog. If the match is not found, . Once a data source has been linked to an external catalog, it can be seen on the data source's detail page.
Pull and apply tags in Immuta: Using the link established in the first step, Immuta polls the external catalog to ingest and apply tags to each data source and its columns. Immuta checks every 24 hours for any relevant metadata changes in the connected external catalog. If the external catalog sync for a particular data source is unsuccessful, this will be reflected in the . For data sources where the last external catalog sync failed, Immuta will reattempt the sync every hour until successful. Tags originating from an external catalog can be found on the tags list page and on the columns tab for each data source.
See below for more information about the way Immuta integrates with each supported external catalog provider.
Immuta's Alation integration supports importing both tags and custom fields, Alation's two primary ways of allowing data stewards to apply metadata to data assets.
Tags: Tags are a single word or phrase that can be attached to most Alation objects by nearly anyone. For instance, users can add a PCI tag for financial data.
Custom fields: Custom fields are key-value pairs that can only be attached and removed by authorized users. Unlike tags, custom fields can have multiple values associated with a single key. For example, the custom field DK_STEWARD could have MARKETING, FINANCE, and CUSTOMER values associated with it. Using Alation custom fields allows you to explicitly control who can modify information associated with that field inside of Alation, whereas Alation standard tags are modifiable by any user inside of Alation. The following custom field data types are supported and will be applied to Immuta data sources as tags: pickers, multi-select pickers, object sets, people sets, references, and dates.
When pulled into Immuta, Alation tags and custom fields will be applied to data sources as either column or data source tags in Immuta. Importing both Alation tags and custom fields into Immuta provides full flexibility for organizations leveraging the Alation enterprise data catalog, no matter what operating model they choose to document their metadata in Alation.
Linking to data sources and columns in Alation: Immuta links data sources to assets in Alation by looking up the fully qualified name of an object via the Alation API.
Pull and apply tags in Immuta from Alation: Immuta polls Alation every 24 hours for all tags.
The Atlan catalog integration with Immuta supports ingestion of tags and descriptions from Atlan assets.
Linking to data sources and columns in Atlan: Immuta links data sources to assets in Atlan by looking up the fully qualified name of an entity and its corresponding host information using Atlan APIs. All of the following conditions must be fulfilled for an Immuta data source to successfully link to an Atlan asset:
The Immuta data source name must match the Atlan asset name.
The host URL of the Snowflake account or Databricks workspace used to onboard the data source into Immuta must match the host URL used to onboard the asset into Atlan.
Pull and apply tags in Immuta from Atlan: Immuta checks Atlan every 24 hours for any relevant metadata changes. Based on these changes, Immuta then only polls and ingests tags from Atlan for the relevant data sources. However, if Immuta observes more than 25,000 metadata changes in Atlan within 24 hours, it will poll all data sources for tags during that run of external catalog tag synchronization.
Custom metadata fields from Atlan do not get ingested as tags into Immuta
The current implementation only supports Databricks Unity Catalog or Snowflake data sources and their associated columns.
Immuta's Collibra integration supports importing tags, data classifications, and attributes. Additionally, data source and column descriptions from the connected Collibra catalog will be pulled into Immuta.
Tags: Tags are a single word or phrase that can be attached to objects in Collibra. For instance, users can add a PHI tag on health-related data assets.
Data classifications: Data classifications are a label in Collibra on the asset type column that describe the content of data and are separate from tags in Collibra. Immuta will ingest accepted data classifications from Collibra and apply these classifications as tags on the appropriate columns. All data classifications from Collibra will be under the Data classification parent tag.
Attributes: Attributes in Collibra are a characteristic that describes an asset with an individual field. Unlike tags, attributes can have multiple values associated with a single key. For example, the attribute region could have emea, apac, and nala values associated with it. Using Collibra attributes allows you to explicitly control who can modify information associated with that field inside of Collibra, whereas Collibra standard tags are modifiable by any user inside of Collibra.
When pulled into Immuta, Collibra tags, data classifications, and attributes will be applied to data sources as either column or data source tags in Immuta. Importing Collibra tags, data classifications, and attributes into Immuta provides full flexibility for organizations leveraging the Collibra data catalog, no matter what operating model they choose to document their metadata in Collibra.
Linking to data sources and columns in Collibra: Immuta links data sources to assets in Collibra by looking up the full name. The Immuta data source name must match the Collibra table asset name for the table to successfully link.
Pull and apply tags in Immuta from Collibra: Immuta checks Collibra every 24 hours by observing the linked assets history for any relevant metadata changes. Based on these changes, Immuta then only polls and ingests objects from Collibra for the relevant data sources. However, if Immuta observes more than 25,000 metadata changes in Collibra within 24 hours, it will poll all data sources for tags, data classifications, and attributes during that run of external catalog tag synchronization.
The catalog auto-linking method will only auto-link Collibra assets where the asset's full name follows the Collibra Edge naming convention. Any assets following a different naming convention must be linked manually instead.
Columns must have a direct relation to their parent asset in Collibra. Indirect/inherited relations are not supported and will result in column tags and attributes not being ingested into Immuta.
The Microsoft Purview catalog integration with Immuta currently supports ingestion of Classifications and Managed attributes of type single or multiple choice as tags. Additionally, data source and column descriptions from the connected Microsoft Purview catalog will be pulled into Immuta.
Linking to data sources and columns in Microsoft Purview: Immuta links data sources to assets in Microsoft Purview by looking up the fully qualified name of an entity. The composition of the fully qualified name in Microsoft Purview differs depending on the technology type backing the data source.
Pull and apply tags in Immuta from Microsoft Purview: Immuta polls Microsoft Purview every 24 hours for all tags.
Standard tags from Purview do not get ingested into Immuta
The current implementation only supports Databricks Unity Catalog, Snowflake, and Azure Synapse Analytics data sources and their associated columns
If a managed attribute is applied to an Immuta data source but later expires, it will still appear as a tag on the data source. Expired attributes must be removed from the object in Purview for the tag to be removed from the Immuta data source.
If users have an unsupported catalog, or have customized their catalog integration, they can connect through the REST Catalog using the Immuta API.
For more details about using a custom REST catalog with Immuta, see the Custom REST catalog interface endpoints page.
The AWS Lake Formation integration can ingest Lake Formation Tags and apply them to Immuta data sources.
To learn more about AWS Lake Formation tag ingestion, see the AWS Lake Formation reference guide.
To configure tag ingestion, see the Register an AWS Lake Formation connection page.
Users can connect their Databricks Unity Catalog account to allow Immuta to ingest Databricks tags and apply them to Databricks data sources.
To learn more about Databricks Unity Catalog tag ingestion, see the Databricks Unity Catalog reference guide.
To configure tag ingestion, see the Register a Databricks Unity Catalog connection page.
Users can connect a Snowflake account to allow Immuta to ingest Snowflake tags onto Snowflake data sources.
To learn more about Snowflake tag ingestion, see the Snowflake reference guide.
To configure tag ingestion, see the Enable Snowflake tag ingestion page.
This table lists the supported external catalogs and their supported authentication methods. Data platform tag ingestion uses the data platform credentials. For more details about a catalog, see the linked section:
❌
✅
✅
Tags ingested from external catalogs cannot be edited within Immuta. To edit, delete, or add a tag from an external catalog to a data source or column, make the change in the external catalog.
You can configure multiple external catalogs within a single tenant of Immuta, but only one external catalog can be linked to a data source.
Immuta searches all external catalog providers once per day and links data sources without an external catalog attached to them to the first catalog that matches.
S3 data sources can currently only be to external catalogs.
The following catalog-related events are audited and can be found on the audit page in the UI:
ConfigurationUpdated: The configuration on the Immuta app settings page is updated, including when an external catalog configuration is added or deleted.
DatasourceCatalogSynced: An external catalog is linked and synced for the data source.
To configure an external catalog, see the Configuration how-to guide.
To learn more about how Immuta can automatically tag your data, see the Data discovery introduction.
Best practice: Use a single catalog; having more than one can lead to multiple truths and data leaks.
Alation tags and custom fields (except people sets, since those are represented as email addresses) containing values with a dot "." delimiter will be transformed into hierarchical tags in Immuta. To learn more about the benefits of hierarchical tags for policy authoring, see .
Private preview
The Atlan catalog integration is only available to select accounts. Contact your Immuta representative to enable this feature.
Collibra objects using the dot "." delimiter will be transformed into hierarchical tags in Immuta. To learn more about the benefits of hierarchical tags for policy authoring, see .
Private preview: This feature is available to select accounts. Contact your Immuta representative to enable this feature.
Private preview: This feature is only available to select accounts. Contact your Immuta representative to enable this feature.
Private preview: This feature is only available to select accounts. Contact your Immuta representative to enable this feature.
❌
❌
✅
✅
✅
❌
❌
✅
❌
Learn about how data identification discovers and tags your data
Identification uses data patterns to determine what type of data your column represents. Using identifiers within domains, Immuta evaluates your data and can assign the appropriate tags to your data source columns based on what it finds. This saves the time of identifying your data manually and provides the benefit of a standard taxonomy across all your data sources in Immuta.
Identification runs identifiers to discover data. These identifiers are grouped into domains with data sources. Each identifier contains a single criteria and the tags that will be applied when the criteria's conditions have been met.
There are two types of identifiers in Immuta:
Reference identifiers: This is a library of the identifiers that can be added to domains. When added to a domain, a copy of the reference identifier is made as the domain-specific identifier.
Immuta comes with to discover common categories of data. These cannot be modified or deleted.
Data governors can create their own reference identifiers for use within your organization.
Domain-specific identifiers: These identifiers only exist within a specific domain and are checked against the data sources in that domain when identification runs.
Users with the Manage Identifiers permission can create these identifiers or add them to a domain from a reference identifier.
If a domain-specific identifier was copied over from a reference identifier, there is no lineage and any edits to the reference identifier will not be reflected in the domain-specific copy.
Criteria are the conditions in an identifier that need to be met for resulting tags to be applied to data.
Competitive criteria analysis: This criteria is a process that will review all the regex and dictionary criteria within the identifiers of the domain and search for the identifier with the best fit. In this review, each competitive criteria analysis identifier in the domain competes against each other to find the best and most specific identifier that fits the data. The resulting tags for the best identifier are then applied to the column. Only one competitive criteria analysis identifier for each domain will apply per column. Competitive criteria identifiers, both built-in and custom, must match at least 90% of the data sampled. To learn more about the competitive nature, see the .
Regex: This criteria contains a case-insensitive regular expression (regex) that searches for matches against column values. Immuta only supports regular expressions written in RE2 syntax.
Create a new identifier in the or with the .
The way Immuta runs identification depends on your :
Dictionary and regex identifiers: To evaluate your data for matches to dictionary and regex identifiers, Immuta generates a SQL query using a domain's identifiers. The Immuta system account then executes that query in the remote technology (e.g., in Snowflake) to match any regex and dictionary identifiers. Immuta receives the query result which contains the column name and the matching identifiers but no raw data values.
Column name regex identifiers: To evaluate your data for matches to column name regex identifiers, Immuta does not need to query your remote technology. Instead, column name identifiers are matched with the column metadata within Immuta (i.e., the column names of your tables).
The results of these processes are then used to apply the resulting tags to the appropriate columns.
This evaluating and tagging process occurs when identification runs and happens automatically from the following event:
A new data source is added to a domain with identifiers (either manually or )
The following actions will also trigger identification:
Column detection is enabled, and new columns are detected on data sources within a domain with identifiers. Here, identification will only run on new columns, and no existing tags will be removed or changed.
. Note, this will use the identifiers that already applied to the data source.
.
Identification has varied support for from different technologies based on the identifier type.
When identification is manually triggered by a data owner, all column tags previously applied by identification are removed and the tags prescribed by the latest run are applied. However, if identification is triggered because a new column is detected by schema monitoring or object sync, tags will only be applied to the new column, and no tags will be modified on existing columns. Additionally, governors, data source owners, and data source experts can to prevent them from being used and auto-tagged on that data source in the future.
The amount of time it takes to run identification on a data source depends on several factors:
Columns: The time to run identification grows nearly linearly with the number of text columns in the data source.
Identifiers: The number of identifiers being used the time to run identification.
Row count: Performance of identification may vary depending on the sampling method used by each technology. For Snowflake, the number of rows has little impact on the time because data sampling has near-constant performance.
The time it takes to run identification for all newly onboarded data sources in Immuta is not limited by identification's performance but by the execution of background jobs in Immuta. when onboarding a large number of data sources to ensure the advanced settings are set appropriately for your organization.
For users interested in testing identification, note that the built-in identifiers by Immuta require a 90% match to data to be assigned to a column. This means that with synthetic data, there may be situations where the data is not real enough to fit the confidence needed to match identifiers. To test identification, use a dev environment and create copies of your tables.
The following identification-related events are and can be found on the :
: An identifier is created.
: An identifier is deleted.
: An identifier's criteria, description, name, or tag is updated.
Deleting the built-in Discovered tags is not recommended: If you do delete built-in Discovered tags and use the built-in identifiers without editing the tags, then when the identifier is matched the column will not be tagged. As an alternative, tags can be disabled on a , or identification won't run if you do not add identifiers to domains.
Regex patterns with nested wildcards are not recommended: When creating regexes for identifiers, it is best to avoid nested wildcards. They can be too complex and cause internal timeouts. As an alternative, break up the regexes into simpler patterns. Then after they are tagged, use a to group them under a parent tag.
*Two built-in patterns support and match based on additional data types:
DATE: Columns will match this identifier if they are string and the regex matches or if the data type is date, date+time, or timestamp.
TIME: Columns will match this identifier if they are string and the regex matches or if the data type is time. Note that if the date is included in the data, it will not match this identifier.
The size of the identification query for dictionary patterns, which are compiled into a regex and regex patterns, is limited by the backing technology:
For Snowflake, the .
For Starburst (Trino), the default query character limit is 1,000,000 characters. However, .
Immuta will start up a Databricks cluster to complete the identification job if one is not already running. This can cause unnecessary costs if the cluster becomes idle. Follow to automatically terminate inactive clusters after a set period of time.
The following Databricks Unity Catalog securable objects are supported with Immuta, but cannot be used with identification:
Volumes (external and managed)
The Redshift cluster must be up and running for identification to successfully run.
To use AWS access key authentication on a Redshift data source and have competitive criteria analysis identifiers supported,
The AWS access key used to register the data source must be able to do a minimum of the following :
redshift-data:BatchExecuteStatement
redshift-data:CancelStatement
Column name: This criteria includes a case-insensitive regular expression (regex) matched against column names, not against the values in the column. The identifier's tags will be applied to the column where the name is found. Multiple column name identifiers can match a column and be applied. Immuta only supports regular expressions written in RE2 syntax.
✅
✅
Amazon S3
❌
❌
✅
AWS Lake Formation
❌
❌
✅
Azure Synapse Analytics
❌
❌
✅
Databricks Lakebase
❌
❌
✅
Databricks Spark
✅
✅
✅
Databricks Unity Catalog
✅
✅
✅
Google BigQuery view-based
❌
❌
✅
Google BigQuery viewless
❌
❌
❌
MariaDB
❌
❌
✅
MySQL
❌
❌
✅
Oracle
❌
❌
✅
PostgreSQL
❌
❌
✅
Snowflake
✅
✅
✅
SQL Server
❌
❌
✅
Starburst (Trino)
✅
✅
✅
Teradata
❌
❌
✅
Views: Performance on views is limited by the performance of the query that defines the view. Running identification on complex views with large amounts of data is more likely to result in timeouts. Immuta recommends running identification on the underlying base tables.
actor.name.Immuta System Account and will include the related identifiers in the event as relatedResources.type.CLASSIFIERS.TagRemoved: A tag is removed from a data source or column. Tag events from identification will have actor.name.Immuta System Account and will include the related identifiers in the event as relatedResources.type.CLASSIFIERS.
Dictionary
Text string columns
Can be toggled in the identifier definition
Models
Functions
Using a large number of files to store the data in a table with a large number of rows may result in the Databricks planner scanning the entire table, resulting in a slow performing query.
redshift-data:DescribeStatement
redshift-data:ExecuteStatement
redshift-data:GetStatementResult
redshift-data:ListStatements
The AWS access key used to register the data source must have redshift:GetClusterCredentials for the cluster, user, and database that they onboard their data sources with.
If using a custom URL, then the data source registered with the AWS access key must have the region and clusterid included in the additional connection string options formatted like the following example:
Redshift Serverless data sources are not supported for competitive criteria analysis identifiers with the AWS access key authentication method.
Amazon Redshift
✅
✅
✅
Amazon Redshift Spectrum
Data regex*
Text string columns
Case-sensitive
Column name regex
Any column
Default 15-minute timeout
Identification queries will timeout after 15 minutes to avoid overconsumption of resources and reduce the cost of running identification. If your identification run was not completed because of this timeout, submit a support ticket to change the default setting.
✅
Not case-sensitive
region=us-east-2;clusterid=12345Understand the common data types built-in identifiers detect and tag
Immuta comes with a pack of built-in identifiers that look for common data types. These identifiers were written by Immuta's research and development team and cannot be deleted or edited by users. However, users can add these built-in identifiers to their own domains and edit the tags applied by them.
Identifiers must match at least 90% of the sampled data to be tagged, with three exceptions noted below. See the How competitive pattern analysis works guide for more information about sampling and thresholds.
ARGENTINA_DNI_NUMBER
Detects strings consistent with Argentina's National Identity (DNI) Number. Requires an eight-digit number with periods after the second and fifth digits.
Discovered.Country.Argentina
Discovered.Entity.DNI Number
AUSTRALIA_MEDICARE_NUMBER Improved
Detects numeric strings consistent with Australian Medicare Number. Requires a ten- or eleven-digit number. The starting digit must be between 2 and 6, inclusive. Spaces must be placed between the fourth and fifth and ninth and tenth digits. Optional eleventh digit separated by a / or a space. Examples
Discovered.Country.Australia
Discovered.Entity.Medicare Number
AUSTRALIA_PASSPORT Improved
Detects strings consistent with the Australian Passport number. A string of 8 or 9 characters is required, with a starting uppercase character (A, B, C, D, E, F, G, H, J, L, M, N, R, X, or U) or a two-character alphabetic prefix (P followed by A, B, C, D, E, F, U, W, X, or Z) followed by seven numeric digits. Examples
Discovered.Country.Australia
Discovered.Entity.Passport
BELGIUM_NATIONAL_ID_CARD_NUMBER
Detects numeric strings consistent with Belgium's National ID card. Requires a twelve-digit number with a required hyphen (-) between the third and fourth digits. Allows for an optional hyphen between the tenth and eleventh digits.
Discovered.Country.Belgium
Discovered.Entity.National ID Card Number
BELGIUM_NATIONAL_REGISTRATION_NUMBER New
Detects numeric strings consistent with Belgium's National Registration Number. Requires 11 characters in the form YY.MM.DD-NNN-XX, where YY.MM.DD corresponds to birth date, NNN is a number, and XX is a checksum digit. Example
Discovered.Country.Belgium
Discovered.Entity.National Registration Number
BITCOIN_INVOICE_ADDRESS
Detects strings consistent with the following Bitcoin Invoice Address formats: P2PKH, P2SH, and Bech32.
Discovered.Entity.CRYPTO
BRAZIL_CPF_NUMBER Improved
Detects a numeric string consistent with Brazil's CPF (Cadastro de Pessoas Físicas) number. An eleven-digit numeric string with optional non-numeric separators (., -, or space) after the third, sixth, and ninth digits. Examples
Discovered.Country.Brazil
Discovered.Entity.CPF Number
CANADA_BC_PHN
Detects numeric strings consistent with British Columbia's Personal Health Number (PHN). Requires a ten-digit numeric string with hyphens (-) or spaces after the fourth and seventh digits.
Discovered.Country.Canada
Discovered.Entity.British Columbia Health Network Number
CANADA_OHIP
Detects alphanumeric strings consistent with Ontario's Health Insurance Plan (OHIP). Requires a twelve-digit capitalized alphanumeric code. Optional hyphens (-) or spaces can appear after the fourth, seventh, and tenth digits.
Discovered.Country.Canada
Discovered.Entity.Ontario Health Insurance Number
CANADA_PASSPORT Improved
Detects strings consistent with the Canadian Passport Number format. Allows for two formats. One format requires two capital letters followed by six digits. The other format requires one letter, followed by six digits, and ends in two letters. Examples
Discovered.Country.Canada
Discovered.Entity.Passport
CANADA_QUEBEC_HIN
Detects alphanumeric strings consistent with Quebec's Health Insurance Number (HIN). Requires four alphabetic characters followed by an optional space or hyphen (-), and then eight digits with an optional hyphen or space after the fourth digit.
Discovered.Country.Canada
Discovered.Entity.Quebec Health Insurance Number
COUNTRY New
Detects strings consistent with the names of all countries in the world. This identifier is case-insensitive.
Discovered.Entity.Location
CREDIT_CARD_NUMBER Improved
Detects strings consistent with a credit card number with prefixes matching major credit card companies.
Discovered.Entity.Credit Card Number
DATE Improved
Detects strings consistent with dates in over 30 different formats or date type: date, date+time, or timestamp. This identifier is case-insensitive.
Discovered.Entity.Date
DOMAIN_NAME Improved
Detects strings that begin with a letter and are no more than 225 characters. A full domain can have one to four labels separated by a .. Each label can be one to 63 alphanumeric characters long. And each label after the first must be in the dictionary list of possible labels. This identifier is case-insensitive.
Discovered.Entity.Domain Name
EMAIL_ADDRESS
Detect strings consistent with an email address. Usernames are required to be fewer than 255 characters, follow by @, a domain of fewer than 255 characters, and a top level domain of between 2 and 20 characters.
Discovered.Entity.Electronic Mail Address
ETHNIC_GROUP
Detects strings consistent with the US Census race designations. This identifier allows for dashes to be used in place of spaces and is case-insensitive.
Discovered.Entity.Ethnic Group
FDA_CODE Improved
Detects a string consistent with a drug or ingredient registered with the Food and Drug Administration (FDA). Must start with between 4 to 5 digits, followed by a hyphen, followed by 3 to 4 digits, followed by a hyphen, and finishing with 1 to 2 digits.
Discovered.Country.US
Discovered.Entity.FDA Code
FINANCIAL_INSTITUTIONS New
Detects strings consistent with names of financial institutions based on lists provided by the FDIC and OCC, includes alternative names.
Discovered.Entity.Financial Institutions
FRANCE_NIR Improved
Detects numeric strings consistent with France's National ID number (Numéro d'Inscription au Répertoire). Requires a fifteen-digit numeric string. An optional hyphen (-) or space can appear after the 13th digit.
Discovered.Country.France
Discovered.Entity.NIR
FRANCE_PASSPORT
Detects alphanumeric strings consistent with the French Passport number. Requires two numbers followed by two uppercase letters and ends with five digits.
Discovered.Country.France
Discovered.Entity.Passport
GENDER Improved
Detects strings consistent with gender types and common abbreviations. This identifier is case-insensitive.
Discovered.Entity.Gender
GERMANY_DRIVERS_LICENSE_NUMBER
Detects alphanumeric strings consistent with Germany's driver's license number. Requires an eleven-element string of the format CDDCCCCCCDC where C is an uppercase Latin letter and D is a numeric digit.
Discovered.Country.Germany
Discovered.Entity.Drivers License Number
GREAT_BRITAIN_DRIVERS_LICENSE New
Detects alphanumeric strings consistent with the United Kingdom's driver's license number. Requires either a 16- or 18-character string. The first five characters represent the driver's surname, padded with 9s, followed by a single digit for decade of birth, two digits for month of birth (incremented by 50 for female drivers), two digits for day of birth, one digit for year of birth, two letters, an arbitrary digit, and two digits. Two additional digits can be present for each license issuance. Examples
Discovered.Country.UK
Discovered.Entity.Drivers License Number
IBAN_CODE
Detects strings consistent with an International Bank Account Number (IBAN). Requires a string in the form ZZ-DD-BBAN, where ZZ is a country code, DD is two numeric digits, and BBAN is a Basic Bank Account Number comprising two to seven groups of three to five uppercase alphanumeric characters, optionally separated by space or dash, and optionally followed by a final group of length one to three.
Discovered.Entity.IBAN Code
ICD10_CODE Improved
Detects strings consistent with codes from the International Statistical Classification of Diseases and Related Health Problems (ICD), as drawn from the Clinical Modification lexicon from the year 2025. This identifier is case-insensitive.
Discovered.Entity.ICD10 Code
ICD_10_PCS New
Detects strings consistent with procedure codes from the International Statistical Classification of Diseases and Related Health Problems (ICD), as drawn from the Clinical Modification lexicon from 2020. Example
Discovered.Entity.ICD10 Procedure Code
IMEI_HARDWARE_ID Improved
Detects strings consistent with an International Mobile Equipment Identity (IMEI) number. Must contain 15 or 16 digits with optional hyphens or spaces after the 2nd, 8th, and 14th digits. Examples
Discovered.Entity.IMEI
IP_ADDRESS
Detects IP Addresses in the V4 and V6 formats. This identifier is case-insensitive.
Discovered.Entity.IP Address
LOCATION
Detects ISO3166 formatted locations. This identifier must match at least 80% of the data sampled.
Discovered.Entity.Location
MAC_ADDRESS Improved
Detects strings consistent with a Media Access Control (MAC) address. Must contain twelve hexadecimal digits, with every two digits separated by a colon or hyphen. Examples
Discovered.Entity.MAC Address
NAICS_CODE New
Detects strings consistent with North American Industry Classification System (NAICS). A two-digit number represents a basic sector and each preceding digit represents a more specific sub sector with a maximum of six digits. Examples
Discovered.Entity.NAICS Code
PERSON_NAME Improved
Detects strings consistent with a dictionary of people's names. The name dictionary is US-centric with person names drawn from the US Social Security database, covering 80% of the U.S. population. This identifier must match at least 45% of the data sampled. This identifier is case-insensitive.
Discovered.Entity.Person Name
PHONE_NUMBER Improved
Detects strings consistent with telephone numbers. Primarily looks for strings consistent with the United States telephone numbers naming convention. Optional area codes allowed.
Discovered.Entity.Telephone Number
POSTAL_CODE Improved
Detects strings consistent with a valid US Zip code with an optional +4 separated by a dash. Only valid five-digit zip codes are detected. This identifier is case-insensitive.
Discovered.Entity.Postal Code
SEC_STOCK_TICKER New
Detects strings consistent with the stock tickers recognized by the U.S. Securities and Exchange Commission (SEC).
Discovered.Entity.Stock Ticker Symbol
SPAIN_NIF_NUMBER Improved
Detects strings consistent with Spain's Tax Identification number. Requires a string with nine alphanumeric characters. Requires either eight digits followed by an optional hyphen or space and a single uppercase letter or the initial character must be X, Y, or Z, followed by an optional dash or space, seven numeric digits, followed by an optional dash or space, and finally, by a single uppercase letter. Examples
Discovered.Country.Spain
Discovered.Entity.NIF Number
SPAIN_PASSPORT
Detects string consistent with Spain's Passport Number. Requires a eight- or nine-character string starting with either two or three uppercase letters followed by six numeric digits.
Discovered.Country.Spain
Discovered.Entity.Passport
SWIFT_CODE
Detects alphanumeric strings consistent with a SWIFT code (or Bank Identifier Code (BIC)) format. Requires values consistent with AAAAAACCDDD, where A is an uppercase letter, C is an uppercase letter or numeric digit, and DDD is an optional three-character sequence of uppercase letters or numeric digits.
Discovered.Entity.Swift Code
TIME Improved
Detects strings consistent with times in various formats or data type: time. If date is included in the time, it will not match. Use the DATE identifier instead.
Discovered.Entity.Date
UK_NATIONAL_INSURANCE_NUMBER Improved
Detects alphanumeric strings consistent with the United Kingdom's National Insurance Number. Requires a nine-character string. The first two digits must be uppercase letters, followed by an optional space, then six digits with optional spaces or hyphens (-) every two digits, ending with A, B, C, or D.
Discovered.Country.UK
Discovered.Entity.National Insurance Number
URL Improved
Detects string consistent with a URL. String must begin with a common schema, followed a string and ending with a top level domain of no more than 128 alphanumeric characters.
Discovered.Entity.URL
US_DEA_NUMBER
Detects alphanumeric strings consistent a Drug Enforcement Administration (DEA) number is assigned to a health care provider. It must have a length of nine characters. The first two digits must be uppercase alphanumeric characters, and the last seven characters are numeric digits. The first character may not be I, N, O, Q, V, W, Y, or Z.
Discovered.Country.US
Discovered.Entity.DEA Number
US_EMPLOYER_IDENTIFICATION_NUMBER
Detects numeric string consistent United States Employer Identification Number (EIN). Strings must contain nine digits with a hyphen after the second digit.
Discovered.Country.US
Discovered.Entity.Employer ID Number
US_HEALTHCARE_NPI Improved
Detects 10-digit numeric strings consistent with US National Provider Identifier (NPI). It must either start with 80840 followed by a 1 or 2, or it must begin with a 1 or 2.
Discovered.Country.US
Discovered.Entity.Healthcare NPI
US_PERSON_FULL_NAME New
Detects strings consistent with a person's {first name} space {last name}. Uses the same names from the PERSON_NAME identifier. This identifier must match at least 20% of the data sampled and is case-insensitive.
Discovered.Entity.Person Name
US_PREPARER_TAXPAYER_IDENTIFICATION_NUMBER
Detects strings consistent with a Preparer Taxpayer ID number. Strings must have nine characters, starting with a P that is followed by eight digits.
Discovered.Country.US
Discovered.Entity.Preparer Taxpayer ID Number
US_SOCIAL_SECURITY_NUMBER Improved
Detects strings consistent with a US Social Security Number. Strings must contain nine digits and comprise three parts: the three left-most digits designating the area number, the middle two digits designating the group number, and the four right-most digits designating the serial number. For a column to be tagged, none of these parts can contain all zeroes, and area numbers must not be 666 or in the range of 900-999. Examples
Discovered.Country.US
Discovered.Entity.Social Security Number
US_STATE Improved
Detects strings consistent with either a full name or two-letter abbreviation of a US state or territory.
Discovered.Country.US
Discovered.Entity.State
US_STREET_ADDRESS New
Detects strings consistent with U.S. street addresses. Requires the street naming convention of {address_number} {street_name} {unit number (optional)} with an optional road suffix after the street name. The maximum length for street name is 20 alphanumeric characters. This identifier must match at least 80% of the data sampled and is case-insensitive.
Discovered.Entity.Location
VEHICLE_IDENTIFICATION_NUMBER
Detects strings consistent with Vehicle Identification Numbers. A valid World Manufacturer Identifier is required.
Discovered.Country.US
Discovered.Entity.Vehicle Identifier or Serial Number