This page outlines how to connect an external catalog on the Immuta app settings page. For details on external catalogs in Immuta, see the External catalog reference guide.

Link an Alation catalog

Requirements:

• APPLICATION_ADMIN Immuta permission

• An Alation API access token connected to a user with the Server Admin permission

1. Navigate to the App Settings page.

2. Scroll to 2 External Catalogs, and click Add Catalog.

3. Enter a Display Name and select Alation from the dropdown menu.

4. Complete the URL and API key fields. The API key must be an API access token for your Alation instance connected to a user with the Server Admin permission.

5. Configure whether or not Alation tags and custom fields are imported as Immuta tags:

   • Link Alation tags: When selected, Immuta imports Alation tags as Immuta tags.

   • Link Alation Custom Fields: When selected, Immuta imports Alation custom fields as Immuta tags. Follow the Alation documentation to create an Alation custom field, add permissions to your custom field, and apply custom fields to tables and columns.

6. Opt to select Upload Certificates.

   1. Upload the Certificate Authority, Certificate File, and Key File.

   2. Opt to enable Strict SSL by selecting the checkbox.

7. Click the Test Connection button.

8. Once the connection is successful, click Save.

Link a Collibra catalog

Requirement: APPLICATION_ADMIN Immuta permission

1. Navigate to the App Settings page.

2. Scroll to 2 External Catalogs, and click Add Catalog.

3. Enter the Display Name and select Collibra from the dropdown menu.

4. Enter the HTTP endpoint of the catalog in the URL field.

5. Complete the Username and Password fields. Note: This is the username and the password that Immuta can use to connect to the external catalog.

6. Complete the Asset Mappings modal to set which Collibra asset types align to the Immuta data source and column. Immuta will only link data sources from the asset types you specify.

7. Complete the Attributes as Tags modal to specify which Collibra attributes you want in Immuta. These attributes will come in as parent tags with their values as children tags.

8. Opt to select Upload Certificates.

   1. Upload the Certificate Authority, Certificate File, and Key File.

   2. Opt to enable Strict SSL by selecting the checkbox.

9. Click the Test Connection button.

10. Once the connection is successful, click Save.

Link a Microsoft Purview external catalog

Requirement: APPLICATION_ADMIN Immuta permission

Prerequisite

Register an app in the Azure portal with the with the following settings:

• Supported account type: "Accounts in this organizational directory only"

• Microsoft-Graph: User.Read API permission

• A client secret

Using that registered app, navigate to Immuta and complete the following:

1. Navigate to the App Settings page.

2. Scroll to 2 External Catalogs, and click Add Catalog.

3. Enter the Display Name and select Microsoft Purview from the dropdown menu.

4. Complete the following fields:

   1. Enter the Microsoft Purview endpoint URL including the Azure Account Name, like https://<ACCOUNTNAME>.purview.azure.com, in the Purview Endpoint URL field.

   2. Complete the Microsoft Entra Directory (tenant) ID and Microsoft Entra (client) ID fields.

   3. Enter the Microsoft Entra Application Client Secret ID for Immuta to authenticate and connect to the Purview API. The secret cannot be expired.

5. Click the Test Connection button.

6. Once the test is successful, click Save.

Link a custom REST catalog

Requirement: APPLICATION_ADMIN Immuta permission

Integrating a custom REST catalog service with Immuta requires implementing a REST interface. For details about the necessary endpoints that must be serviced, see the Custom REST catalog interface endpoints page.

1. Navigate to the App Settings page.

2. Scroll to 2 External Catalogs, and click Add Catalog.

3. Enter the Display Name and select Rest from the dropdown menu.

4. Select the Internal Plugin checkbox if the catalog has been uploaded to Immuta as a custom server plugin.

5. Complete the following fields:

   1. Enter the HTTP endpoint of the catalog in the URL field.

   2. Complete the Username and Password fields.

   3. Enter the path of the Tags Endpoint.

   4. Enter the path of the Data Source Endpoint.

   5. Enter the path to the information page for a data source in the Data Source Link Template field.

6. Opt to enter the path to the information page for a column in the Column Link Template field.

7. Opt to upload a Catalog Image.

8. Opt to select Upload Certificates.

   1. Upload the Certificate Authority, Certificate File, and Key File.

   2. Opt to enable Strict SSL by selecting the checkbox.

9. Click the Test Connection button.

10. Click the Test Data Source Link.

11. Once both tests are successful, click Save.

Enable Snowflake tag ingestion

See the Configure a Snowflake integration page for guidance on configuring tag ingestion.

If Snowflake data sources existed before configuring tag ingestion, Immuta will automatically sync those data sources to the catalog and apply tags to them. Immuta will automatically check the external catalog for changes and sync data sources to the catalog every 24 hours.

Enable Databricks Unity Catalog tag ingestion

See the Configure a Databricks Unity Catalog integration page for guidance on configuring tag ingestion.

If Databricks Unity Catalog data sources existed before configuring tag ingestion, Immuta will automatically sync those data sources to the catalog and apply tags to them. Immuta will automatically check the external catalog for changes and sync data sources to the catalog every 24 hours.

Manually link catalogs to data sources

You can manually link and remove external catalogs from data sources on the data source details tab.

1. Navigate to your data source.

2. In the connection information section, click the Link Catalog icon (or Unlink Catalog to remove an external catalog from a data source).

3. Select your external catalog from the dropdown menu.

4. Click Link to confirm.

Manually sync external catalog tags

1. Navigate to your data source and click the data source Health dropdown menu.

2. Click Re-run in the External Catalog section.

Connect an external catalog to use tagging capabilities outside of Immuta and pull tags from external table schemas. Once the catalog has been connected, Immuta ingests a data dictionary from the catalog and applies data source and column tags directly to the data source. These tags can then be used to create policies.

Getting started

This getting started guide outlines how to use external catalogs in Immuta.

How-to guide

Configure an external catalog: Configure Alation, Collibra, or a custom REST catalog to ingest tags into Immuta.

Reference guides

• External catalog integrations: This reference guide describes the requirements of the external catalogs Immuta supports.

• Custom REST catalog introduction: This reference guide describes the custom catalog option for users to make API calls to retrieve metadata on their data.

• Custom REST catalog interface endpoints: This reference guide describes the endpoints for configuring a custom REST catalog.

The how-to guides linked on this page illustrate how to link an external catalog with Immuta to ingest tags.

Link an Alation external catalog

Requirement: A catalog with tags that correspond to your Immuta data sources

Configuration process

1. Connect your Alation catalog to Immuta.

2. When changes are made to the external catalog, refresh external tags.

Link a Collibra external catalog

Requirements:

• A physical data dictionary with assets that correspond to your Immuta data sources

• The Collibra global role Catalog or Catalog Author

Configuration process

1. Connect your Collibra catalog to Immuta.

2. When changes are made to the external catalog, refresh external tags.

Link a Microsoft Purview external catalog

Requirements:

• A catalog with assets that correspond to your Immuta data sources

• The ability to create a registered app in the Azure portal

Configuration process

1. Connect your Microsoft Purview catalog to Immuta.

2. When changes are made to the external catalog, refresh external tags.

Link a custom REST catalog

Requirements:

• A catalog with tags that correspond to your Immuta data sources

• Authenticate with the Immuta API

Configuration process

1. Connect your custom REST Catalog to Immuta.

2. When changes are made to the external catalog, refresh external tags.

Link Databricks Unity Catalog for tag ingestion

Requirements:

• Fewer than 2,500 Databricks Unity Catalog data sources registered in Immuta

• Databricks privileges listed on the Configure a Databricks Unity Catalog integration page

Configuration process

1. Configure Databricks Unity Catalog integration.

2. Configure Databricks Unity tag ingestion in Immuta.

3. Register Databricks data sources in Immuta.

Once you register data sources, table and column tags from Databricks Unity Catalog will be ingested and applied to the corresponding data sources in Immuta.

Link Snowflake for tag ingestion

Requirements:

• A Snowflake user who can set the following permissions:

  1. GRANT IMPORTED PRIVILEGES ON DATABASE snowflake

  2. GRANT APPLY TAG ON ACCOUNT

• Snowflake Enterprise Edition or higher

Configuration process

1. Configure Snowflake tag ingestion in Immuta.

2. When changes are made to the tags in Snowflake, refresh external tags

Immuta allows you to automate discovering and tagging data across your data platform. Tagging is critical for two reasons:

• It allows you to define data sensitivity, which in turn allows you to monitor where you have potential data security issues and gaps in your security posture.

• It allows you to abstract your physical structure from your access policy logic. For example, you can build access policies like mask all columns tagged Person Name (where Person Name was auto-tagged by Discover) rather than much less scalable policies that must be knowledgeable of your physical layers like mask column x in database y in data platform z.

Challenge and goals

Today’s sensitive data discovery tools give you a shallow overview of your data corpus across a long list of platforms. They give you pointers on where you have sensitive data without the granularity to drive your column- or row-level access controls. They help you understand what data you possess according to a regulatory framework, like HIPAA or PCI but without the details needed to automate your audits or compliance reporting. Knowing that you need to drive east to west on a road map from New York to California is helpful but ultimately insufficient to get you from a specific location to another.

Existing tools promise a high degree of automation, yet their many false positives result in painful manual work that never stops. Although data gets scanned automatically, performance breaks down at scale, or you manually need to fine-tune the computing resources of the scanners. Last but not least, your security team objects to the agent-based processing that requires taking data out of your data platform, and the associated data residency concerns may give you pause.

At Immuta, we believe that data security should not be painful. We believe that you can innovate and move quickly, while at the same time protecting your data and adhering to your internal policies and external regulations. Technology and automation allow you to make the right trade-off decisions quickly. It all starts with highly accurate and actionable metadata. If you trust your metadata and if it’s actionable, you can leverage it to automatically grant access to data, mask sensitive information, and automate your audit reporting.

Immuta Discover was built to tackle those challenges and address them through a unique architecture that was designed in collaboration with the largest financial institutions, healthcare companies, and government agencies in the world. The cloud and AI paradigm requires a fundamentally different approach. You must assume that your data is dynamic, unique, and collected in a multitude of different geographies and legal jurisdictions. Immuta Discover is built for this new world and its specific demands.

How does it work?

Scalability through in-platform processing

Identifying and classifying data requires analyzing and looking at the data - there’s no way around it. Immuta Discover does all the analysis and processing inside the remote technology. It takes advantage of those platforms’ inherent scalability to enable you to analyze large amounts of data quickly, efficiently, and without the need for separate resource optimization for containers or virtual machines.

Data residency compliance by design

By processing data directly inside the data platform, Immuta Discover automatically adheres to data residency and locality requirements. If you run your data warehouse or lake globally - across North America, the European Union, and Asia - Immuta processes the data in the region where your data is stored. No data ever leaves the data platform, and it will never move across different cloud regions.

Improved security and simplicity through agentless scanning

In-platform processing greatly reduces risk and improves your data security posture. Provisioning agents, whether they’re in a container, virtual machine, or Amazon Machine Image (AMI), create complexity and an unnecessary security risk. Not only can those agents become compromised, but their misconfiguration might lead to data leaks to other parts of your cloud infrastructure. An agentless approach can better leverage data platform optimizations to process data instead of transferring it out to re-optimize and analyze. This simplifies operations and increases efficiency for your infrastructure teams.

Cross-platform consistency

The advantages of in-platform processing are abundant, but implementing it across a multitude of platforms is challenging. Immuta helps bypass the obstacles by doing all the heavy lifting for you and building in specific implementations for each technology. Although all those implementations are ultimately different, Immuta abstracts the results to one standardized taxonomy, so you can have consistently accurate and granular metadata across all your data stores.

Granular query-level classification

Immuta Discover classifies data on a column level and instantaneously identifies schema changes. Only with that level of granularity and automation can you adhere to your audit requirements and understand what actions have been taken on your data. For example, if non-sensitive data is joined with sensitive data at query time, Immuta Discover will monitor and record that for your review. Continuous schema monitoring ensures schema changes never result in holes in your access controls and data security posture.

Highly accurate and actionable metadata

Trust in your metadata is critical for data security.

To unblock your data consumers, you need to automate your data access controls; this requires trusting that your classification and metadata are accurate and actionable. Immuta Discover provides you with highly accurate metadata and tags out-of-the-box and assists you in fine-tuning the classification mechanism to deal with false positives quickly. That enables you to build policies that dynamically grant or restrict access to protected data (like PHI or PII) depending on who is accessing it and what protections you want to apply.

Components of Discover

Immuta Discover works in three phases: identification, categorization, and classification.

1. Identification: In this first phase, data is identified by its kind – for example, a name or an age. This identification can be manually performed, externally provided by a catalog, or automatically determined by Immuta Discover through column-level analysis of patterns.

2. Categorization: In the second phase, data is categorized in the context of where it appears, subject to your active frameworks. For example, a record occurring in a clinical context containing both a name and individual health data is protected health information (PHI) under HIPAA.

   This categorization of data helps to understand the context it is in, including information like whether or not a record pertains to an individual, the composition and kinds of identifiers present, the data subject, whether the data belongs to any controlled data categories under certain legislation, etc.

3. Classification: In the third and final phase, data is classified according to its sensitivity level (e.g., Customer Financial Data is Highly Sensitive) and the risk associated to the data subject. Detect dashboards support 3 sensitivity levels. However, customers are free to customize the sensitivity names for the tags as needed.

Requirements:

• Immuta permission GOVERNANCE

• Registered data sources; see the reference page for supported technologies

This how-to guide is for enabling sensitive data discovery (SDD) for the first time. For additional information on sensitive data discovery, see the Data discovery page.

Turn on SDD

Requirement: Immuta permission APPLICATION_ADMIN

1. Navigate to the App Settings page and scroll to the Sensitive Data Discovery section.

2. Select the Enable Sensitive Data Discovery (SDD) checkbox to enable SDD.

3. Click Save and then click Confirm to apply your changes. Note that the Immuta tenant will have a system restart.

Note that the global framework is not set by default, so SDD will not run automatically on any data sources. Set a global framework to have identification automatically run on all new data sources.

Configure the global framework

Requirement: Immuta permission APPLICATION_ADMIN

1. Navigate to the App Settings page and scroll to the Sensitive Data Discovery section.

2. Enter the request-friendly name of your global identification framework in the Global SDD Template Name field. This name can be found in the URL when you navigate to the identification framework's page.

3. Click Save, and then Confirm your changes.

Create a new framework with identifiers

Once SDD is enabled on your tenant, SDD will automatically run when new data sources are added, but it must be manually run for all existing data sources. This allows you to test out SDD with a select few data sources without worrying that it will add tags throughout all your data sources.

For this step, you will pick the identifiers to match the data that matters to your organization. For example, for international data, you may want to enable many different identifiers for many countries, like the "Australia Passport" identifier and the "Finland National ID Number" identifier. However, if you are dealing with United States domestic financial data, those identifiers would be irrelevant. In that case, it would be better to identify the data likely to appear, like Bitcoin or US Bank Routing MICR.

First, create an empty framework,

1. Navigate to Discover and Identification.

2. Select Create New.

3. Enter a Name and Description for your new identification framework.

4. Select Create empty framework.

Then, add a new identifier to that framework,

1. Navigate to Discover and Identifiers.

2. Use the checkboxes to select all the identifiers relevant to your data. Tip: From the overview page you can see the name and the tags that will be applied by the identifier. To better understand the data it will match, click the name to read the description.

3. Once you have checked the identifiers you want in your framework, click Add to Framework.

4. Type the framework name in the text box.

5. Click Add to Framework.

Run identification on your data sources

Once you have created a framework relevant to your data, it is time to test it on your data and customize it. Run identification on a select number of data sources where you understand the data to assess and adjust the tags to reflect what you expect to see.

Add those select data sources to your new framework,

1. Navigate to Discover and Identification.

2. Click your new framework name.

3. Navigate to the Data Sources tab.

4. Click Add Data Sources.

5. Check the checkboxes for the select data sources you want to try SDD on.

6. Click Add Data Source(s).

Then, run identification on those data sources,

1. Navigate to Discover and Identification.

2. Click the action menu for your new framework.

3. Click Run Identification.

View the identification results

After identification runs, you will receive a notification that the job is complete. Then, you can view the results from the data source dictionary.

1. Navigate to the data source overview page of the data source you added to the framework.

2. Click the Data Dictionary tab.

3. Assess whether the Discovered tags are applied as expected.

4. If you are happy with the Discovered tags, follow the Assign data sources to frameworks guide to add the rest of your data sources to the framework and follow the Run identification guide to run identification on all your data sources.

5. If you want additional tags, follow the Create an identifier guide to create identifiers that matter to your data.

Identifiers in domains allows you to use the same domains you already organize your data in to hold identifiers and run sensitive data discovery (SDD) without having to use identification frameworks. See the Identifiers in domains guide for more information about the feature and limitations.

Prerequisites

Identifiers can be added and SDD can be run in any of your current domains. However, if you are not already using domains, set up a domain specifically to run SDD:

1. Enable SDD.

2. Register your data sources.

3. Create a domain.

4. Grant the appropriate users the Manage Identifiers permission for the domain.

Add identifiers to a domain

1. Navigate to the Identifiers tab of your domain.

2. Click Get Started.

3. Add reference identifiers to your domain that are relevant to your data by clicking the checkboxes. Note: When added to your domain, the identifier is a point-in-time copy of the reference identifier. It has the same name, pattern, and tags.

4. Click Add Identifiers.

Create a new identifier

This can be done within a domain from the Identifiers tab to create a domain-specific identifier, or it can be done from the Discover Identifiers page to create a reference identifier.

1. Click Create New.

2. Enter a name and description for your identifier.

3. Click Next.

4. Enter criteria: Select the Type of criteria.

   1. For regex, enter a regex to be matched against column values. The default criteria encoding is case-sensitive. You can change this encoding using the regex criteria. The regex must use RE2.

   2. For column name regex, enter a regex to be matched against column names. The default criteria encoding is case-insensitive. You can change this encoding using the regex criteria. The regex must use RE2 syntax.

   3. For a dictionary, enter the values in a comma-separated list to match against column values. Opt to toggle the Case insensitive switch to on if you want the dictionary to be case sensitive.

5. Click Next.

6. Select the tags to apply: Use the text box to search for a tag under the "Discovered" hierarchy or type a tag name to create a new tag under the "Discovered.Entity" hierarchy to apply to columns that match your identifier.

7. Click Next to review your new identifier and click Create Identifier to create it.

Immuta can scan your data sources and apply relevant tags when data is recognized. This eliminates a manual tagging process for your data, saving you time and providing standard taxonomy across all your data sources.

Requirements

• Registered data sources

• Immuta permission GOVERNANCE

Implement SDD

Sensitive data discovery (SDD) is an Immuta Discover feature that identifies your data sources and applies relevant tags when data is recognized. This eliminates a manual tagging process for your data, saving you time and providing standard taxonomy across all your data sources.

To learn more, see the Data discovery page.

Enable sensitive data discovery

Enable sensitive data discovery on your tenant. Opt to have SDD run automatically for new data sources by setting a global framework, or run SDD granularly by applying data sources to specific frameworks.

Configure SDD for your data

For additional control, create your own identifiers to recognize the data that matters to you. Add these identifiers to new frameworks and specify the data sources that need this framework. This fine-level control creates automatic tagging that is relevant and accurate to your data, requiring fewer manual adjustments to the resulting tags.

1. Customize SDD for your data:

   1. Create identifiers

   2. Create a framework

2. Assign your framework to specific data sources

Adjust Discovered tags

If you have any tags that are applied to your data sources by SDD that you don't want, you can easily disable these tags for each data source. This ensures that they will not be applied to the data source again if identification is re-run.

1. Verify tags

2. Disable Discovered tags

Reference pages:

Immuta comes with a default framework containing built-in Discovered tags and built-in identifiers. These identifiers and tags can be used in your own frameworks.

1. Built-in Discovered tags

2. Built-in identifiers

Implement classification

Classification is an Immuta Discover feature that categorizes your data based on the content and the associated risk the data poses. This increases your understanding of your data and allows you to make faster decisions about it.

Configure classification for your data

To create or manage a framework using the Immuta API, see the Frameworks API reference page.

Adjust classification tags

If you have any tags that are applied to your data sources by classification that you don't want, you can easily disable these tags for each data source. This ensures that they will not be applied to the data source again when classification is re-run.

1. Assess your data source tags.

2. Tune your data dictionaries.

Immuta uses tags primarily to enforce policies, but tags can also be used for generating Immuta reports and search results in the Immuta UI.

Connect external catalogs

Integrate your existing data catalog with Immuta.

Data discovery

Automatically discover and tag data based on its content (like column names).

Data classification

Automatically tag data based on its sensitivity and the associated risk level.

Manage tags

Create and manage tags in Immuta.

Requirements:

• Sensitive data discovery (SDD) enabled

• Immuta permission GOVERNANCE

Create an identification framework

Create an identification framework with no identifiers

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Click Create New.

3. Enter a Name and Description for the identification framework.

4. Select the option to Create empty framework.

5. Click Create.

After you create the identification framework, you can create new identifiers.

Copy an existing identification framework and its identifiers

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Click Create New.

3. Enter a Name and Description for the identification framework.

4. Select the option to Create identifiers from an existing framework.

5. Select the checkbox for the framework you want to copy. You can only copy a single framework. For more information about a framework, click the framework name to open a new tab with details about the framework.

6. Click Create.

Manage an identification framework's identifiers

Add an identifier to a framework

To add an identifier to a framework,

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Select the framework name for the identification framework you want to edit.

3. Click Add Identifier.

4. Choose in the dropdown to add an identifier from those already in Immuta or create a new identifier for the framework.

   • For existing identifiers: Opt to edit the tags. Then click Add Identifier.

   • For new identifiers:

     1. Fill out a Name and Description.

     2. Enter criteria: Select the Type of criteria.

        1. For regex, enter a regex to be matched against column values. The default criteria encoding is case-sensitive. You can change this encoding using the regex criteria. The regex must use RE2.

        2. For column name regex, enter a regex to be matched against column names. The default criteria encoding is not case-sensitive. You can change this encoding using the regex criteria. The regex must use RE2 syntax.

        3. For a dictionary, enter the values in a comma-separated list to match against column values. Opt to toggle the Case insensitive switch to on if you want the dictionary to be case sensitive.

     3. Select the tags to apply: Use the text box to search for a tag under the "Discovered" hierarchy or type a tag name to create a new tag under the "Discovered" hierarchy to apply to columns that match your identifier.

     4. Click Next to review your new identifier and click Create Identifier to create it.

Edit an identifier in a framework

Only tags can be edited within a framework. Edits made to an identifier within a framework will only impact that specific identifier. To fully edit an identifier (including the name, description, or criteria) for all frameworks, use the Edit an identifier how-to guide.

To edit the tags applied by an identifier for a framework,

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Select the framework name for the identification framework you want to edit.

3. Click the more actions icon for an identifier and select Edit tags.

4. Remove the tags or type a tag name to add tags.

5. Click Save.

Delete an identifier from a framework

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Select the framework name for the identification framework you want to edit.

3. Click the more actions icon for an identifier and select Delete.

4. Click Delete again in the modal.

Manage an identification framework's data sources

Assign an identification framework to data sources

To assign a framework to run on specific data sources,

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Select the framework you want to assign and navigate to the Data Sources tab.

3. Click Add Data Sources.

4. Select the checkbox for the data source you want this framework to run on. You may select more than one.

5. Click Add Data Source(s).

Remove data sources from an identification framework

After a data source is removed from a framework, it will use the global framework for any SDD scans and the tags applied by the removed framework will be replaced. The global framework is signified by the globe icon.

To remove data sources from a framework,

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Select the framework you want to remove data sources from and navigate to the Data Sources tab.

3. Select the checkbox for the data source you want to remove from the framework. You may select more than one.

4. Select Remove and click Remove again in the modal.

Delete an identification framework

Requirement: No data sources assigned to the framework

To delete a framework,

1. Click the Discover icon in the navigation menu and select the Identification tab.

2. Click the more actions icon in the Action column for the framework you want to delete. The global framework cannot be deleted. If you want to delete it, configure a different framework as the global framework.

3. Select Delete and click Delete again in the modal.

Sensitive data discovery (SDD) runs identifiers to discover data. These identifiers are grouped into domains with data sources. Each identifier contains a single criteria and the tags that will be applied when the criteria's conditions have been met.

Identifier

There are two types of identifiers in Immuta:

1. Reference identifiers: These identifiers are a library of the identifiers that can be added to domains. When added to a domain, reference identifiers are copied over and become domain-specific identifiers.

   1. Immuta comes with built-in identifiers to discover common categories of data. These cannot be modified or deleted.

   2. Data governors can create their own reference identifiers for use within your organization.

2. Domain-specific identifiers: These identifiers only exist within a specific domain and are checked against the data sources in that domain when SDD runs.

   1. Users with the Manage Identifiers permission can create these identifiers or add them to a domain from a reference identifier.

   2. If a domain-specific identifier was copied over from a reference identifier, there is no lineage and any edits to the reference identifier will not be reflected in the domain-specific copy.

Criteria

Criteria are the conditions in an identifier that need to be met for resulting tags to be applied to data.

SDD only supports regular expressions (regex) written in RE2 syntax.

Supported criteria types for identifiers

• Competitive criteria analysis: This criteria is a process that will review all the regex and dictionary criteria within the identifiers of the domain and search for the identifier with the best fit. In this review, each competitive criteria analysis identifier in the domain competes against each other to find the best and most specific identifier that fits the data. The resulting tags for the best identifier are then applied to the column. Only one competitive criteria analysis identifier for each domain will apply per column. Competitive criteria identifiers, both built-in and custom, must match at least 90% of the data sampled. To learn more about the competitive nature, see the How competitive criteria analysis works guide.

  • Regex: This criteria contains a case-insensitive regular expression that searches for matches against column values.

  • Dictionary: This criteria contains a list of words and phrases to match against column values.

• Column name: This criteria includes a case-insensitive regular expression matched against column names, not against the values in the column. The identifier's tags will be applied to the column where the name is found. Multiple column name identifiers can match a column and be applied.

Create a new identifier in the Immuta UI or with the sdd/classifier endpoint.

What has changed with SDD when using this new feature?

If you used SDD prior to this feature release in January 2025, there are some differences:

1. There are now two types of identifiers:

   1. Reference identifiers

   2. Domain identifiers

   See information about these in the Identifiers section.

2. There is a new permission to manage identifiers within domains: Manage Identifiers. The permission allows you to do the following:

   1. Create an identifier within your domain

   2. View the reference identifiers in Immuta

   3. Add, edit, and delete identifiers within your domain

3. The following have been removed:

   1. Identification frameworks: Previously, all identifiers had to be contained within a framework and that framework had to be assigned to a data source to run. Now, identifiers are added to domains with data sources.

   2. Global framework: Previously, a global framework could be set to run SDD automatically on all new data sources. This behavior cannot be achieved with identifiers in domains.

Limitations

See the table below for information on when SDD runs with the SDD feature before vs after with identifiers in domains.

Requirements:

• Sensitive data discovery (SDD) enabled

• Immuta permission GOVERNANCE

• Registered data sources; see the reference page for supported technologies

Create an identifier

1. Click the Discover icon in the navigation menu and select the Identifiers tab.

2. Click Create New.

3. Enter a Name and Description for the new identifier.

4. Enter criteria: Select the Type of criteria.

   1. For regex, enter a regex to be matched against column values. The default criteria encoding is case-sensitive. You can change this encoding using the regex criteria. The regex must use RE2. These identifiers are only supported on Snowflake, Databricks, Starburst (Trino), and Redshift data sources.

   2. For a dictionary, enter the values in a comma-separated list to match against column values. Opt to toggle the Case insensitive switch to on if you want the dictionary to be case sensitive. These identifiers are only supported on Snowflake, Databricks, Starburst (Trino), and Redshift data sources.

   3. For column name regex, enter a regex to be matched against column names. The default criteria encoding is case-insensitive. You can change this encoding using the regex criteria. The regex must use RE2 syntax.

5. Select the tags to apply: Use the text box to search for a tag under the "Discovered" hierarchy or type a tag name to create a new tag under the "Discovered" hierarchy to apply to columns that match your identifier.

6. Click Next to review your new identifier and click Create Identifier to create it.

7. See the Manage identification frameworks page to add your new identifier to a framework.

Note that all user-created identifiers must be a 90% match or greater for the contents of the column to be tagged.

Edit an identifier

Editing the details or criteria of an identifier from the identifiers menu will affect any framework with that identifier throughout Immuta. Editing the tags will only affect new frameworks the identifier is added to.

To edit an identifier,

1. Click the Discover icon in the navigation menu and select the Identifiers tab.

2. Click the name of the identifier you want to edit.

3. Click Edit.

4. Edit the field you want to change.

5. Click Save.

Built-in identifiers cannot be edited.

Delete an identifier

Deleting an identifier will remove it from all the frameworks it is in throughout Immuta.

To delete an identifier,

1. Click the Discover icon in the navigation menu and select the Identifiers tab.

2. Click the more actions icon in the Action column for the identifier you want to delete.

3. Select Delete and click Delete again in the modal.

Built-in identifiers cannot be deleted.

Requirements:

• Sensitive data discovery enabled

• Registered data sources; see the reference page for supported technologies

• Immuta permission GOVERNANCE

Identification (or sensitive data discovery (SDD)) runs automatically. If you want to re-run identification when a new global framework is set or when new identifiers have been added to a framework, you can manually run it for all data sources using the API or from the UI by following a how-to below.

Run identification using a specific framework

1. Click the Discover icon and the Identification tab in the navigation menu.

2. Select the more actions icon.

3. Select Run Identification and then select it again in the modal.

Run identification on a data source

1. Navigate to the data source overview page.

2. Click the health status.

3. Select Re-run next to Sensitive Data Discovery (SDD).

Verify discovered tags

Disable Discovered tags from the data dictionary

If a governor, data owner, or data source expert disables a Discovered tag from the data dictionary, the column will not be re-tagged next time identification (or SDD) runs. When a Discovered tag is disabled, it will not completely disappear, and it can be manually enabled through the tag side sheet.

To disable a discovered tag,

1. Navigate to a data source and click the Data Dictionary tab.

2. Scroll to the column you want to remove the tag from and click the tag you want to remove.

3. Click Disable in the side sheet and then click Confirm.

Immuta comes with a pack of built-in identifiers that look for common data types. And since the first pack was released, improvements have been made. These improvements are now available in this improved pack, which includes some unchanged identifiers, but also many new and improved versions of legacy identifiers. These identifiers were written by Immuta's research and development team and cannot be deleted or edited by users. However, users can add these built-in identifiers to their own frameworks and edit the tags applied by them.

Identifiers must match at least 90% of the sampled data to be tagged, with three exceptions noted below. See the How competitive pattern analysis works guide for more information about sampling and thresholds.

Identifier descriptions and default resulting tags

Sensitive data discovery (SDD) is an Immuta feature that uses data patterns to determine what type of data your column represents. Using identification frameworks and identifiers, Immuta evaluates your data and can assign the appropriate tags to your data dictionary based on what it finds. This saves the time of identifying your data manually and provides the benefit of a standard taxonomy across all your data sources in Immuta.

Architecture

To evaluate your data, SDD generates a SQL query using the identification framework's identifiers; the Immuta system account then executes that query in the remote technology. Immuta receives the query result, containing the column name and the matching identifiers but no raw data values. These results are then used to apply the resulting tags to the appropriate columns.

This evaluating and tagging process occurs when identification runs and happens automatically from the following events, if a global framework is set:

• A new data source is created.

• Schema monitoring is enabled, and a new data source is detected.

The following actions will also trigger identification:

• Column detection is enabled, and new columns are detected. Here, SDD will only run on new columns, and no existing tags will be removed or changed. Note, this will use the identification framework that already ran on the data source.

• A user manually triggers it from the data source health check menu. Note, this will use the identification framework that already applies to the data source or the global framework, if set.

• A user manually triggers it from the identification frameworks page.

• A user manually triggers it through the API.

Users can manually run identification from a data source's overview page or the identification frameworks page.

Components

Sensitive data discovery (SDD) runs frameworks to discover data. These frameworks are a collection of identifiers. These identifiers contain a single criteria and the tags that will be applied when the criteria's conditions have been met. See the sections below for more information on each component.

Identification framework

An identification framework is a group of identifiers that will look for particular criteria and tag any columns where those conditions are met.

While organizations can have multiple frameworks, only one may be applied to each data source. Immuta has the built-in "Default Framework," which contains all the built-in identifiers and assigns the built-in Discovered tags.

For a how-to on the framework actions users can take, see the Manage frameworks page.

Global framework

Each organization can set a global framework to apply to all the data sources in Immuta by default unless they have a different framework assigned. It is labeled on the frameworks page with a globe icon. If a global framework is set, identification will run on all new data sources. If a global framework is not set, identification will only run on data sources manually applied to an identification framework.

Users can set any framework as the global framework or leave the global framework field blank.

Identifier

An identifier is a criteria and the tags to apply to data that matches the criteria. When Immuta recognizes that criteria, it can tag the data to describe the type.

Immuta comes with built-in identifiers to discover common categories of data. These identifiers cannot be modified or deleted. Users can also create their own unique identifiers to find their specific data.

For a how-to on the identifier actions users can take, see the Create an identifier page.

Supported criteria types for identifiers

• Competitive criteria analysis: This criteria is a process that will review all the regex and dictionary criteria within the identifiers of the framework and search for the identifier with the best fit. In this review, each competitive criteria analysis identifier in the framework competes against each other to find the best and most specific identifier that fits the data. The resulting tags for the best identifier are then applied to the column. Only one competitive criteria analysis identifier will apply per column. Competitive criteria identifiers, both built-in and custom, must match at least 90% of the data sampled. To learn more about the competitive nature, see the How competitive criteria analysis works guide.

  • Regex: This criteria contains a case-insensitive regular expression that searches for matches against column values. SDD only supports regular expressions (regex) written in RE2 syntax.

  • Dictionary: This criteria contains a list of words and phrases to match against column values.

• Column name: This criteria includes a case-insensitive regular expression matched against column names, not against the values in the column. The identifier's tags will be applied to the column where the name is found. Multiple column name identifiers can match a column and be applied. SDD only supports regular expressions (regex) written in RE2 syntax.

Supported technologies

Sensitive data discovery has varied support for data sources from different technologies based on the identifier type.

Configuration

Only application admins can enable sensitive data discovery (SDD) globally on the Immuta app settings page. Then, data source creators can disable SDD on a data-source-by-data-source basis.

Tag mutability

When SDD is manually triggered by a data owner, all column tags previously applied by SDD are removed and the tags prescribed by the latest run are applied. However, if SDD is triggered because a new column is detected by schema monitoring, tags will only be applied to the new column, and no tags will be modified on existing columns. Additionally, governors, data source owners, and data source experts can disable any unwanted Discovered tags in the data dictionary to prevent them from being used and auto-tagged on that data source in the future.

Performance

The amount of time it takes to run identification on a data source depends on several factors:

• Columns: The time to run identification grows nearly linearly with the number of text columns in the data source.

• Identifiers: The number of identifiers being used weakly impacts the time to run identification.

• Row count: Performance of identification may vary depending on the sampling method used by each technology. For Snowflake, the number of rows has little impact on the time because data sampling has near-constant performance.

• Views: Performance on views is limited by the performance of the query that defines the view.

The time it takes to run identification for all newly onboarded data sources in Immuta is not limited by SDD performance but by the execution of background jobs in Immuta. Consult your Immuta account manager when onboarding a large number of data sources to ensure the advanced settings are set appropriately for your organization.

Testing

For users interested in testing SDD, note that the built-in identifiers by Immuta require a 90% match to data to be assigned to a column. This means that with synthetic data, there may be situations where the data is not real enough to fit the confidence needed to match identifiers. To test SDD, use a dev environment, create copies of your tables, or use the API to run a dryRun and see the tags that would be applied to your data by SDD.

Considerations

Deleting the built-in Discovered tags is not recommended: If you do delete built-in Discovered tags and use the Default Framework, then when the identifier is matched the column will not be tagged. As an alternative, tags can be disabled on a column-by-column basis from the data dictionary, or SDD can be turned off on a data-source-by-data-source basis when creating a data source.

Supported data types and casing

*Two built-in patterns support and match based on additional data types:

• DATE: Columns will match this identifier if they are string and the regex matches or if the data type is date, date+time, or timestamp.

• TIME: Columns will match this identifier if they are string and the regex matches or if the data type is time. Note that if the date is included in the data, it will not match this identifier.

Limitations with dictionary patterns

Immuta compiles dictionary patterns into a regex that is sent in the body of a query.

For Snowflake, the size of the dictionary is limited by the overall query text size limit in Snowflake of 1 MB.

Databricks limitation

For Databricks, Immuta will start up a Databricks cluster to complete the SDD job if one is not already running. This can cause unnecessary costs if the cluster becomes idle. Follow Databricks best practices to automatically terminate inactive clusters after a set period of time.

Starburst (Trino) limitation

Redshift limitations

• The Redshift cluster must be up and running for SDD to successfully run

• Redshift Spectrum is only supported with column name regex identifiers

Redshift supported authentication methods

AWS access key limitations

To use AWS access key authentication on a Redshift data source and have competitive criteria analysis identifiers supported,

• The AWS access key used to register the data source must be able to do a minimum of the following redshift-data API actions:

  • redshift-data:BatchExecuteStatement

  • redshift-data:CancelStatement

  • redshift-data:DescribeStatement

  • redshift-data:ExecuteStatement

  • redshift-data:GetStatementResult

  • redshift-data:ListStatements

• The AWS access key used to register the data source must have redshift:GetClusterCredentials for the cluster, user, and database that they onboard their data sources with.

• If using a custom URL, then the data source registered with the AWS access key must have the region and clusterid included in the additional connection string options formatted like the following:

  Copy

    region=us-east-2;clusterid=12345

• Redshift Serverless data sources are not supported for competitive criteria analysis identifiers with the AWS access key authentication method.

Legacy SDD

Legacy SDD was available before October 2023. It is no longer available, but some users may still see the term "legacy SDD" in the context of their data tags applied to specific data sources. These tags can be disabled from data sources but cannot be removed.

Immuta comes with a set of built-in identifiers that look for common data types. These identifiers were written by Immuta's research and development team and cannot be deleted or edited by users. However, users can add these built-in identifiers to their own frameworks and edit the tags applied by them.

Identifiers must match at least 90% of the sampled data to be tagged, with two exceptions noted below. See the How competitive pattern analysis works guide for more information about sampling and thresholds.

Identifier descriptions and default resulting tags

Add tags to data sources

1. Click the Data icon in the navigation menu and select the Data Sources tab.

2. Select a data source.

3. Click the Add Tags button on the Details tab.

4. Begin typing a tag name in the Search by Name field and select the tag from the dropdown list.

5. Click Add. A list of the applied tags will populate on the Details tab.

6. Repeat as necessary for other data sources and tags.

Remove tags from data sources

1. Click the Data icon in the navigation menu and select the Data Sources tab.

2. Select a data source.

3. Scroll to the Tags section on the Details tab, and click on the tag you want to remove.

4. Click Delete in the side sheet and then click Confirm.

Manage data dictionary tags

The data dictionary lists the columns within the data source and the value type of the data within each column. From this page, governors can add tags to or remove them from specific columns in a data source.

Add tags to the data dictionary

1. Navigate to a data source and click the Data Dictionary tab.

2. Scroll to the column you want to add a tag to and click Add Tags.

3. Begin typing in the Search by Name field and select the tag from the dropdown list.

4. Click Add. The applied tag will appear below the column name in the data dictionary.

Remove tags from the data dictionary

1. Navigate to a data source and click the Data Dictionary tab.

2. Scroll to the column you want to remove the tag from and click on the tag you want to delete.

3. Click Delete in the side sheet and then click Confirm.

Manage project tags

1. Click the Data icon and select Projects in the navigation menu.

2. Select a project.

3. Click the Add Tags button on the Project Overview tab.

4. Begin typing in the Search by Name field that appears, and then select the tag from the dropdown list.

5. Click Add. A list of the applied tags will populate on the project overview.

Remove tags from projects

1. Click the Data icon and select Projects in the navigation menu.

2. Select a project.

3. Scroll to the Tags section on the Overview tab, and then click the tag you want to delete.

4. Click Delete in the side sheet and then click Confirm.

Related guides

Reference guides

For information about data sources and tags, see the following guides:

How-to guides

In addition to adding and managing data source tags as outlined above, data owners can manage data source

The custom REST catalog integration allows Immuta to make a defined set of API calls to a Custom REST service you develop to retrieve metadata. The Custom REST service receives Immuta's calls, and then collects the relevant information and delivers it back to Immuta.

The diagram below highlights the main feature of Immuta's Custom REST Catalog integration.

Through a Custom REST Catalog, you can build and maintain your own solutions that provide metadata required to effectively use Immuta within your organization.

Section Contents

• API Interface Specification Documentation: This page details the endpoints and data schemas of the API and contains example requests and responses.

Of sensitive data discovery's , regex and dictionary are competitive. This means that when assessing your data, if multiple identifiers could match, only one with competitive criteria will be chosen to tag the data. To better understand how Immuta executes this competition, read further.

Discover employs a three-phased competitive criteria analysis approach for sensitive data discovery (SDD):

1. : No data is moved, and Immuta checks the identifiers against a sample of data from the table.

2. : Identifiers with a criteria match of less than a 90% match are filtered out.

3. : The remaining identifiers are compared with one another to find the most specific criteria that qualifies and matches the sample.

In the end, competitive criteria analysis aims to find a single identifier for each column that best describes the data format.

Sampling

In the sampling process, no database contents are transmitted to Immuta; instead, Immuta receives only the column-wise hit rate (the number of times the criteria has matched a value in the column) information for each active identifier. To do this, Discover instructs a remote database to measure column-wise hit rate information for all active identifiers over a row sample.

The sample size is decided based on the number of identifiers and the data size, when available. In the most simplified case, the requested number of sampled rows depends only on the number of regex and dictionary criteria being run in the framework, not the data size. The sample size dependence on the number of identifiers is weak and will not exceed 13,000 rows.

Sampling considerations

In practice, the number of sampled values for each column may be less than the requested number of rows because columns are not independently sampled but rather projected from a row-wise sample. This can impact the sample when the target table has less than the requested number of rows, when some of the column values are null, or because of technology-specific limitations.

• Snowflake and Starburst (Trino): Discover implements table sampling by row count.

• Databricks and Redshift: Due to technology limitations and the inability to predict the size of the table, Discover implements a best-effort sampling strategy comprising a flat 10% row sample capped at the first 10,000 sampled rows. In particular, under-sampling may occur on tables with less than 100,000 rows. Moreover, the resulting sample is biased towards earlier records.

• All platforms: Sampling from views can have significantly slower performance that varies by the performance of the query that defines the view.

• All platforms: Any null values included in the sample will not count towards the qualification or scoring when included in the sample. However, it will lower the number of available values to match against the patterns, as the sample size is not dynamic based on the ignored null values.

Qualifying

During the qualification phase, identifiers that do not agree with the data are disqualified. An identifier agrees with the data if the hit rate on the remote sample exceeds the predefined threshold. This threshold is 90% match for most built-in identifiers; however, a few built-in identifiers have lower threshold requirements. The 90% threshold is standard for all custom identifiers as well to ensure the criteria matches the data within the column and to avoid false positives. Note that threshold calculations are relative to the number of non-null entries for each column.

If no identifiers qualify, then no identifier is assessed for scoring and the column is not tagged.

Scoring

During the scoring phase, a machine inference is carried out among all qualified identifiers, combining criteria-derived complexity information with hit rate information to determine which identifier best describes the sample data. This process prefers the more restrictive of two competing identifiers since the ability to satisfy the more difficult-to-satisfy identifier itself serves as evidence that it is more likely. This phase ends by returning a single most likely identifier per the inference process.

Example

Here are a set of regex identifiers and a sample of data:

Identifiers:

1. [a-zA-Z0-9]{3} - This regex will match 3 character strings with the characters a-z, lowercase or uppercase, or digits 0-9.

2. [a-c]{3} - This regex will match 3 character strings with the characters a-c, lowercase.

3. (a|b|d){3} - This regex will match 3 character strings with the characters a, b, or d, lowercase.

When qualifying the identifiers, Identifier 1 and Identifier 3 both match 90% or more of the data. Identifier 2 does not, and is disqualified.

Then the qualified identifiers are scored. Here, Identifier 1, despite matching 100% of the data, is unspecific and could match over 200,000 values. On the other hand, Identifier 3 matches just at 90% but is very specific with only 27 available values.

Therefore, with the specificity taken into account, Identifier 3 would be the match for this column, and its tags would be applied to the data source in Immuta.

Important notes

• Dictionaries are part of the competitive process, while column-name regex are not.

• Scoring ties are rare but can occur if the same criteria (either dictionary or regex) is specified more than once (even in different forms). Scoring ties are inconclusive, and the scoring phase will not return an identifier in the case of a tie.

• Criteria complexity analysis is sensitive to the total number of strings an identifier accepts or, equivalently for dictionaries, the number of entries. Therefore, identifiers that accept much more than is necessary to describe the intended column data format may perform more poorly in the competitive analysis because they are easier to satisfy.

Immuta is pre-configured with a set of tags that can be used to write global policies before data sources even exist. See a list of the built-in Discovered tags below and the for information about where these tags will be applied by the built-in identifiers.

Country tags

All the tags below belong to the Country parent. For example, the full tag name will appear as Discovered . Country . Argentina.

Entity tags

All the tags below belong to the Entity parent. For example, the full tag name will appear as Discovered . Entity . Aadhaar Individual.

Identifier tags

None of the tags below have an additional parent or child tag. For example, the full tag name will appear as Discovered . Identifier Direct.

Personal information tags

None of the tags below have an additional parent or child tag. For example, the full tag name will appear as Discovered . PCI.

Architectural Overview

The diagram below contrasts Immuta's provided catalog integration architecture with this Customer REST Catalog interface - which gives the customer tremendous control over the metadata being provided to Immuta.

The custom-developed service must be built to receive and handle calls to the REST endpoints specified below. Immuta will call these endpoints as detailed below when certain events occur and at various intervals. The required responses to complete the connection are also detailed.

General Concepts

Tags in Immuta

Tags are attributes applied to data - either at the top, data source, level or at the individual column level.

Tags in Immuta take the form of a nested tree structure. There are "parents", "children", "grand-children", etc.:

| Parent (root)
|\_ Child1
|   \_ Grandchild1 (leaf)
 \_ Child2
    \_ Grandchild1 (leaf)

The REST Catalog interface interprets a tag's relationship mapping from a string based on a standard "dot" (.) notation, like:

"Parent.Child1.Grandchild1"

Tags returned must meet the following constraints:

• They must be no longer than 500 characters. Longer tags will not throw an error but will be truncated silently at 500 characters.

• They must be composed of letters, digits, underscores, dashes, and whitespace characters. A period (.) is used as a separator as described above. Other special characters are not supported.

A tag object has a single id property, which is used to uniquely identify the tag within the catalog. This id may be of either a string or integer type, and its value is completely up to the designer of the REST Catalog service. Common examples include: a standard integer value, a UUID, or perhaps a hash of the tag's string value (if it is unique within the system).

For this Customer REST Catalog interface, tags are represented in JSON like:

"<string>[.<string>[.<string>...]]": {
    "id": "<unique identifier, string or int>"
},

For example, the object below specifies 3 different tags:

"REST_Catalog_Root": {
    "id": "id_is_set_by_catalog_and_can_be_int_or_string"
},
"REST_Catalog_Root.Child1": {
    "id": "d3e859da-40e9-43d2-a302-294458e79a64"
},
"REST_Catalog_Root.Child2.Grandchild1": {
    "id": 10
}

Descriptions in Immuta

Descriptions are strings that, like tags, can be applied to either a data source or an individual column. These strings support UTF-8, including special and various language characters.

Authentication

Immuta can make requests to your REST Catalog service using any of the following authentication methods:

• Username and password: Immuta can send requests with a username and a password in the Authorization HTTP header. In this case, the custom REST service will need to be able to parse a Basic Authorization Header and validate the credentials sent with it.

• PKI Certificate: Immuta can also send requests using a CA certificate, a certificate, and a key.

• NO Authentication: Immuta can make unauthenticated requests to your REST Catalog service. However, this should only be used if you have other security measures in place (e.g., if the service is in an isolated network that's reachable only by your Immuta environment).

Endpoint Specification

GET /tags

Overview

The /tags endpoint is used to collect ALL the tags the catalog can provide. It is used by Immuta to populate Immuta's tags list in the Governance section. These tags can then be used for policy creation ahead of actual data sources being created that make use of them. This enables policies to immediately apply when data sources are registered with Immuta.

As with all external catalogs, tags ingested by Immuta from the REST catalog interface are not able to be modified locally within Immuta as this catalog becomes the "source of truth" for them. This results in the tags showing in Immuta with either a lock icon next to them, or without the delete button that would allow a user to manually remove them from an assigned data source or column.

Request

The /tags endpoint receives a simple GET request from Immuta. No payload nor query parameters are required.

Example request:

curl http://<your_custom_rest_catalog>/tags \
     --header 'Authorization: Basic <base64 of username:password>'

Response

Example response:

{
  "REST_Catalog_Root": {
      "id": "id_is_set_by_catalog_and_can_be_int_or_string"
  },
  "REST_Catalog_Root.Child1": {
      "id": "d3e859da-40e9-43d2-a302-294458e79a64"
  },
  "REST_Catalog_Root.Child2.Grandchild1": {
      "id": 10
  }
}

POST /dataSource

Overview

The /dataSource endpoint does the vast majority of the work. It receives a POST request from Immuta, and returns the mapping of a data source and its columns to the applied tags and descriptions.

Immuta will try to fetch metadata for a data source in the system at various times:

1. During data source creation. During data source creation, Immuta will send metadata to the REST Catalog service, most notably the connection details of the data source, which includes the schema and table name. It is important that the Custom REST service implemented can parse this information and search its records for an appropriate record to return with an ID unique to this data source in its catalogMetadata object.

2. When a user manually links the data source. Data sources that either fail to auto-link, or that were created prior to the Custom REST catalog being configured, can still be manually linked. To do so, a data source owner can provide the ID of the asset as defined by the Custom REST Catalog via the Immuta UI. In order for this to work, the Custom REST Catalog service must support matching data source assets by unique ID.
3. During various refreshes. Once linked, Immuta will periodically call the /dataSource endpoint to ensure information is up to date.

Request

Immuta's POST requests to the /dataSource endpoint will consist of a payload containing many of the elements outlined below:

This object must be parsed by the in Custom REST Catalog order to determine the specific data source metadata being requested.

For the most part, Immuta will provide the id of the data source as part of the catalogMetadata. This should be used as the primary metadata lookup value.

{
  "catalogMetadata": {
    "id": <unique integer or string value>
  }
}

When a data source is being created, such an id will not yet be known to Immuta. Immuta will instead send handlerInfo information as part of the request.

{
  "handlerInfo": {
    "schema": "schema_name",
    "table": "table_name"
  }
}

When an id is not specified, the schema and table name elements should be parsed in an attempt to identify the desired catalog entry and provide an appropriate id. If such a lookup is successful and an id is returned to Immuta in the catalogMetadata section, Immuta will establish an automatic link between the the new data source and the catalog entry, and future references will use that id.

Response

Example response:

  "catalogMetadata": {
    "id": <unique integer or string>
  },
  "description": <string>,
  "tags": {
    "Parent": {
      "id": <tag_id1>
    },
  },
  "dictionary": {
    "some_column_name": {
      "catalogMetadata": {
        "id": <col_id1>
      },
      "description": "This column has example data in it",
      "tags": {
        "Parent.Child1": {
          "id": <tag_id2>
        },
        "Parent.Child1.Grandchild1": {
          "id": <tag_id3>
        }
      }
    }
  }
}

GET /dataSource/page/{id}

Overview

This endpoint returns a human-readable information page from the REST catalog for the data source associated with {id}. Immuta provides this as a mechanism for allowing the REST catalog to provide additional information about the data source that may not be directly ingested by or visible within Immuta. This link is accessed in the Immuta UI when a user clicks the catalog logo associated with the data source.

Request

Immuta will send a GET request to the /dataSource/page/{id} endpoint, where {id} will be:

Example request:

curl http://<your_custom_rest_catalog>/dataSource/page/123

Response

The Custom REST Catalog can either provide such a page directly, or can redirect the user to any resource where the appropriate page would be provided - for example a backing full service catalog such as Collibra, if this Custom REST catalog is simply being used to support a custom data model.

Example response:

<html> 
  <head> 
    <title>data source 123</title> 
  </head> 
  <body> data source 123 is a data source that was created just for documentation.
  </body> 
</html>

GET /column/{id}

Overview

This endpoint returns the catalog's human-readable information page for the column associated with {id}. Immuta provides this as a mechanism for allowing the REST catalog to provide additional information about the specific column that may not be directly ingested by or visible within Immuta.

Request

Immuta will send a GET request to the /column/{id} endpoint, where {id} will be:

Example request:

curl http://<your_custom_rest_catalog>/column/10

Response

The Custom REST Catalog can either provide such a page directly, or can redirect the user to any resource where the appropriate page would be provided - for example a backing full service catalog such as Collibra, if this Custom REST catalog is simply being used to support a custom data model.

Example response:

<html>
  <head>
    <title>data source 123 Column 10</title>
  </head>
  <body>
    Column 10 is full of example data for documentation reasons.
  </body>
</html>

Users who want to use tags from outside of Immuta can connect an external catalog to automatically pull and apply them to Immuta data sources. These tags can then be used to drive or .

Supported external catalogs

Immuta supports the following external catalogs:

To configure an external catalog, see the .

Architecture

Once an external catalog has been configured on the Immuta app settings page, there are two recurring process steps:

1. Linking to data sources and columns: Whenever a new data source is created or an external catalog is set up, Immuta will attempt to automatically link data sources to their corresponding assets in the external catalog. This is done by comparing the fully qualified name of a data source in Immuta with its corresponding asset name in the external catalog, so data sources must have the same name in Immuta and the external catalog. Alternatively, a user can also a data source to an asset in an external catalog. Once a data source has been linked to an external catalog, it can be seen on the data source's detail page.

2. Pull and apply tags in Immuta: Using the link established in the first step, Immuta polls the external catalog to ingest and apply tags to each data source and its columns. Immuta checks every 24 hours for any relevant metadata changes in the connected external catalog. Tags originating from an external catalog can be found on the tags list page and on the data dictionary page for each data source.

See below for more information about the way Immuta integrates with each supported external catalog provider.

Alation

Immuta's Alation integration supports importing both tags and custom fields, Alation's two primary ways of allowing data stewards to apply metadata to data assets.

• Tags: Tags are a single word or phrase that can be attached to most Alation objects by nearly anyone. For instance, users can add a PCI tag for financial data.

• Custom fields: Custom fields are key-value pairs that can only be attached and removed by authorized users. Unlike tags, custom fields can have multiple values associated with a single key. For example, the custom field DK_STEWARD could have MARKETING, FINANCE, and CUSTOMER values associated with it. Using Alation custom fields allows you to explicitly control who can modify information associated with that field inside of Alation, whereas Alation standard tags are modifiable by any user inside of Alation.

When pulled into Immuta, Alation tags and custom fields will be applied to data sources as either column or data source tags in Immuta. Importing both Alation tags and custom fields into Immuta provides full flexibility for customers leveraging the Alation enterprise data catalog, no matter what operating model they choose to document their metadata in Alation.

Collibra

Immuta's Collibra integration supports importing both tags and attributes. Additionally, data source and column descriptions from the connected Collibra catalog will be pulled into Immuta.

• Tags: Tags are a single word or phrase that can be attached to objects in Collibra. For instance, users can add a PHI tag on health-related data assets.

• Attributes: Attributes in Collibra are a characteristic that describes an asset with an individual field. Unlike tags, attributes can have multiple values associated with a single key. For example, the attribute classification could have non sensitive, sensitive, and highly sensitive values associated with it. Using Collibra attributes allows you to explicitly control who can modify information associated with that field inside of Collibra, whereas Collibra standard tags are modifiable by any user inside of Collibra.

When pulled into Immuta, Collibra tags and attributes will be applied to data sources as either column or data source tags in Immuta. Importing both Collibra tags and attributes into Immuta provides full flexibility for customers leveraging the Collibra data catalog, no matter what operating model they choose to document their metadata in Collibra.

How Immuta gets metadata from Collibra

Limitations

• Collibra assets must have unique full names in order for Immuta to guarantee exact matching. If there are multiple Collibra assets with the same name, Immuta will link to the first asset it matches to.

• Columns must have a direct relation to their parent asset in Collibra. Indirect/inherited relations are not supported and will result in column tags and attributes not being ingested into Immuta.

Microsoft Purview catalog

How Immuta gets metadata from Microsoft Purview

• Linking to data sources and columns in Microsoft Purview: Immuta links data sources to assets in Microsoft Purview by looking up the fully qualified name of an entity. The composition of the fully qualified name in Microsoft Purview differs depending on the technology type backing the data source.

• Pull and apply tags in Immuta from Microsoft Purview: Immuta polls Microsoft Purview every 24 hours for all tags.

Limitations

• Standard tags from Purview do not get ingested into Immuta

• The current implementation only supports Databricks Unity Catalog, Snowflake and Azure Synapse Analytics data sources and their associated columns

• Managed attributes are supported, but have the following limitations:

  • If a managed attribute is applied to an Immuta data source but later expires, it will still appear as a tag on the data source. Expired attributes must be removed from the object in Purview for the tag to be removed from the Immuta data source.

  • The following managed attribute data types are not supported and will not be applied to Immuta data sources as tags:

    • Dates

    • Number types

    • Rich text

Custom REST catalog

If users have an unsupported catalog, or have customized their catalog integration, they can connect through the REST Catalog using the Immuta API.

Databricks Unity Catalog tag ingestion

Snowflake tag ingestion

External catalog behaviors

• Tags ingested from external catalogs cannot be edited within Immuta. To edit, delete, or add a tag from an external catalog to a data source or column, make the change in the external catalog.

• Immuta searches all external catalog providers once per day and links data sources without an external catalog attached to them to the first catalog that matches.

• S3 data sources cannot currently be linked to external catalogs.

Resources

Classification is the process in which data is categorized by the content and the associated risk level based on context. Classification complements sensitive data discovery (SDD), and the tags classification applies can give additional information in the Detect dashboards for data sources.

How-to guides

• : Use the API to activate a classification framework.

• : Create a classification framework using a provided template.

Reference guide

: This reference guide describes classification frameworks and how classification works in Immuta.

Classification is the process in which data is categorized by the content and the associated risk level based on context. To classify your data, Discover evaluates your data in two phases:

1. Sensitive data discovery (SDD) runs to identify your data by content type. The data is discovered and evaluated by the identifier it matches and is tagged.

2. Classification runs to classify your data by its context. The data is classified by the rules within a framework and the tags currently applied to the column and table. Once the data is classified, it's tagged with special tags with additional metadata used in the as sensitivity and visualize when that sensitive data is accessed.

Both phases of classification in Immuta can be customized to find and tag the data your organization cares about. After data is classified, classification tags can be used to or .

Using Discover classification to assign risk and sensitivity levels to your data and Detect dashboards to visualize the risk levels offers these benefits:

• Increasing the semantic understanding of your data to better meet compliance requirements

• Reducing the time to make decisions about what data access is allowed under what purposes

• Reducing the effort and time to respond to auditors about data access in your company

• Reducing the labor of classifying data to enumerate what data is within the scope of security or regulatory compliance frameworks

What is the difference between entity tags and classification tags?

Both entity and classification tags describe the content of data on a per-column basis, and you can use them to and . However, there are key differences between the two:

• Entity tags are applied through identification and describe what the data is. SDD applies entity tags to columns based on the patterns of the data.

• Classification tags are applied through categorization and risk assessment and describe the context of the data and the risk it poses. Using classification frameworks, classification tags are applied to columns based on the entity tags previously applied by SDD. Additional classification tags can then be applied, providing even more context or expressing the property of the record rather than just the column.

Why isn’t entity tagging sufficient for classification?

Entity tags describe the contents of individual columns, in isolation. But you don't access individual columns in isolation, so why would you determine their sensitivity that way? Entity tags do not attempt to and cannot contextualize column contents with neighboring columns' contents. This means that connections between data are lost if they cannot be identified through a pattern within the column itself. Classification tags describe the contents of a table with the context of all its columns, providing a holistic view of the risk of the data for what it is, rather than the pattern it fits. Context is necessary to understand whether your data is public or private data, risky or safe to have ungoverned access, or sensitive and creating toxic joins when accessed with other tables.

For example, under HIPAA, a list of procedures a doctor performed is only considered protected health information (PHI) if it can be associated with the identity of patients. Since entity tagging operates on a single column-by-column basis, it cannot reason whether or not a column containing procedure codes merits classification as PHI. Therefore, entity tagging will not tag procedure codes as PHI. But classification tagging will tag it PHI if it detects patient identity information in the other columns of the table.

Additionally, entity tagging does not indicate how sensitive the data is, but classification tags can carry a sensitivity level. For example, an entity tag may identify a column that contains telephone numbers, but the entity tag alone cannot say that the column is sensitive. A phone number associated with a person may be classified as sensitive, while the publicly listed phone number of a company might not be considered sensitive.

After you understand what entities your data contains using SDD, you need to adopt frameworks that determine what combinations of data constitute sensitive data and their level of sensitivity.

What is a framework?

Frameworks are a set of data categories and a set of classification rules to place data into those categories. In Immuta, the data categories are represented by tags, and when data fits a classification rule the tag is applied:

• Classification rules determine how each classification tag is applied. These rules can apply tags based on tags already on the column, tags applied to neighboring columns, and tags applied to the data source. This means that the complete data source is considered when classifying your data sources, and even tags applied to individual columns can affect the risk level of the entire data source.

Frameworks are often built off of an interpretation of regulatory frameworks or standards, such as the US Health Insurance Portability and Accountability Act (HIPAA) and the PCI standard. However, organizations can also build frameworks that represent their internal business processes. When used in Immuta, they automate data tagging and provide information about what data you have immediately after it is registered in Immuta.

What are the benefits of classification?

Data classification is a process, and with Immuta, much of it is automated. This means that you can reap the benefits of classified and tagged data quicker and easier than manually classifying and tagging it:

• Build data platform compliance: Create classification frameworks to identify and classify your data based on the industry practices and regulations your organization needs to abide by. Once the frameworks are built, they will automatically tag data as it's registered, ensuring your data sources are properly tagged to abide by the regulations you care about.

Requirements:

• Registered

• Immuta permission GOVERNANCE

Immuta Discover provides identifiers out-of-the-box to recognize and tag data. Users can then utilize classification frameworks and build them to apply tags based off those identifier tags and their own catalog tags.

Tune identification frameworks and identifiers first to adjust where Discovered tags are applied. Because classification frameworks can apply classification tags from the Discovered tags, tuning SDD should come first and will have trickle-down effects on classification. Customizing SDD requires some initial work but will automate data tagging for all data sources in the future.

Follow the steps below to tune SDD for your data:

1. .

2. .

3. .

4. : This will remove the tags from any previous identification frameworks and re-run identification with your new framework. From here, either continue to edit identifiers to reconfigure the applied tags, or if you are happy with the results, proceed to the next step.

5. .

After SDD has applied entity tags, any active classification frameworks will automatically reapply their tags to account for any changes to Discovered tags. It may be necessary to adjust the classification tags based on your organization's data, security, and compliance needs.

Assess your data source tags

Requirement: Immuta permission GOVERNANCE or data owner

Target some data sources to manually review tags:

1. Navigate to the data dictionary for the data source by opening the Data Sources page and selecting a data source. Click the Data Dictionary tab to open the data dictionary.

2. The data dictionary lists the data source columns, with details about the name, data type, and a list of the tags on each column. Assess whether the tags are accurate to your data.

If you find that too many tags are applied

Tags may be unexpected but still accurate to your data. Additionally, they may have been applied because they were found to be the best match from the identifiers in the framework.

If you want to improve SDD and personalize it to your data, assess why the tag was applied to your data:

2. Is the identifier incorrectly matching this specific column, but correct in other places? It must have been the most correct match found by identification. Create a better match by completing the following steps:

If you want to remove the unexpected tags, use one of the following how-to guides:

2. Ensure the Discovered tags are applied properly by adjusting SDD.

If you find that tags are missing

If you were expecting some sensitive data to be tagged and it is not, enable additional tags using one of the following how-to guides:

2. Ensure the Discovered tags are applied properly by adjusting SDD.

Tune your data dictionaries

Requirement: Immuta permissions GOVERNANCE and AUDIT

1. Navigate to the Data Sources page and select the data sources that you assessed and noted issues.

2. Click the Data Dictionary tab.

3. Delete unnecessary tags by clicking on the tag you want to remove from the column, and select Disable from the tag side sheet.

4. To add tags,

   1. Click Add Tags in the Actions column.

   2. Begin typing the name of the tag you want to add in the Search by Name field and select the tag from the dropdown list.

   3. Click Add.

After you have registered data sources in Immuta, you can start automating data classification of a column based on its context, which is the combination of

• associated tags already applied to the column

• tags applied to the neighboring columns and

• table tags on the data source.

The starter framework in this how-to is built to map a classification scale of restricted, confidential, internal, and public to Immuta's three-level scale, which is used in the data source and query event dashboards.

Follow this guide to map your tags to the example framework, or consult the for more information about the framework schema.

Customize the framework

Using the example framework below, customize the framework for your organization's classification tags:

{
  "shortName": "ECMC Framework",
  "name": "External Catalog Mapping Classification Framework",
  "description": "This framework maps the classification tags the organization has in Collibra to Immuta data sources.",
  "tags": [
    {
      "name": "ECMC.Confidentiality.Highly Sensitive",
      "source": "curated",
      "sensitivities": [
        {
          "dimension": "confidentiality",
          "sensitivity": 2
        }
      ]
    },
    {
      "name": "ECMC.Confidentiality.Sensitive",
      "source": "curated",
      "sensitivities": [
        {
          "dimension": "confidentiality",
          "sensitivity": 1
        }
      ]
    },
    {
      "name": "ECMC.Confidentiality.Nonsensitive",
      "source": "curated",
      "sensitivities": []
    }
  ],
  "rules": [
    {
      "name": "ECMC 00001",
      "classificationTag": {
        "name": "ECMC.Confidentiality.Highly Sensitive",
        "source": "curated"
      },
      "columnTags": [
        {
          "name": "Restricted",
          "source": "collibra"
        }
      ],
      "neighborColumnTags": [],
      "tableTags": []
    },
    {
      "name": "ECMC 00002",
      "classificationTag": {
        "name": "ECMC.Confidentiality.Sensitive",
        "source": "curated"
      },
      "columnTags": [
        {
          "name": "Confidential",
          "source": "collibra"
        }
      ],
      "neighborColumnTags": [],
      "tableTags": []
    },
    {
      "name": "ECMC 00003",
      "classificationTag": {
        "name": "ECMC.Confidentiality.Sensitive",
        "source": "curated"
      },
      "columnTags": [
        {
          "name": "Internal",
          "source": "collibra"
        }
      ],
      "neighborColumnTags": [],
      "tableTags": []
    },
    {
      "name": "ECMC 00004",
      "classificationTag": {
        "name": "ECMC.Confidentiality.Nonsensitive",
        "source": "curated"
      },
      "columnTags": [
        {
          "name": "Public",
          "source": "curated"
        }
      ],
      "neighborColumnTags": [],
      "tableTags": []
    }
  ],
  "active": true
}

Parameters

1. tags: These tags are automatically created in Immuta with the sensitivity you assign. They must not already exist in Immuta. All tags used in the classificationTag parameter should be defined here.

2. tags.sensitivities: This is metadata for the sensitivity of the new tag. Use confidentiality for dimension. Options for sensitivity are 1 (shown as sensitive in Detect dashboards) and 2 (shown as highly sensitive in Detect dashboards). For nonsensitive, leave this parameter empty.

3. rules: These are the rules for applying the tags defined above. Each rule contains the classification tag to apply if the requirements are met and the requirements: the column tags, neighboring column tags, and table tags that must be present. All requirements within each defined rule must be met for the classification tag to be applied.

4. rules.classificationTag: The name and source of the tag you want applied if the rule requirements are met. This classification tag must be defined in tags. The source is curated.

5. rules.columnTags: These are the required tags for a column. If the tags defined here are found on a column, and the other tag rules are met, then the rule's classificationTag will be applied to the same column.

6. rules.neighborColumnTags: These are the required tags on other columns in the data source (or in the query if dynamic query classification is enabled). If the tags defined here are found on any column in the data source, and the other tag rules are met, then the rule's classificationTag will be applied to all the neighboring columns.

7. rules.tableTags: These are the required tags on the data source. If the tags defined here are found on the data source, and the other tag rules are met, then the rule's classificationTag will be applied to all the columns in that data source.

8. active: When true the framework is active and will apply tags when the rules are met.

How to edit rules

Follow the example below to map your tags to the rules in the example framework.

This example framework has a rule where columns tagged DSF.Interpretation.Credentials.Secret by sensitive data discovery will be tagged RAF.Confidentiality.High:

"rules": [
{
    "name": "RAF 00004",
    "classificationTag": {
      "name": "RAF.Confidentiality.High",
      "source": "curated"
    },
    "columnTags": [
    {
        "name": "DSF.Interpretation.Credentials.Secret",
        "source": "curated"
    }
    ],
    "neighborColumnTags": [],
    "tableTags": []
}
]

To translate this to your tags, replace the name and source value of the columnTags, neighborColumnTags, or tableTags with your own. This new example is for a Collibra tag from the external catalog that an organization uses for confidential data. This rule now states: Apply the classification tag RAF.Confidentiality.High to a column if it has the collibra tag Confidential. Repeat this for your organization's remaining classification levels.

"rules": [
{
    "name": "RAF 00004",
    "classificationTag": {
      "name": "RAF.Confidentiality.High",
      "source": "curated"
    },
    "columnTags": [
    {
        "name": "Confidential",
        "source": "collibra"
    }
    ],
    "neighborColumnTags": [],
    "tableTags": []
}
]

Find the name and source for your tags

If you do not know the name or source for your tags, you can list your tags using the Immuta API:

curl \
    --request GET \
    --header "accept: application/json" \
    --header "Authorization: Bearer <your-token." \
    https://your-immuta-url.com/tag

This request will list all the tags in your Immuta environment, similar to this example response:

[
  {
    "id": 114,
    "name": "DataProperties.Cross-Sectional",
    "source": "curated",
    "deleted": false,
    "systemCreated": true
  },
  {
    "id": 2,
    "name": "Discovered.Country.Argentina",
    "source": "curated",
    "deleted": false,
    "systemCreated": true
  },
  {
    "id": 9,
    "name": "Discovered.Country.Australia",
    "source": "collibra",
    "deleted": false,
    "systemCreated": true
  }
]

Activate your new framework

Requirement: Immuta permission GOVERNANCE

Once you have made all the customizations to the example framework, make the following request using the Immuta API, with your full customized framework as the payload.

curl \
    --request POST \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer <your-token>" \
    --data @example-payload.json \
    https://your.immuta.com/frameworks/

Your new framework will now be visible in the Immuta UI by navigating the the Classification section under Discover.

Requirements:

• Registered

• Immuta permission GOVERNANCE

To activate a classification framework,

1. Navigate to Discover and select the Classification tab.

2. Click the more actions icon in the Actions column for the framework you want to activate.

3. Select Activate.