Learn about connecting external catalogs
Connect an external catalog to use tagging capabilities outside of Immuta and pull tags from external table schemas. Once the catalog has been connected, Immuta ingests a data dictionary from the catalog and applies data source and column tags directly to the data source. These tags can then be used to create policies.
Configure an external catalog: Configure an external catalog to ingest tags into Immuta.
External catalog integrations: This reference guide describes the requirements of the external catalogs Immuta supports.
Custom REST catalog interface endpoints: This reference guide describes the endpoints for configuring a custom REST catalog.
Connect your external catalog to pull data metadata into Immuta
This page outlines how to connect an external catalog to Immuta. For details on external catalogs in Immuta, see the External catalog reference guide.
Requirements:
APPLICATION_ADMIN Immuta permission
An connected to a user with the permission
Navigate to the App Settings page.
Scroll to 2 External Catalogs, and click Add Catalog.
Enter a Display Name and select Alation from the dropdown menu.
Requirements:
APPLICATION_ADMIN Immuta permission
An with that correspond to Immuta data sources
Navigate to the App Settings page.
Scroll to 2 External Catalogs, and click Add Catalog.
Enter a Display Name and select Atlan from the dropdown menu.
Requirements:
APPLICATION_ADMIN Immuta permission
A Collibra user with visibility on all assets relevant to Immuta data sources (Collibra global role Catalog or Catalog Author)
A Collibra physical data dictionary with assets that correspond to your Immuta data sources
Navigate to the App Settings page.
Scroll to 2 External Catalogs, and click Add Catalog.
Enter the Display Name and select Collibra from the dropdown menu.
Requirements:
APPLICATION_ADMIN Immuta permission
A Microsoft Purview catalog with assets that correspond to your Immuta data sources
The ability to create a registered app in the Azure portal. See the .
with the with the following settings:
Supported account type: "Accounts in this organizational directory only"
Microsoft-Graph: User.Read API permission
A client secret
Using that registered app, navigate to Immuta and complete the following:
Navigate to the App Settings page.
Scroll to 2 External Catalogs, and click Add Catalog.
Enter the Display Name and select Microsoft Purview from the dropdown menu.
Requirements:
APPLICATION_ADMIN Immuta permission
An external catalog with tags that correspond to your Immuta data sources
Integrating a custom REST catalog service with Immuta requires implementing a REST interface. For details about the necessary endpoints that must be serviced, see the .
Navigate to the App Settings page.
Scroll to 2 External Catalogs, and click Add Catalog.
Enter the Display Name and select Rest from the dropdown menu.
You can manually link and remove external catalogs from data sources on the data source details tab.
Navigate to your data source.
In the connection information section, click the Link Catalog icon (or Unlink Catalog to remove an external catalog from a data source).
Select your external catalog from the dropdown menu and enter the appropriate ID:
Navigate to your data source and click the data source Health dropdown menu.
Click Re-run in the External Catalog section.
Learn about supported external catalogs and how they interact with Immuta
Server Admin permission.Configure whether or not Alation tags and custom fields are imported as Immuta tags:
Link Alation tags: When selected, Immuta imports Alation tags as Immuta tags.
Link Alation Custom Fields: When selected, Immuta imports Alation custom fields as Immuta tags. Follow the Alation documentation to create an Alation custom field, add permissions to your custom field, and apply custom fields to tables and columns.
Opt to select Upload Certificates.
Upload the Certificate Authority, Certificate File, and Key File.
Opt to enable Strict SSL by selecting the checkbox.
Click the Test Connection button.
Once the connection is successful, click Save.
Click the Test Connection button.
Once the connection is successful, click Save.
Select an authentication method from the dropdown menu. Immuta will use the credentials provided to connect to the external catalog:
Username and password: Complete the Username and Password fields.
OAuth 2.0:
Collibra OAuth provider: Generate the client ID and client secret in Collibra. Immuta will use these credentials to communicate with Collibra. .
Fill out the Client ID. This is a combination of letters, numbers, or symbols used as a public identifier.
Enter the Client Secret you created above.
Leave the
External OAuth provider: Use your external OAuth provider client ID and client secret. Immuta will use these credentials to request an access token from Collibra's token endpoint. Then, Immuta will use that returned access token as the bearer token in API calls with Collibra.
Set up Collibra to accept and validate external tokens in the JWT format.
Fill out the Client ID. This is a combination of letters, numbers, or symbols used as a public identifier.
Complete the Asset Mappings modal to set which Collibra asset types align to the Immuta data source and column. Immuta will only link data sources from the asset types you specify.
Complete the Attributes as Tags modal to specify which Collibra attributes you want in Immuta. These attributes will come in as parent tags with their values as children tags.
Opt to select Upload Certificates.
Upload the Certificate Authority, Certificate File, and Key File.
Opt to enable Strict SSL by selecting the checkbox.
Click the Test Connection button.
Once the connection is successful, click Save.
Enter the Microsoft Purview endpoint URL including the Azure Account Name, like https://<ACCOUNTNAME>.purview.azure.com, in the Purview Endpoint URL field.
Complete the Microsoft Entra Directory (tenant) ID and Microsoft Entra (client) ID fields.
Enter the Microsoft Entra Application Client Secret ID for Immuta to authenticate and connect to the Purview API. The secret cannot be expired.
Click the Test Connection button.
Once the test is successful, click Save.
Complete the following fields:
Enter the HTTP endpoint of the catalog in the URL field.
Complete the Username and Password fields.
Enter the path of the Tags Endpoint.
Enter the path of the Data Source Endpoint.
Enter the path to the information page for a data source in the Data Source Link Template field.
Opt to enter the path to the information page for a column in the Column Link Template field.
Opt to upload a Catalog Image.
Opt to select Upload Certificates.
Upload the Certificate Authority, Certificate File, and Key File.
Opt to enable Strict SSL by selecting the checkbox.
Click the Test Connection button.
Click the Test Data Source Link.
Once both tests are successful, click Save.
Alation: Enter the ID of the asset from Alation into the Catalog Id field.
Atlan: Enter the GUID of the asset from Atlan into the Catalog Id field.
Collibra: Enter the UUID of the asset from Collibra into the Catalog Id field.
Microsoft Purview: Enter the GUID of the asset from Microsoft Purview into the Catalog Id field.
Click Link to confirm.
To change the default expiration period for your Alation catalog's API tokens, see configure the expiration period for Alation API tokens.
Private preview: This feature is available to select accounts. Contact your Immuta representative to enable this feature.
Private preview: This feature is available to select accounts. Contact your Immuta representative to enable this feature.
You can also ingest tags from the following data platforms:
To configure an external catalog, see the Configure an external catalog guide.
Once an external catalog has been configured on the Immuta app settings page, there are two recurring process steps:
Linking to data sources and columns: Immuta will attempt to automatically link data sources to their corresponding assets in an external catalog from two actions:
A new data source is created
An external catalog is set up
This is done by comparing the fully qualified name of a data source in Immuta with its corresponding asset name in the external catalog, so data sources must have the same database, schema, and object name in Immuta and the external catalog. If the match is not found, . Once a data source has been linked to an external catalog, it can be seen on the data source's detail page.
Pull and apply tags in Immuta: Using the link established in the first step, Immuta polls the external catalog to ingest and apply tags to each data source and its columns. Immuta checks every 24 hours for any relevant metadata changes in the connected external catalog. If the external catalog sync for a particular data source is unsuccessful, this will be reflected in the . For data sources where the last external catalog sync failed, Immuta will reattempt the sync every hour until successful. Tags originating from an external catalog can be found on the tags list page and on the columns tab for each data source.
See below for more information about the way Immuta integrates with each supported external catalog provider.
Immuta's Alation integration supports importing both tags and custom fields, Alation's two primary ways of allowing data stewards to apply metadata to data assets.
Tags: Tags are a single word or phrase that can be attached to most Alation objects by nearly anyone. For instance, users can add a PCI tag for financial data.
Custom fields: Custom fields are key-value pairs that can only be attached and removed by authorized users. Unlike tags, custom fields can have multiple values associated with a single key. For example, the custom field DK_STEWARD could have MARKETING, FINANCE, and CUSTOMER values associated with it. Using Alation custom fields allows you to explicitly control who can modify information associated with that field inside of Alation, whereas Alation standard tags are modifiable by any user inside of Alation. The following custom field data types are supported and will be applied to Immuta data sources as tags: pickers, multi-select pickers, object sets, people sets, references, and dates.
When pulled into Immuta, Alation tags and custom fields will be applied to data sources as either column or data source tags in Immuta. Importing both Alation tags and custom fields into Immuta provides full flexibility for organizations leveraging the Alation enterprise data catalog, no matter what operating model they choose to document their metadata in Alation.
Linking to data sources and columns in Alation: Immuta links data sources to assets in Alation by looking up the fully qualified name of an object via the Alation API.
Pull and apply tags in Immuta from Alation: Immuta polls Alation every 24 hours for all tags.
The Atlan catalog integration with Immuta supports ingestion of tags and descriptions from Atlan assets.
Linking to data sources and columns in Atlan: Immuta links data sources to assets in Atlan by looking up the fully qualified name of an entity and its corresponding host information using Atlan APIs. All of the following conditions must be fulfilled for an Immuta data source to successfully link to an Atlan asset:
The Immuta data source name must match the Atlan asset name.
The host URL of the Snowflake account or Databricks workspace used to onboard the data source into Immuta must match the host URL used to onboard the asset into Atlan.
Pull and apply tags in Immuta from Atlan: Immuta checks Atlan every 24 hours for any relevant metadata changes. Based on these changes, Immuta then only polls and ingests tags from Atlan for the relevant data sources. However, if Immuta observes more than 25,000 metadata changes in Atlan within 24 hours, it will poll all data sources for tags during that run of external catalog tag synchronization.
Custom metadata fields from Atlan do not get ingested as tags into Immuta
The current implementation only supports Databricks Unity Catalog or Snowflake data sources and their associated columns.
Immuta's Collibra integration supports importing tags, data classifications, and attributes. Additionally, data source and column descriptions from the connected Collibra catalog will be pulled into Immuta.
Tags: Tags are a single word or phrase that can be attached to objects in Collibra. For instance, users can add a PHI tag on health-related data assets.
Data classifications: Data classifications are a label in Collibra on the asset type column that describe the content of data and are separate from tags in Collibra. Immuta will ingest accepted data classifications from Collibra and apply these classifications as tags on the appropriate columns. All data classifications from Collibra will be under the Data classification parent tag.
Attributes: Attributes in Collibra are a characteristic that describes an asset with an individual field. Unlike tags, attributes can have multiple values associated with a single key. For example, the attribute region could have emea, apac, and nala values associated with it. Using Collibra attributes allows you to explicitly control who can modify information associated with that field inside of Collibra, whereas Collibra standard tags are modifiable by any user inside of Collibra.
When pulled into Immuta, Collibra tags, data classifications, and attributes will be applied to data sources as either column or data source tags in Immuta. Importing Collibra tags, data classifications, and attributes into Immuta provides full flexibility for organizations leveraging the Collibra data catalog, no matter what operating model they choose to document their metadata in Collibra.
Linking to data sources and columns in Collibra: Immuta links data sources to assets in Collibra by looking up the full name. The Immuta data source name must match the Collibra table asset name for the table to successfully link.
Pull and apply tags in Immuta from Collibra: Immuta checks Collibra every 24 hours by observing the linked assets history for any relevant metadata changes. Based on these changes, Immuta then only polls and ingests objects from Collibra for the relevant data sources. However, if Immuta observes more than 25,000 metadata changes in Collibra within 24 hours, it will poll all data sources for tags, data classifications, and attributes during that run of external catalog tag synchronization.
The catalog auto-linking method will only auto-link Collibra assets where the asset's full name follows the Collibra Edge naming convention. Any assets following a different naming convention must be linked manually instead.
Columns must have a direct relation to their parent asset in Collibra. Indirect/inherited relations are not supported and will result in column tags and attributes not being ingested into Immuta.
The Microsoft Purview catalog integration with Immuta currently supports ingestion of Classifications and Managed attributes of type single or multiple choice as tags. Additionally, data source and column descriptions from the connected Microsoft Purview catalog will be pulled into Immuta.
Linking to data sources and columns in Microsoft Purview: Immuta links data sources to assets in Microsoft Purview by looking up the fully qualified name of an entity. The composition of the fully qualified name in Microsoft Purview differs depending on the technology type backing the data source.
Pull and apply tags in Immuta from Microsoft Purview: Immuta polls Microsoft Purview every 24 hours for all tags.
Standard tags from Purview do not get ingested into Immuta
The current implementation only supports Databricks Unity Catalog, Snowflake, and Azure Synapse Analytics data sources and their associated columns
If a managed attribute is applied to an Immuta data source but later expires, it will still appear as a tag on the data source. Expired attributes must be removed from the object in Purview for the tag to be removed from the Immuta data source.
If users have an unsupported catalog, or have customized their catalog integration, they can connect through the REST Catalog using the Immuta API.
For more details about using a custom REST catalog with Immuta, see the Custom REST catalog interface endpoints page.
The AWS Lake Formation integration can ingest Lake Formation Tags and apply them to Immuta data sources.
To learn more about AWS Lake Formation tag ingestion, see the AWS Lake Formation reference guide.
To configure tag ingestion, see the Register an AWS Lake Formation connection page.
Users can connect their Databricks Unity Catalog account to allow Immuta to ingest Databricks tags and apply them to Databricks data sources.
To learn more about Databricks Unity Catalog tag ingestion, see the Databricks Unity Catalog reference guide.
To configure tag ingestion, see the Register a Databricks Unity Catalog connection page.
Users can connect a Snowflake account to allow Immuta to ingest Snowflake tags onto Snowflake data sources.
To learn more about Snowflake tag ingestion, see the Snowflake reference guide.
To configure tag ingestion, see the Configure a Snowflake integration page.
This table lists the supported external catalogs and their supported authentication methods. Data platform tag ingestion uses the data platform credentials. For more details about a catalog, see the linked section:
❌
❌
✅
❌
Tags ingested from external catalogs cannot be edited within Immuta. To edit, delete, or add a tag from an external catalog to a data source or column, make the change in the external catalog.
You can configure multiple external catalogs within a single tenant of Immuta, but only one external catalog can be linked to a data source.
Immuta searches all external catalog providers once per day and links data sources without an external catalog attached to them to the first catalog that matches.
S3 data sources can currently only be to external catalogs.
The following catalog-related events are audited and can be found on the audit page in the UI:
ConfigurationUpdated: The configuration on the Immuta app settings page is updated, including when an external catalog configuration is added or deleted.
DatasourceCatalogSynced: An external catalog is linked and synced for the data source.
To configure an external catalog, see the Configuration how-to guide.
To learn more about how Immuta can automatically tag your data, see the Data discovery introduction.
Best practice: Use a single catalog; having more than one can lead to multiple truths and data leaks.
Alation tags and custom fields (except people sets, since those are represented as email addresses) containing values with a dot "." delimiter will be transformed into hierarchical tags in Immuta. To learn more about the benefits of hierarchical tags for policy authoring, see .
Private preview
The Atlan catalog integration is only available to select accounts. Contact your Immuta representative to enable this feature.
Collibra objects using the dot "." delimiter will be transformed into hierarchical tags in Immuta. To learn more about the benefits of hierarchical tags for policy authoring, see .
Private preview: This feature is available to select accounts. Contact your Immuta representative to enable this feature.
Private preview: This feature is only available to select accounts. Contact your Immuta representative to enable this feature.
Private preview: This feature is only available to select accounts. Contact your Immuta representative to enable this feature.
Learn about the custom REST catalog integration and its endpoints
The custom REST catalog integration allows Immuta to make a defined set of API calls to a custom REST service you develop to retrieve metadata. The custom REST service receives Immuta's calls, and then collects the relevant information and delivers it back to Immuta, allowing you to build and maintain your own solutions that provide metadata required to effectively use Immuta within your organization.
The custom-developed service must be built to receive and handle calls to the REST endpoints specified below. Immuta will call these endpoints when certain events occur and at various intervals. The required responses to complete the connection are detailed below.
Tags are attributes applied to data at the data-source-level or at the column-level.
Tags in Immuta take the form of a nested tree structure:
The REST catalog interface interprets a tag's relationship mapping from a string based on a standard dot (.) notation:
Tags returned must meet the following constraints:
They must be no longer than 500 characters. Longer tags will not throw an error but will be truncated silently at 500 characters.
They must be composed of letters, digits, underscores, dashes, and whitespace characters. A dot (.) is used as a delimiter as described above. Other special characters are not supported.
A tag object has a single id property, which is used to uniquely identify the tag within the catalog. This id may be either a string or integer type, and its value is up to the designer of the REST catalog service. Common examples include a standard integer value, a UUID, or a hash of the tag's string value (if it is unique within the system).
For this example REST catalog interface, tags are represented in a JSON object. The object below specifies 3 different tags:
For more information on tags and how they are created, managed, and displayed within Immuta, see the .
Descriptions are strings that can be applied to either a data source or an individual column. These strings support UTF-8, including special and various language characters.
Immuta can make requests to your REST catalog service using any of the following authentication methods:
Username and password: Immuta can send requests with a username and a password in the authorization HTTP header. In this case, the custom REST service will need to be able to parse a basic authorization header and validate the credentials sent with it.
PKI certificate: Immuta can also send requests using a CA certificate, a certificate, and a key.
No authentication: Immuta can make unauthenticated requests to your REST catalog service. However, this should only be used if you have other security measures in place (e.g., if the service is in an isolated network that's reachable only by your Immuta environment).
/tagsThe /tags endpoint is used to collect ALL the tags the catalog can provide. It is used by Immuta to populate Immuta's tags list on the tags page. These tags can then be used for policy creation ahead of actual data sources being created that make use of them. This enables policies to immediately apply when data sources are registered with Immuta.
As with all external catalogs, tags ingested by Immuta from the REST catalog interface cannot be modified locally within Immuta, since this catalog is the source of truth for them.
The custom REST service must respond with an object that maps all tag name strings to associated ids. The fully-qualifies the location of the tag in the tree structure, and the id is a globally unique identifier assigned by the REST catalog to that tag.
/dataSourceThe /dataSource endpoint does the majority of the work. It receives a POST request from Immuta, and returns the mapping of a data source and its columns to the applied tags and descriptions.
Immuta will try to fetch metadata for a data source in the system at various times:
During data source creation. During data source creation, Immuta will send metadata to the REST catalog service, most notably the connection details of the data source, which includes the schema and table name. It is important that the custom REST service implemented can parse this information and search its records for an appropriate record to return with an ID unique to this data source in its catalogMetadata object.
. Data sources that either fail to auto-link or that were created prior to the custom REST catalog being configured can still be manually linked. To do so, a data source owner can provide the ID of the asset as defined by the custom REST catalog via the Immuta UI. In order for this to work, the custom REST catalog service must support matching data source assets by unique ID.
Immuta's POST requests to the /dataSource endpoint will consist of a payload containing many of the elements outlined below.
This object must be parsed by the custom REST catalog in order to determine the specific data source metadata being requested.
Immuta will provide the id of the data source as part of the catalogMetadata. This should be used as the primary metadata lookup value.
When a data source is being created, such an id will not yet be known to Immuta. Immuta will instead send handlerInfo information as part of the request.
When an id is not specified, the schema and table name elements should be parsed in an attempt to identify the desired catalog entry and provide an appropriate id. If such a lookup is successful and an id is returned to Immuta in the catalogMetadata section, Immuta will establish an automatic link between the new data source and the catalog entry, and future references will use that id.
The schema for the /dataSource response uses the same tag object structure from the , along with the following set of metadata keys for both data sources and columns.
Example
/dataSource/page/{id}This endpoint returns a human-readable information page from the REST catalog for the data source associated with {id}. Immuta provides this as a mechanism for allowing the REST catalog to provide additional information about the data source that may not be directly ingested by or visible within Immuta. This link is accessed in the Immuta UI when a user clicks the catalog logo associated with the data source.
Immuta will send a GET request to the /dataSource/page/{id} endpoint.
Example
The custom REST catalog can either provide such a page directly, or can redirect the user to any resource where the appropriate page would be provided - for example a backing full service catalog such as Collibra, if this custom REST catalog is simply being used to support a custom data model.
/column/{id}This endpoint returns the catalog's human-readable information page for the column associated with {id}. Immuta provides this as a mechanism for allowing the REST catalog to provide additional information about the specific column that may not be directly ingested by or visible within Immuta.
Immuta will send a GET request to the /column/{id} endpoint.
Example
The custom REST catalog can either provide such a page directly, or it can redirect the user to any resource where the appropriate page would be provided. For example, it could redirect to a backing full service catalog such as Collibra, if the custom REST catalog is simply being used to support a custom data model.
Opt to enter the Scope. The scope limits the operations and roles allowed in Collibra. See the OAuth 2.0 documentation for details about scopes.
Enter the Client Secret. Immuta uses this secret to authenticate with the authorization server when it requests a token.
Add the token endpoint URL in the Token URL field.
Opt to enter the Scope. The scope limits the operations and roles allowed in Collibra. See the OAuth 2.0 documentation for details about scopes.
❌
✅
✅
✅
❌
❌
✅
❌
/dataSource endpoint to ensure information is up to date.string
The name of the data source in the catalog.
handlerInfo
dictionary
Object holding the data source's connection details.
handlerInfo.schema
string
The data source’s schema name in the source system.
handlerInfo.table
string
The data source’s table name in the source system.
handlerInfo.hostname
string
The data source’s connection schema in the source storage system.
handlerInfo.port
integer
The data source’s connection port in the source storage system.
handlerInfo.query
string
The data source’s connection schema in the source storage system, if applicable.
dataSource
dictionary
Object holding general data source information from Immuta. This can be viewed with debugging, but is not usually required for catalog purposes.
string
The name of the data source in the catalog.
description
string
A description of the data source.
tags
<tags object>
Object containing the data source-level tags.
dictionary
dictionary
Object containing the column names of the data source as its keys.
dictionary.<column>
dictionary
Object containing a single column's metadata.
dictionary.<column>.catalogMetadata.id
string or integer
The unique identifier of the column in the catalog.
dictionary.<column>.description
string
A description of the column.
dictionary.<column>.tags
<tags object>
Object containing the column-level tags as keys.
catalogMetadata
dictionary
Object holding the data source's catalog metadata.
catalogMetadata.id
string or integer
The unique identifier of the data source in the catalog.
catalogMetadata
dictionary
Object holding the data source's catalog metadata.
catalogMetadata.id
string or integer
The unique identifier of the data source in the catalog.
id
URL parameter, integer, or string
The unique identifier of the data source in the remote catalog system.
id
URL parameter, integer, or string
The unique identifier of the column in the remote catalog system.
Authentication and specific endpoints
When accessing the /dataSource and /tags endpoints, Immuta will use the configured username and password. If you choose to also protect the human-readable pages with authentication, users will be prompted to authenticate when they first visit those pages.
catalogMetadata.name
catalogMetadata.name
| Parent (root)
|\_ Child1
| \_ Grandchild1 (leaf)
\_ Child2
\_ Grandchild1 (leaf)"Parent.Child1.Grandchild1""REST_Catalog_Root": {
"id": "id_is_set_by_catalog_and_can_be_int_or_string"
},
"REST_Catalog_Root.Child1": {
"id": "d3e859da-40e9-43d2-a302-294458e79a64"
},
"REST_Catalog_Root.Child2.Grandchild1": {
"id": 10
}curl http://<your_custom_rest_catalog>/tags \
--header 'Authorization: Basic <base64 of username:password>'{
"REST_Catalog_Root": {
"id": "id_is_set_by_catalog_and_can_be_int_or_string"
},
"REST_Catalog_Root.Child1": {
"id": "d3e859da-40e9-43d2-a302-294458e79a64"
},
"REST_Catalog_Root.Child2.Grandchild1": {
"id": 10
}
}{
"catalogMetadata": {
"id": <unique integer or string value>
}
}{
"handlerInfo": {
"schema": "schema_name",
"table": "table_name"
}
} "catalogMetadata": {
"id": <unique integer or string>
},
"description": <string>,
"tags": {
"Parent": {
"id": <tag_id1>
},
},
"dictionary": {
"some_column_name": {
"catalogMetadata": {
"id": <col_id1>
},
"description": "This column has example data in it",
"tags": {
"Parent.Child1": {
"id": <tag_id2>
},
"Parent.Child1.Grandchild1": {
"id": <tag_id3>
}
}
}
}
}curl http://<your_custom_rest_catalog>/dataSource/page/123<html>
<head>
<title>data source 123</title>
</head>
<body> data source 123 is a data source that was created just for documentation.
</body>
</html>curl http://<your_custom_rest_catalog>/column/10<html>
<head>
<title>data source 123 Column 10</title>
</head>
<body>
Column 10 is full of example data for documentation reasons.
</body>
</html>