Supported technologies

• Databricks Unity Catalog

• Snowflake

Requirements

• A native integration enabled on the Immuta app settings page

• Data sources registered

To complete your upgrade,

1. Select Upgrade Manager in the navigation. This tab will only be available if you have integrations ready for upgrade.

2. Click Start Upgrade.

3. Enter a Connection Key. The connection key represents the unique name of your connection and will be used as prefix in the name for all data objects associated with this connection. It will also appear as the display name in the UI and will be used in all API calls made to update or delete the connection.

4. Click Next.

5. Ensure Immuta has the correct credentials to connect to Databricks Unity Catalog or Snowflake. Select the tab below for more information:

1. Click Next.

2. Click Upgrade Connection.

3. Click the link to the docs to understand the impacts of the upgrade.

4. Click the checkbox to confirm understanding of the upgrade effects, and click Yes, Upgrade Connection.

Requirements

The following permissions and personas are used in the registration process:

• Immuta permission: CREATE_DATA_SOURCE

• Snowflake permissions for the user registering the connection and running the script:

  • CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

  • CREATE ROLE ON ACCOUNT WITH GRANT OPTION

  • CREATE USER ON ACCOUNT WITH GRANT OPTION

  • MANAGE GRANTS ON ACCOUNT WITH GRANT OPTION

  • APPLY MASKING POLICY ON ACCOUNT WITH GRANT OPTION

  • APPLY ROW ACCESS POLICY ON ACCOUNT WITH GRANT OPTION

  • REFERENCES on all tables

  • USAGE on the schema and database to register data sources

• Snowflake permissions for the new Immuta system user that is created:

  • APPLY MASKING POLICY ON ACCOUNT

  • APPLY ROW ACCESS POLICY ON ACCOUNT

  • Additional grants associated with the IMMUTA database

Prerequisite

No Snowflake native integration configured in Immuta. If your Snowflake native integration is already configured on the app settings page, follow the Use the connection upgrade manager guide.

Register a connection

To register a Snowflake connection, follow the instructions below.

1. Click Data and select the Infrastructure tab in the navigation menu.

2. Click the + Add Host button.

3. Select the Snowflake data platform tile.

4. Enter the connection information:

   • Host: The URL of your Snowflake account.

   • Port: Your Snowflake port.

   • Warehouse: The warehouse the Immuta system account user will use to run queries and perform Snowflake operations.

   • Immuta Database: The new, empty database for Immuta to manage. This is where system views, user entitlements, row access policies, column-level policies, procedures, and functions managed by Immuta will be created and stored.

   • Role: The default Snowflake role for the Immuta system account user.

   • Connection Key: The connection key represents the unique name of your connection and will be used as prefix in the name for all data objects associated with this connection. It will also appear as the display name in the UI and will be used in all API calls made to update or delete the connection.

5. Click Next.

6. Select an authentication method from the dropdown menu. This authentication information will be included in the script populated later on the page.

   1. Username and password: Choose one of the following options.

      1. Select Immuta Generated to have Immuta populate the system account name and password.

      2. Select User Provided to enter your own name and password for the Immuta system account.

   2. Snowflake External OAuth:

      1. Fill out the Token Endpoint, which is where the generated token is sent. It is also known as aud (audience) and iss (issuer).

      2. Fill out the Client ID, which is the subject of the generated token. It is also known as sub (subject).

      3. Opt to fill out the Resource field with a URI of the resource where the requested token will be used.

      4. Enter the x509 Certificate Thumbprint. This identifies the corresponding key to the token and is often abbreviated as x5t or is called kid (key identifier).

      5. Upload the PEM Certificate, which is the client certificate that is used to sign the authorization request.

   3. Key Pair Authentication:

      1. Complete the Username field. This username will be used to connect to the remote database and retrieve records for this data source.

      2. If using a private key, enter the Private Key Password.

      3. Click Select a File, and upload a Snowflake key pair file.

7. The Role is prepopulated from the entry on the previous page.

8. Copy the provided script and run it in Snowflake with the following Snowflake permissions:

   • CREATE DATABASE ON ACCOUNT WITH GRANT OPTION

   • CREATE ROLE ON ACCOUNT WITH GRANT OPTION

   • CREATE USER ON ACCOUNT WITH GRANT OPTION

   • MANAGE GRANTS ON ACCOUNT WITH GRANT OPTION

   • APPLY MASKING POLICY ON ACCOUNT WITH GRANT OPTION

   • APPLY ROW ACCESS POLICY ON ACCOUNT WITH GRANT OPTION

9. Click Test Connection.

10. If the connection is successful, click Next. If there are any errors, check the connection details and credentials to ensure they are correct and try again.

11. Ensure all the details are correct in the summary and click Complete Setup.

If you attempted the upgrade and receive the message that your upgrade is Partially Complete, find the un-upgraded data sources by navigating to the Upgrade Manager and clicking the number in the Available column for the relevant connection.

Use the options below to resolve those un-upgraded data sources in order to finish your upgrade. See the linked how-to's for more details on the actions to take.

Note that these un-upgraded data sources still exist and are still protected by policy.

1. Delete the remaining data sources: The easiest solution is to delete the data sources that did not upgrade. Note that disabled data sources that no longer exist in your data platform will never be upgraded. Only do this if you no longer need these data sources in Immuta.

2. Adjust the privileges of the system user used to connect Immuta and your data platform: Ensure that the Immuta system user can also access all remaining un-upgraded data sources in your data platform.

   1. Expand permissions in Snowflake or Databricks (recommended): Extend the Immuta system user's permissions in your data platform by granting it access to all remaining un-upgraded data sources.

   2. Change the system user credentials used by Immuta: You can also provide Immuta with a different set of credentials that already have the required permissions on the un-upgraded data sources.

Required Snowflake permissions

Ensure that the role you specified in the upgrade has at least the following permissions:

• USAGE on parent databases and schemas of objects registered as Immuta data sources

• REFERENCES on all objects registered as Immuta data sources

And has been granted to the Immuta system user.

Required Databricks Unity Catalog permissions

Ensure the Databricks service principal you created and connected with Immuta has at least the following permissions:

• USE CATALOG and USE SCHEMA on parent catalogs and schemas of objects registered as Immuta data sources

• SELECT and MODIFY on all objects registered as Immuta data sources

Delete the data sources

Expand permissions in Snowflake

GRANT ROLE <name> TO USER <user_name>;

Expand permissions in Databricks Unity Catalog

Change the system user credentials used by Immuta

If you have another set of credentials on hand with wider permissions, you can edit the connection to use these credentials instead to resolve the un-upgraded data sources.

Requirement: Immuta permission CREATE_DATA_SOURCE

Prerequisite: A connection for Snowflake or Databricks Unity Catalog

Crawl a connection

1. Click Data and select the Infrastructure tab in the navigation menu.

2. Click the more actions menu for the connection you want and select Run Object Sync.

3. Opt to click the checkbox to Also scan inactive objects.

4. Click Run Object Sync.

Crawl a database

1. Click Data and select the Infrastructure tab in the navigation menu.

2. Select the connection.

3. Click the more actions menu in the Action column for the database you want to crawl and select Run Object Sync.

4. Opt to click the checkbox to Also scan inactive objects.

5. Click Run Object Sync.

Crawl a schema

1. Click Data and select the Infrastructure tab in the navigation menu.

2. Select the connection.

3. Select the database.

4. Click the more actions menu in the Action column for the schema you want to crawl and select Run Object Sync.

5. Opt to click the checkbox to Also scan inactive objects.

6. Click Run Object Sync.

Connections are an improvement from the existing process for not only onboarding your data sources but also managing the integration. However, there are some differences between the two processes that should be noted and understood before you start with the upgrade.

1. API changes: See the API changes pages for a complete breakdown of the APIs that will not work once you begin the upgrade. These changes will mostly affect users with automated API calls around schema monitoring and data source registration.

2. Automated data source names: Previously, you could name data sources manually. However, data sources from connections are automatically named using the information (database, schema, table) from your data platform.

3. Schema projects phased out: With integrations, many settings and the connection info for data sources were controlled in the schema project. This functionality is no longer needed with connections and now you can control connection details in a central spot.

4. New hierarchy display: With integrations, tables were brought in as data sources and presented as a flat list on the data source list page. With connections, databases and schemas are displayed as objects too.

5. Change from schema monitoring to object sync: Object metadata synchronization between Immuta and your data platform is no longer optional but always required:

   1. If schema monitoring is off before the upgrade: Once the connection is registered, everything the system user can see will be pulled into Immuta and, if it didn't already exist in Immuta, it will be an inactive object. These inactive objects exist so you can see them, but policy is not protecting the objects, and they will not appear as data sources.

   2. If schema monitoring is on before the upgrade: Once the connection is registered, everything the system user can see will be pulled into Immuta. If it already existed in Immuta, it will be an active object and continue to appear as data source.

Connections allow you to register your data objects in a technology through a single connection, making data registration more scalable for your organization. Instead of registering schema and databases individually, you can register them all at once and allow Immuta to monitor your data platform for changes so that data sources are added and removed automatically to reflect the state of data on your data platform.

Once you register your connection, Immuta presents a hierarchical view of your data that reflects the hierarchy of objects in your data platform:

• Account (Snowflake) or Metastore (Databricks Unity Catalog)

• Database

• Schema

• Tables: These represent the individual objects in your data platform, and when active, become data sources

Beyond making the registration of your data more intuitive, connections provides more control. Instead of performing operations on individual schemas or tables, you can perform operations (such as object sync) at the connection level.

Requirements

See the Snowflake or Databricks Unity Catalog connection registration how-to guides for a list of requirements.

Connection registration and object sync

With connections, you configure the integration and register data sources simultaneously. Once you save your connection, Immuta manages and applies Snowflake or Unity Catalog governance features to data objects registered in Immuta.

Then, Immuta crawls your connection to register all tables within every schema and database that the Snowflake role or Databricks account credentials you provided during the registration has access to. The object metadata, user metadata, and policy definitions are stored in the Immuta metadata database, and this metadata is used to enforce policies for users accessing this data.

After initial registration, your connection can be crawled in two ways:

• Periodic crawl: This crawl happens once every 24 hours. Currently, updating this schedule is not configurable.

• Manual crawl: You can manually trigger a crawl of your connection.

During these subsequent crawls of your connection, Immuta identifies tables, schemas, or databases that have been added or removed. If tables are added, new data sources are created in Immuta. If remote tables are deleted, the corresponding data sources and data objects will be removed from Immuta.

For more information about the Snowflake or Databricks Unity Catalog integration and and how policies are enforced, see the Snowflake integration reference guide or Databricks Unity Catalog integration reference guide.

Data source requests

When there is an active policy that targets the New tag, Immuta sends validation requests to data owners for the following changes made in the remote data platform:

• Column added: Immuta applies the New tag on the column that has been added and sends a request to the data owner to validate if the new column contains sensitive data. Once the data owner confirms they have validated the content of the column, Immuta removes the New tag from it and as a result any policy that targets the New column tag no longer applies.

• Column data type changed: Immuta applies the New tag on the column where the data type has been changed and sends a request to the data owner to validate if the column contains sensitive data. Once the data owner confirms they have validated the content of the column, Immuta removes the New tag from it and as a result any policy that targets the New column tag no longer applies.

• Column deleted: Immuta deletes the column from the data source's data dictionary in Immuta. Then, Immuta sends a request to the data owner to validate the deleted column.

• Data source created: Immuta applies the New tag on the data source that has been newly created and sends a request to the data owner to validate if the new data source contains sensitive data. Once the data owner confirms they have validated the content of the data source, Immuta removes the New tag from it and as a result any policy that targets the New data source tag no longer applies.

For instructions on how to view and manage your tasks and requests in the Immuta UI, see the Manage access requests guide. To view and manage your tasks and requests via the Immuta API, see the Manage data source requests section of the API documentation.

Default settings

When registering a connection, Immuta sets the connection to the recommended default settings to protect your data. The recommended settings are described below:

• Object sync: This setting allows Immuta to monitor the connection for changes. When Immuta identifies a new table, a data source will automatically be created. Similarly, if remote tables are deleted, the corresponding data sources and data objects will be deleted in Immuta. This setting is enabled by default and cannot be disabled.

• Default run schedule: This sets the time interval for Immuta to check for new objects. By default, this schedule is set to 24 hours.

• Sensitive data discovery: This setting enables sensitive data discovery and allows you to select the sensitive data discovery framework that Immuta will apply to your data objects. This setting is enabled by default to use the preconfigured or global framework.

• Impersonation: This setting enable and defines the role for user impersonation in Snowflake. User impersonation is not supported in the Databricks Unity Catalog integration. This setting is disabled by default.

• Project workspaces: This setting enables Snowflake project workspaces. If you use Snowflake secure data sharing with Immuta, enable this setting, as project workspaces are required. If you use Snowflake table grants, disable this setting; project workspaces cannot be used when Snowflake table grants are enabled. Project workspaces are not supported in the Databricks Unity Catalog integration. This setting is disabled by default.

Deregistering a connection

Deregistering a connection automatically deletes all of its child objects in Immuta. However, Immuta will not remove the objects in your Snowflake or Databricks account.

Limitations

• Snowflake and Databricks Unity Catalog are currently the only integrations that support connections

• Databricks Unity Catalog: Delta shares are unsupported.

Related guides

How-to guides

• Register a Snowflake connection

• Register a Databricks connection

Reference guides

• Snowflake integration reference guide

• Databricks Unity Catalog reference guide

• Policies in Immuta

• Data sources in Immuta

Connections allow you to register your data objects in a technology through a single connection, making data registration more scalable for your organization. Instead of registering schema and databases individually, you can register them all at once and allow Immuta to monitor your data platform for changes so that data sources are added and removed automatically to reflect the state of data on your data platform.

How-to guides

• Register a Snowflake connection: Register a connection with a Snowflake account and register the data objects within it.

• Register a Databricks Unity Catalog connection: Register a connection with a Databricks Unity Catalog metastore and register the data objects within it.

• Crawl a connection or object: Trigger a manual crawl of the entire connection or a single object to sync your remote data platform objects with Immuta.

• Use the connection upgrade manager: Complete the upgrade path from the existing native integrations and data sources to a connection.

Reference guides

• Connections: This reference guide discusses the major concepts, design, and settings of connections.

• Upgrading to a connection: This reference guide discusses the differences when upgrading from the existing integrations and data sources to a connection.

Requirements

The following permissions and personas are used in the registration process:

• Immuta permission: CREATE_DATA_SOURCE

• Databricks privileges for the user registering the connection and running the script:

  • Account or workspace admin

  • CREATE CATALOG privilege on the Unity Catalog metastore to create an Immuta-owned catalog and tables

• Databricks privileges for the service principal you create:

  • OWNER privilege on the Immuta catalog you configure.

  • OWNER privilege on catalogs with schemas and tables registered as Immuta data sources so that Immuta can administer Unity Catalog row-level and column-level security controls. This privilege can be applied by granting OWNER on a catalog to a Databricks group that includes the Immuta service principal to allow for multiple owners. If the OWNER privilege cannot be applied at the catalog- or schema-level, each table registered as an Immuta data source must individually have the OWNER privilege granted to the Immuta service principal.

  • USE CATALOG and USE SCHEMA on parent catalogs and schemas of tables registered as Immuta data sources so that the Immuta service principal can interact with those tables.

  • SELECT and MODIFY on all tables registered as Immuta data sources so that the Immuta service principal can grant and revoke access to tables and apply Unity Catalog row- and column-level security controls.

  • USE CATALOG on the system catalog for native query audit.

  • USE SCHEMA on the system.access schema for native query audit.

  • SELECT on the following system tables for native query audit:

    • system.access.audit

    • system.access.table_lineage

    • system.access.column_lineage

Prerequisites

• Unity Catalog metastore created and attached to a Databricks workspace. See the Databricks Unity Catalog reference guide for information on workspaces and catalog isolation support with Immuta.

• Unity Catalog enabled on your Databricks cluster or SQL warehouse. All SQL warehouses have Unity Catalog enabled if your workspace is attached to a Unity Catalog metastore. Immuta recommends linking a SQL warehouse to your Immuta tenant rather than a cluster for both performance and availability reasons.

Register a connection

1. Click Data and select the Infrastructure tab in the navigation menu.

2. Click the + Add Host button.

3. Select the Databricks data platform tile.

4. Enter the connection information:

   • Host: The hostname of your Databricks workspace.

   • Port: Your Databricks port.

   • HTTP Path: The HTTP path of your Databricks cluster or SQL warehouse.

   • Immuta Catalog: The name of the catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.

   • Connection Key: The connection key represents the unique name of your connection and will be used as prefix in the name for all data objects associated with this connection. It will also appear as the display name in the UI and will be used in all API calls made to update or delete the connection.

5. Click Next.

6. Select Access Token authentication method from the dropdown menu.

7. Enter the Access Token in the Immuta System Account Credentials section. This is the access token for the Immuta service principal. This service principal must have the metastore privileges listed in the requirements section at the top of this page for the metastore associated with the Databricks workspace. If this token is configured to expire, update this field regularly for the integration to continue to function. This authentication information will be included in the script populated later on the page.

8. Copy the provided script and run it in Databricks as a user with the CREATE CATALOG privilege on the Unity Catalog metastore.

9. Click Validate Connection.

10. If the connection is successful, click Next. If there are any errors, check the connection details and credentials to ensure they are correct and try again.

11. Ensure all the details are correct in the summary and click Complete Setup.

Connections allow you to register your data objects in a technology through a single connection, making data registration more scalable for your organization. Instead of registering schema and databases individually, you can register them all at once and allow Immuta to monitor your data platform for changes so that data sources are added and removed automatically to reflect the state of data on your data platform.

Native integrations

Supported technology and authorization methods

Snowflake

• Snowflake OAuth

• Username and password

• Key pair

Databricks

• Token

*M2M OAuth is not yet supported.

Supported features

The tables below outline Immuta features, their availability with integrations, and their availability with connections.

Snowflake

Databricks Unity Catalog

Data sources

Supported object types

The following object types are supported for data sources registered with connections. However, when applying read and write access policies to these data sources, the privileges granted by Immuta vary depending on the object type. See an outline of privileges granted by Immuta on Snowflake and Databricks Unity Catalog object types on the Subscription policy access types page.

Snowflake

• Table

• View

• Materialized view

• External table

• Event table

• Iceberg table

• Dynamic table

Databricks Unity Catalog

• Table

• View

• Materialized view

• Streaming table

• External table

Hierarchy

With connections, your data sources are ingested and presented to reflect the infrastructure hierarchy of your connected data platform. For example, this is what the new hierarchy will look like for a Snowflake connection:

Tags

Consideration

The V2 /data endpoint allows users to register data sources and attach a tag automatically when the data sources are registered in Immuta.

The V2 /data endpoint is not supported with a connection, and there is no substitution for this behavior at this time. If you require default tags for newly onboarded data sources, please reach out to your Immuta support professional before upgrading.

Users and permissions

With integrations

With connections

Schema monitoring

During object sync, Immuta crawls your connection to ingest metadata for every database, schema, and table that the Snowflake role or Databricks account credentials you provided during the configuration has access to. Upon completion of the upgrade, the tables' states depend on your previous schema monitoring settings:

• If you had schema monitoring enabled on a schema: All tables from that schema will be registered in Immuta as active data sources.

• If you had schema monitoring disabled on a schema: All tables from that schema (that were not already registered in Immuta) will be registered as inactive data sources. They are visible from the infrastructure view, but are not listed as data sources until they are activated.

After the initial upgrade, your connection is periodically crawled every 24 hours to keep your tables in Immuta in sync. Additionally, users can also manually crawl metadata via the UI or API.

Additional settings

Object sync provides additional controls compared to schema monitoring:

• Object status: Connections, databases, schemas and tables can be marked active, which for tables make them appear as data sources, or inactive. These statuses are inherited to all lower objects by default, but that can be overridden. For example, if you make a database inactive, all schemas and tables within that database will also be inactive. However, if you want one of those tables to be a data source, you can manually activate it.

• Activate new data objects: This setting controls what state new objects are registered as in Immuta when found by object sync.

  • Active: New data objects found by object sync will automatically be activated and tables will be registered as data sources.

  • Inactive: This is the default. New data objects found by object sync will be inactive.

Comparison

Databricks Unity Catalog

Data sources with integrations, required users to manually create the schema monitoring job in Databricks. However, this job has been fully automated on data sources with connections, and this step is no longer necessary.

APIs

Consolidating integration setup and data source registration into a single connection significantly simplifies programmatic interaction with the Immuta APIs. Actions that used to be managed through multiple different endpoints can now be achieved through one simple and standardized one. As a result, multiple API endpoints are blocked once a user has upgraded their connection.

All blocked APIs will send an error indicating "400 Bad Request - [...]. Use the /data endpoint." This error indicates that you will need to update your processes that are calling the Immuta APIs to leverage the new /data endpoint instead. For details, see the API changes page.