Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Public preview
This feature is public preview and available to select accounts. Reach out to your Immuta support professional to enable it on your tenant.
Databricks Unity Catalog
Snowflake
An integration enabled on the Immuta app settings page
Data sources registered
Immuta global GOVERNANCE
permission
To complete your upgrade,
Select Upgrade Manager in the navigation menu. This tab will only be available if you have integrations ready for upgrade.
Click Start Upgrade.
Enter a Connection Key. The connection key represents the unique name of your connection and will be used as prefix in the name for all data objects associated with this connection. It will also appear as the display name in the UI and will be used in all API calls made to update or delete the connection.
Click Next.
Ensure Immuta has the correct credentials to connect to Databricks Unity Catalog or Snowflake. Select the tab below for more information:
Click Validate Credentials to ensure the access token can connect Immuta and Databricks Unity Catalog.
Create a Snowflake role with a minimum of the following permissions:
USAGE
on all databases and schemas with registered data sources
REFERENCES
on all tables and views registered in Immuta
Grant the new Snowflake role to the in your Snowflake environment.
Enter the new Snowflake role in the textbox.
Click Validate Credentials to ensure the role has been granted to the right user.
Click Next.
Click Upgrade Connection.
Click the link to the docs to understand the impacts of the upgrade.
Click the checkbox to confirm understanding of the upgrade effects, and click Yes, Upgrade Connection.
Connections allow you to register your data objects in a technology through a single connection, making data registration more scalable for your organization. Instead of registering schema and databases individually, you can register them all at once and allow Immuta to monitor your data platform for changes so that data sources are added and removed automatically to reflect the state of data on your data platform.
Register a Snowflake connection: Register a connection with a Snowflake account and register the data objects within it.
Register a Databricks Unity Catalog connection: Register a connection with a Databricks Unity Catalog metastore and register the data objects within it.
Crawl a connection or object: Trigger a manual crawl of the entire connection or a single object to sync your remote data platform objects with Immuta.
Use the connection upgrade manager: Complete the upgrade path from the existing integrations and data sources to a connection.
Connections: This reference guide discusses the major concepts, design, and settings of connections.
Upgrading to a connection: This reference guide discusses the differences when upgrading from the existing integrations and data sources to a connection.
If you attempted the upgrade and receive the message that your upgrade is Partially Complete, find the un-upgraded data sources by navigating to the Upgrade Manager and clicking the number in the Available column for the relevant connection.
Use the options below to resolve those un-upgraded data sources in order to finish your upgrade. See the linked how-to's for more details on the actions to take.
Note that these un-upgraded data sources still exist and are still protected by policy.
Delete the remaining data sources: The easiest solution is to delete the data sources that did not upgrade. Note that disabled data sources that no longer exist in your data platform will never be upgraded. Only do this if you no longer need these data sources in Immuta.
Adjust the privileges of the system user used to connect Immuta and your data platform: Ensure that the Immuta system user can also access all remaining un-upgraded data sources in your data platform.
Expand permissions in Snowflake or Databricks (recommended): Extend the Immuta system user's permissions in your data platform by granting it access to all remaining un-upgraded data sources.
Change the system user credentials used by Immuta: You can also provide Immuta with a different set of credentials that already have the required permissions on the un-upgraded data sources.
Ensure that has at least the following permissions:
USAGE
on parent databases and schemas of objects registered as Immuta data sources
REFERENCES
on all objects registered as Immuta data sources
And has been granted to the .
Ensure the Databricks service principal you created and connected with Immuta has at least the following permissions:
USE CATALOG
and USE SCHEMA
on parent catalogs and schemas of objects registered as Immuta data sources
SELECT
and MODIFY
on all objects registered as Immuta data sources
Schema monitoring must be turned off in the schema project to disable and delete data sources that did not upgrade.
View the data sources that were not upgraded
Find the un-upgraded data sources by navigating to the Upgrade Manager and clicking the number in the Available column.
Disable the data sources
From this data source list page, disable all the data sources to delete.
Check the top checkbox in the data source list table. Deselect the checkbox for any data sources you do not want to delete.
Click More Actions.
Click Disable and then Confirm.
Delete the data sources
From this data source list page, delete the data sources.
Check the top checkbox in the data source list table. Deselect the checkbox for any data sources you do not want to delete.
Click More Actions.
Click Disable and then Confirm.
Finalize the upgrade
Once the un-upgraded data sources are deleted, you should be able to complete the upgrade.
Navigate to the Upgrade Manager.
Click Finalize.
Check your role permissions
To find the role you specified, do the following in the Immuta UI:
Navigate to Connections.
Select the connection you are trying to upgrade.
Navigate to the Connections tab.
See the Role.
Now, ensure that role has the required permissions for each data source that was not successfully upgraded. Add the permissions where needed.
Grant your role to the system account
To find the system account you specified, do the following in the Immuta UI:
Navigate to Connections.
Select the connection you are trying to upgrade.
Navigate to the Connections tab.
See the Setup: Username.
Now, in Snowflake, grant the role to the system account:
Run object sync
Navigate to Connections.
Click on the more actions menu for the connection you are trying to upgrade.
Select Run Object Sync.
Click the checkbox to Also scan inactive objects.
Click Run Object Sync.
Now, navigate back to the Upgrade Manager tab, and if all your data sources are successfully upgraded, finalize the upgrade.
Finalize the upgrade
Once the un-upgraded data sources are resolved, you can complete the upgrade.
Navigate to the Upgrade Manager.
Click Finalize.
Check your service principal privileges
To find the service principal you specified, do the following in the Immuta UI:
Navigate to Connections.
Select the connection you are trying to upgrade.
Navigate to the Connections tab.
Now, ensure that service principal has the required privileges for each data source that was not successfully upgraded. Add the privileges where needed.
Run object sync
Navigate to Connections.
Click on the more actions menu for the connection you are trying to upgrade.
Select Run Object Sync.
Click the checkbox to Also scan inactive objects.
Click Run Object Sync.
Now, navigate back to the Upgrade Manager tab, and if all your data sources are successfully upgraded, finalize the upgrade.
Finalize the upgrade
Once the un-upgraded data sources are resolved, you can complete the upgrade.
Navigate to the Upgrade Manager.
Click Finalize.
If you have another set of credentials on hand with wider permissions, you can edit the connection to use these credentials instead to resolve the un-upgraded data sources.
Edit the connection
Navigate to Connections.
Select the connection you are trying to upgrade.
Navigate to the Connections tab.
Click Edit and then Next
Enter the new credentials in the textbox and continue to the end to save.
Run object sync
Navigate to Connections.
Click on the more actions menu for the connection you are trying to upgrade.
Select Run Object Sync.
Click the checkbox to Also scan inactive objects.
Click Run Object Sync.
Now, navigate back to the Upgrade Manager tab, and if all your data sources are successfully upgraded, finalize the upgrade.
Finalize the upgrade
Once the un-upgraded data sources are resolved, you can complete the upgrade.
Navigate to the Upgrade Manager.
Click Finalize.
Connections are an improvement from the existing process for not only onboarding your data sources but also managing the integration. However, there are some differences between the two processes that should be noted and understood before you start with the upgrade.
API changes: See the API changes pages for a complete breakdown of the APIs that will not work once you begin the upgrade. These changes will mostly affect users with automated API calls around schema monitoring and data source registration.
Automated data source names: Previously, you could name data sources manually. However, data sources from connections are automatically named using the information (database, schema, table) from your data platform.
Schema projects phased out: With integrations, many settings and the connection info for data sources were controlled in the schema project. This functionality is no longer needed with connections and now you can control connection details in a central spot.
New hierarchy display: With integrations, tables were brought in as data sources and presented as a flat list on the data source list page. With connections, databases and schemas are displayed as objects too.
Change from schema monitoring to object sync: Object metadata synchronization between Immuta and your data platform is no longer optional but always required:
If schema monitoring is off before the upgrade: Once the connection is registered, everything the system user can see will be pulled into Immuta and, if it didn't already exist in Immuta, it will be an inactive object. These inactive objects exist so you can see them, but policy is not protecting the objects, and they will not appear as data sources.
If schema monitoring is on before the upgrade: Once the connection is registered, everything the system user can see will be pulled into Immuta. If it already existed in Immuta, it will be an active object and continue to appear as data source.
Create a single data source
Step 1: Ensure your system user has been granted access to the relevant object in the data platform.
Step 2: Wait until the next object sync or manually trigger a metadata crawl using .
Step 3: If the parent schema has activateNewChildren: false
,
with settings: isActive: true
.
Bulk create data sources
Step 1: Ensure your system user has been granted access to the relevant object in the data platform.
Step 2: Wait until the next object sync or manually trigger a metadata crawl using .
Step 3: If the parent schema has activateNewChildren: false
,
with settings: isActive: true
.
Edit a data source connection
No substitute. Data sources no longer have their own separate connection details but are tied to the parent connection.
Bulk edit data source's connections
No substitute. Data sources no longer have their own separate connection details but are tied to the parent connection.
Run schema detection (object sync)
Delete a data source
Bulk delete data sources
Enable a single data source
with settings: isActive: true
Bulk enable data sources
with settings: isActive: true
Disable a single data source
with settings: isActive: false
Bulk disable data sources
with settings: isActive: false
Edit a data source name
No substitute. Data source names are automatically generated based on information from your data platform.
Edit a connection key
No substitute. Data sources no longer have their own separate connection details but are tied to the parent connection.
Override a host name
No substitute. Data sources no longer have their own separate connection details but are tied to the parent connection.
Create an integration/connection
Update an integration/connection
Delete an integration/connection
Delete and update a data dictionary
PUT
No substitute. Data source dictionaries are automatically generated based on information from your data platform.
Update a data source owner
with settings: dataOwners
Response to a data source owner request
with settings: dataOwners
Requirement: GOVERNANCE
global permission or Infrastructure Admin or Data Owner within the hierarchy
Prerequisite: A connection for Snowflake or Databricks Unity Catalog
Click Data and select the Connections tab in the navigation menu.
Click the more actions menu for the connection you want and select Run Object Sync.
Opt to click the checkbox to Also scan inactive objects.
Click Run Object Sync.
Click Data and select the Connections tab in the navigation menu.
Select the connection.
Click the more actions menu in the Action column for the database you want to crawl and select Run Object Sync.
Opt to click the checkbox to Also scan inactive objects.
Click Run Object Sync.
Click Data and select the Connections tab in the navigation menu.
Select the connection.
Select the database.
Click the more actions menu in the Action column for the schema you want to crawl and select Run Object Sync.
Opt to click the checkbox to Also scan inactive objects.
Click Run Object Sync.
Public preview
This feature is public preview and available to select accounts. Reach out to your Immuta support professional to enable it on your tenant.
Connections allow you to register your data objects in a technology through a single connection, making data registration more scalable for your organization. Instead of registering schema and databases individually, you can register them all at once and allow Immuta to monitor your data platform for changes so that data sources are added and removed automatically to reflect the state of data on your data platform.
Once you register your connection, Immuta presents a hierarchical view of your data that reflects the hierarchy of objects in your data platform:
Account (Snowflake) or Metastore (Databricks Unity Catalog)
Database
Schema
Tables: These represent the individual objects in your data platform, and when active, become data sources
Beyond making the registration of your data more intuitive, connections provides more control. Instead of performing operations on individual schemas or tables, you can perform operations (such as object sync) at the connection level.
See the Snowflake or Databricks Unity Catalog connection registration how-to guides for a list of requirements.
Immuta will ensure the objects in your database stay synchronous with the registered objects in Immuta. To do this, Immuta uses the account credentials provided during registration to check the remote technology for object changes like a table being created, new columns being added to a table, or a table being deleted.
If tables are added, new data sources are created in Immuta. If remote tables are deleted, the corresponding data sources and data objects will be removed from Immuta. And if a column changes in a table, those changes will be reflected in the Immuta data source data dictionary.
Your connection can be synced in two ways:
Periodic object sync: This happens once every 24 hours (at 1:00 AM UTC). Currently, updating this schedule is not configurable.
Manual object sync: You can manually run object sync on your connection.
When new columns are detected and added to Immuta, they will be automatically tagged with the New
tag. This allows governors to use the seeded New Column Added
global policy to mask columns with the New
tag, since they could contain sensitive data.
The New Column Added
global policy is staged (inactive) by default. See the Clone, activate, or stage a global policy guide to activate this seeded global policy if you want any columns with the New
tag to be automatically masked.
Without connections, schema monitoring would also tag new data sources with the New
tag. However this behavior is exclusive to schema monitoring and will not happen with object sync. Object sync only tags new columns of known data sources with the New
tag.
When there is an active policy that targets the New
tag, Immuta sends validation requests to data owners for the following changes made in the remote data platform:
Column added: Immuta applies the New
tag on the column that has been added and sends a request to the data owner to validate if the new column contains sensitive data. Once the data owner confirms they have validated the content of the column, Immuta removes the New
tag from it and as a result any policy that targets the New
column tag no longer applies.
Column deleted: Immuta deletes the column from the data source's data dictionary in Immuta. Then, Immuta sends a request to the data owner to validate the deleted column.
For instructions on how to view and manage your tasks and requests in the Immuta UI, see the Manage access requests guide. To view and manage your tasks and requests via the Immuta API, see the Manage data source requests section of the API documentation.
When registering a connection, Immuta sets the connection to the recommended default settings to protect your . The recommended settings are described below:
Object sync: This setting allows Immuta to monitor the connection for changes. When Immuta identifies a new table, a data source will automatically be created. Similarly, if remote tables are deleted, the corresponding data sources and data objects will be deleted in Immuta. This setting is enabled by default and cannot be disabled.
Default run schedule: This sets the time interval for Immuta to check for new objects. By default, this schedule is set to 24 hours.
Sensitive data discovery: This setting enables sensitive data discovery and allows you to select the sensitive data discovery framework that Immuta will apply to your data objects. This setting is enabled by default to use the preconfigured or global framework.
Impersonation: This setting enable and defines the role for user impersonation in Snowflake. User impersonation is not supported in the Databricks Unity Catalog integration. This setting is disabled by default.
Project workspaces: This setting enables Snowflake project workspaces. If you use Snowflake secure data sharing with Immuta, enable this setting, as project workspaces are required. If you use Snowflake table grants, disable this setting; project workspaces cannot be used when Snowflake table grants are enabled. Project workspaces are not supported in the Databricks Unity Catalog integration. This setting is disabled by default.
Deregistering a connection automatically deletes all of its child objects in Immuta. However, Immuta will not remove the objects in your Snowflake or Databricks account.
Snowflake and Databricks Unity Catalog are currently the only integrations that support connections
Databricks Unity Catalog: Delta shares are unsupported.