Ctrlk
Book a demoStatus page
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Data Sources Reference Guide

Learn about what a data source is and which registration method to use for your data platform

Data owners expose their data across their organization to other users by registering that data in Immuta as a data source. Data sources are collections of metadata about your tables or data objects and allow for Immuta actions like the following:

  • Apply tags to data sources to enforce access controls

  • Apply data policies to a data source's columns

  • Restrict the users who can query a data source with subscription policies

  • Gather your data sources into various domains for delegation

  • Publish data products containing your data sources in the Request app

When data is registered, Immuta does not affect existing policies on those tables in the remote system for non-Immuta users, so users who had access to a table before it was registered can still access that data without interruption. However, this behavior is different for Immuta users on an integration-by-integration basis, so see the integration reference guides for more details.

Data source registration

For policies to properly apply to data sources, there must be an integration configured in Immuta, and that integration's connection details must match the data source's connection details. This allows for Immuta to natively enforce policies on that table in your data platform. There are two ways to configure integrations and register data based on your technology.

Schema projects

Schema projects are automatically created and managed by Immuta when data sources are created. They group all the data sources of the schema, and when new data sources are created, manually or with schema monitoring, they are automatically added to the schema project. They work as a tool to organize all the data sources within a schema, which is particularly helpful with schema monitoring enabled.

Schema projects are created when tables are registered as data sources in Immuta. The user creating the data source does not need the CREATE_PROJECT permission to have the project auto-create because no data sources can be added by the owner. Instead, new data sources are managed by Immuta. The user can manage subscription policies for schema projects, but they cannot apply data policies or purposes to them.

The schema settings, such as schema evolution and connection information, can be edited from the project overview tab. Note: Deleting the project will delete all of the data sources within it as well.

Schema project actions

Schema settings are edited from the project overview tab:

  • Schema project connection details: Editing these details will update them for all the data sources within the schema project.

  • Data source naming convention: When schema monitoring is enabled, new data sources will be automatically detected and added to the schema project. Updating the naming convention will change how these newly detected data sources are named by Immuta.

  • Schema detection owner: When schema monitoring is enabled, a user is assigned to be the owner of any detected and Immuta created data source.

  • Disable or delete your schema project: Deleting the project will delete all of the data sources within it as well.

Data source health checks

When an Immuta data source is created, background jobs use the connection information provided to compute health checks dependent on the type of data source created and how it was configured. These data source health checks include the following options depending on the technology and registration method:

Status
Description
Action
Supported technologies
Registration type

External catalog

Indicates whether or not the external catalog was successfully linked to the data source. This status will appear if you have an external catalog configured.

Re-run

All

All

Fingerprint

Indicates whether or not the data source fingerprint was successfully generated.

Re-run

Snowflake

Connections

Identification

Indicates whether identification was successfully run on the data source.

Re-run

All

All

Row count

Indicates whether the number of rows in the data source was successfully calculated.

Re-run

All

All

High cardinality column

Indicates whether the data source's high cardinality column was successfully calculated.

Re-run

All

All

Global policies applied

Indicates whether global policies were successfully applied to the data source.

No action

See the integrations support matrix for the technologies that support policies.

All

Object sync

Indicates whether object sync has successfully run on the data source data object.

Re-run

See the integrations support matrix for the technologies on connections.

Connections

Schema detection

Indicates whether the job run to determine if a remote table was added or removed from the schema was successful.

No action

  • Redshift Spectrum

  • Azure Synapse Analytics

  • Google BigQuery

Legacy onboarding

Column detection

Indicates whether the job run to determine if a column was added or removed from the remote table registered as an Immuta data source was successful.

Trigger Detection

  • Redshift Spectrum

  • Azure Synapse Analytics

  • Google BigQuery

Legacy onboarding

Framework classification

Indicates whether classification was successfully run on the data source to determine the sensitivity of the data source.

Re-run

All

All

Databricks

Indicates whether Databricks governance policies have been successfully synced.

Sync all policies

Databricks

Connections

Snowflake

Indicates whether Snowflake governance policies have been successfully synced.

Sync data policies

Snowflake

Connections

  • Redshift View

  • Azure Synapse Analytics View

  • Google BigQuery View

Indicates whether views were properly created for tables registered in Immuta.

Create

  • Redshift Spectrum

  • Azure Synapse Analytics

  • Google BigQuery

Legacy onboarding

After these jobs complete, the health status for each is updated to indicate whether the status check passed, was skipped, is unknown, or failed.

These background jobs can be disabled during data source creation by adding a specific tag to prevent automatic table statistics. This prevent statistics tag can be set on the app settings page by a system administrator. However, with automatic table statistics disabled these policies will be unavailable until the data source owner manually generates the fingerprint (available only for Snowflake data sources):

  • Masking with format preserving masking

  • Masking using randomized response

Unhealthy Databricks data sources

Unhealthy data sources may fail their row count queries if they run against a cluster that has the Databricks query watchdog enabled.

Health check limitations

Data sources with over 1600 columns will not have health checks run, but will still appear as healthy. The health check cannot be run automatically or manually.

Data source management

Schema monitoring

With schema monitoring enabled, Immuta monitors your organization's servers to find when new tables or columns are created or deleted and automatically registers (or disables) those tables in Immuta.

See the Schema monitoring section for more details.

Data source user roles

There are various roles users and groups can play relating to each data source. These roles are managed through the members tab of the data source. Roles include the following types:

  • Owners: Those who create and manage new data sources and their users, documentation, column tags, and column descriptions.

  • Subscribers: Those who have access to the data source data. With the appropriate data accesses and attributes, these users and groups can view files, run queries, and generate analytics against the data source data. All users and groups granted access to a data source have subscriber status.

  • Experts: Those who are knowledgeable about the data source data and can elaborate on it. They are responsible for managing the data source's documentation, column tags, and column descriptions.

See Manage data source members for a tutorial on modifying user roles.

Data source columns

The columns tab provides information about the columns within the data source, including column names and value types.

Columns are automatically generated when the data source is created. However, data owners and experts can tag columns and add descriptions to these entries.

Column icons

The columns tab displays icons on columns that have a masking policy applied to them. The appearance of these icons varies depending on the permission of the user.

Governors and data owners

If you have the GOVERNANCE permission or are the data source owner, the column icons will appear in these ways:

  • No icon: No masking policy applies to the column.

  • Yellow eye: A masking policy applies to the column, but the column is unmasked for the current user because they meet the exception criteria for the policy.

  • Red eye: A policy on the column masks it for the current user.

All other users

The column icons will appear in these ways for all other users:

  • No icon: Either no masking policy applies to the column or a masking policy applies to the column, but the column is unmasked for the current user because they meet the exception criteria for the policy.

  • Red eye: A policy on the column masks it for the current user.

Nested columns

You can create Databricks Spark data sources with nested columns when you enable complex data types. When complex types are enabled, Databricks Spark data sources can have columns that are arrays, maps, or structs that can be nested.

Audit

The following events related to data sources are audited and can be found on the audit page in the UI:

PreviousRun Schema Monitoring and Column Detection JobsNextDisable Immuta from Sampling Raw Data

Last updated

Was this helpful?