Metadata Ingestion
Last updated
Was this helpful?
Last updated
Was this helpful?
For Immuta to enforce policies, it needs to catalog the resources policies are being applied to by performing metadata ingestion. Metadata ingestion is the process that occurs when you create an Immuta data source where Immuta gathers details about your tables. However, Immuta does not need access to the data within the tables in order to protect it, with the exception of a few specific and advanced masking policies detailed below.
Immuta collects and stores the following kinds of information in Immuta's Metadata Database for policy enforcement. Further, policy information may be transmitted to data source host systems for enforcement purposes as part of a query or to enable the host system to perform policy enforcement.
Identity Management Information: Usernames, group information, and other kinds of personal identifiers may be stored and referenced for the purposes of performing authentication and access control and may be retained in audit logs. When such information is relevant for access determination under policy, it may be retained as part of the policy definition.
Schema Information: Data source metadata such as schema, column data types, and information about the host.
Immuta's Metadata Database can also contain the following forms of metadata for policy enforcement. These forms contain sample data from your tables and can be disabled if you do not want Immuta to have access to the data being protected.
Fingerprints: When enabled, additional statistical queries made during the health check are distilled into summary statistics, called fingerprints. During this process, statistical query results and data samples (which may contain PII) are temporarily held in memory by the Fingerprint Service.
k-Anonymization Policies: When a k-anonymization policy is applied, the columns under the k-anonymization policy are queried within a separate fingerprinting process which generates rules enforcing k-anonymity. The results of this query, which may contain PII, are temporarily held in memory by the Fingerprint Service. The final rules are stored for enforcement. Immuta requires that you opt in to use this masking policy type. To enable k-anonymization for your account, see the .
Randomized Response Policies: If the list of substitution values for a categorical column is not part of the policy specification (e.g., when specified via the API), a list is obtained via query and merged into the policy definition.
If no metadata collection types have been disabled, data is processed in the following workflow to support data source creation, health checks, policy enforcement, and dictionary features.
A System Administrator configures the integration in Immuta.
A Data Owner registers data sources from their remote data platform with Immuta. Note: Data Owners can see sample data when editing a data source. However, this action requires the database password, and the small sample of data visible is only displayed in the UI and is not stored in Immuta.
When a data source is created or updated, the Metadata Database pulls in and stores statistics about the data source, including row count and high cardinality calculations.
The data source health check runs daily to ensure existing tables are still valid.
If an external catalog is enabled, the daily health check will pull in data source attributes (e.g., tags and definitions) and store them in the Metadata Database.
Immuta requires certain privileges to perform metadata ingestion. The user connecting a table to Immuta as a data source must have privileges specific to their data platform to perform metadata ingestion.
For example, a user registering a Snowflake table as an Immuta data source must have the REFERENCES privilege to view the structure of the table and allow Immuta access to that information as well. This does not require the user (or Immuta) to have access to view the data itself.