Public Preview
This feature is currently in public preview and available to all accounts.
Native query audit for Unity Catalog captures user data access activity within Unity Catalog as Immuta audit logs. Immuta audits the activity of all Unity Catalog users and tables. Multiple access options are supported for audit:
Cluster queries with the following supported languages: SQL, Scala, Python, and R.
SQL Warehouse queries
Deprecation notice
Support for the audit endpoint and UI has been deprecated. Instead, pull audit logs from Kubernetes and push them to your SIEM.
Databricks Unity Catalog integration with native query audit enabled
A Databricks deployment with system tables capabilities
Best practices: Store audit logs
By default most Immuta audit logs expire after 60 days, so store audit logs outside of Immuta in order to retain the audits long term.
The following audit record types do not expire after 60 days: blobFetch, dataSourceSubscription, globalPolicyApproved, globalPolicyApprovalRescinded, globalPolicyChangeRequested, globalPolicyConflictResolved, globalPolicyCreate, globalPolicyDelete, globalPolicyDisabled, globalPolicyUpdate, policyExemption, policyHandlerCreate, policyHandlerUpdate, prestoQuery, spark, and sqlQuery.
Immuta collects audit records at the frequency configured when enabling the integration, which is between 1 and 24 hours. The frequency is a global setting based on integration type, so organizations with multiple Unity Catalog integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records; however, there could be performance and cost impacts from the frequent jobs.
To manually request native query audit ingestion, click Load Audit Events on the Immuta audit page.
Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below. These audit logs are stored with the recordType: nativeQuery.
Enrichment of audit logs with Immuta entitlements information is not supported. While you will see these entitlements in the Databricks Spark audit logs, the following will not be in the native query audit for Unity Catalog:
Immuta policies information
User attributes
Groups
Immuta determines unauthorized events based on error messages within Unity Catalog records. When the error messages contain expected language, unauthorized events will be available for native query audit for Unity Catalog. In other cases, it is not possible to determine the cause of an error.
Audit for cluster queries do not support UNAUTHORIZED status. If a cluster query is unauthorized, it will show FAILURE.
Data source information will be provided when available:
For some queries, Databricks Unity Catalog does not report the target data source for the data access operation. In these cases the activity is audited, yet the audit record in Immuta will not include the target data source information.
Data source information is not available for unauthorized queries and events.
Column information from the query is not currently supported.
The cluster for the Unity Catalog integration must always be running for Immuta to audit activity.
Immuta audit records include unregistered data sources and users; however, activity from them will not appear in any governance reports.
Deprecation notice
Support for the audit endpoint and UI has been deprecated. Instead, pull audit logs from Kubernetes and push them to your SIEM.
In addition to the executed Spark plan, the tables, and the tables' underlying paths for every audited Spark job, Immuta captures the code or query that triggers the Spark plan. Immuta audits the activity of Immuta users on Immuta data sources.
Databricks users registered as Immuta users: Note that the users' Databricks usernames must be mapped to Immuta. Without this, Immuta will not know the users are Immuta users and will not collect audit events for their data access activity.
Best Practices: Store Audit Records
By default Immuta audit records expire after 60 days, so store audit records outside of Immuta in order to retain the audits long term.
Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below. These audit records are stored with the recordType: spark.
| Property | Description | Example |
|---|---|---|
Below is an example of the queryText, which contains the full notebook cell (since the query was the result of a notebook). If the query had been from a JDBC connection, the queryText would contain the full SQL query.
This notebook cell had multiple audit records associated with it, but the example audit record in the tab to the right corresponds to the filteredDf.write.saveAsTable('{}.audit_cell'.format(testDb)) line.
This example audit record contains two fields under extra:
queryText: The queryText will contain either the full notebook cell (when the query is the result of a notebook) or the full SQL query (when it is a query from a JDBC connection).
queryLanguage: The queryLanguage corresponds to the programming language used: SQL, Python, Scala, or R. Audited JDBC queries will indicate that it came from JDBC here.
Beyond raw audit events (such as “John Doe queried Table X in Databricks"), the Databricks audit records include the policy information enforced during the query execution, even if a query was denied.
Queries will be denied if at least one of the conditions below is true:
User does not meet policy conditions.
User is not subscribed to the data source.
Data source is not in the user's current project.
Data source is in the user's current project, but the user is not subscribed to the data source.
Data source is not registered in Immuta.
Access denied audit records include details about the user (accessControls.entitlements) and the Subscription policy (accessControls.policySet) that blocked access to the table.
The user's entitlements represent the state at the time of the query. This includes the following fields:
project: The user's current project.
attributes: The user's attributes.
groups: The user's groups.
impersonatedUsers: The user that the current user is impersonating.
The policySet includes the following fields:
subscriptionPolicyType: The type of Subscription policy (such as MANUAL, ADVANCED, or ENTITLEMENTS).
type: Indicates whether the policy is a SUBSCRIPTION or DATA policy. Query denied records will always be a Subscription policy type.
ruleAppliedForUser: Indicates whether or not the policy was applied for the user. If false, the user was an exception to the policy.
rationale: The policy rationale written by the policy creator.
global: Indicates whether or not the policy was a Global policy. When false, the policy is Local.
mergedPolicies: Shows the policy information for each of the merged Global Subscription policies.
Query Scenario
Before examining the audit log excerpt below, review the user's entitlements and the policy on the patient_transactions data source.
User Entitlements
groups: none
attributes: SpecialAccess.Addresses, OfficeLocation.Maryland
current project: Medical Claims
Data Source Policy
Subscription Policy: Allow individual users you select to access the data source.
The Databricks user described above attempts to query a patient_transactions table, which has not been added to a project. The user's query is denied, and the audit logs reveal the policy details. Focus specifically on the entitlements, policySet, actionStatus, and actionStatusReason sections below:
Although the user was subscribed to the data source, the project field shows that this user is currently working under the Medical Claims project, and the actionStatusReason indicates that the data source has not been added to that project. Consequently, the actionStatus field shows the query was UNAUTHORIZED.
The user's entitlements includes the following fields:
project: The user's current project.
attributes: The user's attributes.
groups: The user's groups.
impersonatedUsers (when relevant): The username the current user is impersonating.
For a successful query, policySet includes all policies applied to the data source:
subscriptionPolicyType: The type of Subscription policy (such as MANUAL, ADVANCED, or ENTITLEMENTS).
type: Indicates whether the policy is a SUBSCRIPTION or DATA policy.
ruleAppliedForUser: Indicates whether or not a policy was applied for the user. If false, the user was an exception to the policy.
rationale: Provides the rationale given for the policy by the policy creator.
global: Indicates whether or not the policy was a Global policy. When false, the policy is Local.
mergedPolicies: Shows the policy information for each of the merged Global Subscription policies.
Query Scenario
Before examining the audit log excerpt below, review the user's entitlements and the policy on the patients data source.
User Entitlements
groups: none
attributes: SpecialAccess.Addresses, OfficeLocation.Maryland
current project: None
Data Source Policies
Subscription Policy: Allow individual users you select to access the data source.
Data Policy: Mask by making null the value in the column(s) lastname for everyone.
Data Policy: Mask by hashing the value in the column(s) address for everyone except users who possess the attribute SpecialAccess.Addresses.
The Databricks user described above attempts to query a patients table. The user's query is successful, and the audit logs reveal the policy details. Focus specifically on the entitlements, policySet, and actionStatus sections below:
This user is subscribed to the data source and the actionStatus shows their query was a SUCCESS.
The first masking policy in policySet masks the field lastname to NULL for everyone, as no exceptions are listed. Furthermore, the ruleAppliedForUser field is true, indicating that the querying user sees NULL values for this column.
However, the ruleAppliedForUser for the second masking policy in the policySet is false, indicating that the querying user can see the values in the address column in the clear because one of their entitlements is listed under exceptions to the policy: SpecialAccess.Addresses.
Query Scenario
Before examining the audit log excerpt below, review the user's entitlements and the policy on the maryland_employees data source.
User Entitlements
groups: none
attributes: SpecialAccess.Addresses, OfficeLocation.Maryland
current project: None
Data Source Policy
Merged Subscription Policy: Allow users to subscribe if they have the attribute OfficeLocation.Maryland or are a member of the group Human Resources.
The Databricks user described above attempts to query the maryland_employees table. The user's query is successful, and the audit logs reveal the policy details. Focus specifically on the entitlements, mergedPolicies, and actionStatus sections below:
This user is subscribed to the data source and the actionStatus shows their query was a SUCCESS.
The mergedPolicies section illustrates both Global policies that were merged and applied to the data source. Although the querying user is not a member of the group Human Resources, the ruleAppliedForUser field is still true and they are subscribed to the data source because they met one requirement of the subscription policy: they possess the attribute OfficeLocation.Maryland.
Deprecation notice
Support for the audit endpoint and UI has been deprecated. Instead, pull audit logs from Kubernetes and push them to your SIEM.
With the Event Listener enabled, users can view audit records for queries made in Starburst against Immuta data sources on the Audit page. Immuta audits the activity of Immuta users on Immuta data sources.
Starburst (Trino) users registered as Immuta users: Note that the users' Starburst (Trino) usernames must be mapped to Immuta. Without this, Immuta will not know the users are Immuta users and will not collect audit events for their data access activity.
Best Practices: Store Audit Records
By default Immuta audit records expire after 60 days, so store audit records outside of Immuta in order to retain the audits long term.
Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below. These audit records are stored with the recordType: prestoQuery.
| Property | Description | Example |
|---|---|---|
Deprecation notice
Support for the audit endpoint and UI has been deprecated. Instead, and push them to your SIEM.
Snowflake query audit logs is a feature that audits queries that users run natively in Snowflake and turns them into Immuta audit logs. Immuta uses the Snowflake QUERY_HISTORY and ACCESS_HISTORY tables and translates them into the audit logs that can be viewed and downloaded within the Immuta UI or using the Immuta API. Immuta audits the activity of Immuta users on Immuta data sources.
Snowflake Enterprise Edition or higher
: Note that the users' . Without this, Immuta will not know the users are Immuta users and will not collect audit events for their data access activity.
Best Practices: Store Audit Records
By default Immuta audit records expire after 60 days, so store audit records outside of Immuta in order to retain the audits long term.
Immuta collects audit records at the frequency , which is between 1 and 24 hours. The frequency is a global setting based on integration type, so organizations with multiple Snowflake integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records; however, there could be performance and cost impacts from the frequent jobs.
To manually request native query audit ingestion, click Load Audit Events on the Immuta audit page.
Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below. These audit records are stored with the recordType: nativeQuery.
| Property | Description | Example |
|---|---|---|
| Property | Description | Example |
|---|
ID
integer
b0000000-1234-abcd-11111111111111
DateTime
integer or string The timestamp for when the record was created. This may be an ISO-8601 timestamp string or an epoch timestamp.
2504188066580 or 2017-08-31T14:01:15.607Z
Month
integer
1455
ProfileID
integer The profile ID of the user who made the query.
1
UserID
string The user ID of the user who made the query.
jane.doe@immuta.com
DataSourceID
integer The ID of the data source that was queried.
12
DataSourceName
string The name of the data source that was queried.
Public Customer Data
ProjectID
integer The ID of the project the data source is in.
18
ProjectName
string The name of the project the data source is in.
Project 1
PurposeID
integer The ID of the project's purpose(s).
22
RecordType
string The type of record captured.
Databricks query audit records will always be spark.
Success
boolean If true, the query was successful.
true or false
Component
string The Immuta component that generated the record.
nativeSql
AccessType
string Indicates whether access was granted to an individual blob or if this was a query potentially encompassing many blobs.
query
Query
string The query that was run in the integration.
See the example below
actionStatus
string Indicates whether or not the user was granted access to the data.
UNAUTHORIZED or SUCCESS. See the Enriched Databricks Audit Logs section for details.
actionStatusReason
string When a user's query is denied, this property explains why. When a query is successful, this value is null.
policySet
array Provides policy details.
entitlements
array Includes the user's groups, attributes, and current project at the time of the query.
ID
string
b0000000-1234-abcd-11111111111111
DateTime
integer or string The timestamp for when the record was created. This may be an ISO-8601 timestamp string or an epoch timestamp.
2504188066580 or 2017-08-31T14:01:15.607Z
Month
integer
1455
ProfileID
integer The profile ID of the user who made the query.
1
UserID
string The user ID of the user who made the query.
jane.doe@immuta.com
DataSourceID
integer The ID of the data source that was queried.
12
DataSourceName
string The name of the data source that was queried.
Public Customer Data
ProjectID
integer The ID of the project the data source is in.
18
ProjectName
string The name of the project the data source is in.
Project 1
RecordType
string The type of record captured.
Trino query audit records will always be prestoQuery.
Success
boolean If true, the query was successful.
true or false
Component
string The Immuta component that generated the record.
nativeSql
AccessType
string Indicates whether access was granted to an individual blob or if this was a query potentially encompassing many blobs.
query
Query
string The query that was run in the integration.
select * from immuta.public. \"case\" limit 50
Extra
array Information on the query, including viewSql, direct, and maskedColumns.
See example audit record below.
DataSourceSchemaName
string
The name of the schema that the data source that was queried came from.
DataSourceTableName
string The name of the table of the data source that was queried.
case
sqlUser
string The Starburst (Trino) username of the user who made the query.
kris
id integer
The unique ID of the audit record.
b0000000-1234-abcd-11111111111111
dateTime integer or string
The timestamp for when the record was created. This may be an ISO-8601 timestamp string or an epoch timestamp.
2504188066580 or 2017-08-31T14:01:15.607Z
profileId integer
When available, the profile ID of the user who made the query. If the user is not registered with Immuta, the profileId will be -1. In this case, actorEmail can provide the user's Unity Catalog email.
1
userId string
When available, the Immuta user ID of the user who made the query. If the user is not registered with Immuta, userId will be null. In this case, actorEmail can provide the user's Unity Catalog email.
taylor@immuta.com
dataSourceId integer
When available, the Immuta ID of the data source that was queried. If the data source is not registered with Immuta, dataSourceId will be undefined. In this case, nativeObject and nativeObjectFullName can provide the name of the object in Unity Catalog.
12
dataSourceName string
When available, the Immuta name of the data source that was queried. If the data source is not registered with Immuta, dataSourceName will be undefined. In this case, nativeObject and nativeObjectFullName can provide the name of the object in Unity Catalog.
Immuta_Sample_Data
recordType string
The type of record captured based on query type. Databricks Unity Catalog query audit logs will always be nativeQuery.
nativeQuery
success boolean
If TRUE, the query was successful.
TRUE
component string
The Immuta component that generated the record.
nativeSql
accessType string
Unity Catalog audit logs will always be query.
query
query string
The command text of the query.
select * from samples_catalog.samples_schema.databricks_sample_data limit 52;
queryId integer
The unique ID of the query. If the query joins multiple tables, each table will appear as a separate log with a matching query ID.
869500255746459
extra.handler string
The integration technology. Databricks Unity Catalog query audit logs will always be Databricks.
Databricks
extra.startTime date
The date and time the query started in UTC.
2023-06-06 13:27:51
extra.endTime date
The date and time the query ended in UTC.
2023-06-06 13:28:02
extra.duration string
The time the query took in milliseconds.
557
extra.nativeObject string
When available, the name of the Unity Catalog object that was queried.
databricks_sample_data
extra.nativeObjectFullName string
When available, the fully qualified Unity Catalog object that was queried.
samples_catalog.samples_schema.databricks_sample_data
extra.host string
The host that the integration is connected to.
immuta-example.cloud.databricks.com
extra.actionStatus string
Indicates whether or not the user was granted access to the data. Options are UNAUTHORIZED, FAILURE, or SUCCESS.
SUCCESS
extra.actionStatusReason string
When available and a user's query is denied, this property contains the Unity Catalog system's reason for failure. When a query is successful, this value is null.
InnerExecutionException: null
extra.errorCode integer
When available, the error code for the unauthorized query. For unauthorized queries options are 1 and 400.
1
extra.actionName string
The environment data was accessed in. If commandSubmit, the audit log is from a SQL warehouse. If runCommand, the audit log is from a cluster.
runCommand
extra.service string
The environment data was accessed in. Options include spark, cluster, and warehouse.
cluster
extra.accountId string
The Unity Catalog account ID.
52e863bc-ea7f-46a9-8e17-6aed7541832d
extra.notebookId integer
For cluster queries, the ID of the Unity Catalog notebook the query was made in, otherwise null.
869500255746458
extra.clusterId string
For cluster queries, the ID of the Unity Catalog cluster the query was made in, otherwise null.
0511-134653-3eykv2e7
extra.requestedId string
The Unity Catalog request ID.
6435b58c-9557-4395-ae73-dc8ebe83c15f
extra.sessionId string
For warehouse queries, the ID of the Unity Catalog session the query was made in, otherwise null.
01ee14d9-cab3-1ef6-9cc4-f0c315a53788
extra.warehouseId string
For warehouse queries, the ID of the Unity Catalog warehouse the query was made in, otherwise null.
559483c6eac0359f
extra.queryLanguage string
For cluster queries, the programming language used: SQL, Python, Scala, or R.
sql
extra.actorIp string
When available, the IP address of the Spark cluster the request is coming from.
100.40.110.136
extra.actorEmail string
The Unity Catalog email of the user making the query.
taylor@immuta.com
extra.userAgent string
When available, the Unity Catalog client information of the user who made the query.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36
extra.workspaceId integer
The ID of the Unity Catalog workspace the query was made in.
8765531160949612
DateTime |
|
|
ProfileID |
|
|
UserID |
|
|
DataSourceID |
|
|
DataSourceName |
|
|
RecordType |
| Native query audit records will always be |
Success |
|
|
Component |
|
|
AccessType |
|
|
Query |
|
|
Handler |
|
|
StartTime |
|
|
EndTime |
|
|
Duration |
|
|
NativeObject |
|
|
NativeObjectType |
|
|
Host |
|
|
Database |
|
|
SQLUser |
|
|