Databricks Unity Catalog Query Audit Logs
Native query audit for Databricks Unity Catalog captures user data access within Unity Catalog and presents them in a universal format as Immuta audit logs. Multiple access options are supported for audit:
Cluster queries with the following supported languages: SQL, Scala, Python, and R.
SQL warehouse queries
Immuta audits the activity of all Unity Catalog users and tables regardless of whether they are registered in Immuta.
Requirements
A Databricks deployment with system tables capabilities
Store audit logs
By default Databricks Unity Catalog audit logs expire after 90 days. Export the universal audit model (UAM) logs to S3 or ADLS Gen2, and store audit logs outside of Immuta in order to retain the audit logs long-term.
Audit frequency
Immuta collects audit records at the frequency configured when enabling the integration, which is between 1 and 24 hours. The frequency is a global setting based on integration type, so organizations with multiple Unity Catalog integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records; however, there could be performance and cost impacts from the frequent jobs. Immuta will start a Databricks cluster to complete the audit ingest job if one is not already running.
To manually prompt the native query audit, click Load Audit Events on the Immuta audit page.
Audit scope
Immuta audits all data sources and users in Unity Catalog. An administrator can configure the integration to just ingest specific workspaces when enabling the integration.
Audit schema
Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below.
Example audit record
Limitations
Enrichment of audit logs with Immuta entitlements information is not supported. While you will see these entitlements in the Databricks Spark audit logs, the following will not be in the native query audit for Unity Catalog:
Immuta policies information
User attributes
Groups
Immuta determines unauthorized events based on error messages within Unity Catalog records. When the error messages contain expected language, unauthorized events will be available for native query audit for Unity Catalog. In other cases, it is not possible to determine the cause of an error.
Audit for cluster queries do not support
UNAUTHORIZED
status. If a cluster query is unauthorized, it will showFAILURE
.Data source information will be provided when available:
For some queries, Databricks Unity Catalog does not report the target data source for the data access operation. In these cases the activity is audited, yet the audit record in Immuta will not include the target data source information.
Data source information is not available for unauthorized queries and events.
Column information from the query is not currently supported.
Immuta audit records include unregistered data sources and users; however, activity from them will not appear in any governance reports.
Last updated