Databricks Unity Catalog Audit Logs

Public Preview

This feature is currently in public preview and available to all accounts.

Native query audit for Unity Catalog captures user data access activity within Unity Catalog as Immuta audit logs. Immuta audits the activity of all Unity Catalog users and tables. Multiple access options are supported for audit:

• Cluster queries with the following supported languages: SQL, Scala, Python, and R.

• SQL Warehouse queries

Deprecation notice

Support for the audit endpoint and UI has been deprecated. Instead, pull audit logs from Kubernetes and push them to your SIEM.

Requirements

• Databricks Unity Catalog integration with native query audit enabled

• A Databricks deployment with system tables capabilities

• Provide the Immuta system account with access to audit-related Unity Catalog system tables

Best practices: Store audit logs

By default most Immuta audit logs expire after 60 days, so store audit logs outside of Immuta in order to retain the audits long term.

The following audit record types do not expire after 60 days: blobFetch, dataSourceSubscription, globalPolicyApproved, globalPolicyApprovalRescinded, globalPolicyChangeRequested, globalPolicyConflictResolved, globalPolicyCreate, globalPolicyDelete, globalPolicyDisabled, globalPolicyUpdate, policyExemption, policyHandlerCreate, policyHandlerUpdate, prestoQuery, spark, and sqlQuery.

Audit frequency

Immuta collects audit records at the frequency configured when enabling the integration, which is between 1 and 24 hours. The frequency is a global setting based on integration type, so organizations with multiple Unity Catalog integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records; however, there could be performance and cost impacts from the frequent jobs.

To manually request native query audit ingestion, click Load Audit Events on the Immuta audit page.

Audit messages

Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below. These audit logs are stored with the recordType: nativeQuery.

Property	Description	Example
id integer	The unique ID of the audit record.	b0000000-1234-abcd-11111111111111
dateTime integer or string	The timestamp for when the record was created. This may be an ISO-8601 timestamp string or an epoch timestamp.	2504188066580 or 2017-08-31T14:01:15.607Z
profileId integer	When available, the profile ID of the user who made the query. If the user is not registered with Immuta, the profileId will be -1. In this case, actorEmail can provide the user's Unity Catalog email.	1
userId string	When available, the Immuta user ID of the user who made the query. If the user is not registered with Immuta, userId will be null. In this case, actorEmail can provide the user's Unity Catalog email.	[e-mail]
dataSourceId integer	When available, the Immuta ID of the data source that was queried. If the data source is not registered with Immuta, dataSourceId will be undefined. In this case, nativeObject and nativeObjectFullName can provide the name of the object in Unity Catalog.	12
dataSourceName string	When available, the Immuta name of the data source that was queried. If the data source is not registered with Immuta, dataSourceName will be undefined. In this case, nativeObject and nativeObjectFullName can provide the name of the object in Unity Catalog.	Immuta_Sample_Data
recordType string	The type of record captured based on query type. Databricks Unity Catalog query audit logs will always be nativeQuery.	nativeQuery
success boolean	If TRUE, the query was successful.	TRUE
component string	The Immuta component that generated the record.	nativeSql
accessType string	Unity Catalog audit logs will always be query.	query
query string	The command text of the query.	select * from samples_catalog.samples_schema.databricks_sample_data limit 52;
queryId integer	The unique ID of the query. If the query joins multiple tables, each table will appear as a separate log with a matching query ID.	869500255746459
extra.handler string	The integration technology. Databricks Unity Catalog query audit logs will always be Databricks.	Databricks
extra.startTime date	The date and time the query started in UTC.	2023-06-06 13:27:51
extra.endTime date	The date and time the query ended in UTC.	2023-06-06 13:28:02
extra.duration string	The time the query took in milliseconds.	557
extra.nativeObject string	When available, the name of the Unity Catalog object that was queried.	databricks_sample_data
extra.nativeObjectFullName string	When available, the fully qualified Unity Catalog object that was queried.	samples_catalog.samples_schema.databricks_sample_data
extra.host string	The host that the integration is connected to.	immuta-example.cloud.databricks.com
extra.actionStatus string	Indicates whether or not the user was granted access to the data. Options are UNAUTHORIZED, FAILURE, or SUCCESS.	SUCCESS
extra.actionStatusReason string	When available and a user's query is denied, this property contains the Unity Catalog system's reason for failure. When a query is successful, this value is null.	InnerExecutionException: null
extra.errorCode integer	When available, the error code for the unauthorized query. For unauthorized queries options are 1 and 400.	1
extra.actionName string	The environment data was accessed in. If commandSubmit, the audit log is from a SQL warehouse. If runCommand, the audit log is from a cluster.	runCommand
extra.service string	The environment data was accessed in. Options include spark, cluster, and warehouse.	cluster
extra.accountId string	The Unity Catalog account ID.	52e863bc-ea7f-46a9-8e17-6aed7541832d
extra.notebookId integer	For cluster queries, the ID of the Unity Catalog notebook the query was made in, otherwise null.	869500255746458
extra.clusterId string	For cluster queries, the ID of the Unity Catalog cluster the query was made in, otherwise null.	0511-134653-3eykv2e7
extra.requestedId string	The Unity Catalog request ID.	6435b58c-9557-4395-ae73-dc8ebe83c15f
extra.sessionId string	For warehouse queries, the ID of the Unity Catalog session the query was made in, otherwise null.	01ee14d9-cab3-1ef6-9cc4-f0c315a53788
extra.warehouseId string	For warehouse queries, the ID of the Unity Catalog warehouse the query was made in, otherwise null.	559483c6eac0359f
extra.queryLanguage string	For cluster queries, the programming language used: SQL, Python, Scala, or R.	sql
extra.actorIp string	When available, the IP address of the Spark cluster the request is coming from.	100.40.110.136
extra.actorEmail string	The Unity Catalog email of the user making the query.	[e-mail]
extra.userAgent string	When available, the Unity Catalog client information of the user who made the query.	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36
extra.workspaceId integer	The ID of the Unity Catalog workspace the query was made in.	8765531160949612

Example audit record

{
  "id": "9f542dfd-5099-4362-a72d-8377306db3b8",
  "dateTime": "1686072470865",
  "month": 1481,
  "profileId": 999111223,
  "userId": "[e-mail]",
  "dataSourceId": 21,
  "dataSourceName": "Immuta_Sample_Data",
  "count": 1,
  "recordType": "nativeQuery",
  "success": true,
  "component": "nativeSql",
  "accessType": "query",
  "query": "select * from samples_catalog.samples_schema.databricks_sample_data limit 52;\n",
  "queryId": "869500255746459",
  "extra": {
    "handler": "Databricks",
    "startTime": "2023-06-06 13:27:50.865",
    "endTime": "2023-06-06 13:27:51.678",
    "duration": "557",
    "nativeObject": "databricks_sample_data",
    "nativeObjectFullName": "samples_catalog.samples_schema.databricks_sample_data",
    "host": "immuta-example.cloud.databricks.com",
    "actionStatus": "SUCCESS",
    "actionStatusReason": null,
    "errorCode": null,
    "actionName": "runCommand",
    "accountId": "52e863bc-ea7f-46a9-8e17-6aed7541832d",
    "notebookId": "869500255746458",
    "clusterId": "0511-134653-3eykv2e7",
    "requestId": "6435b58c-9557-4395-ae73-dc8ebe83c15f",
    "sessionId": null,
    "warehouseId": null,
    "queryLanguage": "sql",
    "actorIp": "100.40.110.136",
    "actorEmail": "[e-mail]",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
    "workspaceId": 8765531160949612
  },
  "createdAt": "2023-06-07T15:14:41.950Z",
  "updatedAt": "2023-06-07T15:14:41.950Z"
}

Limitations

• Enrichment of audit logs with Immuta entitlements information is not supported. While you will see these entitlements in the Databricks Spark audit logs, the following will not be in the native query audit for Unity Catalog:

  • Immuta policies information

  • User attributes

  • Groups

• Immuta determines unauthorized events based on error messages within Unity Catalog records. When the error messages contain expected language, unauthorized events will be available for native query audit for Unity Catalog. In other cases, it is not possible to determine the cause of an error.

• Audit for cluster queries do not support UNAUTHORIZED status. If a cluster query is unauthorized, it will show FAILURE.

• Data source information will be provided when available:

  • For some queries, Databricks Unity Catalog does not report the target data source for the data access operation. In these cases the activity is audited, yet the audit record in Immuta will not include the target data source information.

  • Data source information is not available for unauthorized queries and events.

• Column information from the query is not currently supported.

• The cluster for the Unity Catalog integration must always be running for Immuta to audit activity.

• Immuta audit records include unregistered data sources and users; however, activity from them will not appear in any governance reports.