Databricks Unity Catalog Query Audit Logs

Native query audit for Unity Catalog captures user data access activity within Unity Catalog as Immuta audit logs. Immuta audits the activity of all Unity Catalog users and tables. Multiple access options are supported for audit:

• Cluster queries with the following supported languages: SQL, Scala, Python, and R.
• SQL Warehouse queries

Requirements

• Databricks Unity Catalog integration with native query audit enabled
• A Databricks deployment with system tables capabilities
• Provide the Immuta system account with access to audit-related Unity Catalog system tables

Audit frequency

Immuta collects audit records at the frequency configured when enabling the integration, which is between 1 and 24 hours. The frequency is a global setting based on integration type, so organizations with multiple Unity Catalog integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records; however, there could be performance and cost impacts from the frequent jobs.

Audit messages

Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below. These audit logs are stored with the recordType: nativeQuery.

Example audit record

{
  "id": "9f542dfd-5099-4362-a72d-8377306db3b8",
  "dateTime": "1686072470865",
  "month": 1481,
  "profileId": 999111223,
  "userId": "taylor@immuta.com",
  "dataSourceId": 21,
  "dataSourceName": "Immuta_Sample_Data",
  "count": 1,
  "recordType": "nativeQuery",
  "success": true,
  "component": "nativeSql",
  "accessType": "query",
  "query": "select * from samples_catalog.samples_schema.databricks_sample_data limit 52;\n",
  "queryId": "869500255746459",
  "extra": {
    "handler": "Databricks",
    "startTime": "2023-06-06 13:27:50.865",
    "endTime": "2023-06-06 13:27:51.678",
    "duration": "557",
    "nativeObject": "databricks_sample_data",
    "nativeObjectFullName": "samples_catalog.samples_schema.databricks_sample_data",
    "host": "immuta-example.cloud.databricks.com",
    "actionStatus": "SUCCESS",
    "actionStatusReason": null,
    "errorCode": null,
    "actionName": "runCommand",
    "accountId": "52e863bc-ea7f-46a9-8e17-6aed7541832d",
    "notebookId": "869500255746458",
    "clusterId": "0511-134653-3eykv2e7",
    "requestId": "6435b58c-9557-4395-ae73-dc8ebe83c15f",
    "sessionId": null,
    "warehouseId": null,
    "queryLanguage": "sql",
    "actorIp": "100.40.110.136",
    "actorEmail": "taylor@immuta.com",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
    "workspaceId": 8765531160949612
  },
  "createdAt": "2023-06-07T15:14:41.950Z",
  "updatedAt": "2023-06-07T15:14:41.950Z"
}

Limitations

• Enrichment of audit logs with Immuta entitlements information is not supported. While you will see these entitlements in the Databricks Spark audit logs, the following will not be in the native query audit for Unity Catalog:
  • Immuta policies information
  • User attributes
  • Groups
• Immuta determines unauthorized events based on error messages within Unity Catalog records. When the error messages contain expected language, unauthorized events will be available for native query audit for Unity Catalog. In other cases, it is not possible to determine the cause of an error.
• Audit for cluster queries do not support UNAUTHORIZED status. If a cluster query is unauthorized, it will show FAILURE.
• Data source information will be provided when available:
  • For some queries, Databricks Unity Catalog does not report the target data source for the data access operation. In these cases the activity is audited, yet the audit record in Immuta will not include the target data source information.
  • Data source information is not available for unauthorized queries and events.
• Column information from the query is not currently supported.
• The cluster for the Unity Catalog integration must always be running for Immuta to audit activity.
• Immuta audit records include unregistered data sources and users; however, activity from them will not appear in any governance reports.