Databricks Unity Catalog Audit Logs

Public Preview

This feature is currently in public preview and available to all accounts.

Native query audit for Unity Catalog captures user data access activity within Unity Catalog as Immuta audit logs. Immuta audits the activity of all Unity Catalog users and tables. Multiple access options are supported for audit:

• Cluster queries with the following supported languages: SQL, Scala, Python, and R.

• SQL Warehouse queries

Deprecation notice

Support for the audit endpoint and UI has been deprecated. Instead, pull audit logs from Kubernetes and push them to your SIEM.

Requirements

• Databricks Unity Catalog integration with native query audit enabled

• A Databricks deployment with system tables capabilities

• Provide the Immuta system account with access to audit-related Unity Catalog system tables

Best practices: Store audit logs

By default most Immuta audit logs expire after 60 days, so store audit logs outside of Immuta in order to retain the audits long term.

The following audit record types do not expire after 60 days: blobFetch, dataSourceSubscription, globalPolicyApproved, globalPolicyApprovalRescinded, globalPolicyChangeRequested, globalPolicyConflictResolved, globalPolicyCreate, globalPolicyDelete, globalPolicyDisabled, globalPolicyUpdate, policyExemption, policyHandlerCreate, policyHandlerUpdate, prestoQuery, spark, and sqlQuery.

Audit frequency

Immuta collects audit records at the frequency configured when enabling the integration, which is between 1 and 24 hours. The frequency is a global setting based on integration type, so organizations with multiple Unity Catalog integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records; however, there could be performance and cost impacts from the frequent jobs.

To manually request native query audit ingestion, click Load Audit Events on the Immuta audit page.

Audit messages

Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below. These audit logs are stored with the recordType: nativeQuery.

Example audit record

{
  "id": "9f542dfd-5099-4362-a72d-8377306db3b8",
  "dateTime": "1686072470865",
  "month": 1481,
  "profileId": 999111223,
  "userId": "[e-mail]",
  "dataSourceId": 21,
  "dataSourceName": "Immuta_Sample_Data",
  "count": 1,
  "recordType": "nativeQuery",
  "success": true,
  "component": "nativeSql",
  "accessType": "query",
  "query": "select * from samples_catalog.samples_schema.databricks_sample_data limit 52;\n",
  "queryId": "869500255746459",
  "extra": {
    "handler": "Databricks",
    "startTime": "2023-06-06 13:27:50.865",
    "endTime": "2023-06-06 13:27:51.678",
    "duration": "557",
    "nativeObject": "databricks_sample_data",
    "nativeObjectFullName": "samples_catalog.samples_schema.databricks_sample_data",
    "host": "immuta-example.cloud.databricks.com",
    "actionStatus": "SUCCESS",
    "actionStatusReason": null,
    "errorCode": null,
    "actionName": "runCommand",
    "accountId": "52e863bc-ea7f-46a9-8e17-6aed7541832d",
    "notebookId": "869500255746458",
    "clusterId": "0511-134653-3eykv2e7",
    "requestId": "6435b58c-9557-4395-ae73-dc8ebe83c15f",
    "sessionId": null,
    "warehouseId": null,
    "queryLanguage": "sql",
    "actorIp": "100.40.110.136",
    "actorEmail": "[e-mail]",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
    "workspaceId": 8765531160949612
  },
  "createdAt": "2023-06-07T15:14:41.950Z",
  "updatedAt": "2023-06-07T15:14:41.950Z"
}

Limitations

• Enrichment of audit logs with Immuta entitlements information is not supported. While you will see these entitlements in the Databricks Spark audit logs, the following will not be in the native query audit for Unity Catalog:

  • Immuta policies information

  • User attributes

  • Groups

• Immuta determines unauthorized events based on error messages within Unity Catalog records. When the error messages contain expected language, unauthorized events will be available for native query audit for Unity Catalog. In other cases, it is not possible to determine the cause of an error.

• Audit for cluster queries do not support UNAUTHORIZED status. If a cluster query is unauthorized, it will show FAILURE.

• Data source information will be provided when available:

  • For some queries, Databricks Unity Catalog does not report the target data source for the data access operation. In these cases the activity is audited, yet the audit record in Immuta will not include the target data source information.

  • Data source information is not available for unauthorized queries and events.

• Column information from the query is not currently supported.

• The cluster for the Unity Catalog integration must always be running for Immuta to audit activity.

• Immuta audit records include unregistered data sources and users; however, activity from them will not appear in any governance reports.