1 of 1

Databricks Unity Catalog Audit Logs

Native query audit for Databricks Unity Catalog captures user data access within Unity Catalog and presents them in a universal format as Immuta audit logs. Multiple access options are supported for audit:

Cluster queries with the following supported languages: SQL, Scala, Python, and R.
SQL warehouse queries

Immuta audits the activity of all Unity Catalog users and tables.

Requirement

Databricks Unity Catalog integration with native query audit enabled

Store audit logs

By default Immuta audit logs expire after 7 days. Export the universal audit model (UAM) logs to S3 or ADLS Gen 2, and store audit logs outside of Immuta in order to retain the audit logs long-term.

Audit frequency

Immuta collects audit records at the frequency configured when enabling the integration, which is between 1 and 24 hours. The frequency is a global setting based on integration type, so organizations with multiple Unity Catalog integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records; however, there could be performance and cost impacts from the frequent jobs. Immuta will start a Databricks cluster to complete the audit ingest job if one is not already running.

To manually prompt the native query audit, click Load Audit Events on the Immuta audit page.

Audit schema

Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below.

Property

Description

Example

Example audit record

{
  "action": "QUERY",
  "actor": {
    "type": "USER_ACTOR",
    "id": "taylor@immuta.com",
    "name": "Taylor",
    "identityProvider": "bim",
    "profileId": "10"
  },
  "sessionId": "01ee14d9-cab3-1ef6-9cc4-f0c315a53788",
  "requestId": "504b8fd9-38c1-4a90-966e-7445a6675f79",
  "actionStatus": "SUCCESS",
  "actionStatusReason": null,
  "eventTimestamp": "2023-06-27T11:03:59.000Z",
  "id": "01ee14da-517a-1670-afce-0c3e0fdcf7d4",
  "tenantId": "your-immuta.com",
  "userAgent": "",
  "targetType": "DATASOURCE",
  "targets": [
      {
      "type": "DATASOURCE",
      "id": "2034",
      "name": "University Art Gallery Exhibition",
      "technology": "DATABRICKS"
    }
  ],
  "relatedResources": [],
  "auditPayload": {
    "type": "QueryAuditPayload",
    "queryId": "01ee14da-517a-1670-afce-0c3e0fdcf7d4",
    "query": "SELECT VERSION AS `version` FROM `sample-data`.`__immuta_version`",
    "startTime": "2023-06-27T11:03:59.000Z",
    "duration": 23.568,
    "errorCode": null,
    "technologyContext": {
      "type": "DatabricksContext",
      "clusterId": null,
      "workspaceId": "3841033049363283",
      "service": "SQL",
      "warehouseId": "559483c6eac0359f",
      "notebookId": null,
      "account": {
        "id": "52e863bc-ea7f-46a9-8e17-6aed7541832d",
        "username": "taylor@databricks.com"
      },
      "host": "deployment-name.cloud.databricks.com",
      "clientIp": "0.0.0.0"
    },
    "objectsAccessed": [],
    "securityProfile": {
      "sensitivity": {
        "score": "INDETERMINATE"
      }
    },
    "version": 1
  },
  "receivedTimestamp": "2023-06-27T15:18:22.314Z"
}

Limitations

Enrichment of audit logs with Immuta entitlements information is not supported. While you will see these entitlements in the Databricks Spark audit logs, the following will not be in the native query audit for Unity Catalog:
- Immuta policies information
- User attributes
- Groups
Immuta determines unauthorized events based on error messages within Unity Catalog records. When the error messages contain expected language, unauthorized events will be available for native query audit for Unity Catalog. In other cases, it is not possible to determine the cause of an error.
Audit for cluster queries do not support UNAUTHORIZED status. If a cluster query is unauthorized, it will show FAILURE.
Data source information will be provided when available:
- For some queries, Databricks Unity Catalog does not report the target data source for the data access operation. In these cases the activity is audited, yet the audit record in Immuta will not include the target data source information.
- Data source information is not available for unauthorized queries and events.
Column information from the query is not currently supported.
Immuta audit records include unregistered data sources and users; however, activity from them will not appear in any governance reports.