Databricks Unity Catalog Query Audit Logs

Native query audit for Databricks Unity Catalog captures user data access within Unity Catalog and presents them in a universal format as Immuta audit logs. Multiple access options are supported for audit:

• Cluster queries with the following supported languages: SQL, Scala, Python, and R.

• SQL warehouse queries

Immuta audits the activity of all Unity Catalog users and tables regardless of whether they are registered in Immuta.

Requirements

• Databricks Unity Catalog integration with native query audit enabled

• A Databricks deployment with system tables capabilities

Store audit logs

By default Databricks Unity Catalog audit logs expire after 90 days. Export the universal audit model (UAM) logs to S3 or ADLS Gen2, and store audit logs outside of Immuta in order to retain the audit logs long-term.

Audit frequency

Immuta collects audit records at the frequency configured when enabling the integration, which is between 1 and 24 hours. The frequency is a global setting based on integration type, so organizations with multiple Unity Catalog integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records; however, there could be performance and cost impacts from the frequent jobs. Immuta will start a Databricks cluster to complete the audit ingest job if one is not already running.

To manually prompt the native query audit, click Load Audit Events on the Immuta audit page.

Audit scope

Immuta audits all data sources and users in Unity Catalog. An administrator can configure the integration to just ingest specific workspaces when enabling the integration.

Audit schema

Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below.

Example audit record

{
  "action": "QUERY",
  "actor": {
    "type": "USER_ACTOR",
    "id": "[e-mail]",
    "name": "Taylor",
    "identityProvider": "bim",
    "profileId": "10"
  },
  "sessionId": "01ee14d9-cab3-1ef6-9cc4-f0c315a53788",
  "requestId": "504b8fd9-38c1-4a90-966e-7445a6675f79",
  "actionStatus": "SUCCESS",
  "actionStatusReason": null,
  "eventTimestamp": "2023-06-27T11:03:59.000Z",
  "id": "01ee14da-517a-1670-afce-0c3e0fdcf7d4",
  "tenantId": "your-immuta.com",
  "userAgent": "",
  "targetType": "DATASOURCE",
  "targets": [
      {
      "type": "DATASOURCE",
      "id": "2034",
      "name": "University Art Gallery Exhibition",
      "technology": "DATABRICKS"
    }
  ],
  "relatedResources": [],
  "auditPayload": {
    "type": "QueryAuditPayload",
    "queryId": "01ee14da-517a-1670-afce-0c3e0fdcf7d4",
    "query": "SELECT VERSION AS `version` FROM `sample-data`.`__immuta_version`",
    "startTime": "2023-06-27T11:03:59.000Z",
    "duration": 23.568,
    "errorCode": null,
    "technologyContext": {
      "type": "DatabricksContext",
      "clusterId": null,
      "workspaceId": "3841033049363283",
      "service": "SQL",
      "warehouseId": "559483c6eac0359f",
      "notebookId": null,
      "account": {
        "id": "52e863bc-ea7f-46a9-8e17-6aed7541832d",
        "username": "[e-mail]"
      },
      "host": "deployment-name.cloud.databricks.com",
      "clientIp": "0.0.0.0"
    },
    "objectsAccessed": [],
    "securityProfile": {
      "sensitivity": {
        "score": "INDETERMINATE"
      }
    },
    "version": 1
  },
  "receivedTimestamp": "2023-06-27T15:18:22.314Z"
}

Limitations

• Enrichment of audit logs with Immuta entitlements information is not supported. While you will see these entitlements in the Databricks Spark audit logs, the following will not be in the native query audit for Unity Catalog:

  • Immuta policies information

  • User attributes

  • Groups

• Immuta determines unauthorized events based on error messages within Unity Catalog records. When the error messages contain expected language, unauthorized events will be available for native query audit for Unity Catalog. In other cases, it is not possible to determine the cause of an error.

• Audit for cluster queries do not support UNAUTHORIZED status. If a cluster query is unauthorized, it will show FAILURE.

• Data source information will be provided when available:

  • For some queries, Databricks Unity Catalog does not report the target data source for the data access operation. In these cases the activity is audited, yet the audit record in Immuta will not include the target data source information.

  • Data source information is not available for unauthorized queries and events.

• Column information from the query is not currently supported.

• Immuta audit records include unregistered data sources and users; however, activity from them will not appear in any governance reports.