Skip to content

You are viewing documentation for Immuta version 2023.3.

For the latest version, view our documentation for Immuta SaaS or the latest self-hosted version.

Databricks Unity Catalog Query Audit Logs

Deprecation notice

Support for the audit endpoint and UI has been deprecated. Instead, pull audit logs from Kubernetes and push them to your SIEM.

Public Preview

This feature is currently in public preview and available to all accounts.

Native query audit for Unity Catalog captures user data access activity within Unity Catalog as Immuta audit logs. Immuta audits the activity of all Unity Catalog users and tables. Multiple access options are supported for audit:

  • Cluster queries with the following supported languages: SQL, Scala, Python, and R.
  • SQL Warehouse queries

Requirements

  • Databricks Unity Catalog integration with native query audit enabled

Best practices: Store audit logs

By default most Immuta audit logs expire after 60 days, so store audit logs outside of Immuta in order to retain the audits long term.

The following audit record types do not expire after 60 days: blobFetch, dataSourceSubscription, globalPolicyApproved, globalPolicyApprovalRescinded, globalPolicyChangeRequested, globalPolicyConflictResolved, globalPolicyCreate, globalPolicyDelete, globalPolicyDisabled, globalPolicyUpdate, policyExemption, policyHandlerCreate, policyHandlerUpdate, prestoQuery, spark, and sqlQuery.

Audit frequency

Immuta collects audit records at the frequency configured when enabling the integration, which is between 1 and 24 hours. The frequency is a global setting based on integration type, so organizations with multiple Unity Catalog integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records; however, there could be performance and cost impacts from the frequent jobs.

To manually prompt the native query audit, click Native Query Audit on the Immuta audit page.

Audit messages

Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below. These audit logs are stored with the recordType: nativeQuery.

Property Description Example
id integer The unique ID of the audit record. b0000000-1234-abcd-11111111111111
dateTime integer or string The timestamp for when the record was created. This may be an ISO-8601 timestamp string or an epoch timestamp. 2504188066580 or 2017-08-31T14:01:15.607Z
profileId integer When available, the profile ID of the user who made the query. If the user is not registered with Immuta, the profileId will be -1. In this case, actorEmail can provide the user's Unity Catalog email. 1
userId string When available, the Immuta user ID of the user who made the query. If the user is not registered with Immuta, userId will be null. In this case, actorEmail can provide the user's Unity Catalog email. taylor@immuta.com
dataSourceId integer When available, the Immuta ID of the data source that was queried. If the data source is not registered with Immuta, dataSourceId will be undefined. In this case, nativeObject and nativeObjectFullName can provide the name of the object in Unity Catalog. 12
dataSourceName string When available, the Immuta name of the data source that was queried. If the data source is not registered with Immuta, dataSourceName will be undefined. In this case, nativeObject and nativeObjectFullName can provide the name of the object in Unity Catalog. Immuta_Sample_Data
recordType string The type of record captured based on query type. Databricks Unity Catalog query audit logs will always be nativeQuery. nativeQuery
success boolean If TRUE, the query was successful. TRUE
component string The Immuta component that generated the record. nativeSql
accessType string Unity Catalog audit logs will always be query. query
query string The command text of the query. select * from samples_catalog.samples_schema.databricks_sample_data limit 52;
queryId integer The unique ID of the query. If the query joins multiple tables, each table will appear as a separate log with a matching query ID. 869500255746459
extra.handler string The integration technology. Databricks Unity Catalog query audit logs will always be Databricks. Databricks
extra.startTime date The date and time the query started in UTC. 2023-06-06 13:27:51
extra.endTime date The date and time the query ended in UTC. 2023-06-06 13:28:02
extra.duration string The time the query took in milliseconds. 557
extra.nativeObject string When available, the name of the Unity Catalog object that was queried. databricks_sample_data
extra.nativeObjectFullName string When available, the fully qualified Unity Catalog object that was queried. samples_catalog.samples_schema.databricks_sample_data
extra.host string The host that the integration is connected to. immuta-example.cloud.databricks.com
extra.actionStatus string Indicates whether or not the user was granted access to the data. Options are UNAUTHORIZED, FAILURE, or SUCCESS. SUCCESS
extra.actionStatusReason string When available and a user's query is denied, this property contains the Unity Catalog system's reason for failure. When a query is successful, this value is null. InnerExecutionException: null
extra.errorCode integer When available, the error code for the unauthorized query. For unauthorized queries options are 1 and 400. 1
extra.actionName string The environment data was accessed in. If commandSubmit, the audit log is from a SQL warehouse. If runCommand, the audit log is from a cluster. runCommand
extra.service string The environment data was accessed in. Options include spark, cluster, and warehouse. cluster
extra.accountId string The Unity Catalog account ID. 52e863bc-ea7f-46a9-8e17-6aed7541832d
extra.notebookId integer For cluster queries, the ID of the Unity Catalog notebook the query was made in, otherwise null. 869500255746458
extra.clusterId string For cluster queries, the ID of the Unity Catalog cluster the query was made in, otherwise null. 0511-134653-3eykv2e7
extra.requestedId string The Unity Catalog request ID. 6435b58c-9557-4395-ae73-dc8ebe83c15f
extra.sessionId string For warehouse queries, the ID of the Unity Catalog session the query was made in, otherwise null. 01ee14d9-cab3-1ef6-9cc4-f0c315a53788
extra.warehouseId string For warehouse queries, the ID of the Unity Catalog warehouse the query was made in, otherwise null. 559483c6eac0359f
extra.queryLanguage string For cluster queries, the programming language used: SQL, Python, Scala, or R. sql
extra.actorIp string When available, the IP address of the Spark cluster the request is coming from. 100.40.110.136
extra.actorEmail string The Unity Catalog email of the user making the query. taylor@immuta.com
extra.userAgent string When available, the Unity Catalog client information of the user who made the query. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36
extra.workspaceId integer The ID of the Unity Catalog workspace the query was made in. 8765531160949612

Example audit record

{
  "id": "9f542dfd-5099-4362-a72d-8377306db3b8",
  "dateTime": "1686072470865",
  "month": 1481,
  "profileId": 999111223,
  "userId": "taylor@immuta.com",
  "dataSourceId": 21,
  "dataSourceName": "Immuta_Sample_Data",
  "count": 1,
  "recordType": "nativeQuery",
  "success": true,
  "component": "nativeSql",
  "accessType": "query",
  "query": "select * from samples_catalog.samples_schema.databricks_sample_data limit 52;\n",
  "queryId": "869500255746459",
  "extra": {
    "handler": "Databricks",
    "startTime": "2023-06-06 13:27:50.865",
    "endTime": "2023-06-06 13:27:51.678",
    "duration": "557",
    "nativeObject": "databricks_sample_data",
    "nativeObjectFullName": "samples_catalog.samples_schema.databricks_sample_data",
    "host": "immuta-example.cloud.databricks.com",
    "actionStatus": "SUCCESS",
    "actionStatusReason": null,
    "errorCode": null,
    "actionName": "runCommand",
    "accountId": "52e863bc-ea7f-46a9-8e17-6aed7541832d",
    "notebookId": "869500255746458",
    "clusterId": "0511-134653-3eykv2e7",
    "requestId": "6435b58c-9557-4395-ae73-dc8ebe83c15f",
    "sessionId": null,
    "warehouseId": null,
    "queryLanguage": "sql",
    "actorIp": "100.40.110.136",
    "actorEmail": "taylor@immuta.com",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
    "workspaceId": 8765531160949612
  },
  "createdAt": "2023-06-07T15:14:41.950Z",
  "updatedAt": "2023-06-07T15:14:41.950Z"
}

Limitations

  • Enrichment of audit logs with Immuta entitlements information is not supported. While you will see these entitlements in the Databricks Spark audit logs, the following will not be in the native query audit for Unity Catalog:
    • Immuta policies information
    • User attributes
    • Groups
  • Immuta determines unauthorized events based on error messages within Unity Catalog records. When the error messages contain expected language, unauthorized events will be available for native query audit for Unity Catalog. In other cases, it is not possible to determine the cause of an error.
  • Audit for cluster queries do not support UNAUTHORIZED status. If a cluster query is unauthorized, it will show FAILURE.
  • Data source information will be provided when available:
    • For some queries, Databricks Unity Catalog does not report back the target data source for the data access operation. In these cases the activity is audited, yet the audit record in Immuta will not include the target data source information.
    • Data source information is not available for unauthorized queries and events.
  • Column information from the query is not currently supported.
  • The cluster for Unity Catalog integration must always be running for Immuta to audit activity.
  • Immuta audit records include unregistered data sources and users; however, activity from them will not appear in any governance reports.

Enable native query audit for Unity Catalog

To enable native query audit for Unity Catalog, complete the following steps:

  1. Enable the integration as normal, and the Enable Native Query Audit checkbox will be selected by default.
  2. Enable a system schema where the <SCHEMA_NAME> is access.
  3. Grant your Immuta system account user access to the Databricks Unity Catalog system tables. Immuta must have access to the following tables:

    1. system.access.audit
    2. system.access.table_lineage
    3. system.access.column_lineage
  4. Enable verbose audit logs in Unity Catalog.