Skip to content

Databricks Unity Catalog UAM Logs

Public preview

This feature is currently in public preview and available to all accounts.

Native query audit for Databricks Unity Catalog captures user data access within Unity Catalog and presents them in a universal format as Immuta audit logs. Immuta audits the activity of all Unity Catalog users and tables. Multiple access options are supported for audit:

  • Cluster queries with the following supported languages: SQL, Scala, Python, and R.
  • SQL warehouse queries

Requirements

Best practices: Store audit logs

By default Databricks Unity Catalog audit logs expire after 90 days. Export the universal audit model (UAM) logs to S3, and store audit logs outside of Immuta in order to retain the audit logs long-term.

Audit frequency

Immuta collects audit records at the frequency configured when enabling the integration, which is between 1 and 24 hours. The frequency is a global setting based on integration type, so organizations with multiple Unity Catalog integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records; however, there could be performance and cost impacts from the frequent jobs.

To manually prompt the native query audit, click Load Audit Events on the Immuta audit page.

Audit scope

Immuta audits all data sources and users in Unity Catalog. An administrator can configure the integration to just ingest specific workspaces when enabling the integration.

Audit schema

Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below.

Property Description Example
action The action associated with the audit log. QUERY
actor.type The Immuta user type of the actor who made the query. When the actor is not registered with Immuta, the type, id, and name fields will be unknown. USER_ACTOR
actor.id The Immuta user ID of the actor who made the query. When the actor is not registered with Immuta, the type, id, and name fields will be unknown. taylor@databricks.com
actor.name The Immuta name of the user who made the query. When the user is not registered with Immuta, the type, id, and name fields will be unknown. Taylor
actor.identityProvider The IAM the user is registered in. bim is the built-in Immuta IAM. When the user is not registered with Immuta, this field will be omitted. bim
actor.profileId The profile ID of the user who made the query. When the user is not registered with Immuta, this field will be omitted. 10
sessionId The session ID of the user who performed the action. 01ee14d9-cab3-1ef6-9cc4-f0c315a53788
requestId The API request ID that triggered the action, if applicable. 504b8fd9-38c1-4a90-966e-7445a6675f79
actionStatus Indicates whether or not the user was granted access to the data. Possible values are UNAUTHORIZED, FAILURE, or SUCCESS. SUCCESS
actionStatusReason When available, the reason from Unity Catalog that the user’s query was denied. null if actionStatus is SUCCESS
eventTimestamp The time the query occurred. 2023-06-27T11:03:59.000Z
id The unique ID of the audit record. 9f542dfd-5099-4362-a72d-8377306db3b8
tenantId The Immuta SaaS tenant ID. your-immuta.com
userAgent Client information of the user who made the query. -
targetType The type of targets affected by the query; this value will always be DATASOURCE. DATASOURCE
targets A list of the targets affected by the query. See the example below
auditPayload.type The type of audit record; this value will always be: QueryAuditPayload. QueryAuditPayload
auditPayload.queryId The unique ID of the query. If the query joins multiple tables, each table will appear as a separate log, but all will have the same query ID. 01ee14da-517a-1670-afce-0c3e0fdcf7d4
auditPayload.query The command text of the query that was run in the integration. Immuta truncates the query text to the first 2048 characters. SELECT VERSION AS 'version' FROM 'sample-data'.'__immuta_version'
auditPayload.startTime The date and time the query started in UTC. 2023-06-27T11:03:59.000Z
auditPayload.duration The time the query took in seconds. 0.557
auditPayload.errorCode The errorCode for the denied query. null if actionStatus is SUCCESS
auditPayload.technologyContext.type The technology the query was made in. DatabricksContext
auditPayload.technologyContext.clusterId The Unity Catalog cluster ID. null
auditPayload.technologyContext.workspaceId The Unity Catalog workspace ID. 8765531160949612
auditPayload.technologyContext.service Where in Unity Catalog the query was made. Possible values are SQL for SQL warehouses and NOTEBOOK for notebooks. SQL
auditPayload.technologyContext.warehouseId The Unity Catalog warehouse ID. 559483c6eac0359f
auditPayload.technologyContext.notebookId The Unity Catalog notebook ID. 869500255746458
auditPayload.technologyContext.account.id The actor’s Unity Catalog account ID 52e863bc-ea7f-46a9-8e17-6aed7541832d
auditPayload.technologyContext.account.username The actor’s Unity Catalog username. taylor@databricks.com
auditPayload.technologyContext.host The Unity Catalog host. deployment-name.cloud.databricks.com
auditPayload.technologyContext.clientIp The IP address of the Spark cluster the request is coming from. 0.0.0.0
auditPayload.objectsAccessed The Unity Catalog objects accessed. []
auditPayload.securityProfile.sensitivity.score The sensitivity score of the query. Classification must be configured for this field. INDETERMINATE
auditPayload.version The version of the audit event schema. 1
receivedTimestamp The timestamp of when the audit event was received and stored by Immuta. 2023-06-27T15:18:22.314Z

Example audit record

{
  "action": "QUERY",
  "actor": {
    "type": "USER_ACTOR",
    "id": "taylor@immuta.com",
    "name": "Taylor",
    "identityProvider": "bim",
    "profileId": "10"
  },
  "sessionId": "01ee14d9-cab3-1ef6-9cc4-f0c315a53788",
  "requestId": "504b8fd9-38c1-4a90-966e-7445a6675f79",
  "actionStatus": "SUCCESS",
  "actionStatusReason": null,
  "eventTimestamp": "2023-06-27T11:03:59.000Z",
  "id": "01ee14da-517a-1670-afce-0c3e0fdcf7d4",
  "tenantId": "your-immuta.com",
  "userAgent": "",
  "targetType": "DATASOURCE",
  "targets": [
      {
      "type": "DATASOURCE",
      "id": "2034",
      "name": "University Art Gallery Exhibition",
      "technology": "DATABRICKS"
    }
  ],
  "relatedResources": [],
  "auditPayload": {
    "type": "QueryAuditPayload",
    "queryId": "01ee14da-517a-1670-afce-0c3e0fdcf7d4",
    "query": "SELECT VERSION AS `version` FROM `sample-data`.`__immuta_version`",
    "startTime": "2023-06-27T11:03:59.000Z",
    "duration": 23.568,
    "errorCode": null,
    "technologyContext": {
      "type": "DatabricksContext",
      "clusterId": null,
      "workspaceId": "3841033049363283",
      "service": "SQL",
      "warehouseId": "559483c6eac0359f",
      "notebookId": null,
      "account": {
        "id": "52e863bc-ea7f-46a9-8e17-6aed7541832d",
        "username": "taylor@databricks.com"
      },
      "host": "deployment-name.cloud.databricks.com",
      "clientIp": "0.0.0.0"
    },
    "objectsAccessed": [],
    "securityProfile": {
      "sensitivity": {
        "score": "INDETERMINATE"
      }
    },
    "version": 1
  },
  "receivedTimestamp": "2023-06-27T15:18:22.314Z"
}

Limitations

  • Enrichment of audit logs with Immuta entitlements information is not supported. While you will see these entitlements in the Databricks Spark audit logs, the following will not be in the native query audit for Unity Catalog:
    • Immuta policies information
    • User attributes
    • Groups
  • Immuta determines unauthorized events based on error messages within Unity Catalog records. When the error messages contain expected language, unauthorized events will be available for native query audit for Unity Catalog. In other cases, it is not possible to determine the cause of an error.
  • Audit for cluster queries do not support UNAUTHORIZED status. If a cluster query is unauthorized, it will show FAILURE.
  • Data source information will be provided when available:
    • For some queries, Databricks Unity Catalog does not report the target data source for the data access operation. In these cases the activity is audited, yet the audit record in Immuta will not include the target data source information.
    • Data source information is not available for unauthorized queries and events.
  • Column information from the query is not currently supported.
  • The cluster for the Unity Catalog integration must always be running for Immuta to audit activity.
  • Immuta audit records include unregistered data sources and users; however, activity from them will not appear in any governance reports.

Enable native query audit for Unity Catalog

To enable native query audit for Unity Catalog, complete the following steps:

  1. Enable the integration as normal, and the Enable Native Query Audit checkbox will be selected by default.
  2. Enable a system schema where the <SCHEMA_NAME> is access.
  3. Grant your Immuta system account user access to the Databricks Unity Catalog system tables. Immuta must have access to the following tables:

    1. system.access.audit
    2. system.access.table_lineage
    3. system.access.column_lineage
  4. Enable verbose audit logs in Unity Catalog.