# Databricks Unity Catalog Audit Logs Query audit for Databricks Unity Catalog captures user data access within Unity Catalog and presents them in a universal format as Immuta audit logs. Multiple access options are supported for audit: * Cluster queries with the following supported languages: SQL, Scala, Python, and R. * SQL warehouse queries Immuta audits the activity of all Unity Catalog users and tables regardless of whether they are registered in Immuta. ## Requirements * [Databricks Unity Catalog integration with query audit enabled](https://documentation.immuta.com/2024.2/data-and-integrations/databricks-unity-catalog/how-to-guides/configure) * A Databricks deployment with [system tables](https://docs.databricks.com/en/administration-guide/system-tables/index.html) capabilities {% hint style="info" %} **Store audit logs** By default Immuta audit logs expire after 7 days. Export the universal audit model (UAM) logs to [S3](https://documentation.immuta.com/2024.2/detect-your-activity/audit/how-to-guides/enable-uam) or [ADLS Gen 2](https://documentation.immuta.com/2024.2/detect-your-activity/audit/how-to-guides/export-adls), and store audit logs outside of Immuta in order to retain the audit logs long-term. Alternatively, you can [adjust this default duration when deploying Immuta](https://documentation.immuta.com/2024.2/self-managed-deployment/install/generic-installation#install-immuta). If you need to adjust this setting after deployment, contact your Immuta representative. {% endhint %} ## Audit frequency Immuta collects audit records once an hour by default, and the frequency can be [changed at any time from the Immuta app settings page](https://documentation.immuta.com/2024.2/application-settings/how-to-guides/config-builder-guide#databricks-unity-catalog-configuration). The frequency is a global setting based on integration type, so organizations with multiple Databricks Unity Catalog integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records. However, there could be performance and cost impacts from the frequent jobs because Immuta will start a Databricks cluster to complete the audit ingest job if one is not already running. To manually prompt the query audit, click **Load Audit Events** on the Immuta audit page. ## Audit scope Immuta audits all data sources and users in Unity Catalog. An administrator can configure the integration to just ingest specific workspaces when [enabling the integration](https://documentation.immuta.com/2024.2/data-and-integrations/databricks-unity-catalog/how-to-guides/configure). The audit is intended to ingest user queries for data access, so queries with terms[^1] that signify transformation queries are filtered out of audit. ## Audit schema Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below. | Property | Description | Example | | --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | | **action** | The action associated with the audit log. | `QUERY` | | actor.**type** | The Immuta user type of the actor who made the query. When the actor is not registered with Immuta, the `type`, `id`, and `name` fields will be `unknown`. | `USER_ACTOR` | | actor.**id** | The Immuta user ID of the actor who made the query. When the actor is not registered with Immuta, the `type`, `id`, and `name` fields will be `unknown`. | `taylor@databricks.com` | | actor.**name** | The Immuta name of the user who made the query. When the user is not registered with Immuta, the `type`, `id`, and `name` fields will be `unknown`. | `Taylor` | | actor.**identityProvider** | The IAM the user is registered in. `bim` is the built-in Immuta IAM. When the user is not registered with Immuta, this field will be omitted. | `bim` | | actor.**profileId** | The profile ID of the user who made the query. When the user is not registered with Immuta, this field will be omitted. | `10` | | **sessionId** | The session ID of the user who performed the action. | `01ee14d9-cab3-1ef6-9cc4-f0c315a53788` | | **requestId** | The API request ID that triggered the action, if applicable. | `504b8fd9-38c1-4a90-966e-7445a6675f79` | | **actionStatus** | Indicates whether or not the user was granted access to the data. Possible values are `UNAUTHORIZED`, `FAILURE`, or `SUCCESS`. | `SUCCESS` | | **actionStatusReason** | When available, the reason from Unity Catalog that the user’s query was denied. | `null` if **actionStatus** is `SUCCESS` | | **eventTimestamp** | The time the query occurred. | `2023-06-27T11:03:59.000Z` | | **id** | The unique ID of the audit record. | `9f542dfd-5099-4362-a72d-8377306db3b8` | | **tenantId** | The Immuta tenant ID. | `your-immuta.com` | | **userAgent** | Client information of the user who made the query. | - | | **targetType** | The type of targets affected by the query; this value will always be `DATASOURCE`. | `DATASOURCE` | | **targets** | A list of the targets affected by the query. | *See the example below* | | auditPayload.**type** | The type of audit record; this value will always be: `QueryAuditPayload`. | `QueryAuditPayload` | | auditPayload.**queryId** | The unique ID of the query. If the query joins multiple tables, each table will appear as a separate log, but all will have the same query ID. | `01ee14da-517a-1670-afce-0c3e0fdcf7d4` | | auditPayload.**query** | The command text of the query that was run in the integration. Immuta truncates the query text to the first 2048 characters. | `SELECT VERSION AS 'version' FROM 'sample-data'.'__immuta_version'` | | auditPayload.**startTime** | The date and time the query started in UTC. | `2023-06-27T11:03:59.000Z` | | auditPayload.**duration** | The time the query took in seconds. | `0.557` | | auditPayload.**errorCode** | The `errorCode` for the denied query. | `null` if **actionStatus** is `SUCCESS` | | auditPayload.technologyContext.**type** | The technology the query was made in. | `DatabricksContext` | | auditPayload.technologyContext.**clusterId** | The Unity Catalog cluster ID. | `null` | | auditPayload.technologyContext.**workspaceId** | The Unity Catalog workspace ID. | `8765531160949612` | | auditPayload.technologyContext.**service** | Where in Unity Catalog the query was made. Possible values are `SQL` for SQL warehouses and `NOTEBOOK` for notebooks. | `SQL` | | auditPayload.technologyContext.**warehouseId** | The Unity Catalog warehouse ID. | `559483c6eac0359f` | | auditPayload.technologyContext.**notebookId** | The Unity Catalog notebook ID. | `869500255746458` | | auditPayload.technologyContext.account.**id** | The actor’s Unity Catalog account ID | `52e863bc-ea7f-46a9-8e17-6aed7541832d` | | auditPayload.technologyContext.account.**username** | The actor’s Unity Catalog username. | `taylor@databricks.com` | | auditPayload.technologyContext.**host** | The Unity Catalog host. | `deployment-name.cloud.databricks.com` | | auditPayload.technologyContext.**clientIp** | The IP address of the Spark cluster the request is coming from. | `0.0.0.0` | | auditPayload.technologyContext.**queryLanguage** | The `queryLanguage` corresponds to the programming language used: SQL, Python, Scala, or R. Audited JDBC queries will indicate that it came from JDBC here. | `python` | | auditPayload.**objectsAccessed** | The Unity Catalog objects accessed. | `[]` | | auditPayload.securityProfile.sensitivity.**score** | The sensitivity score of the query. Classification must be configured for this field. | `INDETERMINATE` | | auditPayload.**version** | The version of the audit event schema. | `1` | | **receivedTimestamp** | The timestamp of when the audit event was received and stored by Immuta. | `2023-06-27T15:18:22.314Z` | ## Example audit record ```json { "action": "QUERY", "actor": { "type": "USER_ACTOR", "id": "taylor@immuta.com", "name": "Taylor", "identityProvider": "bim", "profileId": "10" }, "sessionId": "01ee14d9-cab3-1ef6-9cc4-f0c315a53788", "requestId": "504b8fd9-38c1-4a90-966e-7445a6675f79", "actionStatus": "SUCCESS", "actionStatusReason": null, "eventTimestamp": "2023-06-27T11:03:59.000Z", "id": "01ee14da-517a-1670-afce-0c3e0fdcf7d4", "tenantId": "your-immuta.com", "userAgent": "", "targetType": "DATASOURCE", "targets": [ { "type": "DATASOURCE", "id": "2034", "name": "University Art Gallery Exhibition", "technology": "DATABRICKS" } ], "relatedResources": [], "auditPayload": { "type": "QueryAuditPayload", "queryId": "01ee14da-517a-1670-afce-0c3e0fdcf7d4", "query": "SELECT VERSION AS `version` FROM `sample-data`.`__immuta_version`", "startTime": "2023-06-27T11:03:59.000Z", "duration": 23.568, "errorCode": null, "technologyContext": { "type": "DatabricksContext", "clusterId": null, "workspaceId": "3841033049363283", "service": "SQL", "warehouseId": "559483c6eac0359f", "notebookId": null, "account": { "id": "52e863bc-ea7f-46a9-8e17-6aed7541832d", "username": "taylor@databricks.com" }, "host": "deployment-name.cloud.databricks.com", "clientIp": "0.0.0.0" }, "objectsAccessed": [], "securityProfile": { "sensitivity": { "score": "INDETERMINATE" } }, "version": 1 }, "receivedTimestamp": "2023-06-27T15:18:22.314Z" } ``` ## Limitations * Enrichment of audit logs with Immuta entitlements information is not supported. While you will see these entitlements in the Databricks Spark audit logs, the following will not be in the query audit for Unity Catalog: * Immuta policies information * User attributes * Groups * Immuta determines unauthorized events based on error messages within Unity Catalog records. When the error messages contain expected language, unauthorized events will be available for query audit for Unity Catalog. In other cases, it is not possible to determine the cause of an error. * Audit for cluster queries do not support `UNAUTHORIZED` status. If a cluster query is unauthorized, it will show `FAILURE`. * Data source information will be provided when available: * For some queries, Databricks Unity Catalog does not report the target data source for the data access operation. In these cases the activity is audited, yet the audit record in Immuta will not include the target data source information. * Data source information is not available for unauthorized queries and events. * Column information from the query is not currently supported. * Immuta audit records include unregistered data sources and users; however, activity from them will not appear in any [governance reports](https://documentation.immuta.com/2024.2/detect-your-activity/audit/reference-guides/reports). [^1]: `ADD`, `ALTER`, `ANALYZE`, `CACHE`, `CLEAR`, `COMMENT ON`, `CONVERT`, `COPY`, `CREATE`, `DELETE`, `DESCRIBE`, `DROP`, `EXPLAIN`, `FSCK`, `GENERATE`, `GRANT`, `INSERT`, `LIST`, `LOAD`, `MSCK`, `MERGE`, `OPTIMIZE`, `REFRESH`, `REORG`, `REPAIR`, `REPLACE`, `RESTORE`, `REVOKE`, `SHOW`, `SYNC`, `TRUNCATE`, `UNCACHE`, `UNDROP`, `UPDATE`, `VACCUM`, or `VALUES`