LogoLogo
2024.2
  • Immuta Documentation - 2024.2
  • What is Immuta?
  • Self-Managed Deployment
    • Getting Started
    • Deployment Requirements
    • Install
      • Managed Public Cloud
      • Red Hat OpenShift
      • Generic Installation
      • Immuta in an Air-Gapped Environment
      • Deploy Immuta without Elasticsearch
    • Configure
      • Ingress Configuration
      • Cosign Verification
      • TLS Configuration
      • Immuta in Production
      • External Cache Configuration
      • Rotating Credentials
      • Enabling Legacy Query Engine and Fingerprint
    • Upgrade
      • Upgrade Immuta
      • Upgrade to Immuta 2024.2 LTS
    • Disaster Recovery
    • Troubleshooting
    • Conventions
    • Release Notes
  • Data and Integrations
    • Immuta Integrations
    • Snowflake
      • Getting Started
      • How-to Guides
        • Configure a Snowflake Integration
        • Snowflake Table Grants Migration
        • Edit or Remove Your Snowflake Integration
        • Integration Settings
          • Enable Snowflake Table Grants
          • Use Snowflake Data Sharing with Immuta
          • Configure Snowflake Lineage Tag Propagation
          • Enable Snowflake Low Row Access Policy Mode
            • Upgrade Snowflake Low Row Access Policy Mode
      • Reference Guides
        • Snowflake Integration
        • Snowflake Data Sharing
        • Snowflake Lineage Tag Propagation
        • Snowflake Low Row Access Policy Mode
        • Snowflake Table Grants
        • Warehouse Sizing Recommendations
      • Phased Snowflake Onboarding Concept Guide
    • Databricks Unity Catalog
      • Getting Started
      • How-to Guides
        • Configure a Databricks Unity Catalog Integration
        • Migrate to Unity Catalog
      • Databricks Unity Catalog Integration Reference Guide
    • Databricks Spark
      • How-to Guides
        • Configuration
          • Simplified Databricks Configuration
          • Manual Databricks Configuration
          • Manually Update Your Databricks Cluster
          • Install a Trusted Library
        • DBFS Access
        • Limited Enforcement in Databricks
        • Hide the Immuta Database in Databricks
        • Run spark-submit Jobs on Databricks
        • Configure Project UDFs Cache Settings
        • External Metastores
      • Reference Guides
        • Databricks Spark Integration
        • Databricks Spark Pre-Configuration Details
        • Configuration Settings
          • Cluster Policies
            • Python & SQL
            • Python & SQL & R
            • Python & SQL & R with Library Support
            • Scala
            • Sparklyr
          • Environment Variables
          • Ephemeral Overrides
          • Py4j Security Error
          • Scala Cluster Security Details
          • Databricks Security Configuration for Performance
        • Databricks Change Data Feed
        • Databricks Libraries Introduction
        • Delta Lake API
        • Spark Direct File Reads
        • Databricks Metastore Magic
    • Starburst (Trino)
      • Getting Started
      • How-to Guides
        • Configure Starburst (Trino) Integration
        • Customize Read and Write Access Policies for Starburst (Trino)
      • Starburst (Trino) Integration Reference Guide
    • Redshift
      • Getting Started
      • How-to Guides
        • Configure Redshift Integration
        • Configure Redshift Spectrum
      • Reference Guides
        • Redshift Integration
        • Redshift Pre-Configuration Details
    • Azure Synapse Analytics
      • Getting Started
      • Configure Azure Synapse Analytics Integration
      • Reference Guides
        • Azure Synapse Analytics Integration
        • Azure Synapse Analytics Pre-Configuration Details
    • Amazon S3
    • Google BigQuery
    • Legacy Integrations
      • Securing Hive and Impala Without Sentry
      • Enabling ImmutaGroupsMapping
    • Registering Metadata
      • Data Sources in Immuta
      • Register Data Sources
        • Create a Data Source
        • Create an Amazon S3 Data Source
        • Create a Google BigQuery Data Source
        • Bulk Create Snowflake Data Sources
      • Data Source Settings
        • How-to Guides
          • Manage Data Sources and Data Source Settings
          • Manage Data Source Members
          • Manage Access Requests and Tasks
          • Manage Data Dictionary Descriptions
          • Disable Immuta from Sampling Raw Data
        • Data Source Health Checks Reference Guide
      • Schema Monitoring
        • How-to Guides
          • Run Schema Monitoring and Column Detection Jobs
          • Manage Schema Monitoring
        • Reference Guides
          • Schema Monitoring
          • Schema Projects
        • Why Use Schema Monitoring?
    • Catalogs
      • Getting Started with External Catalogs
      • Configure an External Catalog
      • Reference Guides
        • External Catalogs
        • Custom REST Catalogs
          • Custom REST Catalog Interface Endpoints
    • Tags
      • How-to Guides
        • Create and Manage Tags
        • Add Tags to Data Sources and Projects
      • Tags Reference Guide
  • People
    • Getting Started
    • Identity Managers (IAMs)
      • How-to Guides
        • Microsoft Entra ID
        • Okta LDAP Interface
        • Okta and OpenID Connect
        • Integrate Okta SAML SCIM with Immuta
        • OneLogin with OpenID
        • Configure SAML IAM Protocol
      • Reference Guides
        • Identity Managers
        • SAML Single Logout
        • SAML Protocol Configuration Options
    • Immuta Users
      • How-to Guides
        • Managing Personas and Permissions
        • Manage Attributes and Groups
        • User Impersonation
        • External User ID Mapping
        • External User Info Endpoint
      • Reference Guides
        • Attributes and Groups in Immuta
        • Permissions and Personas
  • Discover Your Data
    • Getting Started
    • Introduction
    • Architecture
    • Data Discovery
      • How-to Guides
        • Enable Sensitive Data Discovery (SDD)
        • Manage Identification Frameworks
        • Manage Patterns
        • Manage Rules
        • Manage SDD on Data Sources
        • Manage Global SDD Settings
        • Migrate From Legacy to Native SDD
      • Reference Guides
        • How Competitive Pattern Analysis Works
        • Built-in Pattern Reference
        • Built-in Discovered Tags Reference
    • Data Classification
      • How-to Guides
        • Activate Classification Frameworks
        • Adjust Identification and Classification Framework Tags
        • How to Use a Built-In Classification Framework with Your Own Tags
      • Built-in Classification Frameworks Reference Guide
  • Detect Your Activity
    • Getting Started
      • Monitor and Secure Sensitive Data Platform Query Activity
        • User Identity Best Practices
        • Integration Architecture
        • Snowflake Roles Best Practices
        • Register Data Sources
        • Automate Entity and Sensitivity Discovery
        • Detect with Discover: Onboarding Guide
        • Using Immuta Detect
      • General Immuta Configuration
        • User Identity Best Practices
        • Integration Architecture
        • Databricks Roles Best Practices
        • Register Data Sources
    • Introduction
    • Audit
      • How-to Guides
        • Export Audit Logs to S3
        • Export Audit Logs to ADLS
        • Run Governance Reports
      • Reference Guides
        • Universal Audit Model (UAM)
        • Snowflake Query Audit Logs
        • Databricks Unity Catalog Audit Logs
        • Databricks Query Audit Logs
        • Starburst (Trino) Query Audit Logs
        • UAM Schema
        • Audit Export CLI
        • Governance Report Types
      • Deprecated Audit Guides
        • Legacy to UAM Migration
        • Download Audit Logs
        • System Audit Logs
    • Detection
      • Use the Detect Dashboards
      • Reference Guides
        • Detect
        • Detect Dashboards
        • Unknown Users in Audit Logs
    • Monitors
      • Manage Monitors and Observations
      • Detect Monitors Reference Guide
  • Secure Your Data
    • Getting Started with Secure
      • Automate Data Access Control Decisions
        • The Two Paths: Orchestrated RBAC and ABAC
        • Managing User Metadata
        • Managing Data Metadata
        • Author Policy
        • Test and Deploy Policy
      • Compliantly Open More Sensitive Data for ML and Analytics
        • Managing User Metadata
        • Managing Data Metadata
        • Author Policy
      • Federated Governance for Data Mesh and Self-Serve Data Access
        • Defining Domains
        • Managing Data Products
        • Managing Data Metadata
        • Apply Federated Governance
        • Discover and Subscribe to Data Products
    • Introduction
      • Scalability and Evolvability
      • Understandability
      • Distributed Stewardship
      • Consistency
      • Availability of Data
    • Authoring Policies in Secure
      • Authoring Policies at Scale
      • Data Engineering with Limited Policy Downtime
      • Subscription Policies
        • How-to Guides
          • Author a Subscription Policy
          • Author an ABAC Subscription Policy
          • Subscription Policies Advanced DSL Guide
          • Author a Restricted Subscription Policy
          • Clone, Activate, or Stage a Global Policy
        • Reference Guides
          • Subscription Policies
          • Subscription Policy Access Types
          • Advanced Use of Special Functions
      • Data Policies
        • Overview
        • How-to Guides
          • Author a Masking Data Policy
          • Author a Minimization Policy
          • Author a Purpose-Based Restriction Policy
          • Author a Restricted Data Policy
          • Author a Row-Level Policy
          • Author a Time-Based Restriction Policy
          • Certifications Exemptions and Diffs
          • External Masking Interface
        • Reference Guides
          • Data Policy Types
          • Masking Policies
          • Row-Level Policies
          • Custom WHERE Clause Functions
          • Data Policy Conflicts and Fallback
          • Custom Data Policy Certifications
          • Orchestrated Masking Policies
    • Domains
      • Getting Started with Domains
      • Domains Reference Guide
    • Projects and Purpose-Based Access Control
      • Projects and Purpose Controls
        • Getting Started
        • How-to Guides
          • Create a Project
          • Create and Manage Purposes
          • Adjust a Policy
          • Project Management
            • Manage Projects and Project Settings
            • Manage Project Data Sources
            • Manage Project Members
        • Reference Guides
          • Projects and Purposes
          • Policy Adjustments
        • Why Use Purposes?
      • Equalized Access
        • Manage Project Equalization
        • Project Equalization Reference Guide
        • Why Use Project Equalization?
      • Masked Joins
        • Enable Masked Joins
        • Why Use Masked Joins?
      • Writing to Projects
        • How-to Guides
          • Create and Manage Snowflake Project Workspaces
          • Create and Manage Databricks Project Workspaces
          • Write Data to the Workspace
        • Reference Guides
          • Project Workspaces
          • Project UDFs (Databricks)
    • Data Consumers
      • Subscribe to a Data Source
      • Query Data
        • Querying Snowflake Data
        • Querying Databricks Data
        • Querying Databricks SQL Data
        • Querying Starburst (Trino) Data
        • Querying Redshift Data
        • Querying Azure Synapse Analytics Data
      • Subscribe to Projects
  • Application Settings
    • How-to Guides
      • App Settings
      • BI Tools
        • BI Tool Configuration Recommendations
        • Power BI Configuration Example
        • Tableau Configuration Example
      • Add a License Key
      • Add ODBC Drivers
      • Manage Encryption Keys
      • System Status Bundle
    • Reference Guides
      • Data Processing, Encryption, and Masking Practices
      • Metadata Ingestion
  • Releases
    • Immuta v2024.2 Release Notes
    • Immuta Release Lifecycle
    • Immuta LTS Changelog
    • Immuta Support Matrix Overview
    • Immuta CLI Release Notes
    • Immuta Image Digests
    • Preview Features
      • Features in Preview
    • Deprecations
  • Developer Guides
    • The Immuta CLI
      • Install and Configure the Immuta CLI
      • Manage Your Immuta Tenant
      • Manage Data Sources
      • Manage Sensitive Data Discovery
        • Manage Sensitive Data Discovery Rules
        • Manage Identification Frameworks
        • Run Sensitive Data Discovery on Data Sources
      • Manage Policies
      • Manage Projects
      • Manage Purposes
    • The Immuta API
      • Integrations API
        • Getting Started
        • How-to Guides
          • Configure an Amazon S3 Integration
          • Configure an Azure Synapse Analytics Integration
          • Configure a Databricks Unity Catalog Integration
          • Configure a Google BigQuery Integration
          • Configure a Redshift Integration
          • Configure a Snowflake Integration
          • Configure a Starburst (Trino) Integration
        • Reference Guides
          • Integrations API Endpoints
          • Integration Configuration Payload
          • Response Schema
          • HTTP Status Codes and Error Messages
      • Immuta V2 API
        • Data Source Payload Attribute Details
        • Data Source Request Payload Examples
        • Create Policies API Examples
        • Create Projects API Examples
        • Create Purposes API Examples
      • Immuta V1 API
        • Authenticate with the API
        • Configure Your Instance of Immuta
          • Get Fingerprint Status
          • Get Job Status
          • Manage Frameworks
          • Manage IAMs
          • Manage Licenses
          • Manage Notifications
          • Manage Sensitive Data Discovery (SDD)
          • Manage Tags
          • Manage Webhooks
          • Search Filters
        • Connect Your Data
          • Create and Manage an Amazon S3 Data Source
          • Create an Azure Synapse Analytics Data Source
          • Create an Azure Blob Storage Data Source
          • Create a Databricks Data Source
          • Create a Presto Data Source
          • Create a Redshift Data Source
          • Create a Snowflake Data Source
          • Create a Starburst (Trino) Data Source
          • Manage the Data Dictionary
        • Manage Data Access
          • Manage Access Requests
          • Manage Data and Subscription Policies
          • Manage Domains
          • Manage Write Policies
            • Write Policies Payloads and Response Schema Reference Guide
          • Policy Handler Objects
          • Search Audit Logs
          • Search Connection Strings
          • Search for Organizations
          • Search Schemas
        • Subscribe to and Manage Data Sources
        • Manage Projects and Purposes
          • Manage Projects
          • Manage Purposes
        • Generate Governance Reports
Powered by GitBook

Other versions

  • SaaS
  • 2024.3

Copyright © 2014-2024 Immuta Inc. All rights reserved.

On this page
  • Requirement
  • Audit frequency
  • Audit schema
  • Example audit record
  • Limitations

Was this helpful?

Export as PDF
  1. Detect Your Activity
  2. Audit
  3. Reference Guides

Databricks Unity Catalog Audit Logs

PreviousSnowflake Query Audit LogsNextDatabricks Query Audit Logs

Last updated 16 days ago

Was this helpful?

Query audit for Databricks Unity Catalog captures user data access within Unity Catalog and presents them in a universal format as Immuta audit logs. Multiple access options are supported for audit:

  • Cluster queries with the following supported languages: SQL, Scala, Python, and R.

  • SQL warehouse queries

Immuta audits the activity of all Unity Catalog users and tables.

Requirement

Store audit logs

By default Immuta audit logs expire after 7 days. Export the universal audit model (UAM) logs to or , and store audit logs outside of Immuta in order to retain the audit logs long-term.

Audit frequency

Immuta collects audit records once an hour by default, and the frequency can be . The frequency is a global setting based on integration type, so organizations with multiple Databricks Unity Catalog integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records. However, there could be performance and cost impacts from the frequent jobs because Immuta will start a Databricks cluster to complete the audit ingest job if one is not already running.

To manually prompt the query audit, click Load Audit Events on the Immuta audit page.

Audit schema

Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below.

Property
Description
Example

action

The action associated with the audit log.

QUERY

actor.type

The Immuta user type of the actor who made the query. When the actor is not registered with Immuta, the type, id, and name fields will be unknown.

USER_ACTOR

actor.id

The Immuta user ID of the actor who made the query. When the actor is not registered with Immuta, the type, id, and name fields will be unknown.

taylor@databricks.com

actor.name

The Immuta name of the user who made the query. When the user is not registered with Immuta, the type, id, and name fields will be unknown.

Taylor

actor.identityProvider

The IAM the user is registered in. bim is the built-in Immuta IAM. When the user is not registered with Immuta, this field will be omitted.

bim

actor.profileId

The profile ID of the user who made the query. When the user is not registered with Immuta, this field will be omitted.

10

sessionId

The session ID of the user who performed the action.

01ee14d9-cab3-1ef6-9cc4-f0c315a53788

requestId

The API request ID that triggered the action, if applicable.

504b8fd9-38c1-4a90-966e-7445a6675f79

actionStatus

Indicates whether or not the user was granted access to the data. Possible values are UNAUTHORIZED, FAILURE, or SUCCESS.

SUCCESS

actionStatusReason

When available, the reason from Unity Catalog that the user’s query was denied.

null if actionStatus is SUCCESS

eventTimestamp

The time the query occurred.

2023-06-27T11:03:59.000Z

id

The unique ID of the audit record.

9f542dfd-5099-4362-a72d-8377306db3b8

tenantId

The Immuta SaaS tenant ID.

your-immuta.com

userAgent

Client information of the user who made the query.

-

targetType

The type of targets affected by the query; this value will always be DATASOURCE.

DATASOURCE

targets

A list of the targets affected by the query.

See the example below

auditPayload.type

The type of audit record; this value will always be: QueryAuditPayload.

QueryAuditPayload

auditPayload.queryId

The unique ID of the query. If the query joins multiple tables, each table will appear as a separate log, but all will have the same query ID.

01ee14da-517a-1670-afce-0c3e0fdcf7d4

auditPayload.query

The command text of the query that was run in the integration. Immuta truncates the query text to the first 2048 characters.

SELECT VERSION AS 'version' FROM 'sample-data'.'__immuta_version'

auditPayload.startTime

The date and time the query started in UTC.

2023-06-27T11:03:59.000Z

auditPayload.duration

The time the query took in seconds.

0.557

auditPayload.errorCode

The errorCode for the denied query.

null if actionStatus is SUCCESS

auditPayload.technologyContext.type

The technology the query was made in.

DatabricksContext

auditPayload.technologyContext.clusterId

The Unity Catalog cluster ID.

null

auditPayload.technologyContext.workspaceId

The Unity Catalog workspace ID.

8765531160949612

auditPayload.technologyContext.service

Where in Unity Catalog the query was made. Possible values are SQL for SQL warehouses and NOTEBOOK for notebooks.

SQL

auditPayload.technologyContext.warehouseId

The Unity Catalog warehouse ID.

559483c6eac0359f

auditPayload.technologyContext.notebookId

The Unity Catalog notebook ID.

869500255746458

auditPayload.technologyContext.account.id

The actor’s Unity Catalog account ID

52e863bc-ea7f-46a9-8e17-6aed7541832d

auditPayload.technologyContext.account.username

The actor’s Unity Catalog username.

taylor@databricks.com

auditPayload.technologyContext.host

The Unity Catalog host.

deployment-name.cloud.databricks.com

auditPayload.technologyContext.clientIp

The IP address of the Spark cluster the request is coming from.

0.0.0.0

auditPayload.technologyContext.queryLanguage

The queryLanguage corresponds to the programming language used: SQL, Python, Scala, or R. Audited JDBC queries will indicate that it came from JDBC here.

python

auditPayload.objectsAccessed

The Unity Catalog objects accessed.

[]

auditPayload.securityProfile.sensitivity.score

The sensitivity score of the query. Classification must be configured for this field.

INDETERMINATE

auditPayload.version

The version of the audit event schema.

1

receivedTimestamp

The timestamp of when the audit event was received and stored by Immuta.

2023-06-27T15:18:22.314Z

Example audit record

{
  "action": "QUERY",
  "actor": {
    "type": "USER_ACTOR",
    "id": "taylor@immuta.com",
    "name": "Taylor",
    "identityProvider": "bim",
    "profileId": "10"
  },
  "sessionId": "01ee14d9-cab3-1ef6-9cc4-f0c315a53788",
  "requestId": "504b8fd9-38c1-4a90-966e-7445a6675f79",
  "actionStatus": "SUCCESS",
  "actionStatusReason": null,
  "eventTimestamp": "2023-06-27T11:03:59.000Z",
  "id": "01ee14da-517a-1670-afce-0c3e0fdcf7d4",
  "tenantId": "your-immuta.com",
  "userAgent": "",
  "targetType": "DATASOURCE",
  "targets": [
      {
      "type": "DATASOURCE",
      "id": "2034",
      "name": "University Art Gallery Exhibition",
      "technology": "DATABRICKS"
    }
  ],
  "relatedResources": [],
  "auditPayload": {
    "type": "QueryAuditPayload",
    "queryId": "01ee14da-517a-1670-afce-0c3e0fdcf7d4",
    "query": "SELECT VERSION AS `version` FROM `sample-data`.`__immuta_version`",
    "startTime": "2023-06-27T11:03:59.000Z",
    "duration": 23.568,
    "errorCode": null,
    "technologyContext": {
      "type": "DatabricksContext",
      "clusterId": null,
      "workspaceId": "3841033049363283",
      "service": "SQL",
      "warehouseId": "559483c6eac0359f",
      "notebookId": null,
      "account": {
        "id": "52e863bc-ea7f-46a9-8e17-6aed7541832d",
        "username": "taylor@databricks.com"
      },
      "host": "deployment-name.cloud.databricks.com",
      "clientIp": "0.0.0.0"
    },
    "objectsAccessed": [],
    "securityProfile": {
      "sensitivity": {
        "score": "INDETERMINATE"
      }
    },
    "version": 1
  },
  "receivedTimestamp": "2023-06-27T15:18:22.314Z"
}

Limitations

  • Enrichment of audit logs with Immuta entitlements information is not supported. While you will see these entitlements in the Databricks Spark audit logs, the following will not be in the query audit for Unity Catalog:

    • Immuta policies information

    • User attributes

    • Groups

  • Immuta determines unauthorized events based on error messages within Unity Catalog records. When the error messages contain expected language, unauthorized events will be available for query audit for Unity Catalog. In other cases, it is not possible to determine the cause of an error.

  • Audit for cluster queries do not support UNAUTHORIZED status. If a cluster query is unauthorized, it will show FAILURE.

  • Data source information will be provided when available:

    • For some queries, Databricks Unity Catalog does not report the target data source for the data access operation. In these cases the activity is audited, yet the audit record in Immuta will not include the target data source information.

    • Data source information is not available for unauthorized queries and events.

  • Column information from the query is not currently supported.

Immuta audit records include unregistered data sources and users; however, activity from them will not appear in any .

governance reports
Databricks Unity Catalog integration with query audit enabled
S3
ADLS Gen 2
changed at any time from the Immuta app settings page