In addition to the executed Spark plan, the tables, and the tables' underlying paths for every audited Spark job, Immuta captures the code or query that triggers the Spark plan. Immuta audits the activity of Immuta users on Immuta data sources.
Databricks users registered as Immuta users: Note that the users' Databricks usernames must be mapped to Immuta. Without this, Immuta will not know the users are Immuta users and will not collect audit events for their data access activity.
Store audit logs
By default Immuta audit logs expire after 7 days. Export the universal audit model (UAM) logs to S3 or ADLS Gen 2, and store audit logs outside of Immuta in order to retain the audit logs long-term.
Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below.
action
The action associated with the audit log.
QUERY
actor.type
The Immuta user type of the actor who made the query.
USER_ACTOR
actor.id
The Immuta user ID of the actor who made the query.
taylor@databricks.com
actor.name
The Immuta name of the user who made the query.
Taylor
actor.identityProvider
The IAM the user is registered in. bim is the built-in Immuta IAM.
bim
sessionId
The session ID of the user who performed the action.
01ee14d9-cab3-1ef6-9cc4-f0c315a53788
actionStatus
Indicates whether or not the user was granted access to the data. Possible values are UNAUTHORIZED, FAILURE, or SUCCESS.
SUCCESS
actionStatusReason
When a user's query is denied, this property explains why. When a query is successful, this value is null.
eventTimestamp
The time the query occurred.
2023-06-27T11:03:59.000Z
id
The unique ID of the audit record.
9f542dfd-5099-4362-a72d-8377306db3b8
targetType
The type of targets affected by the query; this value will always be DATASOURCE.
DATASOURCE
targets
A list of the targets affected by the query.
See the example below
auditPayload.type
The type of audit record; this value will always be: QueryAuditPayload.
QueryAuditPayload
auditPayload.queryId
The unique ID of the query. If the query joins multiple tables, each table will appear as a separate log, but all will have the same query ID.
01ee14da-517a-1670-afce-0c3e0fdcf7d4
auditPayload.query
The query that was run in the integration. Immuta truncates the query text to the first 2048 characters.
See the example below
auditPayload.startTime
The date and time the query started in UTC.
2023-06-27T11:03:59.000Z
auditPayload.duration
The time the query took in seconds.
0.557
auditPayload.accessControls
Includes the user's groups, attributes, and current project at the time of the query.
auditPayload.policySet
Provides policy details.
auditPayload.technologyContext.type
The technology the query was made in.
DatabricksContext
auditPayload.technologyContext.clusterId
The Databricks cluster ID.
null
auditPayload.technologyContext.clusterName
The Databricks cluster name.
databricks-cluster-name
auditPayload.technologyContext.workspaceId
The Databricks workspace ID.
8765531160949612
auditPayload.technologyContext.pathUris
The Databricks URI scheme for the storage type.
["dbfs:/user/hive/warehouse/your_database.db/movies"]
auditPayload.technologyContext.metastoreTables
The Databricks metastore tables.
["your_database.movies"]
auditPayload.technologyContext.queryLanguage
The queryLanguage corresponds to the programming language used: SQL, Python, Scala, or R. Audited JDBC queries will indicate that it came from JDBC here.
python
auditPayload.technologyContext.queryText
The queryText will contain either the full notebook cell (when the query is the result of a notebook) or the full SQL query (when it is a query from a JDBC connection).
See the example below
auditPayload.technologyContext.immutaPluginVersion
The Immuta plugin version for the Databricks Spark integration.
2022.3.0-spark-3.1.1
receivedTimestamp
The timestamp of when the audit event was received and stored by Immuta.
2023-06-27T15:18:22.314Z
Below is an example of the queryText, which contains the full notebook cell (since the query was the result of a notebook). If the query had been from a JDBC connection, the queryText would contain the full SQL query.
This notebook cell had multiple audit records associated with it.
Beyond raw audit events (such as “John Doe queried Table X in Databricks"), the Databricks audit records include the policy information enforced during the query execution, even if a query was denied.
Queries will be denied if at least one of the conditions below is true:
User does not meet policy conditions.
User is not subscribed to the data source.
Data source is not in the user's current project.
Data source is in the user's current project, but the user is not subscribed to the data source.
Data source is not registered in Immuta.
The user's entitlements represent the state at the time of the query. This includes the following fields:
project
The user's current project.
attributes
The user's attributes.
groups
The user's groups.
impersonatedUsers
The user that the current user is impersonating.
The policySet includes the following fields:
subscriptionPolicyType
The type of subscription policy.
MANUAL, ADVANCED, or ENTITLEMENTS
type
Indicates whether the policy is a subscription or data policy. Query denied records will always be a subscription policy type.
SUBSCRIPTION or DATA
ruleAppliedForUser
True if the policy was applied for the user. If false, the user was an exception to the policy.
true or false
rationale
The policy rationale written by the policy creator.
-
global
True if the policy was a global policy. If false, the policy is local.
true or false
mergedPolicies
Shows the policy information for each of the merged global subscription policies, if available.
-
Starburst (Trino) query audit logs is a feature that audits queries that users run natively in Starburst (Trino) and presents them in a universal format as Immuta audit logs. Users can view audit records for queries made in Starburst (Trino) against Immuta data sources on the audit page. Immuta audits the activity of Immuta users on Immuta data sources.
Starburst (Trino) integration with the Starburst or Trino plugin version 443 or newer, or Trino 435 with the Immuta Trino 435.1 plugin
Starburst (Trino) users registered as Immuta users: Note that the users' Starburst (Trino) usernames must be mapped to Immuta. Without this, Immuta will not know the users are Immuta users and will not collect audit events for their data access activity.
Store audit logs
By default Immuta audit logs expire after 7 days. Export the universal audit model (UAM) logs to S3 or ADLS Gen 2, and store audit logs outside of Immuta in order to retain the audit logs long-term.
Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below.
action
The action associated with the audit log.
QUERY
actor.type
The Immuta user type of the actor who made the query.
USER_ACTOR
actor.id
The Immuta user ID of the actor who made the query.
taylor@starburst.com
actor.name
The Immuta name of the user who made the query.
Taylor
actor.identityProvider
The IAM the user is registered in. bim is the built-in Immuta IAM.
bim
actor.profileId
The profile ID of the user who made the query.
10
actionStatus
Indicates whether or not the user was granted access to the data. Possible values are FAILURE or SUCCESS. Unauthorized access is not audited for Starburst (Trino).
SUCCESS
eventTimestamp
The time the query occurred.
2023-06-27T11:03:59.000Z
id
The unique Immuta ID of the audit record. This will match the Trino query ID.
20240221_200952_00200_qhadw
tenantId
The Immuta SaaS tenant ID.
your-immuta.com
targetType
The type of targets affected by the query; this value will always be DATASOURCE.
DATASOURCE
targets
A list of the targets affected by the query.
See the example below
auditPayload.type
The type of audit record; this value will always be: QueryAuditPayload.
QueryAuditPayload
auditPayload.queryId
The unique Starburst (Trino) ID of the query.
20240221_200952_00200_qhadw
auditPayload.query
The command text of the query that was run in the integration. Immuta truncates the query text to the first 2048 characters.
select * from lineitem l join orders o on l.orderkey = o.orderkey limit 10
auditPayload.startTime
The date and time the query started in UTC.
2023-06-27T11:03:59.000Z
auditPayload.duration
The time the query took in seconds.
0.557
auditPayload.objectsAccessed
An array of the data sources accessed in the query.
See example below.
auditPayload.objectsAccessed.name
The name of the data source accessed in the query.
\"tpch\".\"tiny\".\"customer\"
auditPayload.objectsAccessed.datasourceId
The Immuta data source ID.
17
auditPayload.objectsAccessed.databaseName
The name of the Starburst (Trino) catalog.
tpch
auditPayload.objectsAccessed.schemaName
The name of the Starburst (Trino) schema.
tiny
auditPayload.objectsAccessed.type
Specifies if the queried data source is a table or view. Starburst (Trino) queries are always LOGICAL_TABLE, which could be either.
LOGICAL_TABLE
auditPayload.objectsAccessed.columns
An array of the columns accessed in the query.
See example below.
auditPayload.objectsAccessed.columns.name
The name of the column.
custkey
auditPayload.objectsAccessed.columns.tags
An array of the tags on the column.
See example below.
auditPayload.objectsAccessed.columns.securityProfile
See example below.
auditPayload.objectsAccessed.columns.inferred
If true, the column accessed has been determined by Immuta based on the available audit information from Starburst (Trino) and query parsing. It was not explicitly provided.
true
auditPayload.objectsAccessed.securityProfile
See example below.
auditPayload.technologyContext.type
The technology the query was made in.
TrinoContext
auditPayload.technologyContext.trinoUsername
The Starburst (Trino) user ID for the user who made the query.
taylor@starburst.com
auditPayload.technologyContext.immutaPluginVersion
The version of the Immuta plugin in Starburst (Trino).
437-SNAPSHOT
auditPayload.technologyContext.rowsProduced
The number of rows returned in the query.
3
auditPayload.version
The version of the audit event schema.
1
receivedTimestamp
The timestamp of when the audit event was received and stored by Immuta.
2023-06-27T15:18:22.314Z
objectsAccessed is not available with Hive or Iceberg views.
columnsAccessed will include columns related to the query that were not actually accessed in some cases:
For row access policies that rely on a column in the queried table, even if that column was not a part of the query, it will be included in the columnsAccessed.
For conditional masking, if the policy protects a column accessed, then the conditional column will be included in the columnsAccessed.
Snowflake query audit logs is a feature that audits queries that users run natively in Snowflake and presents them in a universal format as Immuta audit logs. Immuta uses the Snowflake QUERY_HISTORY and ACCESS_HISTORY tables and translates them into the audit logs that can be viewed at query events in the Immuta UI or exported to S3. Immuta audits the activity of Immuta users on Immuta data sources.
Snowflake Enterprise Edition or higher
Snowflake users registered as Immuta users: Note that the users' Snowflake usernames must be mapped to Immuta. Without this, Immuta will not know the users are Immuta users and will not collect audit events for their data access activity.
Store audit logs
By default Immuta audit logs expire after 7 days. Export the universal audit model (UAM) logs to S3 or ADLS Gen 2, and store audit logs outside of Immuta in order to retain the audit logs long-term.
Immuta collects audit records at the frequency configured when enabling the integration, which is between 1 and 24 hours. The frequency is a global setting based on integration type, so organizations with multiple Snowflake integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records; however, there could be performance and cost impacts from the frequent jobs.
To manually request native query audit ingestion, click Load Audit Events on the Immuta audit page.
Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below.
action
The action associated with the audit log.
QUERY
actor.type
The Immuta user type of the actor who made the query.
USER_ACTOR
actor.id
The Immuta user ID of the actor who made the query.
taylor@snowflake.com
actor.name
The Immuta name of the user who made the query.
Taylor
actor.identityProvider
The IAM the user is registered in. bim is the built-in Immuta IAM.
bim
sessionId
The session ID of the user who performed the action.
01ee14d9-cab3-1ef6-9cc4-f0c315a53788
actionStatus
Indicates whether or not the user was granted access to the data. Possible values are UNAUTHORIZED, FAILURE, or SUCCESS.
SUCCESS
actionStatusReason
When available, the reason from Unity Catalog that the user’s query was denied.
null if actionStatus is SUCCESS
eventTimestamp
The time the query occurred.
2023-06-27T11:03:59.000Z
id
The unique ID of the audit record.
9f542dfd-5099-4362-a72d-8377306db3b8
userAgent
Client information of the user who made the query.
Snowflake Web App
tenantId
The Immuta SaaS tenant ID.
your-immuta.com
targetType
The type of targets affected by the query; this value will always be DATASOURCE.
DATASOURCE
targets
A list of the targets affected by the query.
See the example below
auditPayload.type
The type of audit record; this value will always be: QueryAuditPayload.
QueryAuditPayload
auditPayload.queryId
The unique ID of the query. If the query joins multiple tables, each table will appear as a separate log, but all will have the same query ID.
01ee14da-517a-1670-afce-0c3e0fdcf7d4
auditPayload.query
The command text of the query that was run in the integration. Immuta truncates the query text to the first 2048 characters.
SELECT VERSION AS 'version' FROM 'sample-data'.'__immuta_version'
auditPayload.startTime
The date and time the query started in UTC.
2023-06-27T11:03:59.000Z
auditPayload.duration
The time the query took in seconds.
0.557
auditPayload.errorCode
The errorCode for the denied query.
null if actionStatus is SUCCESS
auditPayload.technologyContext.type
The technology the query was made in.
SnowflakeContext
auditPayload.technologyContext.host
The host that the integration is connected to.
your-hostname.snowflake.computing.com
auditPayload.technologyContext.snowflakeUsername
The user's Snowflake username.
taylor@snowflake.com
auditPayload.technologyContext.rowsProduced
The number of rows returned in the query. Note that rows produced will show 0 for cached queries.
3
auditPayload.technologyContext.roleName
The Snowflake role the user used to make the query.
ACCOUNTADMIN
auditPayload.technologyContext.warehouseId
The ID of the warehouse where the query was made.
null
auditPayload.technologyContext.warehouseName
The name of the warehouse where the query was made.
null
auditPayload.technologyContext.clusterNumber
The number of the cluster where the query was made.
0
auditPayload.objectsAccessed
An array of the data sources accessed in the query.
See example below.
auditPayload.objectsAccessed.name
The name of the data source accessed in the query.
\"tpch\".\"tiny\".\"customer\"
auditPayload.objectsAccessed.datasourceId
The Immuta data source ID.
17
auditPayload.objectsAccessed.databaseName
The name of the Snowflake database.
tpch
auditPayload.objectsAccessed.schemaName
The name of the Snowflake schema.
tiny
auditPayload.objectsAccessed.type
Specifies if the queried data source is a table or view.
TABLE
auditPayload.objectsAccessed.columns
An array of the columns accessed in the query.
See example below.
auditPayload.objectsAccessed.columns.name
The name of the column.
custkey
auditPayload.objectsAccessed.columns.tags
An array of the tags on the column.
See example below.
auditPayload.objectsAccessed.columns.securityProfile
See example below.
auditPayload.objectsAccessed.columns.inferred
If true, the column accessed has been determined by Immuta using query parsing; false if it is explicitly provided.
false
auditPayload.objectsAccessed.securityProfile
See example below.
auditPayload.securityProfile.sensitivity.score
The sensitivity score of the query. Classification must be configured for this field.
INDETERMINATE
receivedTimestamp
The timestamp of when the audit event was received and stored by Immuta.
2023-06-27T15:18:22.314Z
See the .
See the .
See the .
Details about the sensitivity of the column. Available when .
A classification for all the columns accessed together. Available when .
Details about the sensitivity of the column. Available when .
A classification for all the columns accessed together. Available when .
Native query audit for Databricks Unity Catalog captures user data access within Unity Catalog and presents them in a universal format as Immuta audit logs. Multiple access options are supported for audit:
Cluster queries with the following supported languages: SQL, Scala, Python, and R.
SQL warehouse queries
Immuta audits the activity of all Unity Catalog users and tables.
Databricks Unity Catalog integration with native query audit enabled
Store audit logs
By default Immuta audit logs expire after 7 days. Export the universal audit model (UAM) logs to S3 or ADLS Gen 2, and store audit logs outside of Immuta in order to retain the audit logs long-term.
Immuta collects audit records at the frequency configured when enabling the integration, which is between 1 and 24 hours. The frequency is a global setting based on integration type, so organizations with multiple Unity Catalog integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records; however, there could be performance and cost impacts from the frequent jobs. Immuta will start a Databricks cluster to complete the audit ingest job if one is not already running.
To manually prompt the native query audit, click Load Audit Events on the Immuta audit page.
Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below.
action
The action associated with the audit log.
QUERY
actor.type
The Immuta user type of the actor who made the query. When the actor is not registered with Immuta, the type, id, and name fields will be unknown.
USER_ACTOR
actor.id
The Immuta user ID of the actor who made the query. When the actor is not registered with Immuta, the type, id, and name fields will be unknown.
taylor@databricks.com
actor.name
The Immuta name of the user who made the query. When the user is not registered with Immuta, the type, id, and name fields will be unknown.
Taylor
actor.identityProvider
The IAM the user is registered in. bim is the built-in Immuta IAM. When the user is not registered with Immuta, this field will be omitted.
bim
actor.profileId
The profile ID of the user who made the query. When the user is not registered with Immuta, this field will be omitted.
10
sessionId
The session ID of the user who performed the action.
01ee14d9-cab3-1ef6-9cc4-f0c315a53788
requestId
The API request ID that triggered the action, if applicable.
504b8fd9-38c1-4a90-966e-7445a6675f79
actionStatus
Indicates whether or not the user was granted access to the data. Possible values are UNAUTHORIZED, FAILURE, or SUCCESS.
SUCCESS
actionStatusReason
When available, the reason from Unity Catalog that the user’s query was denied.
null if actionStatus is SUCCESS
eventTimestamp
The time the query occurred.
2023-06-27T11:03:59.000Z
id
The unique ID of the audit record.
9f542dfd-5099-4362-a72d-8377306db3b8
tenantId
The Immuta SaaS tenant ID.
your-immuta.com
userAgent
Client information of the user who made the query.
-
targetType
The type of targets affected by the query; this value will always be DATASOURCE.
DATASOURCE
targets
A list of the targets affected by the query.
See the example below
auditPayload.type
The type of audit record; this value will always be: QueryAuditPayload.
QueryAuditPayload
auditPayload.queryId
The unique ID of the query. If the query joins multiple tables, each table will appear as a separate log, but all will have the same query ID.
01ee14da-517a-1670-afce-0c3e0fdcf7d4
auditPayload.query
The command text of the query that was run in the integration. Immuta truncates the query text to the first 2048 characters.
SELECT VERSION AS 'version' FROM 'sample-data'.'__immuta_version'
auditPayload.startTime
The date and time the query started in UTC.
2023-06-27T11:03:59.000Z
auditPayload.duration
The time the query took in seconds.
0.557
auditPayload.errorCode
The errorCode for the denied query.
null if actionStatus is SUCCESS
auditPayload.technologyContext.type
The technology the query was made in.
DatabricksContext
auditPayload.technologyContext.clusterId
The Unity Catalog cluster ID.
null
auditPayload.technologyContext.workspaceId
The Unity Catalog workspace ID.
8765531160949612
auditPayload.technologyContext.service
Where in Unity Catalog the query was made. Possible values are SQL for SQL warehouses and NOTEBOOK for notebooks.
SQL
auditPayload.technologyContext.warehouseId
The Unity Catalog warehouse ID.
559483c6eac0359f
auditPayload.technologyContext.notebookId
The Unity Catalog notebook ID.
869500255746458
auditPayload.technologyContext.account.id
The actor’s Unity Catalog account ID
52e863bc-ea7f-46a9-8e17-6aed7541832d
auditPayload.technologyContext.account.username
The actor’s Unity Catalog username.
taylor@databricks.com
auditPayload.technologyContext.host
The Unity Catalog host.
deployment-name.cloud.databricks.com
auditPayload.technologyContext.clientIp
The IP address of the Spark cluster the request is coming from.
0.0.0.0
auditPayload.technologyContext.queryLanguage
The queryLanguage corresponds to the programming language used: SQL, Python, Scala, or R. Audited JDBC queries will indicate that it came from JDBC here.
python
auditPayload.objectsAccessed
The Unity Catalog objects accessed.
[]
auditPayload.securityProfile.sensitivity.score
The sensitivity score of the query. Classification must be configured for this field.
INDETERMINATE
auditPayload.version
The version of the audit event schema.
1
receivedTimestamp
The timestamp of when the audit event was received and stored by Immuta.
2023-06-27T15:18:22.314Z
Enrichment of audit logs with Immuta entitlements information is not supported. While you will see these entitlements in the Databricks Spark audit logs, the following will not be in the native query audit for Unity Catalog:
Immuta policies information
User attributes
Groups
Immuta determines unauthorized events based on error messages within Unity Catalog records. When the error messages contain expected language, unauthorized events will be available for native query audit for Unity Catalog. In other cases, it is not possible to determine the cause of an error.
Audit for cluster queries do not support UNAUTHORIZED status. If a cluster query is unauthorized, it will show FAILURE.
Data source information will be provided when available:
For some queries, Databricks Unity Catalog does not report the target data source for the data access operation. In these cases the activity is audited, yet the audit record in Immuta will not include the target data source information.
Data source information is not available for unauthorized queries and events.
Column information from the query is not currently supported.
Immuta audit records include unregistered data sources and users; however, activity from them will not appear in any governance reports.