1 of 1

Databricks Spark Query Audit Logs

In addition to the executed Spark plan, the tables, and the tables' underlying paths for every audited Spark job, Immuta captures the code or query that triggers the Spark plan. Immuta audits the activity of Immuta users on Immuta data sources.

Requirements

Databricks Spark integration with native query audit enabled
Databricks tables registered as Immuta data sources
Databricks users registered as Immuta users: Note that the users' Databricks usernames must be mapped to Immuta. Without this, Immuta will not know the users are Immuta users and will not collect audit events for their data access activity.

Store audit logs

By default Immuta audit logs expire after 7 days. Export the universal audit model (UAM) logs to S3 or ADLS Gen 2, and store audit logs outside of Immuta in order to retain the audit logs long-term.

Audit schema

Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below.

Property

Description

Example

action

The action associated with the audit log.

QUERY

actor.type

The Immuta user type of the actor who made the query.

USER_ACTOR

actor.id

The Immuta user ID of the actor who made the query.

taylor@databricks.com

actor.name

The Immuta name of the user who made the query.

Taylor

actor.identityProvider

The IAM the user is registered in. bim is the built-in Immuta IAM.

bim

sessionId

The session ID of the user who performed the action.

01ee14d9-cab3-1ef6-9cc4-f0c315a53788

actionStatus

Indicates whether or not the user was granted access to the data. Possible values are UNAUTHORIZED, FAILURE, or SUCCESS.

SUCCESS

actionStatusReason

When a user's query is denied, this property explains why. When a query is successful, this value is null.

eventTimestamp

The time the query occurred.

2023-06-27T11:03:59.000Z

The unique ID of the audit record.

9f542dfd-5099-4362-a72d-8377306db3b8

targetType

The type of targets affected by the query; this value will always be DATASOURCE.

DATASOURCE

targets

A list of the targets affected by the query.

See the example below

auditPayload.type

The type of audit record; this value will always be: QueryAuditPayload.

QueryAuditPayload

auditPayload.queryId

The unique ID of the query. If the query joins multiple tables, each table will appear as a separate log, but all will have the same query ID.

01ee14da-517a-1670-afce-0c3e0fdcf7d4

auditPayload.query

The query that was run in the integration. Immuta truncates the query text to the first 2048 characters.

See the example below

auditPayload.startTime

The date and time the query started in UTC.

2023-06-27T11:03:59.000Z

auditPayload.duration

The time the query took in seconds.

0.557

auditPayload.accessControls

Includes the user's groups, attributes, and current project at the time of the query.

auditPayload.policySet

Provides policy details.

auditPayload.technologyContext.type

The technology the query was made in.

DatabricksContext

auditPayload.technologyContext.clusterId

The Databricks cluster ID.

null

auditPayload.technologyContext.clusterName

The Databricks cluster name.

databricks-cluster-name

auditPayload.technologyContext.workspaceId

The Databricks workspace ID.

8765531160949612

auditPayload.technologyContext.pathUris

The Databricks URI scheme for the storage type.

["dbfs:/user/hive/warehouse/your_database.db/movies"]

auditPayload.technologyContext.metastoreTables

The Databricks metastore tables.

["your_database.movies"]

auditPayload.technologyContext.queryLanguage

The queryLanguage corresponds to the programming language used: SQL, Python, Scala, or R. Audited JDBC queries will indicate that it came from JDBC here.

python

auditPayload.technologyContext.queryText

The queryText will contain either the full notebook cell (when the query is the result of a notebook) or the full SQL query (when it is a query from a JDBC connection).

See the example below

auditPayload.technologyContext.immutaPluginVersion

The Immuta plugin version for the Databricks Spark integration.

2022.3.0-spark-3.1.1

receivedTimestamp

The timestamp of when the audit event was received and stored by Immuta.

2023-06-27T15:18:22.314Z

Example queryText

Below is an example of the queryText, which contains the full notebook cell (since the query was the result of a notebook). If the query had been from a JDBC connection, the queryText would contain the full SQL query.

testTable = 'default.crime_data_delta'
testDb = 'test'

df = spark.table(testTable)
df.limit(1).collect()

filteredDf = df.filter('victim_age > 20')

filteredDf.write.saveAsTable('{}.audit_cell'.format(testDb))
spark.table('{}.audit_cell'.format(testDb)).limit(1).collect()

spark.sql('DROP TABLE IF EXISTS {}.audit_cell'.format(testDb))

This notebook cell had multiple audit records associated with it.

Example audit record

{
  "action": "QUERY",
  "actor": {
    "type": "USER_ACTOR",
    "name": "Taylor",
    "id": "taylor@immuta.com",
    "identityProvider": "okta",
    "impersonatedBy": null
  },
  "sessionId": "abc123456589",
  "actionStatus": "SUCCESS",
  "actionStatusReason": null,
  "actorIp": "1.2.3.4",
  "eventTimestamp": "2022-10-13T20:03:41.013Z",
  "id": "abc123",
  "customerId": "abc123",
  "targetType": "DATASOURCE",
  "targets": [{
    "id": "4",
    "name": "Movies",
    "technology": "DATABRICKS"
  }],
  "auditPayload": {
    "type": "QueryAuditPayload",
    "queryId": "81fe4385-1329-444a-b6d9-b26bce5c8dc7",
    "query": "Project [director#778904]\n+- Filter ((YEAR#778903L = 1999) OR (YEAR#778903L = 2000))\n   +- Relation[movie_id#778901L,Title#778902,Year#778903L,Director#778904,Budget_million#778905,Gross_worldwide#778906L] parquet\n",
    "startTime": "2022-10-13T20:03:41.013Z",
    "endTime": null,
    "duration": null,
    "accessControls": {
      "entitlements": {
        "groups": [],
        "attributes": []
      },
      "policySet": [{
        "type": "SUBSCRIPTION",
        "global": false,
        "subscriptionPolicyType": "MANUAL",
        "ruleAppliedForUser": true
      }]
    },
    "technologyContext": {
      "type": "DatabricksContext",
      "clusterId": "1006-194110-8j0shd5d",
      "clusterName": "databricks-cluster-name",
      "workspaceId": "123456789",
      "pathUris": [
        "dbfs:/user/hive/warehouse/your_database.db/movies"
      ],
      "metastoreTables": ["your_database.movies"],
      "queryLanguage": "python",
      "queryText": "query_success = []\nnum_queries_run = 0\nimpersonate_probability = .20\nspark.sql(\"set immuta.impersonate.user=\")\n\ndef make_fail_query(query):\n  try:\n    spark.sql(\"set immuta.impersonate.user=taylor@databricks.com\")\n    spark.sql(query).toPandas()\n  except: \n    pass\n  \nfor index, query in enumerate(new_queries.values):\n  if(num_queries_run % 100 == 0):\n    print(f\"Queries Successfully Ran: {num_queries_run}/2000, out of total queries ran: {index+1}\")\n  to_impersonate = random.randrange(100)\n  if to_impersonate < impersonate_probability * 100:\n    make_fail_query(query)\n    spark.sql(\"set immuta.impersonate.user=\")\n    num_queries_run += 1\n  else:\n    try:\n      spark.sql(query).toPandas()\n      query_success.append((query, True))\n      num_queries_run += 1\n      if num_queries_run == 2000:\n        break\n    except Exception as e:\n      query_success.append((query, False))\n      \n    ",
      "immutaPluginVersion": "2022.3.0-spark-3.1.1"
    }
  },
  "receivedTimestamp": "2022-10-13T20:03:41.044Z"
}

Enriched Databricks audit logs

Beyond raw audit events (such as “John Doe queried Table X in Databricks"), the Databricks audit records include the policy information enforced during the query execution, even if a query was denied.

Queries will be denied if at least one of the conditions below is true:

User does not meet policy conditions.
User is not subscribed to the data source.
Data source is not in the user's current project.
Data source is in the user's current project, but the user is not subscribed to the data source.
Data source is not registered in Immuta.

User entitlements

The user's entitlements represent the state at the time of the query. This includes the following fields:

Property

Description

project

The user's current project.

attributes

The user's attributes.

groups

The user's groups.

impersonatedUsers

The user that the current user is impersonating.

Policy information

The policySet includes the following fields:

Property

Description

Possible values

subscriptionPolicyType

The type of subscription policy.

MANUAL, ADVANCED, or ENTITLEMENTS

type

Indicates whether the policy is a subscription or data policy. Query denied records will always be a subscription policy type.

SUBSCRIPTION or DATA

ruleAppliedForUser

True if the policy was applied for the user. If false, the user was an exception to the policy.

true or false

rationale

The policy rationale written by the policy creator.

global

True if the policy was a global policy. If false, the policy is local.

true or false

mergedPolicies

Shows the policy information for each of the merged global subscription policies, if available.

Databricks Spark Query Audit Logs

Requirements

Databricks Spark integration with native query audit enabled
Databricks tables registered as Immuta data sources
Databricks users registered as Immuta users: Note that the users' Databricks usernames must be mapped to Immuta. Without this, Immuta will not know the users are Immuta users and will not collect audit events for their data access activity.

Store audit logs

By default Immuta audit logs expire after 7 days. Export the universal audit model (UAM) logs to S3 or ADLS Gen 2, and store audit logs outside of Immuta in order to retain the audit logs long-term.

Audit schema

Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below.

Property

Description

Example

action

The action associated with the audit log.

QUERY

actor.type

The Immuta user type of the actor who made the query.

USER_ACTOR

actor.id

The Immuta user ID of the actor who made the query.

taylor@databricks.com

actor.name

The Immuta name of the user who made the query.

Taylor

actor.identityProvider

The IAM the user is registered in. bim is the built-in Immuta IAM.

bim

sessionId

The session ID of the user who performed the action.

01ee14d9-cab3-1ef6-9cc4-f0c315a53788

actionStatus

Indicates whether or not the user was granted access to the data. Possible values are UNAUTHORIZED, FAILURE, or SUCCESS.

SUCCESS

actionStatusReason

When a user's query is denied, this property explains why. When a query is successful, this value is null.

See the .

eventTimestamp

The time the query occurred.

2023-06-27T11:03:59.000Z

The unique ID of the audit record.

9f542dfd-5099-4362-a72d-8377306db3b8

targetType

The type of targets affected by the query; this value will always be DATASOURCE.

DATASOURCE

targets

A list of the targets affected by the query.

See the example below

auditPayload.type

The type of audit record; this value will always be: QueryAuditPayload.

QueryAuditPayload

auditPayload.queryId

The unique ID of the query. If the query joins multiple tables, each table will appear as a separate log, but all will have the same query ID.

01ee14da-517a-1670-afce-0c3e0fdcf7d4

auditPayload.query

The query that was run in the integration. Immuta truncates the query text to the first 2048 characters.

See the example below

auditPayload.startTime

The date and time the query started in UTC.

2023-06-27T11:03:59.000Z

auditPayload.duration

The time the query took in seconds.

0.557

auditPayload.accessControls

Includes the user's groups, attributes, and current project at the time of the query.

See the .

auditPayload.policySet

Provides policy details.

See the .

auditPayload.technologyContext.type

The technology the query was made in.

DatabricksContext

auditPayload.technologyContext.clusterId

The Databricks cluster ID.

null

auditPayload.technologyContext.clusterName

The Databricks cluster name.

databricks-cluster-name

auditPayload.technologyContext.workspaceId

The Databricks workspace ID.

8765531160949612

auditPayload.technologyContext.pathUris

The Databricks URI scheme for the storage type.

["dbfs:/user/hive/warehouse/your_database.db/movies"]

auditPayload.technologyContext.metastoreTables

The Databricks metastore tables.

["your_database.movies"]

auditPayload.technologyContext.queryLanguage

The queryLanguage corresponds to the programming language used: SQL, Python, Scala, or R. Audited JDBC queries will indicate that it came from JDBC here.

python

auditPayload.technologyContext.queryText

The queryText will contain either the full notebook cell (when the query is the result of a notebook) or the full SQL query (when it is a query from a JDBC connection).

See the example below

auditPayload.technologyContext.immutaPluginVersion

The Immuta plugin version for the Databricks Spark integration.

2022.3.0-spark-3.1.1

receivedTimestamp

The timestamp of when the audit event was received and stored by Immuta.

2023-06-27T15:18:22.314Z

Example queryText

testTable = 'default.crime_data_delta'
testDb = 'test'

df = spark.table(testTable)
df.limit(1).collect()

filteredDf = df.filter('victim_age > 20')

filteredDf.write.saveAsTable('{}.audit_cell'.format(testDb))
spark.table('{}.audit_cell'.format(testDb)).limit(1).collect()

spark.sql('DROP TABLE IF EXISTS {}.audit_cell'.format(testDb))

This notebook cell had multiple audit records associated with it.

Example audit record

{
  "action": "QUERY",
  "actor": {
    "type": "USER_ACTOR",
    "name": "Taylor",
    "id": "taylor@immuta.com",
    "identityProvider": "okta",
    "impersonatedBy": null
  },
  "sessionId": "abc123456589",
  "actionStatus": "SUCCESS",
  "actionStatusReason": null,
  "actorIp": "1.2.3.4",
  "eventTimestamp": "2022-10-13T20:03:41.013Z",
  "id": "abc123",
  "customerId": "abc123",
  "targetType": "DATASOURCE",
  "targets": [{
    "id": "4",
    "name": "Movies",
    "technology": "DATABRICKS"
  }],
  "auditPayload": {
    "type": "QueryAuditPayload",
    "queryId": "81fe4385-1329-444a-b6d9-b26bce5c8dc7",
    "query": "Project [director#778904]\n+- Filter ((YEAR#778903L = 1999) OR (YEAR#778903L = 2000))\n   +- Relation[movie_id#778901L,Title#778902,Year#778903L,Director#778904,Budget_million#778905,Gross_worldwide#778906L] parquet\n",
    "startTime": "2022-10-13T20:03:41.013Z",
    "endTime": null,
    "duration": null,
    "accessControls": {
      "entitlements": {
        "groups": [],
        "attributes": []
      },
      "policySet": [{
        "type": "SUBSCRIPTION",
        "global": false,
        "subscriptionPolicyType": "MANUAL",
        "ruleAppliedForUser": true
      }]
    },
    "technologyContext": {
      "type": "DatabricksContext",
      "clusterId": "1006-194110-8j0shd5d",
      "clusterName": "databricks-cluster-name",
      "workspaceId": "123456789",
      "pathUris": [
        "dbfs:/user/hive/warehouse/your_database.db/movies"
      ],
      "metastoreTables": ["your_database.movies"],
      "queryLanguage": "python",
      "queryText": "query_success = []\nnum_queries_run = 0\nimpersonate_probability = .20\nspark.sql(\"set immuta.impersonate.user=\")\n\ndef make_fail_query(query):\n  try:\n    spark.sql(\"set immuta.impersonate.user=taylor@databricks.com\")\n    spark.sql(query).toPandas()\n  except: \n    pass\n  \nfor index, query in enumerate(new_queries.values):\n  if(num_queries_run % 100 == 0):\n    print(f\"Queries Successfully Ran: {num_queries_run}/2000, out of total queries ran: {index+1}\")\n  to_impersonate = random.randrange(100)\n  if to_impersonate < impersonate_probability * 100:\n    make_fail_query(query)\n    spark.sql(\"set immuta.impersonate.user=\")\n    num_queries_run += 1\n  else:\n    try:\n      spark.sql(query).toPandas()\n      query_success.append((query, True))\n      num_queries_run += 1\n      if num_queries_run == 2000:\n        break\n    except Exception as e:\n      query_success.append((query, False))\n      \n    ",
      "immutaPluginVersion": "2022.3.0-spark-3.1.1"
    }
  },
  "receivedTimestamp": "2022-10-13T20:03:41.044Z"
}

Enriched Databricks audit logs

Queries will be denied if at least one of the conditions below is true:

User does not meet policy conditions.
User is not subscribed to the data source.
Data source is not in the user's current project.
Data source is in the user's current project, but the user is not subscribed to the data source.
Data source is not registered in Immuta.

User entitlements

The user's entitlements represent the state at the time of the query. This includes the following fields:

Property

Description

project

The user's current project.

attributes

The user's attributes.

groups

The user's groups.

impersonatedUsers

The user that the current user is impersonating.

Policy information

The policySet includes the following fields:

Property

Description

Possible values

subscriptionPolicyType

The type of subscription policy.

MANUAL, ADVANCED, or ENTITLEMENTS

type

Indicates whether the policy is a subscription or data policy. Query denied records will always be a subscription policy type.

SUBSCRIPTION or DATA

ruleAppliedForUser

True if the policy was applied for the user. If false, the user was an exception to the policy.

true or false

rationale

The policy rationale written by the policy creator.

global

True if the policy was a global policy. If false, the policy is local.

true or false

mergedPolicies

Shows the policy information for each of the merged global subscription policies, if available.