Skip to content

Databricks Query Audit Logs

In addition to the executed Spark plan, the tables, and the tables' underlying paths for every audited Spark job, Immuta captures the code or query that triggers the Spark plan. Immuta audits the activity of Immuta users on Immuta data sources.

Requirements

Best practices: Store audit logs

By default Immuta audit logs expire after 7 days. Export the universal audit model (UAM) logs to S3 or ADLS Gen 2, and store audit logs outside of Immuta in order to retain the audit logs long-term.

Audit schema

Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below.

Property Description Example
action The action associated with the audit log. QUERY
actor.type The Immuta user type of the actor who made the query. USER_ACTOR
actor.id The Immuta user ID of the actor who made the query. taylor@databricks.com
actor.name The Immuta name of the user who made the query. Taylor
actor.identityProvider The IAM the user is registered in. bim is the built-in Immuta IAM. bim
sessionId The session ID of the user who performed the action. 01ee14d9-cab3-1ef6-9cc4-f0c315a53788
actionStatus Indicates whether or not the user was granted access to the data. Possible values are UNAUTHORIZED, FAILURE, or SUCCESS. SUCCESS
actionStatusReason When a user's query is denied, this property explains why. When a query is successful, this value is null. See the Enriched Databricks Audit Logs section for details.
eventTimestamp The time the query occurred. 2023-06-27T11:03:59.000Z
id The unique ID of the audit record. 9f542dfd-5099-4362-a72d-8377306db3b8
targetType The type of targets affected by the query; this value will always be DATASOURCE. DATASOURCE
targets A list of the targets affected by the query. See the example below
auditPayload.type The type of audit record; this value will always be: QueryAuditPayload. QueryAuditPayload
auditPayload.queryId The unique ID of the query. If the query joins multiple tables, each table will appear as a separate log, but all will have the same query ID. 01ee14da-517a-1670-afce-0c3e0fdcf7d4
auditPayload.query The query that was run in the integration. Immuta truncates the query text to the first 2048 characters. See the example below
auditPayload.startTime The date and time the query started in UTC. 2023-06-27T11:03:59.000Z
auditPayload.duration The time the query took in seconds. 0.557
auditPayload.accessControls Includes the user's groups, attributes, and current project at the time of the query. See the Enriched Databricks Audit Logs section for details.
auditPayload.policySet Provides policy details. See the Enriched Databricks Audit Logs section for details.
auditPayload.technologyContext.type The technology the query was made in. DatabricksContext
auditPayload.technologyContext.clusterId The Databricks cluster ID. null
auditPayload.technologyContext.clusterName The Databricks cluster name. databricks-cluster-name
auditPayload.technologyContext.workspaceId The Databricks workspace ID. 8765531160949612
auditPayload.technologyContext.pathUris The Databricks URI scheme for the storage type. ["dbfs:/user/hive/warehouse/your_database.db/movies"]
auditPayload.technologyContext.metastoreTables The Databricks metastore tables. ["your_database.movies"]
auditPayload.technologyContext.queryLanguage The queryLanguage corresponds to the programming language used: SQL, Python, Scala, or R. Audited JDBC queries will indicate that it came from JDBC here. python
auditPayload.technologyContext.queryText The queryText will contain either the full notebook cell (when the query is the result of a notebook) or the full SQL query (when it is a query from a JDBC connection). See the example below
auditPayload.technologyContext.immutaPluginVersion The Immuta plugin version for the Databricks integration. 2022.3.0-spark-3.1.1
receivedTimestamp The timestamp of when the audit event was received and stored by Immuta. 2023-06-27T15:18:22.314Z

Example queryText

Below is an example of the queryText, which contains the full notebook cell (since the query was the result of a notebook). If the query had been from a JDBC connection, the queryText would contain the full SQL query.

testTable = 'default.crime_data_delta'
testDb = 'test'

df = spark.table(testTable)
df.limit(1).collect()

filteredDf = df.filter('victim_age > 20')

filteredDf.write.saveAsTable('{}.audit_cell'.format(testDb))
spark.table('{}.audit_cell'.format(testDb)).limit(1).collect()

spark.sql('DROP TABLE IF EXISTS {}.audit_cell'.format(testDb))

This notebook cell had multiple audit records associated with it.

Example audit record

{
  "action": "QUERY",
  "actor": {
    "type": "USER_ACTOR",
    "name": "Taylor",
    "id": "taylor@immuta.com",
    "identityProvider": "okta",
    "impersonatedBy": null
  },
  "sessionId": "abc123456589",
  "actionStatus": "SUCCESS",
  "actionStatusReason": null,
  "actorIp": "1.2.3.4",
  "eventTimestamp": "2022-10-13T20:03:41.013Z",
  "id": "abc123",
  "customerId": "abc123",
  "targetType": "DATASOURCE",
  "targets": [{
    "id": "4",
    "name": "Movies",
    "technology": "DATABRICKS"
  }],
  "auditPayload": {
    "type": "QueryAuditPayload",
    "queryId": "81fe4385-1329-444a-b6d9-b26bce5c8dc7",
    "query": "Project [director#778904]\n+- Filter ((YEAR#778903L = 1999) OR (YEAR#778903L = 2000))\n   +- Relation[movie_id#778901L,Title#778902,Year#778903L,Director#778904,Budget_million#778905,Gross_worldwide#778906L] parquet\n",
    "startTime": "2022-10-13T20:03:41.013Z",
    "endTime": null,
    "duration": null,
    "accessControls": {
      "entitlements": {
        "groups": [],
        "attributes": []
      },
      "policySet": [{
        "type": "SUBSCRIPTION",
        "global": false,
        "subscriptionPolicyType": "MANUAL",
        "ruleAppliedForUser": true
      }]
    },
    "technologyContext": {
      "type": "DatabricksContext",
      "clusterId": "1006-194110-8j0shd5d",
      "clusterName": "databricks-cluster-name",
      "workspaceId": "123456789",
      "pathUris": [
        "dbfs:/user/hive/warehouse/your_database.db/movies"
      ],
      "metastoreTables": ["your_database.movies"],
      "queryLanguage": "python",
      "queryText": "query_success = []\nnum_queries_run = 0\nimpersonate_probability = .20\nspark.sql(\"set immuta.impersonate.user=\")\n\ndef make_fail_query(query):\n  try:\n    spark.sql(\"set immuta.impersonate.user=taylor@databricks.com\")\n    spark.sql(query).toPandas()\n  except: \n    pass\n  \nfor index, query in enumerate(new_queries.values):\n  if(num_queries_run % 100 == 0):\n    print(f\"Queries Successfully Ran: {num_queries_run}/2000, out of total queries ran: {index+1}\")\n  to_impersonate = random.randrange(100)\n  if to_impersonate < impersonate_probability * 100:\n    make_fail_query(query)\n    spark.sql(\"set immuta.impersonate.user=\")\n    num_queries_run += 1\n  else:\n    try:\n      spark.sql(query).toPandas()\n      query_success.append((query, True))\n      num_queries_run += 1\n      if num_queries_run == 2000:\n        break\n    except Exception as e:\n      query_success.append((query, False))\n      \n    ",
      "immutaPluginVersion": "2022.3.0-spark-3.1.1"
    }
  },
  "receivedTimestamp": "2022-10-13T20:03:41.044Z"
}

Enriched Databricks audit logs

Beyond raw audit events (such as “John Doe queried Table X in Databricks"), the Databricks audit records include the policy information enforced during the query execution, even if a query was denied.

Queries will be denied if at least one of the conditions below is true:

  • User does not meet policy conditions.
  • User is not subscribed to the data source.
  • Data source is not in the user's current project.
  • Data source is in the user's current project, but the user is not subscribed to the data source.
  • Data source is not registered in Immuta.

User entitlements

The user's entitlements represent the state at the time of the query. This includes the following fields:

Property Description
project The user's current project.
attributes The user's attributes.
groups The user's groups.
impersonatedUsers The user that the current user is impersonating.

Policy information

The policySet includes the following fields:

Property Description Possible values
subscriptionPolicyType The type of subscription policy. MANUAL, ADVANCED, or ENTITLEMENTS
type Indicates whether the policy is a subscription or data policy. Query denied records will always be a subscription policy type. SUBSCRIPTION or DATA
ruleAppliedForUser True if the policy was applied for the user. If false, the user was an exception to the policy. true or false
rationale The policy rationale written by the policy creator. -
global True if the policy was a global policy. If false, the policy is local. true or false
mergedPolicies Shows the policy information for each of the merged global subscription policies, if available. -