Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Immuta reports allow data governors to use a natural language builder to instantly create reports that delineate user activity across Immuta. These reports can be based on various entity types, including users, groups, projects, data sources, purposes, policy types, or connection types.
User reports can be run for all users or for individual users who have been registered in Immuta. Non-registered users activity will not appear in reports.
Data sources subscribed to. This report lists data sources each user is subscribed to and includes user roles, subscription types, when users last subscribed, who approved the users' subscriptions to the data sources, when the subscriptions expire, what attributes the users possess and the groups the users belong to.
Status of all users. This report lists account information of all users in the system, including the users' full names, usernames, IAMs, HDFS Principals, and last login dates.
Groups the user belongs to. This report lists the names of the groups the user belongs to and the dates that groups were joined.
Data sources the user subscribes to. This report details the data source names, the user's roles, when the user last subscribed, who approved the subscriptions, when the subscriptions expire (if applicable) and the reasons for subscribing (if applicable).
Projects the user is currently a member of. This report lists the project names, whether the projects are public or private, the user's roles in the projects, the creator of the projects, when the projects were created, and when the user joined the projects.
All data sources ever accessed by the user. This report lists the data source names, when the data sources were first accessed by the user (or "read date"), and when the data sources were last accessed by the user. By default, this report only displays the last month of results. (You can download the full report by clicking Export to CSV.) The time period can be configured in the date field at the top of report's page.
Attributes the user has. This report lists the current attributes a user has and the values assigned to each attribute.
Purposes for accessing data. This report lists all purposes under which the user has accessed data sources. By default, this report only displays the last month of results. (The full report can be downloaded by clicking Export to CSV.) The time period can be configured in the date field at the top of report's page.
Group Reports can be run for all groups or for individual groups.
Data sources that members of this group are subscribed to. This report lists the data source names, the group's role, when the group last subscribed to the data sources, who approved the subscriptions, and the expiration dates (if applicable) and reasons (if applicable) for the subscriptions.
Users who belong to the group. This report lists the names of users and the dates the users joined the group.
Data sources that members of this group are subscribed to. This report lists the data source names, the group's role, when the group last subscribed to the data sources, who approved the subscriptions, and the expiration dates (if applicable) and reasons (if applicable) for the subscriptions.
Projects that users in this group are members of. This report includes the names of the projects, whether the projects are public or private, the group's role in the projects, the names of the project creators, when the projects were created, and when the group joined the projects.
Users and groups who are members of the project. This report includes usernames, email addresses, user roles in the project, when the users joined, and the subscription types. The subscription types may be "Individual User," indicating that the user joined the project directly, or it might be "Group," in which case the name of the group will be stated. Group subscriptions occur when an entire group is added to a project.
Data sources that are part of the project. This report lists the data source names, the reasons given when added to the project (if applicable), the users who added the data sources, and when the data sources were added to the project.
Purpose of the project. This report includes the purpose name, the user who added the purpose, and when the purpose was added to the project.
Data source reports can be run for all data sources or for individual data sources that are registered in Immuta. Activity to non-registered tables will not appear in the reports.
Users and groups subscribed to data sources. This report lists all users and groups subscribed to every data source and includes usernames, email addresses, subscription types, user roles, subscription dates, who approved the subscriptions, expiration dates, and user attributes.
Users and groups subscribed to the data source. This report lists the names of users, reasons for accessing the data sources (if applicable), user roles, email addresses, when users last subscribed, who approved the subscriptions, when the subscriptions expire (if applicable), and the subscription types. A subscription type may be "Individual User," indicating that the user subscribed to the data sources directly, or it might be "Group," in which case the name of the group will be stated. Group subscriptions occur when an entire group is added to a data source.
Projects that contain the data source. This report lists the project names, the users who added the data source to projects, when the data source was added to projects, the reasons for adding the data sources (if applicable), whether the projects are public or private, who created the projects, and when the projects were created.
Purposes of all projects that contain the data source. This report states the purpose names, the users who assigned the purposes to the projects, the dates the purposes were assigned, the names of the projects, the reasons the purposes were added (if applicable), whether the projects are public or private, who created the projects, and when the projects were created.
All users who have accessed the data source. This report lists usernames, email addresses, each user's latest query, and the date of the last access. By default, this report only displays the last month of results. (The full report can be downloaded by clicking Export to CSV.) The time period can be configured in the date field at the top of report's page.
All purposes for data source access. This report lists users who have accessed the data source and the purposes under which they were working. By default, this report only displays the last month of results. (The full report can be downloaded by clicking Export to CSV.) The time period can be configured in the date field at the top of report's page.
All users who have subscribed to the data source. This report lists users or groups, email addresses, when users subscribed, reasons for subscriptions (if applicable), who approved the subscriptions, when the subscriptions expire, and the dates and reasons users unsubscribed (if applicable). By default, this report only displays the last month of results. (The full report can be downloaded by clicking Export to CSV.)
Users who are members of projects with this purpose. This report lists usernames, email addresses, their roles in the project, the names of the projects, whether the projects are public or private, the creators of the projects, when the projects were created, when users joined, and their subscription types (individual or group).
Data sources that are part of projects with this purpose. This report lists the names of the data sources, who created the data sources, the project names, whether the projects are public or private, the creators of the projects, whether the projects have other purposes, and when the projects were created. Note that whether projects have other purposes will be assigned as "True" or "False." Immuta Governors can configure whether projects can have more than one purpose in the Settings tab in the Governance console.
Whether any other purposes have been combined with this purpose. If multiple purposes are allowed in one project, this report lists the names of the other purposes, the project name, the users who added each purpose, the project creator, whether the project is public or private, and the date the project was created.
Projects that have this purpose. This report lists the names of the projects, the users who added the purpose, whether the projects are public or private, creators of the projects, whether the projects have other purposes, and when the projects were created.
Data sources that have been accessed for this purpose. This report lists the names of the data sources, the users who accessed data sources for this purpose, the project names, and whether projects have other purposes. By default, this report only displays the last month of results, but the time period can be configured in the date field at the top of this report's page.
Data sources with this policy type. Immuta supports a range of policy types, such as masking, WHERE clauses, purpose restrictions, and more. This report lists every data source with this policy type, including when they were created, who created the data sources, who created the policy, and when the policy was created.
Global policy reports can be run for all global policies or for individual global policies.
Global policies that have been disabled. This report details the names of the policies, the policies themselves, the policy types, the data sources from which the policies were disabled, who disabled the policies, when they were disabled, the justifications the users provided for disabling the policies, who created the policies, when the policies were created, and how the policies were associated with the data sources.
Global policies that cannot currently be applied. This report details the names of the policies, the policies themselves, the policy types, the names of the data sources the policies cannot be applied to, when the data sources were created, when the policies were created, the reasons the policies cannot be applied, who created the policies, and how the policies are associated with the data sources.
Data sources impacted by the policy. This report lists the data sources, when the data sources were created, and whether or not the policy is fully applied to the data sources.
Data sources impacted by the policy that have not been certified: This report lists the data sources that have not been certified, when the global policy was applied, and the data owner.
Data sources impacted by the policy that have been certified: This report lists the data sources that have been certified, the user that certified it, when the global policy was applied, and when it was certified.
Data sources with this connection type. This report lists the data sources, each data source's creator, the creation date, and the tables or queries used by the connection selected.
Data sources this tag has been assigned to. This report generates a list of data sources associated with that tag and includes the columns tagged, the value types of the data tagged, who tagged the data sources, when the data sources were tagged, and when the data sources were created.
Purposes associated with data sources containing this tag. This report generates a list of purposes under which users have accessed data sources containing this tag. By default, this report only displays the last month of results. (The full report can be downloaded by clicking Export to CSV.) The time period can be configured in the date field at the top of report's page.
Users who have accessed data sources containing this tag. This report lists users who have accessed data sources with this tag, their email addresses, when they queried the data, and when the data sources were created.
Projects that contain data with this tag. This report details the projects associated with this tag, whether or not the projects are public or private, when the projects were created, the data sources in the projects, and when the data sources were created.
Deprecation notice
Support for the audit endpoint and UI has been deprecated. Instead, pull audit logs from Kubernetes and push them to your SIEM.
With the Event Listener enabled, users can view audit records for queries made in Starburst against Immuta data sources on the Audit page. Immuta audits the activity of Immuta users on Immuta data sources.
Starburst (Trino) users registered as Immuta users: Note that the users' Starburst (Trino) usernames must be mapped to Immuta. Without this, Immuta will not know the users are Immuta users and will not collect audit events for their data access activity.
Best Practices: Store Audit Records
By default Immuta audit records expire after 60 days, so store audit records outside of Immuta in order to retain the audits long term.
Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below. These audit records are stored with the recordType: prestoQuery.
| Property | Description | Example |
|---|---|---|
Deprecation notice
Support for the audit endpoint and UI has been deprecated. Instead, and push them to your SIEM.
In addition to the executed Spark plan, the tables, and the tables' underlying paths for every audited Spark job, Immuta captures the code or query that triggers the Spark plan. Immuta audits the activity of Immuta users on Immuta data sources.
: Note that the users' . Without this, Immuta will not know the users are Immuta users and will not collect audit events for their data access activity.
Best Practices: Store Audit Records
By default Immuta audit records expire after 60 days, so store audit records outside of Immuta in order to retain the audits long term.
Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below. These audit records are stored with the recordType: spark.
| Property | Description | Example |
|---|
Below is an example of the queryText, which contains the full notebook cell (since the query was the result of a notebook). If the query had been from a JDBC connection, the queryText would contain the full SQL query.
This notebook cell had multiple audit records associated with it, but the example audit record in the tab to the right corresponds to the filteredDf.write.saveAsTable('{}.audit_cell'.format(testDb)) line.
This example audit record contains two fields under extra:
queryText: The queryText will contain either the full notebook cell (when the query is the result of a notebook) or the full SQL query (when it is a query from a JDBC connection).
queryLanguage: The queryLanguage corresponds to the programming language used: SQL, Python, Scala, or R. Audited JDBC queries will indicate that it came from JDBC here.
Beyond raw audit events (such as “John Doe queried Table X in Databricks"), the Databricks audit records include the policy information enforced during the query execution, even if a query was denied.
Queries will be denied if at least one of the conditions below is true:
User does not meet policy conditions.
User is not subscribed to the data source.
Data source is not in the user's current project.
Data source is in the user's current project, but the user is not subscribed to the data source.
Data source is not registered in Immuta.
The user's entitlements represent the state at the time of the query. This includes the following fields:
project: The user's current project.
attributes: The user's attributes.
groups: The user's groups.
impersonatedUsers: The user that the current user is impersonating.
The policySet includes the following fields:
subscriptionPolicyType: The type of Subscription policy (such as MANUAL, ADVANCED, or ENTITLEMENTS).
type: Indicates whether the policy is a SUBSCRIPTION or DATA policy. Query denied records will always be a Subscription policy type.
ruleAppliedForUser: Indicates whether or not the policy was applied for the user. If false, the user was an exception to the policy.
rationale: The policy rationale written by the policy creator.
global: Indicates whether or not the policy was a Global policy. When false, the policy is Local.
Query Scenario
Before examining the audit log excerpt below, review the user's entitlements and the policy on the patient_transactions data source.
User Entitlements
groups: none
attributes: SpecialAccess.Addresses, OfficeLocation.Maryland
current project: Medical Claims
Data Source Policy
Subscription Policy: Allow individual users you select to access the data source.
The Databricks user described above attempts to query a patient_transactions table, which has not been added to a project. The user's query is denied, and the audit logs reveal the policy details. Focus specifically on the entitlements, policySet, actionStatus, and actionStatusReason sections below:
Although the user was subscribed to the data source, the project field shows that this user is currently working under the Medical Claims project, and the actionStatusReason indicates that the data source has not been added to that project. Consequently, the actionStatus field shows the query was UNAUTHORIZED.
The user's entitlements includes the following fields:
project: The user's current project.
attributes: The user's attributes.
groups: The user's groups.
impersonatedUsers (when relevant): The username the current user is impersonating.
For a successful query, policySet includes all policies applied to the data source:
subscriptionPolicyType: The type of Subscription policy (such as MANUAL, ADVANCED, or ENTITLEMENTS).
type: Indicates whether the policy is a SUBSCRIPTION or DATA policy.
ruleAppliedForUser: Indicates whether or not a policy was applied for the user. If false, the user was an exception to the policy.
rationale: Provides the rationale given for the policy by the policy creator.
global: Indicates whether or not the policy was a Global policy. When false, the policy is Local.
mergedPolicies: Shows the policy information for each of the merged Global Subscription policies.
Query Scenario
Before examining the audit log excerpt below, review the user's entitlements and the policy on the patients data source.
User Entitlements
groups: none
attributes: SpecialAccess.Addresses, OfficeLocation.Maryland
current project: None
Data Source Policies
Subscription Policy: Allow individual users you select to access the data source.
Data Policy: Mask by making null the value in the column(s) lastname for everyone.
Data Policy: Mask by hashing the value in the column(s) address for everyone except users who possess the attribute SpecialAccess.Addresses.
The Databricks user described above attempts to query a patients table. The user's query is successful, and the audit logs reveal the policy details. Focus specifically on the entitlements, policySet, and actionStatus sections below:
This user is subscribed to the data source and the actionStatus shows their query was a SUCCESS.
The first masking policy in policySet masks the field lastname to NULL for everyone, as no exceptions are listed. Furthermore, the ruleAppliedForUser field is true, indicating that the querying user sees NULL values for this column.
However, the ruleAppliedForUser for the second masking policy in the policySet is false, indicating that the querying user can see the values in the address column in the clear because one of their entitlements is listed under exceptions to the policy: SpecialAccess.Addresses.
Query Scenario
Before examining the audit log excerpt below, review the user's entitlements and the policy on the maryland_employees data source.
User Entitlements
groups: none
attributes: SpecialAccess.Addresses, OfficeLocation.Maryland
current project: None
Data Source Policy
Merged Subscription Policy: Allow users to subscribe if they have the attribute OfficeLocation.Maryland or are a member of the group Human Resources.
The Databricks user described above attempts to query the maryland_employees table. The user's query is successful, and the audit logs reveal the policy details. Focus specifically on the entitlements, mergedPolicies, and actionStatus sections below:
This user is subscribed to the data source and the actionStatus shows their query was a SUCCESS.
The mergedPolicies section illustrates both Global policies that were merged and applied to the data source. Although the querying user is not a member of the group Human Resources, the ruleAppliedForUser field is still true and they are subscribed to the data source because they met one requirement of the subscription policy: they possess the attribute OfficeLocation.Maryland.
Audience: Immuta users with the
AUDITpermission and Data OwnersContent Summary: This page instructs how to view and filter audit logs.
Deprecation notice
Support for the audit page has been deprecated. Instead, and push them to your SIEM.
To view all of the audit logs, click on the Audit icon displayed in the left side panel (only users with permission to access the page will see the icon).
To filter the audit logs by purpose,
Start typing the name of your desired purpose and select one of the auto-completed results beneath the text box.
Note: If a list of purposes does not display, then no purposes matching the text exist.
Click the Filter button. Relevant records will display in the right panel.
To remove the filter and display all results, click the Clear button.
To filter by blobId, type in the exact blobId in the filter box in the left panel.
Click the Filter button. Relevant records will display in the right panel.
To remove the filter and display all results, click the Clear button.
To filter by remote query id, type in the remote query id in the filter box in the left panel.
Click the Filter button. Relevant records will display in the right panel.
To remove the filter and display all results, click the Clear button.
To filter by user, record type, project, data source or the entry timestamp, just click one or more of the corresponding facets listed on the left side of the center pane.
To sort entries in ascending (oldest first) or descending (latest first) order, click the corresponding tabs along the top of the center pane.
To view more than 50 entries per page, click the desired number in the upper right of the center pane.
To get back to all of the log entries, simply click on the Audit icon displayed in the left side panel.
Public Preview
This feature is currently in public preview and available to all accounts.
Native query audit for Unity Catalog captures user data access activity within Unity Catalog as Immuta audit logs. Immuta audits the activity of all Unity Catalog users and tables. Multiple access options are supported for audit:
Cluster queries with the following supported languages: SQL, Scala, Python, and R.
SQL Warehouse queries
Deprecation notice
Support for the audit endpoint and UI has been deprecated. Instead, and push them to your SIEM.
Databricks Unity Catalog integration with native query audit enabled
A Databricks deployment with capabilities
Best practices: Store audit logs
By default most Immuta audit logs expire after 60 days, so store audit logs outside of Immuta in order to retain the audits long term.
The following audit record types do not expire after 60 days: blobFetch, dataSourceSubscription, globalPolicyApproved, globalPolicyApprovalRescinded, globalPolicyChangeRequested, globalPolicyConflictResolved, globalPolicyCreate, globalPolicyDelete, globalPolicyDisabled, globalPolicyUpdate, policyExemption, policyHandlerCreate, policyHandlerUpdate, prestoQuery, spark, and sqlQuery.
To manually request native query audit ingestion, click Load Audit Events on the Immuta audit page.
Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below. These audit logs are stored with the recordType: nativeQuery.
Enrichment of audit logs with Immuta entitlements information is not supported. While you will see these entitlements in the Databricks Spark audit logs, the following will not be in the native query audit for Unity Catalog:
Immuta policies information
User attributes
Groups
Immuta determines unauthorized events based on error messages within Unity Catalog records. When the error messages contain expected language, unauthorized events will be available for native query audit for Unity Catalog. In other cases, it is not possible to determine the cause of an error.
Audit for cluster queries do not support UNAUTHORIZED status. If a cluster query is unauthorized, it will show FAILURE.
Data source information will be provided when available:
For some queries, Databricks Unity Catalog does not report the target data source for the data access operation. In these cases the activity is audited, yet the audit record in Immuta will not include the target data source information.
Data source information is not available for unauthorized queries and events.
Column information from the query is not currently supported.
The cluster for the Unity Catalog integration must always be running for Immuta to audit activity.
Data governors can create reports of data usage using natural language builder, and those with the AUDIT permission and Data Owners can pull audit logs directly in the Immuta UI.
This section includes reference and how-to guides for viewing audit logs and reports of data access. Some of these guides are provided below. See the left navigation for a complete list of resources.
Query audit logs:
Access include details about the user (accessControls.entitlements) and the Subscription policy (accessControls.policySet) that blocked access to the table.
mergedPolicies: Shows the policy information for each of the .
Immuta collects audit records at the frequency , which is between 1 and 24 hours. The frequency is a global setting based on integration type, so organizations with multiple Unity Catalog integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records; however, there could be performance and cost impacts from the frequent jobs.
| Property | Description | Example |
|---|
Immuta audit records include unregistered data sources and users; however, activity from them will not appear in any .
ID
string
b0000000-1234-abcd-11111111111111
DateTime
integer or string The timestamp for when the record was created. This may be an ISO-8601 timestamp string or an epoch timestamp.
2504188066580 or 2017-08-31T14:01:15.607Z
Month
integer
1455
ProfileID
integer The profile ID of the user who made the query.
1
UserID
string The user ID of the user who made the query.
jane.doe@immuta.com
DataSourceID
integer The ID of the data source that was queried.
12
DataSourceName
string The name of the data source that was queried.
Public Customer Data
ProjectID
integer The ID of the project the data source is in.
18
ProjectName
string The name of the project the data source is in.
Project 1
RecordType
string The type of record captured.
Trino query audit records will always be prestoQuery.
Success
boolean If true, the query was successful.
true or false
Component
string The Immuta component that generated the record.
nativeSql
AccessType
string Indicates whether access was granted to an individual blob or if this was a query potentially encompassing many blobs.
query
Query
string The query that was run in the integration.
select * from immuta.public. \"case\" limit 50
Extra
array Information on the query, including viewSql, direct, and maskedColumns.
See example audit record below.
DataSourceSchemaName
string
The name of the schema that the data source that was queried came from.
DataSourceTableName
string The name of the table of the data source that was queried.
case
sqlUser
string The Starburst (Trino) username of the user who made the query.
kris
id | The unique ID of the audit record. |
|
dateTime | The timestamp for when the record was created. This may be an ISO-8601 timestamp string or an epoch timestamp. |
|
profileId | When available, the profile ID of the user who made the query. If the user is not registered with Immuta, the |
|
userId | When available, the Immuta user ID of the user who made the query. If the user is not registered with Immuta, |
|
dataSourceId | When available, the Immuta ID of the data source that was queried. If the data source is not registered with Immuta, |
|
dataSourceName | When available, the Immuta name of the data source that was queried. If the data source is not registered with Immuta, |
|
recordType | The type of record captured based on query type. Databricks Unity Catalog query audit logs will always be |
|
success | If |
|
component | The Immuta component that generated the record. |
|
accessType | Unity Catalog audit logs will always be |
|
query | The command text of the query. |
|
queryId | The unique ID of the query. If the query joins multiple tables, each table will appear as a separate log with a matching query ID. |
|
extra.handler | The integration technology. Databricks Unity Catalog query audit logs will always be |
|
extra.startTime | The date and time the query started in UTC. |
|
extra.endTime | The date and time the query ended in UTC. |
|
extra.duration | The time the query took in milliseconds. |
|
extra.nativeObject | When available, the name of the Unity Catalog object that was queried. |
|
extra.nativeObjectFullName | When available, the fully qualified Unity Catalog object that was queried. |
|
extra.host | The host that the integration is connected to. |
|
extra.actionStatus | Indicates whether or not the user was granted access to the data. Options are |
|
extra.actionStatusReason | When available and a user's query is denied, this property contains the Unity Catalog system's reason for failure. When a query is successful, this value is |
|
extra.errorCode | When available, the error code for the unauthorized query. For unauthorized queries options are |
|
extra.actionName | The environment data was accessed in. If |
|
extra.service | The environment data was accessed in. Options include |
|
extra.accountId | The Unity Catalog account ID. |
|
extra.notebookId | For cluster queries, the ID of the Unity Catalog notebook the query was made in, otherwise |
|
extra.clusterId | For cluster queries, the ID of the Unity Catalog cluster the query was made in, otherwise |
|
extra.requestedId | The Unity Catalog request ID. |
|
extra.sessionId | For warehouse queries, the ID of the Unity Catalog session the query was made in, otherwise |
|
extra.warehouseId | For warehouse queries, the ID of the Unity Catalog warehouse the query was made in, otherwise |
|
extra.queryLanguage | For cluster queries, the programming language used: SQL, Python, Scala, or R. |
|
extra.actorIp | When available, the IP address of the Spark cluster the request is coming from. |
|
extra.actorEmail | The Unity Catalog email of the user making the query. |
|
extra.userAgent | When available, the Unity Catalog client information of the user who made the query. |
|
extra.workspaceId | The ID of the Unity Catalog workspace the query was made in. |
|
All activity in Immuta is audited. This process provides rich audit logs that detail who subscribes to each data source , why they subscribe, when they access data, what SQL queries and blob fetches they run, and which files they access. Audit logs can be used for a number of intentions, including insider threat surveillance and data access monitoring for billing purposes. Audit logs can also be shipped to your enterprise auditing capability.
Best Practices: Store Audit Records
Store audit records outside of Immuta in order to retain the audit logs long-term.
By default, Immuta stores most audit records for 60 days. The following audit record types do not expire after 60 days:
blobFetch
dataSourceSubscription
globalPolicyApproved
globalPolicyApprovalRescinded
globalPolicyChangeRequested
globalPolicyConflictResolved
globalPolicyCreate
globalPolicyDelete
globalPolicyDisabled
globalPolicyUpdate
nativeQuery
policyExemption
policyHandlerCreate
policyHandlerUpdate
prestoQuery
spark
sqlQuery
Immuta writes all logs to stdout in Kubernetes. Users will get all logs here, but should use the Common Message Types to parse for specifics.
When running Immuta on Docker-based installations, all logs from the Immuta Docker containers will be sent to the Docker log driver.
The Immuta log file will contain messages that are one-line JSON, as described in Log Formats.
Logs messages from the Immuta platform typically will be one line JSON and contain all of the common JSON properties. Depending on the message type, more JSON properties may be present. See Common Message Types for more details.
level: This is a string representation of the log level. Acceptable values are "debug", "info", "warning", "error", and "audit".
timestamp: This is a timestamp for when the message occurred. The timestamp format is YYYY-MM-DDTHH:mm:ss.sssZ (ISO 8601).
message: This is the log message, which may be used to determine common message types.
Generally, any query that causes multiple background queries will have an audit record created for each. The audited plan should be different, however, for each record. For example, subqueries will generate two audit records: one for the subquery and one for the outer query.
Immuta does not audit any notebook cells that don’t query data.
Each audit message from the Immuta platform will be a one-line JSON object containing the common JSON properties and the Audit JSON properties. Depending on the recordType, an audit message may contain additional data.
In order to discover audit messages using your analysis tool, you may search the object using the criteria below:
level: "audit"
message: "Audit - *
dateTime:
description: The timestamp for when the record was created. This may be an ISO-8601 timestamp string or a ms since epoch timestamp.
type: integer or string
example: 1504188066580 or "2017-08-31T14:01:15.607Z"
component:
description: The Immuta component that generated this record. Possible values are "console", "featureStore", "dataSource", "bim", "audit", "script", "policy", "project", "plugin", and "governance".
type: string
instanceId:
description: The instance ID of the component generating this record.
type: string
profileId:
description: The profile ID of the user generating the action.
type: integer
userId:
description: The user ID of the user generating the action.
type: string, null
sqlUser:
description: The database account generating the action.
type: string
dataAccess:
description: Describes access to an individual blob or a query that may grant access to multiple blobs.
type: object
sessionId:
description: If this record is generated in response to a user action and if that user's session ID is known, record that session ID here.
type : string
dataSource:
description: If the record creation is associated with a data source, the data source name should be recorded here.
type: string, null
dataSourceId:
description: If the record creation is associated with a data source, the data source ID should be recorded here.
type: integer, null
projectName:
description: If the record creation is associated with a project, the project name should be recorded here.
type: string, null
projectId:
description: If the record creation is associated with a project, the project ID should be recorded here.
type: integer, null
purposeIds:
description: If the action being taken by the user involves data and is happening for a specific person, the purpose IDs should be recorded here.
type: array[integer], null
success:
description: Denotes whether the action being audited was successful.
type: boolean
failureReason:
description: Describes the reason that this audit event failed. Possible values are "systemError", "insufficientAuthorizations", "insufficientPermissions", and "userError".
type: string
failureDetails:
description: If the audit event failed, details can be provided in this free text field to examine later.
type: string or object
recordType:
description: The type of audit event being captured. This also corresponds to the additional information in the record field. Possible values are "auditQuery", "blobVisibility", "blobFetch", "blobIndex", "blobDelete", "blobCatalogFetch", "blobCatalogFetchDate", "blobUpdateFeatures", "blobUpdateTags", "createQuery", "modifyQuery", "consoleDataSourceView", "sqlAccess", "sqlCreateUser", "sqlDeleteUser", "sqlResetPassword", "featureList", "sqlQuery", "dataSourceCreate", "dataSourceDelete", "dataSourceSave", "dataSourceGet", "dataSourceListMine", "dataSourceGetTags", "dataSourceSubscription", "dataSourceGetUsers", "dataSourceTest", "dictionaryCreate", "dictionaryDelete", "dictionaryUpdate", "projectCreate", "projectUpdate", "projectDelete", "addToProject", "removeFromProject", "acknowledgePurposes", "comment", "userVisibilities", "accessUser", "accessGroup", "searchAuthorizations", "apiKey", "scriptCopy", "scriptSave", "scriptGet", "scriptGetForks", "scriptGetVersions", "scriptVersionGet", "scriptUpdate", "scriptDelete", "scriptVersionDelete","scriptVersionUpdate", "scriptDataSourceGet", "scriptDataSourceUpdate", "scriptSaveContent", "scriptGetContent", "userKernelCreate", "userKernelUpdate", "userKernelDelete", "querySampleData", "authenticate", "checkPendingRequest", "policyExemption", "governanceUpdate", "purposeCreate", "purposeUpdate", and "purposeDelete".
type: string
record:
description: The component-defined type of record. For example, it could be something like 'data source access request'.
type: object
extra:
description: A JSON object representing the additional information to be logged/audited.
type: object
API Key Object
keyIamId:
description: The IAM ID for the user who owns the API key accessed.
type: string
keyId:
description: The API key ID.
type: integer
keyUserId:
description: The user who owns the API key accessed.
type: string
keyAction:
description: Denotes how the specified user was accessed. Possible values are "get" and "delete".
type: string
Data Access Object
accessType:
description: Indicates whether access was granted to an individual blob or if this was a query potentially encompassing many blobs. Possible values are "blob"and "query".
type: string
blobId:
description: If accessType==blob, this is the blobId.
visibility:
description: If the accessType==blob, this is the visibility. If the accessType==query, this is an array of the visibilities the user had when querying.
type: object, array
query:
description: If the accessType==blob, this is not present. If the accessType==query, this is the query.
type: string
dataSourceTableName:
description: The data source table name queried in the audit record.
type: string
Blob Fetch Object
blobSize:
description: The size (in bytes) of the blob being fetched.
type: integer
Blob Visibility Object
newVisibility:
description: This is the new visibility for the blob.
type: object
SQL Access Object
action:
description: Denotes whether access was granted or revoked. Possible values are "revoked"and "granted".
type: string
sqlUser:
description: The username of the user whose access is being manipulated.
type: string
SQL Create User Object
sqlUser:
description: The username of the user whose access is being manipulated.
type: string
Data Source Subscription Object
dataSourceSubscriptionState:
description: If the record type is dataSourceSubscription, this field must be present and indicate the state to which the record was changed (dataSources.length must be 1 in this case). Possible values are "denied", "subscribed", "expert", "owner", "ingest", and "unsubscribed".
type: string
accessedId:
description: The user identifier of the user who is being acted upon.
type: integer
accessedIdType:
description: Type of user being acted upon. Possible values are "user" and "group".
type: string
Comment Object
commentAction:
description: Describes the commenting action. Possible values are "reply", "create", "delete", and "resolve".
type: string
commentId:
description: The comment ID of the comment created / deleted.
type: integer
commentParentId:
description: The parent comment of the comment, if any.
type: integer
Data Source Delete Object
hardDelete:
description: Denotes whether this was a hard delete.
type: boolean
Access User Object
accessedUserId:
description: The user being accessed.
type: string
accessedIamId:
description: The IAM ID for the user being accessed.
type: string
accessType:
description: Denotes how the specified user was accessed. Possible values are "update", "get", "search", "create", "delete", "complete", "newToken", and "clone".
type: string
Access Group Object
accessedUserId:
description: The user being accessed.
type: string
accessedGroupId:
description: The group being accessed.
type: integer
groupAccessType:
description: Denotes how the specified group was accessed. Possible values are "update", "get", "search", "create", "delete", "addUser", and "removeUser".
type: string
groupIamId:
description: The IAM ID for the group being accessed.
type: string
Each error message response from the Immuta platform will be a one-line JSON object containing the common JSON properties and the error JSON properties below. Error message responses represent error responses that have been sent to clients.
NOTE: It is possible for a similar message referencing the same error to appear in the logs, as this log message represents only that an error response was sent to a client.
In order to discover error response message using your analysis tool, search the object using the criteria below:
message: Error Response Sent
id:
description: A unique ID assigned to each request.
type: string
method:
The HTTP method used for the request.
type: string
path:
description: This is the HTTP path used for the request.
type: string
query:
description: An object containing the parsed query string used for the query.
type: object
responseSentTime:
description: This is the duration from the time a request was received until the time that the server responded.
type: integer
responseTime:
description: This is the duration from the time a request was received until server post-processing.
type: integer
stack:
description: This is a string representation of a stack trace if one exists.
type: string
statusCode:
description: This is the HTTP status code that was sent to the client.
type: integer
Each request message response from the Immuta platform will be a one-line JSON object containing the common JSON properties and the error JSON properties below. Request message responses represent responses that have been sent to clients.
In order to discover request response messages using your analysis tool, search the object using the criteria below:
message: Response Sent
id:
description: A unique ID assigned to each request.
type: string
method:
The HTTP method used for the request.
type: string
path:
description: This is the HTTP path used for the request.
type: string
query:
description: An object containing the parsed query string used for the query.
type: object
responseSentTime:
description: This is the duration from the time a request was received until the time that the server responded.
type: integer
responseTime:
description: This is the duration from the time a request was received until server post-processing.
type: integer
statusCode:
description: This is the HTTP status code that was sent to the client.
type: integer
Audience: Data Governors
Content Summary: This page outlines the process of building Immuta Reports. For descriptions of these reports, navigate to the Immuta Reports page.
Use Case
The compliance team is required to monitor data use and access for all projects. To do so, they can build Immuta reports.
Immuta Reports allow Data Governors to use a natural language builder to instantly create reports that detail user activity across Immuta.
Click select entity and choose the option you would like the report based on from the dropdown menu. Your options include User, Group, Project, Data Source, Purpose, Policy Type, Connection, or Tag.
After making your selection, type your entity name in the enter name field.
Select the name from the dropdown menu that appears. Once the entity name has been selected, a number of reports will populate the center window.
Click a tile with the description of the report to run that report.
Once you've run the report, you can click the Export to CSV button in the top right of the page to download the report.
Note: If you would like to switch reports from this page, you can make changes by clicking the dropdown menu and then Refresh to run a new report. Otherwise, click Back to Report Builder in the top right of the page to return to the full Report Builder.
ID |
|
|
DateTime |
|
|
Month |
|
|
ProfileID |
|
|
UserID |
|
|
DataSourceID |
|
|
DataSourceName |
|
|
ProjectID |
|
|
ProjectName |
|
|
PurposeID |
|
|
RecordType |
| Databricks query audit records will always be |
Success |
|
|
Component |
|
|
AccessType |
|
|
Query |
| See the example below |
actionStatus |
|
actionStatusReason |
|
policySet |
|
entitlements |
|
Deprecation notice
Support for the audit endpoint and UI has been deprecated. Instead, pull audit logs from Kubernetes and push them to your SIEM.
Snowflake query audit logs is a feature that audits queries that users run natively in Snowflake and turns them into Immuta audit logs. Immuta uses the Snowflake QUERY_HISTORY and ACCESS_HISTORY tables and translates them into the audit logs that can be viewed and downloaded within the Immuta UI or using the Immuta API. Immuta audits the activity of Immuta users on Immuta data sources.
Snowflake Enterprise Edition or higher
Snowflake users registered as Immuta users: Note that the users' Snowflake usernames must be mapped to Immuta. Without this, Immuta will not know the users are Immuta users and will not collect audit events for their data access activity.
Best Practices: Store Audit Records
By default Immuta audit records expire after 60 days, so store audit records outside of Immuta in order to retain the audits long term.
Immuta collects audit records at the frequency configured when enabling the integration, which is between 1 and 24 hours. The frequency is a global setting based on integration type, so organizations with multiple Snowflake integrations will have the same audit frequency for all of them. The more frequent the audit records are ingested, the more current the audit records; however, there could be performance and cost impacts from the frequent jobs.
To manually request native query audit ingestion, click Load Audit Events on the Immuta audit page.
Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below. These audit records are stored with the recordType: nativeQuery.
Audience: System Administrators, Data Owners, and users with the AUDIT permission
Content Summary: This page outlines how to download the audit logs for your instance of Immuta through the API.
Deprecation notice
Support for the audit endpoint has been deprecated. Instead, pull audit logs from Kubernetes and push them to your SIEM.
Generate your API key on the API Keys tab on your profile page and save the API key somewhere secure.
You will pass this API key in the authorization header when you make a request, as illustrated in the example below:
Endpoint
| Method | Path | Purpose |
|---|---|---|
Query Parameters
| Attribute | Description | Required |
|---|---|---|
Response Parameters
The response returns the audit records. See below for an example.
The request below saves 50 audit logs for https://your-immuta-url.immuta.com in the file audit-logs-file.son, with the audit records sorted by time in descending order.
UNAUTHORIZED or SUCCESS. See the .
See the .
See the .
See the .
| Property | Description | Example |
|---|---|---|
DateTime
integer or string The timestamp for when the record was created. This may be an ISO-8601 timestamp string or an epoch timestamp.
2504188066580 or 2017-08-31T14:01:15.607Z
ProfileID
integer The profile ID of the user who made the query.
1
UserID
string The user ID of the user who made the query.
jane.doe@immuta.com
DataSourceID
integer The ID of the data source that was queried.
12
DataSourceName
string The name of the data source that was queried.
Public Customer Data
RecordType
string The type of record captured.
Native query audit records will always be nativeQuery.
Success
boolean If true, the query was successful.
true or false
Component
string The Immuta component that generated the record.
nativeSql
AccessType
string Indicates whether access was granted to an individual blob or if this was a query potentially encompassing many blobs.
query
Query
string The query that was run in the integration.
select * from IMMUTA.PUBLIC.CUSTOMER;
Handler
string The integration type.
Snowflake
StartTime
timestamp The date and time the query started in UTC.
2021-09-20 17:20:00.39100000 +0000
EndTime
timestamp The date and time the query ended in UTC.
2021-09-20 17:20:02.42100000 +0000
Duration
integer The time the query took in milliseconds.
2030
NativeObject
string The fully qualified object that was queried.
IMMUTA.PUBLIC.CUSTOMER
NativeObjectType
string The type of the object that was queried.
view or table
Host
string The host that the integration is connected to.
your-hostname.snowflake.computing.com
Database
string The database that the native object resides in.
IMMUTA
SQLUser
string The integration username of the user who made the query.
jane.doe@immuta.com
GET
/audit
Gets the audit records for the instance of Immuta.
dataSourceId
array[integer] Returns records for the data source with the specified ID.
No
projectId
array[integer] Returns records for the project with the specified ID.
No
profileId
array[integer] Returns records for the user with the specified profile ID.
No
recordType
string Returns the type of audit record specified. See recordTypes in the response example below for details.
No
outcome
string Returns records based on the specified outcome: success, userError, systemError, insufficientAuthorizations, failure, or insufficientPermissions.
No
minDate
date The date of the oldest audit record to include.
No
maxDate
date The date of the most recent audit record to include.
No
blobId
string Returns audit records for the specified blobId.
No
purpose
integer Returns audit records for the purpose with the specified ID.
No
offset
integer Used in combination with size to fetch pages. Example: "size=10&offset=10" would return the second page or "size=10&offset=20" would return the third page.
No
size
integer The number of results to return per page. Default is 50.
No
sortField
string Sorts the records by the specified field. Default is dateTime.
No
sortOrder
string Sorts records in ascending (asc) or descending (desc) order. Default is desc.
No