Detect Monitors Reference Guide

Public preview

This feature is available to all accounts. Contact your Immuta representative to enable this feature.

Immuta Detect monitors automates manual aggregation and calculation of user activity metrics based on query events. It can then notify you when the metrics exceed your intended operating thresholds. When a user's query activity metric exceeds a configured threshold within a timeframe, Immuta creates an observation that links the user's query audit events that contributed to the breached activity metric. You can specify monitor conditions that work with the information in the query event context to compute each user's activity metrics. Monitors can be configured with multiple thresholds to assign an observation severity which allows you to control the severity of observations Immuta Detect sends a webhook notification for.

When to use Detect monitors

Create monitors to gain awareness of when your users' behavior changes, maintain data availability through data platform policy changes, ensure access patterns remain consistent for your data controls to remain effective, and be notified about anomalies. See the following example use cases:

• Change management: Configure a monitor to notify you when unauthorized or failed queries to production data sources exceed a baseline threshold, indicating potential data outage from a recent subscription policy change.

• Know when a new user has gained access to sensitive data: Configure a monitor to notify the first time a user has accessed any data source that's classified sensitive or highly-sensitive and tagged specifically within a controlled set of data sources.

• Find high-volume data activity: Use Detect to review typical query count over a supported timeframe, and configure a monitor for when a user has issued more than the typical number of queries.

• Surface forbidden data combinations: For data sources or columns that must never be joined in a query, configure a monitor with query tag conditions to notify you when forbidden queries may have occurred.

• Track data platform cost and experience deviations: Configure a monitor to watch for long-running queries that dominate resource or need optimization.

Requirements

Immuta permission AUDIT

Supported integrations

Detect monitors can be created for data sources from the following integrations:

• Snowflake integration with native query audit enabled

• Databricks Spark integration

• Databricks Unity Catalog with native query audit enabled

• Starburst (Trino) integration

Query event context

The query event context includes a set of metadata that is used by Detect monitors to compute each user's query activity metrics. Query audit events in Immuta can include metadata such as the sensitivity classification, query execution outcome, queried column tag, queried table tag, data source name, and schema. The sensitivity classification is set if Immuta Discover was configured at the time of query audit processing. For example, the sensitivity measures emitted by Discover's Data Security Framework rules that inspect adjacent columns work in Detect to help you assess sensitivity of each query.

You can review the full context of a query audit event in the Query detail page in Detect.

Monitors

Monitors are organization-defined thresholds for access to your organization's data by users. Within monitors, an organization can define what user actions should trigger an observation. If a user activity metric exceeds the configured severity thresholds in the specified timeframe, Immuta creates an observation.

Monitors options

Monitors can be scoped to individual data sources, all the data sources within a schema, or all registered data sources in Immuta.

Additionally, monitors can selectively assess user activity based on matching conditions in the query event context. You can create monitors for

• when a user accesses more than a set number of rows of data with specific tags or sensitivity.

• when a user has made more than a set number of queries of data with specific tags or sensitivity.

• when a user has made more than a set number of queries, each over a specific query duration.

• when a user has more than a set number of unauthorized or failed queries against any data source in Immuta.

All specified conditions must be satisfied for the query event to contribute to the user activity metric. For example, you can specify both highly sensitivity and a specific tag for the event to count towards the selected user activity metric.

Query duration details

The query duration option for monitors provides notifications for long-running queries:

• When monitoring the user query count metric, each query must meet the query duration condition to be included in a user's query count. When the user query count metric crosses the monitor thresholds, an observation will be created.

• When monitoring rows accessed, each query must meet the query duration condition for that query row count to be included in the user's rows accessed metric. When the user's rows accessed metric crosses the monitor thresholds, an observation will be created.

Monitors resolution

Best practice: Ensure the audit frequency for each integration is set to a lowest value possible for your organization when using monitors. For Snowflake and Databricks Unity Catalog integrations, frequent audit record ingestion will lead to highly current audit records; however, there could be performance and cost impacts from the frequent jobs.

Monitors create observations when a user's query activity metrics exceed the specified threshold within the configured timeframe. Evaluating whether the user activity metrics have exceeded monitor thresholds is performed using a sliding window algorithm in increments of 10 minutes aligned to query events' timestamps.

Only completed queries are evaluated for the monitors. Once they are audited, they then fall into the timeframe window for the time that they complete.

Observations

An observation is based on the results of monitors and a user's query activity metric. Each observation can be marked open or acknowledged for review tracking purposes.

Observations surface the user query activity being monitored once the acceptable threshold has been exceeded. When creating a monitor, you can choose to be notified about the observations in one of three ways:

1. Never: Every time the monitor is exceeded, an observation will be created in Immuta, but you will not receive any webhook events from them.

2. Notify each time an Observation is generated: Every time the monitor is exceeded, it will create an observation, and a webhook event will notify you of that creation.

3. Notify the first time an Observation is generated for each user: Every time the monitor is exceeded, it will create an observation, and a webhook event will notify you when the first observation is created for a user. You will not receive notifications for observations from that monitor again for that specific user.

Observation contents

Observations contain the following information:

• Open or acknowledged label: A quick visual of the state of the observation.

• Severity label: This label aligns with the severity specified by the threshold within the monitor.

• The user name: The user who exceeded the severity thresholds of the monitor.

• Message: An Immuta-generated message describing what created the observation.

• Monitor type: The type of monitor that created the observation.

• Related events: A list of audit events that created the observation with links to the event details page for each.

Notifications

Users can use the webhooks configured when creating a monitor to send notifications to outside applications. Applications like Slack and Teams can receive these webhooks and surface them as alerts in the app. The notification contents include the Immuta profile ID for the user that caused the observation, the severity of the observation, a short message about what prompted the observation, and a link to look at the observation in Immuta for additional details. Note that the notification does not show any personally identifiable information about your users. The Immuta profile ID is an integer that represents a user profile in Immuta, but any additional information about the user must be accessed in the Immuta UI.

Example notification payload

{
    "observedResource": {
        "type": "USER",
        "profileId": "8",
        "identityProvider": "bim"
    },
    "severity": "HIGH",
    "link": "http://your-immuta.com/#/detect/observations/1/detail?utm_source=1&utm_medium=WEBHOOK",
    "metric": "1",
    "message": "Greater than 1000 query events in the last 1 hour.",
    "timestamp": "2023-12-13T22:05:30.612Z",
}

Snowflake data platform limitation

Due to a Snowflake limitation, a monitor for unauthorized query outcome cannot be combined with specific tags or sensitivities. It must be created for all data sources and cannot be scoped to a schema or specific data source. Successful query outcomes can be combined with specific tags or sensitivities and scoped to a specific schema or data source.