Google BigQuery Viewless Integration Reference Guide

Learn about how the Google BigQuery viewless integration works and what Immuta does in your environment to register your data and keep it in sync

Public preview: This integration is available to all accounts that request to enable it for their tenant. Contact your Immuta representative to enable it.

The Google BigQuery viewless integration allows you to register data from Google BigQuery in Immuta and enforce subscription policies and row-level data policies on that data. The sequence diagram below outlines the events that occur when an Immuta user queries a Google BigQuery table that has been registered as an Immuta data source.

What does Immuta do in my environment?

Registering a connection

Google BigQuery is configured and data is registered through connections, an Immuta feature that allows you to register your data objects through a single connection to make data registration more scalable for your organization. Instead of registering schema and databases individually, you can register them all at once and allow Immuta to monitor your data platform for changes so that data sources are added and removed automatically to reflect the state of data in your data platform.

When the connection is registered, Immuta ingests and stores connection metadata in the Immuta metadata database. In the example below, the Immuta application administrator connects the database that contains marketing-data, research-data, and cs-data tables. Immuta registers these tables as data sources and stores the table metadata in the Immuta metadata database.

Immuta presents a hierarchical view of your data that reflects the hierarchy of objects in Google BigQuery after registration is complete:

• Project
• Dataset
• Data object

Beyond making the registration of your data more intuitive, connections provides more control. Instead of performing operations on individual schemas or tables, you can perform operations (such as object sync) at the connection level.

See the Connections reference guide for details about connections and how to manage them. To configure your Google BigQuery viewless integration and register data, see the Register a Google BigQuery connection guide.

Applying policies

After objects are registered in Immuta, you can author subscription policies or row-level data policies in Immuta to enforce access controls. The Google BigQuery viewless integration is additive: Immuta does not update or override existing access controls on securables unless an Immuta policy explicitly dictates an access change.

Subscription policies

Immuta enforces read and write subscription policies on Google BigQuery objects by updating allow policies on Google BigQuery resources that grant and revoke access to objects according to the allow policy.

When a subscription policy is activated on an Immuta data source,

1. Immuta creates a custom role in the Google BigQuery project with the necessary permissions on the data object.
2. Immuta calculates the users affected by the subscription policy and creates a Google group for those users.
3. Immuta adds the appropriate Google groups to a Google BigQuery allow policy and binds them to the custom role. This binding grants the custom role to those Google groups, which then permits users in those groups to access the data object.

The Protecting data in Google BigQuery page provides an example of this policy enforcement. See the Subscription policy access types page for details about the Google BigQuery privileges granted to users when they are subscribed to a data source protected by a subscription policy.

Data policies

Immuta applies Google BigQuery row-level security controls that are enforced natively within Google BigQuery.

When a row-level data policy is applied to a table registered in Immuta, Immuta creates a Google BigQuery row-level access policy that includes the target table and a filter_expression that compares the querying user's entitlements (saved in the immuta_profiles table) against those specified in the data policy. Then, when a user queries that data source, the rows returned are filtered according to what the policy dictates that they're allowed to see.

The Protecting data in Google BigQuery page provides an example of this policy enforcement.

Required Google BigQuery privileges

The privileges that the Google BigQuery integration requires align to the least privilege security principle. The table below describes each privilege required by the setup user and the service account.

Google BigQuery users with access to Immuta-governed Google BigQuery data sources will be able to read the immuta_profiles table

Google BigQuery states that grantees must have the bigquery.tables.getData permission on the target and referenced tables in policies. Because Immuta policy predicates read immuta_profiles for every entitlement check, Immuta grants every user querying a protected table read access to immuta_profiles.

Google BigQuery privilege	User requiring the privilege	Explanation
roles/iam.serviceAccountCreator roles/iam.serviceAccountAdmin roles/resourcemanager.projectIamAdmin on the project (or be a project owner)	Setup user	This privilege is required so that the setup user can create the service account and custom role for Immuta.
bigquery.datasets.get bigquery.tables.get bigquery.tables.list	Service account	These privileges allow the Immuta service account to get metadata for datasets and their objects.
At the GCP organization level: resourcemanager.organizations.get At the Google Workspace level (also scoped to the organization, but a different control plane): Group Create privilege and privilege required to apply security label on group resource Group Editor privilege scoped to Access to Security Groups Only At the GCP project level: bigquery.datasets.create bigquery.datasets.delete bigquery.datasets.update bigquery.jobs.create bigquery.routines.create bigquery.routines.delete bigquery.routines.get bigquery.routines.update bigquery.rowAccessPolicies.create bigquery.rowAccessPolicies.delete bigquery.rowAccessPolicies.list bigquery.rowAccessPolicies.setIamPolicy bigquery.rowAccessPolicies.update bigquery.tables.create bigquery.tables.delete bigquery.tables.getData bigquery.tables.getIamPolicy bigquery.tables.setIamPolicy bigquery.tables.update bigquery.tables.updateData iam.roles.create iam.roles.get iam.roles.list iam.roles.undelete iam.roles.update resourcemanager.projects.getIamPolicy resourcemanager.projects.setIamPolicy	Service account	These privileges allow the Immuta service account to create Google groups to enforce read and write subscription policies and row-level data policies on data objects registered as Immuta data sources. Immuta does not manage or have access to your existing Google groups managed outside Immuta.

Maintaining state with Google BigQuery

The following user actions spur various processes in the Google BigQuery viewless integration so that Immuta data remains synchronous with data in Google BigQuery:

• Data source created or updated: Immuta registers data source metadata and stores that metadata in the Immuta metadata database.
• Data source deleted: Immuta deletes the data source metadata from the metadata database.
• User account is mapped to Immuta: When a user account is mapped to Immuta, their metadata is stored in the metadata database.
• User subscribed to a data source: When users are added to a data source by a data owner or through a subscription policy, Immuta calculates the users affected by the subscription policy and creates a Google group for those users. Then Immuta grants Google BigQuery privileges to their group through an allow policy.
• Automatic subscription policy applied to or updated on a data source: Immuta calculates the users and data sources affected by the policy change and grants or revokes users' privileges on the data object by adding them to or removing them from the Immuta-managed Google group that has access to the securable. See the Protecting data in Google BigQuery page for details about this process.
• Subscription policy deleted: Immuta calculates the users and Google groups affected by the policy and revokes privileges on the Google BigQuery object by removing users from the Immuta-created Google group. If there are no longer any members of an Immuta-created Google group, that group is deleted by Immuta.
• User removed from a data source: Immuta removes the user from the Immuta-managed Google group.

Google BigQuery destructive operations that remove row access policies

Google BigQuery silently removes row access policies on tables that have any of the following Google BigQuery operations run against them:

• CREATE OR REPLACE TABLE
• bq load --replace
• WRITE_TRUNCATE jobs
• bq cp -f

If you run any of these operations in Google BigQuery against objects protected by Immuta, you must manually trigger Immuta policies to resync. See the Google Cloud documentation for details about this behavior.

Supported object types

Object type	Subscription policy support	Data policy support	Request app support
Tables	✅	✅	✅
Views	✅	❌	✅
Materialized views	✅	❌	✅

Supported policies

The Google BigQuery integration allows users to author subscription and row-level data policies to enforce access controls. See the corresponding pages for details about specific types of policies supported:

• Subscription policy access types
• Data policies reference guide

See the applying policies section for details about policy enforcement.

Security and compliance

See the Security and compliance in Google BigQuery guide for details about security, supported authentication methods, and audit.

User registration and ID mapping

The built-in Immuta IAM can be used as a complete solution for authentication and user entitlement. However, you can connect your existing identity management provider to Immuta to use that system for authentication and user entitlement instead. Each of the supported IAM protocols includes a set of configuration options that enable Immuta to communicate with the IAM system and map the users, permissions, groups, and attributes into Immuta.

For policies to impact the right users, the user account in Immuta must be mapped to the user account in Google BigQuery. You can ensure these accounts are mapped correctly in the following ways:

• Automatically: If usernames in Google BigQuery align with usernames in the external IAM and those accounts align with an IAM attribute, you can enter that IAM attribute on the app settings page to automatically map user IDs in Immuta to Google BigQuery.
• Manually: You can manually map user IDs for individual users.

For guidance on connecting your IAM to Immuta, see the how-to guide for your protocol.

Limitations and known issues

The following Immuta features are unsupported:

• Identification
• Select data policies

  • Limit to purpose
  • Masking
  • Minimization
  • Time-based restrictions
• Tag ingestion
• Query audit

Google BigQuery has the following limitations:

• Limit of one allow policy per resource
• Limit of 250 Google groups in an allow policy
• Limit of 1500 groups Immuta can own
• Limit of 400 row-access policies per table
• Destructive operations that remove row access policies: Google BigQuery silently removes row access policies on tables that have any of the following Google BigQuery operations run against them:

  • CREATE OR REPLACE TABLE
  • bq load --replace
  • WRITE_TRUNCATE jobs
  • bq cp -f

  If you run any of these operations in Google BigQuery against objects protected by Immuta, you must manually trigger Immuta policies to resync. See the Google Cloud documentation for details about this behavior.