Databricks Lakebase Integration
The Databricks Lakebase integration registers data from Databricks Lakebase in Immuta and enforces subscription policies on that data when queried in PostgreSQL. The sequence diagram below outlines the events that occur when an Immuta user who is subscribed to a data source queries that data.
What does Immuta do in my environment?
Registering a connection
Databricks Lakebase is configured and data is registered through connections, an Immuta feature that allows you to register your data objects through a single connection to make data registration more scalable for your organization. Instead of registering schema and catalogs individually, you can register them all at once and allow Immuta to monitor your data platform for changes so that data sources are added and removed automatically to reflect the state of data in your data platform.
During connection registration, you provide Immuta credentials with the privileges outlined on the Register a Databricks Lakebase connection page. When the connection is registered, Immuta ingests and stores connection metadata in the Immuta metadata database.
In the example below, the Immuta application administrator connects the database that contains marketing-data , research-data , and cs-data tables. Immuta these tables as data sources and stores the table metadata in the Immuta metadata database.
Immuta presents a hierarchical view of your data that reflects the objects in PostgreSQL hosted on Databricks Lakebase after registration is complete:
Lakebase database
Database
Schema
Table
Beyond making the registration of your data more intuitive, connections provides more control. Instead of performing operations on individual schemas or tables, you can perform operations (such as object sync) at the connection level.
See the Connections reference guide for details about connections and how to manage them. To configure your Databricks Lakebase connection, see the Register a Databricks Lakebase connection guide.
Applying policies
Immuta enforces read and write subscription policies on Databricks Lakebase tables by issuing SQL statements in PostgreSQL that grant and revoke access to tables according to the policy.
When a user is subscribed to a table registered in Immuta,
Immuta creates a role for that user in PostgreSQL, if one doesn't already exist.
PostgreSQL stores that role in its internal system catalog.
Immuta issues grants to that user's role in PostgreSQL to enforce policy. The Protecting data page provides an example of this policy enforcement.
See the Subscription policy access types page for details about the privileges granted to users when they are subscribed to a data source protected by a subscription policy.
Databricks Lakebase privileges
The privileges that the Databricks Lakebase integration requires align to the least privilege security principle. The table below describes each privilege required by the user.
databricks_superuser
This privilege is required so that Immuta can create and grant permissions to PostgreSQL roles.
CREATEROLE
Because privileges are granted to roles, this privilege is required so that Immuta can create PostgreSQL roles and manage role membership to enforce access controls for Databricks Lakebase objects.
Maintaining state with Databricks Lakebase
The following user actions spur various processes in the Databricks Lakebase integration so that Immuta data remains synchronous with data in Databricks Lakebase:
Data source created: Immuta registers data source metadata and stores that metadata in the Immuta metadata database.
Data source deleted: Immuta deletes the data source metadata from the metadata database and removes subscription policies from that table.
User account is mapped to Immuta: When a user account is mapped to Immuta, their metadata is stored in the metadata database.
User subscribed to a data source: When a user is added to a data source by a data owner or through a subscription policy, Immuta creates a role for that user (if a role for them does not already exist) and grants PostgreSQL privileges to their role.
Automatic subscription policy applied to or updated on a data source: Immuta calculates the users and data sources affected by the policy change and grants or revokes users' privileges on the PostgreSQL table. See the Protecting data page for details about this process.
Subscription policy deleted: Immuta revokes privileges from the affected roles.
User removed from a data source: Immuta revokes privileges from the user's role.
The database instance must be up and running for state to be maintained and object sync to successfully complete. If the database instance is stopped, object sync will fail.
Supported object types
Databricks Lakebase holds PostgreSQL objects. See the PostgreSQL supported object types section for details about the PostgreSQL objects that Immuta supports.
Supported policies
Immuta supports Databricks Lakebase policies through PostgreSQL. See the PostgreSQL supported policies section for details about the policies that Immuta supports.
Security and compliance
Authentication method
The Databricks Lakebase integration supports OAuth machine-to-machine (M2M) authentication to register a connection.
The Databricks Lakebase connection authenticates as a Databricks identity and generates an OAuth token. Immuta then uses that token as a password when connecting to PostgreSQL. To enable secure, automated machine-to machine access to the database instance, the connection must obtain an OAuth token using a Databricks service principal. See the Databricks OAuth machine-to-machine (M2M) authentication page for more details.
User registration and ID mapping
The built-in Immuta IAM can be used as a complete solution for authentication and user entitlement. However, you can connect your existing identity management provider to Immuta to use that system for authentication and user entitlement instead. Each of the supported IAM protocols includes a set of configuration options that enable Immuta to communicate with the IAM system and map the users, permissions, groups, and attributes into Immuta.
For policies to impact the right users, the user account in Immuta must be mapped to the user account in PostgreSQL. You can ensure these accounts are mapped correctly in the following ways:
Automatically: If usernames in PostgreSQL align with usernames in the external IAM and those accounts align with an IAM attribute, you can enter that IAM attribute on the app settings page to automatically map user IDs in Immuta to PostgreSQL.
Manually: You can manually map user IDs for individual users.
For guidance on connecting your IAM to Immuta, see the how-to guide for your protocol.
Limitations and known issues
The following Immuta features are unsupported:
Data policies
Impersonation
Tag ingestion
Query audit
Subscription policies on partitioned tables
Last updated
Was this helpful?