AWS PrivateLink for Databricks

AWS PrivateLink provides private connectivity from the Immuta SaaS platform to Databricks accounts hosted on AWS. It ensures that all traffic to the configured endpoints only traverses private networks.

This front-end PrivateLink connection allows users to connect to the Databricks web application, REST API, and Databricks Connect API over a VPC interface endpoint. For details about AWS PrivateLink in Databricks and the network flow in a typical implementation, explore the Databricks documentation.

This feature is supported in most regions across Immuta's global segments (NA, EU, and AP); contact your Immuta representative if you have questions about availability.

Requirements

Databricks

Ensure that your accounts meet the following requirements:

• Your Databricks account is on the E2 version of the platform.

• Your Databricks account is on the Enterprise pricing tier.
• You have your Databricks account ID from the account console.
• You have an Immuta SaaS tenant.

• AWS PrivateLink for Databricks has been enabled.

Databricks workspace

Ensure that your workspace meets the following requirements:

• Your workspace must be in an AWS region that supports the E2 version of the platform.
• Your Databricks workspace must use a customer-managed VPC to add any PrivateLink connection.
• Your workspaces must be configured with private_access_settings objects.

You cannot configure a connection to your workspace over the public internet if PrivateLink is enabled.

If you have PrivateLink configured on your workspace, Databricks will update the DNS records for that workspace URL to resolve to <region>.privatelink.cloud.databricks.com. Immuta SaaS uses these publicly-resolvable records to direct traffic to a PrivateLink endpoint on our network.

This means that if you have PrivateLink enabled on your workspace, you must follow these instructions to configure your integration. Even if your workspace is also publicly-routable, Databricks's DNS resolution forces the traffic over PrivateLink.

The two supported configurations are

• A workspace with no PrivateLink configuration, which resolves to public IP addresses.

• A workspace with PrivateLink configuration, which allows access from the Immuta SaaS regional endpoint (listed below).

Enablement

Contact your Databricks representative to enable AWS PrivateLink on your account.

Configure Databricks with AWS PrivateLink

1. Register the Immuta VPC endpoint for the applicable AWS region with your Databricks workspaces. The Immuta VPC endpoint IDs are listed in the table below.

AWS region	VPC endpoint ID
ap-northeast-1 Asia Pacific (Tokyo)	vpce-08cadda15f0f70462
ap-south-1 Asia Pacific (Mumbai)	vpce-0efef886a4fbd9532
ap-southeast-1 Asia Pacific (Singapore)	vpce-07e9890053f5084b2
ap-southeast-2 Asia Pacific (Sydney)	vpce-0d363d9ea82658bec
ca-central-1 Canada (Central)	vpce-01933bcf30ac4ed19
eu-central-1 Europe (Frankfurt)	vpce-0048e36edfb27d0aa
eu-west-1 Europe (Ireland)	vpce-0783d9412b046df1f
eu-west-2 Europe (London)	vpce-0f546cc413bf70baa
us-east-1 US East (Virginia)	vpce-0c6e8f337e0753aa9
us-east-2 US East (Ohio)	vpce-00ba42c4e2be20721
us-west-2 US West (Oregon)	vpce-029306c6a510f7b79

1. Identify your private access level (either ACCOUNT or ENDPOINT) and configure your Databricks workspace accordingly.

   1. If the private_access_level on your private_access_settings object is set to ACCOUNT, no additional configuration is required.

   2. If the private_access_level on your private_access_settings object is set to ENDPOINT, using the table above, you will need to add it to the allowed_vpc_endpoint_ids list inside your private_access_settings object in Databricks. For example,

"private_access_settings_name": "immuta-access",
"region": "us-east-1",
"public_access_enabled": false,
"private_access_level": "ENDPOINT",
"allowed_vpc_endpoint_ids": [
        "vpce-0fe5b17a0707d6fa5"
]

1. Configure Databricks depending on your integration type:

   1. Configure the Databricks Unity Catalog integration using your standard cloud.databricks.com workspace URL as the Host.
   2. Configure the Databricks Spark integration using your standard cloud.databricks.com URL. And register your tables as Immuta data sources using the cloud.databricks.com as the Server when registering data sources.