SaaS
Search powered by AI

Configure a Databricks Unity Catalog Integration

Immuta’s integration with Unity Catalog allows you to manage multiple Databricks workspaces through Unity Catalog while protecting your data with Immuta policies. Instead of manually creating UDFs or granting access to each table in Databricks, you can author your policies in Immuta and have Immuta manage and enforce Unity Catalog access-control policies on your data in Databricks clusters or SQL warehouses.

Use the /integrations endpoint to

Requirements

Permissions

Several different accounts are used to set up and maintain the Databricks Unity Catalog integration. The permissions required for each are outlined below.

  • Immuta account (required): This user configures the integration on the app settings page in Immuta. To access the app settings page, this user needs the following permission:

    • APPLICATION_ADMIN Immuta permission

  • Databricks service principal (required): This service principal is used continuously by Immuta to orchestrate Unity Catalog policies and maintain state between Immuta and Databricks. In the automatic setup option, Immuta also uses this service principal to create the Immuta-managed catalog. This service principal needs the following Databricks privileges:

    • CREATE CATALOG privilege on the Unity Catalog metastore. This is only required if you have Immuta automatically configure the integration in Databricks for you. If a separate user will run the Immuta script in Databricks to manually configure the integration, that Databricks user account needs this privilege instead.

    • OWNER permission on the Immuta catalog you configure.

    • OWNER privilege on one of the securables below so that Immuta can administer Unity Catalog row-level and column-level security controls.

      • on catalogs with schemas and tables registered as Immuta data sources. This permission could also be applied by granting OWNER on a catalog to a Databricks group that includes the Immuta service principal to allow for multiple owners.

      • on schemas with tables registered as Immuta data sources.

      • on all tables registered as Immuta data sources - if the OWNER permission cannot be applied at the catalog- or schema-level. In this case, each table registered as an Immuta data source must individually have the OWNER permission granted to the Immuta service principal.

    • USE CATALOG and USE SCHEMA on parent catalogs and schemas of tables registered as Immuta data sources so that the Immuta service principal can SELECT and MODIFY securables within the parent catalog and schema.

    • SELECT and MODIFY on all tables registered as Immuta data sources so that the Immuta service principal can grant and revoke access to tables and apply Unity Catalog row- and column-level security controls.

    • For native query audit (optional)

      • USE CATALOG on the system catalog

      • USE SCHEMA on the system.access schema

      • SELECT on the following system tables:

        • system.access.audit

        • system.access.table_lineage

        • system.access.column_lineage

  • Databricks account (recommended): This user account can manually configure the integration in Databricks to create the Immuta-managed catalog. To do so, this account requires the following Databricks privileges:

    • CREATE CATALOG on the Unity Catalog metastore

    • ACCOUNT ADMIN on the Unity Catalog metastore for native query audit (optional)

Authentication

  • Access token authentication: If using this method, generate a personal access token for the service principal that Immuta will use to manage policies in Unity Catalog. This service principal must have the privileges listed above for the metastore associated with the Databricks workspace.

  • OAuth machine-to-machine (M2M) authentication: If using this method, follow Databricks documentation to create a client secret for the Immuta service principal. This service principal must have the privileges listed above for the metastore associated with the Databricks workspace.

Configure the integration

  1. In Databricks, create a service principal with the privileges listed above.

  2. Opt to enable native query audit for Unity Catalog:

    1. Enable a system schema where the <SCHEMA_NAME> is access.

    2. Enable verbose audit logs in Unity Catalog.

    3. If you will configure the integration using the manual setup option, the Immuta script you will generate includes the SQL statements for granting required privileges to the service principal, so you can skip this step and continue to the manual setup section. Otherwise, manually grant the Immuta service principal access to the Databricks Unity Catalog system tables. For Databricks Unity Catalog audit to work, the service principal must have the following access at minimum:

      • USE CATALOG on the system catalog

      • USE SCHEMA on the system.access schema

      • SELECT on the following system tables:

        • system.access.audit

        • system.access.table_lineage

        • system.access.column_lineage

  3. You have two options for configuring your Databricks Unity Catalog integration. Select the method you prefer below to navigate to configuration instructions:

    • Automatic setup: Immuta creates the catalogs, schemas, tables, and functions using the service principal you created.

    • Manual setup: Run the Immuta script in Databricks yourself to create the catalog. You can also modify the script to customize your storage location for tables, schemas, or catalogs. The user running the script must have the Databricks privileges listed above.

Automatic setup

Copy the request example, and replace the values with your own as directed to configure the integration settings. The examples provided use JSON format, but the request also accepts YAML.

See the config object description for parameter definitions, value types, and additional configuration options.

curl -X 'POST' \
    'https://www.organization.immuta.com/integrations' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Databricks",
    "autoBootstrap": true,
    "config": {
      "workspaceUrl": "www.example-workspace.cloud.databricks.com",
      "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
      "authenticationType": "token",
      "token": "REDACTED",
      "catalog": "immuta"
    }
    }'

  1. Replace the Immuta URL and API key with your own.

  2. Change the config values to your own, where

    • workspaceUrl is your Databricks workspace URL.

    • httpPath is the HTTP path of your Databricks cluster or SQL warehouse.

    • token is the Databricks personal access token. This is the access token for the Immuta service principal.

    • catalog is the name of the Databricks catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.

Response

The response returns the status of the Databricks Unity Catalog integration configuration connection. See the response schema reference for details about the response schema.

A successful response includes the validation tests statuses.

{
  "id": "123456789",
  "status": "creating",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Metastore validation",
      "status": "passed"
    }, {
      "name": "Basic Connection Test",
      "result": [
      {
        "1": 1
      }
      ],
      "status": "passed"
    }
    ]
  }
}

Manual setup

To manually configure the integration, complete the following steps:

  1. Generate the Immuta script and run it in your Databricks environment.

  2. Configure the integration in Immuta.

Generate the script

Copy the request example, and replace the values with your own as directed to configure the integration settings. The examples provided use JSON format, but the request also accepts YAML.

See the config object description for parameter definitions, value types, and additional configuration options.

curl -X 'POST' \
    'https://www.organization.immuta.com/integrations/scripts/create' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Databricks",
    "autoBootstrap": false,
    "config": {
      "workspaceUrl": "www.example-workspace.cloud.databricks.com",
      "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
      "authenticationType": "token",
      "token": "REDACTED",
      "catalog": "immuta"
    }
    }'

  1. Replace the Immuta URL and API key with your own.

  2. Change the config values to your own, where

    • workspaceUrl is your Databricks workspace URL.

    • httpPath is the HTTP path of your Databricks cluster or SQL warehouse.

    • token is the Databricks personal access token. This is the access token for the Immuta service principal.

    • catalog is the name of the Databricks catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.

  3. Run the script returned in the response in your Databricks environment.

Response

The response returns the script for you to run in your environment.

Configure the integration in Immuta

Copy the request example, and replace the values with your own as directed to configure the integration settings. The examples provided use JSON format, but the request also accepts YAML. The payload you provide must match the payload sent when generating the script.

See the config object description for parameter definitions, value types, and additional configuration options.

curl -X 'POST' \
    'https://www.organization.immuta.com/integrations' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Databricks",
    "autoBootstrap": false,
    "config": {
      "workspaceUrl": "www.example-workspace.cloud.databricks.com",
      "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
      "authenticationType": "token",
      "token": "REDACTED",
      "catalog": "immuta"
    }
    }'

  1. Replace the Immuta URL and API key with your own.

  2. Pass the same payload you sent when generating the script, where

    • workspaceUrl is your Databricks workspace URL.

    • httpPath is the HTTP path of your Databricks cluster or SQL warehouse.

    • token is the Databricks personal access token. This is the access token for the Immuta service principal.

    • catalog is the name of the Databricks catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.

Response

The response returns the status of the Databricks Unity Catalog integration configuration connection. See the response schema reference for details about the response schema.

A successful response includes the validation tests statuses.

{
  "id": "123456789",
  "status": "creating",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Metastore validation",
      "status": "passed"
    }, {
      "name": "Basic Connection Test",
      "result": [
      {
        "1": 1
      }
      ],
      "status": "passed"
    }
    ]
  }
}

Get an integration

curl -X 'GET' \
    'https://www.organization.immuta.com/integrations/{id}' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f'

  1. Copy the request example.

  2. Replace the Immuta URL and API key with your own.

  3. Replace the {id} request parameter with the unique identifier of the integration you want to get. Alternatively, you can get a list of all integrations and their IDs with the GET /integrations endpoint.

Response

The response returns a Databricks Unity Catalog integration configuration. See the response schema reference for details about the response schema. An unsuccessful request returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.

{
  "id": "123456789",
  "status": "enabled",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Metastore validation",
      "status": "passed"
    }, {
      "name": "Basic Connection Test",
      "result": [
      {
        "1": 1
      }
      ],
      "status": "passed"
    }
    ]
  },
  "type": "Databricks",
  "autoBootstrap": false,
  "config": {
    "port": 443,
    "workspaceUrl": "www.example-workspace.cloud.databricks.com",
    "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
    "authenticationType": "token",
    "token": "REDACTED",
    "audit": {
      "enabled": false
    },
    "catalog": "immuta"
  }
}

Get all integrations

curl -X 'GET' \
    'https://www.organization.immuta.com/integrations' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f'

  1. Copy the request example.

  2. Replace the Immuta URL and API key with your own.

Response

The response returns the configuration for all integrations. See the response schema reference for details about the response schema. An unsuccessful request returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.

[
  {
    "id": "1",
    "status": "enabled",
    "validationResults": {
      "status": "passed",
      "validationTests": [
      {
        "name": "Initial Validation: Basic Connection Test",
        "status": "passed"
      },
      {
        "name": "Initial Validation: Default Warehouse Access Test",
        "status": "passed",
        "result": []
      },
      {
        "name": "Initial Validation: Validate access to Privileged Role",
        "status": "passed",
        "result": []
      },
      {
        "name": "Validate Automatic: Database Does Not Exist",
        "status": "passed"
      },
      {
        "name": "Validate Automatic: Impersonation Role Does Not Exist",
        "status": "skipped"
      },
      {
        "name": "Validate Automatic Bootstrap User Grants",
        "status": "passed"
      }
    ] },
    "type": "Snowflake",
    "autoBootstrap": true,
    "config": {
      "host": "organization.us-east-1.snowflakecomputing.com",
      "warehouse": "SAMPLE_WAREHOUSE",
      "database": "SNOWFLAKE_SAMPLE_DATA",
      "port": 443,
      "audit": {
          "enabled": false
        },
      "workspaces": {
        "enabled": false
      },
      "impersonation": {
        "enabled": false
      },
      "lineage": {
        "enabled": false
      },
      "authenticationType": "userPassword",
      "username": "<REDACTED>",
      "password": "<REDACTED>",
      "role": "ACCOUNTADMIN"
    },
    {
      "id": "2",
      "status": "enabled",
      "type": "Databricks",
      "validationResults": {
        "status": "passed",
        "validationTests": [
        {
          "name": "Metastore validation",
          "status": "passed"
        },
        {
          "name": "Basic Connection Test",
          "result": [
          {
            "1": 1
          }
          ],
          "status": "passed"
        }
        ]
      },
      "autoBootstrap": true,
      "config": {
        "workspaceUrl": "www.example-workspace.cloud.databricks.com",
        "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
        "authenticationType": "token",
        "token": "REDACTED",
        "audit": {
          "enabled": false
        },
        "catalog": "immuta"
      }
    }
  }
]

Update an integration configuration

You have two options for updating your integration. Follow the steps that match your initial configuration of autoBootstrap:

Automatic update

Copy the request example, and replace the values with your own as directed to configure the integration settings. The examples provided use JSON format, but the request also accepts YAML.

See the config object description for parameter definitions, value types, and additional configuration options.

This example updates the access token.

curl -X 'PUT' \
    'https://www.organization.immuta.com/integrations/{id}' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Databricks",
    "autoBootstrap": true,
    "config": {
      "workspaceUrl": "www.example-workspace.cloud.databricks.com",
      "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
      "authenticationType": "token",
      "token": "REDACTED",
      "groupPattern": {
        "deny": "admins"
      },
      "catalog": "immuta"
    }
    }'

  1. Replace the Immuta URL and API key with your own.

  2. Replace the {id} request parameter with the unique identifier of the integration you want to update.

  3. Change the config values to your own, where

    • workspaceUrl is your Databricks workspace URL.

    • httpPath is the HTTP path of your Databricks cluster or SQL warehouse.

    • token is the Databricks personal access token. This is the access token for the Immuta service principal.

    • catalog is the name of the Databricks catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.

Response

The response returns the status of the Databricks Unity Catalog integration configuration connection. See the response schema reference for details about the response schema.

A successful response includes the validation tests statuses.

{
  "id": "123456789",
  "status": "editing",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Metastore validation",
      "status": "passed"
    }, {
      "name": "Basic Connection Test",
      "result": [
      {
        "1": 1
      }
      ],
      "status": "passed"
    }
    ]
  }
}

Manual update

To manually update the integration, complete the following steps:

  1. Generate the updated Immuta script and run it in your Databricks environment.

  2. Update the integration in Immuta.

Generate the updated script

Copy the request example, and replace the values with your own as directed to generate the script. The example provided uses JSON format, but the request also accepts YAML.

See the config object description for parameter definitions, value types, and additional configuration options.

This example updates the access token.

curl -X 'POST' \
    'https://www.organization.immuta.com/integrations/{id}/scripts/edit' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Databricks",
    "autoBootstrap": false,
    "config": {
      "workspaceUrl": "www.example-workspace.cloud.databricks.com",
      "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
      "authenticationType": "token",
      "token": "REDACTED",
      "catalog": "immuta"
    }
    }'

  1. Replace the Immuta URL and API key with your own.

  2. Replace the {id} request parameter with the unique identifier of the integration you want to update.

  3. Change the config values to your own, where

    • workspaceUrl is your Databricks workspace URL.

    • httpPath is the HTTP path of your Databricks cluster or SQL warehouse.

    • token is the Databricks personal access token. This is the access token for the Immuta service principal.

    • catalog is the name of the Databricks catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.

  4. Run the script returned in the response in your Databricks environment.

Response

The response returns the script for you to run in your Databricks environment.

Update the integration in Immuta

Copy the request example, and replace the values with your own as directed to update the integration settings. The example provided uses JSON format, but the request also accepts YAML. The payload you provide must match the payload sent when generating the script.

See the config object description for parameter definitions, value types, and additional configuration options.

This example updates the access token.

curl -X 'PUT' \
    'https://www.organization.immuta.com/integrations/{id}' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Databricks",
    "autoBootstrap": false,
    "config": {
      "workspaceUrl": "www.example-workspace.cloud.databricks.com",
      "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
      "authenticationType": "token",
      "token": "REDACTED",
      "catalog": "immuta"
    }
    }'

  1. Replace the Immuta URL and API key with your own.

  2. Replace the {id} request parameter with the unique identifier of the integration you want to update.

  3. Pass the same payload you sent when updating the script, where

    • workspaceUrl is your Databricks workspace URL.

    • httpPath is the HTTP path of your Databricks cluster or SQL warehouse.

    • token is the Databricks personal access token. This is the access token for the Immuta service principal.

    • catalog is the name of the Databricks catalog Immuta will create to store internal entitlements and other user data specific to Immuta. This catalog will only be readable for the Immuta service principal and should not be granted to other users. The catalog name may only contain letters, numbers, and underscores and cannot start with a number.

Response

The response returns the status of the Databricks Unity Catalog integration configuration connection. See the response schema reference for details about the response schema.

A successful response includes the validation tests statuses.

{
  "id": "123456789",
  "status": "editing",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Metastore validation",
      "status": "passed"
    }, {
      "name": "Basic Connection Test",
      "result": [
      {
        "1": 1
      }
      ],
      "status": "passed"
    }
    ]
  }
}

Delete an integration

curl -X 'DELETE' \
    'https://www.organization.immuta.com/integrations/{id}' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f'

  1. Copy the request example.

  2. Replace the Immuta URL and API key with your own.

  3. Replace the {id} request parameter with the unique identifier of the integration you want to delete.

Response

The response returns the status of the Databricks Unity Catalog integration configuration that has been deleted. See the response schema reference for details about the response schema. An unsuccessful request returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.

{
  "id": "123456789",
  "status": "deleting",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Metastore validation",
      "status": "passed"
    }, {
      "name": "Basic Connection Test",
      "result": [
      {
        "1": 1
      }
      ],
      "status": "passed"
    }
    ]
  }
}
PreviousConfigure an Azure Synapse Analytics IntegrationNextConfigure a Google BigQuery Integration

Last updated

Self-managed versions

2024.32024.22024.1

Copyright © 2014-2024 Immuta Inc. All rights reserved.