2024.2
Search powered by AI

Configure a Google BigQuery Integration

Private preview: This integration is available to select accounts. Reach out to your Immuta representative for details.

The Google BigQuery resource allows you to create, configure, and manage your Google BigQuery integration. In this integration, Immuta generates policy-enforced views in your configured Google BigQuery dataset for tables registered as Immuta data sources.

Use the /integrations endpoint to

Requirements

  • APPLICATION_ADMIN Immuta permission

  • Google BigQuery integration enabled in Immuta (work with your Immuta representative to enable this integration)

  • To execute the Immuta script from your command line to create a Google Cloud service account and role, you must be authenticated to the gcloud CLI utility as a user with all of the following roles:

    • roles/iam.roleAdmin

    • roles/iam.serviceAccountAdmin

    • roles/serviceusage.serviceUsageAdmin

Prerequisite

Create a Google Cloud service account and role by either using the Google Cloud console or the provided Immuta script.

Configure the integration

curl -X 'POST' \
    'https://www.organization.immuta.com/integrations' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Google BigQuery",
    "autoBootstrap": false,
    "config": {
      "role": "immuta",
      "datasetSuffix": "_secureView",
      "dataset": "immuta",
      "location": "us-east1",
      "credential": "{\"type\":\"service_account\",\"project_id\":\"innate-conquest-123456\",\"private_key_id\":\"9163c12345690924f5dd218ff39\",\"private_key\":\"-----BEGIN PRIVATE KEY-----\nXXXXXXXro0s\n/yQlPQijowkccmrmWJyr93kdLnwJzBvLHCto/+W\ncvF2ygX9oM/dyUK//z//4nptMp+Ck//Yw3D4rIBwGu4DWiR1qRnf\nDoGyXfThPTQ==\n-----END PRIVATE KEY-----\n\",\"client_email\":\"service-account-id@innate-conquest-123456.iam.gserviceaccount.com\",\"client_id\":\"1166290***432952487857\",\"auth_uri\":\"https://accounts.google.com/o/oauth2/auth\",\"token_uri\":\"https://oauth2.googleapis.com/token\",\"auth_provider_x509_cert_url\":\"https://www.googleapis.com/oauth2/v1/certs\",\"client_x509_cert_url\":\"https://www.googleapis.com/robot/v1/metadata/x509/service-accound-id%40innate-conquest-123456.iam.gserviceaccount.com\",\"universe_domain\":\"googleapis.com\"}"
    }
    }'

  1. Copy the request example. The example uses JSON format, but the request also accepts YAML.

  2. Replace the Immuta URL and API key with your own.

  3. Change the config values to your own, where

    • role is the Google Cloud role used to connect to Google BigQuery.

    • datasetSuffix is the suffix to postfix to the name of each dataset created to store secure views. This string must start with an underscore.

    • dataset is the name of the BigQuery dataset to provision inside of the project for Immuta metadata storage.

    • location is the dataset's location, which can be any valid GCP location (such as us-east1).

    • credential is the Google BigQuery service account JSON keyfile credential content. See the Google documentation for guidance on generating and downloading this keyfile.

See the config object description for parameter definitions, value types, and additional configuration options.

Response

The response returns the status of the Google BigQuery integration configuration connection. See the response schema reference for details about the response schema.

A successful response includes the validation tests statuses.

{
  "id": "123456789",
  "status": "creating",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Basic Validation: Verify service account not being used for data source connection credentials",
      "status": "passed"
    },
    {
      "name": "Basic Validation: Immuta Service Account postfix",
      "status": "passed"
    },
    {
      "name": "Basic Validation: Non-matching service account in key file",
      "status": "passed"
    },
    {
      "name": "Basic Validation: Connection can be made to BigQuery",
      "status": "passed"
    },
    {
      "name": "Initialize Validation: [Dataset - exists] innate-conquest-123456.immuta",
      "status": "passed"
    },
    {
      "name": "Validate Manual: [Dataset - create] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144",
      "status": "passed"
    },
    {
      "name": "Validate Manual: [Table - create] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE",
      "status": "passed"
    },
    {
      "name": "Validate Manual: [Table - insert] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE",
      "status": "warning",
      "message": "Billing has not been enabled for this project. Enable billing at https://console.cloud.google.com/billing. DML queries are not allowed in the free tier. Set up a billing account to remove this restriction.. Confirm that the bootstrap script was executed without any modifications. This warning will not prevent the connection but continue at your own risk as some features may not function."
    },
    {
      "name": "Validate Manual: [Table - get] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE (After insert)",
      "status": "skipped"
    },
    {
      "name": "Validate Manual: [Table - update] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE",
      "status": "skipped"
    },
    {
      "name": "Validate Manual: [Table - get] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE (After update)",
      "status": "skipped"
    },
    {
      "name": "Validate Manual: [Table - delete] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE",
      "status": "passed"
    },
    {
      "name": "Validate Manual: [Dataset - delete] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144",
      "status": "passed"
    }
    ]
  }
}

Get an integration

curl -X 'GET' \
    'https://www.organization.immuta.com/integrations/{id}' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f'

  1. Copy the request example.

  2. Replace the Immuta URL and API key with your own.

  3. Replace the {id} request parameter with the unique identifier of the integration you want to get. Alternatively, you can get a list of all integrations and their IDs with the GET /integrations endpoint.

Response

The response returns a Google BigQuery integration configuration. See the response schema reference for details about the response schema. An unsuccessful response returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.

{
  "id": "123456789",
  "status": "enabled",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Basic Validation: Verify service account not being used for data source connection credentials",
      "status": "passed"
    },
    {
      "name": "Basic Validation: Immuta Service Account postfix",
      "status": "passed"
    },
    {
      "name": "Basic Validation: Non-matching service account in key file",
      "status": "passed"
    },
    {
      "name": "Basic Validation: Connection can be made to BigQuery",
      "status": "passed"
    },
    {
      "name": "Initialize Validation: [Dataset - exists] innate-conquest-123456.immuta",
      "status": "passed"
    },
    {
      "name": "Validate Manual: [Dataset - create] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144",
      "status": "passed"
    },
    {
      "name": "Validate Manual: [Table - create] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE",
      "status": "passed"
    },
    {
      "name": "Validate Manual: [Table - insert] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE",
      "status": "warning",
      "message": "Billing has not been enabled for this project. Enable billing at https://console.cloud.google.com/billing. DML queries are not allowed in the free tier. Set up a billing account to remove this restriction.. Confirm that the bootstrap script was executed without any modifications. This warning will not prevent the connection but continue at your own risk as some features may not function."
    },
    {
      "name": "Validate Manual: [Table - get] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE (After insert)",
      "status": "skipped"
    },
    {
      "name": "Validate Manual: [Table - update] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE",
      "status": "skipped"
    },
    {
      "name": "Validate Manual: [Table - get] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE (After update)",
      "status": "skipped"
    },
    {
      "name": "Validate Manual: [Table - delete] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE",
      "status": "passed"
    },
    {
      "name": "Validate Manual: [Dataset - delete] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144",
      "status": "passed"
    }
    ]
  },
  "type": "Google BigQuery",
  "autoBootstrap": false,
  "config": {
    "port": 443,
    "role": "immuta",
    "datasetSuffix": "_secureView",
    "dataset": "immuta",
    "location": "us-east1",
    "credential": {
      "type": "service_account",
      "project_id": "innate-conquest-123456",
      "private_key_id": "9163c12345690924f5dd218ff39",
      "private_key": "-----BEGIN PRIVATE KEY-----\nXXXXXXXro0s\n/yQlPQijowkccmrmWJyr93kdLnwJzBvLHCto/+W\ncvF2ygX9oM/dyUK//z\//4nptMp+Ck//Yw3D4rIBwGu4DWiR1qRnf\nDoGyXfThPTQ==\n-----END PRIVATE KEY-----\n",
      "client_email": "service-account-id@innate-conquest-123456.iam.gserviceaccount.com",
      "client_id": "1166290***432952487857",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://oauth2.googleapis.com/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-accound-id%40innate-conquest-123456.iam.gserviceaccount.com",
      "universe_domain": "googleapis.com"
    }
  }
  }

Get all integrations

curl -X 'GET' \
    'https://www.organization.immuta.com/integrations' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f'

  1. Copy the request example.

  2. Replace the Immuta URL and API key with your own.

Response

The response returns the configuration for all integrations. See the response schema reference for details about the response schema. An unsuccessful request returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.

[
  {
    "id": "1",
    "status": "enabled",
    "validationResults": {
      "status": "passed",
      "validationTests": [
      {
        "name": "Initial Validation: Basic Connection Test",
        "status": "passed"
      },
      {
        "name": "Initial Validation: Default Warehouse Access Test",
        "status": "passed",
        "result": []
      },
      {
        "name": "Initial Validation: Validate access to Privileged Role",
        "status": "passed",
        "result": []
      },
      {
        "name": "Validate Automatic: Database Does Not Exist",
        "status": "passed"
      },
      {
        "name": "Validate Automatic: Impersonation Role Does Not Exist",
        "status": "skipped"
      },
      {
        "name": "Validate Automatic Bootstrap User Grants",
        "status": "passed"
      }
    ] },
    "type": "Snowflake",
    "autoBootstrap": true,
    "config": {
      "host": "organization.us-east-1.snowflakecomputing.com",
      "warehouse": "SAMPLE_WAREHOUSE",
      "database": "SNOWFLAKE_SAMPLE_DATA",
      "port": 443,
      "audit": {
          "enabled": false
        },
      "workspaces": {
        "enabled": false
      },
      "impersonation": {
        "enabled": false
      },
      "lineage": {
        "enabled": false
      },
      "authenticationType": "userPassword",
      "username": "<REDACTED>",
      "password": "<REDACTED>",
      "role": "ACCOUNTADMIN"
    },
    {
      "id": "2",
      "status": "enabled",
      "type": "Databricks",
      "validationResults": {
        "status": "passed",
        "validationTests": [
        {
          "name": "Metastore validation",
          "status": "passed"
        },
        {
          "name": "Basic Connection Test",
          "result": [
          {
            "1": 1
          }
          ],
          "status": "passed"
        }
        ]
      },
      "autoBootstrap": true,
      "config": {
        "workspaceUrl": "www.example-workspace.cloud.databricks.com",
        "httpPath": "sql/protocolv1/o/0/0000-00000-abc123",
        "token": "REDACTED",
        "audit": {
          "enabled": false
        },
        "catalog": "immuta"
      }
    }
  }
]

Update an integration configuration

curl -X 'PUT' \
    'https://www.organization.immuta.com/integrations/{id}' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f' \
    -d '{
    "type": "Google BigQuery",
    "autoBootstrap": false,
    "config": {
      "role": "immuta",
      "datasetSuffix": "_secureView",
      "dataset": "immuta",
      "location": "us-east1",
      "credential": "{\"type\":\"service_account\",\"project_id\":\"innate-conquest-123456\",\"private_key_id\":\"9163c12345690924f5dd218ff39\",\"private_key\":\"-----BEGIN PRIVATE KEY-----\nXXXXXXXro0s\n/yQlPQijowkccmrmWJyr93kdLnwJzBvLHCto/+W\ncvF2ygX9oM/dyUK//z//4nptMp+Ck//Yw3D4rIBwGu4DWiR1qRnf\nDoGyXfThPTQ==\n-----END PRIVATE KEY-----\n\",\"client_email\":\"service-account-id@innate-conquest-123456.iam.gserviceaccount.com\",\"client_id\":\"1166290***432952487857\",\"auth_uri\":\"https://accounts.google.com/o/oauth2/auth\",\"token_uri\":\"https://oauth2.googleapis.com/token\",\"auth_provider_x509_cert_url\":\"https://www.googleapis.com/oauth2/v1/certs\",\"client_x509_cert_url\":\"https://www.googleapis.com/robot/v1/metadata/x509/service-accound-id%40innate-conquest-123456.iam.gserviceaccount.com\",\"universe_domain\":\"googleapis.com\"}"
    }
    }'

  1. Copy the request example, which updates the private key. The example uses JSON format, but the request also accepts YAML.

  2. Replace the Immuta URL and API key with your own.

  3. Replace the {id} request parameter with the unique identifier of the integration you want to update.

  4. Change the config values to your own, where

    • role is the Google Cloud role used to connect to Google BigQuery.

    • datasetSuffix is the suffix to postfix to the name of each dataset created to store secure views. This string must start with an underscore.

    • dataset is the name of the BigQuery dataset to provision inside of the project for Immuta metadata storage.

    • location is the dataset's location, which can be any valid GCP location (such as us-east1).

    • credential is the Google BigQuery service account JSON keyfile credential content. See the Google documentation for guidance on generating and downloading this keyfile.

See the config object description for parameter definitions, value types, and additional configuration options.

Response

The response returns the status of the Google BigQuery integration configuration connection. See the response schema reference for details about the response schema.

A successful response includes the validation tests statuses.

{
  "id": "123456789",
  "status": "editing",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Basic Validation: Verify service account not being used for data source connection credentials",
      "status": "passed"
    },
    {
      "name": "Basic Validation: Immuta Service Account postfix",
      "status": "passed"
    },
    {
      "name": "Basic Validation: Non-matching service account in key file",
      "status": "passed"
    },
    {
      "name": "Basic Validation: Connection can be made to BigQuery",
      "status": "passed"
    },
    {
      "name": "Initialize Validation: [Dataset - exists] innate-conquest-123456.immuta",
      "status": "passed"
    },
    {
      "name": "Validate Manual: [Dataset - create] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144",
      "status": "passed"
    },
    {
      "name": "Validate Manual: [Table - create] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE",
      "status": "passed"
    },
    {
      "name": "Validate Manual: [Table - insert] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE",
      "status": "warning",
      "message": "Billing has not been enabled for this project. Enable billing at https://console.cloud.google.com/billing. DML queries are not allowed in the free tier. Set up a billing account to remove this restriction.. Confirm that the bootstrap script was executed without any modifications. This warning will not prevent the connection but continue at your own risk as some features may not function."
    },
    {
      "name": "Validate Manual: [Table - get] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE (After insert)",
      "status": "skipped"
    },
    {
      "name": "Validate Manual: [Table - update] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE",
      "status": "skipped"
    },
    {
      "name": "Validate Manual: [Table - get] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE (After update)",
      "status": "skipped"
    },
    {
      "name": "Validate Manual: [Table - delete] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE",
      "status": "passed"
    },
    {
      "name": "Validate Manual: [Dataset - delete] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144",
      "status": "passed"
    }
    ]
  }
}

Delete an integration

curl -X 'DELETE' \
    'https://www.organization.immuta.com/integrations/{id}' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: 846e9e43c86a4ct1be14290d95127d13f'

  1. Copy the request example.

  2. Replace the Immuta URL and API key with your own.

  3. Replace the {id} request parameter with the unique identifier of the integration you want to delete.

Response

The response returns the status of the Google BigQuery integration configuration that has been deleted. See the response schema reference for details about the response schema. An unsuccessful request returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.

{
  "id": "123456789",
  "status": "deleting",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "Basic Validation: Verify service account not being used for data source connection credentials",
      "status": "passed"
    },
    {
      "name": "Basic Validation: Immuta Service Account postfix",
      "status": "passed"
    },
    {
      "name": "Basic Validation: Non-matching service account in key file",
      "status": "passed"
    },
    {
      "name": "Basic Validation: Connection can be made to BigQuery",
      "status": "passed"
    }
    ]
  }
}
PreviousConfigure a Databricks Unity Catalog IntegrationNextConfigure a Redshift Integration

Last updated

Other versions

SaaS2024.32024.1

Copyright © 2014-2024 Immuta Inc. All rights reserved.