Private preview: This integration is available to select accounts. Reach out to your Immuta representative for details.
The Google BigQuery resource allows you to create, configure, and manage your Google BigQuery integration. In this integration, Immuta generates policy-enforced views in your configured Google BigQuery dataset for tables registered as Immuta data sources.
Google BigQuery integration enabled in Immuta (work with your Immuta representative to enable this integration)
To execute the Immuta script from your command line to create a Google Cloud service account and role, you must be authenticated to the gcloud CLI utility as a user with all of the following roles:
role is the Google Cloud role used to connect to Google BigQuery.
datasetSuffix is the suffix to postfix to the name of each dataset created to store secure views. This string must start with an underscore.
dataset is the name of the BigQuery dataset to provision inside of the project for Immuta metadata storage.
location is the dataset's location, which can be any valid GCP location (such as us-east1).
credential is the Google BigQuery service account JSON keyfile credential content. See the Google documentation for guidance on generating and downloading this keyfile.
See the config object description for parameter definitions, value types, and additional configuration options.
Response
The response returns the status of the Google BigQuery integration configuration connection. See the response schema reference for details about the response schema.
A successful response includes the validation tests statuses.
{"id":"123456789","status":"creating","validationResults": {"status":"passed","validationTests": [ {"name":"Basic Validation: Verify service account not being used for data source connection credentials","status":"passed" }, {"name":"Basic Validation: Immuta Service Account postfix","status":"passed" }, {"name":"Basic Validation: Non-matching service account in key file","status":"passed" }, {"name":"Basic Validation: Connection can be made to BigQuery","status":"passed" }, {"name":"Initialize Validation: [Dataset - exists] innate-conquest-123456.immuta","status":"passed" }, {"name":"Validate Manual: [Dataset - create] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144","status":"passed" }, {"name":"Validate Manual: [Table - create] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE","status":"passed" }, {"name":"Validate Manual: [Table - insert] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE","status":"warning", "message": "Billing has not been enabled for this project. Enable billing at https://console.cloud.google.com/billing. DML queries are not allowed in the free tier. Set up a billing account to remove this restriction.. Confirm that the bootstrap script was executed without any modifications. This warning will not prevent the connection but continue at your own risk as some features may not function."
}, { "name": "Validate Manual: [Table - get] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE (After insert)",
"status":"skipped" }, {"name":"Validate Manual: [Table - update] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE","status":"skipped" }, { "name": "Validate Manual: [Table - get] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE (After update)",
"status":"skipped" }, {"name":"Validate Manual: [Table - delete] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE","status":"passed" }, {"name":"Validate Manual: [Dataset - delete] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144","status":"passed" } ] }}
An unsuccessful request returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.
{"statusCode":409,"error":"Conflict","message":"Google BigQuery integration already exists on host organization.cloud.google.com (id = 123456789)"}
Replace the {id} request parameter with the unique identifier of the integration you want to get. Alternatively, you can get a list of all integrations and their IDs with the GET /integrations endpoint.
Response
The response returns a Google BigQuery integration configuration. See the response schema reference for details about the response schema. An unsuccessful response returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.
{"id":"123456789","status":"enabled","validationResults": {"status":"passed","validationTests": [ {"name":"Basic Validation: Verify service account not being used for data source connection credentials","status":"passed" }, {"name":"Basic Validation: Immuta Service Account postfix","status":"passed" }, {"name":"Basic Validation: Non-matching service account in key file","status":"passed" }, {"name":"Basic Validation: Connection can be made to BigQuery","status":"passed" }, {"name":"Initialize Validation: [Dataset - exists] innate-conquest-123456.immuta","status":"passed" }, {"name":"Validate Manual: [Dataset - create] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144","status":"passed" }, {"name":"Validate Manual: [Table - create] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE","status":"passed" }, {"name":"Validate Manual: [Table - insert] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE","status":"warning", "message": "Billing has not been enabled for this project. Enable billing at https://console.cloud.google.com/billing. DML queries are not allowed in the free tier. Set up a billing account to remove this restriction.. Confirm that the bootstrap script was executed without any modifications. This warning will not prevent the connection but continue at your own risk as some features may not function."
}, { "name": "Validate Manual: [Table - get] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE (After insert)",
"status":"skipped" }, {"name":"Validate Manual: [Table - update] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE","status":"skipped" }, { "name": "Validate Manual: [Table - get] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE (After update)",
"status":"skipped" }, {"name":"Validate Manual: [Table - delete] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE","status":"passed" }, {"name":"Validate Manual: [Dataset - delete] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144","status":"passed" } ] },"type":"Google BigQuery","autoBootstrap":false,"config": {"port":443,"role":"immuta","datasetSuffix":"_secureView","dataset":"immuta","location":"us-east1","credential": {"type":"service_account","project_id":"innate-conquest-123456","private_key_id":"9163c12345690924f5dd218ff39", "private_key": "-----BEGIN PRIVATE KEY-----\nXXXXXXXro0s\n/yQlPQijowkccmrmWJyr93kdLnwJzBvLHCto/+W\ncvF2ygX9oM/dyUK//z\//4nptMp+Ck//Yw3D4rIBwGu4DWiR1qRnf\nDoGyXfThPTQ==\n-----END PRIVATE KEY-----\n",
"client_email":"service-account-id@innate-conquest-123456.iam.gserviceaccount.com","client_id":"1166290***432952487857","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-accound-id%40innate-conquest-123456.iam.gserviceaccount.com",
"universe_domain":"googleapis.com" } } }
The response returns the configuration for all integrations. See the response schema reference for details about the response schema. An unsuccessful request returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.
Replace the {id} request parameter with the unique identifier of the integration you want to update.
Change the config values to your own, where
role is the Google Cloud role used to connect to Google BigQuery.
datasetSuffix is the suffix to postfix to the name of each dataset created to store secure views. This string must start with an underscore.
dataset is the name of the BigQuery dataset to provision inside of the project for Immuta metadata storage.
location is the dataset's location, which can be any valid GCP location (such as us-east1).
credential is the Google BigQuery service account JSON keyfile credential content. See the Google documentation for guidance on generating and downloading this keyfile.
See the config object description for parameter definitions, value types, and additional configuration options.
Response
The response returns the status of the Google BigQuery integration configuration connection. See the response schema reference for details about the response schema.
A successful response includes the validation tests statuses.
{"id":"123456789","status":"editing","validationResults": {"status":"passed","validationTests": [ {"name":"Basic Validation: Verify service account not being used for data source connection credentials","status":"passed" }, {"name":"Basic Validation: Immuta Service Account postfix","status":"passed" }, {"name":"Basic Validation: Non-matching service account in key file","status":"passed" }, {"name":"Basic Validation: Connection can be made to BigQuery","status":"passed" }, {"name":"Initialize Validation: [Dataset - exists] innate-conquest-123456.immuta","status":"passed" }, {"name":"Validate Manual: [Dataset - create] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144","status":"passed" }, {"name":"Validate Manual: [Table - create] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE","status":"passed" }, {"name":"Validate Manual: [Table - insert] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE","status":"warning", "message": "Billing has not been enabled for this project. Enable billing at https://console.cloud.google.com/billing. DML queries are not allowed in the free tier. Set up a billing account to remove this restriction.. Confirm that the bootstrap script was executed without any modifications. This warning will not prevent the connection but continue at your own risk as some features may not function."
}, { "name": "Validate Manual: [Table - get] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE (After insert)",
"status":"skipped" }, {"name":"Validate Manual: [Table - update] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE","status":"skipped" }, { "name": "Validate Manual: [Table - get] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE (After update)",
"status":"skipped" }, {"name":"Validate Manual: [Table - delete] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144.IMMUTA_VERIFY_TABLE","status":"passed" }, {"name":"Validate Manual: [Dataset - delete] innate-conquest-123456.IMMUTA_VERIFY_DATASET_144","status":"passed" } ] }}
An unsuccessful request returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.
{"statusCode":409,"error":"Conflict","message":"Unable to edit integration with ID 123456789 in current state editing."}
Replace the {id} request parameter with the unique identifier of the integration you want to delete.
Response
The response returns the status of the Google BigQuery integration configuration that has been deleted. See the response schema reference for details about the response schema. An unsuccessful request returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.
{"id":"123456789","status":"deleting","validationResults": {"status":"passed","validationTests": [ {"name":"Basic Validation: Verify service account not being used for data source connection credentials","status":"passed" }, {"name":"Basic Validation: Immuta Service Account postfix","status":"passed" }, {"name":"Basic Validation: Non-matching service account in key file","status":"passed" }, {"name":"Basic Validation: Connection can be made to BigQuery","status":"passed" } ] }}
