Requirements

• Snowflake Enterprise Edition

• Snowflake X-Large or Large warehouse is strongly recommended

Create Snowflake data sources

1. Set the default subscription policy to None for bulk data source creation. This will simplify the data source creation process by not automatically applying policies.

2. Make a request to the Immuta V2 API create data source endpoint, as the Immuta UI does not support creating more than 1000 data sources. The following options must be specified in your request to ensure the maximum performance benefits of bulk data source creation. The Skip Stats Job tag is only required if you are using specific policies that require stats; otherwise, Snowflake data sources automatically skip the stats job.

   Copy

   "options": {
       "disableSensitiveDataDiscovery": true,
       "tableTags": [
           "Skip Stats Job"
       ]
   }

Specifying disableSensitiveDataDiscovery as true ensures that sensitive data discovery will not be applied when the new data sources are created in Immuta, regardless of how it is configured for the Immuta tenant. Disabling sensitive data discovery improves performance during data source creation.

Applying the Skip Stats Job tag using the tableTag value will ensure that some jobs that are not vital to data source creation are skipped, specifically the fingerprint and high cardinality check jobs.

When the Snowflake bulk data source creation feature is configured, the create data source endpoint operates asynchronously and responds immediately with a bulkId that can be used for monitoring progress.

Monitor progress

To monitor the progress of the background jobs for the bulk data source creation, make the following request using the bulkId from the response of the previous step:

curl \
    --request POST \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    --data @example_payload.json
    https://your-immuta-url.com/jobs?bulkId=<your-bulkId>

The response will contain a list of job states and the number of jobs currently in each state. If errors were encountered during processing, a list of errors will be included in the response:

    {
      "total":"99893",
      "completed":"99892",
      "failed":"0",
      "pending":"1",
      "errors":null
    }

With these recommended configurations, bulk creating 100,000 Snowflake data sources will take between six and seven hours for all associated jobs to complete.

Requirements

• CREATE_DATA_SOURCE Immuta permission

• Google BigQuery roles:

  • roles/bigquery.metadataViewer on the source table (if managed at that level) or dataset

  • roles/bigquery.dataViewer (or higher) on the source table (if managed at that level) or dataset

  • roles/bigquery.jobUser on the project

Prerequisites

• Configure the Google BigQuery integration

• Upload the Google BigQuery ODBC driver

Create a Google Cloud service account for creating Google BigQuery data sources

Google BigQuery data sources in Immuta must be created using a Google Cloud service account rather than a Google Cloud user account. If you do not currently have a service account for the Google Cloud project separate from the Google Cloud service account you created when configuring the Google BigQuery integration, you must create a Google Cloud service account with privileges to view and run queries against the tables you are protecting.

You have two options to create the required Google Cloud service account:

• Create a service account by using Google Cloud Console.

• Create a service account by using gcloud.

Create a service account using the Google Cloud web console

1. Using the Google Cloud documentation, create a service account with the following roles:

   • BigQuery User

   • BigQuery Data Viewer

2. Using the Google Cloud documentation, generate a service account key for the account you just created.

Create a service account using gcloud

1. Copy the script below and update the SERVICE_ACCOUNT, PROJECT_ID, and IMMUTA_GCP_KEY_FILE values.

   • SERVICE_ACCOUNT is the name for the new service account.

   • PROJECT_ID is the project ID for the Google Cloud Project that is integrated with Immuta.

   • IMMUTA_GCP_KEY_FILE is the path to a new output file for the private key.

2. Use the script below in the gcloud command line. This script is a template; change values as necessary:

   Copy

   # Fill these out
   # Please use .json extension for key
   export SERVICE_ACCOUNT=datasource-account
   export PROJECT_ID=project123
   export IMMUTA_GCP_KEY_FILE=~/GCP_${SERVICE_ACCOUNT}_key.json
   
   # Create service account for creating data sources
   gcloud iam service-accounts create ${SERVICE_ACCOUNT} --project ${PROJECT_ID}
   
   # Generate keyfile
   gcloud iam service-accounts keys create ${IMMUTA_GCP_KEY_FILE} --iam-account=${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com
   
   # Allow account to execute queries
   #gcloud projects add-iam-policy-binding ${PROJECT_ID} \
   #--member="serviceAccount:${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com" --role=projects/${PROJECT_ID}/roles/bigquery.user
   gcloud projects add-iam-policy-binding ${PROJECT_ID} \
   --member="serviceAccount:${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com" --role=roles/bigquery.user
   
   # Allow account to view data
   gcloud projects add-iam-policy-binding ${PROJECT_ID} \
   --member="serviceAccount:${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com" --role=roles/bigquery.dataViewer
   
   echo if something went wrong and you want to delete the service account, run:
   echo gcloud iam service-accounts delete ${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com --project ${PROJECT_ID}

Register data sources in Immuta

1. Click the + button in the top-left corner of the screen and select New Data Source.

2. Select the Google BigQuery tile in the Data Platform section.

3. Complete these fields in the Connection Information box:

   • Account Email Address: Enter the email address of a user with access to the dataset and tables. This is the account created in the Google BigQuery configuration guide.

   • Project: Enter the name of the project that has been integrated with Immuta.

   • Dataset: Enter the name of the dataset with the tables you want Immuta to ingest.

4. Upload a BigQuery Key File in the modal. Note that the account in the key file must match the account email address entered in the previous step.

5. Click the Test Connection button. If the connection is successful, a check mark and successful connection notification will appear and you will be able to proceed. If an error occurs when attempting to connect, the error will be displayed in the UI. In order to proceed to the next step of data source creation, you must be able to connect to this data source using the connection information that you just entered.

6. Decide how to virtually populate the data source by selecting one of the options:

   • Create sources for all tables in this database: This option will create data sources and keep them in sync for every table in the dataset. New tables will be automatically detected and new Immuta views will be created.

   • Schema / Table: This option will allow you to specify tables or datasets that you want Immuta to register.

7. Provide basic information about your data source to make it discoverable to users.

   • Enter the SQL Schema Name Format to be the SQL name that the data source exists under in Immuta. For BigQuery the schema will be the BigQuery dataset. The format must include a schema macro but you may personalize it using lowercase letters, numbers, and underscores to personalize the format. It can have up to 255 characters.

   • Enter the Schema Project Name Format to be the name of the schema project in the Immuta UI. This is an Immuta project that will hold all of the metadata for the tables in a single dataset.

     • When selecting Create sources for all tables in this database and monitor for changes, you may personalize this field as you wish, but it must include a schema macro to represent the dataset name.

     • When selecting Schema/Table, this field is pre-populated with the recommended project name and you can edit freely.

   • Select the Data Source Name Format, which will be the format of the name of the data source in the Immuta UI.

     • <Tablename>: The Immuta data source will have the same name as the original table.

     • <Schema><Tablename>: The Immuta data source will have both the dataset and original table name.

     • Custom: This is a template you create to make the data source name. You may personalize this field as you wish, but it must include a tablename macro. The case of the macro will apply to the data source name (i.e., <Tablename> will result in "Data Source Name," <tablename> will result in "data source name," and <TABLENAME> will result in "DATA SOURCE NAME").

   • Enter the SQL Table Name Format, which will be the format of the name of the table in Immuta. It must include a table name macro, but you may personalize the format using lowercase letters, numbers, and underscores. It may have up to 255 characters.

8. When selecting the Schema/Table option, you can opt to enable schema monitoring by selecting the checkbox in this section. This step will only appear if all tables within a server have been selected for creation.

9. Optional Advanced Settings:

   • Column Detection: To enable, select the checkbox in this section. This setting monitors when remote tables' columns have been changed, updates the corresponding data sources in Immuta, and notifies data owners of these changes. See schema projects overview to learn more about column detection.

   • Data Source Tags: Adding tags to your data source allows users to search for the data source using the tags and governors to apply global policies to the data source. Note if schema detection is enabled, any tags added now will also be added to the tables that are detected.

     • Click the Edit button in the Data Source Tags section.

     • Begin typing in the Search by Tag Name box to select your tag, and then click Add.

10. Click Create to save the data source(s).

Next steps

With data sources registered in Immuta, your organization can now start

• building global subscription and data policies to govern data.

• creating projects to collaborate.

Requirement

CREATE_S3_DATA_SOURCE Immuta permission

Prerequisite

Configure the Amazon S3 integration

Register S3 data

1. Navigate to the My Data Sources page in Immuta.

2. Click New Data Source.

3. Select the Native S3 tile in the data platform section.

4. Select your AWS Account/Region from the dropdown menu.

5. If domains are enabled for your Immuta tenant, opt to select a default domain to which data sources will be assigned.

6. Opt to add default tags to the data sources.

7. Click Next.

8. The prefix field is populated with the base path. Add to this prefix to create a data source for a prefix, bucket, or object.

   • If the data source prefix ends in a wildcard (*), it protects all items starting with that prefix. For example, a base location of s3:// and a data source prefix surveys/2024* would protect paths like s3://surveys/2024-internal/research-dept.txt or s3://surveys/2024-customer/april/us.csv.

   • If the data source prefix ends without a wildcard (*), it protects a single object. For example, a base location path of s3:// and a data source prefix of research-data/demographics would only protect the object that exactly matches s3://research-data/demographics.

9. Click Add Prefix, and then click Next.

10. Verify that your prefixes are correct and click Complete Setup.

For a complete list of supported databases, see the Immuta Support Matrix.

Requirements

• CREATE_DATA_SOURCE Immuta permission

• Snowflake data source requirements:

  • USAGE Snowflake privilege on the schema and database

  • REFERENCES Snowflake privilege on the tables

• Databricks Spark integration requirements: Ensure that at least one of the traits below is true.

  • The user exposing the tables has READ_METADATA and SELECT permissions on the target views/tables (specifically if Table ACLs are enabled).

  • The user exposing the tables is listed in the immuta.spark.acl.whitelist configuration on the target cluster.

  • The user exposing the tables is a Databricks workspace administrator.

• Databricks Unity Catalog integration requirements: When exposing a table from Databricks Unity Catalog, be sure the credentials used to register the data sources have the Databricks privileges listed below.

  • The following privileges on the parent catalogs and schemas of those tables:

    • SELECT

    • USE CATALOG

    • USE SCHEMA

  • USE SCHEMA on system.information_schema

Enter connection information

1. Navigate to the My Data Sources page.

2. Click the New Data Source button in the top right corner.

3. Select the data platform containing the data you wish to expose by clicking a tile.

4. Input the connection parameters to the database you're exposing. Click the tabs below for guidance for select data platforms.

1. Click the Test Connection button.

Select virtual population

1. Decide how to virtually populate the data source by selecting Create sources for all tables in this database and monitor for changes or Schema/Table.

2. Complete the workflow for Create sources for all tables in this database and monitor for changes or Schema/Table selection, which are outlined on the tabs below:

Enter basic information

Provide information about your source to make it discoverable to users.

1. Enter the SQL Schema Name Format to be the SQL name that the data source exists under in the Immuta Query Engine. It must include a schema macro but you may personalize it using lowercase letters, numbers, and underscores to personalize the format. It may have up to 255 characters.

2. Enter the Schema Project Name Format to be the name of the schema project in the Immuta UI. If you enter a name that already exists, the name will automatically be incremented. For example, if the schema project Customer table already exists and you enter that name in this field, the name for this second schema project will automatically become Customer table 2 when you create it.

   1. When selecting Create sources for all tables in this database and monitor for changes you may personalize this field as you wish, but it must include a schema macro.

   2. When selecting Schema/Table this field is prepopulated with the recommended project name and you can edit freely.

3. Select the Data Source Name Format, which will be the format of the name of the data source in the Immuta UI.

1. Enter the SQL Table Name Format, which will be the format of the name of the table in Immuta. It must include a table name macro, but you may personalize the format using lowercase letters, numbers, and underscores. It may have up to 255 characters.

Enable or disable schema monitoring

When selecting the Schema/Table option you can opt to enable Schema Monitoring by selecting the checkbox in this section.

Note: This step will only appear if all tables within a server have been selected for creation.

Create a schema detection job in Databricks

In most cases, Immuta’s schema detection job runs automatically from the Immuta web service. For Databricks, that automatic job is disabled because of the ephemeral nature of Databricks clusters. In this case, Immuta requires users to download a schema detection job template (a Python script) and import that into their Databricks workspace.

1. Enable Schema Monitoring or Detect Column Changes on the Data Source creation page.

2. Click Download Schema Job Detection Template.

3. Click the Click Here To Download text.

4. Before you can run the script, create the correct scope and secret by running these commands in the CLI using the Immuta API Key generated on your user profile page:

   Copy

       databricks secrets create-scope --scope auth
       databricks secrets put --scope auth --key apikey

5. Import the Python script you downloaded into a Databricks workspace as a notebook. Note: The job template has commented out lines for specifying a particular database or table. With those two lines commented out, the schema detection job will run against ALL databases and tables in Databricks. Additionally, if you need to add proxy configuration to the job template, the template uses the Python requests library, which has a simple mechanism for configuring proxies for a request.

6. Schedule the script as part of a notebook job to run as often as required. Each time the job runs, it will make an API call to Immuta to trigger schema detection queries, and these queries will run on the cluster from which the request was made. Note: Use the api_immuta cluster for this job. The job in Databricks must use an Existing All-Purpose Cluster so that Immuta can connect to it over ODBC. Job clusters do not support ODBC connections.

Create the data source

Opt to configure settings in the Advanced Options section (outlined below), and then click Create to save the data source(s).

Advanced options

None of the following options are required. However, completing these steps will help maximize the utility of your data source.