# Data Source Payload Attribute Details ## `connectionKey` The `connectionKey` is a unique identifier for the collection of data sources being created. If an existing `connectionKey` is used with new connection information, it will delete the old data sources and create new ones from the new information in the payload. ## `connection` {% tabs %} {% tab title="Snowflake data source" %} | Attribute | Description | Required or optional | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- | | handler | `Snowflake` | **Required** | | ssl | `boolean` Set to `true` to enable SSL communication with the remote database. | Optional | | database | `string` The database name. | **Required** | | schema | `string` The schema in the remote database. | Optional | | hostname | `string` The hostname of the remote database instance. | **Required** | | port | `number` The port of the remote database instance. | Optional | | warehouse | `string` The default pool of compute resources Immuta will use to run queries and other Snowflake operations. | **Required** | | connectionStringOptions | `string` Additional connection string options to be used when connecting to the remote database. | Optional | | authenticationMethod | `string` The type of authentication method to use. Options include `userPassword`, `keyPair`, and `oAuthClientCredentials`. | **Required** | | username | `string` The username used to connect to the remote database. | Required if using `userPassword` or `keyPair`. | | password | `string` The password used to connect to the remote database. | Required if using `userPassword`. | | useCertificate | `boolean` Set to `true` when using client certificate credentials to request an access token. Otherwise, set to `false` to use client secret. | Required if using `oAuthClientCredentials`. | | userFiles | `object` Details about the files required for the request. | Required if using `keyPair` or `oAuthClientCredentials` with `useCertificate` set to `true`. | | keyName | `string` The connection name of the key file. Must be `PRIV_KEY_FILE` if using `keyPair`, or must be `oauth client certificate` if using `oAuthClientCredentials`. | Required if using `keyPair` or `oAuthClientCredentials` with `useCertificate` set to `true`. | | content | `string` The content of the file, base-64 encoded. | Required if using `keyPair` or `oAuthClientCredentials` with `useCertificate` set to `true`. | | userFilename | `string` The name of the file - for display purposes in the UI. | Required if using `keyPair` or `oAuthClientCredentials` with `useCertificate` set to `true`. | | {% endtab %} | | | {% tab title="Databricks data source" %} | Attribute | Description | Required or optional | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | | handler | `Databricks` | **Required** | | ssl | `boolean` Set to `true` to enable SSL communication with the remote database. | Optional | | database | `string` The database name. | Optional | | hostname | `string` The hostname of the remote database instance. | **Required** | | port | `number` The port of the remote database instance. | Optional | | connectionStringOptions | `string` Additional connection string options to be used when connecting to the remote database. | Optional | | authenticationMethod | `string` The type of authentication method to use. Options include `oAuthM2M` and `token`. | **Required** | | token | `string` The Databricks personal access token for the service principal created for Immuta. | Required if using `token` authentication. | | useCertificate | `boolean` True when using client certificate credentials to request an access token. Otherwise, client secret. | Required if using `oAuthM2M`. | | clientId | `string` The client identifier of the Immuta service principal you configured. This is the client ID displayed in Databricks when creating the client secret for the service principal. | Required if using `oAuthM2M`. | | audience | `string` The audience for the OAuth Client Credential token request. | Required if using `oAuthM2M`. | | clientSecret | `string` An application password an app can use in place of a certificate to identity itself. | Required if using `oAuthM2M` and `useCertificate` is set to `false`. | | certificateThumbprint | `string` The certificate thumbprint to use to generate the JWT for the OAuth Client Credential request. | Required if using `oAuthM2M` and `useCertificate` is set to `true`. | | scope | `clientSecret` The scope limits the operations and roles allowed in Databricks by the access token. See the [OAuth 2.0 documentation](https://oauth.net/2/scope/) for details about scopes. | Optional | | httpPath | `string` The HTTP path of your Databricks cluster or SQL warehouse. | **Required** | | {% endtab %} | | | {% tab title="Redshift data source" %} | Attribute | Description | Required or optional | | ----------------------- | ------------------------------------------------------------------------------------------------ | ------------------------- | | handler | `Redshift` | **Required** | | ssl | `boolean` Set to `true` to enable SSL communication with the remote database. | Optional | | database | `string` The database name. | Optional | | schema | `string` The schema in the remote database. | **Required** | | connectionStringOptions | `string` Additional connection string options to be used when connecting to the remote database. | Optional | | hostname | `string` The hostname of the remote database instance. | **Required** | | port | `number` The port of the remote database instance. | Optional | | authenticationMethod | `string` The type of authentication method to use. Options include `userPassword` and `okta`. | **Required** | | username | `string` The username used to connect to the remote database. | **Required** | | password | `string` The password used to connect to the remote database. | **Required** | | idpHost | `string` The Okta identity provider host URL. | Required if using `okta`. | | appID | `string` The Okta application ID. | Required if using `okta`. | | role | `string` The Okta role. | Required if using `okta`. | | {% endtab %} | | | {% tab title="Other data sources" %} | Attribute | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | handler | `Google BigQuery`, `Presto`, and `Trino` | | ssl | `boolean` Set to `true` to enable SSL communication with the remote database. | | database | `string` The database name. | | schema | `string` The schema in the remote database. | | userFiles | `array` Array of objects; each object must have `keyName` (corresponds to a connection string option), `content` (base-64 encoded content), and `userFilename` (the name of the file - for display purposes in the app). | | connectionStringOptions | `string` Additional connection string options to be used when connecting to the remote database. | | hostname | `string` The hostname of the remote database instance. | | port | `number` The port of the remote database instance. | | authenticationMethod | `string` The type of authentication method to use. Options include `userPassword`, `keyPair`, `oAuthClientCredentials`, `token`, `oAuthM2M`, `keyFile`, `auto` and `okta`. | | username | `string` The username used to connect to the remote database. | | password | `string` The password used to connect to the remote database. | | sid | `string` For Google BigQuery, the BigQuery project used to build the connection string. | | {% endtab %} | | | {% endtabs %} | | ### Special Cases * **Athena**: Also requires `region` and `queryResultLocationBucket`. `queryResultLocationDirectory` is optional. `authenticationMethod` can be `none`, `accessKey` (default: username = access key, password = secret key), or `instanceRole`. * **BigQuery**: Does not require `hostname` and `password`. Requires `sid`, which is the GCP project ID, and `userFiles` with the `keyName` of `KeyFilePath` and the base64-encoded `keyfile.json`. * **Trino**: `authenticationMethod` can be `No Authentication`, `LDAP Authentication`, or `Kerberos Authentication`. ## `nameTemplate` | Attribute | Description | | ----------------------- | ----------------------------------------------------------------------------------- | | dataSourceFormat | `string` Format to be used to name the data sources created in this group. | | schemaFormat | `string` Format to be used to name the Immuta schema created in this group. | | tableFormat | `string` Format to be used to name the Immuta table created in this group. | | schemaProjectNameFormat | `string` Format to be used to name the Immuta schema project created in this group. | Available templates include * `` * `` * `` All cases of the name in Immuta should be lowercase. For example, consider a table `TPC.CUSTOMER` that is given the following `nameTemplate`: ```yaml dataSourceFormat: tableFormat: schemaFormat: schemaProjectNameFormat: ``` This `nameTemplate` will produce a data source named `tpc.customer` in a schema project named `tpc`. ## `options` | Attribute | Description | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | staleDataTolerance | `integer` The length in seconds that data for these sources can be cached. | | disableSensitiveDataDiscovery | `boolean` If true, Immuta will not perform [sensitive data discovery](https://documentation.immuta.com/2024.2/discover-your-data/data-discovery). Default: `false`. | | domainCollectionId | `string` The ID of the domain to assign the data sources to. Use the [GET /collection endpoint](https://documentation.immuta.com/2024.2/developer-guides/immuta-v1-api/manage-data-access/domains-api#get-collection) to retrieve domains and domain IDs. | | hardDelete | `boolean` If `true`, when the table backing the data source is no longer available, the data source in Immuta is deleted. If this is `false`, the data source will be disabled. Default: `false`. | | tableTags | `array` An array of tags (strings) to place at the data source level on every data source. | ## `owners` | Attribute | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | type | `group or user` The type of owner that is being added. | | name | `string` The name of the group or the user (username they log in with). | | iam (optional) | `string` The ID of the identity manager system the user or group comes from. If excluded, any user/group that matches will be added as an owner. | ## `sources` {% hint style="info" %} **Best practice:** Use Subscription Policies to Control Access If you are not tagging individual columns, omit `sources` to create data sources for all tables in the schema or database, and then use Subscription Policies to control access to the tables instead of excluding them from Immuta. {% endhint %} This attribute configures which sources are created. If `sources` is not provided, all sources from the given connection will be created. There are 3 types of sources than can be specified: * [all tables](#recommended-specify-all-tables) * [query](#specify-a-query) * [table](#specify-a-table) ### Recommended: Specify All Tables If you specify any sources (either tables or queries), but you still want to create data sources for the rest of the tables in the schema or database, you can specify `all` as a source: ```yaml sources: - all: true ``` {% hint style="info" %} **Best practice**: Use schema monitoring Excluding sources or specifying `all: true` will turn on automatic schema monitoring in Immuta. As tables are added or removed, Immuta will look for those changes on a schedule (by default, once a day) and either disable or delete data sources for removed tables or create data sources for new tables. New tables will be tagged `New` so that you can build a policy to restrict access to new tables until they are evaluated by data owners. Data owners will be notified of new tables, and all subscribers will be notified if data sources are disabled or deleted. {% endhint %} ### Specify a Query Immuta recommends creating a view in your remote database instead of using this option, but if that is not possible, you can create data sources based on SQL statements: ```yaml sources: - query: “select * from table” naming: datasource: “My Source”, table: “my_source”, schema: “queries” ``` ### Specify a Table If you want to select specific tables to be created as data sources, or if you want to tag individual data sources or columns within a data source, you need to leverage this parameter: ```yaml sources: - table: name_of_table schema: name_of_schema ``` ### Additional Options When specifying a table or query there are other options that can be specified: | Option | Description | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | columnDescriptions | [See details below](#column-descriptions). | | description | A short description for the data source. | | documentation | Markdown-supported documentation for the data source. | | naming | See the example above in [Specify a Query](#specify-a-query). This is required for query-based sources, but is optional for table-based sources and can be used to override the `nameTemplate` provided for the whole database/schema. | | owners | Specify owners for an individual data source. The payload is the same as owners at the root level. | | tags | [See details below](#tags). | #### Columns * If any columns are specified, those are the only columns that will be available in the data source. * If no columns are specified, Immuta will look for new or removed columns on a schedule (by default, once a day) and add or remove columns from the data sources automatically as needed. * New columns will be tagged `New`, so you can build a policy to automatically mask new columns until they are approved. * Data Owners will be notified when columns are added or removed. `columns` is an array of objects for each column: | Attribute | Description | | ----------- | ------------------------------------------------------------ | | name | The column name. | | dataType | The data type. | | nullable | Whether or not the column contains `null`. | | remoteType | The actual data type in the remote database. | | primaryKey | Specify whether this is the primary key of the remote table. | | description | Describe the column. | #### Column Descriptions You can add descriptions to columns without having to specify all the columns in the data source. `columnDescriptions` is an array of objects with the following schema: | Attribute | Description | | ----------- | --------------------------------------- | | columnName | `string` The column name. | | description | `string` The description of the column. | ```yaml columnDescriptions: - columnName: acct_num description: The account number ``` #### Tags You can add tags to columns or data sources. `tags` is an object with the following schema: | Attribute | Description | | --------- | ------------------------------------------------------------------------------------------- | | table | `array` An array of tags (strings) to add to this table. | | columns | `array` An array of objects that specifies columnName (string) and tags (an array to tags). | ```yaml tags: table: - Sensitive - Marketing columns: - columnName: acct_num tags: - unique_id ```