connectionKeyThe connectionKey is a unique identifier for the collection of data sources being created. If an existing connectionKey is used with new connection information, it will delete the old data sources and create new ones from the new information in the payload.
connectionAthena: Also requires region and queryResultLocationBucket. queryResultLocationDirectory is optional. authenticationMethod can be none, accessKey (default: username = access key, password = secret key), or instanceRole.
BigQuery: Does not require hostname
nameTemplateAvailable templates include
<tablename>
<schema>
<database>
All cases of the name in Immuta should be lowercase.
For example, consider a table TPC.CUSTOMER that is given the following nameTemplate:
This nameTemplate will produce a data source named tpc.customer in a schema project named tpc.
optionsownerssourcesThis attribute configures which sources are created. If sources is not provided, all sources from the given connection will be created.
There are 3 types of sources than can be specified:
If you specify any sources (either tables or queries), but you still want to create data sources for the rest of the tables in the schema or database, you can specify all as a source:
Immuta recommends creating a view in your remote database instead of using this option, but if that is not possible, you can create data sources based on SQL statements:
If you want to select specific tables to be created as data sources, or if you want to tag individual data sources or columns within a data source, you need to leverage this parameter:
When specifying a table or query there are other options that can be specified:
If any columns are specified, those are the only columns that will be available in the data source.
If no columns are specified, Immuta will look for new or removed columns on a schedule (by default, once a day) and add or remove columns from the data sources automatically as needed.
New columns will be tagged New, so you can build a policy to automatically mask new columns until they are approved.
Data Owners will be notified when columns are added or removed.
columns is an array of objects for each column:
You can add descriptions to columns without having to specify all the columns in the data source. columnDescriptions is an array of objects with the following schema:
You can add tags to columns or data sources. tags is an object with the following schema:
Optional
hostname
string The hostname of the remote database instance.
Required
port
number The port of the remote database instance.
Optional
warehouse
string The default pool of compute resources Immuta will use to run queries and other Snowflake operations.
Required
connectionStringOptions
string Additional connection string options to be used when connecting to the remote database.
Optional
authenticationMethod
string The type of authentication method to use. Options include userPassword, keyPair, and oAuthClientCredentials.
Required
username
string The username used to connect to the remote database.
Required if using userPassword or keyPair.
password
string The password used to connect to the remote database.
Required if using userPassword.
useCertificate
boolean Set to true when using client certificate credentials to request an access token. Otherwise, set to false to use client secret.
Required if using oAuthClientCredentials.
userFiles
object Details about the files required for the request.
Required if using keyPair or oAuthClientCredentials with useCertificate set to true.
keyName
string The connection name of the key file. Must be PRIV_KEY_FILE if using keyPair, or must be oauth client certificate if using oAuthClientCredentials.
Required if using keyPair or oAuthClientCredentials with useCertificate set to true.
content
string The content of the file, base-64 encoded.
Required if using keyPair or oAuthClientCredentials with useCertificate set to true.
userFilename
string The name of the file - for display purposes in the UI.
Required if using keyPair or oAuthClientCredentials with useCertificate set to true.
handler
Databricks
Required
ssl
boolean Set to true to enable SSL communication with the remote database.
Optional
database
string The database name.
Optional
hostname
handler
Redshift
Required
ssl
boolean Set to true to enable SSL communication with the remote database.
Optional
database
string The database name.
Optional
schema
handler
Google BigQuery, Presto, and Trino
ssl
boolean Set to true to enable SSL communication with the remote database.
database
string The database name.
schema
string The schema in the remote database.
userFiles
array Array of objects; each object must have keyName (corresponds to a connection string option), content (base-64 encoded content), and userFilename (the name of the file - for display purposes in the app).
connectionStringOptions
passwordsiduserFileskeyNameKeyFilePathkeyfile.jsonTrino: authenticationMethod can be No Authentication, LDAP Authentication, or Kerberos Authentication.
handler
Snowflake
Required
ssl
boolean Set to true to enable SSL communication with the remote database.
Optional
database
string The database name.
Required
schema
dataSourceFormat
string Format to be used to name the data sources created in this group.
schemaFormat
string Format to be used to name the Immuta schema created in this group.
tableFormat
string Format to be used to name the Immuta table created in this group.
schemaProjectNameFormat
string Format to be used to name the Immuta schema project created in this group.
staleDataTolerance
integer The length in seconds that data for these sources can be cached.
disableSensitiveDataDiscovery
boolean If true, Immuta will not perform sensitive data discovery. Default: false.
domainCollectionId
string The ID of the domain to assign the data sources to. Use the GET /collection endpoint to retrieve domains and domain IDs.
hardDelete
boolean If true, when the table backing the data source is no longer available, the data source in Immuta is deleted. If this is false, the data source will be disabled. Default: false.
tableTags
array An array of tags (strings) to place at the data source level on every data source.
type
group or user The type of owner that is being added.
name
string The name of the group or the user (username they log in with).
iam (optional)
string The ID of the identity manager system the user or group comes from. If excluded, any user/group that matches will be added as an owner.
columnDescriptions
description
A short description for the data source.
documentation
Markdown-supported documentation for the data source.
naming
See the example above in Specify a Query. This is required for query-based sources, but is optional for table-based sources and can be used to override the nameTemplate provided for the whole database/schema.
owners
Specify owners for an individual data source. The payload is the same as owners at the root level.
tags
name
The column name.
dataType
The data type.
nullable
Whether or not the column contains null.
remoteType
The actual data type in the remote database.
primaryKey
Specify whether this is the primary key of the remote table.
description
Describe the column.
columnName
string The column name.
description
string The description of the column.
table
array An array of tags (strings) to add to this table.
columns
array An array of objects that specifies columnName (string) and tags (an array to tags).
string The schema in the remote database.
dataSourceFormat: <schema> <tablename>
tableFormat: <tablename>
schemaFormat: <schema>
schemaProjectNameFormat: <schema>sources:
- all: truesources:
- query: “select * from table”
naming:
datasource: “My Source”,
table: “my_source”,
schema: “queries”sources:
- table: name_of_table
schema: name_of_schemacolumnDescriptions:
- columnName: acct_num
description: The account numbertags:
table:
- Sensitive
- Marketing
columns:
- columnName: acct_num
tags:
- unique_idstring The hostname of the remote database instance.
Required
port
number The port of the remote database instance.
Optional
connectionStringOptions
string Additional connection string options to be used when connecting to the remote database.
Optional
authenticationMethod
string The type of authentication method to use. Options include oAuthM2M and token.
Required
token
string The Databricks personal access token for the service principal created for Immuta.
Required if using token authentication.
useCertificate
boolean True when using client certificate credentials to request an access token. Otherwise, client secret.
Required if using oAuthM2M.
clientId
string The client identifier of the Immuta service principal you configured. This is the client ID displayed in Databricks when creating the client secret for the service principal.
Required if using oAuthM2M.
audience
string The audience for the OAuth Client Credential token request.
Required if using oAuthM2M.
clientSecret
string An application password an app can use in place of a certificate to identity itself.
Required if using oAuthM2M and useCertificate is set to false.
certificateThumbprint
string The certificate thumbprint to use to generate the JWT for the OAuth Client Credential request.
Required if using oAuthM2M and useCertificate is set to true.
scope
clientSecret The scope limits the operations and roles allowed in Databricks by the access token. See the OAuth 2.0 documentation for details about scopes.
Optional
httpPath
string The HTTP path of your Databricks cluster or SQL warehouse.
Required
string The schema in the remote database.
Required
connectionStringOptions
string Additional connection string options to be used when connecting to the remote database.
Optional
hostname
string The hostname of the remote database instance.
Required
port
number The port of the remote database instance.
Optional
authenticationMethod
string The type of authentication method to use. Options include userPassword and okta.
Required
username
string The username used to connect to the remote database.
Required
password
string The password used to connect to the remote database.
Required
idpHost
string The Okta identity provider host URL.
Required if using okta.
appID
string The Okta application ID.
Required if using okta.
role
string The Okta role.
Required if using okta.
string Additional connection string options to be used when connecting to the remote database.
hostname
string The hostname of the remote database instance.
port
number The port of the remote database instance.
authenticationMethod
string The type of authentication method to use. Options include userPassword, keyPair, oAuthClientCredentials, token, oAuthM2M, keyFile, auto and okta.
username
string The username used to connect to the remote database.
password
string The password used to connect to the remote database.
sid
string For Google BigQuery, the BigQuery project used to build the connection string.