Manage Data Sources

Additional fields may be included in some responses you receive; however, these attributes are for internal purposes and are therefore undocumented.

Endpoints and methods

Method	Endpoint	Purpose
PUT	/dataSource/{dataSourceId}	Update a data source.
PUT	/dataSource/bulk/{type}	Update multiple data sources.
POST	/dataSource/bulkRefreshViews	Sync the data policies and grants for data sources in the remote data platform.
PUT	/dataSource/detectRemoteChanges	Trigger the schema monitoring job for the specified detection group, or all groups if no ID is given.
GET	/dataSource/{dataSourceId}/test	Refresh tags from an external catalog on a data source. The external catalog must be linked to the data source.
GET	/dataSource/blobHandlerTypes	Retrieve all blob handlers the current user is allowed to create.
GET	/dataSource/byPurposes	Get data sources that match a set of purposes.
GET	/dataSource/rpc/mine	Retrieve all the data sources the current user has access to.
GET	/dataSource/{dataSourceId}/activities	Get all of the recent policy activities for a given data source.
GET	/dataSource/{dataSourceId}/contacts	Get the profiles for the data source owners and experts.
GET	/dataSource/{dataSourceId}/tags	Get the tags for a data source.
DELETE	/dataSource/{dataSourceId}	Delete a data source. This will perform a soft delete on the first call and a hard delete the second time.
DELETE	/dataSource/tasks/{taskId}	Delete the specified task.
DELETE	/dataSource/{dataSourceId}/unsubscribe	Unsubscribe from a data source.

PUT /dataSource/{dataSourceId}

The /dataSource/{dataSourceId} endpoint cannot be used for connection-backed data sources when updating deleted or name. To complete these actions on any connection-backed data sources, use the /data endpoint.

Update a data source.

Required: The global GOVERNANCE permission or be the data source owner

curl \
    --request PUT \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    --data @example-payload.json \
    https://your-immuta-url.com/dataSource/22

Payload example

{
  "name": "Public Credit Accounts for Training"
}

Path parameters

Attribute	Description	Required
dataSourceId	integer The data source ID.	Yes

Payload parameters

Attribute	Description	Required
blobHandler	object Blob handler configuration for the data source (for example, url, optional ca, and manualDictionary).	No
blobHandlerType	string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL).	No
recordFormat	string The data format of blobs in the data source, such as json, xml, html, or jpeg.	No
type	string The type of data source: queryable (metadata is dynamically queried).	No
name	string The name of the data source. It must be unique within the Immuta tenant.	No
sqlTableName	string A string that represents this data source's table in Immuta.	No
organization	string The organization that owns the data source.	No
category	string The category of the data source.	No
description	string The description of the data source.	No
hasExamples	boolean When true, the data source contains examples.	No

Response

Returns a data source object for the updated data source.

Attribute	Description
private	boolean When false, the data source will be publicly available in the Immuta UI.
blobHandler	object Blob handler configuration for the data source (for example, url, optional ca, and manualDictionary).
blobHandlerType	string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL).
recordFormat	string The data format of blobs in the data source, such as json, xml, html, or jpeg.
type	string The type of data source: queryable (metadata is dynamically queried).
name	string The name of the data source. It must be unique within the Immuta tenant.
sqlTableName	string A string that represents this data source's table in Immuta.
organization	string The organization that owns the data source.
description	string The description of the data source.
policyHandler	object The policy handler details and policies enforced on the data source.
subscriptionPolicy	object Details about the subscription policy enforced on the data source, including policy type and approvals/exceptions.

{
  "name": "Public Credit Accounts for Training",
  "recordFormat": "Not Provided",
  "description": null,
  "policyHandler": {
    "handlerId": 21,
    "visibilitySchema": {
      "fields": [],
      "version": "2021-09-15T17:44:20.678Z"
    },
    "previousHandlerId": 20,
    "dataSourceId": 1
  },
  "sqlSchemaName": "public",
  "sqlTableName": "credit_accounts",
  "blobHandler": {
    "url": "https://1.1.1.1:1111/snowflake/handler/1",
    "ca": {
      "name": "Certificate Authority Bundle"
    },
    "manualDictionary": false
  },
  "createdBy": 2,
  "deleted": false,
  "type": "queryable",
  "recordCount": 0,
  "rowCount": "100000",
  "statsExpiration": "2021-09-22T13:51:46.646Z",
  "id": 1,
  "blobHandlerType": "Snowflake",
  "policyHandlerType": "Builder",
  "subscriptionType": "approval",
  "subscriptionPolicy": {
    "type": "subscription",
    "approvals": [{
      "requiredPermission": "OWNER",
      "specificApproverRequired": false
    }]
  },
  "globalPolicies": null,
  "status": "failed",
  "statusInfo": {
    "sql": {
      "status": "passed",
      "message": "Passed"
    },
    "stats": {
      "status": "passed",
      "lastAttempted": "2021-09-21T13:51:46.674Z"
    },
    "fingerprint": {
      "status": "passed",
      "lastAttempted": "2021-09-09T14:12:25.177Z"
    },
    "lastAttempt": {
      "date": "2021-09-20T19:35:21.929Z"
    },
    "globalPolicy": {
      "status": "passed",
      "lastAttempted": "2021-09-17T19:07:38.092Z"
    },
    "highCardinality": {
      "status": "failed",
      "message": "Error could not connect to remote database",
      "lastAttempted": "2021-09-20T16:43:19.426Z"
    }
  },
  "expiration": null,
  "catalogMetadata": null,
  "workspace": null,
  "seeded": false,
  "schemaEvolutionId": 1,
  "columnEvolutionEnabled": true,
  "createdAt": "2021-09-09T14:12:09.511Z",
  "updatedAt": "2021-09-21T13:52:42.908Z"
}

PUT /dataSource/bulk/{type}

The dataSource/bulk/{type} endpoint cannot be used with connection-backed data sources when bulk updating delete, restore or disable. To complete these actions on any connection-backed data sources, use the /data endpoint.

Update data sources.

Required: The global GOVERNANCE permission or be the data source owner for all data sources included in the payload

curl \
    --request PUT \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    --data @example-payload.json \
    https://your-immuta-url.com/dataSource/bulk/tags

Payload example

{
  "ids": [49, 50],
  "update": {
    "tags": [{
      "name": "Address.email",
      "source": "curated"
    }]
  }
}

Path parameters

Attribute	Description	Required
type	string The action to perform on the data sources: add-users, disable, restore, delete, or tags.	Yes

Payload parameters

Attribute	Description	Required
ids	array[integer] The IDs of the data sources to update.	Yes
update	array[object] Only required for add-users (includes metadata about the users' profiles: id and state) and tags (includes metadata about the tags: name and source) types.	No

Response

Returns an ID to track the jobs created to update the data sources.

Attribute	Description
bulkId	string The ID of the bulk data source update.
jobsCreated	integer The number of jobs created.

{
  "bulkId": "bulk_ds_update_4896d300e04c4a8f89493ebf125c5c6b",
  "jobsCreated": 2
}

POST /dataSource/bulkRefreshViews

Sync the data policies and grants for data sources in the remote data platform.

Required: The global GOVERNANCE permission or be the data source owner for all data sources included in the payload

curl \
    --request POST \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer <TOKEN>" \
    --data @example-payload.json \
    https://your-immuta-url.com/dataSource/bulkRefreshViews

Payload example

{
  "dataSourceIds": [202]
}

Payload parameters

Attribute	Description	Required
dataSourceIds	array[integer] The IDs of the data sources to sync the policies.	Yes

PUT /dataSource/detectRemoteChanges

The /dataSource/detectRemoteChanges endpoint cannot be used with connection-backed data sources. For any connection-backed data sources, use the /data/crawl/{objectPath} endpoint.

Trigger the schema monitoring job for the specified detection group, or all groups if no payload parameters are given.

Required: The global GOVERNANCE permission or be the data source owner for all data sources included in the payload

curl \
    --request PUT \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    --data @example-payload.json \
    https://your-immuta-url.com/dataSource/detectRemoteChanges

Payload example

The tabs below illustrate payloads for triggering schema monitoring on a host, database, or table. The request will run schema monitoring for all databases registered under the hostname provided in the payload.

Host

The request will run schema monitoring for all databases registered under the hostname provided in the payload.

{
  "hostname": "organization.us-east-1.snowflakecomputing.com"
}

Database

Table

Payload parameters

Attribute	Description	Required
dataSourceIds	array[integer] The data source IDs to run the column detection job on. Leave empty to run this job globally on all data sources. This parameter cannot be included in the payload if schemaEvolutionId or any combination of hostname, database, port, or table is included.	No
hostname	string The hostname of the data sources. This parameter cannot be included in the payload if dataSourceIds or schemaEvolutionId is included.	No
port	integer The port used to connect the data sources to Immuta. This parameter cannot be included in the payload if dataSourceIds or schemaEvolutionId is included.	No
database	string The database name. This runs schema monitoring on the database provided. If data sources were initially registered via the V2 API, including this parameter will locate new schemas that contain tables Immuta has the ability to access, and Immuta will create a new schema project associated with these newly discovered schemas and create data sources for each table located. If data sources were initially registered via the V1 API, including this parameter will only update the columns and tables of registered schema and tables of the specified database; it will not register any new schemas. This parameter cannot be included in the payload if dataSourceIds or schemaEvolutionId is included.	No
table	string The table name. This will run column detection to just update the columns in this table. This parameter cannot be included in the payload if dataSourceIds or schemaEvolutionId is included.	No
schemaEvolutionId	integer The ID of the schema to run the schema monitoring job on. This will run on all tables associated with the specified ID. The schema ID can be found in the response body of /dataSource/{dataSourceId}. This parameter cannot be included in the payload if dataSourceIds or any combination of hostname, database, port, or table is included.	No
skipColumnDetection	boolean When true, Immuta will only pull new tables from the source server. This parameter can only be paired with schemaEvolutionId.	No
overrides.httpPath	string If Databricks ephemeral overrides are configured, provide the alternative HTTP path to trigger schema monitoring on that ephemeral cluster.	No

Response

Returns details about the schema monitoring and column detection jobs created.

Attribute	Description
schemaDetection	object Includes details about the resulting schema detection jobs.
columnDetection	object Includes details about the resulting column detection jobs.
bulkId	string The unique identifier of the jobs running schema monitoring and column detection.

Responses may include bulkId, schemaDetection, or columnDetection objects, depending on the payload.

The tabs below illustrate the example response for each example payload provided above.

Host

{
  "bulkId": "31ab4312-b90a-49a6-baf8-70f87cd92a89"
}

Database

Table

GET /dataSource/{dataSourceId}/test

Refreshes the data source health checks and refreshes tags from an external catalog on a data source. The external catalog must be linked to the data source.

Required: The global GOVERNANCE permission or be the data source owner

curl \
  --request GET \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer dea464c07bd07300095caa8" \
  https://your-immuta-url.com/dataSource/23/test

Path parameters

Attribute	Description	Required
dataSourceId	integer The data source ID.	Yes

Response

Returns health check details for the specified data source.

Attribute	Description
blob	object Indicates whether or not the blob was successfully crawled.
columnEvolution	object Indicates whether or not the job run to check for columns added or removed from the data source passed and when it was last run.
externalCatalog	object Indicates whether or not the external catalog was successfully linked to the data source.
fingerprint	object Indicates whether or not the fingerprint job was successful (passed) and when it was last run. The fingerprint captures summary statistics of a Snowflake data source.
framework	object Indicates whether or not the classification was successfully run on the data source to determine its sensitivity.
globalPolicy	object Indicates whether or not global policies were successfully applied to the data source.
highCardinality	object Indicates whether or not the job run to calculate the data source's high cardinality column passed and when it was last run.
schemaEvolution	object Indicates whether or not the job run to check if a new table had been added in the remote database passed and when it was last run. If a new table was added, Immuta automatically creates a new data source. Correspondingly, if a remote table is removed, that data source will be disabled in the console.
sdd	object Indicates whether or not identification was successfully run on the data source.
sql	object Indicates whether or not the SQL query run to check the data source's health passed and when it was last run.
stats	object Indicates whether or not the job run to calculate the number of rows in the data source passed and when it was last run.

{
  "sql": {
      "status": "passed",
      "message": "Passed"
  },
  "stats": {
      "status": "passed",
      "lastAttempted": "2021-09-09T13:43:43.948Z"
  },
  "fingerprint": {
      "status": "passed",
      "lastAttempted": "2021-07-22T14:12:01.525Z"
  },
  "lastAttempt": {
      "date": "2021-09-09T16:47:05.173Z"
  },
  "columnEvolution": {
      "status": "passed",
      "lastAttempted": "2021-09-08T16:36:05.557Z"
  },
  "highCardinality": {
      "status": "passed",
      "lastAttempted": "2021-07-22T14:11:58.439Z"
  },
  "schemaEvolution": {
      "status": "passed",
      "lastAttempted": "2021-09-08T16:35:57.867Z"
  },
  "status": "passed"
}

GET /dataSource/blobHandlerTypes

Retrieve all blob handlers the current user is allowed to create.

curl \
  --request GET \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer dea464c07bd07300095caa8" \
  https://your-immuta-url.com/dataSource/blobHandlerTypes

Response

Returns blob handler types the current user can create.

Attribute	Description
name	string The name of the blob handler.
baseUrl	string The base URL for the data source.
config	array Includes information about the connection configuration.
port	integer The port number.
driver	string The name of the driver.

{
  "name": "Azure Synapse Analytics",
  "pluginType": "queryable",
  "clientUrl": "/asa",
  "baseUrl": "https://0.0.0.0:8823/asa",
  "config": {
  "port": 1433,
  "allowSSLToggle": false
  },
  "displayOrder": 41,
  "requiresHashPhraseForDP": true,
  "driver": "Azure Synapse Analytics"
}

GET /dataSource/byPurposes

Get data sources that match a set of purposes.

curl \
  --request GET \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer dea464c07bd07300095caa8" \
  https://your-immuta-url.com/dataSource/byPurposes?purposes=Data+Analysis

Query parameters

Attribute	Description	Required
purposes	array[string] The purposes to filter the data sources by.	Yes
excludedProjects	array[integer] Excludes data sources associated with specified project IDs.	No

Response

Returns data sources that match the requested purposes.

Attribute	Description
id	integer The data source ID.
name	array The name of the data source.
policyId	integer The policy ID.
restrictions	array Details regarding the operator (and or or) and purposes.

[{
  "id": 42,
  "name": "Army Army Records",
  "policyId": 56,
  "restrictions": [{
    "operator": "and",
    "purposes": ["Data Analysis"]
  }]
}]

GET /dataSource/rpc/mine

Retrieves all the data sources the current user has access to.

curl \
  --request GET \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer dea464c07bd07300095caa8" \
  https://your-immuta-url.com/dataSource/rpc/mine

Response

Returns data sources the current user can access.

Attribute	Description
id	integer The data source ID.
name	string The data source name.
type	string The type of data source: queryable.
sqlTableName	string The name of the table in Immuta.
sqlSchemaName	string The name of the schema in Immuta.
blobHandlerType	string The type of handler, such as Snowflake.
sparkUseJDBC	boolean When true, uses a JDBC driver.

{
  "id": 23,
  "name": "Public Record",
  "type": "queryable",
  "sqlTableName": "record",
  "sqlSchemaName": "public",
  "blobHandlerType": "Snowflake",
  "sparkUseJDBC": true
}

GET /dataSource/{dataSourceId}/activities

Get all of the recent policy activities for a given data source.

Required: The data source must be publicly visible, you must own the data source, or you must have access to see the data source through the GOVERNANCE permission or a subscription policy

curl \
  --request GET \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer dea464c07bd07300095caa8" \
  https://your-immuta-url.com/dataSource/23/activities

Path parameters

Attribute	Description	Required
dataSourceId	integer The data source ID.	Yes

Query parameters

Attribute	Description	Required
offset	integer The number of results to skip (for paging).	No
size	integer The number of results to return per page.	No

Response

Returns recent policy activity for the specified data source.

Attribute	Description
count	integer The number of results.
activities	array Includes details about the policy and the data source, including the policy and data source type, when the activity notification was triggered, and whether or not the policy change was triggered by a Global policy.
actionBy	array Details about who triggered the action.
targetUser	array Information about the user who received the notification.

{
  "count": 3,
  "activities": [{
    "modelType": "datasource",
    "modelId": "23",
    "createdAt": "2021-09-07T16:13:25.197Z",
    "notificationType": "policyUpdated",
    "metadata": {
      "dataSourceName": "Public Foobar",
      "triggeredByGlobal": false,
      "conflictCount": 0,
      "policyType": "data",
      "handlerId": 3,
      "previousHandlerId": 2,
      "dataSourceType": "queryable"
    },
    "read": false,
    "id": 256,
    "updatedAt": "2021-09-07T16:13:25.197Z",
    "actionBy": {
      "id": 2,
      "name": "first last",
      "email": "[e-mail]"
    },
    "targetUser": {
      "id": 2,
      "name": "first last",
      "email": "[e-mail]"
    }
  }]
}

GET /dataSource/{dataSourceId}/contacts

Gets the profiles for the data source owners and experts.

curl \
  --request GET \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer dea464c07bd07300095caa8" \
  https://your-immuta-url.com/dataSource/23/contacts

Path parameters

Attribute	Description	Required
dataSourceId	integer The data source ID.	Yes

Response

Returns owner and expert profiles for the specified data source.

Attribute	Description
id	integer The data source ID.
state	string The user's data source role, such as owner or subscribed.
name	string The user's name.
email	string The user's email.
profile	integer The user's profile ID.

{
  "type": "profile",
  "id": 23,
  "state": "owner",
  "name": "first last",
  "email": "[e-mail]",
  "profile": 2
},
{
  "type": "profile",
  "id": 23,
  "state": "owner",
  "name": "Tommy Test",
  "email": "[e-mail]",
  "profile": 3
}

GET /dataSource/{dataSourceId}/tags

Get the tags for a data source.

Required: The data source must be publicly visible, you must own the data source, or you must have access to see the data source through the GOVERNANCE permission or a subscription policy

curl \
  --request GET \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer dea464c07bd07300095caa8" \
  https://your-immuta-url.com/dataSource/4/tags

Path parameters

Attribute	Description	Required
dataSourceId	integer The data source ID.	Yes

Query parameters

Attribute	Description	Required
blobId	string Returns the tags for the specified blob.	No
blobTagsOnly	boolean When true, will only display blob tags associated with a data source.	No

Response

Returns tags for the specified data source.

Attribute	Description
tags	array Includes details about the tags, such as the name, source, and the profile ID of the user who added the tag.

{
  "tags": [
    {
      "name": "Discovered.Entity.Medicare Number",
      "source": "curated",
      "modelType": "datasource",
      "modelId": "4",
      "addedBy": 1,
      "deleted": false
    }
  ]
}

DELETE /dataSource/{dataSourceId}

The /dataSource/{dataSourceId} endpoint cannot be used with connection-backed data sources. For any connection-backed data sources, use the /data/object/{objectPath} endpoint.

Delete a data source. This will perform a soft delete on the first call and a hard delete the second time.

Required: The global GOVERNANCE permission or be the data source owner

curl \
  --request DELETE \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer dea464c07bd07300095caa8" \
  https://your-immuta-url.com/dataSource/23

Path parameters

Attribute	Description	Required
dataSourceId	integer The data source ID.	Yes

Response

Returns delete status details for the specified data source.

Attribute	Description
success	boolean If true, the request to disable or delete the data source was successful.
id	integer The data source ID.
schemaEvolutionId	integer The schema evolution ID.
name	string The data source name.
disabled	boolean If true, the data source is disabled.
handlerDeleteErrorMessage	string The delete error message.

{
  "success": true,
  "id": 23,
  "schemaEvolutionId": 1,
  "name": "Public Foobar",
  "disabled": true,
  "handlerDeleteErrorMessage": null
}

DELETE /dataSource/tasks/{taskId}

Delete the specified task.

curl \
  --request DELETE \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer dea464c07bd07300095caa8" \
  https://your-immuta-url.com/dataSource/tasks/8

Path parameters

Attribute	Description	Required
taskId	integer Target task ID.	Yes

Response

Returns details for the deleted task.

Attribute	Description
id	integer The deleted task ID.
state	string The state of the deleted task, such as pending.
type	string The type of deleted task, such as columnAdded.
targetNames	array[string] The names of users who received the request.
targetEmails	array[string] The emails of users who received the request.
requester	metadata Details regarding the requesting profile.
dataSource	metadata details regarding the data source.
metadata	array Details about the deleted task.

{
  "fullCount": 1,
  "id": 8,
  "state": "pending",
  "type": "columnAdded",
  "reason": "Immuta has detected a new column \"org\" in the remote table and has added it to \"Public Customer\"",
  "targetNames": ["Katie"],
  "targetEmails": ["[e-mail]"],
  "requester": {
    "name": "Immuta System Account",
    "id": 1
  },
  "dataSource": {
    "id": 11,
    "name": "Public Customer"
  },
  "createdAt": "2021-10-12T19:28:04.999Z"
}

DELETE /dataSource/{dataSourceId}/unsubscribe

Unsubscribe from a data source.

curl \
  --request DELETE \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer dea464c07bd07300095caa8" \
  https://your-immuta-url.com/dataSource/23/unsubscribe

Path parameters

Attribute	Description	Required
dataSourceId	integer The data source ID.	Yes

Response

Returns subscription status for the requesting user.

Attribute	Description
success	boolean If true, the requesting user is unsubscribed from the data source.

{
  "success": true
}