Subscribe to and Manage Data Sources

Data Source API reference guide

This page describes the dataSource endpoint, through which users can subscribe to data sources, make unmasking requests, and manage data source tasks. To create data sources, see the specific handler endpoints.

Additional fields may be included in some responses you receive; however, these attributes are for internal purposes and are therefore undocumented.

Data source workflow

Search data sources and data source details

Method

Path

Purpose

GET

Search for data sources.

GET

Get a data source based on the ID.

GET

Get data source based on the name.

GET

Get data source based on the short name.

GET

Get parent and child relationship records for derived data sources using a specified data source ID.

GET

Retrieve a blob.

GET

Get all users with the provided access level for this data source.

GET

Retrieves the visibilities, masking information, and filters that the passed in user has access to in the specified data source.

GET

Retrieves a summary of total records, total visibilities, and visibilities a given user has access to.

GET

Retrieves a summary of total records, total visibilities, and visibilities the current user has access to for a specified data source.

Search for data sources

GET /dataSource

Search for data sources.

Query parameters

Attribute

Description

Required

blobHandlerType

array[string] Describes the type of underlying blob handler that will be used with this data source (e.g., Custom, MS SQL).

subscription

array[string] The requesting user's subscription status: pending, owner, subscribed, not_subscribed, expert, or ingest.

status

array[string] The data source status: passed or failed.

tag

array[string] Filters data sources by tags associated with the data sources.

searchText

string Searches for data source names using the provided string.

column

array[string] Searches for data source column names.

connectionString

array[string] Searches by connection string.

schema

string Searches for data source schema.

nameOnly

boolean When true, searchText will only search data source names. Default is false.

idOnly

boolean When true, only returns the ID Of the data source and the user's subscription status.

dataSourceIds

array[integer] Searches for the provided data source IDs.

selectFields

array[string] This field accepts the values id, name, and columnEvolutionEnabled. When id or name are provided, the request will return only the ID or name of the data source and the subscription status. If columnEvolutionEnabled is provided, the response will also include information about the policies, policy conflicts, and workspaces associated with the data sources.

offset

integer Used in combination with size to fetch pages. Default is 0.

size

integer The number of results to return per page. Default is 10.

sortField

string Used to sort results by field, which must be createdAt, name, blobHandlerType, subscriptionStatus, recordCount, status, policy, or editable.

sortOrder

string Sorts results by order, which must be asc or desc.

excludedProjects

array[integer] Filter out any data sources that belong to the specified projects.

ephemeral

boolean When true, returns ephemeral data sources.

clusterName

string The name of the remote cluster the data source is connected to.

mode

integer Specifies the query mode, which must be 0 (FULL), 1 (COUNT), 4 (TAG), 5 (MIN_MAX), or 6 (STATUS).

globalPolicy

string Filter by data sources that have this Global Policy applied.

hostname

string Searches data sources by hostname.

determinePolicyContflicts

boolean When true, filters results to return the data sources with policy conflicts.

collectionId

string Filter by data sources in the domain with this ID.

collectionExclude

string When true, filters results to return the data sources in the domain of the provided ID.

sddTemplateName

string Filter by data sources that have the specified SDD identification framework applied.

excludeSddTemplateName

string Filter by data sources that do not have the specified SDD identification framework applied.

Response schema

Attribute

Description

name

string Data source name.

integer Data source ID.

deleted

boolean If true the data source is a deleted data source.

description

string The data source description.

createdAt

timestamp The date and time the data source was created.

subscriptionPolicy

array Details the type of Subscription Policy applied to the data source.

schemaEvolutionId

integer The schema evolution ID.

recordCount

integer The record count.

status

array[string] Accepted statuses are passed or failed.

subscriptionStatus

array[string] Accepted statuses are subscribed or unsubscribed.

blobHandlerType

array[string] Describes the type of underlying blob handler of this data source (e.g., Custom, MS SQL).

subscriptionType

string The type of subscription policy on the project. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added).

connectionString

string The connection string information.

sqlSchemaName

string The schema name.

policy

string When this value is none, there are no data policies applied to the data source. Otherwise, this field indicates whether or not there are policy conflicts among the data policies applied to the data source.

policyHandlerType

string The policy handler type, such as None or Builder.

Request example

The following request returns 2 data sources.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource?size=2

Response example

{
  "name": "Public Barfoo",
  "id": 22,
  "recordFormat": "Not Provided",
  "deleted": false,
  "description": null,
  "createdAt": "2021-07-22T14:11:55.539Z",
  "subscriptionPolicy": {
      "type": "subscription",
      "automaticSubscription": true
  },
  "schemaEvolutionId": 1,
  "recordCount": 0,
  "status": "passed",
  "subscriptionStatus": "subscribed",
  "blobHandlerType": "Snowflake",
  "subscriptionType": "automatic",
  "connectionString": "test@immuta.connection.source.com:###/test",
  "sqlSchemaName": "public",
  "policy": "None",
  "policyHandlerType": "None",
  "native": null,
  "workspace": null
},
{
  "name": "Public Aaa Tpc",
  "id": 39,
  "recordFormat": "Not Provided",
  "deleted": false,
  "description": null,
  "createdAt": "2023-08-21T10:39:00.250Z",
  "subscriptionPolicy": {
      "type": "subscription",
      "exceptions": {
          "operator": "and",
          "conditions": [
              {
                  "type": "groups",
                  "group": {
                      "name": "alpha"
                  }
              }
          ]
      },
      "allowDiscovery": true,
      "automaticSubscription": true
  },
  "schemaEvolutionId": 1,
  "recordCount": 0,
  "blobHandlerType": "Snowflake",
  "subscriptionType": "policy",
  "sqlSchemaName": "public",
  "status": "passed",
  "subscriptionStatus": "owner",
  "connectionString": "test@immuta.connection.source.com:###/test",
  "remoteTable": "tpc",
  "remoteSchema": "public",
  "domainId": null,
  "domainName": null,
  "policy": "None",
  "policyHandlerType": "None",
  "native": null,
  "workspace": null
}

Get a data source by ID

GET /dataSource/{dataSourceId}

Get a data source based on the ID.

Query parameters

Attribute

Description

Required

dataSourceId

integer The data source ID .

Yes

Response Schema

Attribute

Description

name

string The data source name.

recordFormat

string The data format of blobs in the data source, such as json, xml, html, or jpeg.

description

string The description of the data source.

policyHandler

array The ID of the policy handler and details about the data policies enforced on the data source.

sqlSchemaName

string A string that represents this data source's schema name in Immuta.

sqlTableName

string The SQL table name in Immuta.

blobHandler

array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source.

blobHandlerType

string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL).

createdBy

integer The ID of the profile creating the data source.

deleted

boolean If true, the data source was deleted.

type

string The data source type, such as queryable or ingested.

rowCount

integer The number of rows.

documentation

string Documentation associated with the data source.

integer The data source ID.

policyHandlerType

string The type of policy handler applied to the data source: Builder.

subscriptionType

string The type of subscription policy on the data source. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added).

subscriptionPolicy

array Details about the Subscription Policy applied to the data source.

globalPolicies

string Details about the Global Policies applied to the data source.

status

string The data source health status.

Request example

The following request gets a data source based on the ID 22.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/22

Response example

{
  "name": "Public Barfoo",
  "recordFormat": "Not Provided",
  "description": null,
  "policyHandler": null,
  "sqlSchemaName": "public",
  "sqlTableName": "barfoo",
  "blobHandler": {
      "url": "https://your-url/snowflake/handler/22",
      "ca": {
          "name": "Certificate Authority Bundle"
  },
      "manualDictionary": false
  },
  "createdBy": 2,
  "deleted": false,
  "type": "queryable",
  "recordCount": 0,
  "rowCount": 6,
  "documentation": "# Public Barfoo",
  "statsExpiration": "2021-08-27T16:34:47.846Z",
  "id": 22,
  "blobHandlerType": "Snowflake",
  "policyHandlerType": "None",
  "subscriptionType": "automatic",
  "subscriptionPolicy": {
  "type": "subscription",
      "automaticSubscription": true
  },
  "globalPolicies": null,
  "status": "passed",
  "statusInfo": {
  "sql": {
      "status": "passed",
      "message": "Passed"
   }
  }
}

Get data source by name

GET /dataSource/name/{dataSourceName}

Get a data source based on the name.

Query parameters

Attribute

Description

Required

dataSourceName

string The data source name.

Yes

Response schema

Attribute

Description

name

string The data source name.

recordFormat

string The data format of blobs in the data source, such as json, xml, html, or jpeg.

description

string The description of the data source.

policyHandler

array The ID of the policy handler and details about the data policies enforced on the data source.

sqlSchemaName

string A string that represents this data source's schema name in Immuta.

sqlTableName

string The SQL table name in Immuta.

blobHandler

array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source.

blobHandlerType

string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL).

createdBy

integer The ID of the profile creating the data source.

deleted

boolean If true, the data source was deleted.

type

string The data source type, such as queryable or ingested.

rowCount

integer The number of rows.

documentation

string Documentation associated with the data source.

integer The data source ID.

policyHandlerType

string The type of policy handler applied to the data source: Builder.

subscriptionType

subscriptionPolicy

array Details about the Subscription Policy applied to the data source.

globalPolicies

string Details about the Global Policies applied to the data source.

status

string The data source health status.

Request example

The following request gets a data source based on the name Public Barfoo.

curl \
   --request GET \
   --header "Content-Type: application/json" \
   --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/name/Public%20Barfoo

Response example

{
  "name": "Public Barfoo",
  "recordFormat": "Not Provided",
  "description": null,
  "policyHandler": null,
  "sqlSchemaName": "public",
  "sqlTableName": "barfoo",
  "blobHandler": {
      "url": "https://your-url/snowflake/handler/22",
      "ca": {
          "name": "Certificate Authority Bundle"
  },
      "manualDictionary": false
      },
  "createdBy": 2,
  "deleted": false,
  "type": "queryable",
  "recordCount": 0,
  "rowCount": 6,
  "documentation": "# Public Barfoo",
  "statsExpiration": "2021-08-27T16:34:47.846Z",
  "id": 22,
  "blobHandlerType": "Snowflake",
  "policyHandlerType": "None",
  "subscriptionType": "automatic",
  "subscriptionPolicy": {
      "type": "subscription",
      "automaticSubscription": true
  },
  "globalPolicies": null,
  "status": "passed",
  "statusInfo": {
      "sql": {
      "status": "passed",
      "message": "Passed"
      }
  }
}

Get a data source by the short name

GET /dataSource/sqlTableName/{shortName}

Get a data source based on the SQL table name.

Query parameters

Attribute

Description

Required

shortName

string The data source SQL table name.

Yes

Response schema

Attribute

Description

name

string The data source name.

recordFormat

string The data format of blobs in the data source, such as json, xml, html, or jpeg.

description

string The description of the data source.

policyHandler

array The ID of the policy handler and details about the data policies enforced on the data source.

sqlSchemaName

string A string that represents this data source's schema name in Immuta.

sqlTableName

string The SQL table name in Immuta.

blobHandler

array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source.

blobHandlerType

string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL).

createdBy

integer The ID of the profile creating the data source.

deleted

boolean If true, the data source was deleted.

type

string The data source type, such as queryable or ingested.

rowCount

integer The number of rows.

documentation

string Documentation associated with the data source.

integer The data source ID.

policyHandlerType

string The type of policy handler applied to the data source: Builder.

subscriptionType

subscriptionPolicy

array Details about the Subscription Policy applied to the data source.

globalPolicies

string Details about the Global Policies applied to the data source.

status

string The data source health status.

Request example

The following request gets a data source based on the SQL table name customer_data.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/sqlTableName/customer_data

Response example

    {
  "name": "Dbo Customer Data",
  "recordFormat": "Not Provided",
  "description": null,
  "policyHandler": {
    "visibilitySchema": {
      "fields": [],
      "version": "2021-10-06T19:39:39.145Z"
    },
    "handlerId": 55,
    "dataSourceId": 26
  },
  "sqlSchemaName": "dbo",
  "sqlTableName": "customer_data",
  "blobHandler": {
    "url": "https://1.1.1.1.1/snowflake/testhandler/26",
    "ca": {
      "name": "Certificate Authority Bundle"
    },
    "manualDictionary": false
  },
  "createdBy": 2,
  "deleted": false,
  "type": "queryable",
  "recordCount": 0,
  "rowCount": 1000,
  "documentation": "# Dbo Customer Data",
  "statsExpiration": "2021-11-05T19:37:43.270Z",
  "id": 26,
  "blobHandlerType": "Snowflake",
  "policyHandlerType": "Builder",
  "subscriptionType": "automatic",
  "subscriptionPolicy": {
    "type": "subscription",
    "automaticSubscription": false
  },
  "globalPolicies": null,
  "status": "passed",
  "statusInfo": {
    "sql": {
      "status": "passed",
      "message": "Passed"
    },
    "stats": {
      "status": "passed",
      "lastAttempted": "2021-10-06T19:37:43.337Z"
    },
    "lastAttempt": {
      "date": "2021-10-06T19:39:39.821Z"
    },
    "highCardinality": {
      "status": "passed",
      "lastAttempted": "2021-10-06T19:37:43.337Z"
      }
  },
  "expiration": null,
  "catalogMetadata": null,
  "workspace": null,
  "seeded": false,
  "schemaEvolutionId": 4,
  "columnEvolutionEnabled": true,
  "createdAt": "2021-10-01T14:23:27.225Z",
  "updatedAt": "2021-10-06T19:39:39.145Z",
  "subscribedAsUser": true,
  "subscriptionId": 45,
  "acknowledgeRequired": false,
  "subscriptionStatus": "owner",
  "requestedState": "owner",
  "approved": true,
  "subscriptionExpiration": null,
  "filterId": null
}

Get data source relationships

GET /dataSource/{dataSourceId}/lineage/{type}

Get parent and child relationship records for derived data sources using a specified data source ID.

Query parameters

Attribute

Description

Required

type

string The type of lineage records to return. Options include: parents, children, and all.

Yes

dataSourceId

integer The target data source ID.

Yes

Response schema

Attribute

Description

children

array Details of the child data source, including dataSourceId, dataSourceName, projectId, policyHandlerDiff, deleted, createdBy, and createdAt.

parents

array Details of the parent data source, including dataSourceId, dataSourceName, projectId, policyHandlerDiff, deleted, createdBy, and createdAt.

Request example

The following request gets the parent relationship records for the derived data source with the data source ID 4.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/4/lineage/parents

Response example

{
  "parents": [{
    "dataSourceId": 3,
    "dataSourceName": "Public Healthcare Data",
    "deleted": false,
    "createdAt": "2022-08-17T13:41:38.381Z",
    "projectId": 2,
    "projectName": "Derived Data Source",
    "createdBy": "Your Username",
    "policyHandlerDiff": {
      "dsType": "queryable",
      "currentHandlerId": null,
      "previousHandlerId": null
    }
  }]
}

Retrieve a Blob

GET /dataSource/{dataSourceId}/blob/{blobid*}

Retrieve a blob.

Query parameters

Attribute

Description

Required

dataSourceId

integer The data source ID.

Yes

blobId

string The blob ID.

Yes

Response schema

The response will download the blobs in a file you specify.

Request example

The following request retrieves a blob.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    --output ./the-blobs-will-be-saved-here
    https://your-immuta-url.com/dataSource/22/blob/22

Response example

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  215k  100  215k    0     0   541k      0 --:--:-- --:--:-- --:--:--  541k

Get users by access level

GET /dataSource/{dataSourceId}/access

Get all users with the provided access level for this data source.

Query parameters

Attribute

Description

Required

dataSourceId

integer The data source ID.

Yes

states

Array[string] The status levels to include when querying for user access.

approved

boolean Denotes whether the returned access objects should be approved.

searchText

string A string used to filter returned users. The query is executed with a wildcard prefix and suffix.

size

integer The number of results to return.

offset

integer The number of results to skip (for paging).

sortField

string The field on which to sort the result set.

sortOrder

string The order in which to sort the results.

expandGroups

boolean If true will return individual members of any group subscribed.

ignoreSystemGenerated

boolean If true, will not return system generated accounts.

filterBySchemaEvolution

boolean If true, will only return users who have the specified level of access across ALL data sources within the same schema evolution group as this one.