Subscribe to and Manage Data Sources

Data Source API reference guide

This page describes the dataSource endpoint, through which users can subscribe to data sources and manage data source tasks. To create data sources, see the specific handler endpoints.

Additional fields may be included in some responses you receive; however, these attributes are for internal purposes and are therefore undocumented.

Data source workflow

Search data sources and data source details

MethodPathPurpose

GET

Search for data sources.

GET

Get a data source based on the ID.

GET

Get data source based on the name.

GET

Get data source based on the short name.

GET

Get parent and child relationship records for derived data sources using a specified data source ID.

GET

Retrieve a blob.

GET

Get all users with the provided access level for this data source.

GET

Retrieves the visibilities, masking information, and filters that the passed in user has access to in the specified data source.

GET

Retrieves a summary of total records, total visibilities, and visibilities a given user has access to.

GET

Retrieves a summary of total records, total visibilities, and visibilities the current user has access to for a specified data source.

Search for data sources

GET /dataSource

Search for data sources.

Query parameters

AttributeDescriptionRequired

blobHandlerType

array[string] Describes the type of underlying blob handler that will be used with this data source (e.g., Custom, MS SQL).

No

subscription

array[string] The requesting user's subscription status: pending, owner, subscribed, not_subscribed, expert, or ingest.

No

status

array[string] The data source status: passed or failed.

No

tag

array[string] Filters data sources by tags associated with the data sources.

No

searchText

string Searches for data source names using the provided string.

No

column

array[string] Searches for data source column names.

No

connectionString

array[string] Searches by connection string.

No

schema

string Searches for data source schema.

No

nameOnly

boolean When true, searchText will only search data source names. Default is false.

No

idOnly

boolean When true, only returns the ID Of the data source and the user's subscription status.

No

dataSourceIds

array[integer] Searches for the provided data source IDs.

No

selectFields

array[string] This field accepts the values id, name, and columnEvolutionEnabled. When id or name are provided, the request will return only the ID or name of the data source and the subscription status. If columnEvolutionEnabled is provided, the response will also include information about the policies, policy conflicts, and workspaces associated with the data sources.

No

offset

integer Used in combination with size to fetch pages. Default is 0.

No

size

integer The number of results to return per page. Default is 10.

No

sortField

string Used to sort results by field, which must be createdAt, name, blobHandlerType, subscriptionStatus, recordCount, status, policy, or editable.

No

sortOrder

string Sorts results by order, which must be asc or desc.

No

excludedProjects

array[integer] Filter out any data sources that belong to the specified projects.

No

ephemeral

boolean When true, returns ephemeral data sources.

No

clusterName

string The name of the remote cluster the data source is connected to.

No

mode

integer Specifies the query mode, which must be 0 (FULL), 1 (COUNT), 4 (TAG), 5 (MIN_MAX), or 6 (STATUS).

No

globalPolicy

string Filter by data sources that have this Global Policy applied.

No

hostname

string Searches data sources by hostname.

No

determinePolicyContflicts

boolean When true, filters results to return the data sources with policy conflicts.

No

collectionId

string Filter by data sources in the domain with this ID.

No

collectionExclude

string When true, filters results to return the data sources in the domain of the provided ID.

No

sddTemplateName

string Filter by data sources that have the specified SDD identification framework applied.

No

excludeSddTemplateName

string Filter by data sources that do not have the specified SDD identification framework applied.

No

Response schema

AttributeDescription

name

string Data source name.

id

integer Data source ID.

deleted

boolean If true the data source is a deleted data source.

description

string The data source description.

createdAt

timestamp The date and time the data source was created.

subscriptionPolicy

array Details the type of Subscription Policy applied to the data source.

schemaEvolutionId

integer The schema evolution ID.

recordCount

integer The record count.

status

array[string] Accepted statuses are passed or failed.

subscriptionStatus

array[string] Accepted statuses are subscribed or unsubscribed.

blobHandlerType

array[string] Describes the type of underlying blob handler of this data source (e.g., Custom, MS SQL).

subscriptionType

string The type of subscription policy on the project. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added).

connectionString

string The connection string information.

sqlSchemaName

string The schema name.

policy

string When this value is none, there are no data policies applied to the data source. Otherwise, this field indicates whether or not there are policy conflicts among the data policies applied to the data source.

policyHandlerType

string The policy handler type, such as None or Builder.

Request example

The following request returns 2 data sources.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource?size=2

Response example

{
  "name": "Public Barfoo",
  "id": 22,
  "recordFormat": "Not Provided",
  "deleted": false,
  "description": null,
  "createdAt": "2021-07-22T14:11:55.539Z",
  "subscriptionPolicy": {
      "type": "subscription",
      "automaticSubscription": true
  },
  "schemaEvolutionId": 1,
  "recordCount": 0,
  "status": "passed",
  "subscriptionStatus": "subscribed",
  "blobHandlerType": "Snowflake",
  "subscriptionType": "automatic",
  "connectionString": "test@immuta.connection.source.com:###/test",
  "sqlSchemaName": "public",
  "policy": "None",
  "policyHandlerType": "None",
  "native": null,
  "workspace": null
},
{
  "name": "Public Aaa Tpc",
  "id": 39,
  "recordFormat": "Not Provided",
  "deleted": false,
  "description": null,
  "createdAt": "2023-08-21T10:39:00.250Z",
  "subscriptionPolicy": {
      "type": "subscription",
      "exceptions": {
          "operator": "and",
          "conditions": [
              {
                  "type": "groups",
                  "group": {
                      "name": "alpha"
                  }
              }
          ]
      },
      "allowDiscovery": true,
      "automaticSubscription": true
  },
  "schemaEvolutionId": 1,
  "recordCount": 0,
  "blobHandlerType": "Snowflake",
  "subscriptionType": "policy",
  "sqlSchemaName": "public",
  "status": "passed",
  "subscriptionStatus": "owner",
  "connectionString": "test@immuta.connection.source.com:###/test",
  "remoteTable": "tpc",
  "remoteSchema": "public",
  "domainId": null,
  "domainName": null,
  "policy": "None",
  "policyHandlerType": "None",
  "native": null,
  "workspace": null
}

Get a data source by ID

GET /dataSource/{dataSourceId}

Get a data source based on the ID.

Query parameters

AttributeDescriptionRequired

dataSourceId

integer The data source ID .

Yes

Response Schema

AttributeDescription

name

string The data source name.

recordFormat

string The data format of blobs in the data source, such as json, xml, html, or jpeg.

description

string The description of the data source.

policyHandler

array The ID of the policy handler and details about the data policies enforced on the data source.

sqlSchemaName

string A string that represents this data source's schema name in Immuta.

sqlTableName

string The SQL table name in Immuta.

blobHandler

array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source.

blobHandlerType

string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL).

createdBy

integer The ID of the profile creating the data source.

deleted

boolean If true, the data source was deleted.

type

string The data source type: queryable.

rowCount

integer The number of rows.

documentation

string Documentation associated with the data source.

id

integer The data source ID.

policyHandlerType

string The type of policy handler applied to the data source: Builder.

subscriptionType

string The type of subscription policy on the data source. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added).

subscriptionPolicy

array Details about the Subscription Policy applied to the data source.

globalPolicies

string Details about the Global Policies applied to the data source.

status

string The data source health status.

Request example

The following request gets a data source based on the ID 22.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/22

Response example

{
  "name": "Public Barfoo",
  "recordFormat": "Not Provided",
  "description": null,
  "policyHandler": null,
  "sqlSchemaName": "public",
  "sqlTableName": "barfoo",
  "blobHandler": {
      "url": "https://your-url/snowflake/handler/22",
      "ca": {
          "name": "Certificate Authority Bundle"
  },
      "manualDictionary": false
  },
  "createdBy": 2,
  "deleted": false,
  "type": "queryable",
  "recordCount": 0,
  "rowCount": 6,
  "documentation": "# Public Barfoo",
  "statsExpiration": "2021-08-27T16:34:47.846Z",
  "id": 22,
  "blobHandlerType": "Snowflake",
  "policyHandlerType": "None",
  "subscriptionType": "automatic",
  "subscriptionPolicy": {
  "type": "subscription",
      "automaticSubscription": true
  },
  "globalPolicies": null,
  "status": "passed",
  "statusInfo": {
  "sql": {
      "status": "passed",
      "message": "Passed"
   }
  }
}

Get data source by name

GET /dataSource/name/{dataSourceName}

Get a data source based on the name.

Query parameters

AttributeDescriptionRequired

dataSourceName

string The data source name.

Yes

Response schema

AttributeDescription

name

string The data source name.

recordFormat

string The data format of blobs in the data source, such as json, xml, html, or jpeg.

description

string The description of the data source.

policyHandler

array The ID of the policy handler and details about the data policies enforced on the data source.

sqlSchemaName

string A string that represents this data source's schema name in Immuta.

sqlTableName

string The SQL table name in Immuta.

blobHandler

array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source.

blobHandlerType

string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL).

createdBy

integer The ID of the profile creating the data source.

deleted

boolean If true, the data source was deleted.

type

string The data source type:queryable.

rowCount

integer The number of rows.

documentation

string Documentation associated with the data source.

id

integer The data source ID.

policyHandlerType

string The type of policy handler applied to the data source: Builder.

subscriptionType

string The type of subscription policy on the data source. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added).

subscriptionPolicy

array Details about the Subscription Policy applied to the data source.

globalPolicies

string Details about the Global Policies applied to the data source.

status

string The data source health status.

Request example

The following request gets a data source based on the name Public Barfoo.

curl \
   --request GET \
   --header "Content-Type: application/json" \
   --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/name/Public%20Barfoo

Response example

{
  "name": "Public Barfoo",
  "recordFormat": "Not Provided",
  "description": null,
  "policyHandler": null,
  "sqlSchemaName": "public",
  "sqlTableName": "barfoo",
  "blobHandler": {
      "url": "https://your-url/snowflake/handler/22",
      "ca": {
          "name": "Certificate Authority Bundle"
  },
      "manualDictionary": false
      },
  "createdBy": 2,
  "deleted": false,
  "type": "queryable",
  "recordCount": 0,
  "rowCount": 6,
  "documentation": "# Public Barfoo",
  "statsExpiration": "2021-08-27T16:34:47.846Z",
  "id": 22,
  "blobHandlerType": "Snowflake",
  "policyHandlerType": "None",
  "subscriptionType": "automatic",
  "subscriptionPolicy": {
      "type": "subscription",
      "automaticSubscription": true
  },
  "globalPolicies": null,
  "status": "passed",
  "statusInfo": {
      "sql": {
      "status": "passed",
      "message": "Passed"
      }
  }
}

Get a data source by the short name

GET /dataSource/sqlTableName/{shortName}

Get a data source based on the SQL table name.

Query parameters

AttributeDescriptionRequired

shortName

string The data source SQL table name.

Yes

Response schema

AttributeDescription

name

string The data source name.

recordFormat

string The data format of blobs in the data source, such as json, xml, html, or jpeg.

description

string The description of the data source.

policyHandler

array The ID of the policy handler and details about the data policies enforced on the data source.

sqlSchemaName

string A string that represents this data source's schema name in Immuta.

sqlTableName

string The SQL table name in Immuta.

blobHandler

array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source.

blobHandlerType

string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL).

createdBy

integer The ID of the profile creating the data source.

deleted

boolean If true, the data source was deleted.

type

string The data source type: queryable.

rowCount

integer The number of rows.

documentation

string Documentation associated with the data source.

id

integer The data source ID.

policyHandlerType

string The type of policy handler applied to the data source: Builder.

subscriptionType

string The type of subscription policy on the data source. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added).

subscriptionPolicy

array Details about the Subscription Policy applied to the data source.

globalPolicies

string Details about the Global Policies applied to the data source.

status

string The data source health status.

Request example

The following request gets a data source based on the SQL table name customer_data.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/sqlTableName/customer_data

Response example

    {
  "name": "Dbo Customer Data",
  "recordFormat": "Not Provided",
  "description": null,
  "policyHandler": {
    "visibilitySchema": {
      "fields": [],
      "version": "2021-10-06T19:39:39.145Z"
    },
    "handlerId": 55,
    "dataSourceId": 26
  },
  "sqlSchemaName": "dbo",
  "sqlTableName": "customer_data",
  "blobHandler": {
    "url": "https://1.1.1.1.1/snowflake/testhandler/26",
    "ca": {
      "name": "Certificate Authority Bundle"
    },
    "manualDictionary": false
  },
  "createdBy": 2,
  "deleted": false,
  "type": "queryable",
  "recordCount": 0,
  "rowCount": 1000,
  "documentation": "# Dbo Customer Data",
  "statsExpiration": "2021-11-05T19:37:43.270Z",
  "id": 26,
  "blobHandlerType": "Snowflake",
  "policyHandlerType": "Builder",
  "subscriptionType": "automatic",
  "subscriptionPolicy": {
    "type": "subscription",
    "automaticSubscription": false
  },
  "globalPolicies": null,
  "status": "passed",
  "statusInfo": {
    "sql": {
      "status": "passed",
      "message": "Passed"
    },
    "stats": {
      "status": "passed",
      "lastAttempted": "2021-10-06T19:37:43.337Z"
    },
    "lastAttempt": {
      "date": "2021-10-06T19:39:39.821Z"
    },
    "highCardinality": {
      "status": "passed",
      "lastAttempted": "2021-10-06T19:37:43.337Z"
      }
  },
  "expiration": null,
  "catalogMetadata": null,
  "workspace": null,
  "seeded": false,
  "schemaEvolutionId": 4,
  "columnEvolutionEnabled": true,
  "createdAt": "2021-10-01T14:23:27.225Z",
  "updatedAt": "2021-10-06T19:39:39.145Z",
  "subscribedAsUser": true,
  "subscriptionId": 45,
  "acknowledgeRequired": false,
  "subscriptionStatus": "owner",
  "requestedState": "owner",
  "approved": true,
  "subscriptionExpiration": null,
  "filterId": null
}

Get data source relationships

GET /dataSource/{dataSourceId}/lineage/{type}

Get parent and child relationship records for derived data sources using a specified data source ID.

Query parameters

AttributeDescriptionRequired

type

string The type of lineage records to return. Options include: parents, children, and all.

Yes

dataSourceId

integer The target data source ID.

Yes

Response schema

AttributeDescription

children

array Details of the child data source, including dataSourceId, dataSourceName, projectId, policyHandlerDiff, deleted, createdBy, and createdAt.

parents

array Details of the parent data source, including dataSourceId, dataSourceName, projectId, policyHandlerDiff, deleted, createdBy, and createdAt.

Request example

The following request gets the parent relationship records for the derived data source with the data source ID 4.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/4/lineage/parents

Response example

{
  "parents": [{
    "dataSourceId": 3,
    "dataSourceName": "Public Healthcare Data",
    "deleted": false,
    "createdAt": "2022-08-17T13:41:38.381Z",
    "projectId": 2,
    "projectName": "Derived Data Source",
    "createdBy": "Your Username",
    "policyHandlerDiff": {
      "dsType": "queryable",
      "currentHandlerId": null,
      "previousHandlerId": null
    }
  }]
}

Retrieve a Blob

GET /dataSource/{dataSourceId}/blob/{blobid*}

Retrieve a blob.

Query parameters

AttributeDescriptionRequired

dataSourceId

integer The data source ID.

Yes

blobId

string The blob ID.

Yes

Response schema

The response will download the blobs in a file you specify.

Request example

The following request retrieves a blob.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    --output ./the-blobs-will-be-saved-here
    https://your-immuta-url.com/dataSource/22/blob/22

Response example

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  215k  100  215k    0     0   541k      0 --:--:-- --:--:-- --:--:--  541k

Get users by access level

GET /dataSource/{dataSourceId}/access

Get all users with the provided access level for this data source.

Query parameters

AttributeDescriptionRequired

dataSourceId

integer The data source ID.

Yes

states

Array[string] The status levels to include when querying for user access.

No

approved

boolean Denotes whether the returned access objects should be approved.

No

searchText

string A string used to filter returned users. The query is executed with a wildcard prefix and suffix.

No

size

integer The number of results to return.

No

offset

integer The number of results to skip (for paging).

No

sortField

string The field on which to sort the result set.

No

sortOrder

string The order in which to sort the results.

No

expandGroups

boolean If true will return individual members of any group subscribed.

No

ignoreSystemGenerated

boolean If true, will not return system generated accounts.

No

filterBySchemaEvolution

boolean If true, will only return users who have the specified level of access across ALL data sources within the same schema evolution group as this one.

No

Response schema

AttributeDescription

count

integer The number of users with access to the data source.

users

string The metadata regarding the users with access to the data source.

Request example

The following request gets all users with the provided access level for this data source.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/22/access?sortOrder=desc

Response example

{
  "count": 2,
  "users": [
    {
      "profile": 2,
      "name": "First Last",
      "iamid": "bim",
      "userid": "first.last@immuta.com",
      "email": "first.last@immuta.com",
      "type": "user",
      "admin": "First Last",
      "approved": true,
      "state": "owner",
      "systemGenerated": false,
      "lastExternalRefresh": "2021-10-06T14:58:46.983Z",
      "subscriptionId": 586,
      "createdAt": "2021-10-05T14:33:01.518Z",
      "updatedAt": "2021-10-05T14:33:01.518Z",
      "approvals": [
        {
          "requiredPermission": "OWNER",
          "state": "approved",
          "approverId": null,
          "ownerModelId": null,
          "approver": "First Last",
          "ownerModelName": null
        }
      ],
      "currentUserCanApprove": false
    },
    {
      "profile": 3,
      "name": "Tommy Test",
      "iamid": "bim",
      "userid": "tommy.test@immuta.com",
      "email": "tommy.test@immuta.com",
      "type": "user",
      "admin": "First Last",
      "approved": true,
      "state": "subscribed",
      "systemGenerated": false,
      "lastExternalRefresh": "2021-09-07T16:16:29.957Z",
      "subscriptionId": 649,
      "createdAt": "2021-10-06T14:58:31.366Z",
      "updatedAt": "2021-10-06T14:58:31.366Z",
      "approvals": [
        {
          "requiredPermission": "OWNER",
          "state": "approved",
          "approverId": null,
          "ownerModelId": null,
          "approver": "First Last",
          "ownerModelName": null
        }
      ],
      "currentUserCanApprove": false
    }
  ]
}

Get user access info for a data source

GET /dataSource/{dataSourceId}/users/{profileId}/policyInfo

Retrieves the visibilities, masking information, and filters that the passed in user has access to in the specified data source.

Query parameters

AttributeDescriptionRequired

dataSourceId

integer The data source ID.

Yes

profileId

integer The profile ID of the user.

Yes

projectId

integer The project ID. If provided, this project will be used when evaluating the user's visibilities.

No

Response schema

AttributeDescription

visibilities

array Details of the user's visibilities, including anyKey.

visibilityRuleApplies

boolean If true, a visibility rule exists and the user is not excepted from it.

masked

array Masking information for the data source, including metadata, name, type, and actionType.

additionalFilters

array Policy information for the data source, including customWhere, differentialPrivacy, eventTimeColumn, minimization, time, filterSeconds, and isOlderOrNewer.

Request example

The following request gets the visibility information for the user with the profile ID 2 on the data source with the data source ID 16.

curl \