SaaS
Search powered by AI

Subscribe to and Manage Data Sources

Data Source API reference guide

This page describes the dataSource endpoint, through which users can subscribe to data sources and manage data source tasks. To create data sources, see the specific handler endpoints.

Additional fields may be included in some responses you receive; however, these attributes are for internal purposes and are therefore undocumented.

Data source workflow

  1. Search data sources and data source details.

  2. Access data sources and make data source requests.

  3. Manage data source requests.

  4. Update data sources.

  5. Delete a data source.

Search data sources and data source details

Method
Path
Purpose

GET

Search for data sources.

GET

Get a data source based on the ID.

GET

Get data source based on the name.

GET

Get data source based on the short name.

GET

Get parent and child relationship records for derived data sources using a specified data source ID.

GET

Retrieve a blob.

GET

Get all users with the provided access level for this data source.

GET

Retrieves the visibilities, masking information, and filters that the passed in user has access to in the specified data source.

GET

Retrieves a summary of total records, total visibilities, and visibilities a given user has access to.

GET

Retrieves a summary of total records, total visibilities, and visibilities the current user has access to for a specified data source.

Search for data sources

GET /dataSource

Search for data sources.

Query parameters

Attribute
Description
Required

blobHandlerType

array[string] Describes the type of underlying blob handler that will be used with this data source (e.g., Custom, MS SQL).

No

subscription

array[string] The requesting user's subscription status: pending, owner, subscribed, not_subscribed, expert, or ingest.

No

status

array[string] The data source status: passed or failed.

No

tag

array[string] Filters data sources by tags associated with the data sources.

No

searchText

string Searches for data source names using the provided string.

No

column

array[string] Searches for data source column names.

No

connectionString

array[string] Searches by connection string.

No

schema

string Searches for data source schema.

No

nameOnly

boolean When true, searchText will only search data source names. Default is false.

No

idOnly

boolean When true, only returns the ID Of the data source and the user's subscription status.

No

dataSourceIds

array[integer] Searches for the provided data source IDs.

No

selectFields

array[string] This field accepts the values id, name, and columnEvolutionEnabled. When id or name are provided, the request will return only the ID or name of the data source and the subscription status. If columnEvolutionEnabled is provided, the response will also include information about the policies, policy conflicts, and workspaces associated with the data sources.

No

offset

integer Used in combination with size to fetch pages. Default is 0.

No

size

integer The number of results to return per page. Default is 10.

No

sortField

string Used to sort results by field, which must be createdAt, name, blobHandlerType, subscriptionStatus, recordCount, status, policy, or editable.

No

sortOrder

string Sorts results by order, which must be asc or desc.

No

excludedProjects

array[integer] Filter out any data sources that belong to the specified projects.

No

ephemeral

boolean When true, returns ephemeral data sources.

No

clusterName

string The name of the remote cluster the data source is connected to.

No

mode

integer Specifies the query mode, which must be 0 (FULL), 1 (COUNT), 4 (TAG), 5 (MIN_MAX), or 6 (STATUS).

No

globalPolicy

string Filter by data sources that have this Global Policy applied.

No

hostname

string Searches data sources by hostname.

No

determinePolicyContflicts

boolean When true, filters results to return the data sources with policy conflicts.

No

collectionId

string Filter by data sources in the domain with this ID.

No

collectionExclude

string When true, filters results to return the data sources in the domain of the provided ID.

No

sddTemplateName

string Filter by data sources that have the specified SDD identification framework applied.

No

excludeSddTemplateName

string Filter by data sources that do not have the specified SDD identification framework applied.

No

Response schema

Attribute
Description

name

string Data source name.

id

integer Data source ID.

deleted

boolean If true the data source is a deleted data source.

description

string The data source description.

createdAt

timestamp The date and time the data source was created.

subscriptionPolicy

array Details the type of Subscription Policy applied to the data source.

schemaEvolutionId

integer The schema evolution ID.

recordCount

integer The record count.

status

array[string] Accepted statuses are passed or failed.

subscriptionStatus

array[string] Accepted statuses are subscribed or unsubscribed.

blobHandlerType

array[string] Describes the type of underlying blob handler of this data source (e.g., Custom, MS SQL).

subscriptionType

string The type of subscription policy on the project. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added).

connectionString

string The connection string information.

sqlSchemaName

string The schema name.

policy

string When this value is none, there are no data policies applied to the data source. Otherwise, this field indicates whether or not there are policy conflicts among the data policies applied to the data source.

policyHandlerType

string The policy handler type, such as None or Builder.

Request example

The following request returns 2 data sources.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource?size=2

Response example

{
  "name": "Public Barfoo",
  "id": 22,
  "recordFormat": "Not Provided",
  "deleted": false,
  "description": null,
  "createdAt": "2021-07-22T14:11:55.539Z",
  "subscriptionPolicy": {
      "type": "subscription",
      "automaticSubscription": true
  },
  "schemaEvolutionId": 1,
  "recordCount": 0,
  "status": "passed",
  "subscriptionStatus": "subscribed",
  "blobHandlerType": "Snowflake",
  "subscriptionType": "automatic",
  "connectionString": "test@immuta.connection.source.com:###/test",
  "sqlSchemaName": "public",
  "policy": "None",
  "policyHandlerType": "None",
  "native": null,
  "workspace": null
},
{
  "name": "Public Aaa Tpc",
  "id": 39,
  "recordFormat": "Not Provided",
  "deleted": false,
  "description": null,
  "createdAt": "2023-08-21T10:39:00.250Z",
  "subscriptionPolicy": {
      "type": "subscription",
      "exceptions": {
          "operator": "and",
          "conditions": [
              {
                  "type": "groups",
                  "group": {
                      "name": "alpha"
                  }
              }
          ]
      },
      "allowDiscovery": true,
      "automaticSubscription": true
  },
  "schemaEvolutionId": 1,
  "recordCount": 0,
  "blobHandlerType": "Snowflake",
  "subscriptionType": "policy",
  "sqlSchemaName": "public",
  "status": "passed",
  "subscriptionStatus": "owner",
  "connectionString": "test@immuta.connection.source.com:###/test",
  "remoteTable": "tpc",
  "remoteSchema": "public",
  "domainId": null,
  "domainName": null,
  "policy": "None",
  "policyHandlerType": "None",
  "native": null,
  "workspace": null
}

Get a data source by ID

GET /dataSource/{dataSourceId}

Get a data source based on the ID.

Query parameters

Attribute
Description
Required

dataSourceId

integer The data source ID .

Yes

Response Schema

Attribute
Description

name

string The data source name.

recordFormat

string The data format of blobs in the data source, such as json, xml, html, or jpeg.

description

string The description of the data source.

policyHandler

array The ID of the policy handler and details about the data policies enforced on the data source.

sqlSchemaName

string A string that represents this data source's schema name in Immuta.

sqlTableName

string The SQL table name in Immuta.

blobHandler

array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source.

blobHandlerType

string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL).

createdBy

integer The ID of the profile creating the data source.

deleted

boolean If true, the data source was deleted.

type

string The data source type: queryable.

rowCount

integer The number of rows.

documentation

string Documentation associated with the data source.

id

integer The data source ID.

policyHandlerType

string The type of policy handler applied to the data source: Builder.

subscriptionType

string The type of subscription policy on the data source. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added).

subscriptionPolicy

array Details about the Subscription Policy applied to the data source.

globalPolicies

string Details about the Global Policies applied to the data source.

status

string The data source health status.

Request example

The following request gets a data source based on the ID 22.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/22

Response example

{
  "name": "Public Barfoo",
  "recordFormat": "Not Provided",
  "description": null,
  "policyHandler": null,
  "sqlSchemaName": "public",
  "sqlTableName": "barfoo",
  "blobHandler": {
      "url": "https://your-url/snowflake/handler/22",
      "ca": {
          "name": "Certificate Authority Bundle"
  },
      "manualDictionary": false
  },
  "createdBy": 2,
  "deleted": false,
  "type": "queryable",
  "recordCount": 0,
  "rowCount": 6,
  "documentation": "# Public Barfoo",
  "statsExpiration": "2021-08-27T16:34:47.846Z",
  "id": 22,
  "blobHandlerType": "Snowflake",
  "policyHandlerType": "None",
  "subscriptionType": "automatic",
  "subscriptionPolicy": {
  "type": "subscription",
      "automaticSubscription": true
  },
  "globalPolicies": null,
  "status": "passed",
  "statusInfo": {
  "sql": {
      "status": "passed",
      "message": "Passed"
   }
  }
}

Get data source by name

GET /dataSource/name/{dataSourceName}

Get a data source based on the name.

Query parameters

Attribute
Description
Required

dataSourceName

string The data source name.

Yes

Response schema

Attribute
Description

name

string The data source name.

recordFormat

string The data format of blobs in the data source, such as json, xml, html, or jpeg.

description

string The description of the data source.

policyHandler

array The ID of the policy handler and details about the data policies enforced on the data source.

sqlSchemaName

string A string that represents this data source's schema name in Immuta.

sqlTableName

string The SQL table name in Immuta.

blobHandler

array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source.

blobHandlerType

string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL).

createdBy

integer The ID of the profile creating the data source.

deleted

boolean If true, the data source was deleted.

type

string The data source type:queryable.

rowCount

integer The number of rows.

documentation

string Documentation associated with the data source.

id

integer The data source ID.

policyHandlerType

string The type of policy handler applied to the data source: Builder.

subscriptionType

string The type of subscription policy on the data source. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added).

subscriptionPolicy

array Details about the Subscription Policy applied to the data source.

globalPolicies

string Details about the Global Policies applied to the data source.

status

string The data source health status.

Request example

The following request gets a data source based on the name Public Barfoo.

curl \
   --request GET \
   --header "Content-Type: application/json" \
   --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/name/Public%20Barfoo

Response example

{
  "name": "Public Barfoo",
  "recordFormat": "Not Provided",
  "description": null,
  "policyHandler": null,
  "sqlSchemaName": "public",
  "sqlTableName": "barfoo",
  "blobHandler": {
      "url": "https://your-url/snowflake/handler/22",
      "ca": {
          "name": "Certificate Authority Bundle"
  },
      "manualDictionary": false
      },
  "createdBy": 2,
  "deleted": false,
  "type": "queryable",
  "recordCount": 0,
  "rowCount": 6,
  "documentation": "# Public Barfoo",
  "statsExpiration": "2021-08-27T16:34:47.846Z",
  "id": 22,
  "blobHandlerType": "Snowflake",
  "policyHandlerType": "None",
  "subscriptionType": "automatic",
  "subscriptionPolicy": {
      "type": "subscription",
      "automaticSubscription": true
  },
  "globalPolicies": null,
  "status": "passed",
  "statusInfo": {
      "sql": {
      "status": "passed",
      "message": "Passed"
      }
  }
}

Get a data source by the short name

GET /dataSource/sqlTableName/{shortName}

Get a data source based on the SQL table name.

Query parameters

Attribute
Description
Required

shortName

string The data source SQL table name.

Yes

Response schema

Attribute
Description

name

string The data source name.

recordFormat

string The data format of blobs in the data source, such as json, xml, html, or jpeg.

description

string The description of the data source.

policyHandler

array The ID of the policy handler and details about the data policies enforced on the data source.

sqlSchemaName

string A string that represents this data source's schema name in Immuta.

sqlTableName

string The SQL table name in Immuta.

blobHandler

array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source.

blobHandlerType

string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL).

createdBy

integer The ID of the profile creating the data source.

deleted

boolean If true, the data source was deleted.

type

string The data source type: queryable.

rowCount

integer The number of rows.

documentation

string Documentation associated with the data source.

id

integer The data source ID.

policyHandlerType

string The type of policy handler applied to the data source: Builder.

subscriptionType

string The type of subscription policy on the data source. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added).

subscriptionPolicy

array Details about the Subscription Policy applied to the data source.

globalPolicies

string Details about the Global Policies applied to the data source.

status

string The data source health status.

Request example

The following request gets a data source based on the SQL table name customer_data.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/sqlTableName/customer_data

Response example

    {
  "name": "Dbo Customer Data",
  "recordFormat": "Not Provided",
  "description": null,
  "policyHandler": {
    "visibilitySchema": {
      "fields": [],
      "version": "2021-10-06T19:39:39.145Z"
    },
    "handlerId": 55,
    "dataSourceId": 26
  },
  "sqlSchemaName": "dbo",
  "sqlTableName": "customer_data",
  "blobHandler": {
    "url": "https://1.1.1.1.1/snowflake/testhandler/26",
    "ca": {
      "name": "Certificate Authority Bundle"
    },
    "manualDictionary": false
  },
  "createdBy": 2,
  "deleted": false,
  "type": "queryable",
  "recordCount": 0,
  "rowCount": 1000,
  "documentation": "# Dbo Customer Data",
  "statsExpiration": "2021-11-05T19:37:43.270Z",
  "id": 26,
  "blobHandlerType": "Snowflake",
  "policyHandlerType": "Builder",
  "subscriptionType": "automatic",
  "subscriptionPolicy": {
    "type": "subscription",
    "automaticSubscription": false
  },
  "globalPolicies": null,
  "status": "passed",
  "statusInfo": {
    "sql": {
      "status": "passed",
      "message": "Passed"
    },
    "stats": {
      "status": "passed",
      "lastAttempted": "2021-10-06T19:37:43.337Z"
    },
    "lastAttempt": {
      "date": "2021-10-06T19:39:39.821Z"
    },
    "highCardinality": {
      "status": "passed",
      "lastAttempted": "2021-10-06T19:37:43.337Z"
      }
  },
  "expiration": null,
  "catalogMetadata": null,
  "workspace": null,
  "seeded": false,
  "schemaEvolutionId": 4,
  "columnEvolutionEnabled": true,
  "createdAt": "2021-10-01T14:23:27.225Z",
  "updatedAt": "2021-10-06T19:39:39.145Z",
  "subscribedAsUser": true,
  "subscriptionId": 45,
  "acknowledgeRequired": false,
  "subscriptionStatus": "owner",
  "requestedState": "owner",
  "approved": true,
  "subscriptionExpiration": null,
  "filterId": null
}

Get data source relationships

GET /dataSource/{dataSourceId}/lineage/{type}

Get parent and child relationship records for derived data sources using a specified data source ID.

Query parameters

Attribute
Description
Required

type

string The type of lineage records to return. Options include: parents, children, and all.

Yes

dataSourceId

integer The target data source ID.

Yes

Response schema

Attribute
Description

children

array Details of the child data source, including dataSourceId, dataSourceName, projectId, policyHandlerDiff, deleted, createdBy, and createdAt.

parents

array Details of the parent data source, including dataSourceId, dataSourceName, projectId, policyHandlerDiff, deleted, createdBy, and createdAt.

Request example

The following request gets the parent relationship records for the derived data source with the data source ID 4.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/4/lineage/parents

Response example

{
  "parents": [{
    "dataSourceId": 3,
    "dataSourceName": "Public Healthcare Data",
    "deleted": false,
    "createdAt": "2022-08-17T13:41:38.381Z",
    "projectId": 2,
    "projectName": "Derived Data Source",
    "createdBy": "Your Username",
    "policyHandlerDiff": {
      "dsType": "queryable",
      "currentHandlerId": null,
      "previousHandlerId": null
    }
  }]
}

Retrieve a Blob