2024.1
Search powered by AI

Subscribe to and Manage Data Sources

This page describes the dataSource endpoint, through which users can subscribe to data sources, make unmasking requests, and manage data source tasks. To create data sources, see the specific handler endpoints.

Additional fields may be included in some responses you receive; however, these attributes are for internal purposes and are therefore undocumented.

Data source workflow

  1. Search data sources and data source details.

  2. Access data sources and make data source requests.

  3. Manage data source requests.

  4. Update data sources.

  5. Delete a data source.

Search data sources and data source details

MethodPathPurpose

GET

/dataSource

Search for data sources.

GET

/dataSource/{dataSourceId}

Get a data source based on the ID.

GET

/dataSource/name/{dataSourceName}

Get data source based on the name.

GET

/dataSource/sqlTableName/{shortName}

Get data source based on the short name.

GET

/dataSource/{dataSourceId}/lineage/{type}

Get parent and child relationship records for derived data sources using a specified data source ID.

GET

/dataSource/{dataSourceId}/blob/{blobId*}

Retrieve a blob.

GET

/dataSource/{dataSourceId}/comments/{commentId}

Get a comment by ID for the data source.

GET

/dataSource/{dataSourceId}/comments

Get all the comments for the data source.

GET

/dataSource/{dataSourceId}/comments/count

Count the comments for a data source.

GET

/dataSource/{dataSourceId}/access

Get all users with the provided access level for this data source.

GET

/dataSource/{dataSourceId}/users/{profileId}/policyInfo

Retrieves the visibilities, masking information, and filters that the passed in user has access to in the specified data source.

GET

/dataSource/{dataSourceId}/users/{profileId}/visibilityReport

Retrieves a summary of total records, total visibilities, and visibilities a given user has access to.

GET

/dataSource/{dataSourceId}/visibilityReport

Retrieves a summary of total records, total visibilities, and visibilities the current user has access to for a specified data source.

Search for data sources

GET /dataSource

Search for data sources.

Query Parameters

AttributeDescriptionRequired

blobHandlerType

array[string] Describes the type of underlying blob handler that will be used with this data source (e.g., Custom, MS SQL).

No

subscription

array[string] The requesting user's subscription status: pending, owner, subscribed, not_subscribed, expert, or ingest.

No

status

array[string] The data source status: passed or failed.

No

tag

array[string] Filters data sources by tags associated with the data sources.

No

searchText

string Searches for data source names using the provided string.

No

column

array[string] Searches for data source column names.

No

connectionString

array[string] Searches by connection string.

No

schema

string Searches for data source schema.

No

nameOnly

boolean When true, searchText will only search data source names. Default is false.

No

idOnly

boolean When true, only returns the ID Of the data source and the user's subscription status.

No

dataSourceIds

array[integer] Searches for the provided data source IDs.

No

selectFields

array[string] This field accepts the values id, name, and columnEvolutionEnabled. When id or name are provided, the request will return only the ID or name of the data source and the subscription status. If columnEvolutionEnabled is provided, the response will also include information about the policies, policy conflicts, and workspaces associated with the data sources.

No

offset

integer Used in combination with size to fetch pages. Default is 0.

No

size

integer The number of results to return per page. Default is 10.

No

sortField

string Used to sort results by field, which must be createdAt, name, blobHandlerType, subscriptionStatus, recordCount, status, policy, or editable.

No

sortOrder

string Sorts results by order, which must be asc or desc.

No

excludedProjects

array[integer] Filter out any data sources that belong to the specified projects.

No

ephemeral

boolean When true, returns ephemeral data sources.

No

clusterName

string The name of the remote cluster the data source is connected to.

No

mode

integer Specifies the query mode, which must be 0 (FULL), 1 (COUNT), 4 (TAG), 5 (MIN_MAX), or 6 (STATUS).

No

globalPolicy

string Filter by data sources that have this Global Policy applied.

No

hostname

string Searches data sources by hostname.

No

Response Parameters

AttributeDescription

name

string Data source name.

id

integer Data source ID.

deleted

boolean If true the data source is a deleted data source.

description

string The data source description.

createdAt

timestamp The date and time the data source was created.

subscriptionPolicy

array Details the type of Subscription Policy applied to the data source.

schemaEvolutionId

integer The schema evolution ID.

recordCount

integer The record count.

status

array[string] Accepted statuses are passed or failed.

subscriptionStatus

array[string] Accepted statuses are subscribed or unsubscribed.

blobHandlerType

array[string] Describes the type of underlying blob handler of this data source (e.g., Custom, MS SQL).

subscriptionType

string The type of subscription policy on the project. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added).

connectionString

string The connection string information.

sqlSchemaName

string The schema name.

policy

string When this value is none, there are no data policies applied to the data source. Otherwise, this field indicates whether or not there are policy conflicts among the data policies applied to the data source.

policyHandlerType

string The policy handler type, such as None or Builder.

Request example

The following request returns 2 data sources.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource?size=2

Response example

{
  "name": "Public Barfoo",
  "id": 22,
  "recordFormat": "Not Provided",
  "deleted": false,
  "description": null,
  "createdAt": "2021-07-22T14:11:55.539Z",
  "subscriptionPolicy": {
      "type": "subscription",
      "automaticSubscription": true
  },
  "schemaEvolutionId": 1,
  "recordCount": 0,
  "status": "passed",
  "subscriptionStatus": "subscribed",
  "blobHandlerType": "Snowflake",
  "subscriptionType": "automatic",
  "connectionString": "test@immuta.connection.source.com:###/test",
  "sqlSchemaName": "public",
  "policy": "None",
  "policyHandlerType": "None",
  "native": null,
  "workspace": null
},
{
  "name": "Public Aaa Tpc",
  "id": 39,
  "recordFormat": "Not Provided",
  "deleted": false,
  "description": null,
  "createdAt": "2023-08-21T10:39:00.250Z",
  "subscriptionPolicy": {
      "type": "subscription",
      "exceptions": {
          "operator": "and",
          "conditions": [
              {
                  "type": "groups",
                  "group": {
                      "name": "alpha"
                  }
              }
          ]
      },
      "allowDiscovery": true,
      "automaticSubscription": true
  },
  "schemaEvolutionId": 1,
  "recordCount": 0,
  "blobHandlerType": "Snowflake",
  "subscriptionType": "policy",
  "sqlSchemaName": "public",
  "status": "passed",
  "subscriptionStatus": "owner",
  "connectionString": "test@immuta.connection.source.com:###/test",
  "remoteTable": "tpc",
  "remoteSchema": "public",
  "domainId": null,
  "domainName": null,
  "policy": "None",
  "policyHandlerType": "None",
  "native": null,
  "workspace": null
}

Get a data source by ID

GET /dataSource/{dataSourceId}

Get a data source based on the ID.

Query Parameters

AttributeDescriptionRequired

dataSourceId

integer The data source ID .

Yes

Response Parameters

AttributeDescription

name

string The data source name.

recordFormat

string The data format of blobs in the data source, such as json, xml, html, or jpeg.

description

string The description of the data source.

policyHandler

array The ID of the policy handler and details about the data policies enforced on the data source.

sqlSchemaName

string A string that represents this data source's schema name in the Query Engine.

sqlTableName

string The SQL table name in the Query Engine.

blobHandler

array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source.

blobHandlerType

string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL).

createdBy

integer The ID of the profile creating the data source.

deleted

boolean If true, the data source was deleted.

type

string The data source type, such as queryable or ingested.

rowCount

integer The number of rows.

documentation

string Documentation associated with the data source.

id

integer The data source ID.

policyHandlerType

string The type of policy handler applied to the data source: Builder.

subscriptionType

string The type of subscription policy on the data source. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added).

subscriptionPolicy

array Details about the Subscription Policy applied to the data source.

globalPolicies

string Details about the Global Policies applied to the data source.

status

string The data source health status.

Request example

The following request gets a data source based on the ID 22.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/22

Response example

{
  "name": "Public Barfoo",
  "recordFormat": "Not Provided",
  "description": null,
  "policyHandler": null,
  "sqlSchemaName": "public",
  "sqlTableName": "barfoo",
  "blobHandler": {
      "url": "https://your-url/snowflake/handler/22",
      "ca": {
          "name": "Certificate Authority Bundle"
  },
      "manualDictionary": false
  },
  "createdBy": 2,
  "deleted": false,
  "type": "queryable",
  "recordCount": 0,
  "rowCount": 6,
  "documentation": "# Public Barfoo",
  "statsExpiration": "2021-08-27T16:34:47.846Z",
  "id": 22,
  "blobHandlerType": "Snowflake",
  "policyHandlerType": "None",
  "subscriptionType": "automatic",
  "subscriptionPolicy": {
  "type": "subscription",
      "automaticSubscription": true
  },
  "globalPolicies": null,
  "status": "passed",
  "statusInfo": {
  "sql": {
      "status": "passed",
      "message": "Passed"
   }
  }
}

Get data source by name

GET /dataSource/name/{dataSourceName}

Get a data source based on the name.

Query Parameters

AttributeDescriptionRequired

dataSourceName

string The data source name.

Yes

Response Parameters

AttributeDescription

name

string The data source name.

recordFormat

string The data format of blobs in the data source, such as json, xml, html, or jpeg.

description

string The description of the data source.

policyHandler

array The ID of the policy handler and details about the data policies enforced on the data source.

sqlSchemaName

string A string that represents this data source's schema name in the Query Engine.

sqlTableName

string The SQL table name in the Query Engine.

blobHandler

array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source.

blobHandlerType

string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL).

createdBy

integer The ID of the profile creating the data source.

deleted

boolean If true, the data source was deleted.

type

string The data source type, such as queryable or ingested.

rowCount

integer The number of rows.

documentation

string Documentation associated with the data source.

id

integer The data source ID.

policyHandlerType

string The type of policy handler applied to the data source: Builder.

subscriptionType

string The type of subscription policy on the data source. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added).

subscriptionPolicy

array Details about the Subscription Policy applied to the data source.

globalPolicies

string Details about the Global Policies applied to the data source.

status

string The data source health status.

Request example

The following request gets a data source based on the name Public Barfoo.

curl \
   --request GET \
   --header "Content-Type: application/json" \
   --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/name/Public%20Barfoo

Response example

{
  "name": "Public Barfoo",
  "recordFormat": "Not Provided",
  "description": null,
  "policyHandler": null,
  "sqlSchemaName": "public",
  "sqlTableName": "barfoo",
  "blobHandler": {
      "url": "https://your-url/snowflake/handler/22",
      "ca": {
          "name": "Certificate Authority Bundle"
  },
      "manualDictionary": false
      },
  "createdBy": 2,
  "deleted": false,
  "type": "queryable",
  "recordCount": 0,
  "rowCount": 6,
  "documentation": "# Public Barfoo",
  "statsExpiration": "2021-08-27T16:34:47.846Z",
  "id": 22,
  "blobHandlerType": "Snowflake",
  "policyHandlerType": "None",
  "subscriptionType": "automatic",
  "subscriptionPolicy": {
      "type": "subscription",
      "automaticSubscription": true
  },
  "globalPolicies": null,
  "status": "passed",
  "statusInfo": {
      "sql": {
      "status": "passed",
      "message": "Passed"
      }
  }
}

Get a data source by the short name

GET /dataSource/sqlTableName/{shortName}

Get a data source based on the SQL table name.

Query Parameters

AttributeDescriptionRequired

shortName

string The data source SQL table name.

Yes

Response Parameters

AttributeDescription

name

string The data source name.

recordFormat

string The data format of blobs in the data source, such as json, xml, html, or jpeg.

description

string The description of the data source.

policyHandler

array The ID of the policy handler and details about the data policies enforced on the data source.

sqlSchemaName

string A string that represents this data source's schema name in the Query Engine.

sqlTableName

string The SQL table name in the Query Engine.

blobHandler

array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source.

blobHandlerType

string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL).

createdBy

integer The ID of the profile creating the data source.

deleted

boolean If true, the data source was deleted.

type

string The data source type, such as queryable or ingested.

rowCount

integer The number of rows.

documentation

string Documentation associated with the data source.

id

integer The data source ID.

policyHandlerType

string The type of policy handler applied to the data source: Builder.

subscriptionType

string The type of subscription policy on the data source. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added).

subscriptionPolicy

array Details about the Subscription Policy applied to the data source.

globalPolicies

string Details about the Global Policies applied to the data source.

status

string The data source health status.

Request example

The following request gets a data source based on the SQL table name customer_data.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/sqlTableName/customer_data

Response example

    {
  "name": "Dbo Customer Data",
  "recordFormat": "Not Provided",
  "description": null,
  "policyHandler": {
    "visibilitySchema": {
      "fields": [],
      "version": "2021-10-06T19:39:39.145Z"
    },
    "handlerId": 55,
    "dataSourceId": 26
  },
  "sqlSchemaName": "dbo",
  "sqlTableName": "customer_data",
  "blobHandler": {
    "url": "https://1.1.1.1.1/snowflake/testhandler/26",
    "ca": {
      "name": "Certificate Authority Bundle"
    },
    "manualDictionary": false
  },
  "createdBy": 2,
  "deleted": false,
  "type": "queryable",
  "recordCount": 0,
  "rowCount": 1000,
  "documentation": "# Dbo Customer Data",
  "statsExpiration": "2021-11-05T19:37:43.270Z",
  "id": 26,
  "blobHandlerType": "Snowflake",
  "policyHandlerType": "Builder",
  "subscriptionType": "automatic",
  "subscriptionPolicy": {
    "type": "subscription",
    "automaticSubscription": false
  },
  "globalPolicies": null,
  "status": "passed",
  "statusInfo": {
    "sql": {
      "status": "passed",
      "message": "Passed"
    },
    "stats": {
      "status": "passed",
      "lastAttempted": "2021-10-06T19:37:43.337Z"
    },
    "lastAttempt": {
      "date": "2021-10-06T19:39:39.821Z"
    },
    "highCardinality": {
      "status": "passed",
      "lastAttempted": "2021-10-06T19:37:43.337Z"
      }
  },
  "expiration": null,
  "catalogMetadata": null,
  "workspace": null,
  "seeded": false,
  "schemaEvolutionId": 4,
  "columnEvolutionEnabled": true,
  "createdAt": "2021-10-01T14:23:27.225Z",
  "updatedAt": "2021-10-06T19:39:39.145Z",
  "subscribedAsUser": true,
  "subscriptionId": 45,
  "acknowledgeRequired": false,
  "subscriptionStatus": "owner",
  "requestedState": "owner",
  "approved": true,
  "subscriptionExpiration": null,
  "filterId": null
}

Get data source relationships

GET /dataSource/{dataSourceId}/lineage/{type}

Get parent and child relationship records for derived data sources using a specified data source ID.

Query Parameters

AttributeDescriptionRequired

type

string The type of lineage records to return. Options include: parents, children, and all.

Yes

dataSourceId

integer The target data source ID.

Yes

Response Parameters

AttributeDescription

children

array Details of the child data source, including dataSourceId, dataSourceName, projectId, policyHandlerDiff, deleted, createdBy, and createdAt.

parents

array Details of the parent data source, including dataSourceId, dataSourceName, projectId, policyHandlerDiff, deleted, createdBy, and createdAt.

Request example

The following request gets the parent relationship records for the derived data source with the data source ID 4.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/4/lineage/parents

Response example

{
  "parents": [{
    "dataSourceId": 3,
    "dataSourceName": "Public Healthcare Data",
    "deleted": false,
    "createdAt": "2022-08-17T13:41:38.381Z",
    "projectId": 2,
    "projectName": "Derived Data Source",
    "createdBy": "Your Username",
    "policyHandlerDiff": {
      "dsType": "queryable",
      "currentHandlerId": null,
      "previousHandlerId": null
    }
  }]
}

Retrieve a Blob

GET /dataSource/{dataSourceId}/blob/{blobid*}

Retrieve a blob.

Query Parameters

AttributeDescriptionRequired

dataSourceId

integer The data source ID.

Yes

blobId

string The blob ID.

Yes

Response Parameters

The response will download the blobs in a file you specify.

Request example

The following request retrieves a blob.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    --output ./the-blobs-will-be-saved-here
    https://your-immuta-url.com/dataSource/22/blob/22

Response example

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  215k  100  215k    0     0   541k      0 --:--:-- --:--:-- --:--:--  541k

Get a comment by ID

GET /dataSource/{dataSourceId}/comments/{commentId}

Get a comment by ID for the data source.

Query Parameters

AttributeDescriptionRequired

dataSourceId

integer The data source ID.

Yes

commentId

integer The comment ID.

Yes

Response Parameters

AttributeDescription

author

integer The user ID of the user who posted the comment.

parentId

integer The parent comment ID.

resolved

boolean If true, the comment has been resolved.

body

string The text of the comment.

id

integer The comment ID.

createdAt

timestamp When the comment was created.

updatedAt

timestamp When the comment was last updated.

Request example

The following request gets a comment by ID for the data source.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/1/comments/2

Response example

{
  "author": 1,
  "parentId": null,
  "resolved": false,
  "body": "Should this data source be part of the Medical Claims project?",
  "id": 2,
  "createdAt": "2021-09-02T14:14:31.228Z",
  "updatedAt": "2021-09-02T14:14:31.228Z"
}

Get All Comments for a Data Source

GET /dataSource/{dataSourceId}/comments

Get all the comments for the data source.

Query Parameters

AttributeDescriptionRequired

dataSourceId

integer The data source ID.

Yes

Response Parameters

AttributeDescription

author

array The id, name, and email of the author.

resolved

boolean If true, the comment has been resolved.

id

integer The comment ID.

createdAt

timestamp The date and time the comment was created.

updatedAt

timestamp The date and time the comment was updated.

models

array The modelType (such as datasource), modelId, and modelName.

totalreplies

integer The number of replies to the comment.

lastreply

timestamp The date and time of the last reply.

public

boolean If true, the comment is public.

Request example

The following request adds a comment to the data source.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://your-immuta-url.com/dataSource/22/comments

Response example

[{
        "author": {
            "id": 2,
            "name": "Jane Doe",
            "email": "jane.doe@immuta.com"
        },
        "body": "Actually, Billing does not need access, but Customer Service does.",
        "resolved": false,
        "id": 8,
        "createdAt": "2021-10-21T17:03:31.174Z",
        "updatedAt": "2021-10-21T17:03:31.174Z",
        "models": [{
            "modelType": "datasource",
            "modelId": "22",
            "modelName": "Fake Medical Claims 2017"
        }],
        "totalreplies": 0,
        "lastreply": "0001-01-01T00:00:00.000Z",
        "public": true
    },
    {
        "author": {
            "id": 2,
            "name": "Jane Doe",
            "email": "jane.doe@immuta.com"
        },
        "body": "This data source should be accessible to the Docs team and Billing.",
        "resolved": false,
        "id": 7,
        "createdAt": "2021-10-21T17:02:41.390Z",
        "updatedAt": "2021-10-21T17:02:41.390Z",
        "models": [{
            "modelType": "datasource",
            "modelId": "22",
            "modelName": "Fake Medical Claims 2017"
        }],
        "totalreplies": 0,
        "lastreply": "0001-01-01T00:00:00.000Z",
        "public": true
    }
]

Count the comments for a data source

GET /dataSource/{dataSourceId}/comments/count

Count the comments for a data source.

Query Parameters

AttributeDescriptionRequired

dataSourceId

integer The data source ID.

Yes

columns

boolean When true, retrieves comments for columns.

No

queries

array[string] The queries for which to retrieve comments.

No

resolved

boolean If true, will retrieve only resolved comments. If false, will retrieve only unresolved comments. If not set, will retrieve all comments.

No

Response Parameters

AttributeDescription

modelId

integer The model ID.

modelType