Azure Data Lake Storage Gen2 Data Source API Reference Guide

Deprecation notice

Support for this database has been deprecated.

The adls-gen2 endpoint allows you to connect and manage Azure Data Lake Storage Gen2 data sources in Immuta.

Note

Additional fields may be included in some responses you receive; however, these attributes are for internal purposes and are therefore undocumented.

Create a data source

EndpointQuery ParametersPayload ParametersResponse Parameters

Endpoint

Method	Path	Purpose
POST	`/adls-gen2/handler`	Save the provided connection information as a data source.

Query Parameters

None.

Payload Parameters

Attribute	Description	Required
private	`boolean` When `false`, the data source will be publicly available in the Immuta UI.	Yes
blobHandler	`array[object]` A list of full URLs providing the locations of all blob store handlers to use with this data source.	Yes
blobHandlerType	`string` Describes the type of underlying blob handler that will be used with this data source (e.g., `MS SQL`).	Yes
recordFormat	`string` The data format of blobs in the data source, such as `json`, `xml`, `html`, or `jpeg`.	Yes
type	`string` The type of data source: `ingested` (metadata will exist in Immuta) or `queryable` (metadata is dynamically queried).	Yes
name	`string` The name of the data source. It must be unique within the Immuta instance.	Yes
sqlTableName	`string` A string that represents this data source's table in the Query Engine.	Yes
organization	`string` The organization that owns the data source.	Yes
category	`string` The category of the data source.	No
description	`string` The description of the data source.	No
hasExamples	`boolean` When `true`, the data source contains examples.	No

Response Parameters

Attribute	Description
id	`integer` The handler ID.
dataSourceId	`integer` The ID of the data source.
warnings	`string` This message describes issues with the created data source, such as the data source being unhealthy.
connectionString	`string` The connection string used to connect the data source to Immuta.

Request example

This request creates two Azure Data Lake Storage Gen2 data sources.

curl \
    --request POST \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    --data @example-payload.json \
    https://demo.immuta.com/adls-gen2/handler

Payload example

{
  "handler": {
    "metadata": {
      "tagAttributes": [],
      "eventTimeAttribute": "",
      "useDirectoryForTags": false,
      "sasToken": "your-sas-token",
      "sasTokenUrl": "your-sas-token-url",
      "authenticationMethod": "sasToken",
      "fileSystem": "bens-test-data"
    }
  },
  "dataSource": {
    "blobHandler": {
      "scheme": "https",
      "url": ""
    },
    "expiration": "2021-11-19T04:59:59.999Z",
    "blobHandlerType": "Azure Data Lake Storage Gen2",
    "recordFormat": "",
    "type": "ingested",
    "name": "Test-Data",
    "sqlTableName": "test_data"
  }
}

Response example

{
  "id": 17,
  "dataSourceId": 17
}

Get information about a data source

EndpointQuery ParametersResponse Parameters

Endpoint

Method	Path	Purpose
GET	`/adls-gen2/handler/{handlerId}`	Get the handler metadata associated with the provided handler ID.

Query Parameters

Attribute	Description	Required
handlerId	`integer` The ID of the handler.	Yes
skipCache	`boolean` When `true`, will skip the handler cache when retrieving metadata.	No

Response Parameters

Attribute	Description
body	`array[object]` Metadata about the data source, including the data source ID, schema, database, and connection string.

Request example

This request returns metadata for the handler with the ID 17.

curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    https://demo.immuta.com/adls-gen2/handler/17

Response example

{
  "dataSourceId": 17,
  "metadata": {
    "fileSystem": "bens-test-data",
    "accountName": "your-account-name",
    "sasTokenUrl": "https://your.blob.core.windows.net/",
    "ingestUserId": "azure data lake storage gen2_indexer_17",
    "tagAttributes": [],
    "dataSourceName": "Test-Data",
    "refreshInterval": 0,
    "eventTimeAttribute": "",
    "useDirectoryForTags": false,
    "authenticationMethod": "sasToken"
  },
  "type": "azureDataLakeStorageGen2Handler",
  "connectionString": "your-connection-string",
  "remoteTableDescription": null,
  "id": 17,
  "createdAt": "2021-10-19T16:41:16.161Z",
  "updatedAt": "2021-10-19T16:41:16.390Z"
}

Update a specific data source

EndpointQuery ParametersPayload ParametersResponse Parameters

Endpoint

Method	Path	Purpose
PUT	`/adls-gen2/handler/{handlerId}`	Update the data source metadata associated with the provided handler ID. This endpoint does not perform partial updates, but will allow the dictionary to be omitted. In this case, it uses the current dictionary.

Query Parameters

Attribute	Description	Required
handlerId	`integer` The ID of the handler.	Yes
skipCache	`boolean` When `true`, will skip the handler cache when retrieving metadata.	No

Payload Parameters

Attribute	Description	Required
handler	`metadata` Includes metadata about the handler, such as `ssl`, `port`, `database`, `hostname`, `username`, and `password`.	Yes
connectionString	`string` The connection string used to connect to the data source.	Yes

Response Parameters

Attribute	Description
id	`integer` The ID of the handler.
ca	`string` The certificate authority.
columns	`array[object]` This is a Data Dictionary object, which provides metadata about the columns in the data source, including the name and data type of the column.

Request example

This request updates the metadata for the data source with the handler ID 17.

curl \
    --request PUT \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    --data @example-payload.json \
    https://demo.immuta.com/adls-gen2/handler/17

Payload example

The payload below updates the dataSourceName to Customer Details.

{
  "dataSourceId": 17,
  "metadata": {
    "fileSystem": "bens-test-data",
    "accountName": "your-account-name",
    "sasTokenUrl": "https://your.blob.core.windows.net/",
    "ingestUserId": "azure data lake storage gen2_indexer_17",
    "tagAttributes": [],
    "dataSourceName": "Customer Details",
    "refreshInterval": 0,
    "eventTimeAttribute": "",
    "useDirectoryForTags": false,
    "authenticationMethod": "sasToken"
  },
  "type": "azureDataLakeStorageGen2Handler",
  "connectionString": "your-connection-string",
  "remoteTableDescription": null,
  "id": 17,
  "createdAt": "2021-10-19T16:41:16.161Z",
  "updatedAt": "2021-10-19T16:41:16.390Z"
}

Response example

{
  "id": 17,
  "ca": ["-----BEGIN CERTIFICATE-----\ncertificatedata\n-----END CERTIFICATE-----"]
}