REST Catalog Interface

Audience: Integration Developers

Content Summary: Building a REST Catalog shim to integrate a catalog with Immuta requires implementing a simple REST interface. This page documents these endpoints and provides example requests and responses.

Authentication

Immuta requires HTTP Basic Authentication for REST Catalogs. When accessing the /dataSource and /tags endpoints, Immuta will use the configured username and password. If you choose to also protect the human-readable pages with authentication, users will have to authenticate when they visit the page.

Endpoints

/tags

Description

The tags endpoint returns all the tags the catalog can provide. It is used when refreshing Immuta's tag list imported from this catalog.

Schema

This endpoint responds to a GET request with a map of tags to tag objects. In Immuta, a tag is a string and can be nested. This nested structure is encoded in the /tags response by joining multiple tags with a .. For more information on tags and how they are created, managed, and displayed within Immuta, please see our tag documentation.

For the purpose of REST Catalog integrations, a tag has the following format.

<string>[.<string>[.<string>...]]

Tags returned via a REST Catalog must meet the following constraints.

• They must be no longer than 500 characters. Longer tags will not throw an error but will be truncated silently at 500 characters.
• They must be composed of letters, digits, underscores, dashes, and white space characters. A period is semantic as described above.

A tag object has a single property: id, which should be unique across the catalog. If the catalog does not have an ID field available, hashing the tag should provide a unique value that will work for this purpose.

{"id": <integer or string>}

The complete response looks like this.

{
  <Tag>: <Tag Object>[, <Tag>: <Tag Object>[, <Tag>: <Tag Object>...]]
}

Example

curl http://my.custom.catalog/tags

{
  "tag.one": {"id": 1},
  "tag.two": {"id": 2},
  "parent.child": {"id": 3},
  "triple.nested.tag": {"id": 4}
}

/dataSource

Description

The data source endpoint does most of the work. It receives a POST request and returns the mapping of a data source and its columns to the applied tags.

Schema

Request

The request from Immuta is a POST. The body is a simple object containing only the ID of the data source within the catalog.

{
  "catalogMetadata": {
    "id": <integer>
  }
}

Response

The schema for the data source response reuses the tag list from above and also uses the following set of metadata keys for both data sources and columns.

  "catalogMetadata": {
     # The ID of the data source on the catalog.  It is expected to be the same
     # ID that was POSTed in the request.
    "id": <integer>
  },
  # This description will be displayed in the Immuta UI
  "description": <string>,
  "tags": {
    <Tag>: <Tag Object>[, <Tag>: <Tag Object>[, <Tag>: <Tag Object>...]]
  }

{
  <Metadata>, # for the data source overall
  # This maps the column names to metadata objects
  "dictionary": {
    <string>: {
      <Metadata>
    }
  }
}

Examples

curl -X POST http://my.custom.catalog/dataSource \
     --header "Content-Type: application/json" \
     --data '{"catalogMetadata": {"id": 123}}'

{
  "catalogMetadata": {
    "name": "My Test Catalog",
    "id": 123
  },
  "description": "This catalog was created as a documentation example",
  "tags": {
    "tag.one": {
      "id": 1
    },
    "tag.two": {
      "id": 2
    },
    "parent.child": {
      "id": 3
    }
  },
  "dictionary": {
    "some_column_name": {
      "catalogMetadata": {
        "id": 10
      },
      "description": "This column has example data in it",
      "tags": {
        "tag.one": {
          "id": 1
        },
        "triple.nested.tag": {
          "id": 4
        }
      }
    }
  }
}

/dataSource/page/{ID}

Description

This endpoint returns the catalog's human-readable information page for the data source. Immuta links to this page as additional information about the data source.

If the catalog already provides such a page, this endpoint can redirect the user to that page.

Examples

curl http://my.custom.catalog/dataSource/page/123

<html>
  <head>
    <title>data source 123</title>
  </head>
  <body>
    data source 123 is a data source that was created just for documentation.
  </body>
</html>

/column/{ID}

Description

This endpoint returns the catalog's human-readable information page for the column. Immuta links to this page as additional information about the column.

If the catalog already provides such a page, this endpoint can redirect the user to that page.

Examples

curl http://my.custom.catalog/column/10

<html>
  <head>
    <title>data source 123 Column 10</title>
  </head>
  <body>
    Column 10 is full of example data for documentation reasons.
  </body>
</html>