1 of 1

Legacy S3 Access Pattern

Audience: Data Owners, Data Users, and System Administrators
Content Summary: Immuta supports an S3-style REST API, which allows you to communicate with Immuta the same way you would with S3. Consequently, Immuta easily integrates with tools you may already be using to work with S3.

S3 as a Filesystem

In this integration, Immuta implements a single bucket (with data sources broken up as sub-directories under that bucket), since some S3 tools only support the new virtual-hosted style requests.

The three APIs (outlined below) used in this integration support basic AWS functionality; the requests and responses for each are identical to those in S3.

GET Bucket

This request returns the bucket configured within Immuta.

Method

Path

Successful Status Code

GET Bucket Contents

This request returns the contents of the given bucket.

Method

Path

Successful Status Code

GET Object

This request returns a stream from the requested object within Immuta.

Method

Path

Successful Status Code

Example Request:

curl \
    --request GET \
    --header "Authorization: AWS <API KEY>:immuta" \
    https://demo.immuta.com/s3p/immuta/my_data_source/path/to/file/myfile.json

Example: HTTP Request and Response

GET Bucket Example Request:

curl \
    --request GET \
    --header "Authorization: AWS <API KEY>:immuta" \
    https://demo.immuta.com/s3p/immuta?delimiter=/&prefix=my_data_source/path/to/file

Note: There is a single file in the requested directory.

GET Bucket Example Response:

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://doc.s3.amazonaws.com/2006-03-01/">
    <IsTruncated>false</IsTruncated>
    <Marker></Marker>
    <Name>immuta</Name>
    <Prefix>my_data_source/path/to/file</Prefix>
    <MaxKeys>1000</MaxKeys>
    <Delimiter>/</Delimiter>
    <Contents>
        <Key>my_data_source/path/to/file/myfile.json</Key>
        <LastModified>2018-11-05T21:25:04.000Z</LastModified>
        <ETag>5b0810c82a69a70e552cece19b20585fc94b67fe4eaa8b</ETag>
        <Size>389</Size>
        <StorageClass>STANDARD</StorageClass>
        <Owner>
            <ID>Immuta</ID>
            <DisplayName>Immuta</DisplayName>
        </Owner>
    </Contents>
</ListBucketResult>

Example: Using Boto 3 to Download Objects

Boto 3 is the official Amazon Web Services client SDK for Python and is widely used by developers for accessing S3 objects. With Immuta's S3 integration, Immuta users can use boto3 to download policy-enforced files or tables.

The first step is to create a Session object that points to your Immuta endpoint and is authenticated with a user-specific API Key.

import boto3

session = boto3.session.Session()

s3_client = session.client(
    service_name = 's3',
    aws_access_key_id = '<YOUR_USER_API_KEY>',
    aws_secret_access_key = 'immuta',
    endpoint_url = 'https://<YOUR_IMMUTA_URL>:443/s3p'
)

To find out what objects are available for download, you can list the objects in the immuta bucket. To filter down to a particular data source, pass in a Prefix that corresponds to the SQL table name of your Immuta data source.

bucket_contents = s3_client.list_objects(
    Bucket = 'immuta',
    Delimiter = '/',
    Prefix = '<SQL_TABLE_NAME>'
).get("Contents")

print(bucket_contents[0])
    {
        'Key': '<SQL_TABLE_NAME>/<SINGLE_OBJECT_KEY>',
        'ETag': 'aa0492082b95c5d8bb90377a006e...',
        'StorageClass': 'STANDARD',
        'Owner': {'DisplayName': 'Immuta', 'ID': 'Immuta'}
    }

Once you have an object key, you can use the download_file method to download the object to your local development environment.

s3_client.download_file(
    Bucket = "immuta",
    Key = "<SQL_TABLE_NAME>/<SINGLE_OBJECT_KEY>",
    Filename = "<OUTPUT_FILE_PATH>"
)

Legacy S3 Access Pattern

Audience: Data Owners, Data Users, and System Administrators
Content Summary: Immuta supports an S3-style REST API, which allows you to communicate with Immuta the same way you would with S3. Consequently, Immuta easily integrates with tools you may already be using to work with S3.

S3 as a Filesystem

In this integration, Immuta implements a single bucket (with data sources broken up as sub-directories under that bucket), since some S3 tools only support the new virtual-hosted style requests.

The three APIs (outlined below) used in this integration support basic AWS functionality; the requests and responses for each are identical to those in S3.

GET Bucket

This request returns the bucket configured within Immuta.

Method

Path

Successful Status Code

GET Bucket Contents

This request returns the contents of the given bucket.

Method

Path

Successful Status Code

GET Object

This request returns a stream from the requested object within Immuta.

Method

Path

Successful Status Code

Example Request:

curl \
    --request GET \
    --header "Authorization: AWS <API KEY>:immuta" \
    https://demo.immuta.com/s3p/immuta/my_data_source/path/to/file/myfile.json

Example: HTTP Request and Response

GET Bucket Example Request:

curl \
    --request GET \
    --header "Authorization: AWS <API KEY>:immuta" \
    https://demo.immuta.com/s3p/immuta?delimiter=/&prefix=my_data_source/path/to/file

Note: There is a single file in the requested directory.

GET Bucket Example Response:

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://doc.s3.amazonaws.com/2006-03-01/">
    <IsTruncated>false</IsTruncated>
    <Marker></Marker>
    <Name>immuta</Name>
    <Prefix>my_data_source/path/to/file</Prefix>
    <MaxKeys>1000</MaxKeys>
    <Delimiter>/</Delimiter>
    <Contents>
        <Key>my_data_source/path/to/file/myfile.json</Key>
        <LastModified>2018-11-05T21:25:04.000Z</LastModified>
        <ETag>5b0810c82a69a70e552cece19b20585fc94b67fe4eaa8b</ETag>
        <Size>389</Size>
        <StorageClass>STANDARD</StorageClass>
        <Owner>
            <ID>Immuta</ID>
            <DisplayName>Immuta</DisplayName>
        </Owner>
    </Contents>
</ListBucketResult>

Example: Using Boto 3 to Download Objects

The first step is to create a Session object that points to your Immuta endpoint and is authenticated with a user-specific API Key.

import boto3

session = boto3.session.Session()

s3_client = session.client(
    service_name = 's3',
    aws_access_key_id = '<YOUR_USER_API_KEY>',
    aws_secret_access_key = 'immuta',
    endpoint_url = 'https://<YOUR_IMMUTA_URL>:443/s3p'
)

bucket_contents = s3_client.list_objects(
    Bucket = 'immuta',
    Delimiter = '/',
    Prefix = '<SQL_TABLE_NAME>'
).get("Contents")

print(bucket_contents[0])
    {
        'Key': '<SQL_TABLE_NAME>/<SINGLE_OBJECT_KEY>',
        'ETag': 'aa0492082b95c5d8bb90377a006e...',
        'StorageClass': 'STANDARD',
        'Owner': {'DisplayName': 'Immuta', 'ID': 'Immuta'}
    }

Once you have an object key, you can use the download_file method to download the object to your local development environment.

s3_client.download_file(
    Bucket = "immuta",
    Key = "<SQL_TABLE_NAME>/<SINGLE_OBJECT_KEY>",
    Filename = "<OUTPUT_FILE_PATH>"
)