Configure an Amazon S3 Integration and Create an S3 Data Source

The Amazon S3 resource allows you to create, configure, and manage your S3 integration. In this integration, Immuta provides coarse-grained access controls for data in S3 by performing permission grants using the Access Grants API so that users don't have to manage individual IAM policies themselves.

Use the /integrations endpoint to

• configure an Amazon S3 integration

• get an Amazon S3 integration

• get all integrations

• update an Amazon S3 integration

• delete an Amazon S3 integration

• create an Amazon S3 data source

Requirements

• S3 integration enabled in Immuta; contact your Immuta representative to enable this integration

• Write policies private preview enabled for your account; contact your Immuta representative to get this feature enabled

• No location is registered in your AWS Access Grants instance before configuring the integration in Immuta

Permissions

• APPLICATION_ADMIN Immuta permission to configure the integration

• CREATE_S3_DATASOURCE Immuta permission to register S3 prefixes

• The AWS account credentials or optional AWS IAM role you provide Immuta to configure the integration must

  • have ownership of the buckets Immuta will enforce policies on

  • have the permissions to perform the following actions to create locations and issue grants:

    • accessgrantslocation resource:

      • s3:CreateAccessGrant

      • s3:DeleteAccessGrantsLocation

      • s3:GetAccessGrantsLocation

      • s3:UpdateAccessGrantsLocation

    • accessgrantsinstance resource:

      • s3:CreateAccessGrantsInstance

      • s3:CreateAccessGrantsLocation

      • s3:DeleteAccessGrantsInstance

      • s3:GetAccessGrantsInstance

      • s3:GetAccessGrantsInstanceForPrefix

      • s3:GetAccessGrantsInstanceResourcePolicy

      • s3:ListAccessGrants

      • s3:ListAccessGrantsLocations

    • accessgrant resource:

      • s3:DeleteAccessGrant

      • s3:GetAccessGrant

    • bucket resource: s3:ListBucket

    • role resource:

      • iam:GetRole

      • iam:PassRole

    • all resources: s3:ListAccessGrantsInstances

Set up S3 Access Grants instance

1. Follow AWS documentation to create an Access Grants instance using the S3 console, AWS CLI, AWS SDKs, or the REST API. AWS supports one Access Grants instance per region per AWS account.

2. Follow the instructions at the top of the "Register a location" page in AWS documentation to create an AWS IAM role and edit the trust policy to give the S3 Access Grants service principal access to this role in the resource policy file. You will add this role to your integration configuration in Immuta so that Immuta can register this role with your Access Grants location. The policy should include at least the following permissions, but might need additional permissions depending on other local setup factors. An example trust policy is provided below.

   • sts:AssumeRole

   • sts:SetSourceIdentity

{
  "Version": "2012-10-17",
    "Statement": [
    {
      "Sid": "Stmt1234567891011",
      "Effect": "Allow",
      "Principal": {
        "Service":"access-grants.s3.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole", 
        "sts:SetSourceIdentity"
      ]
    }
  ]
}           

1. Follow the instructions at the top of the "Register a location" page in AWS documentation to create an IAM policy with the following permissions, and attach the policy to the IAM role you created to grant the permissions to the role. The policy should include the following permissions. An example policy is provided below.

• s3:GetObject

• s3:GetObjectVersion

• s3:GetObjectAcl

• s3:GetObjectVersionAcl

• s3:ListMultipartUploadParts

• s3:PutObject

• s3:PutObjectAcl

• s3:PutObjectVersionAcl

• s3:DeleteObject

• s3:DeleteObjectVersion

• s3:AbortMultipartUpload

• s3:ListBucket

• s3:ListAllMyBuckets

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ObjectLevelReadPermissions",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetObjectAcl",
                "s3:GetObjectVersionAcl",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                <bucket arn>
            ]
        },
        {
            "Sid": "ObjectLevelWritePermissions",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectVersionAcl",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                <bucket arn>
            ]
        },
        {
            "Sid": "BucketLevelReadPermissions",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:ListBucket"
            ],
            "Resource": [
                <bucket arn>
            ]
        }
    ]
}

If you use server-side encryption with AWS Key Management Service (AWS KMS) keys to encrypt your data, the following permissions are required for the IAM role in the policy. If you do not use this feature, do not include these permissions in your IAM policy:

• kms:Decrypt

• kms:GenerateDataKey

1. Opt to create an AWS IAM role that Immuta can use to create Access Grants locations and issue grants. This role must have the S3 permissions listed in the permissions section. An example policy is provided below.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "RolePermissions",
            "Effect": "Allow",
            "Action": [
                "iam:GetRole",
                "iam:PassRole"
            ],
            "Resource": "<role_arn>"
        },
        {
            "Sid": "AccessGrants",
            "Effect": "Allow",
            "Action": [
                "s3:CreateAccessGrant",
                "s3:DeleteAccessGrantsLocation",
                "s3:GetAccessGrantsLocation",
                "s3:CreateAccessGrantsLocation",
                "s3:GetAccessGrantsInstance",
                "s3:GetAccessGrantsInstanceForPrefix",
                "s3:GetAccessGrantsInstanceResourcePolicy",
                "s3:ListAccessGrants",
                "s3:ListAccessGrantsLocations",
                "s3:ListAccessGrantsInstances",
                "s3:DeleteAccessGrant",
                "s3:GetAccessGrant"
            ],
            "Resource": [
                "<access_grants_instance_arn>"
            ]
        }
    ]
}

Configure the integration in Immuta

curl -X 'POST' \
    'https://<your-immuta-url.com>/integrations' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: <your-api-key>' \
    -d '{
    "type": "Native S3",
    "autoBootstrap": false,
    "config": {
      "name": "<name-of-your-integration>",
      "awsAccountId": "<your-aws-account-id>",
      "awsRegion": "<your-aws-region>",
      "awsLocationRole": "<arn:aws:iam::your-location-role-arn:role/access-grants-instance-role>",
      "awsLocationPath": "<your-s3-location-path>",
      "authenticationType": "accessKey",
      "awsAccessKeyId": "<your-access-key-id>",
      "awsSecretAccessKey": "<your-secret-access-key>"
    }
    }'

curl -X 'POST' \
    'https://<your-immuta-url.com>/integrations' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: <your-api-key>' \
    -d '{
    "type": "Native S3",
    "autoBootstrap": false,
    "config": {
      "name": "<name-of-your-integration>",
      "awsAccountId": "<your-aws-account-id>",
      "awsRegion": "<your-aws-region>",
      "awsLocationRole": "<arn:aws:iam::your-location-role-arn:role/access-grants-instance-role>",
      "awsLocationPath": "<your-s3-location-path>",
      "authenticationType": "auto"
    }
    }'

See the config object description for parameter definitions, value types, and additional configuration options.

Response

The response returns the status of the S3 integration configuration connection. See the response schema reference for details about the response schema.

{
  "id": "123456789",
  "status": "creating",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "There is no existing integration matching this configuration",
      "status": "passed"
    },
    {
      "name": "The provided integration name is unique across Immuta S3 integrations",
      "status": "passed"
    },
    {
      "name": "The provided access grants location role is a valid ARN format",
      "status": "passed"
    },
    {
      "name": "The provided AWS credentials allow fetching the caller's identity via the AWS STS API",
      "status": "passed"
    },
    {
      "name": "An AWS Access Grants instance is configured in the provided AWS account and region",
      "status": "passed"
    },
    {
      "name": "The provided S3 path exists and Immuta can list prefixes",
      "status": "passed"
    },
    {
      "name": "An AWS Access Grants location does not yet exist for the provided path",
      "status": "passed"
    }
    ]
  }
}

{
  "statusCode": 409,
  "error": "Conflict",
  "message": "Access Grants location already exists on provided path."
}

Get an integration

curl -X 'GET' \
    'https://<your-immuta-url.com>/integrations/{id}' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: <your-api-key>'

1. Copy the request example.

2. Replace the Immuta URL and API key with your own.

3. Replace the {id} request parameter with the unique identifier of the integration you want to get. Alternatively, you can get a list of all integrations and their IDs with the GET /integrations endpoint.

Response

The response returns an S3 integration configuration. See the response schema reference for details about the response schema. An unsuccessful response returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.

{
  "id": "123456789",
  "status": "enabled",
  "validationResults": {
    "status": "passed",
    "validationTests": [
      {
        "name": "There is no existing integration matching this configuration",
        "status": "passed"
      },
      {
        "name": "The provided integration name is unique across Immuta S3 integrations",
        "status": "passed"
      },
      {
        "name": "The provided access grants location role is a valid ARN format",
        "status": "passed"
      },
      {
        "name": "The provided AWS credentials allow fetching the caller's identity via the AWS STS API",
        "status": "passed"
      },
      {
        "name": "An AWS Access Grants instance is configured in the provided AWS account and region",
        "status": "passed"
      },
      {
        "name": "The provided S3 path exists and Immuta can list prefixes",
        "status": "passed"
      },
      {
        "name": "An AWS Access Grants location does not yet exist for the provided path",
        "status": "passed"
      }
    ]
  },
  "type": "Native S3",
  "autoBootstrap": false,
  "config": {
    "port": 443,
    "name": "S3 integration",
    "awsAccountId": "123456789",
    "awsRegion": "us-east-1",
    "awsLocationRole": "arn:aws:iam::123456789:role/access-grants-instance-role",
    "awsLocationPath": "s3://",
    "authenticationType": "accessKey",
    "awsAccessKeyId": "123456789",
    "awsSecretAccessKey": "123456789"
  }
  }

Get all integrations

curl -X 'GET' \
    'https://<your-immuta-url.com>/integrations' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: <your-api-key>'

1. Copy the request example.

2. Replace the Immuta URL and API key with your own.

Response

The response returns the configuration for all integrations. See the response schema reference for details about the response schema. An unsuccessful request returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.

[
  {
    "id": "1",
    "status": "enabled",
    "validationResults": {
      "status": "passed",
      "validationTests": [
      {
        "name": "Initial Validation: Basic Connection Test",
        "status": "passed"
      },
      {
        "name": "Initial Validation: Default Warehouse Access Test",
        "status": "passed",
        "result": []
      },
      {
        "name": "Initial Validation: Validate access to Privileged Role",
        "status": "passed",
        "result": []
      },
      {
        "name": "Validate Automatic: Database Does Not Exist",
        "status": "passed"
      },
      {
        "name": "Validate Automatic: Impersonation Role Does Not Exist",
        "status": "skipped"
      },
      {
        "name": "Validate Automatic Bootstrap User Grants",
        "status": "passed"
      }
    ] },
    "type": "Snowflake",
    "autoBootstrap": true,
    "config": {
      "host": "organization.us-east-1.snowflakecomputing.com",
      "warehouse": "SAMPLE_WAREHOUSE",
      "database": "SNOWFLAKE_SAMPLE_DATA",
      "port": 443,
      "audit": {
        "enabled": false
        },
      "workspaces": {
        "enabled": false
      },
      "impersonation": {
        "enabled": false
      },
      "lineage": {
        "enabled": false
      },
      "authenticationType": "userPassword",
      "username": "<REDACTED>",
      "password": "<REDACTED>",
      "role": "ACCOUNTADMIN"
    }
    },
  {
    "id": "123456789",
    "status": "enabled",
    "validationResults": {
      "status": "passed",
      "validationTests": [
        {
          "name": "There is no existing integration matching this configuration",
          "status": "passed"
        },
        {
          "name": "The provided integration name is unique across Immuta S3 integrations",
          "status": "passed"
        },
        {
          "name": "The provided access grants location role is a valid ARN format",
          "status": "passed"
        },
        {
          "name": "The provided AWS credentials allow fetching the caller's identity via the AWS STS API",
          "status": "passed"
        },
        {
          "name": "An AWS Access Grants instance is configured in the provided AWS account and region",
          "status": "passed"
        },
        {
          "name": "The provided S3 path exists and Immuta can list prefixes",
          "status": "passed"
        },
        {
          "name": "An AWS Access Grants location does not yet exist for the provided path",
          "status": "passed"
        }
      ] },
      "type": "Native S3",
      "autoBootstrap": false,
      "config": {
        "port": 443,
        "name": "S3 integration",
        "awsAccountId": "123456789",
        "awsRegion": "us-east-1",
        "awsLocationRole": "arn:aws:iam::123456789:role/access-grants-instance-role",
        "awsLocationPath": "s3://",
        "authenticationType": "accessKey",
        "awsAccessKeyId": "123456789",
        "awsSecretAccessKey": "123456789"
      }
  }
]

Update an integration configuration

Copy the request example, and replace the values with your own as directed to update the integration settings. The example provided uses JSON format, but the request also accepts YAML.

See the config object description for parameter definitions, value types, and additional configuration options.

curl -X 'PUT' \
    'https://<your-immuta-url.com>/integrations/{id}' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: <your-api-key>' \
    -d '{
    "type": "Native S3",
    "autoBootstrap": false,
    "config": {
      "name": "<your-edited-integration-name>",
      "awsAccountId": "<your-aws-account-id>",
      "awsRegion": "<your-aws-region>",
      "awsLocationRole": "<arn:aws:iam::your-location-role-arn:role/access-grants-instance-role>",
      "awsLocationPath": "<your-s3-location-path>",
      "authenticationType": "<accessKey-or-auto>",
      "awsAccessKeyId": "<your-aws-access-key-id>",
      "awsSecretAccessKey": "<your-secret-access-key>"
    }
    }'

1. Replace the Immuta URL and API key with your own.

2. Replace the {id} request parameter with the unique identifier of the integration you want to update.

3. Change the config values from above to your own. The editable values listed below are the only parameters that can change from the integration's existing configuration. The required parameters listed below must match the integration's existing configuration.

   • editable values:

     • name is the name for the integration that is unique across all Amazon S3 integrations configured in Immuta.

     • authenticationType is the method used to authenticate with AWS when configuring the S3 integration (accepted values are auto or accessKey).

     • awsAccessKeyId is the AWS access key ID for the AWS account editing the integration.

     • awsSecretAccessKey is the AWS secret access key for the AWS account editing the integration.

   • required values from existing configuration:

     • awsRoleToAssume is the optional AWS IAM role ARN Immuta assumes when interacting with AWS.

     • awsAccountId is the ID of your AWS account.

     • awsRegion is the account's AWS region (such as us-east1).

     • awsLocationRole is the AWS IAM role ARN assigned to the base access grants location. This is the role the AWS Access Grants service assumes to vend credentials to the grantee.

     • awsLocationPath is the base S3 location that Immuta will use for this connection when registering S3 data sources. This path must be unique across all S3 integrations configured in Immuta.

Response

The response returns the status of the Amazon S3 integration configuration connection. See the response schema reference for details about the response schema.

{
  "id": "123456789",
  "status": "editing",
  "validationResults": {
    "status": "passed",
    "validationTests": [
    {
      "name": "There is no existing integration matching this configuration",
      "status": "passed"
    },
    {
      "name": "The provided integration name is unique across Immuta S3 integrations",
      "status": "passed"
    },
    {
      "name": "The provided access grants location role is a valid ARN format",
      "status": "passed"
    },
    {
      "name": "The provided AWS credentials allow fetching the caller's identity via the AWS STS API",
      "status": "passed"
    },
    {
      "name": "An AWS Access Grants instance is configured in the provided AWS account and region",
      "status": "passed"
    },
    {
      "name": "The provided S3 path exists and Immuta can list prefixes",
      "status": "passed"
    },
    {
      "name": "An AWS Access Grants location does not yet exist for the provided path",
      "status": "passed"
    }
    ]
  }
}

{
  "statusCode": 409,
  "error": "Conflict",
  "message": "Unable to edit integration with ID 123456789 in current state editing."
}

Delete an integration

curl -X 'DELETE' \
    'https://<your-immuta-url.com>/integrations/{id}' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: <your-api-key>'

1. Copy the request example.

2. Replace the Immuta URL and API key with your own.

3. Replace the {id} request parameter with the unique identifier of the integration you want to delete.

Response

The response returns the status of the S3 integration configuration that has been deleted. See the response schema reference for details about the response schema. An unsuccessful request returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.

{
  "id": "123456789",
  "status": "deleting",
  "validationResults": {
    "status": "passed",
    "validationTests": [
      {
        "name": "The provided access grants location role is a valid ARN format",
        "status": "passed"
      },
      {
        "name": "The provided AWS credentials allow fetching the caller's identity via the AWS STS API",
        "status": "passed"
      },
      {
        "name": "An AWS Access Grants instance is configured in the provided AWS account and region",
        "status": "passed"
      },
      {
        "name": "The provided S3 path exists and Immuta can list prefixes",
        "status": "passed"
      }
    ]
  }
}

Create a data source

curl -X 'POST' \
    'https://<your-immuta-url.com>/native-s3/handler' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: <your-api-key>' \
    -d '{
    "type": "Native S3",
    "integrationId": <id-of-your-s3-integration>,
    "dataSources": [
      {
        "dataSourceName": "<name-of-data-source>",
        "prefix": "</data-source-prefix>"
      }
    ]
    }'

1. Copy the request example. The example uses JSON format, but the request also accepts YAML.

2. Replace the Immuta URL and API key with your own.

3. Change the integrationsID to the ID of your S3 integration. This ID can be retrieved with the GET /integrations request.

4. Change the dataSources values to your own, where

   • dataSourceName is the name of your data source.

   • prefix creates a data source for the prefix, bucket, or object provided in the path. If the data source prefix ends in a wildcard (), it protects all items starting with that prefix. If the data source prefix ends without a wildcard (), it protects a single object.

See the S3 data source payload description for parameter definitions and value types.

Response

The response returns the ID, name, and prefix of the data source. See the response schema reference for details about the response schema. An unsuccessful response returns the status code and an error message. See the HTTP status codes and error messages for a list of statuses, error messages, and troubleshooting guidance.