Amazon EMR

Audience: System Administrators
Content Summary: This tutorial will guide you through the process of spinning up an Amazon Elastic Map Reduce cluster with Immuta's Hadoop and Spark security plugins installed.

Deprecation notice

Support for this integration has been deprecated.

Introduction

This tutorial contains examples using the AWS CLI. These examples are conceptual in nature and will require modification to adapt to your exact deployment needs. If you wish to quickly familiarize yourself with Immuta's EMR integration, please visit the Quickstart Installation Guide for Immuta on AWS EMR.

Supported EMR Versions

This deployment is tested and known to work on the EMR releases listed below.

5.17.0
5.18.0
5.19.0
5.20.0
5.21.0
5.22.0
5.23.0
5.24.0
5.25.0
5.26.0
5.27.0
5.28.0
5.29.0
5.30.0
5.31.0
5.32.0

Create Prerequisite AWS Resources

In addition to the EMR cluster itself, Immuta requires a handful of additional AWS resources in order to function properly.

Immuta Bootstrap Bucket

In order to bootstrap the EMR cluster with Immuta's software bundle and startup scripts, you will need to create an S3 bucket to hold these artifacts.

In this guide, the bucket is referenced by the placeholder $BOOTSTRAP_BUCKET. You should substitute this bucket name for a unique bucket name of your choosing. The bucket must contain all artifacts listed below. These artifacts can be found at Immuta Downloads.

s3://$BOOTSTRAP_BUCKET/immuta-bootstrap
s3://$BOOTSTRAP_BUCKET/immuta-bootstrap.tar.gz
s3://$BOOTSTRAP_BUCKET/immuta_bundle-$IMMUTA_VERSION.tar.gz
s3://$BOOTSTRAP_BUCKET/install.sh

Immuta Data IAM Role

Immuta's Spark integration relies on an IAM role policy that has access to the S3 buckets where your sensitive data is stored. Note that the EC2 Instance Roles for your EMR cluster should not have access to these buckets. Immuta will broker access to the data in these buckets to authorized users.

Create Immuta Data IAM Policy

Modify the JSON data below to include the correct name of your data bucket(s), and save as immuta_data_iam_policy.json.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:Head*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::$DATA_BUCKET_1",
                "arn:aws:s3:::$DATA_BUCKET_2",
                "arn:aws:s3:::$DATA_BUCKET_1/*",
                "arn:aws:s3:::$DATA_BUCKET_2/*"
            ]
        }
    ]
}

If you are leveraging Immuta's Native S3 Workspace capability, you must also give the Immuta data IAM role full control of the workspace bucket or folder.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:Head*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::$DATA_BUCKET_1",
                "arn:aws:s3:::$DATA_BUCKET_2",
                "arn:aws:s3:::$DATA_BUCKET_1/*",
                "arn:aws:s3:::$DATA_BUCKET_2/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::$WORKSPACE_BUCKET",
                "arn:aws:s3:::$WORKSPACE_BUCKET/*"
            ]
        }
    ]
}

Now you can run the following command to create the Immuta IAM user policy.

aws iam create-policy \
    --policy-name immuta_emr_data_policy \
    --policy-document file://immuta_data_iam_policy.json

Create Immuta Data IAM Role

The IAM role that brokers access to S3 data must be able to assume the cluster node instance roles, and vice versa. Since this a cycle, you will need to create both roles with generic trust policies, and then update them after both roles are created.

Create a file called immuta_data_role_trust_policy_generic.json as seen below.

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Principal":{
            "AWS":"arn:aws:iam::$AWS_ACCOUNT_ID:role/EMR_EC2_DefaultRole"
         },
         "Action":"sts:AssumeRole"
      }
   ]
}

After creating the immuta_data_role_trust_policy_generic.json file from above, run the following command to create the Immuta data IAM role. Note that you will be using the generic IAM role trust policy that you created in the previous step. This will be updated when both the data and instance IAM roles are created.

aws iam create-role \
  --role-name immuta_emr_data_role \
  --assume-role-policy-document "file://immuta_data_role_trust_policy_generic.json"

Next you will need to attach the IAM policy that allows access to your protected data in S3.

aws iam attach-role-policy \
    --policy-arn arn:aws:iam::$AWS_ACCOUNT_ID:policy/immuta_emr_data_policy \
    --role-name immuta_emr_data_role

Create Immuta Instance IAM Policy

Modify the JSON data below to include the correct name of your bootstrap bucket, and save as immuta_emr_instance_policy.json.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Resource": "*",
            "Action": [
                "ec2:Describe*",
                "elasticmapreduce:Describe*",
                "elasticmapreduce:ListBootstrapActions",
                "elasticmapreduce:ListClusters",
                "elasticmapreduce:ListInstanceGroups",
                "elasticmapreduce:ListInstances",
                "elasticmapreduce:ListSteps"
            ]
        },
        {
            "Effect": "Allow",
            "Resource": "arn:aws:sqs:*:123456789012:AWS-ElasticMapReduce-*",
            "Action": [
                "sqs:CreateQueue",
                "sqs:DeleteQueue",
                "sqs:DeleteMessage",
                "sqs:DeleteMessageBatch",
                "sqs:GetQueueAttributes",
                "sqs:GetQueueUrl",
                "sqs:PurgeQueue",
                "sqs:ReceiveMessage"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*Object"
            ],
            "Resource": [
                "arn:aws:s3:::$BOOTSTRAP_BUCKET/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::$BOOTSTRAP_BUCKET"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "secretsmanager:*",
            "Resource": [
                "arn:aws:secretsmanager:$AWS_REGION:$AWS_ACCOUNT_ID:secret:immuta-emr-secret-??????",
                "arn:aws:secretsmanager:$AWS_REGION>:$AWS_ACCOUNT_ID:secret:immuta-kerberos-secret-??????"
            ]
        }
    ]
}

Note that the above policy is derived from the Minimal EMR role for EC2 (instance profile) policy described in Amazon's Best Practices for Securing Amazon EMR guide. You may need to tune this policy based on your organization's environment and needs.

After creating the immuta_emr_instance_policy.json file from above, run the following command to create the Immuta EMR Instance policy.

aws iam create-policy \
    --policy-name immuta_emr_instance_policy \
    --policy-document file://immuta_emr_instance_policy.json