Resetting Passwords

The following procedure walks through the process of changing passwords for the database users in the Immuta Database.

The commands outlined here will need to be altered depending on your Helm release name and chosen passwords. Depending on your environment, there may be other changes required for the commands to complete successfully, including, but not limited to, Kubernetes namespace, kubectl context, and Helm values file name.

This process results in downtime.

Helm Values

1. Scale database StatefulSet to 1 replica:

   Copy

   kubectl scale statefulset <release-name>-immuta-database --replicas 1

2. Change database.superuserPassword:

   1. Alter Postgres user password:

      Copy

      kubectl exec <release-name>-immuta-database-0 -- \
        psql -d bometadata -c \
        "ALTER USER postgres WITH ENCRYPTED PASSWORD '<new-password>'

   2. Update database.superuserPassword with <new-password> in immuta-values.yaml.

3. Change database.replicationPassword:

   1. Alter replicator user password:

      Copy

      kubectl exec <release-name>-immuta-database-0 -- \
        psql -d bometadata -c \
        "ALTER USER replicator WITH ENCRYPTED PASSWORD '<new-password>'"

   2. Update database.replicationPassword with <new-password> in immuta-values.yaml.

4. Change database.password:

   1. Alter bometa user password:

      Copy

      kubectl exec <release-name>-immuta-database-0 -- \
        psql -d bometadata -c \
        "ALTER USER bometa WITH ENCRYPTED PASSWORD '<new-password>'"

   2. Update database.password with <new-password> in immuta-values.yaml.

5. Update database.patroniApiPassword with <new-password> in immuta-values.yaml.

6. Run helm upgrade to persist the changes and scale the database StatefulSet up:

   Copy

   helm upgrade --reuse-values <release-name> immuta/immuta

7. Restart web pods:

   Copy

   kubectl rollout restart deployment/<release-name>-immuta-web

Kubernetes Secret

Users have the option to use an existing Kubernetes secret for Immuta database passwords used in Helm installations.

1. Update your existingSecret values in your Kubernetes environment.

2. Get the current replica counts:

   Copy

   DATABASE_REPLICA_COUNT=$(kubectl get statefulset --selector "app.kubernetes.io/component=database" --output=jsonpath='{.items[0].status.replicas}')

3. Scale database StatefulSet to 1 replica:

   Copy

   kubectl scale statefulset <release-name>-immuta-database --replicas 1

4. Change the value corresponding to database.superuserPassword in the existing Kubernetes Secret.

5. Alter Postgres user password:

   Copy

   kubectl exec <release-name>-immuta-database-0 -- \
     psql -d bometadata -c \
     "ALTER USER postgres WITH ENCRYPTED PASSWORD '<new-password>'"

6. Change the value corresponding to database.replicationPassword in the existing Kubernetes Secret.

7. Alter replicator user password:

   Copy

   kubectl exec <release-name>-immuta-database-0 -- \
     psql -d bometadata -c \
     "ALTER USER replicator WITH ENCRYPTED PASSWORD '<new-password>'"

8. Change the value corresponding to database.password in the existing Kubernetes Secret.

9. Alter bometa user password:

   Copy

   kubectl exec <release-name>-immuta-database-0 -- \
     psql -d bometadata -c \
     "ALTER USER bometa WITH ENCRYPTED PASSWORD '<new-password>'"

10. Scale the immuta-database StatefulSet up to the previous replica count determined in the previous step:

    Copy

    kubectl scale statefulset <release-name>-immuta-database --replicas $DATABASE_REPLICA_COUNT

11. Restart web pods:

    Copy

    kubectl rollout restart deployment/<release-name>-immuta-web

Helm Upgrade Process

1. Check Environment

2. Determine Upgrade Path:

   • Method A: Basic Helm Upgrade (Recommended)

   • Method B: Complete Backup and Restore

1 - Check Environment

1.1 - Check Helm Version

• Immuta's Helm Chart requires Helm version 3+.

• New installations of Immuta must use the latest version of Helm 3 and Immuta's latest Chart.

Run helm version to verify the version of Helm you are using:

version.BuildInfo{Version:"v3.0.2", GitCommit:"19e47ee3283ae98139d98460de796c1be1e3975f", GitTreeState:"clean", GoVersion:"go1.13.5"}

Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}

1.2 - Configure Immuta's Helm Chart Repo

In order to deploy Immuta to your Kubernetes cluster, you must be able to access the Immuta Helm Chart Repository and the Immuta Docker Registry. You can obtain credentials from your Immuta support professional.

Run helm repo list to ensure Immuta's Helm Chart repository has been successfully added:

NAME            URL
stable          https://kubernetes-charts.storage.googleapis.com
local           http://127.0.0.1:8879/charts
immuta          https://archives.immuta.com/charts

Run kubectl get secrets to confirm your Kubernetes image pull secret is in place:

NAME                  TYPE                                  DATA   AGE
immuta-registry       kubernetes.io/dockerconfigjson        1      5s

1.3 - Check/Update Your Local Immuta Helm Chart Version

Run helm search repo immuta to check the version of your local copy of Immuta's Helm Chart:

NAME          CHART VERSION APP VERSION DESCRIPTION
immuta/immuta 4.4.1         2.7.0       The Immuta

Update your local Chart by running helm repo update.

To perform an upgrade without upgrading to the latest version of the Chart, run helm list to determine the Chart version of the installed release, and then specify that version using the --version argument of helm repo update.

1.4 - Confirm Connectivity with Your Current Immuta Helm Installation

Run helm list to confirm Helm connectivity and verify the current Immuta installation:

NAME  REVISION  UPDATED                   STATUS    CHART         APP VERSION NAMESPACE
test  1         Tue Dec 17 01:04:36 2019  DEPLOYED  immuta-4.2.3  2.6.0       ns

Make note of:

• NAME - This is the '<YOUR RELEASE NAME>' that will be used in the remainder of these instructions.

• CHART - This is the version of Immuta's Helm Chart that your instance was deployed under.

1.5 - Confirm Access to the Helm Values Used in Your Current Immuta Installation

You will need the Helm values associated with your installation, which are typically stored in an immuta-values.yaml file. If you do not possess the original values file, these can be extracted from the existing installation using:

helm get values <YOUR RELEASE NAME> > immuta-values.yaml

2 - Determine Your Upgrade Path

Select your method:

• Method A - Basic Upgrade (Recommended)

• Method B - Backup and Restore: This method is intended primarily for recovery scenarios and is only to be used if you have been advised to by an Immuta representative. Reach out to your Immuta representative for instructions.

Basic Upgrade (Recommended)

After you make any desired changes in your immuta-values.yaml file, you can apply these changes by running helm upgrade:

helm upgrade <YOUR RELEASE NAME> immuta/immuta \
    --values immuta-values.yaml
    --version <YOUR DESIRED CHART VERSION>

Note: Errors in upgrades can result when upgrading Chart versions on the installation. These are typically easily resolved by making slight modifications of your values to accommodate the changes in the Chart. Downloading an updated copy of the immuta-values.yaml and comparing to your existing values is often a great way to debug such occurrences.

If you are on Kubernetes 1.22+, remove nginxIngress.controller.image.tag=v0.49.3 when upgrading; otherwise, your ingress service may not start after the upgrade.

Software Versions

• Helm 3.2 or greater

• Kubernetes: See the Install Immuta page for a list of versions Immuta supports.

Helm and Immuta's Helm Chart

Immuta uses Helm to manage and orchestrate Kubernetes deployments. Check the Helm Release Notes to ensure you are using the correct Helm Chart with your version of Immuta.

Storage

Database Backups

Database backups for the metadata database and Query Engine may be stored in either cloud-based blob storage or a Persistent Volume in Kubernetes.

Cloud Storage (Recommended)

Backups may be stored using one of the following cloud-based blob storage services:

• AWS S3

  • Supports authentication via AWS Access Key ID / Secret Key, IAM Roles via kube2iam or kiam, or IAM Roles in EKS.

• Azure Blob Storage

  • Supports authentication via Azure Storage Key, Azure SAS Token, or Azure Managed Identities.

• Google Cloud Storage

  • Supports authentication via Google Service Account Key

Database Persistence

When database persistence is enabled, Immuta requires access to PersistentVolumes through the use of a persistent volume claim template. These volumes should normally be provided by a block device, such as AWS EBS, Azure Disk, or GCE Persistent Disk.

Additionally, when database persistence is enabled, Immuta requires the ability to run an initContainer as root. When PodSecurityPolicies are in place, service accounts must be granted access to use a PodSecurityPolicy with the ability to RunAsAny user.

RBAC

The Immuta Helm Chart supports RBAC and will try to create all needed RBAC roles by default.

Ingress

Immuta needs Ingress for two services:

1. Immuta Web Service (HTTP)

2. Immuta Query Engine (TCP)

The Immuta Helm Chart creates Ingress resources for HTTP services (the Immuta Web Service), but because of limitations with Kubernetes Ingress resources TCP ingress must be configured separately. The configuration for TCP ingress is dependent on the Ingress Controller that you are using in your cluster. Also, the configuration for TCP ingress is optional if you will only integrations, and it can be disabled.

To simplify the configuration for cluster Ingress, the Immuta Helm Chart contains an optional Nginx Ingress component that may be used to configure a Nginx Ingress Controller to be used specifically for Immuta. Contact your Immuta Support Professional for more information.

Worker Node Size

Immuta’s suggested minimum node size has 4 CPUs and 16GB RAM. The default Immuta Helm deployment requires at least 3 nodes.

TLS Certificates

All Immuta services use TLS certificates to enable communication over HTTPS. In order to support many configurations, the Immuta Helm Chart has the ability to configure internal and external communication independently. If TLS is enabled, by default, a certificate authority will be generated then used to sign a certificate for both internal and external communications. See Enabling TLS for instructions to configuring TLS.

Internal HTTPS communication refers to all communication between Immuta services. External HTTPS communication refers to communication between clients and the Immuta Query Engine and Web Service, which is configured using a Kubernetes Ingress resource.

Audience: System Administrators

Content Summary: The Immuta Helm installation integrates well with Kubernetes on AWS. This guide walks through the various components that can be set up.

Prerequisite: An Amazon EKS cluster with a recommended minimum of 3 m5.xlarge worker nodes.

Deployment

As of Kubernetes 1.23+ on EKS, you have to configure the EBS CSI driver in order for the Immuta Helm deployment to be able to request volumes for storage. Follow these instructions:

1. Upon cluster creation, create an IAM policy and role and associate it with the cluster. See AWS documentation for details: Creating the Amazon EBS CSI driver IAM role for service accounts.

2. Upon cluster creation, add the EBS CSI driver as an add-on to the cluster. See AWS documentation for details: Managing the Amazon EBS CSI driver as an Amazon EKS add-on.

For deploying Immuta on a Kubernetes cluster using the AWS cloud provider, you can mostly follow the Kubernetes Helm installation guide.

The only deviations from that guide are in the custom values file(s) you create. You will want to incorporate any changes referenced throughout this guide, particularly in the Backups and Load Balancing sections below.

Backups

Best Practice: Use S3 for Shared Storage

On AWS Immuta recommends that you use S3 for shared storage.

AWS IAM Best Practices

When using AWS IAMs make sure that you are using the best practices outlined here: AWS IAM Best Practices.

To begin, you will need an IAM role that Immuta can use to access the S3 bucket from your Kubernetes cluster. There are four options for role assumption:

1. IAM Roles for Service Accounts : recommended for EKS.

2. Kube2iam or kiam: recommended if you have other workloads running in the cluster.

3. Instance profile: recommended if only Immuta is running in the cluster.

4. AWS secret access keys: simplest set-up if access keys and secrets are allowed in your environment.

Necessary IAM Permissions

The role you choose above must have at least the following IAM permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ImmutaS3Backups00",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket>",
                "arn:aws:s3:::<bucket>/<prefix>/database/*",
                "arn:aws:s3:::<bucket>/<prefix>/query-engine/*"
            ]
        }
    ]
}

Sample Helm Values

backup:
  enabled: true
  type: s3
  # Uncomment to enable restore
  # restore:
  #   enabled: true
  # Uncomment if using kube2iam or kiam
  # podAnnotations:
  #   iam.amazonaws.com/role: <ROLE NAME>
  # Uncomment if using EKS
  # serviceAccountAnnotations:
  #   eks.amazonaws.com/role-arn: <ROLE ARN>
  s3:
    bucket: <BUCKET NAME>
    # Uncomment if using a bucket prefix. Remember to reflect this prefix in your IAM Role Policy.
    # bucketPrefix: <BUCKET PREFIX>
    awsRegion: <REGION>
    # Uncomment if using AWS Access Keys
    # awsAccessKeyId:
    # awsSecretAccessKey:

Load Balancing

The easiest way to expose your Immuta deployment running on Kubernetes with the AWS cloud provider is to set up nginx ingress as serviceType: LoadBalancer and let the chart handle creation of an ELB.

If you are using the included ingress controller, it will create a Kubernetes LoadBalancer Service to expose Immuta outside of your cluster. The following options are available for configuring the LoadBalancer Service:

• nginxIngress.controller.service.annotations: Useful for setting options such as creating an internal load balancer or configuring TLS termination at the load balancer.

• nginxIngress.controller.service.loadBalancerSourceRanges: Used to limit which client IP addresses can access the load balancer.

• nginxIngress.controller.service.externalTrafficPolicy: Useful when working with Network Load Balancers on AWS. It can be set to “Local” to allow the client IP address to be propagated to the Pods.

Possible values for these various settings can be found in the Kubernetes Documentation.

If you would like to use automatic ELB provisioning, you can use the following values:

nginxIngress:
  enabled: true
  controller:
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"

You can then manually edit the ELB configuration in the AWS console to use ACM TLS certificates to ensure your HTTPS traffic is secured by a trusted certificate. For instructions on doing this, please see Amazon's guide on how to Configure an HTTPS Listener for Your Classic Load Balancer

Another option is to set up nginx ingress with serviceType: NodePort and configure load balancers outside of the cluster.

For example,

nginxIngress:
  enabled: true
  controller:
    service:
      type: NodePort

In order to determine the ports to configure the load balancer for, examine the Service configuration:

kubectl get svc \
  -l app.kubernetes.io/name=immuta,app.kubernetes.io/component=nginx-ingress \
  -o go-template='{{range .items}}{{range .spec.ports}}{{ printf "%s: %v\n" .name .nodePort}}{{end}}{{end}}'

This will print out the port name and port. For example,

https: 32757
postgres: 31517

Maintenance

The Immuta deployment to EKS has a very low maintenance burden.

Failure Recovery Scenarios

Ensure that your Immuta deployment is taking regular backups to AWS S3.

Your Immuta deployment is highly available and resilient to failure. For some catastrophic failures, recovery from backup may be required. Below is a list of failure conditions and the steps necessary to ensure Immuta is operational.

• Internal Immuta Service Failure: Because Immuta is running in a Kubernetes deployment, no action should be necessary. Should a failure occur that is not automatically resolved, follow Immuta backup restoration procedures.

• EKS Cluster Failure: Should your EKS cluster experience a failure, simply create a new cluster and follow Immuta backup restoration procedures.

• Availability Zone Failure: Because EKS and ELB as well as the Immuta installation within EKS are designed to tolerate the failure of an availability zone, there are no steps needed to address the failure of an availability zone.

• Region Failure: To provide recovery capability in the unlikely event of an AWS Region failure, Immuta recommends periodically copying database backups into an S3 bucket in a different AWS region. Should you experience a region failure, simply create a new cluster in a working region and follow Immuta backup restoration procedures.

See the AWS Documentation for more information on managing service limits to allow for proper disaster recovery.

Prerequisites

• Kubernetes 1.16 or greater

• Helm 3.2 or greater

1 - Configure the Environment

1.1 - Check Helm Version

Immuta's Helm Chart requires Helm version 3+.

Run helm version to verify the version of Helm you are using:

helm version

version.BuildInfo{Version:"v3.2.3", GitCommit:"8f832046e258e2cb800894579b1b3b50c2d83492", GitTreeState:"clean", GoVersion:"go1.13.12"}

1.2 - Configure Immuta's Helm Chart Repo

In order to deploy Immuta to your Kubernetes cluster, you must be able to access the Immuta Helm Chart Repository and the Immuta Docker Registry. You can obtain credentials from your Immuta support professional.

Run helm repo list to ensure Immuta's Helm Chart repository has been successfully added:

helm repo list

Example Output

NAME            URL
stable          https://kubernetes-charts.storage.googleapis.com
local           http://127.0.0.1:8879/charts
immuta          https://archives.immuta.com/charts

Run kubectl get secrets to confirm your Kubernetes image pull secret is in place:

kubectl get secrets

NAME                  TYPE                                  DATA   AGE
immuta-registry       kubernetes.io/dockerconfigjson        1      5s

1.3 - Check/Update Your Local Immuta Helm Chart Version

Run helm search repo immuta to check the version of your local copy of Immuta's Helm Chart:

helm search repo immuta

NAME            CHART VERSION   APP VERSION   DESCRIPTION
immuta/immuta   4.5.0           2020.2.4      Immuta: Automated Data Governance

Update your local Chart by running helm repo update.

To perform an upgrade without upgrading to the latest version of the Chart, run helm list to determine the Chart version of the installed release, and then specify that version using the --version argument of helm repo update.

2 - Configure Immuta Helm Values

1. Once you have the Immuta Docker Registry and Helm Chart Repository configured, download the immuta-values.yaml file. This file is a recommended starting point for your installation.

2. Modify immuta-values.yaml based on the determined configuration for your Kubernetes cluster and the desired Immuta installation. You can change a number of settings in this file, such as

   • Using an external load balancer/ ingress controller

   • Using an external database

   • Adding TLS certs

   Guidance for configuring persistence, backups, and resource limits are provided below. See Immuta Helm Chart Options for a comprehensive list of configuration options.

3. Replace the placeholder password value "<SPECIFY_PASSWORD_THAT_MEETS_YOUR_ORGANIZATIONS_POLICIES>" with a secure password that meets your organization's password policies.

Avoid these special characters in generated passwords

whitespace, $, &, : , \, /, '

3 - Configure Persistence

If you would like to disable persistence to disk for the database and query-engine components, you can do so by configuring database.persistence.enabled=false and/or queryEngine.persistence.enabled=false in immuta-values.yaml. Disabling persistence can be done for test environments. However, we strongly recommend against disabling persistence in production environments as this leaves your database in ephemeral storage.

By default, database.persistence.enabled and queryEngine.persistence.enabled are set to true and request 120Gi of storage for each component. Recommendations for the Immuta Metadata Database storage size for POV, Staging, and Production deployments are provided in the immuta-values.yaml as shown below. However, the actual size needed is a function of the number of data sources you intend to create and the amount of logging/auditing (and its retention) that will be used in your system.

database:
  persistence:
    enabled: true
    volumeClaimSpec:
      resources:
        requests:
          # Set the requested storage size for database persistence.
          # Recommended sizing:
          # - 20Gi: POV or other small sized deployments, such as development or test environments.
          # - 60Gi: Staging environments that mimic production, but see less use.
          # - 120Gi: Production deployments.
          storage: 120Gi

While the Immuta query engine persistent storage size is configurable as well, the default size of 20Gi should be sufficient for operations in nearly all environments.

queryEngine:
  persistence:
    enabled: true
    volumeClaimSpec:
      resources:
        requests:
          # Set the requested storage size for database persistence.
          # Recommended sizing:
          # - 20Gi: Usually sufficient for most deployments
          storage: 20Gi

4 - Configure Backup and Restoration Values

At this point this procedure forks depending on whether you are installing with the intent of restoring from a backup or not. Use the bullets below to determine which step to follow.

• If this is a new install with no restoration needed, follow Step 4.1.

• If you are upgrading a previous installation using the full backup and restore (Method B), follow Step 4.2.

4.1 - Initial Immuta Kubernetes Installation -- No Backup Restoration

Immuta's Helm Chart has support for taking backups and storing them in a PersistentVolume or copying them directly to cloud provider blob storage including AWS S3, Azure Blob Storage, and Google Storage.

To configure backups with blob storage, reference the backup section in immuta-values.yaml and consult the subsections of this section of the documentation that are specific to your cloud provider for assistance in configuring a compatible resource. If your Kubernetes environment is not represented there, or a workable solution does not appear available, please contact your Immuta representative to discuss options.

If using volumes, the Kubernetes cluster Immuta is being installed into must support PersistentVolumes with an access mode of ReadWriteMany. If such a resource is available, Immuta's Helm Chart will set everything up for you if you enable backups and comment out the volume and claimName.

backup:
  # set to true to enable backups
  enabled: true
  #volume:
    # if claimName is set to an existing PVC no PV/PVC will be created
    #claimName: <YOUR ReadWriteMany PersistentVolumeClaim NAME>

4.2 - Restore From Backup - Immuta Kubernetes Re-Installation

If you are upgrading a previous installation using the full backup and restore procedure (Method B), a valid backup configuration must be available in the Helm values. Enable the functionality to restore from backups by setting the restore.enabled option to true in immuta-values.yaml.

backup:
  # set to true to enable backups
  enabled: true
  restore:
    # set to true to enable restoring from backups
    enabled: true

If using the volume backup type, an existing PersistentVolumeClaim name needs to be configured in your immuta-values.yaml because the persistentVolumeClaimSpec is only used to create a new, empty volume.

backup:
  # set to true to enable backups. requires RWX persistent volume support
  enabled: true
  restore:
    # set to true to enable restoring from backups
    enabled: true
  volume:
    # if claimName is set to an existing PVC no PV/PVC will be created
    claimName: <YOUR ReadWriteMany PersistentVolumeClaim NAME>

If you are unsure of the value for <YOUR ReadWriteMany PersistentVolumeClaim NAME>, the command kubectl get pvc will list it for you

kubectl get pvc

NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-immuta-backup   Bound    pvc-2ca50bff-8c2c-487c-b7bf-75c3764c470a   5Gi        RWX            standard       39h

5 - Set Replicas and Resource Limits

Adhering to the guidelines and best practices for replicas and resource limits outlined below is essential for optimizing performance, ensuring cluster stability, controlling costs, and maintaining a secure and manageable environment. These settings help strike a balance between providing sufficient resources to function optimally and making efficient use of the underlying infrastructure.

5.1 - Configure Replicas

Set the following replica parameters in your Helm Chart to the values listed below:

database:
  replicas: "1"
queryEngine:
  replicas: "1"
web:
  replicas: "3"
fingerprint:
  replicas: "2"
cache:
  replicas: "2"

5.2 - Set Resource Limits

The Immuta Helm Chart supports Resource Limits for all components. Set resources and limits to the database and query engine in the Helm values. Without those limits, the pods will be the first target for eviction, which can cause issues during backup and restore, since this process consumes a lot of memory.

Add this YAML snippet to your Helm values file:

database:
  resource:
    requests:
      memory: "4Gi"
    limits:
      memory: "8Gi"
      cpu: "1"

Database is the only component that needs a lot of resources, especially if you don’t use the query engine. For a small installation, you can set the database resources to 2Gi, and if you see slower performance over time, you can increase this number to improve performance.

Setting CPU resources and limits is optional. Resource contention over CPU is not a common occurrence for Immuta, so it won’t have a significant effect if the CPU resource and limit is set.

6 - Deploy Immuta

Run the following command to deploy Immuta:

helm install <YOUR RELEASE NAME> immuta/immuta \
  --values immuta-values.yaml

Advanced Installations

Manage TLS

HTTP communication using TLS certificates is enabled by default in Immuta's Helm Chart for both internal (inside the Kubernetes cluster) and external (between the Kubernetes ingress and the outside world) communications. This is accomplished through the generation of a local certificate authority (CA) which signs certificates for each service - all handled automatically by the Immuta installation. While not recommended, if TLS must be disabled for some reason, this can be done by setting tls.enabled to false in the values file.

Use Your Own TLS Certificate(s)

Using your own certificates requires you to create a Kubernetes Secret containing the private key, certificate, and certificate authority certificate. This can be easily done using kubectl:

kubectl create secret generic immuta-external-tls \
    --from-file=tls.crt=<path to your.pem cert> \
    --from-file=tls.key=<path to your-key.pem key> \
    --from-file=ca.crt=<path to your-ca.pem cert>

After creating the Kubernetes Secret, specify its use in the external ingress by setting tls.externalSecretName = immuta-external-tls in your immuta-values.yaml file:

tls:
  externalSecretName: immuta-external-tls

Using Argo CD

The Immuta Helm Chart (version 4.5.0+) can be deployed using Argo CD.

For Argo CD versions older than 1.7.0 you must use the following Helm values in order for the TLS generation hook to run successfully.

hooks:
  tlsGeneration:
    hookAnnotations:
      helm.sh/hook-delete-policy: "before-hook-creation"

Starting with Argo CD version 1.7.0 the default TLS generation hook values can be used.

tls.manageGeneratedSecret must be set to true when using Argo CD to deploy Immuta; otherwise, the generated TLS secret will be shown as OutOfSync (requires pruning) in Argo CD. Pruning the Secret would break TLS for the deployment, so it is important to set this value to prevent that from happening.

tls:
  manageGeneratedSecret: true

Troubleshoot

For detailed assistance in troubleshooting your installation, contact your Immuta representative or see Helm Troubleshooting.

Audience: System Administrators

Content Summary: This page outlines instructions for troubleshooting specific issues with Helm.

Pods Frozen with the Status "Pending" or "Init:0/1"

If you encounter Immuta Pods that have had the status Pending or Init:0/1 for an extended period of time, then there may be an issue mounting volumes to the Pods. You may find error messages by describing one of the pods that had the Pending or Init:0/1 status.

kubectl describe pod/<YOUR POD NAME>

If an event with the message pod has unbound PersistentVolumeClaims is seen on the frozen pod, then there is most likely an issue with the database backup storage Persistent Volume Claims. Typically this is caused by the database backup PVC not binding because there are no Kubernetes Storage Classes configured to provide the correct storage type.

helm delete <YOUR RELEASE NAME>
helm install <YOUR RELEASE NAME> immuta/immuta \
    --values immuta-values.yaml

helm delete --purge <YOUR RELEASE NAME>
helm install immuta/immuta \
    --values immuta-values.yaml \
    --name <YOUR RELEASE NAME>

Helm Did Not Cleanup Immuta Kubernetes Resources

Occasionally Helm has bugs or loses track of Kubernetes resources that it has created. Immuta has created a Bash script that you may download and use to cleanup all resources that are tied to an Immuta deployment. This command should only be run after helm delete <YOUR RELEASE NAME>.

Solution

Download cleanup-immuta-deployment.sh.

Run the script:

bash cleanup-immuta-deployment.sh <YOUR RELEASE NAME> <YOUR NAMESPACE (optional)>

Rolling-Restart for Database Pods

After a configuration change or cluster outage you may need to perform a rolling-restart to refresh the database pods without data loss. Use the command below to update a restart annotation on the database pods to instruct the database StatefulSet to roll the pods.

IMMUTA_STATEFULSET=$(kubectl get statefulset -l app.kubernetes.io/name=immuta,app.kubernetes.io/instance=<YOUR RELEASE NAME>,app.kubernetes.io/component=query-engine -o name | head -n 1)
kubectl patch ${IMMUTA_STATEFULSET} -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"restart\":\"$(date '+%Y%m%d%H%M%S')\"}}}}}"

IMMUTA_STATEFULSET=$(kubectl get statefulset -l app.kubernetes.io/name=immuta,app.kubernetes.io/instance=<YOUR RELEASE NAME>,app.kubernetes.io/component=database -o name | head -n 1)
kubectl patch ${IMMUTA_STATEFULSET} -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"restart\":\"$(date '+%Y%m%d%H%M%S')\"}}}}}"

Rolling-Restart for Web Pods

After a configuration change or cluster outage you may need to perform a rolling-restart to refresh the web service pods. Use the command below to update a restart annotation on the web pods to Deployment to roll the pods.

IMMUTA_DEPLOYMENT=$(kubectl get deployment -l app.kubernetes.io/name=immuta,app.kubernetes.io/instance=<YOUR RELEASE NAME>,app.kubernetes.io/component=service -o name | head -n 1)
kubectl patch ${IMMUTA_DEPLOYMENT} -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"restart\":\"$(date '+%Y%m%d%H%M%S')\"}}}}}"

Regenerating Internal TLS Certificates

Should the need arise that you need to regenerate internal TLS certificates follow the instructions below.

IMMUTA_STATEFULSET=$(kubectl get statefulset -l app.kubernetes.io/name=immuta,app.kubernetes.io/instance=<YOUR RELEASE NAME>,app.kubernetes.io/component=query-engine -o name | head -n 1)
kubectl patch ${IMMUTA_STATEFULSET} -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"restart\":\"$(date '+%Y%m%d%H%M%S')\"}}}}}"

IMMUTA_DEPLOY=$(kubectl get statefulset -l app.kubernetes.io/name=immuta,app.kubernetes.io/instance=<YOUR RELEASE NAME>,app.kubernetes.io/component=service -o name | head -n 1)
kubectl patch ${IMMUTA_DEPLOY} -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"restart\":\"$(date '+%Y%m%d%H%M%S')\"}}}}}"

Rotating External TLS Certificates

Should you need to rotate external TLS certificates, follow the instructions below:

Audience: System Administrators

Content Summary: This page outlines how to configure an external metadata database for Immuta instead of using Immuta's built-in PostgreSQL Metadata Database that runs in Kubernetes.

Overview

The Metadata Database can optionally be configured to run outside of Kubernetes, which eliminates the variability introduced by the Kubernetes scheduler and/or scaler without compromising high-availability. This is the preferred configuration, as it offers infrastructure administrators a greater level of control in the event of disaster recovery.

Helm Configuration

1. Enable an external metadata database by setting database.enabled=false in the immuta-values.yaml file and passing the connection information for the PostgreSQL instance under the key externalDatabase.

2. Set queryEngine.rehydration.enabled=true. If set to false, then externalDatabase.superuser.username and externalDatabase.superuser.password must be provided.

database:
  enabled: false
externalDatabase:
  enabled: true
  host: <fqdn>
  username: <username>
  password: <password>
  dbname: <database name>
queryEngine:
  rehydration:
    enabled: true
backup:
  restore:
    enabled: false

External Database Object

The externalDatabase object is detailed below and in the Immuta Helm Chart Options.

Additionally, it is possible to use existingSecret instead of setting externalDatabase.password in the Helm values. These passwords map to the same keys that are used for the built-in database. For example,

apiVersion: v1
kind: Secret
metadata:
  name: immuta-secret
data:
  databasePassword: <password>

First-Time PostgreSQL Setup

CREATE ROLE postgres WITH LOGIN ENCRYPTED PASSWORD '<password>';

1. Log in to the external metadata database as a user with the superuser role attribute (such as the postgres user) using your preferred tool (e.g., psql, pgAdmin).

2. Connect to database postgres, and execute the following SQL.

DROP DATABASE bometadata;
CREATE ROLE bometa WITH LOGIN CREATEDB ENCRYPTED PASSWORD '<password>';
GRANT bometa TO CURRENT_USER;
CREATE DATABASE bometadata OWNER bometa;
GRANT ALL ON DATABASE bometadata TO bometa;
ALTER ROLE bometa SET search_path TO bometadata,public;

1. Connect to database bometadata that was created in the previous step, and execute the following SQL. Azure Database for PostgreSQL: Extensions must be configured in the web portal.

CREATE EXTENSION pgcrypto;

Migrating from Immuta's Built-In PostgreSQL Database to External Metadata Database

For existing deployments, you can migrate from the built-in database to an external database. To migrate, backups must be configured. Reach out to your Immuta representative for instructions.

1. (Optional) Set default namespace:

   Copy

   kubectl config set-context --current --namespace <namespace>

2. Trigger manual backup:

   Copy

   kubectl create job adhoc-backup --from cronjob/<release>-immuta-backup

3. Validate backup succeeded:

   Copy

   kubectl get --no-headers pods --output name | grep "adhoc-backup" | xargs kubectl logs --container database-backup

4. Follow the steps outlined in section First-Time PostgreSQL Setup.

5. Edit immuta-values.yaml to enable the external metadata database and restore.

   Copy

   database:
     enabled: false
   externalDatabase:
     enabled: true
     host: <fqdn>
     username: <username>
     password: <password>
     dbname: <database-name>
   backup:
     enabled: true
     restore:
       enabled: true

6. Apply the immuta-values.yaml changes made in the previous step:

   Copy

   helm upgrade <release> immuta/immuta --values immuta-values.yaml

7. Wait until the Kubernetes resources become ready.

   Copy

   kubectl get all

8. Edit immuta-values.yaml to enable Query Engine rehydration and disable backup/restore.

   Copy

   backup:
     enabled: false
     restore:
       enabled: false
   queryEngine:
     rehydration:
       enabled: true

9. Rerun the previous helm upgrade command to apply the latest immuta-values.yaml changes.

10. Connect to database postgres, and execute the following SQL. Azure Database for PostgreSQL: Delete the previously created role by running DROP ROLE postgres;.

ALTER ROLE bometa WITH NOCREATEDB;
ALTER DATABASE bometadata OWNER TO CURRENT_USER;

Audience: System Administrators

Content Summary: This page outlines how to deploy Immuta on OpenShift.

1 - Create a Project

Run the following command in your terminal:

oc create-project immuta

2 - Determine Values for runAsUser and fsGroup

The Immuta Helm Chart must be configured to set two values within the approved ranges for the OpenShift project Immuta is being deployed into: runAsUser and fsGroup.

• runAsUser: On a Pod SecurityContext, this field defines the user ID that will run the processes within the pod. In the next step, this can be set to any value within the range defined in sa.scc.uid-range. See details below.

• fsGroup: This field defines a group ID that will be added as a supplemental group to the Pod. Files in PersistentVolumes will be writable by this group ID. In the next step, this must be set to the minimum value in the range defined in sa.scc.supplemental-groups. See details below.

View the approved ranges in OpenShift using one of the two methods below:

$ oc get project immuta --output template='{{index .metadata.annotations "openshift.io/sa.scc.uid-range"}}{{"\n"}}'
1000620000/10000

$ oc get project immuta --output template='{{index .metadata.annotations "openshift.io/sa.scc.supplemental-groups"}}{{"\n"}}'
1000620000/10000

In both illustrations above, the first part of the value (leading up to the /) is the assigned user ID/group ID range, and the second part (trailing the /) is the extent of the range.

For example, the minimum UID for sa.scc.uid-range=1000620000/10000 is 1000620000 and the maximum is 1000629999 (1000620000 + 10000).

For the examples throughout the rest of this tutorial, 1000620000 will be set as the value for both runAsUser and fsGroup.

For more details on security context restraints and how the user and group ID ranges are allocated, see the OpenShift documentation.

3 - Create a Values File for Immuta

Set these OpenShift-specific Helm values in a YAML file that will be passed to helm install in the next step:

• externalHostname: Set to a subdomain of the domain configured for the OpenShift Ingress controller. Contact your OpenShift administrator to get the configured domain if it is unknown.

• securityContext.runAsUser: Set this to a user ID in the range specified by the annotation openshift.io/sa.scc.uid-range in the OpenShift project for the following components:

  • backup.securityContext.runAsUser

  • cache.securityContext.runAsUser

  • database.securityContext.runAsUser

  • fingerprint.securityContext.runAsUser

  • queryEngine.securityContext.runAsUser

  • web.securityContext.runAsUser

• securityContext.fsGroup: Set this to the minimum value in the range defined in sa.scc.supplemental-groups in the OpenShift project for the following components:

  • backup.securityContext.fsGroup

  • database.securityContext.fsGroup

  • queryEngine.securityContext.fsGroup

  • web.securityContext.fsGroup

• patroniKubernetes.use_endpoints: Set to false for the components below. This change is required for Patroni to be able to successfully elect a leader.

  • database.patroniKubernetes.use_endpoints

  • queryEngine.patroniKubernetes.use_endpoints

• queryEngine.clientService.type: Set to LoadBalancer so that a LoadBalancer will be created to handle the TCP traffic for the Query Engine. The LoadBalancer that OpenShift creates will have its own hostname/IP address, and you must update the Public Query Engine Hostname in Application Settings (instructions below). This step can be omitted if the Query Engine is not being used.

• web.ingress.enabled: Set to false to disable creation of Ingress resources for the Immuta Web Service. OpenShift provides its own Ingress controller for handling HTTP ingress, and this is configured by creating Routes.

Example Values File

externalHostname: immuta.apps.openshift-example-cluster.example.com
backup:
  securityContext:
    runAsUser: 1000620000
    fsGroup: 1000620000
cache:
  securityContext:
    runAsUser: 1000620000
database:
  patroniKubernetes:
    use_endpoints: false
  securityContext:
    runAsUser: 1000620000
    fsGroup : 1000620000
fingerprint:
  securityContext:
    runAsUser: 1000620000
queryEngine:
  clientService:
    type: LoadBalancer
  patroniKubernetes:
    use_endpoints: false
  securityContext:
    runAsUser: 1000620000
    fsGroup: 1000620000
web:
  ingress:
    enabled: false
  securityContext:
    runAsUser: 1000620000
    fsGroup: 1000620000
nginxIngress:
  enabled: false

4 - Install Immuta

Follow the standard Immuta deployment with Helm, but supply the additional values file using the --values flag in the helm install step.

helm install --values immuta-values.yaml --values openshift-values.yaml immuta immuta/immuta

5 - Configure Ingress

1. To set up ingress for Immuta using the OpenShift Ingress controller, get the CA certificate used by Immuta for internal TLS. This will be used by the OpenShift Ingress controller to validate the upstream TLS connection to Immuta.

   Copy

   kubectl get secret \
   --selector=app.kubernetes.io/name=immuta,app.kubernetes.io/component=tls \
   --output=go-template='{{index .items 0 "data" "ca.crt"|base64decode}}' > ca.crt

2. Create a Route using the OpenShift CLI. The hostname flag should be set to match the value configured for externalHostname in the Helm values file, and it should be a subdomain of the domain that the OpenShift Ingress controller is configured for.

   Copy

   oc create route reencrypt immuta-web \
   --dest-ca-cert=ca.crt \
   --hostname=immuta.apps.openshift-example-cluster.example.com \
   --service=immuta-web

   This will create a route to be served by the OpenShift Ingress controller. At this point, Immuta is installed and should be accessible at the configured hostname.

3. Run kubectl get svc immuta-query-engine-clients to inspect the Query Engine client's service in Kubernetes to get the assigned External IP address. For example,

   Copy

   $ kubectl get svc immuta-query-engine-clients
   NAME                          TYPE           CLUSTER-IP       EXTERNAL-IP           PORT(S)                 AGE
   immuta-query-engine-clients   LoadBalancer   172.30.161.225   example.com   5432:31737/TCP,8008:31285/TCP   65m

4. Copy the External-IP address. You will paste this value in the Immuta App Settings page to update the Public Query Engine Hostname.

5. In the Immuta UI, click the App Settings icon in the left sidebar and scroll to the Public URLs section.

6. Enter the value you copied from the EXTERNAL-IP column in the Public Query Engine Hostname field.
7. Click Save to update the configuration.

Caveat

Nginx Ingress must be disabled to run with the restricted SCC. Immuta's built-in Nginx Ingress controller will not run with the restricted SCC and must be disabled to run in this configuration. OpenShift has its own Ingress controller that can be used for HTTP traffic for the Immuta Web Service. However, since the OpenShift Ingress controller does not support TCP traffic, a separate LoadBalancer service must be used for the Query Engine, and the Public Query Engine Hostname must be updated accordingly.

The Helm Chart includes components that make up your Immuta infrastructure, and you can change these values to tailor your Immuta infrastructure to suit your needs. The tables below include parameter descriptions and default values for all components in the Helm Chart.

When installing Immuta, download immuta-values.yaml and update the values to your preferred settings.

See the Helm installation page for guidance and best practices.

Global

Backup

AWS S3 Backups

These values are used when backup.type=s3.

Azure Blob Storage

These values are used when backup.type=azblob.

Google Cloud Storage

These values are used when backup.type=gs.

Immuta TLS

hooks:
  tlsGeneration:
    hookAnnotations:
      helm.sh/hook: post-install

Web Service

Fingerprint Service

Metadata Database

The Metadata Database component can be configured to use either the built-in Kubernetes deployment or an external PostgreSQL database.

The following Helm values are shared between both built-in and external databases.

Built-in Database

These values are used when database.enabled=true.

External Database

These values are used when database.enabled=false.

Query Engine

Query Engine Port

If you will only use integrations, port 5432 is optional. Using the built-in Ingress Nginx Controller, you can disable it by setting the value to false.

Chart Hooks

Cleanup

The Cleanup hook is a Helm post-delete hook that is responsible for cleaning up some resources that are not deleted by Helm.

Database Initialize

The database initialize hook is used to initialize the external database when database.enabled=false.

TLS Generation

The TLS generation hook is a Helm pre-install hook that is responsible for generating TLS certificates used for connections between the Immuta pods.

Cache

Deploy Tools

Ingress Controller

Memcached

Audience: System Administrators

Content Summary: Before installing Immuta, you will need to spin up your AKS or ACS cluster. This page outlines how to deploy Immuta cluster infrastructure on AKS and ACS.

If you would like to install Immuta on an existing AKS or ACS cluster, you can skip this section. However, we recommend deploying a dedicated resource group and cluster for Immuta if possible.

Once you have deployed your cluster infrastructure, please visit to finish installing Immuta.

Support for AKS and ACS

Create Resource Group

You will need a resource group to deploy your AKS or ACS cluster in:

Note: There is no naming requirement for the Immuta resource group.

Create VM Cluster

Now it is time to spin up your cluster resources in Azure. This step will be different depending on whether you are deploying an AKS or ACS cluster.

After running the command, you will have to wait a few moments as the cluster resources are starting up.

Create AKS Cluster (Recommended):

Create ACS Cluster (Deprecated):

Configure kubectl

You will need to configure the kubectl command line utility to use the Immuta cluster.

Install kubectl through Azure CLI

If you do not have kubectl installed, you can install it through the Azure CLI.

If you are using AKS, run

For ACS clusters, run

Point kubectl to your Immuta cluster

If you are using AKS, run

For ACS clusters, run

External References

Audience: System Administrators

Content Summary: This guide illustrates the deployment of an Immuta cluster on Microsoft Azure Kubernetes Service. Requirements may vary depending on the Azure Cloud environment and/or region. For comprehensive assistance, please contact an Immuta Support Professional.

This guide is intended to supplement the main , which is referred to often throughout this page.

Prerequisites:

• Software: 2.3.0 or greater and 2.0.30 or greater

• Node Size: Immuta's suggested minimum Azure VM size for Azure Kubernetes Service deployments is Standard_D3_v2 (4 vCPU, 14GB RAM, 200 GB SSD) or equivalent. The Immuta helm installation requires a minimum of 3 nodes. Additional nodes can be added on demand.

• TLS Certificates: See the for TLS certificate requirements.

To install Azure CLI 2.0, please visit and follow the instructions for your chosen platform. You can also use the .

For more information on nodes, see the .

Deploy Cluster Infrastructure

Before installing Immuta, you will need to spin up your AKS cluster. If you would like to install Immuta on an existing AKS cluster, you can skip this step. If wish to deploy a dedicated cluster for Immuta, please visit .

Navigate to the installation method of your choice:

Option 1: Install Immuta Manually via Helm

Use the Nginx Ingress Controller as a Load Balancer in Azure Cloud

Since you are deploying Immuta as an Azure cloud application in AKS, you can easily configure the Nginx Ingress Controller that is bundled with the Immuta Helm deployment as a load balancer using the generated hostname from Azure.

1. Confirm that you have the following configurations in your values.yaml file before deploying:

   If you are using the included ingress controller, it will create a Kubernetes LoadBalancer Service to expose Immuta outside of your cluster. The following options are available for configuring the LoadBalancer Service:

   • nginxIngress.controller.service.annotations: Useful for setting options such as creating an internal load balancer or configuring TLS termination at the load balancer.

   • nginxIngress.controller.service.loadBalancerSourceRanges: Used to limit which client IP addresses can access the load balancer.

   • nginxIngress.controller.service.externalTrafficPolicy: Useful when working with Network Load Balancers on AWS. It can be set to “Local” to allow the client IP address to be propagated to the Pods.
2. After running helm install, you can find the public IP address of the nginx controller by running
3. If the public IP address shows up as <pending>, wait a few moments and check again. Once you have the IP address, run the following commands to configure the Immuta Azure Cloud Application to use your ingress controller:
4. Shortly after running these commands, you should be able to reach the Immuta console in your web browser at the configured externalHostName.

Configure Security Groups

Backup

Option 2: Automated Installation

• Prepare the Helm values file,

• Register the required secrets to pull Immuta's Docker images,

• Run the Helm installation, and

• Create the mapping between the external IP address Ingress Controller (the cluster's load balancer) and the cluster's public DNS name.

Download the Script

Download the script:

Make it executable by running:

Deploy to an AKS Cluster

Parameters

Below is the list of the parameters that the script accepts. These parameters are environment variables that are prepended to the execution command.

Run the Deployment

Destroy a Deployment

You can use the same script to destroy a deployment you had previously run with this script, by running the following command:

The value of CLUSTER_NAME should be identical to the name of the CLUSTER_NAME value you've used to deploy Immuta.

External References