Troubleshoot Helm

Audience: System Administrators

Content Summary: This page outlines instructions for troubleshooting specific issues with Helm.

Using a Kubernetes namespace

If deploying Immuta into a Kubernetes namespace other than the default, you must include the --namespace option into the helm and kubectl commands provided throughout this section.

Pods Frozen with the Status "Pending" or "Init:0/1"

If you encounter Immuta Pods that have had the status Pending or Init:0/1 for an extended period of time, then there may be an issue mounting volumes to the Pods. You may find error messages by describing one of the pods that had the Pending or Init:0/1 status.

kubectl describe pod/<YOUR POD NAME>

If an event with the message pod has unbound PersistentVolumeClaims is seen on the frozen pod, then there is most likely an issue with the database backup storage Persistent Volume Claims. Typically this is caused by the database backup PVC not binding because there are no Kubernetes Storage Classes configured to provide the correct storage type.

Solution

Review your backup configuration and ensure that you either have the proper storageClassName or claimName set.

Once you have updated the immuta-values.yaml to contain the proper PVC configuration, you will want to first delete the Immuta deployment, then run helm install.

Helm 3

helm delete <YOUR RELEASE NAME>
helm install <YOUR RELEASE NAME> immuta/immuta \
    --values immuta-values.yaml

Helm 2

helm delete --purge <YOUR RELEASE NAME>
helm install immuta/immuta \
    --values immuta-values.yaml \
    --name <YOUR RELEASE NAME>

Helm Did Not Cleanup Immuta Kubernetes Resources

Occasionally Helm has bugs or loses track of Kubernetes resources that it has created. Immuta has created a Bash script that you may download and use to cleanup all resources that are tied to an Immuta deployment. This command should only be run after helm delete <YOUR RELEASE NAME>.

Solution

Download cleanup-immuta-deployment.sh.

601B

cleanup-immuta-deployment.sh

Run the script:

bash cleanup-immuta-deployment.sh <YOUR RELEASE NAME> <YOUR NAMESPACE (optional)>

Rolling-Restart for Database Pods

After a configuration change or cluster outage you may need to perform a rolling-restart to refresh the database pods without data loss. Use the command below to update a restart annotation on the database pods to instruct the database StatefulSet to roll the pods.

Solution

Immuta Query Engine

IMMUTA_STATEFULSET=$(kubectl get statefulset -l app.kubernetes.io/name=immuta,app.kubernetes.io/instance=<YOUR RELEASE NAME>,app.kubernetes.io/component=query-engine -o name | head -n 1)
kubectl patch ${IMMUTA_STATEFULSET} -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"restart\":\"$(date '+%Y%m%d%H%M%S')\"}}}}}"

Metadata Database

IMMUTA_STATEFULSET=$(kubectl get statefulset -l app.kubernetes.io/name=immuta,app.kubernetes.io/instance=<YOUR RELEASE NAME>,app.kubernetes.io/component=database -o name | head -n 1)
kubectl patch ${IMMUTA_STATEFULSET} -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"restart\":\"$(date '+%Y%m%d%H%M%S')\"}}}}}"

Rolling-Restart for Web Pods

After a configuration change or cluster outage you may need to perform a rolling-restart to refresh the web service pods. Use the command below to update a restart annotation on the web pods to Deployment to roll the pods.

Solution

IMMUTA_DEPLOYMENT=$(kubectl get deployment -l app.kubernetes.io/name=immuta,app.kubernetes.io/instance=<YOUR RELEASE NAME>,app.kubernetes.io/component=service -o name | head -n 1)
kubectl patch ${IMMUTA_DEPLOYMENT} -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"restart\":\"$(date '+%Y%m%d%H%M%S')\"}}}}}"

Regenerating Internal TLS Certificates

Should the need arise that you need to regenerate internal TLS certificates follow the instructions below.

Solution

1. Delete the internal TLS secret

   kubectl delete secret -l app=immuta,component=generated-tls,release=<YOUR RELEASE NAME>

2. Recreate the internal TLS secret by running Helm Upgrade.

Note

If you need to modify any postgres settings such as TLS certificate verification for the Query Engine. Be sure to modify values.yaml file before running this command.

• Helm 3: helm upgrade immuta/immuta --values

• Helm 2: helm upgrade immuta/immuta --values --name

WAIT FOR PODS TO RESTART BEFORE CONTINUING

Restart Query Engine:

IMMUTA_STATEFULSET=$(kubectl get statefulset -l app.kubernetes.io/name=immuta,app.kubernetes.io/instance=<YOUR RELEASE NAME>,app.kubernetes.io/component=query-engine -o name | head -n 1)
kubectl patch ${IMMUTA_STATEFULSET} -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"restart\":\"$(date '+%Y%m%d%H%M%S')\"}}}}}"

WAIT FOR PODS TO RESTART BEFORE CONTINUING

Restart Web Service:

IMMUTA_DEPLOY=$(kubectl get statefulset -l app.kubernetes.io/name=immuta,app.kubernetes.io/instance=<YOUR RELEASE NAME>,app.kubernetes.io/component=service -o name | head -n 1)
kubectl patch ${IMMUTA_DEPLOY} -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"restart\":\"$(date '+%Y%m%d%H%M%S')\"}}}}}"

Rotating External TLS Certificates

Should you need to rotate external TLS certificates, follow the instructions below:

Solution

1. Create a new secret with the relevant TLS files.

   kubectl create secret generic <NEW IMMUTA EXTERNAL TLS NAME> -n <IMMUTA NAMESPACE> \
   --from-file=tls.crt=<YOUR .pem CERT PATH> \
   --from-file=tls.key=<YOUR -key.pem KEY PATH> \
   --from-file=ca.crt=<YOUR -ca.pem CERT PATH>

2. Update your tls.externalSecretName in immuta_values.yaml with the new external TLS secret.

   tls:
       enabledExternal: true
       externalSecretName: <NEW IMMUTA EXTERNAL TLS NAME>

3. Run Helm Upgrade to update the certificates for the deployment.

   helm upgrade <YOUR RELEASE NAME> immuta/immuta --values <YOUR VALUES FILE>

4. Delete the old secret.

   kubectl delete secret <OLD IMMUTA EXTERNAL TLS NAME> -n <IMMUTA NAMESPACE>