Kubernetes Helm Upgrade
Audience: System Administrators
Content Summary: This guide provides two methods for upgrading Helm installs of Immuta in Kubernetes.
- Procedure A is for basic upgrades and is typically acceptable when moving between minor point releases of Immuta (such as
2.7.0
to2.7.1
).- Procedure B is a more involved method that leverages a full backup and restore procedure. This approach is needed when conducting more significant upgrades such as major releases (such as
2.6.x
to2.7.x
), when making large jumps in versions (for example going from2.5.x
to2.7.x
), or simply whenever advised by your Immuta professional.Please note that the backup and restore procedure can be used in all cases, so if in doubt elect that procedure.
If using a Kubernetes namespace...
If deploying Immuta into a Kubernetes namespace other than the default, you must include the
--namespace
option into all helm
and kubectl
commands provided throughout this section.
Prerequisites
In order to conduct any upgrades of Immuta in a Kubernetes environment, you will need Helm access to the Kubernetes cluster running Immuta.
Step 1.) Environment Check
a.) Check Helm version
Immuta's Helm Chart requires Helm version 2.16+ or 3+.
- Helm 2.16+ is only supported for existing Immuta installations.
- New installations of Immuta must use the latest version of Helm 3 and Immuta's latest Chart.
Run helm version
to verify the version of Helm you are using:
helm version
Example Output
version.BuildInfo{Version:"v3.0.2", GitCommit:"19e47ee3283ae98139d98460de796c1be1e3975f", GitTreeState:"clean", GoVersion:"go1.13.5"}
Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
If using Helm 2, ensure your Helm version matches Tiller in the cluster
Keeping your Helm version matched to your Tiller version is best practice and can avoid incompatibility issues.
Run helm version
to compare versions and ensure sync.
$ helm version
Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
b.) Configure Immuta's Helm Chart repo
In order to deploy Immuta to your Kubernetes cluster, you must be able to access the Immuta Helm Chart Repository and the Immuta Docker Registry. You can obtain credentials and instructions to set up both of these by accessing the Immuta Release Portal.
Run helm repo list
to ensure Immuta's Helm Chart repository has been successfully added:
helm repo list
Example Output
NAME URL
stable https://kubernetes-charts.storage.googleapis.com
local http://127.0.0.1:8879/charts
immuta https://archives.immuta.com/charts
Don't forget the image pull secret!
As detailed in Immuta Release Portal, you must create a Kubernetes Image Pull Secret in the namespace that you are deploying Immuta in, or the installation will fail.
Run kubectl get secrets
to confirm your Kubernetes image pull secret is in place:
kubectl get secrets
Example Output
NAME TYPE DATA AGE
immuta-registry kubernetes.io/dockerconfigjson 1 5s
c.) Check/Update your local Immuta Helm Chart version
Run helm search repo immuta
to check the version of your local copy of Immuta's Helm Chart:
helm search repo immuta
Example Output
NAME CHART VERSION APP VERSION DESCRIPTION
immuta/immuta 4.4.1 2.7.0 The Immuta
Update your local Chart by running helm repo update
.
To perform an upgrade without upgrading to the latest version of the Chart, run helm list
to determine the Chart
version of the installed release, and then specify that version using the --version
argument of helm repo update
.
d.) Confirm connectivity with your current Immuta Helm installation
Run helm list
to confirm Helm connectivity and verify the current Immuta installation:
helm list
Example Output
NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
test 1 Tue Dec 17 01:04:36 2019 DEPLOYED immuta-4.2.3 2.6.0 ns
Make note of:
- NAME - This is the '
<YOUR RELEASE NAME>
' that will be used in the remainder of these instructions. - CHART - This is the version of Immuta's Helm Chart that your instance was deployed under.
e.) Confirm access to the Helm values used in your current Immuta installation
You will need the Helm values associated with your installation, which are typically stored in an
immuta-values.yaml
file. If you do not possess the original values file, these can be extracted from the existing
installation using:
helm get values <YOUR RELEASE NAME> > immuta-values.yaml
Determine Your Upgrade Path
Use the following bullets to determine whether Procedure A can be used for your upgrade, or whether you need to elect to use the more thorough Procedure B:
- If your current Immuta version is < 2.6.0, follow Procedure B.
- If your Immuta Helm Chart version is < 4.3, you can upgrade using Procedure A so long as your pin your Chart version to the version of the Chart retrieved in Step 4 above. However, if time is available to do the more involved Procedure B to upgrade the Chart + Immuta, that is the recommended approach.
- For major revisions of Immuta (i.e. 2.7.0 -> 2.8.0), Procedure B is typically recommended, if not required. Your Immuta representative can help guide you if in doubt.
- For minor revisions of Immuta (i.e. 2.8.0 -> 2.8.2), use Procedure A.
PROCEDURE A: Basic Helm Upgrade
Step 1.) Run the Upgrade
After you make any desired changes in your immuta-values.yaml
file, you can apply these changes
by running helm upgrade
:
helm upgrade <YOUR RELEASE NAME> immuta/immuta \
--values immuta-values.yaml
--version <YOUR DESIRED CHART VERSION>
helm upgrade immuta/immuta \
--values immuta-values.yaml \
--name <YOUR RELEASE NAME> \
--version <YOUR DESIRED CHART VERSION>
Note: Errors in upgrades can result when upgrading Chart versions on the installation. These are typically easily
resolved by making slight modifications of your values to accommodate the changes in the Chart. Downloading an
updated copy of the immuta-values.yaml
and comparing to your existing values is often a great way to debug
such occurrences.
PROCEDURE B: Complete Backup and Restore Upgrade
This procedure executes a complete backup and restore of Immuta.
The main steps involved are
- Capture a complete, current backup of your existing Immuta installation.
- Delete your existing Immuta install.
- Update the
immuta-values.yaml
to include all desired changes and reinstall Immuta with the "restore" option set.
Step 1.) Capture a Complete Current Backup
To leverage this procedure, you need Immuta's built-in backup/restore mechanism enabled in your Immuta install if it's not already in place.
a.) Enable Immuta's built-in backup capabilities
Immuta's current Helm Chart provides a built-in backup/restore mechanism based on a PersistentVolumeClaim
with
an access mode of ReadWriteMany
. If such a resource is available, Immuta's Helm Chart will set everything up for you
if you enable backups and comment out the claimName
.
backup:
# set to true to enable backups. requires RWX persistent volume support
enabled: true
# if claimName is set to an existing PVC no PV/PVC will be created
# claimName: <YOUR ReadWriteMany PersistentVolumeClaim NAME>
restore:
# set to true to enable restoring from backups. This should be enabled whenever backups are enabled
enabled: false
If you need to make modifications to these values to enable backups, please do so and then follow Procedure A to commit them into your cluster before proceeding here.
Caution
If your Kubernetes cluster doesn't support PersistentVolumes
with an access mode of ReadWriteMany
,
consult the subsections in this section of the documentation that are specific to your cloud provider for assistance
in configuring a compatible resource. If your Kubernetes environment is not represented there, or a workable solution
does not appear available, please contact your Immuta representative to discuss options.
b.) Check existing backups
After checking/enabling the backup/restore features, you can check to see if any previous backups have run, and if so when, by running
kubectl get pods | grep backup
Example Output
test-immuta-backup-1576368000-bzmcd 0/2 Completed 0 2d
test-immuta-backup-1576454400-gtlrv 0/2 Completed 0 1d
test-immuta-backup-1576540800-5tcg2 0/2 Completed 0 6h
c.) Manually trigger a backup to ensure the latest from your current Immuta installation is captured
Even if you have a fairly recent backup, it's advised to take a current, manual one to ensure nothing is lost between that backup and the present time. To do so, execute an ad-hoc backup with the following commands.
Begin by searching for the default backup Cron job:
kubectl get cronjobs
Example Output
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
test-immuta-backup 0 0 * * * False 0 <none> 14m
Now use that Cron job name to create your own ad-hoc job:
kubectl create job adhoc-backup --from cronjob/test-immuta-backup
Once that job is created, you should see it running in your cluster when you get pods:
kubectl get pods
Example Output
NAME READY STATUS RESTARTS AGE
adhoc-backup-tj8k7 0/2 Completed 0 8h
test-immuta-database-0 1/1 Running 0 9h
test-immuta-database-1 1/1 Running 0 9h
test-immuta-fingerprint-57b5c99f4c-bt5vk 1/1 Running 0 9h
test-immuta-fingerprint-57b5c99f4c-lr969 1/1 Running 0 9h
test-immuta-memcached-0 1/1 Running 0 9h
test-immuta-memcached-1 1/1 Running 0 9h
test-immuta-query-engine-0 1/1 Running 0 9h
test-immuta-query-engine-1 1/1 Running 0 9h
test-immuta-web-b5c7654fb-2brwb 1/1 Running 0 9h
test-immuta-web-b5c7654fb-csm7n 1/1 Running 0 9h
d.) Confirm backup completed successfully
Once that job has completed, you can confirm that everything ran successfully by checking the logs of that pod. There are actually 2 containers in that pod, one that backs up Immuta's metadata database and one that backs up the query engine. If all went as expected your outputs to the following commands should be similar to the examples below.
For the metadata database:
kubectl logs adhoc-backup-<RANDOM> -c database-backup
Example Output
2019-12-17 07:29:55 UTC LOG: Driver staging directory: /var/lib/immuta/odbc/install/
2019-12-17 07:29:55 UTC LOG: Driver install directory: /var/lib/immuta/odbc/drivers/
==> Backing up database roles
==> Backing up role bometadata
==> Creating backup archive
==> Finished creating backup. Backup can be found at /var/lib/immuta/postgresql/backups/immuta-20191217072955.tar.gz
For the query engine database:
kubectl logs adhoc-backup-<RANDOM> -c query-engine-backup
Example Output
2019-12-17 07:29:55 UTC LOG: Driver staging directory: /var/lib/immuta/odbc/install/
2019-12-17 07:29:55 UTC LOG: Driver install directory: /var/lib/immuta/odbc/drivers/
==> Backing up database roles
==> Backing up role immuta
==> Creating backup archive
==> Finished creating backup. Backup can be found at /var/lib/immuta/postgresql/backups/immuta-20191217072955.tar.gz
By default Immuta's backups will be stored in a persistent volume that's typically mounted at
var/lib/immuta/postgresql/backups/
in the Cron job and var/lib/immuta/postgresql/restore/
in the database pods.
In the normal case, this volume will persist across even the deletion of the Immuta installation and be picked back
up during re-installation.
e.) (Optional) Create a local backup of the backup
While optional, out of an abundance of caution, Immuta recommends a manual copy of this backup be made and held for safe-keeping until the upgrade has successfully completed:
kubectl cp test-immuta-database-0:/var/lib/immuta/postgresql/restore/immuta-<LATEST DATE/TIME STAMP>.tar.gz db.tar.gz
kubectl cp test-immuta-query-engine-0:/var/lib/immuta/postgresql/restore/immuta-<LATEST DATE/TIME STAMP>.tar.gz qe.tar.gz
Step 2.) Delete the Current Immuta Installation
a.) Helm delete your existing Immuta installation
Once you have the backups secured, you are ready to delete the current Immuta installation and prepare to reinstall the upgraded version.
Caution!
This command is destructive! Please ensure you have a backup (Step 1) prior to running this.
Deleting the Immuta installation can be done by executing:
helm delete <YOUR RELEASE NAME>
helm delete <YOUR RELEASE NAME> --purge
Example Output
These resources were kept due to the resource policy:
[PersistentVolumeClaim] test-immuta-backup
release "test" deleted
b.) Confirm deletion and resource clean-up
When the helm delete
command finishes, confirm that all associated resources have been torn down by issuing a
kubectl get pods
and that all pods associated with the Immuta installation have been removed.
Step 3.) Update Your Values & Re-Install Immuta
Please see the Kubernetes Helm Installation Instructions for detailed procedures to complete the re-install.
Once in a while Kubernetes resources are not completely freed by the previous helm delete
. If upon reinstall you
have errors relating to conflicting resources, please see Troubleshooting.