Immuta Installation on k3s in Air Gapped environment

While the pattern in this article is useful for development and testing, Immuta does not recommend or support using a single server deployment in production.

Gathering Artifacts

Gather k3s, Elasticsearch, PostgreSQL and Immuta artifacts necessary for transfer to the destination network. This list is not intended to be exhaustive but is a decent minimal option. There are alternatives for Kubernetes, Elasticsearch and PostgreSQL that are not covered here that may suit individual use cases better.

Gathering k3s Artifacts

K3S_RELEASE=v1.30.3%2Bk3s1
wget -qO k3s-install.sh https://get.k3s.io
wget -q https://github.com/k3s-io/k3s/releases/download/${K3S_RELEASE}/k3s-airgap-images-amd64.tar.zst
wget -q https://github.com/k3s-io/k3s/releases/download/${K3S_RELEASE}/k3s

Gathering Elasticsearch Artifacts

The docker pull and docker save commands are used here in lieu of other commands like skopeo in order to easily preserve the container name/tag format for import into containerd.

ECK_VERSION=2.14.0
ELASTIC_VERSION=8.15.0

helm repo add elastic https://helm.elastic.co
helm pull elastic/eck-operator
docker pull docker.elastic.co/eck/eck-operator:${ECK_VERSION} 
docker save docker.elastic.co/eck/eck-operator:${ECK_VERSION} > eck-operator-image-${ECK_VERSION}.tar
docker pull docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION} 
docker save docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION} > elasticsearch-${ELASTIC_VERSION}.tar

Gathering PostgreSQL Artifacts

CRUNCHY_VERSION=5.6.0
CRUNCHY_RELEASE=0
POSTGRES_VERSION=16.3-1
PGBACKREST_VERSION=2.51-1
helm pull oci://registry.developers.crunchydata.com/crunchydata/pgo --version ${CRUNCHY_VERSION}
wget -qO postgres-operator-values.yaml https://raw.githubusercontent.com/CrunchyData/postgres-operator-examples/main/helm/install/values.yaml
docker pull registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-${POSTGRES_VERSION}
docker save registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-${POSTGRES_VERSION} > crunchy-postgres-${POSTGRES_VERSION}.tar
docker pull registry.developers.crunchydata.com/crunchydata/postgres-operator:ubi8-${CRUNCHY_VERSION}-${CRUNCHY_RELEASE} 
docker save registry.developers.crunchydata.com/crunchydata/postgres-operator:ubi8-${CRUNCHY_VERSION}-${CRUNCHY_RELEASE} > postgres-operator-${CRUNCHY_VERSION}.tar
docker pull registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi8-${PGBACKREST_VERSION}
docker save registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi8-${PGBACKREST_VERSION} > pgbackrest-${PGBACKREST_VERSION}.tar

Gathering Immuta Artifacts

IMMUTA_VERSION=2024.2.4
IMMUTA_USER=<username>
IMMUTA_PASSWORD=<password>
skopeo login https://ocir.immuta.com -u ${IMMUTA_USER} -p ${IMMUTA_PASSWORD}
export IMMUTA_IMAGES="audit-service audit-export-cronjob cache classify-service immuta-service"
for image in ${IMMUTA_IMAGES}; do
  docker pull ocir.immuta.com/stable/${image}:${IMMUTA_VERSION} 
  docker save ocir.immuta.com/stable/${image}:${IMMUTA_VERSION} > ${image}-${IMMUTA_VERSION}.tar;
done
helm pull oci://ocir.immuta.com/stable/immuta-enterprise --version ${IMMUTA_VERSION} --username ${IMMUTA_USER} --password ${IMMUTA_PASSWORD}

Artifacts to be transferred to air gapped network

Transfer the files gathered in previous steps along with the additional files included at the bottom of this section to the air gapped deployment network. An example of this list is included for reference:

-rwxr-xr-x 1 ubuntu ubuntu   66359448 Jul 31 16:32 k3s
-rw-r--r-- 1 ubuntu ubuntu  154648289 Jul 31 16:33 k3s-airgap-images-amd64.tar.zst
-rwxr-xr-x 1 ubuntu ubuntu      36426 Aug 19 15:05 k3s-install.sh
-rw-r--r-- 1 ubuntu ubuntu      35150 Aug 19 15:18 immuta-enterprise-2024.2.4.tgz
-rw-r--r-- 1 ubuntu ubuntu     118565 Aug 19 17:26 pgo-5.6.0.tgz
-rw-r--r-- 1 ubuntu ubuntu 1469282304 Aug 20 11:35 audit-service-2024.2.4.tar
-rw-r--r-- 1 ubuntu ubuntu  546697216 Aug 20 11:35 audit-export-cronjob-2024.2.4.tar
-rw-r--r-- 1 ubuntu ubuntu   10576896 Aug 20 11:35 cache-2024.2.4.tar
-rw-r--r-- 1 ubuntu ubuntu  895760896 Aug 20 11:36 classify-service-2024.2.4.tar
-rw-r--r-- 1 ubuntu ubuntu 1136108032 Aug 20 11:36 immuta-service-2024.2.4.tar
-rw-r--r-- 1 ubuntu ubuntu  637336064 Aug 20 11:41 crunchy-postgres-16.3-1.tar
-rw-r--r-- 1 ubuntu ubuntu  156720128 Aug 20 11:42 postgres-operator-5.6.0.tar
-rw-r--r-- 1 ubuntu ubuntu     135928 Aug 20 11:48 eck-operator-2.14.0.tgz
-rw-r--r-- 1 ubuntu ubuntu   77628928 Aug 20 11:48 eck-operator-image-2.14.0.tar
-rw-r--r-- 1 ubuntu ubuntu 1266177024 Aug 20 11:49 elasticsearch-8.15.0.tar
-rw-r--r-- 1 ubuntu ubuntu       2438 Aug 20 13:48 postgres-operator-values.yaml
-rw-r--r-- 1 ubuntu ubuntu  194620928 Aug 20 14:26 pgbackrest-2.51-1.tar
-rw-r--r-- 1 ubuntu ubuntu        734 Aug 20 14:30 postgres-cluster.yaml
-rw-r--r-- 1 ubuntu ubuntu        527 Aug 20 14:59 elasticsearch.yaml
-rw-r--r-- 1 ubuntu ubuntu       1043 Aug 20 15:29 immuta-values.yaml

2KB

postgres-operator-values.yaml

734B

postgres-cluster.yaml

183B

elasticsearch.yaml

1KB

immuta-values.yaml

Deploying Kubernetes

Copy the k3s airgap images to the correct destination and run the install setting the option to bypass downloads:

chmod a+x k3s k3s-install.sh
sudo cp k3s /usr/local/bin
sudo mkdir -p /var/lib/rancher/k3s/agent/images
sudo cp k3s-airgap-images-amd64.tar.zst /var/lib/rancher/k3s/agent/images
INSTALL_K3S_SKIP_DOWNLOAD=true ./k3s-install.sh --write-kubeconfig-mode 644 --embedded-registry

Validating the installation

Example of a sucessful deployment pod list:

kubectl get po -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-576bfc4dc7-bn5fk                  1/1     Running     0          31s
kube-system   helm-install-traefik-78wr4                0/1     Completed   1          31s
kube-system   helm-install-traefik-crd-nsl8c            0/1     Completed   0          31s
kube-system   local-path-provisioner-6795b5f9d8-4cmg8   1/1     Running     0          31s
kube-system   metrics-server-557ff575fb-kfn7h           0/1     Running     0          31s
kube-system   svclb-traefik-72f7e9b4-tx568              2/2     Running     0          7s
kube-system   traefik-5fb479b77-qvh5h                   1/1     Running     0          7s

Importing images to embedded k3s registry

Load the images into the embeded registry using the containerd cli:

for image in *.tar; do 
  sudo ctr -n k8s.io image import ${image}; 
done

Example output

This process can take some time depending on environment conditions

unpacking ocir.immuta.com/stable/audit-export-cronjob:2024.2.4 (sha256:4063c4300ee0ebb045385554685b473cd483910175f17a2ab6b6c0187cd9803c)...done
unpacking ocir.immuta.com/stable/audit-service:2024.2.4 (sha256:25bf5596d9a735c0c6101bd5ea5ced84df8323debcc2f47a7338d318388eb9c7)...done
unpacking ocir.immuta.com/stable/cache:2024.2.4 (sha256:f11ca88d5fb619f8756bf9e2685418760835a9f642110d25092afc9bc1ad633f)...done
unpacking ocir.immuta.com/stable/classify-service:2024.2.4 (sha256:347d35579ac433db68f3c89201b4ac18fce14d6bd6b68af73b94b4ee487b8d22)...done
unpacking registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-16.3-1 (sha256:43e002902abfab51a5f269ef72afb196967427e6eae72c95f7bd8d3775519845)...done
unpacking docker.elastic.co/eck/eck-operator:2.14.0 (sha256:f3f22f4e04491c073b75fbfcf865ef6c2c34173045cdef6c69f2c22770eed06c)...done
unpacking docker.elastic.co/elasticsearch/elasticsearch:8.15.0 (sha256:34ab35b93493d86846f66e92823419f94b9cc16b82b63a55e94fe7429dea3404)...done
unpacking ocir.immuta.com/stable/immuta-service:2024.2.4 (sha256:072f3f9b0f170fd96a15698672b49722a0645328bc4218d9aeb0b632c2f4a7b9)...done
unpacking registry.developers.crunchydata.com/crunchydata/postgres-operator:ubi8-5.6.0-0 (sha256:632ba77c2631c6ffa8cf408b42775ae551d21efa48065eeb570b913403b636af)...done

Installing Elasticsearch

Install the ECK operator into its own namespace and then create a single pod Elasticsearch cluster in the Immuta namespace:

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
helm install elastic-operator ./eck-operator-2.14.0.tgz -n elastic-system --create-namespace
kubectl create namespace immuta
kubectl apply -f elasticsearch.yaml

Validating Elasticsearch

Get the password that was generated for the elastic user and validate connectivity to Elasticsearch with curl:

ELASTIC_PASSWORD=$(kubectl get -n immuta secret elasticsearch-es-elastic-user -o go-template='{{.data.elastic | base64decode}}')
kubectl port-forward -n immuta service/elasticsearch-es-http 9200
curl -u "elastic:$ELASTIC_PASSWORD" -k "https://localhost:9200"

Installing PostgreSQL

Install the Postgres Operator in its own namespace with helm:

helm install -n postgres-operator pgo pgo-5.6.0.tgz -f ./postgres-operator-values.yaml  --create-namespace

Install a postgres cluster in the Immuta namespace:

kubectl apply -f postgres-cluster.yaml

Validating PostgreSQL

Forward the Postgres port back to the localhost:

PG_RELEASE_NAME=immuta
PG_CLUSTER_PRIMARY_POD=$(kubectl get pod -n immuta -o name -l postgres-operator.crunchydata.com/cluster=${PG_RELEASE_NAME},postgres-operator.crunchydata.com/role=master)
kubectl -n immuta port-forward "${PG_CLUSTER_PRIMARY_POD}" 5432:5432

Connect to the Postgres cluster by getting relevant connection string information from kubernetes secrets:

PG_CLUSTER_USER_SECRET_NAME=${PG_RELEASE_NAME}-pguser-${PG_RELEASE_NAME}

export PGHOSTNAME=$(kubectl get secrets -n immuta "${PG_CLUSTER_USER_SECRET_NAME}" -o go-template='{{.data.host | base64decode}}')
export PGPASSWORD=$(kubectl get secrets -n immuta "${PG_CLUSTER_USER_SECRET_NAME}" -o go-template='{{.data.password | base64decode}}')
export PGUSER=$(kubectl get secrets -n immuta "${PG_CLUSTER_USER_SECRET_NAME}" -o go-template='{{.data.user | base64decode}}')
export PGDATABASE=$(kubectl get secrets -n immuta "${PG_CLUSTER_USER_SECRET_NAME}" -o go-template='{{.data.dbname | base64decode}}')
psql -h localhost

Configure the database for use with Immuta:

CREATE EXTENSION pgcrypto;
alter role immuta set search_path to bometadata,public;

Installing Immuta

Note that special characters need to be percent encoded in the postgresql connection string as described here: https://www.prisma.io/dataguide/postgresql/short-guides/connection-uris#percent-encoding-values

Edit the immuta-values.yaml file template provided in the artifact gathering and transfer sections with the correct PostgreSQL and Elasticsearch credentials and then install with:

helm upgrade --install -n immuta immuta ./immuta-enterprise-2024.2.4.tgz -f immuta-values.yaml

Validating Immuta

Forward the immuta-secure service port back to the localhost and validate connectivity. This can also be accessed via a web browser if the destination machine is not headless. Firewall ports may need to be opened otherwise:

kubectl port-forward -n immuta service/immuta-secure 8823
curl http://localhost:8823

Complete pod list post installation

This is an example of the running pods list after Immuta and dependent services are deployed:

NAMESPACE           NAME                                              READY   STATUS      RESTARTS       AGE
elastic-system      elastic-operator-0                                1/1     Running     1 (158m ago)   159m
immuta              elasticsearch-es-default-0                        1/1     Running     0              25m
immuta              immuta-audit-558c8fc45d-vm95f                     1/1     Running     0              33s
immuta              immuta-backup-p8gs-gn722                          0/1     Completed   0              53m
immuta              immuta-cache-69b65f8864-m9bt5                     1/1     Running     0              14m
immuta              immuta-discover-58bc75785f-zrg6w                  1/1     Running     0              14m
immuta              immuta-immuta-q2h5-0                              4/4     Running     0              54m
immuta              immuta-repo-host-0                                2/2     Running     0              54m
immuta              immuta-secure-background-worker-9664dfd74-86dfv   1/1     Running     0              14m
immuta              immuta-secure-database-migrate-g5rln              0/1     Completed   0              62s
immuta              immuta-secure-web-74f45d8767-g5qns                1/1     Running     0              14m
kube-system         coredns-576bfc4dc7-xc4v4                          1/1     Running     0              4h10m
kube-system         helm-install-traefik-crd-djfg2                    0/1     Completed   0              4h10m
kube-system         helm-install-traefik-kcqg8                        0/1     Completed   1              4h10m
kube-system         local-path-provisioner-6795b5f9d8-hwfp6           1/1     Running     0              4h10m
kube-system         metrics-server-557ff575fb-4xw6t                   1/1     Running     0              4h10m
kube-system         svclb-traefik-cd7613a4-ws85j                      2/2     Running     0              4h9m
kube-system         traefik-5fb479b77-sgn9c                           1/1     Running     0              4h9m
postgres-operator   pgo-7578f849bb-skmx5                              1/1     Running     0              92m