Skip to content

You are viewing documentation for Immuta version 2021.3.

For the latest version, view our documentation for Immuta SaaS or the latest self-hosted version.

External Metadata Database Configuration

Audience: System Administrators

Content Summary: This page outlines how to configure an external metadata database for Immuta instead of using Immuta's built-in PostgreSQL Metadata Database that runs in Kubernetes.

External Metadata Database Only Supported in Kubernetes Deployments

Configuring an external metadata database is not supported in the Single Node Docker deployment method.

Overview

You can replace Immuta's built-in PostgreSQL Metadata Database with an external PostgreSQL instance when deploying Immuta in Kubernetes. Running the metadata database outside of Kubernetes eliminates the variability introduced by Kubernetes scaling and scheduling events, and infrastructure administrators have greater control over the configuration of the database overall, such as options for high availability and disaster recovery.

Configuration

When configuring your Helm values, enable an external metadata database by setting database.enabled=false in the immuta-values.yaml file and passing the connection information for the PostgreSQL instance under the key externalDatabase, as illustrated in the example below:

database:
  enabled: false
externalDatabase:
  host: replace-with-your-external-postgres.database.hostname
  password: replace-with-your-bometauserpassword
  superuser:
    username: postgres
    password: postgrespassword

External Database Object

The externalDatabase object is detailed below and in the Immuta Helm Chart Options.

Property Description Default Value
host (required) Hostname of the external PostgreSQL database instance. nil
port Port of the external PostgreSQL database instance. 5432
ssl.enabled Boolean value that controls whether SSL is used for external PostgreSQL connection. true
superuser.username (required) Username for the superuser used to initialize the PostgreSQL instance. nil
superuser.password (required) Password for the superuser used to initialize the PostgreSQL instance. nil
username Username that Immuta creates and uses for the application. bometa
password (required) Password associated with username. nil
dbname Database name that Immuta uses. bometadata
backup.enabled Enable flag for external database backups. Only used when backup.enabled=true. true
restore.enabled Enable flag for the external database restore. Only used when backup.restore.enabled=true. true

Additionally, it is possible to use existingSecret instead of setting externalDatabase.password and externalDatabase.superuser.password in the Helm values. These passwords map to the same keys that are used for the built-in database. For example,

apiVersion: v1
kind: Secret
metadata:
  name: immuta-secret
data:
  databasePassword: <password for externalDatabase.password>
  databaseSuperuserPassword: <password for externalDatabase.superuser.password>

Superuser User Requirements

There are two options for the user used for externalDatabase.superuser:

  • a user with superuser role attribute (such as the postgres user), or
  • a user with reduced privileges (this option requires preliminary setup).

A user with superuser permission is required to use the built-in backup functionality. It is possible to disable built-in backups for the external PostgreSQL instance when using a less-privileged user; however, care must be used to ensure that the Query Engine is in-sync with the external database if backups are restored. One situation in which built-in backups might be disabled for the external database is when database provider backups are used. For example, an AWS RDS instance might be configured to take backups, and the user would rely on those backups for the metadata database.

If using a less-privileged user, the bometadata database must exist ahead of time, and the built-in pgcrypto extension must be created in it. The user must have CREATEROLE permissions and have been granted CONNECT, CREATE, and TEMPORARY (ALL) on the bometadata database:

CREATE DATABASE bometadata;
\connect bometadata
CREATE EXTENSION pgcrypto;
CREATE USER example_immuta_admin WITH CREATEROLE ENCRYPTED PASSWORD '';
GRANT ALL ON DATABASE bometadata TO example_immuta_admin;
CREATE ROLE bometa WITH CREATEROlE LOGIN ENCRYPTED PASSWORD '';
GRANT ALL ON DATABASE bometadata TO bometa;
ALTER ROLE bometa SET search_path TO bometadata,public;

Migrating from Immuta's Built-In PostgreSQL Database to External Metadata Database

For existing deployments, you can migrate from the built-in database to an external database. To migrate, backups must be configured.

  1. Follow step 1 (Capture a Complete Current Backup) of this process to back up your instance. Do not complete the other steps in that tutorial.
  2. Set the external database Helm values outlined above and backup.restore.enabled=true in the immuta-values.yaml file.
  3. Run helm upgrade.