What to do if the pg_wal directory is full? (IHC)

Sometimes Patroni can fail to recover and instead of gracefully crashing it will spool to disk in the form of pg_wal files in an effort to save state.

Exec-ing into the database pod (leader pod if there is one) and checking the sizing of the following directory: /var/lib/immuta/postgresql/data/pg_wal. If that directory is inflated then the likely cause is a patroni or audit event failure that caused replication to spool to disk.

Sometimes audit events that are so large (S3 Crawls) they can generate so many records that it causes the table to inflate and crash Patroni leading to the spooling of pg_wal files. Follow the outlines steps below to recover safely and properly without data loss:

1. If a backup is taken and valid prior to this being noticed:

   1. Perform a backup & restore as per our documentation and best practices

2. If there is no backup available:

   1. DO NOT DELETE OR REMOVE ANY WAL FILES FROM THE pg_wal DIRECTORY!

      1. This can lead to replication issues in the future and could corrupt the database

   2. It is recommended to create and engage the Immuta Support team to get the best path forward if a backup & restore is not possible