Altering the default audit trail record types and collection duration

Immuta has default settings for each type of audit record it collects and the amount of time these are kept within the audit trail. This article looks at how to alter the default retention duration.

The audit records held in the Immuta metadata database can begin to consume large amounts of space as many audit record types are held by Immuta indefinitely. While this may be OK for production environments it can cause problems in lower environments where keeping audit records is not as important.

A example of audit records accumulation from an Immuta instance where auditing is sitting at its defaults. The table and its associated indexes are consuming over 8GB.

bometadata=# select table_schema, table_name, pg_relation_size('"'||table_schema||'"."'||table_name||'"'),pg_total_relation_size('"'||table_schema||'"."'||table_name||'"')
from information_schema.tables
where table_schema in ('pgboss','bometadata')
order by table_schema,3 desc;
 table_schema |           table_name           | pg_relation_size | pg_total_relation_size
--------------+--------------------------------+------------------+------------------------
 bometadata   | auditrecord                    |       5100445696 |             8789712896

The records consuming the larger proportion of the table from a record count perspective.

bometadata=# select "recordType", min("createdAt"), max("createdAt"), count(*) as "recordCount" from bometadata.auditrecord group by "recordType" order by "recordCount" desc;
          recordType          |            min             |            max             | recordCount
------------------------------+----------------------------+----------------------------+-------------
 spark                        | 2020-04-24 16:36:36.643+00 | 2024-04-30 15:28:36.992+00 |    22655076
 sqlQuery                     | 2020-04-21 20:28:52.739+00 | 2024-04-30 15:29:54.958+00 |    13252993

The default retention of audit records in Immuta is 60 days. Having said this many audit record types are exempted from being cleaned up when they are more than 60 days old.

To alter the default auditing behaviour manipulation of this exclusion list can be performed to better suit the audit requirements of the Immuta instance.

The default set of audit records exempt from cleanup is the following:

"cleanUpAuditRecordTypeExclusions": [
                "dataSourceSubscription",
                "sqlQuery",
                "nativeQuery",
                "blobFetch",
                "spark",
                "policyHandlerCreate",
                "policyHandlerUpdate",
                "policyExemption",
                "globalPolicyCreate",
                "globalPolicyUpdate",
                "globalPolicyDelete",
                "globalPolicyDisabled",
                "globalPolicyConflictResolved",
                "prestoQuery",
                "globalPolicyApproved",
                "globalPolicyChangeRequested",
                "globalPolicyApprovalRescinded"
            ]

The default retention period is 60 days and set through the following Immuta application setting.

"expireAuditRecords": 60

These settings can be found in the application-configuration.json created when a system status bundle is generated via the Immuta UI for the Immuta instance.

To alter the default exemption list set for audit record types adding an advanced set of Immuta instance parameters is performed by adding the required exemption list to the advanced configuration settings in the Immuta UI.

For example the following would be added to the Immuta advanced configuration settings to remove audit record types spark and sqlQuery from the exemption list and return them to having only a 60 days retention period.

schedule:
  cleanUpAuditRecordTypeExclusions:
    - dataSourceSubscription
    - nativeQuery
    - blobFetch
    - policyHandlerCreate
    - policyHandlerUpdate
    - policyExemption
    - globalPolicyCreate
    - globalPolicyUpdate
    - globalPolicyDelete
    - globalPolicyDisabled
    - globalPolicyConflictResolved
    - prestoQuery
    - globalPolicyApproved
    - globalPolicyChangeRequested
    - globalPolicyApprovalRescinded