Azure Gov Backups fail due to incorrect Azure API URL

For Azure government customers that wish to migrate/ utilize Azure blob storage for backups, by default will fail to use the blob.core.windows.govapi endpoint thus causing the backups job to fail

Issue

When attempting to setup Azure Blob Storage as the method for remote backup storage for Immuta on Azure Gov Cloud. You might run into an error like the following:

Defaulted container "database-backup" out of: database-backup, query-engine-backup, immuta-tools (init)

INFO : 2023/07/26 16:29:29 db.go:70: Connecting to database //postgres:xxxxx@ immuta-database.immuta:5432/postgres.
INFO : 2023/07/26 16:29:29 backup.go:50: Backing up database: bometadata
INFO : 2023/07/26 16:29:29 backup.go:350: Executing command: pg_dump --username postgres --host immuta-database.immuta --port 5432 --dbname bometadata --format d --compress 0 --file /tmp/idt-backup876353966/bometadata
INFO : 2023/07/26 16:29:30 backup.go:350: Executing command: pg_dumpall --username postgres --host immuta-database.immuta --port 5432 --database=postgres --roles-only --clean --if-exists
INFO : 2023/07/26 16:29:30 blob.go:84: Copying "file:///tmp/idt-backup876353966/immuta-20230726162929z.tar.gz" to "azblob://immuta-backups/database/immuta-20230726162929z.tar.gz"

Error: failed to close destination bucket after copy: blob (key "immuta-20230726162929z.tar.gz") (code=Unknown): write error: -> github. com/Azure/azure-storage-blob-go/azblob.newStorageError, /go/pkg/mod/github. com/!azure//azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=NoAuthenticationInformation) =====
Description=Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:#################################
Time:2023-07-26T16:29:30.7695524Z, Details:
Code: NoAuthenticationInformation
PUT https:// StorageAccountName. blob. core. windows. net/immuta-backups/database/immuta-20230726162929z.tar.gz?KJBKAJbwkajdnakjwbdwJBDLKAJbkJABwkdhb%3D=&blockid=AWJBDKjbdkajwbdkahwbdhabwihbdawbdjjjjjjjJJJJJJJJJ%3D%3D&comp=block&timeout=61
Content-Length: [12345]
User-Agent: [go-cloud/blob/0.1.0 Azure-Storage/0.14 (go1.20.6; linux)]
X-Ms-Client-Request-Id: [#################################]
X-Ms-Version: [2020-04-08]
--------------------------------------------------------------------------------
RESPONSE Status: 401 Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
Content-Length: [123]
Content-Type: [application/xml]
Date: [Wed, 26 Jul 2023 16:29:30 GMT]
Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
Www-Authenticate: [Bearer authorization_uri=https:// login. microsoftonline. com/123456789123456789123456789/oauth2/authorize resource_id=https:// storage. azure. com]
X-Ms-Client-Request-Id: [#################################]
X-Ms-Error-Code: [NoAuthenticationInformation]
X-Ms-Request-Id: [#################################]
X-Ms-Version: [2020-04-08]

Cause

The real cause is due to Azure Gov Cloud using a different government API endpoint for such operations, you can see the commercial one is used in the above error:

Commercial URL Endpoint: https:// <storage_account>. blob. core. windows. net
Government URL Endpoint: https:// <storage_account>. blob.core.windows.govapi

This causes the operation to immediately fail as it will never reach the blob storage account.

Resolution

Adding the domain: parameter to the azblob backup section to override this will fix the issue. The backup section will look something like the following:

backup:
  # set to true to enable backups
  enabled: true
  type: azblob
  restore:
    # set to true to enable restoring from backups on initial install
    enabled: true
  # Cron schedule to run backups on
  schedule: "0 0 * * *"
  azblob:
    # We use the azblob go package, which creates a url like this
    # azblob://container_name/container_prefix/database/ 
    # By using the domain option we can pass the --url-var=domain=foo option to the azblob, changing the base url.
    domain: "blob.core.windows.govapi"
    container: "container_name"
    #containerPrefix: "container_prefix"
    #azStorageAccount: "storage_account"
    #azStorageKey: "storage_key"
    azStorageSASToken: "sas_token"