Stuck Prisma Migrations during Upgrade
Occasionally when upgrading using IHC or IEHC the web pods or audit pods will fail to start due to stuck and/or incomplete prisma migrations. This process will resolve them and bring the pods up.
Last updated
Occasionally when upgrading using IHC or IEHC the web pods or audit pods will fail to start due to stuck and/or incomplete prisma migrations. This process will resolve them and bring the pods up.
Last updated
Review the error within the offending pod and identify which migration it is stating is failing to complete. The format of the error will tell you (Refer to example error above)
In the example above the offending migration is "20240906132648_remove_unused_sdd_conditional_tags" yours will likely be different so note this down
Download the script attached to this article and make the edits needed based upon the error message presented
You can run the script with the following command
kubectl -n <NAMESPACE> apply -f <LOCATION OF DOWNLOADED SCRIPT>
This will generate a new pod that will connect to the database and manually force run the migration that is failing to complete on its own.
Once it is complete (should be instant) re-run the helm upgrade command to see if the web pods come back up.
In the event that the web pods fail once again with a migration, it might be that there were multiple migrations that have failed. In this scenario alter the script again with the new migration that was listed as failed and re-run the script (might need to delete the job)