How to fix “Helm Upgrade Error: has no deployed releases” mystery

Photo by Brusk Dede on Unsplash

Helm upgrade --install is an idempotent way of deploying a Helm Chart depends on if the release exists, which is very useful in an automated CI environment.

However, as of Helm 2.7.1, if you run Helm upgrade --install on a previous failed release, the following error will be printed out:

Error: UPGRADE FAILED: "xxx" has no deployed releases

This new behaviour was introduced by this pull request since Helm 2.7.1 and impact Helm 3 as well.

Why this behaviour has been introduced?

In fact, Helm 2’s tiller doesn’t support 3-ways merge of the current manifest, the new manifest, and the actual state of the resources. Instead, it simply compares the previous deployed manifest (by looking up release revision history) and the new manifest (that is about to deploy) to generate the apply patches. It will not verify whether the state of resource matches the previous deployed manifest or not, but always simply assume so. This often will lead to an unexpected/undesired outcome.

For example, when you had your previous deployment failed (e.g. due to timeout), highly likely there are some resources that have been installed in the cluster yet. Prior to Helm 2.7.1, the helm will not stop you from retrying the deployment. However, as there is no difference between the retried manifest and the failed manifest, Helm will think nothing needs to do with the cluster and do nothing instead of installing those missing resources.

The new behaviour has been introduced to prevent helm from using a previously failed deployment data, which might not be consistent with the cluster state. Instead, Helm will use the latest successful deployment as the baseline for upgrading the deployment. If there is no successful deployment can be found from deployment history, you will then see the has no deployed release error being thrown.

How to fix it

Although the purpose of introducing behaviour was to protect the deployment being damaged, the big problem is that once you see the error, you are pretty much left with no too much options to recover from it.

helm upgrade — force might look like an option. However, behinds the scene, it actually does a helm delete — purge (which means you will lose all your release history) and then installs. This makes it a dangerous option and not suitable for production instances as it will unnecessarily take down all your services for some time. Moreover, it will not always let you bypass the error as well.

Solution for Helm 2

The 100% works option to bypass the error is to run the following command¹:

kubectl -n kube-system patch configmap release-name.v123 --type=merge -p '{"metadata":{"labels":{"STATUS":"DEPLOYED"}}}'

In fact, Helm only tries to filter out failed deployments by only looking at the STATUS field of the deployment history and report the error accordingly. The command above will update the STATUS filed of the deployment history data (stored in configMap) and set it to DEPLOYED. Therefore, the helm will not consider the deployment as failed anymore and attempt to deploy against the deployed manifest of a previously failed one. Here, the release-name is the release name that you used to deploy your helm app and v123 is the version of the release revision in the deployment history.

Please keep in mind, the solution here is only let you bypass the error. Due to the lack of 3-ways merge feature of Helm 2, you might still have to do some manual adjustments to your cluster if it’s not on a par with the previous deployment, in some cases, you may still have to run helm delete --purge to start from scratch if too much work involved.

Solution for Helm 3

Since Helm 3, Helm supports three-way merge patch strategy², which makes it safer to bypass the error for Helm 3. Similarly, you can mark a release revision in deployment history asdeployed to bypass the error. As Helm 3 stores deployment history as k8s Secrets, you will need a different command to achieve that:

kubectl -n app-namespace patch secret release-name.v123 --type=merge -p '{"metadata":{"labels":{"status":"deployed"}}}'

Here, the release-name is the release name that you used to deploy your helm app and v123 is the version of the release revision in the deployment history.

[1] https://github.com/helm/helm/issues/5595#issuecomment-580449247

[2] https://github.com/helm/helm/pull/6124