Using Helm’s Post-Renderer To Inject Magic

The popular Kubernetes deployment tool Helm has a post-renderer flag that lets you make changes to the chart on the fly before deploying.

Jul 10, 2024

The popular Kubernetes deployment tool Helm has a post-renderer flag that lets you make changes to the chart on the fly before deploying.

The flag does what its name implies - make changes after rendering of the chart.

All examples in this article can be found in https://github.com/talonx/helm-examples/tree/main/nginx

Let’s look at how Helm processes a chart while deploying it to Kubernetes. A chart’s structure looks like this:

Templates
Metadata
YAML files with default values
YAML files with env specific values

You can run them from outside the nginx directory. The nginx chart is just the default chart created by ‘helm create’, with additional values files that let you override specific settings.

You can also override settings on the CLI during deployment:

helm -n nginx install nginx nginx -f nginx/dc1-values.yaml --set image=nginx-1.5.0

Helm renders the final YAML files which have the Kubernetes manifests after merging the values files and any CLI values with the templates. With this model you need to have all your settings in either the values files, or have to remember the CLI settings.

A typical workflow for your Helm charts from writing to deployment would look like this:

Helm’s rendering engine merges the chart’s templates and values files and produces the final set of YAMLs that will be sent to Kubernetes. At this point you might want to add something to the charts on the fly to tweak some deployment parameters (see Use Cases below). This can be done using helm’s post-renderer option. This takes a script which can transform the YAMLs on the fly.

A post-renderer script can be in any language. It has to be an executable and accept input from stdin and write the output to stdout.

The example below uses a simple Python script to add a nodeSelector to the deployment. The logic of choosing the node selector is separate from the charts, and can be tweaked independently. In this example it is hard-coded in the Python script but it can be read from, say, a config store or a database.

helm -n nginx install nginx nginx -f nginx/dc1-values.yaml --post-renderer=python/inject-affinity.py

If you run this command you can see that

      nodeSelector:
        nodeType: proxy

has been added to the Pod template spec in the Deployment.

It’s always a good idea to have a uniform way of deploying all your applications to Kubernetes, otherwise you can end up with a mess of deployment tools and flags, making it near impossible to roll-out and debug org-wide changes. In most orgs it’s a common tool like the CI/CD system or something custom-built. This allows you to add the post-renderer flag in one place, ensuring that all Helm deployments go through it.

Common Use Cases

Add node selector and node affinity settings for better bin packing and pod location optimization. These are often dynamically decided at a different layer in the pipeline and not by the chart author. Also, you might be using third-party charts that you wish to deploy to specific nodes.
Add a sidecar container to enable logging or metrics collection. The post-renderer script can add the sidecar container without the chart authors having to know about it (which is not possible with charts developed outside your organization anyway).
Flag vulnerable or blacklisted images in the charts before deployment. The post-renderer script can catch these and fail the deployment.

Post-renderer Contracts

Put your script on the $PATH, or supply an absolute or relative path to the helm command.
Your script should return a non-0 exit code on failure, and 0 on success.

Best Practices

Add the post renderer to your deployment pipeline. This ensures that all helm chart deployments pass through a central point and the changes are injected into every deployment.
Do not use arbitrary post-renderer scripts or binaries from the internet. They can pose a security risk as your chart’s rendered version can have sensitive information.

Debugging

One of the first things you might check to debug the case of a Helm chart value not being what you expect would be to look at the output of the Helm chart’s deployment. When you use a post renderer, there will be a difference between the output of executing helm with a –dry-run flag on your local machine and the actual YAML that is deployed. To see the actual YAML, run

helm -n nginx get manifest nginx

against your running Kubernetes cluster.

When you are developing the post-renderer script, one way to test it is to pipe the output of helm template to the script

helm -n nginx template nginx nginx -f nginx/dc1-values.yaml | python3 python/inject-affinity.py

Note that `helm template` does not send the command to the Kubernetes server unlike –dry-run, and this should be sufficient when you are testing the script.

Plug

I’ve just released a new version of IncidentHub with more channels and services, and a bunch of stability fixes in the backend. If you are looking for an easy way to monitor your third-party cloud and SaaS service vendors, I request you to give it a try.

Check out this year’s top 3 posts so far on Becoming a Better TechOps Engineer