26 August 2021
Timofey Kirillov, software developer

werf vs. Helm: Should you even compare them?

This article attempts to provide a detailed answer to a question we stumble upon every now and then: What is the difference between werf and Helm? At first glance, both of these tools seem to be designed with the same purpose: automating application deployment in Kubernetes. However, the reality is a little more complicated than that.

Relevance for CI/CD

The goals in developing werf and Helm are vastly different if you look at them from the perspective of the full CI/CD cycle.

Helm werf
Building an application (Docker) image
Pushing images to the container registry and cleaning them up after a while
Deploying to Kubernetes Deploying to Kubernetes (based on Helm) supplemented by resource tracking, image integration, Giterminism support, and other features

As you can see, werf does much more in terms of the whole CI/CD pipeline. It is involved in the entire application lifecycle, from building images to deployment to Kubernetes.

Helm is a good but somewhat low-level tool. To use it in CI/CD, you have to supplement it with add-ons and integrate it with other tools. In other words, using it inevitably entails a noticeable increase in the complexity of the infrastructure and processes.

That’s why we consider werf a next-generation tool for delivering applications to Kubernetes. In a sense, Helm works as a component of werf, while the latter integrates it with other standard tools: Git, Docker, and Kubernetes. Consequently, werf can be regarded as a “glue” that simplifies and unifies the organization of CI/CD pipelines based on specialized tools (that have already become the industry standard) and the selected CI system.

What werf can do and Helm can’t

As mentioned above, werf is not limited to deploying. Still, while Helm is good at what it brings to the Kubernetes ecosystem, werf provides several enhancements at that, even if we limit ourselves to the deployment function:

Features Helm werf
Awaiting resources to be ready during the deployment process + +
Resource tracking  and error detection +
Fail-fast system during the deployment process +
Protection against parallel deployment runs of the same release +
Integration with images being built +
Support for auto-attaching annotations and labels for all release resources + / − +
Pushing config files and application images to the container registry (bundles) + / − +
Basic support for secret values +
Giterminism and GitOps support + / − +

Below, we will discuss each item in the table (see the “How werf is different from Helm” below). But first, let’s talk about their origins. What were our goals in creating werf?

werf’s Four Noble Truths

1. werf is the universal tool for any CI/CD pipeline.

It must support any existing CI/CD system and easily integrate with it. Currently, werf has full built-in GitLab CI/CD and GitHub Actions integrations out-of-the-box. It also supports other CI systems — all you need to do is to write an integration script (detailed instructions are provided). E.g., this article is a good starting point to use werf with Jenkins.

2. werf delivers applications to Kubernetes in the most optimal way.

werf actively uses past builds/caches when delivering applications to Kubernetes. It thus only builds images (or layers for those images) that are missing in the container registry. This helps both cut back on build time and save storage space.

After the [updated] application is delivered to Kubernetes, all of its resources are updated to match the new desired state as defined in Git. We call this process Giterminism (from the root word “Git” + “determinism”; more details will follow below). Once again, werf calculates the changes required to synchronize the current state with the desired one and applies them.

3. werf provides clear, transparent feedback.

The werf summary report must contain all the information required to troubleshoot the problem. Then, if something goes wrong during the delivery and deployment process, the user won’t have to run kubectl and manually search for information in the cluster. werf displays from the start and constantly updates the current status of the deployment process. It provides all the information necessary to troubleshoot the problem with no need to involve a system administrator.

4. werf supports (and promotes) the GitOps approach.

GitOps is an approach that uses a single source of truth — the Git repository — to deploy applications to Kubernetes. Thus, you can manage the current state of your Kubernetes-based infrastructure in Git using declarative actions. This pattern works in werf out-of-the-box.

We have our own vision for putting GitOps-based CI/CD into practice — the Giterminism mentioned above. werf’s GitOps implementation supports the storage of Kubernetes manifests in Git as well as Git-linked versioning of the images built. With werf, you do not need to build any images to roll back to the previous application version (provided that the cleanup policy is configured accordingly and the images were already built). Other existing GitOps implementations do not provide such a guarantee.

Why and how werf uses Helm

werf uses and extends Helm to follow the above principles.

Helm is a popular, proven tool. It has its own template engine, charts, etc. There is no point in reinventing the wheel: we just use the same thing that already works perfectly. Instead, we prefer to focus on features that help optimize the CI/CD process.

Note that the Helm codebase is compiled in werf (and that is important for compatibility). You also don’t have to update it manually since it gets regular updates from the upstream (the same is true for werf – it has a built-in version manager called multiwerf which supports automatic upgrades and provides five release channels).

We even participate in improving Helm via the upstream. The annotation (“Delete the previous resource before a new hook is launched”) is an example of our contribution. It was adapted to Helm from werf.

Helm: Pros and cons

Helm has a number of advantages:

  • Helm is the package manager of choice for K8s. Helm has become the industry standard – almost everyone who works with Kubernetes uses Helm as well.
  • Templating: Helm templates are good for replicating manifests to use in various environments (even if some users don’t like Go-templates – but that’s another story).
  • Easy chart management: Helm provides a standard format for describing resources and combining them into charts (as Helm “packages”).
  • Re-using charts: Helm supports pushing charts to the Chart Repository or container registry so they can be reused.
  • Convenient management of the release lifecycle: Helm facilitates managing releases and rolling back to the desired versions.

As for the cons, the main downside is that Helm is just a tiny part of the CI/CD chain and almost has no connections to its other vital segments.

The CI/CD process entails the continuous merging of changes within the application’s main codebase as well as propagating these changes to the end-user. This process includes building new application images with updates, testing those images, and deploying new app versions. Usually, the CI/CD process deals with several environments (production, staging, development, etc.) and the configuration of the application may differ for each environment.

With Helm, you can use parameters to configure the chart (the application image to be used and the target environment). However, the exact method for doing this is not etched in stone. The Helm user has to decide for himself which one to use.

How werf is different from Helm

Now, let’s get back to the table above and analyze each item in more detail.

1. Resource tracking and error detection

Both werf and Helm await resources to become ready during deployment. However, werf provides feedback and is more proactive. It is distinguished by the three following major traits:

1. werf outputs logs for the resources being deployed during the deployment process. For individual resources, logging is disabled automatically after they become Ready. You can disable tracking of certain containers or write logs for specific containers only (while disabling logging for the rest). Furthermore, you can enable or disable the output of Kubernetes service information and customize werf behavior to suit your needs using annotations (as well as specify them in the chart templates).

2. werf follows the fail-fast principle. Thus, when it discovers an error within a particular resource, it immediately exits with an error and a non-zero exit code. werf provides the most helpful error-related information so that the user can see the root of the problem right away by browsing the CI/CD job’s output.

In contrast to werf, Helm waits for a timeout before marking the release as failed (in the event there are configuration problems). And such problems are pretty common in CI/CD. Unfortunately, the only way the user can discover what is causing them is via kubectl. This is inconvenient since:

  • first, kubectl access rights have to be configured;
  • using kubectl requires specialized knowledge and skills: where to look, what to look for, and how to interpret it.

Ideally, the developer will be able to determine the cause of the problem and fix it without involving the Ops engineers just by using the werf output. At that point, all you have to do is create a new commit with the fix in Git.

3. Protection against parallel deployment runs of the same release. Helm may crash with an error if there are parallel ongoing deployment runs of the same release. werf automatically prevents the deployment processes from running in parallel. They can only run one at a time.

2. Integration with images being built

In Helm, you have to explicitly pass the full names of the images via values and be sure to update them when changes occur.

werf does this for the user: you do not have to worry about passing image names since it passes them automatically using values. Furthermore, werf implements an optimal algorithm for image naming – it automatically updates the names for the images that have been changed.

In addition, with werf, you can easily roll back to the previous application version since it keeps images built for this version (so there is no need to rebuild them). Often, static image names (e.g., are passed to Helm. In this case, the image name refers to the last built image version. In this tagging model, you have to rebuild the image corresponding to the previous app version to roll back to it. werf implements a content-based tagging strategy in which the tags depend on the Git history. Among other things, this tagging strategy solves the problem of rolling back to the previous version.

What are the advantages of integration with built images?

  • werf can reuse existing Dockerfiles in the configuration.
  • werf assigns names to the images being built automatically and in an optimal fashion. These names depend on the image content and are only updated when changes occur.
  • There is no more need for unnecessary redeployments of the application components and, hence, downtime is minimized. The redeployment process is fast and only involves the components that have been changed (instead of the entire application).
  • Helm configuration is simplified since the only thing to do there is to use the image names that werf provides via the corresponding values.
  • You can easily roll back to the images of a previous application version.

3. Attaching annotations and labels to the release resources

The post rendering mechanism in Helm allows users to add annotations and labels to resources as well as edit other fields. However, Helm does not have a built-in function for adding labels and annotations specifically.

werf automatically adds annotations (such as a link to the CI/CD job that has been used for the last resource deployment or a link to a Git commit) to all release resources.

Furthermore, the user can specify (via the CLI parameters) custom annotations or labels to add to all release resources. Here is an example of such a command:

werf converge --add-annotation pipeline-id=$CI_PIPELINE_ID --add-annotation git-branch=$CI_COMMIT_REF_NAME

You can use this option:

  • for introspection: when you need to find out the CI/CD job with which the resource version is associated;
  • for monitoring: to collect information about cluster resources using annotations or labels.

4. Bundles: Pushing config files and app images to the container registry

Helm supports pushing charts to the OCI container registry or Chart Repository. However, it does not track images that the chart requires to deploy. Thus, the user must build these images and add the correct tags while the chart must be configured to ensure that the right images are used.

werf supports so-called bundles, which combine charts with images specifically built for them and put the resulting bundle in the container registry as a single entity.

The user does not have to be bothered about naming the images to push them together with the chart: werf does this automatically and does a fantastic job. The unchanged images will be reused; werf only pushes the layers required for the current commit. All the werf user has to do is choose the bundle version and ensure that it is a part of the target Git commit.

5. Built-in basic support for secret values

In Helm, third-party plugins are required for working with secrets. This complicates the installation of Helm on new hosts.

With werf, you can use the AES-128, AES-192, and AES-256 algorithms to encode values – no further steps need to be taken. You can also switch the encryption keys.

6. Giterminism and GitOps support

Helm does not regulate linking configuration files in use with Git commits.

werf (version 1.2 and up) forces the configuration from the current Git commit to being used and implements Giterminism mode (including for the Helm configuration).

werf reads the configuration files of the build context from the current commit in the application repository and excludes external dependencies. This ensures reliability and reproducibility. Configurations are easy to reproduce: developers can use images built in the CI system by switching to the corresponding commit. werf prohibits the use of non-committed or non-monitored files.


A direct comparison of werf vs. Helm is inappropriate since werf supports the entire application delivery lifecycle while Helm is specifically designed for the app’s deployment segment. Helm is an excellent tool for deploying applications to Kubernetes. That is why we built it into werf (with some improvements). Still, werf has several advantages with respect to the deployment process as well. These are primarily aimed at more comprehensive and convenient integration with CI/CD systems.