Blog
DevOps as a Service from Flant Dedicated 24x7x365 support Discover the benefits
5 May 2022
Konstantin Nezhbert, technical writer

Running werf in GitLab CI/CD without a Docker server

werf is an Open Source tool for building applications and deploying them to Kubernetes. This article discusses the new experimental werf operating mode that does not require a Docker server to be run. You will learn how to ensure this mode works properly on your machine, build an image, and use the Kubernetes executor to automate builds in GitLab CI/CD.

Introduction

In order to render the build environment predictable and reproducible, you have to run builds in separate, isolated containers or single-use virtual machines. In the case of GitLab CI/CD, you can use the Docker executor for builds. The Docker executor starts the Docker container on the GitLab Runner. The shell instructions defined in the GitLab CI are then executed in this container. In the case of werf, these instructions include werf build, werf converge, and werf cleanup.

Alternatively, you can use the Kubernetes executor: like its Docker counterpart, it renders the environment for running werf predictable and reproducible, with the building performed in the Kubernetes Pods. This renders scaling easier by moving workloads from GitLab Runners to Kubernetes while allowing other K8s cluster resources to be used.

There are ready-made images to run werf in Docker/Kubernetes. They can also be used to execute GitLab CI instructions in a Docker or Kubernetes executor. We’ll show you how to use them below.

How this works in werf

werf supports two operating modes: with and without a Docker server. The latter mode is experimental.

When operating in the latter mode, werf uses the built-in Buildah in rootless mode instead of the Docker server/client.

Currently, this mode only supports Dockerfile-based building. The alternative Stapel builder will be available in the near future.

Let’s look at how you can use the new mode to build images in Kubernetes without a Docker server. To do so, you first need to set up the environment and make sure the system supports Buildah. After that, you can build the image using the new mode and automate the assembly process using the Kubernetes executor in GitLab CI/CD.

1. Preparing the environment

A Docker server is not required in Linux operating systems where werf can natively use Buildah. In Windows and macOS, you can’t do that without a Docker server due to a compatibility layer (a Docker image containing Buildah) required to run Buildah in these operating systems.

Now let’s look at using this mode to enable and run werf on Linux.

Before enabling the experimental mode, make sure that your system supports the rootless OverlayFS file system. Note that Linux kernel versions 5.11 and up support it by default (strictly speaking, a bugfix to enable rootless OverlayFS in SELinux was first introduced in version 5.13; however, many major Linux distributions have backported it into v5.11). You can use fuse-overlayfs if your system kernel does not support rootless OverlayFS.

As mentioned above, there are pre-made werf images that you can use. The images are kept up to date and follow a release schedule: the trdl package manager delivers werf as part of the release process within various release channels (from alpha to stable).

There are several images available. We recommend using the ghcr.io/werf/werf:latest image. It contains the werf 1.2-stable and is based on Alpine Linux (hence its full name is ghcr.io/werf/werf:1.2-stable-alpine). That’s the image we’re going to use in our example.

You can learn more about all the available images here.

In Buildah mode, werf can also be used without containers, e.g., if you want to build images containing werf yourself instead of using ready-made ones provided by developers. See the documentation to learn more.

2. Choosing the operating mode in the container

Depending on the user’s needs as well as the system parameters, there are three ways to use werf in Buildah mode in containers:

  1. Linux kernel with rootless OverlayFS: In this mode, all you need to do is disable the seccomp and AppArmor profiles in the werf container.
  2. Linux kernel without rootless OverlayFS and a privileged container: In the event that your Linux kernel does not support rootless OverlayFS, Buildah and werf will use fuse-overlayfs instead. Note that you have to run werf in a privileged container in order to enable fuse-overlayfs.
  3. Linux kernel without rootless OverlayFS and a non-privileged container with additional settings: Similar to the previous mode, Buildah and werf will use fuse-overlayfs as the file system. To use fuse-overlayfs without a privileged container, run werf in a container with the following parameters:
    1. The AppArmor and seccomp profiles are disabled.
    2. The /dev/fuse device is enabled.

In our case, we will use the most common and preferred operation mode – #1 (unprivileged container with rootless OverlayFS).

3. Building an image without a Docker server

Run the following command to start the build:

docker run \
    --security-opt seccomp=unconfined --security-opt apparmor=unconfined \
    ghcr.io/werf/werf:latest werf_command

The above command will run the selected container (ghcr.io/werf/werf:latest) and execute the werf command (werf_command) it contains.

4. Setting up a build in GitLab CI/CD that uses the Kubernetes executor

Configure GitLab runner in Kubernetes

Edit the /etc/gitlab-runner/config.toml file:

[[runners]]
  name = "kubernetes-runner-for-werf"
  executor = "kubernetes"
  ...
  [runners.kubernetes]
    ...
    pod_annotations = ["container.apparmor.security.beta.kubernetes.io/werf-converge=unconfined"]

NB. See the official documentation to learn more about additional Kubernetes executor parameters.

Configure access to the Kubernetes cluster

There are two ways to configure access to the cluster in which the application is deployed:

  • The first one involves a Service Account for the Kubenetes executor;
  • The second one is based on the kubeconfig file with the appropriate settings.

Let’s look at each option in more detail.

Service Account

Note that this method is only suitable if the Kubernetes executor is running in the target Kubernetes cluster.

See an example of gitlab-kubernetes-runner-deploy Service Account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-kubernetes-runner-deploy
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gitlab-kubernetes-runner-deploy
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: gitlab-kubernetes-runner-deploy
    namespace: default

Configure the GitLab runner (/etc/gitlab-runner/config.toml) to use this Service Account:

[[runners]]
  name = "kubernetes-runner-for-werf"
  ...
  [runners.kubernetes]
    service_account = "gitlab-kubernetes-runner-deploy"
    ...

Kubeconfig

This method requires assigning the base64-encoded contents of the ~/.kube/config file to the WERF_KUBECONFIG_BASE64 environment variable in the GitLab project. werf will automatically use this configuration to connect to the target Kubernetes cluster.

This method works well if the Kubernetes executor’s cluster and the target Kubernetes cluster are different.

Configure the gitlab-ci.yml file

Now for the final step. Below is an example of a basic task for building and deploying a project in GitLab CI/CD:

stages:
  - build-and-deploy

Build and deploy application:
  stage: build-and-deploy
  image: ghcr.io/werf/werf
  script:
    - source $(werf ci-env gitlab --as-file)
    - werf converge
  tags: ["kubernetes-runner-for-werf"]

There we have it. Now everything is ready to run the build inside a werf container running in a Kubernetes cluster.

Takeaways

In this article, we discussed how werf can be used in GitLab CI/CD without relying on a Docker server, thereby rendering the build environment predictable and reproducible. We used a Kubernetes executor to run the build in a separate Pod in the cluster using the pre-made werf container.

We strongly recommend checking the werf documentation for a detailed description of all the available operating modes that do not require a Docker server to be run, as well as for issues that may arise and solutions for them.

Share