In this article, I will walk you through the process of automating the issuance and renewal of certificates provided by Let’s Encrypt (and other services) for Kubernetes Ingress using the cert-manager add-on. But first, let me start with a brief introduction to the problem.

HTTPS in a nutshell

The HTTP protocol developed in the early 1990s has become an integral part of our daily life. Today, we cannot live a single day without it. However, it does not even provide a basic level of security when exchanging information between the user and the web server. That is when HTTPS (“S” means “secure” here) comes to the rescue. In HTTPS, the exchanged data is encrypted using SSL/TLS — that family of protocols has proven itself well in protecting privacy and data integrity and is being actively promoted by the industry.

For example, Google has been calling for “HTTPS everywhere” since 2014. When prioritizing search results, it takes into account whether sites use secure, encrypted connections. All this propaganda affects ordinary users as well: modern browsers warn their users about insecure connections and invalid SSL certificates.

A certificate for a personal website might cost tens of dollars. However, buying it is not always justified. Fortunately, since late 2015, there is a free alternative in the form of Let’s Encrypt (LE) certificates. This nonprofit authority was created by Mozilla enthusiasts to make internet-wide hassle-free encryption a reality.

The certificate authority issues domain-validated certificates (the most basic ones available on the market) valid for 90 days, and it is also possible to obtain a so-called wildcard certificate for several subdomains.

The algorithms described in the Automated Certification Management Environment (ACME) protocol (designed specifically for Let’s Encrypt) are used to obtain a certificate. With it, the agent can prove control of the domain either by provisioning an HTTP resource (the so-called “HTTP-01 challenge”) or DNS records (“DNS-01 challenge”) — more information about them you may find below.

Cert-manager

Cert-manager is a Kubernetes-native certificate management controller consisting of a set of CustomResourceDefinitions (hence the restriction on the minimum supported version of K8s, v1.12) for configuring CA (certificate authorities) and obtaining certificates. The installation of CRDs in a cluster is straightforward and boils down to applying a single YAML file:

kubectl create ns cert-manager
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.13.0/cert-manager.yaml

(You can also install it with Helm).

Before being able to request certificates, you must create CA resources: Issuer or ClusterIssuer. They are used for signing CSRs (certificate requests). The difference between these resources is that ClusterIssuer is non-namespaced and can be used in multiple namespaces:

• Issuer is used within a single namespace only;
• ClusterIssuer is a global cluster object.

Using cert-manager

Method #1. Self-signed certificate

Let’s start with the simplest case — requesting a self-signed certificate. It is quite common. For example, you can use it in your K8s cluster for testing environments dynamically created for developers needs. It can be also useful in the case of the external load balancer that terminates SSL traffic.

The Issuer resource would look like this:

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}

To issue a certificate, you have to define the Certificate resource that determines the issuance (see the issuerRef section below) and the location of the private key (the secretName field). Then you need to invoke that key in the Ingress (note the tls section in the spec field):

---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: selfsigned-crt
spec:
  secretName: tls-secret
  issuerRef:
    kind: Issuer
    name: selfsigned
  commonName: "yet-another.website"
  dnsNames:
  - "yet-another.website"
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: app
spec:
  tls:
  - hosts:
    - "yet-another.website"
    secretName: tls-secret
  rules:
  - host: "yet-another.website"
    http:
      paths:
      - path: /
        backend:
          serviceName: app
          servicePort: 8080

The certificate will be issued in a few seconds after these resources are added to the cluster. You can see the confirmation in the output of the command:

kubectl -n app describe certificate selfsigned-crt
...
  Normal  GeneratedKey  5s    cert-manager  Generated a new private key
  Normal  Requested     5s    cert-manager  Created new CertificateRequest resource "selfsigned-crt-4198958557"
  Normal  Issued        5s    cert-manager  Certificate issued successfully

Looking at the secret resource itself, you will see:

• the tls.key private key,
• the ca.crt root certificate,
• and the self-signed tls.crt certificate.

You can browse the contents of these files using the openssl utility:

kubectl -n app get secret tls-secret -ojson | jq -r '.data."tls.crt"' | base64 -d | openssl x509 -dates -noout -issuer
notBefore=Feb 10 21:01:59 2020 GMT
notAfter=May 10 21:01:59 2020 GMT
issuer=O = cert-manager, CN = yet-another.website

It is worth noting that clients consuming certificates issued in such a way (i.e., using the self-signed issuer) will not trust them. The reason is simple: this issuer type does not have a CA (see the note on the project website).

To avoid this, specify the path to the secret file containing ca.crt in the Certificate resource. For example, you can use the corporate CA. This way, you will be able to sign certificates issued for your Ingress with a key that is already in use by other server services/information systems.

Method #2. Let’s Encrypt certificate with the HTTP validation

As I mentioned before, there are two types of “challenges” available to prove the control of the domain, HTTP-01 and DNS-01.

The first approach (HTTP-01) involves deploying a tiny web server as a separate deployment. It will be serving some information at the http:///.well-known/acme-challenge/ URL per request of the certification server. Therefore, this method implies the accessibility of Ingress from the outer world via port 80 and the publicity of the domain’s DNS record.

The second challenge (DNS-01) makes sense if there is an API you can use to change the DNS records of your domain. The Issuer uses these tokens to create TXT records for your domain. Then, ACME server gets these records during confirmation. Let’s Encrypt easily integrates with various DNS providers, including CloudFlare, AWS Route53, Google CloudDNS, and others (as well as with the LE’s own DNS implementation, acme-dns).

Note: Let’s Encrypt imposes fairly strict limits on requests to ACME servers. To avoid unnecessary load on LE’s production environment, we recommend using the letsencrypt-staging certificate for testing (the difference is in the ACME server only).

So, let’s describe the resources:

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory 
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
       ingress:
         class: nginx
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: le-crt
spec:
  secretName: tls-secret
  issuerRef:
    kind: Issuer
    name: letsencrypt
  commonName: yet-another.website
  dnsNames:
  - yet-another.website

Note that we use the staging server in acme’s server field for our Issuer. You can replace it with the production one later.

Let’s apply this configuration and trace the entire process of obtaining the certificate:

1. The creation of a Certificate leads to the emergence of a new CertificateRequest resource:

kubectl -n app describe certificate le-crt
...
Created new CertificateRequest resource "le-crt-1127528680"

2. In its description there is a notification about an Order creation:

kubectl -n app describe certificaterequests le-crt-1127528680
…
Created Order resource app/le-crt-1127528680-1805948596

3. The Order contains the description of parameters of the validation and its current status. The validation is performed by the Challenge resource:

kubectl -n app describe order le-crt-1127528680-1805948596
…
Created Challenge resource "le-crt-1127528680-1805948596-1231544594" for domain "yet-another.website"

4. And finally, there is information about the status of the validation itself in resource details:

kubectl -n app describe challenges le-crt-1127528680-1805948596-1231544594
...
  Reason:      Successfully authorized domain                                                                                                                                                                      
...
  Normal  Started         2m45s  cert-manager  Challenge scheduled for processing
  Normal  Presented       2m45s  cert-manager  Presented challenge using http-01 challenge mechanism
  Normal  DomainVerified  2m22s  cert-manager  Domain "yet-another.website" verified with "http-01" validation

The certificate will be issued in less than a minute if all the prerequisites are met: the domain is accessible from the outer world, the rate limits on the LE side are respected, and so on. If the issuance was successful, you should see the message “Certificate issued successfully” in the output of the describe certificate le-tls command.

Now you can safely change the ACME server address to the production one (https://acme-v02.api.letsencrypt.org/directory) and re-issue valid certificates signed by Let's Encrypt Authority X3 instead of Fake LE Intermediate X1.

But first, you have to delete the Certificate resource. Otherwise, the issuance procedure will not start because the certificate already exists, and it is valid. Deleting a secret would immediately result in invalidation of the certificate with the following message in the output of the describe certificate command:

  Normal  PrivateKeyLost  44s                   cert-manager  Lost private key for CertificateRequest 'le-crt-613810377', deleting old resource

Now it is time to apply the production manifest for the Issuer with the Certificate described above (it has not changed):

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
       ingress:
         class: nginx

After receiving the Certificate issued successfully confirmation, let us check it out:

kubectl -n app get secret tls-secret -ojson | jq -r '.data."tls.crt"' | base64 -d | openssl x509 -dates -noout -issuer
notBefore=Feb 10 21:11:48 2020 GMT
notAfter=May 10 21:11:48 2020 GMT
issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3

Method #3. Validating Wildcard LE over DNS

To make one step further we will issue a certificate for all subdomains of the site using another method of validation — the DNS one. We will use CloudFlare as our DNS provider to change the domain records we need.

First, let’s create a token to use the CloudFlare API:

1. Profile → API Tokens → Create Token.

2. Set access rights as follows:

• Permissions:
  • Zone — DNS — Edit
  • Zone — Zone — Read

• Zone Resources:
  • Include — All Zones

3. Copy the token generated (for example, y_JNkgQwkroIsflbbYqYmBooyspN6BskXZpsiH4M).

Create a Secret resource containing the token and describe it in your Issuer:

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token
type: Opaque
stringData:
  api-token: y_JNkgQwkroIsflbbYqYmBooyspN6BskXZpsiH4M
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory 
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - dns01:
        cloudflare:
          email: my-cloudflare-acc@example.com
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token

---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: le-crt
spec:
  secretName: tls-secret
  issuerRef:
    kind: Issuer
    name: letsencrypt
  commonName: yet-another.website
  dnsNames:
  - "yet-another.website"
  - "*.yet-another.website"

(Do not forget to use a staging environment for testing!)

It is time to go through the domain ownership confirmation procedure:

kubectl -n app describe challenges.acme.cert-manager.io le-crt-613810377-1285319347-3806582233
...
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for dns-01 challenge propagation: DNS record for "yet-another.website" not yet propagated
  State:       pending
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Started    54s   cert-manager  Challenge scheduled for processing
  Normal  Presented  53s   cert-manager  Presented challenge using dns-01 challenge mechanism

The TXT record will appear at your DNS dashboard:

… and after a while, the status will change to:

Domain "yet-another.website" verified with "dns-01" validation

Let’s make sure that the certificate is valid for all subdomains:

kubectl -n app get secret tls-secret -ojson | jq -r '.data."tls.crt"' | base64 -d | openssl x509 -dates -noout -text |grep DNS:
          DNS:*.yet-another.website, DNS:yet-another.website

The validation over DNS is usually slow since most DNS providers have a so-called “propagation time” — a period showing how long it takes for an updated DNS record to become available on all DNS servers of the provider.

The ACME standard also supports a combination of both types of validation. You can use it to speed up obtaining a certificate for the main domain. In this case, the description of the Issuer will have the following form:

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - selector:
        dnsNames:
        - "*.yet-another.website"
      dns01:
        cloudflare:
          email: my-cloudflare-acc@example.com
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token            
    - selector:
        dnsNames:
        - "yet-another.website"
      http01:
        ingress:
          class: nginx

If you apply this configuration, two Challenge resources will be created:

kubectl -n app describe orders le-crt-613810377-1285319347
…
  Normal  Created  3m29s  cert-manager  Created Challenge resource "le-crt-613810377-1285319347-3996324737" for domain "yet-another.website"                 
  Normal  Created  3m29s  cert-manager  Created Challenge resource "le-crt-613810377-1285319347-1443470517" for domain "yet-another.website"

Method #4. Using special Ingress annotations

Besides creating certificates directly, you can use cert-manager’s ingress-shim component. It relieves you of the need to create Certificate resources explicitly. The idea is to obtain a certificate automatically using the Issuer specified in the special annotations of Ingress. Here is an example of the respective Ingress resource:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - "yet-another.website"
    secretName: tls-secret
  rules:
  - host: "yet-another.website"
    http:
      paths:
      - path: /
        backend:
          serviceName: app
          servicePort: 8080

In this case, the availability of Issuer is enough to get the work done, meaning that we have to create a lesser number of entities.

Also, there is an obsolete kube-lego annotation — kubernetes.io/tls-acme: "true". The peculiarity about it is that when deploying cert-manager, the default Issuer must be specified via Helm arguments (or by appending arguments of cert-manager’s deployment container).

Personally, we do not use these approaches (mentioned in Method #4) and cannot recommend them due to their opacity (and various associated problems). However, it’s good to mention them in the article to provide a fuller picture.

Key takeaways

We have learned how to obtain auto-renewable, self-signed, and free SSL certificates from Let’s Encrypt for website domains that are managed by Ingresses in the Kubernetes clusters.

The article provides example solutions for the most common problems we face. However, cert-manager features are not limited to those described above. On the project’s website, you can find examples of using it with other services. For example, you can use Vault as a certificate authority or set up external Issuers.