Dhall is a programming language used for generating configuration files for various purposes. This Open Source project made its first release in 2018.
Like any new language for generating configuration files, Dhall seeks to address the limited functionality of YAML, JSON, TOML, XML, and other configuration formats while keeping things simple at the same time. The language has a steadily growing popularity. In 2020, Dhall bindings for Kubernetes were introduced.
Let’s start with a brief description of the language before discussing how Dhall facilitates K8s manifest creation.
How does Dhall differ from other languages?
The authors suggest treating Dhall like an advanced JSON: with all the functions, types, and imports. But why do we need a new format seeing that proven formats already exist?
Well, maybe just because?..
The authors cite the non-programmable nature of JSON and YAML as the main inspiration for developing Dhall. This non-programmable nature narrows these formats’ capabilities and sometimes leads to suboptimal solutions, such as repetitive code.
A focus on DRY
The DRY rule (“Don’t repeat yourself”) is considered good advice among developers. However, it’s hard to avoid repetition in working with JSON and YAML. The functional limitations of the configuration files lead to repetitive configuration fragments, which cannot be simplified or discarded.
Dhall is touted as a language that facilitates following the DRY principle. It allows you to replace a repetitive code block (a common occurrence in JSON or YAML) with a function result or a variable value. Below is an example from Dhall’s documentation that compares two config files in JSON and Dhall format. Each file essentially does the same thing: describe where the public and private SSH user keys are stored.
Here is the source JSON file:
[
{
"privateKey": "/home/john/.ssh/id_rsa",
"publicKey": "/home/john/.ssh/id_rsa.pub",
"user": "john"
},
{
"privateKey": "/home/jane/.ssh/id_rsa",
"publicKey": "/home/jane/.ssh/id_rsa.pub",
"user": "jane"
},
{
"privateKey": "/etc/jenkins/jenkins_rsa",
"publicKey": "/etc/jenkins/jenkins_rsa.pub",
"user": "jenkins"
},
{
"privateKey": "/home/chad/.ssh/id_rsa",
"publicKey": "/home/chad/.ssh/id_rsa.pub",
"user": "chad"
}
]And here is the same config in Dhall format:
-- config0.dhall
let ordinaryUser =
\(user : Text) ->
let privateKey = "/home/${user}/.ssh/id_rsa"
let publicKey = "${privateKey}.pub"
in { privateKey, publicKey, user }
in [ ordinaryUser "john"
, ordinaryUser "jane"
, { privateKey = "/etc/jenkins/jenkins_rsa"
, publicKey = "/etc/jenkins/jenkins_rsa.pub"
, user = "jenkins"
}
, ordinaryUser "chad"
]So far, the files have almost the same number of lines.
Now let’s add a new user, Alice. To do so, you have to insert five more lines into the JSON file:
[
…
{
"privateKey": "/home/alice/.ssh/id_rsa",
"publicKey": "/home/alice/.ssh/id_rsa.pub",
"user": "alice"
}
]Consequently, you risk making mistakes (even with simple copy-pasting): for example, you might copy the config for Chad but forget to change the name to Alice.
In Dhall, all you have to do is call the ordinaryUser function – it takes one line:
…
in [ ordinaryUser "john"
, ordinaryUser "jane"
, { privateKey = "/etc/jenkins/jenkins_rsa"
, publicKey = "/etc/jenkins/jenkins_rsa.pub"
, user = "jenkins"
}
, ordinaryUser "chad"
, ordinaryUser "alice" -- the line we're adding
]The more complex the configuration file, the more obvious JSON’s inflexibility compared to Dhall becomes.
Quick exporting to other formats
All you need is a single command to export the configuration into the desired format. For example, here is how you can convert the above Dhall file into JSON:
dhall-to-json --pretty <<< './config0.dhall'It is just as easy to export Dhall to YAML, XML, and Bash. Yes, you read that right: dhall-bash turns Dhall instructions into Bash scripts (although, alas, only a limited number of Bash functions is supported).
Emphasis on safety
While Dhall is a programming language, it is not a Turing complete one. The authors say that this restriction increases the security of both the code and the resulting configuration files.
Some tools use existing programming languages for configurations. For example, webpack supports TypeScript (and more), Django supports Python, sbt supports Scala, etc. However, the downside of this flexibility is possible problems that may arise related to insecure code, cross-site scripting (XSS), server-side request forgery (SSRF), and other attacks. Dhall is immune to all of this.
Dhall and Kubernetes
Okay, but what about using Dhall to create K8s manifests?
The fact that YAML is not well-suited to dealing with a large number of Kubernetes objects is mainly due to the specifics of its design. Although YAML supports basic templating, it is primarily a format for storing configurations. You can get around its limitations by using Helm templates, for example, but that is not always easy or even feasible — there is a detailed article on this topic in our blog. Another way is to use a different, more flexible configuration language, such as Dhall (we’ll list some alternatives below). The point is Dhall is a language with built-in patterns, which, at the same time, is not strongly typed. As the authors point out, it offers a simple description of the configuration no matter how many abstractions are created.
You can generate Kubernetes objects using Dhall expressions and then export them to YAML using the dhall-to-yaml utility. The public dhall-kubernetes repository contains so-called bindings – Dhall types and functions designed to work with Kubernetes objects. Here is how the Deployment configuration goes:
-- examples/deploymentSimple.dhall
let kubernetes =
https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/master/package.dhall sha256:532e110f424ea8a9f960a13b2ca54779ddcac5d5aa531f86d82f41f8f18d7ef1
let deployment =
kubernetes.Deployment::{
, metadata = kubernetes.ObjectMeta::{ name = Some "nginx" }
, spec = Some kubernetes.DeploymentSpec::{
, selector = kubernetes.LabelSelector::{
, matchLabels = Some (toMap { name = "nginx" })
}
, replicas = Some +2
, template = kubernetes.PodTemplateSpec::{
, metadata = Some kubernetes.ObjectMeta::{ name = Some "nginx" }
, spec = Some kubernetes.PodSpec::{
, containers =
[ kubernetes.Container::{
, name = "nginx"
, image = Some "nginx:1.15.3"
, ports = Some
[ kubernetes.ContainerPort::{ containerPort = +80 } ]
}
]
}
}
}
}
in deploymentExporting it to regular YAML using dhall-to-yamlwill result in the following:
## examples/out/deploymentSimple.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 2
selector:
matchLabels:
name: nginx
template:
metadata:
name: nginx
spec:
containers:
- image: nginx:1.15.3
name: nginx
ports:
- containerPort: 80As for the Dhall “modularity,” the authors consider cases when it is necessary to define: a) some type of MyService with settings for different deployments and b) functions that can be applied to MyService to create objects for K8s. This is handy since it allows you to define services outside the Kubernetes context and to reuse abstractions to handle other types of configuration files. Meanwhile, the DRY principle holds: to make a minor change in the configuration of several objects, all you need to do is change a function in a single Dhall file instead of having to edit all the related YAMLs by hand.
The configuration of the Nginx Ingress controller that sets up TLS certificates and routes for multiple services is an example of such “modularity”:
-- examples/ingress.dhall
let Prelude =
../Prelude.dhall sha256:10db3c919c25e9046833df897a8ffe2701dc390fa0893d958c3430524be5a43e
let map = Prelude.List.map
let kubernetes =
https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/master/package.dhall sha256:532e110f424ea8a9f960a13b2ca54779ddcac5d5aa531f86d82f41f8f18d7ef1
let Service = { name : Text, host : Text, version : Text }
let services = [ { name = "foo", host = "foo.example.com", version = "2.3" } ]
let makeTLS
: Service → kubernetes.IngressTLS.Type
= λ(service : Service) →
{ hosts = Some [ service.host ]
, secretName = Some "${service.name}-certificate"
}
let makeRule
: Service → kubernetes.IngressRule.Type
= λ(service : Service) →
{ host = Some service.host
, http = Some
{ paths =
[ { backend =
{ serviceName = service.name
, servicePort = kubernetes.IntOrString.Int +80
}
, path = None Text
}
]
}
}
let mkIngress
: List Service → kubernetes.Ingress.Type
= λ(inputServices : List Service) →
let annotations =
toMap
{ `kubernetes.io/ingress.class` = "nginx"
, `kubernetes.io/ingress.allow-http` = "false"
}
let defaultService =
{ name = "default"
, host = "default.example.com"
, version = " 1.0"
}
let ingressServices = inputServices # [ defaultService ]
let spec =
kubernetes.IngressSpec::{
, tls = Some
( map
Service
kubernetes.IngressTLS.Type
makeTLS
ingressServices
)
, rules = Some
( map
Service
kubernetes.IngressRule.Type
makeRule
ingressServices
)
}
in kubernetes.Ingress::{
, metadata = kubernetes.ObjectMeta::{
, name = Some "nginx"
, annotations = Some annotations
}
, spec = Some spec
}
in mkIngress servicesThis is the result of exporting to YAML using dhall-to-yaml:
## examples/out/ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.allow-http: 'false'
kubernetes.io/ingress.class: nginx
name: nginx
spec:
rules:
- host: foo.example.com
http:
paths:
- backend:
serviceName: foo
servicePort: 80
- host: default.example.com
http:
paths:
- backend:
serviceName: default
servicePort: 80
tls:
- hosts:
- foo.example.com
secretName: foo-certificate
- hosts:
- default.example.com
secretName: default-certificateIn this case, the defined services function was called twice: with the parameters specified in defaultService (with the default.example.com host) and with the manually passed values (with the foo.example.com host). Thus, the resulting manifest contains the Ingress resource with these two hosts.
Examples of use in K8s community
At the OSDNConf 2021 conference, Oleg Nikolin of Portside shared his story of how Dhall helped out his engineering team. With the migration to Kubernetes and the growing sophistication of the CI/CD process, the number of YAML configurations used in the company took a leap up to 12,000. Adding just one variable to one of the services usually led to changes in 40 different files stored in different repositories. It wasn’t easy to review the code. The problems were piling up. The situation was further complicated because they did not always manifest immediately after deployment. If the service had already been running with an incorrect configuration for some time, it was hard to trace the root of the problem.
After switching to Dhall and refactoring, the team got rid of 50% of the unnecessary configs. Dhall has improved CI/CD security: it is now easier to check K8s manifests and environment variables before deploying. The company has also created a global library with descriptions of all the K8s resources it uses. These steps have reduced the burden on the DevOps engineers and simplified configuration validation.
Christine Dodrill of Tailscale offers another excellent example of how Dhall facilitates the handling of YAML files. She wanted to streamline the checking of the K8s configuration files to make sure they were correct. Neither Helm nor Kustomize could handle that task, so she turned to Dhall. And here is her conclusion: “Dhall is probably the most viable replacement for Helm and other Kubernetes templating tools that I have found in recent memory”.
Other tools for creating manifests
Yes, there are other frameworks and languages that allow you to bypass YAML’s limitations and streamline the usage of Kubernetes manifests. Below is a list of Open Source projects with similar functionality:
- CUE is a language with a rich set of tools for defining, generating, and validating all kinds of data – configuration, APIs, database schemas, etc.
- Jsonnet is a data templating language. As its name suggests, jsonnet is a mix of JSON and sonnet. The language is similar to CUE in many respects as well.
- jk is a data templating tool designed to facilitate creating structured configurations, including K8s manifests.
- HCL is the HashiCorp configuration language. Its description reads: “HCL has both a native syntax, intended to be pleasant to read and write for humans, and a JSON-based variant that is easier for machines to generate and parse.”
- cdk8s is a framework for “programming” Kubernetes manifests using popular programming languages, such as JavaScript, Java, TypeScript, and Python. We’ve recently published a review on it.
Dhall’s criticism
Although Dhall’s syntax is pretty straightforward and its usage options are described in detail in the documentation, some may find it challenging to learn. To learn Dhall reasonably quickly, you need at least some basic experience with programming languages.
Some people agree that the existing languages for creating configurations have flexibility issues; however, they still do not like the solution that Dhall offers. As Andy Chu, Oil Shell programmer and creator, says, “total languages don’t give you any useful engineering properties, but a lack of side effects do (and Dhall also offers that).” This view is echoed by Earthly’s Adam Gordon Bell, who thinks “Dhall is strange,” even more strange than HCL, while the latter “already has mind share in the DevOps community”.
Summary
Despite its relative newness, Dhall is a fairly mature framework that is evolving thanks to the feedback it gets from the community. The project boasts 3,400+ stars on GitHub, a solid contributor base, and regular releases. Dhall is a viable alternative in cases where basic manifests and templates for them no longer do the trick.
Various companies, such as KSF Media, Earnest Research, and IOHK, use Dhall in their production to create and manage Kubernetes manifests.
Comments