Hello Dagger

An example repo demonstrating Dagger.

What is Dagger?

Dagger is not a CI/CD Runner like GitHub Actions or Gitlab CI. It is an abstraction that uses and composes containers to run commands. Think of it as somewhere between just, nix, and writing custom programs for your build plus containers all over the place.

It exists as a cli tool which either starts a Dagger Engine locally or connects to an existing one (in the cloud). This engine runs as a container and runs all tasks inside the engine as a container as well. The cli tool communicates via GraphQL with the engine.

Besides the TUI, Dagger Cloud is a commercial offering that is basically a fancy add-on. It provides web-visualizations of errors during a dagger execution and of performance metrics (profiling), as well as some caching features.

The TUI and engine are published under the Apache 2 License and thus can be used free of charge.

Besides using containers, Dagger's main selling point is the ability to program tasks in high level languages like Go or Typescript instead of configuring them in json or yaml. Dagger provides a DSL-like API that reminds one of Dockerfile commands + FP (even though it isn't). Combined with the ecosystem of the base language nearly anything can be coded and containerized fairly easy.

Walkthrough

This repo uses mise / asdf for setup, see .tool-versions.

Initialize the existing repo with dagger:

dagger init --sdk=typescript --source=./dagger

Typescript is just one option to write the config in, Python and Go are also currently available with more to follow.

Note: For some reason it wants to create a LICENSE file...

LSPs should work properly out of the box as the dagger config is located in the ./dagger directory. When using TS this is a self-sufficient node/yarn project.

To get an overview of available functions to call:

dagger functions

Run a function:

dagger call build --source=.

This runs the build function defined in dagger/src/index.ts and passes . as the source parameter of that function. The return type of this function is a Container type which just prints some general information on the resulting container when called this way.

_type: Container
defaultArgs: []
entrypoint:
    - helloworld
mounts: []
platform: linux/amd64
user: ""
workdir: ""

The Container has to be consumed by other functions to actually export it to some registry or run it. The same is true for other types like File and Directory which also only expose the actual artifacts through 'effectful' functions, thus the FP-like feeling. Only 'simple' strings can be exposed on the shell. Consequently, pipelining on the shell seems not to be a common thing.

Dagger separates the containerized build environment from the host system. Unless specifically specified, like with --source=., dagger does not expose host resources. This adds to the notion of trying to create reproducible actions.

Camel Case function names in the TS setup are available in Kebab Case on the shell (same applies for Go and Python). This does not only apply to your own declared functions but to all functions available on the returned data types.

this.build(source).asService().up({ ports: "8080:9000" });

becomes

dagger call build --source=. as-service up --ports=8080:9000

Dagger calls this function chaining.

After running a dagger command the Dagger Engine stays alive as a local container waiting for the next call.

The Cookbook does contain a lot of extremely useful patterns on how to handle builds and containers. More importantly, it showcases ways to debug everyday problems when working on container builds, see temrinal(), File.contents()

Run golangci-lint:

dagger call lint --source=.

Run trivy:

dagger call security-scan --source=.

Trivy is setup as a dagger module. Modules are most of the time simple wrappers instructing container images with some convenience functions. There is actually not too much magic here.

Modules are extension libraries in dagger that can be written in Go, Python, or Typescript. Either way, they are usable in any other language, e.g. a Python module can be used in a Typescript config. The proper API functions are exposed on the dag object.

Push the container image to some registry:

dagger call build --source=. publish --address=mtr.devops.telekom.de/...

This call makes use of the aforementioned chaining capabilities. One could create a custom function to chain calls within the config or just chain call in a shell call.

Deployment to Kubernetes

Dagger at its core does not provide tooling to deploy artifacts to kubernetes or anywhere else. There are, however, modules that expose APIs for, for example, kubectl and helm. Creating automatic deployments based on these should be straight forward. But at this point one could argue to use better suited tooling for this task rather than to manually code it again.

CI

Dagger does not provide a Job Runner. Thus, it has to run within a system like GitHub Actions, Gitlab CI, or Jenkins. Once it's running, any workload within dagger should run seamlessly like on your local machine thanks to dagger's container abstraction.

A critical problem however is the fact that the dagger engine requires running in privileged mode, see FAQ. Depending on the platform this might be a dealbreaker due to security issues. If the platform does not provide a somewhat isolated vm environment that is capable of running containers itself (in this case the dagger cli would spawn its own instance of the dagger engine), the suggested setup includes hosting one or more shared dagger engines within the platform, see the Gitlab CI kubernetes example. Security concerns of using a shared instance of the dagger engine should be investigated if one wants to go down this road.

Conclusion

Dagger is an extremely neat tool that puts DX front and center. Easy and fast setup on proper dev machines is a big plus. Intuitive APIs, easy programmability and containers make creating a build and test environment an actually nice experience. The debugging capabilities of your build pipeline are excellent.

However, dagger only covers the pipeline up until the point where you push your image to a registry. After that you are more or less on your own. This is actually fine as it is somewhat out of scope.

The requirement of privileged mode and container capabilities in general for the dagger engine is probably a big headache not only in CI but also on weirdly restricted dev machines. Furthermore, during testing, dagger seems to not create actually reproducible builds by default but has to be at least explicitly configured accordingly, but this needs further investigation. Nix might be a worthwhile alternative.