4.8 KiB
4.8 KiB
CI/CD pipeline tools for composable pipeline
Context and Problem Statement
In order to build a composable pipeline that provides a golden path and reusable components, we need to define the tools that will be used to execute the pipeline.
ArgoCD is considered set in stone as the tool to manage the deployment of applications. However, the tools to compose and execute the pipeline are still up for debate.
Note: The pipeline will use many other tools to perform certain actions such as testing, building, and deploying. This ADR is focused on the tools that will be used to compose and execute the pipeline itself.
In general, there are 2 decisions to make:
- What tools should we use to execute the pipeline?
- What tools should we use to compose the pipeline?
The following use-cases should be considered for this decision:
- User who wants to manage their own runners (???)
- User who only wants to use our golden path
- User who wants to use our golden path and add custom actions
- User who wants to use their own templates and import some of our actions
- User who wants to import an existing GitHub repository with a pipeline
Considered Options
- Argo Workflows + Events
- Argo Workflows + Events + Additional Composition tool
- Forgejo Actions
- Forgejo Actions + Additional Composition tool
- Dagger (as Engine)
- Shuttle (as Engine)
Decision Outcome
TBD
Pros and Cons of the Options
Argo Workflows + Events
Pro
- integration with ArgoCD
- ability to trigger additional workflows based on events.
- level of maturity and community support.
Con
- Ability to self-host runners?
- way how composition for pipelines works (based on Kubernetes CRDs)
- Templates must be available in the cluster where the pipelines are executed, so any imported templates must be applied into the cluster before the pipeline can be executed and cannot simply reference a repository
- This makes it difficult to import existing templates from other repositories when using self-hosted runners
- This also makes it difficult to use our golden path, or at least we will need to provide a way to import our golden path into the cluster
- This also makes the split of every component has its own repo very difficult
- additional UI to manage the pipeline
- Additional complexity
Argo Workflows + Events + Additional Composition tool
Pro
- Composability can be offloaded to another tool
Con
- All cons of the previous option (except composability)
- Additional complexity by adding another tool
Forgejo Actions
Pro
- tight integration with GitHub Actions providing a familiar interface for developers and a vast catalog of actions to choose from
- ability to compose pipelines without relying on another tool
- Self-hosting of runners possible
- every component can have its own repository and use different tools (e.g. written in go, bash, python etc.)
Con
- level of maturity - will require additional investments to provide a production-grade system
Forgejo Actions + Additional Tool
Pro
- may be possible to use GitHub actions alongside another tool
Con
- additional complexity by adding another tool
Shuttle
Pro
- Possibility to clearly define interfaces for pipeline steps
- Relatively simple
Con
- basically backed by only one company
- centralized templates, so no mechanism for composing pipelines from multiple repositories
Dagger
Pro
- Pipeline as code
- if it runs it should run anywhere and produce the "same" / somewhat stable results
- build environments are defined within containers / the dagger config. Dagger is the only dependency one has to install on a machine
- DX is extremely nice, especially if you have to debug (image) builds, also type safety due to the ability to code your build in a strong language
- additional tooling, like trivy, is added to a build pipeline with low effort due to containers and existing plugin/wrappers
- you can create complex test environments similar to test containers and docker compose
Con
- relies heavily containers, which might not be available some environments (due to policy etc), it also has an effect on reproducibility and verifiability
- as a dev you need to properly understand containers
- dagger engine has to run privileged locally and/or in the cloud which might be a blocker or at least a big pain in the ...
Suggestion Patrick
- dagger is a heavy weight and might not be as productive in a dev workflow as it seems (setup lsp etc)
- it might be too opinionated to force on teams, especially since it is not near mainstream enough, community might be too small
- it feels like dagger gets you 95% of the way, but the remaining 5% are a real struggle
- if we like it, we should check the popularity in the dev community before further considering as it has a direct impact on teams and their preferences