Skip to content
BranchesTags

Repository files navigation


Argo CD Conductr - GitOps Everything 🧪


Report Bug · Request Feature

Table of Contents
  1. About The Project
  2. Getting Started
  3. TODO
  4. Known Issues
  5. References
  6. License

About The Project

The primary goal of this project is to exercise with Argo CD based GitOps deployment covering the full cycle - up to production via promotion, if you want to. Experimentation and production should not conflict.

The change process starts at localhost. Hence, we consider kind experience very important. Given that, some elements may be useful in CI context. Most things, should play nice productive environments as well.

Demo using opentofu bootstrapping a single node kind cluster showing deployments,statefulsets and daemonsets as they enter their desired state 🪄🎩🐰

Demo

Goals

  • Speed : A fast cycle from localhost to production 🚀
  • Fail early and loud (notifications)
  • Scalability
  • Simplicity (yes, really)
  • Composability
  • Target kind, vanilla Kubernetes and Openshift including crc

Non Goals

Decisions

Stage propagation is hard. Folders, branches, repos, ... you name it. All those come with pros and cons and it ends up being a tradeoff. Apparently, it is so hard that a dedicated project kargo was born to solve it. Long story short:

We have started using kargo, and we are trying the migration following effective processes for monorepos while using a single long lived branch (unchanged as we did before) and the "Rendered Config" pattern. Essential bits appear to be working with the kargo (environment/stage), and we may even get way without changing the folder structure.

We use single level environment staging with one cluster per environment. We do not use names and namespaces in this context, and we don't even dare to do multi-tenancy in a single cluster (OLMv1 drops it). This should help with isolation, loose coupling, support the cattle model and keep things simpler. We want cluster scoped staging. Using another nested level introduces issues ("Matrjoschka Architecture").

We prefer Pull over Push.

We focus on one "Platform Team" managing many clusters using a single repo. It should enable ArgoCD embedding for Application verticals.

Following the App of Apps pattern, our kind-olm root Application is at (envs/kind-olm). The root app kicks off various ApplicationSets covering similarly shaped (e.g. helm/kustomize) apps hosted in apps. Within that folder, we do not want Argo CD resources. This helps with separation and quick testing cycles.

OLM footprint has a bigger footprint than helm and it comes with its own set of issues as well. It is higher level and way more user friendly. With some components (e.g. Argo CD, Loki, LVM) helm is the second class citizen. With others (e.g. Rook), it's the opposite. We prefer first class citizens. Hence, we default to bring in OLM when it is not there initially (such as on kind).

Features

We cover deployments of:

  • Argo CD (self managed)
  • Argo CD Notifications
  • Argo-CD Image-Updater
  • Argo Rollouts
  • Argo Events
  • Operator Lifecycle Management
  • Kargo
  • Kube-Prometheus
  • Loki
  • Velero
  • Cert-Manager
  • Sealed Secrets
  • SOPS Secrets
  • Submariner
  • LitmusChaos

Beyond deployments, we feature:

  • mise aiming at a more uniform environment locally and in CI
  • make based tasks
  • Github Actions integration
  • Prometheus Rule Unit Testing
  • A bare bones alerting application in case want to send alerts to very custom receivers (like Matrix Chat Rooms)
  • Open Cluster Management / Submariner multi-cluster setup (dormant)
  • Kargo promotion pipelines — per-app Rendered Configs and whole-env promotion to a second cluster (see docs/kargo-promotion.md)

(back to top)

Getting Started

Some opinions first:

  • YAML at scale is ... terrible. Unfortunately, there is no way around.
  • CI/CD usually comes with horrible DX: “.. it’s this amalgamation of scripts in YAML tied together with duct tape.”
  • CI/CD should enable basic inner loop local development. Should not be git commit -m hoping-for-the-best && git push
  • Naming ... is hard
  • Joining clusters is hard (e.g. Submariner)
  • Beware of Magic 🎩🪄🐰 (e.g. Argo CD helm release changes when Prometheus CRDs become available)
  • Beware of helm shared values or kustomize base. We deploy main and shared bits kick in on all environments.
  • Versions/Refs: Pin or Float? It depends. We should probably pin things in critical environments and keep things floating a bit more elsewhere
  • Don't try too hard modeling deps and ordering. Failing to start a few times can be perfectly fine. Honor this modeling your alerts.
  • We should propagate to production frequently.
  • Rebuilding whole things automatically from scratch matters a lot. Drift kicks in fast and it helps with Recovery.
  • Bootstrapping OLM is painful - thanks god, there is a helm chart these days.
  • Using Kubernetes bits in Terraform (e.g. helm, kustomize, kubectl, kubernetes providers), only use the bare minimum (because deps are painful) This is an example of how you may give instructions on setting up your project locally. To get a local copy up and running follow these simple example steps.

Prerequisites

  • make
  • kubectl
  • mise (highly recommended)
  • docker (if using kind)
  • opentofu (required to bring up a cluster)
  • helm (only for the deprecated no-opentofu install path)

Usage

For basic demo purposes, you can use this public repo. If you want to run against your own, replace the git server reference with your own.

The root Makefile is the entrypoint. Run make for the full list of targets; the Clusters section brings everything up from scratch (kind cluster → cilium → Argo CD → root app) via the OpenTofu module in ./tf, which it drives for you:

make cluster-up            # default hub cluster (helm flavor) - the usual entrypoint
make cluster-workload-up   # optional second "workload" cluster (pull model)
make kargo-connect         # wire the workload Kargo shard back to the hub
make cluster-down          # tear the current cluster back down

cluster-up picks the right tofu workspace and tfvars per cluster, so you never touch tofu workspace directly. The OLM-preferring kind-olm flavor is built the same way but in its own workspace with tf/terraform-batman.tfvars (see tf/ for the underlying apply/quick-destroy engine, and docs/kargo-promotion.md for the two-cluster promotion flow).

Deprecated: install Argo CD into a pre-existing cluster (no OpenTofu)

Before OpenTofu was required, the root Makefile could install Argo CD into a cluster you had already created (it did not create the kind cluster). Those targets still exist but are deprecated — prefer make cluster-up: make argocd-helm-install-basic argocd-apply-root boots the olm-less kind-helm flavor (bringing in cert-manager, grafana-operator and the OCM cluster-manager as helm charts), while make argocd-olm-install-basic argocd-apply-root ENV=kind-olm boots the OLM-preferring kind-olm flavor.

Our preferred approach to secrets is sealed-secrets (have a look at gen-keys.sh in case you'd like to use sops instead).

If using github, you may want to disable github actions and/or add a public deployment key.

gh repo deploy-key add ...

If you want to see what a target will do before running it, dry-run it first:

make -n cluster-up

Run it without -n once you feel confident to get the ball rolling.

The deployment will deploy a SealedSecret. It will fail during decryption, because we won't be sharing our key. It is meant to be used with Argo Notifications, so it is not critical for a basic demo. Feel free to introduce your own bootstrap secret.

We want lifecycle of things (Create/Destroy) to be as fast as possible. Pulling images can slow things down significantly. Contrary docker a host based solution (such as k3s), challenges are harder with kind. Make sure to understand your the defails of your painpoints before implementing your solution.

Quick sync experiments with the in-cluster Gitea

The kind-olm and kind-helm environments deploy a bare, single-pod Gitea (apps/infra/gitea) to act as an in-cluster git remote for quick Argo CD sync experiments — edit, push, sync without leaving the cluster or waiting on GitHub.

Admin credentials are the chart defaults — gitea_admin / r8sA8CPHD9!bt6d — a local throwaway, fine for kind. Reach the UI/API from the host via

kubectl -n gitea port-forward svc/gitea-http 3000:3000

Create a repo (make it public and Argo CD needs no repo credentials) and push:

curl -su 'gitea_admin:r8sA8CPHD9!bt6d' -X POST localhost:3000/api/v1/user/repos \
  -H 'Content-Type: application/json' -d '{"name": "sync-lab", "auto_init": true}'
git clone 'http://gitea_admin:r8sA8CPHD9%21bt6d@localhost:3000/gitea_admin/sync-lab.git'

From inside the cluster — i.e. as an Application repoURL — the same repo is

http://gitea-http.gitea.svc.cluster.local:3000/gitea_admin/sync-lab.git

Repositories and the sqlite database live on a small PVC, so they survive pod restarts — but they are pruned together with the gitea app itself.

For a scripted end-to-end pass of exactly this loop, run tools/gitea-sync-demo.sh: it creates a repo, pushes manifests, points a throwaway Application at it and verifies initial sync, update, prune and self-heal before cleaning up after itself. The script is deliberately imperative — everything it touches is ephemeral demo state that is removed on exit, so the declarative rule ("edit manifests, commit, let Argo CD sync") still holds for anything durable.

(back to top)

TODO

Tracked work items live in docs/TODO.md.

(back to top)

Known Issues

References

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

  1. Fork the Project
  2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
  3. Commit your Changes (git commit -m 'Add some AmazingFeature')
  4. Push to the Branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

(back to top)

License

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)

(back to top)

About

Argo CD Conductr - GitOps Everything 🧪

Resources

Readme
Apache-2.0 license
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

Packages

Used by

Contributors

Languages