The k8s journey: One leap forward

[Brutus5000]

Rolling Out - Compose to Kubernetes

Since I began working on the FAF infrastructure around 2017, I have successfully completed three full server migrations.
Along with this, we (the FAF server admins) brought many changes like NixOS as declarative operating system or ZFS as
file system. Yet, nothing compares to the change we recently introduced.

After over a year of testing and experimenting, we finally did it: transitioning the core services of FAF
from a Docker Compose stack to Kubernetes (K8s) on Thursday, May 16, 2024. It's worth noting that this was a pure
background change without any visible impact for our users.

Although this significant change brought a few hiccups, the overall transition proved smoother than expected, with
everything seeming to function as before.

Why We Transitioned to K8s: Understanding the Changes and Reasons

Previously we were running all services in a set of three Docker Compose files. Each service was running as a docker
container. Essentially, in Kubernetes everything also runs as Docker containers.

For that we had to run the docker daemon. With Kubernetes, we run k3s as daemon that takes care of everything.

We are still running a single node. We did not set up a large Kubernetes cluster. Also, we still mount volumes directly
into the ZFS filesystem of our host. So why even bother?

Well, Kubernetes was never about a single service in a container, it's about managing a large set of them.

And the FAF stack has become quite big with over 20 services to manage.

With Docker Compose we had to manage the containers itself via the shell on the system. Also configuration and secrets
in files were edited on the file system.

In Kubernetes we can manage the cluster using a web interface without needing to login to the server. The configuration
and services themselves are managed in a Git repository that gets automatically synchronized with the cluster. This
approach is called GitOps. Secrets are no longer managed in files but synchronized via a dedicated service called
Infisical.

I will explain more features in the next following topics.

The non goals and cons of Kubernetes

Kubernetes aims for many goals that we do not pursue. So even though they are often mentioned, the following bullet
points are aspects we did not aim for:

• We did not aim to introduce a multi-node cluster. That would be needed to reach many of the goals below. This
  does not fit the way FAF currently handles it storage. However, a single node is the first step to take if we have
  needs for that in the future.
• We did not aim to become more scalable. FAF does not need to scale to infinity. With the last server move we kind of
  overprovisioned hoping to solve the DDoS issues, which unfortunately did not help.
• We did not aim to enable "scale to zero". This only makes sense in fully managed "serverless" environments, to save
  costs. This does not work on already cost-effective bare metal setups like FAF.
• We did not aim for high availability. The overall dependability hinges on the availability of each required component,
  and the current setup does not facilitate this. FAF has many architectural single points of failure that cannot
  be solved with infrastructure.
• We did not aim to move to a managed cluster like in AWS, GCP or Azure. The current setup is the most cost-efficient.

So now that we declined all the standard reasons to introduce Kubernetes, let us look at which burden we actually
brought to ourselves:

• Kubernetes is much more complex than a plain Docker Compose setup.
  • Kubernetes to Docker is like a Linux distribution to the Linux kernel. It is a whole suite of services running in
    the backend that could have bugs. So, the more complex, the higher the chance there is that we meet bugs.
  • It introduces many new concepts and building blocks that work together from Pod (a single deployable unit
    consisting of one or more Docker containers) to Deployment (a meta-object that handles versioning, rollout and
    scaling of a pod definition), Config Maps up to 3rd party extensions in Custom Resource Definitions. Even though only use a subset of Kubernetes, overall we use a subset of around 20 resource types.
• Kubernetes has a resource overhead. All the additional services mentioned above still need to run.
• For configuring it correctly we rely on added new tooling such as Helm, Infisical and ArgoCD.
• It is (currently) not possible to just spin up a subset of services for local testing and development.

With all that burden, does it look like an unwise decision to go to K8s? Has FAF just become a playground of resume
driven developers?
Of course, not. We had specific problems to solve and goals to achieve, all for a healthier long-term outcome.

Unveiling the Goals and Advantages of Kubernetes

We are running our Docker Compose setup for almost a decade. We optimized it extensively but encountered numerous
limitations in terms of:

• accessing our server,
• operating our services,
• disparity between the described state in the Docker Compose git repository and the real configurations on
  the server.

Thus, our immediate, primary goals were as defined:

• Move from a customized, script-oriented usage of Docker Compose to a well documented, established and widely known
  industry standard workflow
• Manage the cluster state via GitOps (achieved via ArgoCD)
  • All changes to the cluster are committed to a Git repository.
  • These changes are then immediately deployed to the server.
  • Make ssh-login to the server for regular duties obsolete
• Empower application maintainers (without giving ssh server access!)
  • offer a secure git-based workflow to update versions edit configuration themselves
  • make the "real state" of configuration visible
• Automatically scan for application updates (achieved via Renovate on the Git repo)
• Avoid single services to overload the server by applying resource limits
• Improve stability of services
  • use health checks to restart broken services automatically
  • use multiple replicas (where possible)
  • enable configuration changes without downtime

All of these goals can be achieved by Kubernetes today.
This hopefully allows us to distribute the work among more people without handing over the golden key to all private (
GDPR-protected) data.

It also hopefully reduced time spent on the server for future updates.

Are we finished yet?

No. We still need to migrate shared base services, such as MariaDB, RabbitMQ or PostgreSQL.

Here we still need to investigate the best way.
Generally, we have the option to use operators which allow declarative configuration of services, e.g. the allowed
users with passwords.

This would be an improvement over the current script-based approach and would simplify the setup on
local machines. Also, the topic of automated testing on the test server should be investigated. The minimal docker based solution no
longer works.

As we keep working to make FAF's infrastructure better, we are sticking to our promise of doing it really well.

Thank you all for being a part of this journey.

---

Many thanks to @Magge and others for refining this article.

[Defiant]

Thank you for bringing a lot of pleasure to my life by running FAF.

Maybe you'll find this funny: https://www.youtube.com/watch?v=9wvEwPLcLcA

Def