Next server update: 12.02.2021 - Preparation log


The next server update has been scheduled for the 13.02.2021 betwen 8.30 - 10.30 GMT (=UTC).

What is planned so far:

  • Migrate ratings and rating changes to new leaderboard tables
  • Upgrading NixOS to release channel 20.09 (including update of all userspace packages and a reboot)
  • Reconfigure Docker to no longer use native ZFS volumes but transparent ext4/xfs volumes (managed by ZFS). Reason: there are performance and management issues with the native ZFS storage support in Docker
  • Set up ZFS autosnapshot and tune it
  • Update all Docker base images to the latest bugfix release for FAF services (e.g. MySQL, MongoDB, RabbitMQ, Prometheus, ...)
  • Update lobby server to next version containing several bugfixes

What could come if we have enough time to make it happen:

  • Move from MySQL to MariaDB
  • Implement ZFS tuning as researched by Mozilla/Let's Encrypt (* requires MariaDB)

What will probably not come:

  • Update the replay server in order to use Zstandard compression for new replays

Shame on me, just fixed the date. It will be on Friday.


Small recap of todays update:

  • The whole update was done in about 1h 15min, so a lot faster than I expected (mostly thanks to @p4block amazing Linux skills)
  • We didn't receive larger complaints so far, so we didn't break anything. major. There seems to be a smaller issue with the lobby server sending queue pop times we'll need to checkup.
  • Unfortunately the MariaDB update is postponed for now due to additional work required

Side note:
Due to the ZFS reconfiguration we were now able to setup auto-snapshots of the whole system. So there are now dedicated internal backups not just of the db but of everything. We still need to improve external backups (right now only the MySQL database is subject to backup)


I definitely Did not Break the Contribution Page 😉