Sometimes people ask us: Can't you just stop changing things and leave FAF as it is? The answer is no, if we do that FAF will eventually die. And today I'd like to explain why.

Preface

Do you still use Windows XP? According to some FAF users the best operating system ever. Or Winamp? Or Netscape Navigator? Or a device with Android 2.3?

Probably not. And with good reason: Software gets old and breaks. Maybe the installation fails, maybe it throws weird errors, maybe it's unusable now because it was build for tiny screen resolutions, maybe it depends on an internet service that no longer exists. There are all sorts of reasons that software breaks.

But what is the cause? And what does that mean for FAF?

Simplification and division of labor: About programming languages and libraries

People are lazy and want to make their lives easier. When the first computers were produced, they could only be programmed in machine language (assembler). In the 80s and 90s some very successful games like Transport Tycoon were written in assembler. This is still possible, but hardly anyone does it anymore. Effort and complexity are high and it works only on the processor whose dialect you program.

Nowadays we write and develop software in a high level language like C++, Java or Python. Some smart people then came up with the idea that it might not make much sense to program the same thing over and over again in every application: Opening files, loading data from the internet or playing music on the speakers. The idea of the library was born. In software development, a library is a collection of functions in code that any other developer can use without knowing the content in detail.

These libraries have yet another name, which sheds more light on the crux of the matter: dependencies. As soon as I as a developer use a library, my program is dependent on this library. Because without the library I cannot build and start my application. In times of the internet this is not a problem, because nothing gets lost. But the problem is a different one, we will get to that now.

The software life cycle

Even if it sounds banal, every piece of software (including the libraries mentioned) goes through a life cycle.
At the very beginning, the software is still very unstable and has few features. Often one speaks also of alpha and beta versions. This is not relevant for us, because we do not use them in FAF.

After that a software matures. More features. More people start using them. What happens? More bugs are found! Sometimes they are small, e.g. a wrong calculation, but sometimes they are big or security related problems. Those that crash your computer or allow malicious attackers to gain full access to the computer they are running on. Both on the FAF Server and on your computer at home a nightmare. So such bugs have to be fixed. And now?

Scenario A:
A new release is built. But: A new release of a dependency alone does not solve any problems. It must also be used in the applications that build on it! This means that all "upstream" projects based on it must also build a new release. And now imagine you use library X, which uses library Y, which in turn uses library Z. This may take some time. And 3 layers of libraries are still few. Many complex projects have dependencies up to 10 levels deep or more.

Scenario B:
There is no new release.

• The company has discontinued the product, has another new product or is bankrupt.
• The only developer has been hit by a bus or is fed up with his community and now only plays Fortnite.

Finally, all commercial software will end up in scenario B at the end of its life cycle. And in most cases open source software also builds on top commercial software directly or indirectly.

Just a few examples:

• All Windows versions before Windows 10 are no longer developed. They have known security issues and you are advised to no longer use them.
• The latest Direct X versions are only available on the latest Windows
• Almost all Firefox versions older than 1 release are no longer supported (with a few exceptions)

What happens at the end of the lifecycle?
For a short period of time, probably nothing. But at some point shit hits the fan. Real world examples:

• When people upgrade their operating system to Windows XP or newer some older Install Shield Wizards doesn't work anymore. Suddenly your precious Anno 1602 fails to install.
• Your software assume the users Windows installation has a DVD codec or some ancient weird video codec to be installed, but Microsoft stopped shipping it in Windows 10 to save a few bucks.
• There is an incompatibility in a version of Microsofts Visual C++ redistributable (if you ever wondered what that is, it's a C++ library for Windows available in a few hundred partially incompatible versions)

The impact on FAF

FAF has hundreds of dependencies. Some are managed by other organisations (e.g. the Spring framework for Java handles literally thousands of dependencies), but most are managed by ourselves.

A few examples that have cost us a lot of effort:

• Operating system upgrades on the server
• Python 2 is no longer supported, Python 3 is only supported until version 3.4 (affects Python client, lobby server, replay server)
• Qt 4 was no longer supported (affected Python client), we needed to migrate to Qt5
• All Java versions prior to 11 are no longer supported by Oracle (concerns API and Java client)
• Windows updates affects all clients
• Microsofts weird integration of OneDrive into Windows causes weird errors to pop up

Many of these changes required larger changes in the software codebases and also impacted the behavior of code. As source of new bugs to arise.

If we would freeze time and nothing would change, then all this would be no problem. But the software environment changes, whether we as developers want it to or not. You as a user install a new Windows, you download updates, you buy new computers, there is no way (and no reason) for us to prevent this.

And we must and want FAF to run on modern computers. And of course we want to make bug fixes from our dependencies available to you. So we need to adapt. FAF is alive. And life is change. But unfortunately in software change also brings new errors.

Everytime we upgrade a dependency we might introduce new bugs. And since we're not a million dollar company, we have no QA team to find this bugs before shipping.

Today two thousand players encountered once again how fragile our whole server setup sometimes is and how an unlikely chain of events can cause major issues. In this talk I want to shed some light on how the chain started moving.

Start of the journey: The chat user list

We are currently investigating CPU load issues in the client which we could track down to updates of the chat user list in #aeolus. If you open the chat tab you see a partial list of users, as not the whole list fits to the screen.
We are using a ListView component from our UI framework for this. Each row or ListItem (a user or a group name) is one element in this list view. However, it would be very memory hungry, if the application would render each chat user's list item (~2000 on peak times) if you can only see a few dozen of them.
Therefore the list view component only holds a pool of list items that you can actually see and re-uses / redraws / repopulates them as the list changes.
When profiling the application I could observe hundreds of thousands of updates to the list over a period of 10-15 minutes with about 2.000 users logged in to the chat. Something was causing massive changes to the user list and we couldn't find a reason.

Mass changes out of nowhere

On a different topic we noticed an increasing number of errors when trying to login to the IRC on peak times for no obvious reason as well. Sometimes it took multiple minutes to login.

But then somebody using a regular IRC client looked into the logs and found an interesting thing: a lot of people were logging in and out in a very short period with the quit message "Max SendQ exceeded". A little bit of googling revealed, that this error message was thrown, when the IRC server sends data to the client, and the client doesn't load it fast enough.

Using the incredibly long chat log history of our moderators we were able to track down these errors to their first occurences to October 2019 when we roughly breached 1900 users online at once. Since then FAF has been growing (up to 2200 users online at peak) and so grew the problem as well.

So there is a coincidence between the SendQ exceeded error and the amount of relogin/logouts. Now we tried to connect the dots:

• Whenever you join a channel you receive the list of users in this channel.
• If this list goes beyond a certain size the error pops up.
• The error caused lots of logins/logouts.
• Each of these events caused the chat user list to update multiple times.

So the idea was: If we could solve the IRC error, we might solve / reduce the users CPU load. A potential solution was to increase the buffer size before the server sends the error message.

The IRC restart of Doom

The situation was perfect: 2000 users online. A configuration change. All users would automatically reconnect. We could live-test if the change would make the server go away or persist.

So we flipped the switch and restarted the IRC server.

We could observe the IRC numbers rising up from 0 to 1400 and then it drastically stopped. 600 users lost. What happened?

We expected the IRC reconnect to be simple and smooth. But when did anything ever went smooth? A look into the server console showed all cores at 100% load, most utilization by the API and our MySQL database. WTF happened?

The cache that cached too late

The FAF client has a few bad habits. One of them is, that it downloads information about all clans of all people online in the chat. This is a simplified approach to reduce some complexity.

Since we only have a few dozen active clans, that would only be a few dozen simple calls which would/should be in cache already. Or so we thought.

As it turns out, the caching didn't work as intended. The idea behind the caching is the following: You want to lookup a clan XYZ. First you look into the cache, and if you don't find it here, you ask our API. Asking the API does take a few dozen milliseconds.
But what happens, if you have 10 players from the same clan? You should look it up in the cache, and if you don't find it there, you would query it once, right? Well that's what we expected. But since the first API lookup wasn't finished when looking up the clan for 2nd to 10th time, each of these lookups would also ask the API.

Now imagine 2000 users querying the clans for every player with a clan tag (~200?). Boom. Our API and our database are not meant to handle such workloads. It didn't crash, but it slowed down. And it took so much CPU while doing this, that there wasn't much CPU left for the other services. It took the server roundabout 20 minutes to serve all requests and go back to normal.

Once we spotted that bug, it was an easy fix. As it turns out we just needed a sync=true to particular cases. Problem solved. Why oh why is this not the default? Well, as it turns out this option was added in a later version of our Spring framework and due to backwards compatibility it will always be an opt-in option. Ouchy.

Lessons learned

We saw it again: IRC sucks (well we all knew that before right?). But it seems like the IRC configuration change fixed the SendQ exceeded error. Yay, we actually might reduce the CPU usage for our players.
Also we now know that synchronized caching should be the default, at least in the FAF client
Unfortunately it was revealed again that the FAF api can't handle ultra-high workloads. Unfortunately limiting the amount of available CPUs in Docker does not work in out setup. Further solutions need to be evaluated (e.g. circuit breakers).

Recap and next steps

So now we know that restarting the IRC server can be a dangerous thing. So would I do it again? Hell yes. I know it sucks for the playerbase if the server is having massive lags due to running into CPU limit. But today we learned so much about what our software architecture is capable and where we have weaknesses.

We can use this to improve it. And we need to. FAF is growing more than I've ever expected. In the last 12 months our average-online-playerbase grew by ~300 users with the latest peak being 2200 users. When we moved to the new server beginning of the year I thought that our new server can handle around 3000 users, but as it turns out there is more to do.

Many of you won't remember it, but the amount of posts in this forum and the feedback we collected showed us: the onboarding process along with the registration page is a horror for new users (especially non-english native speakers).

Together with our fellow community member Terrorblade, who is a working expert on customer journeys, we worked out weaknesses in the current process and ways to fix them.

Here are few things we are doing horribly wrong:

• The website promotes the client download. But that is not the start of the journey. If you download and install the client as first step, you then figure out that you need to go back to the website to register and steam link first. In worst case you go Website landing page -> Client -> Website registration -> Client -> Website steam linking -> Client
• The website is only available in english
• The information is unstructured and not really offering much help where you need it
• Errors in the process don't give the user any usefull help (e.g. technical api errors are printed in plain text)
• Field validation is not always available (e.g. is the username already taken? You only find out after you click the registration button)
• Hidden steps (you can only Steam link after you logged in!)

Due to the Steam link process and the required client download the FAF onboarding process is much more complicated than the that of other websites. We even modelled a perfect workflow, but that's beyond the scope of the current work. Nevertheless for the brave peopel here I'd like to share it with you, without further explanation:

In the last weeks I had the opportunity in my job to learn frontend development with Angular (and yet more trainings are to come). In order to train my newly gathered skills, I gave it a try. Unfortunately what I still don't have are skills for designing shit with CSS and colors.

All screenshots are representing work in progress:

Failed registration: As long as you didn't fix all issues and selected all checkboxes you can't click the button. The user lookup is done with a 1 second delay

You can dynamically switch the language. Currently I'm only populating english and german. The biggest need however would be russian.

Once you successfully registered we now (and only then) tell you, that you need to click on the email link:

Nevertheless, users are capable of switching between the steps to see whats ahead and what to wait for:

But using the activation url it looks simpler:

So after a successfull activation we can move on:

The Steam linking page needs some more info why this is required I guess:

The steam login itself is not interesting here, so I skip it. After a successfull login we get this:

And the FAF CLient Setup page is yet missing content completely. Not sure if we want to add screenshot per language, that sounds a bit too much work...

So far I have only shown the "happy path". The most important pain point for users is when the Steam linking doesn't work. We need to be able to distinguish between "You don't own the game" and "Your profile seems to be not public", but this needs some API changes as well.

I hope this change will reduce the amount of people we loose on the way to FAF. But it's still a long way and it might need much more time invested. So far I spent around 20 hours to reach this state (yet I'm facing many beginner problems as this is my first bigger Angular project, so it's a lot of basic learning time included).

Maybe I'll do some more Twitch streams and now the 5 people who saw my last (and first) stream know what they actually saw

Please share ideas and feedback guys!

Welcome to the new FAForever forum!

This forum is the replacement for the old phpBB one. You probably have a lot of question, so we try to answer some of them already:

What has changed?

Single Sign On

We now feature a single sign login bound to your FAF account. You do not need to create a forum account, you just login with your FAF username and password. If you change your FAF account name it will also change in this forum.

This allows to always be up to date with playernames. But even more important it is the ultimate spam prevention: a spam bot can't register to the forum without registering FAF first. This prevents all automated spam attack against the underlying software nodebb.

It has a drawback though: If you can't create or access your FAF account you can't ask for help in the forum. We kindly advise everybody to move such request to the #technical-help channel on Discord.

Restructuring

We dropped a lot of sub-forums that weren't really used before. We might reconsider changing more stuff based on your feedback.

Underlying software

We are now running on a software called NodeBB. Nodebb aims to also look nice on mobile devices. Our NodeBB uses MongoDB under the hood which is a much more suitable datastore for a forum (=> better performance)

• NodeBB Documentation
• Community Support Forum
• Project repository

phpBB was running fine, why drop it?

phpBB was not fine! Let me tell you why:

• Recently it encountered a lot of downtimes which we were unable to find a root cause.
• We got spammed really hard and could only partially prevent it (and you suffered from horrible captcha issues!)
• The administration and moderation tools were a usability nightmare.
• phpBB itself was almost a 9 year old version that never received any updates. Undocumented manual changes to the codebase and database to enable features such as the "like"-system made it impossible to upgrade it safely.
• It was running on an outdated php version which is a security threat to our server.

What's going to happen to the old forum?

This is a very good question. We will not migrate it, since this would not work out with the new forum structure and the new account linking (in the old forum we have no verified connection between a forum account and a faf account).

We want to make the forum read-only for now and maybe somebody in this community (wink, wink!) wants to write a script to turn the old forum into a static page archive.

The design of this forum ... ?

... is debatable for sure. It took me 1 1/2 year from the first evaluation to the working solution as of now. Design was no priority topic at this point. But NodeBB offers a lot. If you are interested in helping out, please contact me.

You want to contribute to FAF but you're unsure? You think you lack the skills? Well then read on, todays blog is for you! Even though it's a story about developers, it applies to all contributors! (Also watch my YouTube tutorial on how to start contributing).

I want to tell you 2 independent stories about 2 very different developers:

Story 1

I once met a Java developer at a party in his late twenties who was working in a software consultancy for a few years. From a trustworthy source I knew he was quite talented and he was very unhappy about his salary. Back then my company was looking for developers, so I approached him and asked him about his skills. So he was doing mostly Java frontend (ugh, not so modern) and regular backend stuff. I told him that my company was hiring. Of course he was interested, so I handed in his application.

I met him again before the interview and talked to him. I said "Dude, I saw in your application that you have no experience in software development in the cloud with Kubernetes and Docker and all the fancy shit. I have this open source project called FAForever. I could show it to you and explain you a lot of stuff before the interview." He declined and I asked why. "Well, I spend so much time at work, I don't want to drag it into my own personal life, but of course I will say I'm eager to learn it on the job." He was not hired.

Why am I sharing this story? First of all, I'm not criticizing here. It's an absolute fair point to strictly separate your personal life from your work life. I'm sharing this story because it shows, that not every developer has the same mindset about learning in his free time. It's also obvious that it's not the skillset alone that makes a developer suitable for FAF.

Story 2

Let's take another example. A few years ago there was a dude lurking in the FAF Slack offering help. He knew FAF for quite some time as a player, but recently stumbled over the open source project. He had a vision of bringing back Galactic War, but he had only basic programming experience. But was experienced in database queries as he was working as a consultant for ERP systems. He offered some help on administrative tasks on the server, e.g. running manualy queries (a lot of work was done directly in the database back then), helping on patch days or keeping an eye on the less-important software like the website.

Over time he gained more and more responsibilities and even started more and more on development. But man, did he fuck up things. One day he wanted to backup the website before an upgrade and gzipped all files, not knowing that the gzip command deletes the files afterwards, which resulted in afew hours downtime of the website. Or when he tried to fix some server permissions and locked out the MySQL database from its own files. But he learned. And read docs. And improved stuff. Step by step. He didn't do it all on his own of course. He was mentored by one of the reigning DevOps councilors back then.

After hundreds of hours spent on writing code and scripts, analyzing bugs, dockerizing aps, migrating servers, building/replacing/configuring software he became well sufficient with the way of software development, that he applied for a software developer job. Even though he had no commercial track record, his knowledge on software development and the cloud related technologies (e.g. Docker) was good enough, that he got the job anyway. In the same year, the dude became DevOps councilor. Some of you might have figured it out already: That guy is me. Hello!

So why am I sharing "my FAF life story" with you? Because I wanna show off? Well... maybe a little bit But moooore important in my opinion there are some key learnings that I want to share with you:

• It's not the skillset, that qualifies you for FAF, it's your dedication and your will to improve. (Even though an existing skillset helps of course!)
• You don't need to have learned or studied programming to have an impact on FAF. You don't even need to start with development.
• Mentorship helps. I didn't really learn software development anywhere, besides FAF. And I learned it because when I joined, we had a very fixed focus on quality and Downlord was an unrelenting mentor, who only accepted high quality submissions.
• I didn't learn this in a few weeks by reading a book or watching YouTube tutorials. It was a demanding process and took hundreds, maybe thousands of hours. And as you can imagine, no expert has that amount of time so large parts of this way you need learn on your own. People can give you advise, but eventually you need to solve the problem you want to solve.
• Learning doesn't go without failure. I did some stupid shit back then that I'm not proud about. But I learned from all of these lessons, and after each fuckup I sat down and thought about, how to prevent this error from happening again.

I hope you didn't get the impression that you need to spend hundreds of hours to get into FAF. You don't need to, if you want to contribute in one area. And nobody expects you to.

The most important thing is, that you have fun. I have seen many people leave because they lost fun by overacting. For me it was worth it and it was fun to spend the amount of time. Since I have a kid I have to make more and more cuts in my FAF time. That's also one of the reasons why I am trying to motivate you to become the next generation of FAF developers. The one day will come when I want to pass the baton.

Small sidenote:
Some people claim that FAF is just a playground for developers, so that they have some cool technology to play with for their resume, while breaking important features for the playerbase. While I can't deny that we use our result for resumes (well I wouldn't have gotten job without it), I can assure you, that we do no play around in terms of "resume driven development".

Just for your post I consider bringing back the feature to downvote posts.
ICE is not a failure. If you had a good connection before, than you weren't part of the 1/3 of the FAF community that didn't.
And reconnecting to games after internet loss is a savior.

Why did we choose NodeBB instead of [any another product]?

Upgrading to the latest phpBB

The obvious choice would have been migrating to a newer version of phpBB. But as I mentioned before there are some issues that would have made this transition a walk on a tight rope.

Our phpBB installation is 9 years old, the underlying phpBB 3.0 release even older (according to Wikipedia work on phpBB 3.0.x began in late 2002, first release candidate was in May 2007). The web software development world was a completely different one back then as it is today:

phpBB 3 had no concept of installable mods or plugins. If you wanted to extend your installation you would follow the extension developers guide and manually edit php code files and manually run SQL statements against the database, thus adding fields, tables and whatever the developers came up with. Also version control on the running server was not a thing back then.

Unfortunately our installation made use of this. For example the whole like-system is deeply intrusive into every part of the application. The "solved" feature in the help forum is another example. Eventually no active server admin knows what has been changed to the system and there is no way to look it up.

All of these additional features would instantly break on a new version, as there is no corresponding code handling this in a newer version. Furthermore every change to the database could make a potential upgrade fail. And there is nobody out there to help you upgrade a manually fucked up 3.0 installation throughout version 3.1 and then 3.2. And if you then run into some weird unexplainable issues due to some weird migration leftovers nobody will be able to help you.

In addition to that past experience shows that former administrators seems to have manually fiddled with the database as some features in the database are broken or don't work as intended.

An upgrade would have come with high risk, unknown outcome and lots of efforts. This is the reason why we decided to look for a solution with a clean installation. Furthermore phpBB 3.2 didn't even match our criteria.

Our requirements (and how NodeBB matches them)

Before selecting any kind of software (or product) you should know what you want. Knowing the issues of the old forums and from other issues we had in FAF over the last 5 years, I made a list of requirements before I went to search for the market.

Eventually NodeBB was the only product I found matching all criteria, so I'll leave out the comparison with other software out there.

1. The software is actively maintained. This sounds trivial, but a lot of projects are in a semi-zombie state and we're selecting a board for the next 10 years, right?
   NodeBB is actively maintained, e.g. they just release the v1.14 release series with two bugfix releases while I was setting it up.
2. The software has to be open source.
   We are an open source project. We live open source. Also we don't have hundreds of dollars to throw out of the window.
   NodeBB is open source and available on Github.
3. The software is battle proven by large installations. We don't want to make a bet on some newcomer thingy with fancy features, we want stable enterprise-like software.
   NodeBB is widely adopted. See their website for examples.
4. It needs to have a large community supporting it. Bonus points for a company backing it with support.
   A large community supporting it + open source means even if the company behind it goes out of business, the community will most probably take over.
   NodeBB is also a company, offering cloud installations for money and paid work on support or customization. [So if everything burns we could throw a bit of money at them and they could help us.]
5. It needs to have official docker support by the core team. This is mostly a server administration thingy but FAF has a few examples where this is not the case and it's a pain in the ass to keep it working.
   NodeBB has official images on dockerhub.
6. It needs to offer a way to integrate our own single sign on.
   Single sign on is a nice feature to sync FAF accounts to the forum. We struggle with lots of support request of people who don't give their account name. Another benefit is, that it keeps the spam bots out, as the FAF registration process is completely unique and spam bots only attack default registration processes.
   NodeBB supports additional login system via plugin. For OAuth2 (which FAF is using) they provided an excellent skeleton to work on with. So the plugin is decoupled from the rest of the code and will not interfere with regular updates (unless there is a rare breaking change in exactly this part of the application - but then they publish it in the release notes).
7. It needs to offer a web api for reading and writing stuff. Reading is beneficial for potential integration with our clients and the website to maybe drop Wordpress at some point. Writing is crucial for the one feature I hate FAF the most: the ability to change your username. Changing your account name is very very very uncommon (usually you just have a display name you can change) and it's not supported by regular authentication providers.
   NodeBB offers seperate read and write APIs via plugins. Utilising this I was able to create username synchronisation with our API in an custom plugin (which also handles the single sign on).

Eventually the points 6 and 7 were the dealbreakers on most forum alternatives out there.

On another side note: As a developer I hate JavaScript for being unreadable, unstructured and totally chaotic. However, when I dived into the codebase for writing the plugins I have seen the cleanest JavaScript code I have ever seen in my life. Therefore I can tell you that the NodeBB guys are really professional and know what they are doing. This gives me additional trust in this system. (Something I can't say about phpBB by the way.)

Guys, stay cool.

ThomasHiatt brings some valid points. To dismiss them with the argument "all work is voluntary" does not do justice to the matter.

Nevertheless, I would like to start by emphasizing that we have achieved positive results in recent years. The ICE adapter works and has, in my opinion, brought about considerable improvements. The new replay server no longer crashes once a week and generates thousands of broken replays. We have moved the server several times when the old one crashed under load almost every day. All these changes did not go smoothly. Nevertheless, they have improved FAF for everyone, including ThomasHiatt.

None of these improvements would have been possible without the tireless efforts of our volunteers. And none of these projects would ever have gone live if mistakes were not tolerated.

We all try to meet the wishes of the community. These can be bugfixes, small feature requests, but also big tickets like Team Matchmaker or Galactic War.

The basic idea we follow is that when we touch code for a bug or feature, the new code should align with our big tickets. The biggest change at the moment is the Team Matchmaker and it has a big impact on our rating handling.

Now, unfortunately, FAF has another complexity in the interaction of Server<->Client<->Game. People working on the lobby server usually have little knowledge of the game code and vice versa. Unfortunately there are often misunderstandings in communication or bugs in the interaction, especially if server and game are subject to different release cycles. Another problem is the complexity of testing such changes. It cannot be automated and even manual testing is only possible to a very limited extent.

I hope I could make it a bit clear why bugs happen and will continue to happen. The alternative is to stop making any changes until the software slowly dies (yes, software ages, I plan to write a blog article on this topic to explain it in more detail).

So the crucial question is not how to prevent errors, but how best to handle them. And here in this specific case we as developers and especially I as the responsible councilor have failed on several levels in my opinion:

• We/I did not understand from the community feedback that there is a big problem. For example, I knew that there was a problem, but I had assumed that it only affected a small part of the games.
• We didn't investigate how many games are affected and we didn't keep any metrics (maybe that's a technical improvement point to take away)
• And as a result, we have prioritized the problem solution incorrectly

From my point of view it's okay to make room in the forum or elsewhere to escalate the issue, as long as you do so in a respectful and non-personal way. Alternatively, a clarifying conversation with me would have been more effective. That's what I am here for... well, also... somehow.

To cut a long story short: We have understood your distress and a solution proposal is now available that still needs to be evaluated (testing is difficult again, so it will probably be a shot in the dark...). Hopefully it can be rolled out with the planned server update in 3 days.

Excuse me? What the fuck am I reading here?

Yes, the vault (map & mod) could need some love, but there are quite a few reasons why this is diffcult. Nevertheless, your attitude is astonishing:

• You demand something ASAP while
• you didn't even bother figuring out the right forum to post (which would have been "Suggestions") and of course
• you don't offer any help of your own, but expect other people to do it for you?

There is a matching quote that comes to my mind:

And so, my fellow Commanders: ask not what your community can do for you — ask what you can do for your community.
-- Commander John F. Kennedy

Posts like yours make me question my commitment to FAF.

@foodlfg said in Unexpected chain of events: How a users cpu load issue paralysed the server:

interesting. nice find!

but it feels like especially from the java client side that the technologies that it is using were not properly selected. it does unnecessary things that the programmers have no control over. neither simple, nor fast. working with pre-made packages glued together.

I can't defend each and every choice made in the Java client, but I can tell you if we had a choice, we use battle-proven standard libraries used by thousands of developers. Your statement goes more into a general critic against Spring framework with all its "magic" involved. It can be quite difficult, but that doesn't mean you don't have the control over it. The general problem is complexity, as Spring components try to cover everybodys use case and can be configured in dozens of ways.

However, some libraries turned out to be a real stick in the mud. But you can't always know that beforehand. JavaFX e.g. is a pretty solid library in general. But using the JFoenix library for "nicer" ui controls was a bad choice as the developers stopped updating it to newer versions. Or the current IRC library used served us well for 5 years, but seems to make problems (not sure yet) if we hit 2k+ users in a single channel (who would have known FAF would ever reach that numbers?).

If you would write a new client from scratch in whatever languages you would have the same problems. Again. Which libraries are stable and well maintained for the next 5 years? Should I use the trendy libraries or the stable one? What if the trendy one supersedes the stable one?

as for the server side, i was like wtf when i saw that the API can receive database query like parameters which are a big feature and convenient to use, i guess. but do you guys really need this?

Do we really need this? Yes we do. The library that we use allows us to do basically permissions based proxy access to the FAF db. This includes:

• We can expose everything that we want (not all tables of the FAF db are public but all that are used).
• We can put security filters on what we need to protect. (e.g. some fields or some tables are only visible to privileged people)
• We can also create and update stuff (also permission based)
• And some things are just free: sorting, filtering, including other files

We had a different API before where we wrote individual queries for each use case and it was a huge amount of work even for the starting use cases back then. Do you want to rewrite the sorting and pagination rules each time on your own? Also the include concept would be lost then, as it is very generic and you would need to make dozens of api calls instead.
By now we have the whole moderation client built around this API. The java client uses it extensively. The website uses it (for leaderboards and the league things) and even 3rd party tools make use of it. So I think the whole API concept is a huge success.

the most simple solution would be to write a simple database auery in the DATABASE and an API that calls that one query. the further the query is from the database, the more control you lose and you have less chance to operate the database efficiently.

But it's not one query. It's hundreds of queries. I don't want to write and test these manually if there is a library that can build almost optimal SQL queries if you don't misuse it.

you cannot really prepare for the queries this way for example. there are queries that needs up to date data, there are others that need only a few hours fresh data. you can store the last ones in memory in the database if you know them so you just serve the already queried data.
just ideas, i know it is not a paid project but it is something to learn from...

That is the downside of it yes, indeed. Flexibility comes at a price. We need to use it with care and we need to think about cancelling too expensive queries somehow in the long run

Eventually it always comes down to this: We are a very small core team and there is rarely more than 1 core developer per service. Thus we need to make the right abstractions to keep the code size small enough to handle. If I write hundreds of queries manually to squeeze out maximum performance, I can no longer make changes to the database without fixing dozens of them (apart from the fact that I couldn't do anything else for FAF as I'm busy just with that). We made a trade-off here and choose flexibility over performance.

The live replay is not costing you any additional resources. You stream the replay to the replay server (for storing the replay anyway) and it costs less than 5kb/s.
All viewers stream the live replay from the live replay server, not from your connection.

It's different with observers. The need to run the simulation as well and receive the data from all players.

Ransomfaf. Your FAF is now encrypted unless you pay.

And on Steam, not anywhere else

See here: https://forum.faforever.com/topic/586/unable-to-upload-any-map-since-today-worked-before

Looks like my setup broke. Everytime I try to join a game I'm stuck with the FA mouse cursor and a black screen. Pinning Proton to 5.0-10 did not help. And I can't tab out of the black fullscreen window... Any advice?

Yes, the new action takes place here: https://github.com/FAForever/faf-qai

@Sheikah and I investigated this. The last client release contained a newer library which caused this issue. Updating our dependencies will fix this.

We'll probably release an intermediate release candidate that you can use. Since only map upload is broken, we're not going to do a full release at this causes thousands of players to download the whole 130mb installer.

https://forum.faforever.com/topic/278/i-lost-access-to-my-account-what-should-i-do
Look at case 2.

Python client is no longer officially supported.
Windows 7 is also not supported anymore.

Maybe you find somebody willing to help here: https://forum.faforever.com/topic/317/legacy-client-not-working/20

The hardware is not that relevant as it is exchangeable. All applications are run with a large docker-compose. The related repository is faf-stack