tldr: Unfortunately, this is a dire message. The continuation of FAForever should not be taken for granted. We need help. We need new contributors with the intrinsic motivation to support FAForever’s future. All type of contributions are welcome, but in particular we are looking for contributors with a solid software development background to help address the latest attack vectors used by malicious actors.

The past decade the internet has changed significantly. In my personal view the internet used to be more innocent. As a small open source service provider such as FAForever it was unthinkable that we would be the target of malicious actors. And because of that the infrastructure of FAForever was never designed to be resilient against malicious actors. Of course, there are basic authentication and authorization checks. But in general, our implementation assumed the user consuming the service acted in good faith.

Starting roughly two years ago, FAForever has been under constant attack by one or more malicious actors. This has been, and still is, a difficult struggle. The malicious actors started off with relative innocent DDoS attacks that abused the initial naïve setup of our service. Over the past two years it has grown into a sophisticated, consistent multifaceted cat-and-mouse game between the malicious actors and the server administrators of FAForever. I am talking about an organized effort to take down FAForever. Some of these malicious actors put in the effort to make a recruitment video on YouTube and even managing a Discord server with the sole purpose of learning how to attack the infrastructure of FAForever. As if FAForever is some live target practice to learn the ropes of the trade.

Over the past two years the type of attacks has adapted. Additional defenses against the actions of these malicious actors were put into place. We introduced the use of OAuth to access our previously public API to prevent malicious actors of constantly initiating long, straining tasks on our systems. Cloudflare was implemented to prevent DDoS attacks from flooding legitimate (HTTP) traffic to our systems. Network relays are provided by a third party, for which we pay, as our self-hosted solution could not be adequately defended. A lot more has happened as we tried to constantly adapt to catch the latest attack vector of the malicious actors.

What initially started with abusing our services directly has eventually evolved into targeting individual players. As our defenses for the core services of FAForever became more resilient, the malicious actors appear to be shifting their focus to target the one weakness that is inherent to peer-to-peer networking. That is the underlying network technology for Supreme Commander. Through that the malicious actors can gather the IPs and target individual players with direct DDoS attacks. This started with the targeting of specific streamers as they were playing roughly a year ago. Apparently, it recently expanded to a more systematic approach to target individual players just to occasionally DDoS them at the right moment to break the game that they are playing.

This is just another step in the cat-and-mouse game that has been happening the past two years. Fortunately, there is a technical solution to this. We can force all players to connect to each other via a relay. This would hide their personal IP. We attempted this last week with our current infrastructure, but it was not successful. There is a lot of work being done on a new solution. But the technical complexity of it is staggering. We've come a long way and it appears that games with up to two players are stable. However, games with more then two players are unstable. Reasoning about and investigating of the cause requires a lot of technical knowledge. And we need more of that expertise in order to help stabilize the new solution in order to protect our community.

Our new implementation uses the Interactive Connectivity Establishment (ICE) protocol as the underlying technology. We now use a modern, well maintained library that implements the protocol. The details that prevent us from deploying it are in the interaction between the game, the ICE adapter, the ICE breaker and the relays.

What FAForever needs for the future

We need new contributors who combine technical knowledge with the intrinsic motivation to take initiative and help stabilize FAForever. This means taking the lead in finding and identifying issues, investigating them, and then either reporting observations or submitting pull requests with fixes.

If you want to contribute then please reach out to @Giebmasse to get access to Zulip. You can reach out to him via the forums or via our official Discord server. Once you have access to Zulip you can join dedicated channels to report your observations or ask for help with your investigations and/or pull requests.

What FAForever needs in the short term

In the short term we need people that are already familiar with the technical domain. You should be familiar with investigating (network related) problems. You have the time, the tooling and the understanding to take the initiative and start running tests to make observations. Compare what you see in your tooling, what you see in logs with what you see in the source code to help identify the cause of the instability.

This work is focused primarily on the following repositories:

• WebRTC Network adapter
• https://github.com/FAForever/faf-icebreaker

Related channels on Zulip:

• general
• ice-development

What FAForever needs in the medium-to-long term

We need new application developers and server administrators to help stabilize and improve our services. As mentioned before, the past two years have been a constant cat-and-mouse game. Whenever a vulnerability is exploited, the service becomes unavailable until we implement a fix. FAForever has no full-time engineers; contributor time is limited. This creates a high-pressure environment where vulnerabilities must be patched immediately.

Unfortunately, a fix made in such conditions can sometimes introduce new stability issues in parts of the software that are not critical but relied (in good faith) on the pre-patched behavior. And because the next exploit of a vulnerability usually appears soon after, there is little to no time left for proper refactoring. This is where you can make the difference.

This may involve but is not limited to one of the following repositories:

• The official client
• Our API
• Moderator client
• Shared library of functionality
• Database
• User service
• Lobby service (interacts with the game)
• League service

Infrastructure related repositories:

• Kubernetes stack
• Architectural documentation

For more information, see the following resources on Discord in the Looking for Volunteers:

• Application Developer
• System administrator

Also see the statutes of the DevOps team to learn more about all the available roles and the organization of the team.

Related channels on Zulip:

• general
• java-server
• client
• java-api

What else you can do to help FAForever

If you're not a developer, you can still contribute to FAForever in several important ways.

Do not vent your frustration towards contributors

I understand that the situation these past years can have been frustrating. A perceived stream of never-ending issues that break your flow when playing Supreme Commander via FAForever. Whether it is unable to login, unable to download the latest version of the game or just unable to keep a connection going. It has been frustrating for everyone involved. And I understand that you may feel the need to vent your anger. This occasionally happens by users on our platforms.

But please, do not direct this frustration towards our service or our contributors. Whether this is a server administrator, a moderator or just someone who's trying to help people with technical issues. They are not responsible for the behavior of the malicious actors that are attacking our infrastructure day in day out. These attacks made FAForever feel unstable all the time. Venting your anger towards contributors does not fix the actions of the malicious actors. All it will do is drain the intrinsic motivation of contributors. The same contributors that are the only thing we have at this moment that keeps FAForever going.

Do not share credentials with other users and/or services

Recently, we've seen a surge in accounts that are used for malicious activity such as crashing and/or invalidating in-game lobbies. This includes accounts with a long history of normal behavior. We suspect these accounts may have been hijacked by malicious actors. Help the community to protect your account by using a unique password, especially between FAForever and a fork of the project.

For more information, see also this announcement:

• https://forum.faforever.com/topic/9485/suspicion-of-hijacked-accounts

Help contribute in any of the other teams

A lot of teams are looking for contributors and there are a lot of area's where initiative from someone with the right expertise can create a large, meaningful impact for the community. This is not limited to just developers. If you are for example proficient as a graphics designer or a video editor then there's a lot you can do in FAForever to both help shape our image to the outside world. See also the following links for more information:

• https://forum.faforever.com/topic/8514/join-the-faf-team-exciting-opportunities-for-contributors
• Looking for Volunteers thread in Discord

You can make a post and/or reach out to people with a team lead role on Discord to see if you can contribute towards the goals and purpose of a team that sparks your interest.

Help spread awareness

You can also support us simply by spreading awareness. Share this announcement with people you know who might be able to help. If you’re a streamer, share it with your audience. The more reach we have, the better our chances of finding new contributors who are motivated to help preserve FAForever for the future.

Only together can we ensure the continuity of FAForever. If you care as much about its future as we do, please contribute or help us reach someone who can. Now is the time to make a difference!

Jip
President of the FAForever association