FAQ Authors group

Brutus5000

Make sure after uninstall to delete the remaining files from the program files folder. This is a common problem for whatever reason the installer does not always clean up properly.

Brutus5000

Around October 2023 hostile ex-community members launched the first wave of DDoS attacks. The FAF infrastructure and application landscape was not prepared for that. Services directly connecting to our main servers on many open ports, openly accessible APIs for the benefit of the community. People could run their own IRC bots. People built API scrapers and analytics tools. All of that had to vanish basically overnight.

In a rush the FAF team closed down open ends as good as possible. We migrated the lobby connection from raw TCP to web sockets, we put the formerly open API behind authentication. We changed the IRC server in favour of an implementation supporting HTTP based access. And then we hid almost all services behind Cloudflare proxy servers except for the server itself that was still reachable from the internet.

The weak point remained our ICE server aka Coturn servers. So we started paying for external TURN as a service provider and added more infrastructure around. But the feedback on connectivity was never good. Things never got back to where it was. I started digging into the ice adapter more than any before. Documented features. Tried to refactor it. Tried to rewrite it. We even tried to integrate Cloudflares new TURN service as it went live. The connectivity was horrible and the used payment according to Cloudflare would have ruined FAF financially within a month, so we had to disable it, not knowing what actually happened there.  At some point the DDoS more or less stopped and things settled a little more to normal and the topic of ICE fell into background noise.

Now for over two weeks now DDoS is back with cyber-terrorist alike demands (we either abide to the attackers terms or the DDoS continues forever). But the FAF team stance here is clear: We do not negotiate with terrorists.

Instead we continued hardening our servers. Our main server is no longer reachable directly from the internet, and has to pass multiple firewalls. Yet the bottleneck once again is the ICE connectivity.

So with the accumulated knowledge of the last years, we investigated and analysed the shit out of the ICE adapter with more tooling and (semi-)reproducible test methods. The results were not promising in multiple combinations:

Our Hetzner cloud servers have huge packet loss even outside DDoS (tested on fresh VMs) on ICE-related communication. We don’t know why, but it seems that Hetzner really doesn’t like this kind of traffic. Potential solutions: (a) Report packet loss to Hetzner in a structured way, (b) use ports that are for other traffic and thus more stable, (3) use a different provider
The „coturn“ software spew non-stop errors but was completely useless in logging why these errors occurred. So we tried out a different software called „eturnal“ (love the pun here), which gave us a better hint about problems
A Wireshark capture of a user trying to connect to Cloudflare showed us a single successful connection attempt followed by 80000 (!!) failed connection attempts in a 5 minute interval.

So apart from the Hetzner issues, we could boil it down to problems in the ice adapter. The ice adapter at its core is built around the „ice4j“ library. This is a piece of software that originally built for the Jitsi phone software (even though it was renamed a few times and by now is a commercial service). The only maintainers are Jitsi developers and as such the focus lies on the features of Jitsi. There is a component called Jitsu video bridge that is also open source. When we looked into it, we saw that Jitsi is not using TURN at all, and therefore not a big priority in ice4j. The code of ice4j has no documentation outside of regular Javadocs. And it looks like it was written in a C-programming style from the 90s (while Jitsi components are written in modern Kotlin). The worst part however is, that it is not possible to control or configure from the calling code.

This should not be a problem if the library does what it should. But as far as we can see by now, the TURN code of ice4j does not behave like it should. Whether it is violating the specifications is beyond my understanding. A single example that we could identify is that ice4j tries to establish a TURN session for all the ip addresses it could find: the external ip (that one makes sense), but also all internal network ips (nope, that does not make sense!). And this in particular is the reason why the ice adapter causes an endless log stream of errors in coturn: the attempt to establish a turn connection for a private network address is causing an authentication error… (ok - both software stacks here behave like idiots). That might also be the reason why there are so many Cloudflare login attempts? We don’t know.

What we do know is that we can rewrite the ice adapter all we want. As long as we choose Java or Kotlin we are bound to ice4j as it is the only notable library for ICE.

What we do know, is that ice is a low level protocol which is used by WebRTC. Which is used by every single browser and every single audio/video conferencing tool that runs in a browser. WebRTC is everywhere and its „data channel“ feature allows features that we wished for in the ice adapter in a long time (guaranteed and order submission of packets, keep alive functionality). So why go with ICE alone when we can have WebRTC with ice?

Now, there is a striving project for WebRTC called Pion with 14k stars on Github (ice4j: 500) and around 200 contributors (ice4j: 25) and lots and lots of example code. So where is the catch? The catch is: It is written in Go. And we have no Go developers at FAF - I never used it so far.

So what do you think: Should I stay (on ice4j) or should I Go (learning Go)?

Brutus5000

@Sturmgewehr said in Current connection problems (Preferred Coturns for ICE):

Are they all self-hosted except for GLOBAL? So the servers at Hetzner like Germany 1 or Finland?

Yes. All coturn servers except global are self hosted. Global can be either Xirsys or Cloudflare depending on our configuration right now.

@Sturmgewehr said in Current connection problems (Preferred Coturns for ICE):

During a test (as host -> lobby) I once tried to block all coturn server connections locally (firewall), so that players can only connect directly to me, to roughly understand how much delay / lag / connection problems coturn servers generate.
Couldn't do this though, as once I blocked all coturn servers, I couldn't create a lobby or join other lobbies (tried this several times and with very few players in them to exclude any existing coturn server connections).
I wonder if this is a bug in the software and it is never assumed here that there are simply no coturn servers?

Yes, blocking all coturns for FAF is not a smart move
The ICE protocol requires coturn. If you block all coturns the ICE adapter can only gather your local ip addresses. Since everybody uses a router in between, this will never reveal your public ip to others. And even if it could, it would still be required for hole-punching (you send data out on the port, so your firewall accepts incoming traffic on that port).

Btw: No match worked yesterday. But since the coturn servers Finland is no longer in the list (off?), it works quite well.

Coincidence.

I cannot recommend Hetzner for DDoS protection.

Yes I am quite pissed at Hetzner right now. It's not just their lack of DDoS handlng, is also there basically useless hardware firewall setup from the 90s. You can setup exactly 10 rules and each rule only takes on ip-range. That is not even sufficient to whitelist all cloudflare ip addresses.

I have personally had the best experience with OVH.

FAF used to run on OVH at the very beginning. It was very unreliable. Also we are talking about the one company that had a whole datacenter burn down so far.

Hetzner has incredible good pricing for dedicated machines. But the rest slowly goes to shit.

Brutus5000

US East has the same problems as all other coturns. We tried a very long test set yesterday.

Our self hosted coturns are less reliable than using 3rd party ones. But much worse: the ice adapter (more precise the underlying library) is much more often failing to provide the important self-reflexive and relay candidates than 3rd party browser Turn test apps.

The relay is not needed that often, but if you don't get s self reflexive candidate (aka your own public ip) no connection can be made at all.

Why did it work in the past? We don't know. We use the same ice4j as in the past. We use the same coturn and even tried alternative tools like eturnal yesterday with the same results (even tried different linux distros and a different hoster).

Brutus5000

Put it in the wiki. In the forum it will just disappear eventually

Brutus5000

@Kilatamoro Just because you can't micro with 0.5s delay doesn't mean nobody can. I remember Zock killing two tanks and a few engies with a mech marine.

Also please note that people working on the game develop game logic, not game engineer. There are completely different sets of skills and experience.

Brutus5000

Yeah looks like it got lost. Happens. Feel free to ping people if we don't respond. I get like 50 notifications in FAF a day and can only actually handle like 5...
I kindly asked on the NodeBB discord and they right away fixed a few issues, but only on the next version, which is a major upgrade to 4.0 so I will not roll it out right away.

And in general: I am super happy that you offer and try to fix it for FAF. But even better is fixing it "upstream" with the original authors. This is win-win because:

You have direct contact to the developers and get some feedback and a helping hand
Changes should not get lost or break over time just because the underlying stuff changes
We are making the world better for NodeBB user and not just for FAF
You build a reference track for future projects on Github

https://community.nodebb.org/ -> Follow the Discord link and/or particpate on the forum.
Also the theme we use is under https://github.com/NodeBB/nodebb-theme-persona and as I learned today the themes come from a library called "bootswatch" (just read up the Discord talk in #general from yesterday :D)

Brutus5000

@snoog I enabled it on forum.faforever.xyz, but where should that toogle show up? I do not see it

Brutus5000

I've asked in their Discord where the CSS files for the themes are, so maybe someone can have a look.

But in general I get the demand for dark mode, but it is beyond my capabilities to support this too. Feel free to ask for help in the official nodebb forums. The people there are nice and helpful.

Brutus5000

So we can raise a bug ticket but it's hard for me to tell what you think is wrong. So I assume the separator lines in the icons did not exist before?

What else?
You cannot detect the unread topics because you did not select the unread topics. Not sure what visual indicator you are missing, the default theme does not mark me already read topics in the recent view.

FAQ Authors

Posts

Member List