If you've played a game over the past 2 years, you know it. That little notification that pops up when you join the game telling you that some thing called the “FAF Ice Adapter” has been started (which is going to vanish in the next release). But what the heck even is that and how does it help my game run without disconnects?

Warning: This post is quite long and might go into some details on how the internet works as well as the inner workings of FA's simulation and networking. It is also designed as an in-detail explanation and reference I was suggested to write after having to explain ICE to quite a lot of people including developers, so I won't skip the ICE related details.
I tried to explain them as well as possible, but feel free to skip especially the first two sections if you wanna get to the point quicker. I provide a TL;DR whenever possible.

FA's simulation model

First of all, let us dive a bit into the way the game operates.

When you launch a game of FA, you do not connect to a server. There is no server. Back when the game was built, internet connections weren't capable of handling thousands of units per player, so the game had to be developed in a different way. Instead of using a central server, maintaining and managing the entire game state, every game client knows about the complete state of the game. This avoids sending information about what's going on in the game and synchronizing the positions and states of all units and projectiles over the network. Instead, the only information clients need to exchange are the player inputs. The entire simulation is then built deterministically, so that given the state of the game, as well as all player inputs, every player's computer arrives at the some result after calculating one tick in the game.

This removes the need for a (costly) server but also comes with some drawbacks, mainly off-loading CPU load to the clients, therefore slowing the game down to the simulation speed of the slowest client, as well as having to ensure all player inputs are present when the next tick is to be processed. If a player isn't finished calculating the previous tick/step or their inputs where dropped somewhere, the entire game will lock up and you'll get either stuttering or even a connection issue / “player quiet since” dialog. (a so called “lockstep model”)

TL;DR: FA has no server, each player runs the entire game equally

FA's network model

As there is no server, players somehow need to exchange messages for coordinating the game and sending their input to each other player each tick. Therefore the game uses a peer-to-peer architecture. This means, that each client in game is connected to each other client directly.

To allow for network congestion and high distances, there needs to be a gap between the point in time when the command from one player is sent and the point it is processed at. When you issue a movement order, it is in fact only executed 500 ms later (could also be 250, not completely sure), even though the UI layer of the game conveniently hides that from you by starting to rotate the unit immediately. This is in fact only visual and not present in the actual game simulation.

TL;DR: The take away here is that each command is delayed by 500 ms no matter what, so as long as your round-trip latency (RTT) to each other peer stays below that threshold, the game will run just fine. If it rises above 500, even just 550 for one other player, the entire game will start stuttering. This needs to be the case for each connection between all players. If one connection breaks, the entire game will lock up.

Running multiplayer games

A 2v2 only contains twice as many players compared to a 1v1. A 4v4 doesn't sound to bad, given that it's only 8 players. But when you look at the connections that have to be maintained in the background, the story is quite different. As there has to be a connection between each pair of players, the number of connections actually scales quadratically with the number of players:

Players	Connections
2	1
4	6
8	28
12	66
16	120

Remember, even one of those connections breaking means the entire game locks up.

The ghosts of older times - IPv4 and NAT

Now let's get into the actual networking. The Internet works using addresses. Each device has an address, an Internet Protocol (IP) address. If you know the address, you can talk to it.

When packet switching was first designed, it wasn't conceivable that one day there would be more IP capable devices than humans. 32 bits (0s and 1s) per address seemed enough, you know them as the 0.0.0.0 - 255.255.255.255 notation. That's 4,294,967,296 addresses, of which 86 % are available for public usage.

Right now, each US household owns on average of 10 internet connected devices. There are more end user devices connected to the Internet than humans on this spaceship, not even taking into account business devices, public infrastructure, the servers of service providers on the internet and the infrastructure of the internet itself. All of which, need IP addresses.

Put simply, we're completely out of IPv4 addresses. This problem was discovered early, so a replacement was designed called IPv6. (The protocol identifier for IPv4 was 4, 5 was already in use for ST, so there never was an IPv5). IPv6 uses 128 bit for addresses. That's quite some more addresses. 340,282,366,920,938,463,463,374,607,431,768,211,456 to be exact. We could give each grain of sand on this planet an IPv6 address and still be left with 99.9999 % of addresses.

The big issue: all the existing infrastructure uses IPv4, the transition to IPv6 will take some time. (here's a map)

TL;DR: We don't have enough IPv4 address for everybody.

NAT

important for ICE

In the mean time, we'll have to get tricky to save IP addresses. The major building block in doing that is Network Address Translation (NAT). Instead of giving every device an address, we separate the network into lots of local sub networks and give each of them one address. So your router gets assigned a public IPv4 address and all devices in your household use the same address to communicate. Each device get's a private address that is only valid within your local home network (192.168.X.X, 10.X.X.X, …).

Each time you want to open a website, your device sends a request to your local gateway (router) which translates your local network address and replaces it with its global address. The only issue with this: when it receives an answer from the website addressed to your public address, it needs to figure out which local device to send that packet to. It does so by assigning you a random port X on sending the packet using the public address and remembering that. When the packet comes back on port X, it knows exactly which private/local address to translate it to.

from: https://commons.wikimedia.org/wiki/File:NAT_Concept-en.svg

Sidenote: This prevents you from hosting a webserver, as no one can reach your device without you connecting to them first. You need to tell your router what to do with the packets it receives, which is exactly what port forwarding does. ("If you get a package on your public address addressed to port 80, that should go to PC-X in the local network")

NAT Traversal

This is fine, if your users “behind NAT” only access remote services, but do not have incoming connections. But if you dream of establishing a connection between two people behind a NAT, which is what a peer to peer application like FAF does (66 times for a 4v4), you simply cannot. The routers on both ends just won't know what to do with the packets they get.

To get around this, NAT traversal methods have been developed. One of them is hole punching. Hole punching works by both clients just sending each other packets on a fixed port, therefore convincing their NAT they had an outgoing request, telling it what to do with incoming packets. It requires a third party server, that is publicly reachable telling both clients to start transmitting to each other.

All of that is assuming though that your router actually has a public IPv4 address. Oftentimes this just isn't the case though, e.g. in Dual Stack Lite. There, you actually only have an IPv6 connection and IPv4 traffic get's tunneled via that. Your ISP (internet service provider) will then often use something called a Carrier-Grade NAT (CGNAT) / symmetric NAT. There, multiple customers of the same ISP get put behind a NAT with one or multiple! public IP addresses.

So your requests might use different public IP addresses shared with hundreds of other customers. Hole punching is impossible in that circumstance.

TL;DR: NAT traversal and hole punching can get you through NAT, but only in some situations.

FA Networking

To recap: we need to establish a connection between all players, all of which are located behind a NAT and cannot be reached under a public IP address. If a connection between any two players breaks, we cannot run the game.

The game by itself already does some kind of NAT traversal, it can already use STUN. It's not very reliable, most of the time only works for simple host to host connections and rolls a dice on being able to hole punch.

This meant that most players had to resort to port forwarding, telling their router by modifying a setting which incoming packets to send to their machine. (which is impossible with CGNAT)

Legacy FAF Networking

To mitigate this, FAF implemented a proxy server that allowed players to establish a connection even if they were unable to get a direct connection. On client startup, the server would test if hole punching succeeds, and if not establish a proxy connection for the game to use.

This way, players with a forwarded port or hole punching succeeding got a direct connection, anybody else was still able to connect to other players using the proxy. Somewhat, sometimes, if they got lucky…

A tale of REFAFing

New players might no longer even know this word. The REFAF was an essential part of FAF culture for many years. It was used to tell someone trying to connect to a game that they should rejoin. But not just the game, restart the entire FAF client.

It was pretty common for players not to get a connection to others in lobby, almost all games involved certain players not being able to connect to each other in lobby and having to refaf multiple times, before the game had even started.

If you managed to start the game, in a substantial amount of matches, at least two players would then loose connection during the game (locking the entire game), still able to chat with everyone else, trying to figure which one of the two players is going to leave so the game may continue. (an especially bad debate, if the players happened to be on opposing teams)

Side note: But what the heck does REFAFing actually do? The connectivity type was defined by the test run on client launch. Subsequently, all connections in all games and to all players would use the same connection type. If the server could reach you directly, but you didn't manage to get a direct connection to another player, you were out of luck as you weren't using the proxy. REFAFing reruns the connectivity test, hoping for a different result. (there's likely way more issues at play here, but that's one of the main reasons)

So, after “fixing connections” turned out as the top contender in a survey about what FAF users wanted, it was decided that ICE, being a new, standardized protocol to establish connections was the future of FAF connectivity. And so began the 2.5 year development cycle that successively yielded 5 different, all-most nearly completed, ice adapters.

The ICE Adapter

The FAF ice adapter aims at managing all connections for the game. It's a program running on your local system started/stopped by the FAF client that offers an interface to the game. The game is then told by the FAF server/client/adapter to connect to other players, supplying it a wrong IP address. If you join a FAF game nowadays, your game will think all of your peers it's connected to are located on your local machine. It sends all its traffic to the ice adapter, which then figures out how to forward it to the other players' ice adapters, which then forward it to their games.

The ICE adapter uses the Interactive Connectivity Establishment (ICE, standardized as per RFC5254) to, well, interactively establish connections.

How does ICE compare to the old solution

ICE actually doesn't change that much compared to the old solution. It still uses direct connections via hole punching (or forwarded port, but DON'T do that) and it still uses a relay in case the direct connection doesn't work. The difference is, that ICE is an industry standard that streamlines everything and is way, way more intelligent.

• ICE reruns the connectivity establishment interactively for each peer, you can use different methods (local, direct, relay) for different players
• ICE monitors the connection and reconnects via relay when the direct connection breaks
• ICE monitors the connection and survives a change in IP address, if your power goes out, connecting your laptop to your phone's hotspot will reconnect you
• ICE supports local connections, if you're in the same network as some other player, you'll get a LAN connection (before you had to set and forward different ports for each player and then got a connection via the internet)
• The ICE adapter can pick different relays dependent on your location
• relaying is done using the TURN protocol, which is also standardized and probably more reliable

DO NOT PORT FORWARD WITH ICE

You do no longer need to forward any ports for ICE and won't improve anything by doing so.

TL;DR: DO NOT PORT FORWARD FOR FAF ANYMORE

How does ICE work:

You've made it this far, awesome! Welcome to the interesting part, how the heck does ICE actually work.

For each player you connect to, the ice adapter will run ICE to establish a connection. The process can be broken down into the following steps:

1. Gather candidates
2. Send candidates to the other player's adapter via the FAF server
3. Receive candidates from the other player's adapter via the FAF server
4. Form pairs of candidates (one local candidate, one remote candidate for the other player)
5. Try to reach the other player on all pairs (using the local address, sending to the remote candidate)
6. Pick best pair to succeeded, nominate, confirm (if failed, disable non relay candidates, go to step 1)
7. Start communication
8. (monitor connection, echo requests, once per second, restart on connection loss)

The following types of candidates exist (ignore the last one):

• HOST: a candidate obtained from a local interface, this is an IP address someone in your own network can reach you at (for local connections)
• SRFLX: server reflexive - a candidate obtained by asking a STUN server, where do you see this request coming from - usually your “public IP”
• RELAY: a relay candidate, this is basically a TURN server borrowing you its public IP address and port, you can use it by sending data there through a tunnel
• ( PRFLX**:** peer reflexive - a candidate obtained by asking another player, already connected to, where they see the request coming from - allows connection e.g. within autonomous systems, other WANs, without using the internet )

Side note: STUN - session traversal utilities for NAT, TURN - traversal around NAT using relay networks

So in step 1, your adapter wil gather all possible candidates, e.g.
host 192.168.0.10:6120 (your local IPv4)
host [fe80::2d5d:1a01:9e2b:4ac1]:6121 (your local IPv6)
srflx 1.2.3.4:6122 (your public IP)
relay 116.202.155.226:12345 (faforever.com relay)

It will then send those to the other peer and receive their candidate list (analogous, step 2 and 3).

It will then open a ton of sockets and start talking to the other side (4-7). When it reaches someone, it will attempt to communicate, and then establish connection on the preferred pair, an example list of pairs:
host<->host
host<->host
srflx<->host
host<->srlfx
relay<->srlfx
relay<->relay

A relay connection will ALWAYS succeed, therefore in theory the adapter should always be able to connect.

Side note for developers: One adapter is in offerer mode, the other in answerer mode. Game host is always offerer. Offerer sends candidates first, decides on the chosen candidate pair and monitors the connection.

Development

In theory all of this sounds quite simple. I wrote 95% of the current ICE adapter's code within a single day. The issue is with debugging, testing, figuring out what's wrong.

Every machine is different. Every operating system is different. Every network setup is different. Every ISP is different.

There were five attempts at building an ice adapter for FAF. The second adapter was written in nodeJS using WebRTC (by muellni, before I even joined the ICE project). After doing quite some testing, we couldn't figure out why the CPU load was unbearable.

Therefore the next adapter was written using C++ and a native binding of WebRTC. Debugging that took quite some time. That was when I joined the project, building a test client and server that simulated a running FA game for the adapter and took measurements as well as shipped logs to the sever automatically. From that point on we did weekly ice adapter tests with a dozen of volunteers (I want to thank everyone who helped out on that here , if you didn't get the ICE avatar yet, message a moderator) trying to figure out what was wrong while muellni and vk tried to fix the adapter each week.

After multiple months of that (during which I also wrote a 4th adapter as an exercise using the ice4j library), we finally arrived at a point where we had a stable, lightweight, reliable adapter ready for deployment. A deployment date was scheduled (September 2018), we were doing some final tests.

Then the day came when someone told us their game was lagging. After some investigation, we figured out that their upload bandwidth was too low. But it had worked before. As it turned out, the WebRTC ICE implementation was encrypting all of our packets. Some certain huge internet giants (cough) decided that it's a good idea to enforce packet encryption in WebRTC (ice supports multiple encryption types as well as none). In general I agree with that sentiment. For us, it meant 12 bytes of encryption header added to on average 15 byte long game data packets.

That doesn't sound like much. But in practice it meant, everyone able to play 6v6 might now be stuck with 4v4, everyone who was capable of 4v4 would now be having trouble with 2v2.

That was unacceptable. So we searched for a solution, some hacky way to disable encryption. We found none.

During that investigation I had also written a 5th adapter (somehow I accidentally completely lost the code for the 4th one, it was synced to my cloud and committed into git, I still don't get how that happened). So, back to the start. 6 more months of debugging and testing and countless bugs, issues, and (sometimes self-resolving) problems later and we got ready for deployment.

In total, ICE development took 2.5 years, 5 adapters, a test client and server software for simulation, multiple people joining / quitting the project, weekly tests and lots of volunteers.

Side note: I'm not going to go into detail on the python client vs java client issue which was co-dependent with ICE and caused a huge disaster on the deployment weekend.

Where are we today

There's an unresolved issue with the adapter that occurs in <1/100 cases. It is likely due to the OS not being happy with the adapter spawning hundreds of sockets. It happens in native Windows code and I got no clue what's going on, the reason for socket close supplied by Windows is incorrect. If you are experiencing this (lots of errors about sockets in log), try the following in order (see also the Wiki page on troubleshooting network issues): Deactivate virtual network adapters, use a VPN, reinstall Windows, use Linux.

Relay servers

The ice adapter takes into account mutiple relay servers.

Oceanian and especially australian players are often put behind CGNAT by their ISPs, preventing them from establishing a direct connection. Sometimes this can be resolved by calling your ISP and asking them if it's possible to get you off CGNAT. In this case, the relay via coturn (a STUN+TURN server) is needed. FAF's coturn is located in Nuremberg, Germany.

So two neighbors A and B living next door will be relayed via Europe, if you take into account round trip that means: A -> Europe -> B -> Europe -> A
One round trip is 300 ms at best, so two round trips is 600 ms, above FAF's maximum threshold of 500ms. This made quite a lot of players unable to play.

Therefore one community member offered to host another coturn server in ?Sydney? which is taken into account by the ice adapter. This should (and has, when it worked) allowed nearly all oceanian players to player with everyone else around the world completely fine. (as I said above, as long as the latency is below 500ms)

Current issue

There has been an ongoing issue with the oceanian relay over the past months. The FAF server has been under a denial of service attack for quite some time. This forced the server admins to disable ICMP echo on the server. The ICE adapter uses ICMP echo to determine the nearest relay server.

Therefore all users used the european relay on faforever.com, causing troubles as described in the previous section. We have been trying to fix this but couldn't find a permanent solution.

Server restructuring

To improve the infrastructure, the server admins have been moving the coturn server (relay) from faforever.com over to multiple separate machines. This allows enabling ICMP echo again and building a more robust infrastructure. You might experience some issues in game when playing at the moment, although the ice adapter should reconnect quite quickly. Ideally, we're aiming for multiple TURN servers around the world for the best connectivitiy. At the moment there are 2 in Germany, 1 in Australia and there are investigations about adding one for America.

PS: there is no ping in FA, there is no ICMP echo in FA, it's called a latency or round-trip time (RTT) ;). Otherwise that's like saying “I'm staying in bed today cause I have thermometer”.

Adding to the GDPR thing. You are posting in this forum using a username chosen by you.
This is personally identifying data.
There is no legal way for FAF to show your post to other people on the internet without collecting your permission. (send your complaints to the the european parliament)

Unless you randomize usernames and prohibit people from using free text fields, you can basically not not collect personal data, for which you need people's permission.

This ks actually quite tricky, even my small scripts that present users with a website that remembers them based on IP have to get their consent for GDPR.

In the case of FAF, FAF doesn't collect your data, it doesn't use third party analytics, it doesn't track you or ship advertisements. It actually tells you what exactly is collected in the consent form (see also FAF privacy policy, email for spam prevention, pw reset and contact, username for, well, that should be obvious).

So be happy that you have that control over your data, and if you desire, not to share your username with FAF, then that's your right and you can simply not post here.

Guy who developed the ICE adapter here, Bandwidth of 8 Mbit/s should be fine. I don't have the exact bandwidth requirements in mind, but I remember sth like 20KB/s per player.
It's not just the download that matters, but especially the upload. About the traffic, Client and ICE overhead is insignificant, it's 95% compressed game data.

Back when testing ICE, we had a player with 1 Mbit/s download and they weren't able to play anything 8+. So for dual gap I'd say 4+ Mbit/s will be fine. Beware though that the game is peer to peer and therefore traffic symmetrical, you will need the same thing for uplink as well. (Have been playing 16 player games with 2 Mbit/s up for ages).

About latency, that's a different story. You've come at an unlucky time. The FAF server is currently misconfigured, causing the latency estimation to Faforever.com to fail.

ICE tries to establish a direct connection, but uses a fallback in case that might not be possible (due to e. g. a symmetric NAT as it might be the case in a hotel environment).

We've got a relay server in australia specifically for oceanian players, but due to the mentioned current issue, your ICE adapter will not detect that that should be used.

So it relays your traffic via the server in Nuremeberg, for a round trip from player A->B->A, that means A->Europe->B->Europe->A. So you're going 4 times around half the world. That's the reason your in game latency is currently that high.

We're working on a fix. In the mean time, I've also added a workaround into the ice adapter which uses some fallback servers for the latency estimation (not the relaying, just the figuring out of "should I use the oceanian relay").

You can find the new adapter here: https://github.com/FAForever/java-ice-adapter/releases/tag/v3.1.0

I'm trying to get it inro the release candidate soon, you can test it by replacing the faf-ice-adapter.jar in the natives folder in the install dir of the client.

This will cause your adapter to correctly detect that faforever.com is too far away and use the correct relay. If you're lucky, your relay candidate will be used. It might be the case though for some players in your team, that are in oceania too and still run the old adapter, that their, european relay candidate is used, so you may end up with the same issue, till a fix is released to everyone. (although the issue won't be on your side then)

You can also send me an adapter log after testing, then I can take a look which candidates (relay/direct/...) and which relay were chosen.

If you have any other questions / need help / ideas troubleshooting network issues, feel free to message me on Discord (Official FAF server).

I'll see what I can do, it should work out, but hey, this is FAF, so soon.

@youthenoob said in Is it worth me writing a program to check the network?:

I already work in software development and networking, and am now wondering whether it is worth my time (perhaps a day to two) to write some software to test/record the network performance.

It's always nice to have more data on network performance for investigation.

So before I start investing/wasting too much time on this I thought i would get your opinions. The things I was looking at testing were:

• Ping times to the local gateway, to see if there is jitter (caused by the wifi/bad cabling/s**t router).
• Ping times to a distant IP address to see if there are ISP issues

You can do some latency tests, although I usually just point people to ping.canbeuseful.com for running a latency test during the game, I think weak WiFi links aren't the issue in 99.9% of games, most people are smart enough to think of that themselves.

• iPerf (bandwidth tester), to see the max bandwidth the user has

That would be quite interesting, you could add information to your tool that tells the user what games they will be able to run (2v2, 4v4, 5v5, 6v6, ...). Bandwidth scales rather linearly with peer count, up / down bandwidth is symmetrical.

• The bandwidth used during gameplay, to see if something else is taking all of it (vs total bandwidth).

If you can monitor / obtain info about other system traffic that might be interesting to discover background load, like steam updates.

Traffic in FA consists of 2 packets per peer per tick. 10 Ticks per second. One of those packets is ~15 bytes, the other one is ~25-70 bytes.

• Nvidia driver version (to test for the old driver issue)
• Whether the user is using Global Carrier NAT; games get routed through a relay and ping times are added.

That would also be quite interesting.

A lot of data is also available from the ice adapter debug window, which is currently not visible due to the client not having a JRE with JavaFX.
The ICE adapter already measures latency to each peer.
It can also tell you which candidate pair it chose for each peer (host, prflx, srflx, relay), from which you can infer information about network and NAT architecture.

The best option would probably to add some of those measurements to the ice adapter itself, as it has already gotten access to the traffic as well as quite a bit of other debugging information. You can e.g. just start counting game traffic here: https://github.com/FAForever/java-ice-adapter/blob/master/ice-adapter/src/main/java/com/faforever/iceadapter/ice/PeerIceModule.java#L390

Not sure if that's an option for you though, language-wise as well as taking into account you want to access system traffic info and want to perform measurements to local gateway.

Fix is not released yet but in the works, faforever.com seems to respond to ICMP echo again though. I guess a server admin changed something.

So the old ice adapter should work normal again.

Forwarding ports is unnecessary.
The error you posted is normal.

The main issue is this:

java.util.concurrent.CancellationException: null
	at java.base/java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2468)
	at com.faforever.iceadapter.ice.PeerIceModule.lambda$gatherCandidates$4(PeerIceModule.java:118)
	at com.faforever.iceadapter.util.Executor.lambda$executeDelayed$0(Executor.java:16)
	at java.base/java.lang.Thread.run(Thread.java:832)
Nov 30, 2020 8:17:08 PM com.nbarraille.jjsonrpc.JJsonPeer sendNotification
INFO: Sending Notification:{"method":"onIceConnectionStateChanged","params":[82574,47273,"disconnected"],"jsonrpc":"2.0"}
Nov 30, 2020 8:17:08 PM org.ice4j.util.Logger log
INFO: ICE state changed from Waiting to Terminated. Local ufrag 357851eod9cggh
Nov 30, 2020 8:17:08 PM org.ice4j.util.Logger log
INFO: Failed to receive: java.net.SocketException: socket closed
Nov 30, 2020 8:17:08 PM org.ice4j.util.Logger log
INFO: Closing.
Nov 30, 2020 8:17:08 PM org.ice4j.util.Logger log
INFO: Failed to receive: java.net.SocketException: Socket closed
Nov 30, 2020 8:17:08 PM org.ice4j.util.Logger log
INFO: Failed to receive: java.net.SocketException: Socket closed
Nov 30, 2020 8:17:08 PM org.ice4j.util.Logger log
INFO: Failed to receive: java.net.SocketException: Socket closed
Nov 30, 2020 8:17:08 PM org.ice4j.util.Logger log
INFO: Failed to receive: java.net.SocketException: Socket closed
Nov 30, 2020 8:17:08 PM org.ice4j.util.Logger log
INFO: Failed to receive: java.net.SocketException: Socket closed
Nov 30, 2020 8:17:08 PM org.ice4j.util.Logger log
INFO: Failed to receive: java.net.SocketException: Socket closed

This means your operating system or antivirus is closing the socket and not telling the adapter why.
This only occurs on less than a few percent of devices. I spent a lot of time debugging that and sadly didn't find a solution, the issue lies somewhere within proprietary code of your operating system manufacturer.

Try the options suggested in https://wiki.faforever.com/index.php?title=Connection_issues_and_solutions#Unable_to_connect_to_someone
in the order they're listed in. Each is more likely to fix the issue, e.g. VPN worked on one of my machines when I had that issue.

@askaholic said in Introducing Mapgen Week on Ladder:

Let’s rename it to "Neroxis and Sheikah Map Generator". And while we’re at it let’s rename the faf client to "Downlords and Axels and Brutus’ and Sheikahs and All those other guys FAF Client"

How about Askaholic's FAF server? Can I give the ice adapter a nICE name?
In all seriousness, we should remove people's names from those projects. This applies especially to the client, which only contained Downlord's name as he was asked to distinguish it from the official FAF client and name it unofficially, instead of using sth like "FAF Client 2.0", ... .

@askaholic said in LanCache.NET support:

I think in theory ICE is supposed to be able to discover when you’re playing on LAN and connect directly. Not sure if it actually works though because I haven’t had any luck with it. @Geosearchef would know more about that.

ICE will connect directly in a local network, yes, but this is about caching content requests to the server. I do not really see the point about that though, given the amount of data to be shipped as well as the issue with https encryption.

Just dropping by to say this is awesome! Can't wait to finally play a normal 4v4 using matchmaking. No map picking, no pre determined spots, just pure FA.

@veteranashe That's hard to do implementation wise. People might queue up for that queue in a party of 4. Dynamically scalling that down requires a lot of complexity in the code as well as the algorithm used for matching.

@poop_dynamics said in Bad connection with Wi-Fi:

cable is always better than wifi, you can have the best wifi router and your pc 30 cm away from the router, a 1,95 $ ethernet cable will still beat your 500 $ wifi setup.

Cable is the way to go, but just saying, wifi over cable is pretty reliable too ;). (actually more reliable if you're abusing an old copper cable for ethernet)
Also, please don't put your PC 30 cm away from the transmitter, the polarization might cause you not to get a signal, you should keep at least 1-2 meters of distance.

@askaholic said in Bad connection with Wi-Fi:

I mean in a 1v1 you’re only sending data over a single connection whereas in a 6v6 you’ve got 11 other people who you’re exchanging data with. Plus there are 6x as many people issuing commands which means each connection is transferring about 6x as much data. Multiply those two numbers together and you need about 66x as much bandwidth for a 6v6 as you do for a 1v1.

FA bandwidth requirements scale linearly with player count, they are symmetric. There are 10 ticks a second, each tick you send and receive two packets with each other player. One is 25 byte, the other is 35-70 byte. Total bandwidth requirement for a 6v6 is ~1.2 Mbit/s down, 1.2 Mbit/s up.

@rhyseenz You run the adapter using a JRE that contains JavaFX and double click the tray icon.
Now how to do the first part, euhm, I'm still trying to figure that out. The client's JRE no longer contains JavaFX, the ice adapter would need to be migrated to a newer Java version and ship JFX itself.

Looks to me like either there's a network error causing data to pile up or you are opening too many sockets. You could take a look at how many sockets there are open on your machine and which application created them.

This might also help, although I have no idea about the side effects:
https://customer.precisely.com/s/article/How-to-resolve-java-net-SocketException-No-buffer-space-available-maximum-connections-reached-error-in-Spectrum?language=en_US

Do you know what wait time the algorithms of other games use? No, they don't show you the timer.
So here's my suggestion, just replace the wave countdown with a continously increasing counter,that way everyone's happy.

@Eternal about the 4th point. The server already calculates the amount of land/naval/air vehicles built. It is used for the achievement system, where you can see which types of units you have built in which proportions.

You can find the log directory in the top left of the client. Just send me the ice adapter log of a game im question via pm.

If you've played a game over the past 2 years, you know it. That little notification that pops up when you join the game telling you that some thing called the “FAF Ice Adapter” has been started (which is going to vanish in the next release). But what the heck even is that and how does it help my game run without disconnects?

Warning: This post is quite long and might go into some details on how the internet works as well as the inner workings of FA's simulation and networking. It is also designed as an in-detail explanation and reference I was suggested to write after having to explain ICE to quite a lot of people including developers, so I won't skip the ICE related details.
I tried to explain them as well as possible, but feel free to skip especially the first two sections if you wanna get to the point quicker. I provide a TL;DR whenever possible.

FA's simulation model

First of all, let us dive a bit into the way the game operates.

When you launch a game of FA, you do not connect to a server. There is no server. Back when the game was built, internet connections weren't capable of handling thousands of units per player, so the game had to be developed in a different way. Instead of using a central server, maintaining and managing the entire game state, every game client knows about the complete state of the game. This avoids sending information about what's going on in the game and synchronizing the positions and states of all units and projectiles over the network. Instead, the only information clients need to exchange are the player inputs. The entire simulation is then built deterministically, so that given the state of the game, as well as all player inputs, every player's computer arrives at the some result after calculating one tick in the game.

This removes the need for a (costly) server but also comes with some drawbacks, mainly off-loading CPU load to the clients, therefore slowing the game down to the simulation speed of the slowest client, as well as having to ensure all player inputs are present when the next tick is to be processed. If a player isn't finished calculating the previous tick/step or their inputs where dropped somewhere, the entire game will lock up and you'll get either stuttering or even a connection issue / “player quiet since” dialog. (a so called “lockstep model”)

TL;DR: FA has no server, each player runs the entire game equally

FA's network model

As there is no server, players somehow need to exchange messages for coordinating the game and sending their input to each other player each tick. Therefore the game uses a peer-to-peer architecture. This means, that each client in game is connected to each other client directly.

To allow for network congestion and high distances, there needs to be a gap between the point in time when the command from one player is sent and the point it is processed at. When you issue a movement order, it is in fact only executed 500 ms later (could also be 250, not completely sure), even though the UI layer of the game conveniently hides that from you by starting to rotate the unit immediately. This is in fact only visual and not present in the actual game simulation.

TL;DR: The take away here is that each command is delayed by 500 ms no matter what, so as long as your round-trip latency (RTT) to each other peer stays below that threshold, the game will run just fine. If it rises above 500, even just 550 for one other player, the entire game will start stuttering. This needs to be the case for each connection between all players. If one connection breaks, the entire game will lock up.

Running multiplayer games

A 2v2 only contains twice as many players compared to a 1v1. A 4v4 doesn't sound to bad, given that it's only 8 players. But when you look at the connections that have to be maintained in the background, the story is quite different. As there has to be a connection between each pair of players, the number of connections actually scales quadratically with the number of players:

Players	Connections
2	1
4	6
8	28
12	66
16	120

Remember, even one of those connections breaking means the entire game locks up.

The ghosts of older times - IPv4 and NAT

Now let's get into the actual networking. The Internet works using addresses. Each device has an address, an Internet Protocol (IP) address. If you know the address, you can talk to it.

When packet switching was first designed, it wasn't conceivable that one day there would be more IP capable devices than humans. 32 bits (0s and 1s) per address seemed enough, you know them as the 0.0.0.0 - 255.255.255.255 notation. That's 4,294,967,296 addresses, of which 86 % are available for public usage.

Right now, each US household owns on average of 10 internet connected devices. There are more end user devices connected to the Internet than humans on this spaceship, not even taking into account business devices, public infrastructure, the servers of service providers on the internet and the infrastructure of the internet itself. All of which, need IP addresses.

Put simply, we're completely out of IPv4 addresses. This problem was discovered early, so a replacement was designed called IPv6. (The protocol identifier for IPv4 was 4, 5 was already in use for ST, so there never was an IPv5). IPv6 uses 128 bit for addresses. That's quite some more addresses. 340,282,366,920,938,463,463,374,607,431,768,211,456 to be exact. We could give each grain of sand on this planet an IPv6 address and still be left with 99.9999 % of addresses.

The big issue: all the existing infrastructure uses IPv4, the transition to IPv6 will take some time. (here's a map)

TL;DR: We don't have enough IPv4 address for everybody.

NAT

important for ICE

In the mean time, we'll have to get tricky to save IP addresses. The major building block in doing that is Network Address Translation (NAT). Instead of giving every device an address, we separate the network into lots of local sub networks and give each of them one address. So your router gets assigned a public IPv4 address and all devices in your household use the same address to communicate. Each device get's a private address that is only valid within your local home network (192.168.X.X, 10.X.X.X, …).

Each time you want to open a website, your device sends a request to your local gateway (router) which translates your local network address and replaces it with its global address. The only issue with this: when it receives an answer from the website addressed to your public address, it needs to figure out which local device to send that packet to. It does so by assigning you a random port X on sending the packet using the public address and remembering that. When the packet comes back on port X, it knows exactly which private/local address to translate it to.

from: https://commons.wikimedia.org/wiki/File:NAT_Concept-en.svg

Sidenote: This prevents you from hosting a webserver, as no one can reach your device without you connecting to them first. You need to tell your router what to do with the packets it receives, which is exactly what port forwarding does. ("If you get a package on your public address addressed to port 80, that should go to PC-X in the local network")

NAT Traversal

This is fine, if your users “behind NAT” only access remote services, but do not have incoming connections. But if you dream of establishing a connection between two people behind a NAT, which is what a peer to peer application like FAF does (66 times for a 4v4), you simply cannot. The routers on both ends just won't know what to do with the packets they get.

To get around this, NAT traversal methods have been developed. One of them is hole punching. Hole punching works by both clients just sending each other packets on a fixed port, therefore convincing their NAT they had an outgoing request, telling it what to do with incoming packets. It requires a third party server, that is publicly reachable telling both clients to start transmitting to each other.

All of that is assuming though that your router actually has a public IPv4 address. Oftentimes this just isn't the case though, e.g. in Dual Stack Lite. There, you actually only have an IPv6 connection and IPv4 traffic get's tunneled via that. Your ISP (internet service provider) will then often use something called a Carrier-Grade NAT (CGNAT) / symmetric NAT. There, multiple customers of the same ISP get put behind a NAT with one or multiple! public IP addresses.

So your requests might use different public IP addresses shared with hundreds of other customers. Hole punching is impossible in that circumstance.

TL;DR: NAT traversal and hole punching can get you through NAT, but only in some situations.

FA Networking

To recap: we need to establish a connection between all players, all of which are located behind a NAT and cannot be reached under a public IP address. If a connection between any two players breaks, we cannot run the game.

The game by itself already does some kind of NAT traversal, it can already use STUN. It's not very reliable, most of the time only works for simple host to host connections and rolls a dice on being able to hole punch.

This meant that most players had to resort to port forwarding, telling their router by modifying a setting which incoming packets to send to their machine. (which is impossible with CGNAT)

Legacy FAF Networking

To mitigate this, FAF implemented a proxy server that allowed players to establish a connection even if they were unable to get a direct connection. On client startup, the server would test if hole punching succeeds, and if not establish a proxy connection for the game to use.

This way, players with a forwarded port or hole punching succeeding got a direct connection, anybody else was still able to connect to other players using the proxy. Somewhat, sometimes, if they got lucky…

A tale of REFAFing

New players might no longer even know this word. The REFAF was an essential part of FAF culture for many years. It was used to tell someone trying to connect to a game that they should rejoin. But not just the game, restart the entire FAF client.

It was pretty common for players not to get a connection to others in lobby, almost all games involved certain players not being able to connect to each other in lobby and having to refaf multiple times, before the game had even started.

If you managed to start the game, in a substantial amount of matches, at least two players would then loose connection during the game (locking the entire game), still able to chat with everyone else, trying to figure which one of the two players is going to leave so the game may continue. (an especially bad debate, if the players happened to be on opposing teams)

Side note: But what the heck does REFAFing actually do? The connectivity type was defined by the test run on client launch. Subsequently, all connections in all games and to all players would use the same connection type. If the server could reach you directly, but you didn't manage to get a direct connection to another player, you were out of luck as you weren't using the proxy. REFAFing reruns the connectivity test, hoping for a different result. (there's likely way more issues at play here, but that's one of the main reasons)

So, after “fixing connections” turned out as the top contender in a survey about what FAF users wanted, it was decided that ICE, being a new, standardized protocol to establish connections was the future of FAF connectivity. And so began the 2.5 year development cycle that successively yielded 5 different, all-most nearly completed, ice adapters.

The ICE Adapter

The FAF ice adapter aims at managing all connections for the game. It's a program running on your local system started/stopped by the FAF client that offers an interface to the game. The game is then told by the FAF server/client/adapter to connect to other players, supplying it a wrong IP address. If you join a FAF game nowadays, your game will think all of your peers it's connected to are located on your local machine. It sends all its traffic to the ice adapter, which then figures out how to forward it to the other players' ice adapters, which then forward it to their games.

The ICE adapter uses the Interactive Connectivity Establishment (ICE, standardized as per RFC5254) to, well, interactively establish connections.

How does ICE compare to the old solution

ICE actually doesn't change that much compared to the old solution. It still uses direct connections via hole punching (or forwarded port, but DON'T do that) and it still uses a relay in case the direct connection doesn't work. The difference is, that ICE is an industry standard that streamlines everything and is way, way more intelligent.

• ICE reruns the connectivity establishment interactively for each peer, you can use different methods (local, direct, relay) for different players
• ICE monitors the connection and reconnects via relay when the direct connection breaks
• ICE monitors the connection and survives a change in IP address, if your power goes out, connecting your laptop to your phone's hotspot will reconnect you
• ICE supports local connections, if you're in the same network as some other player, you'll get a LAN connection (before you had to set and forward different ports for each player and then got a connection via the internet)
• The ICE adapter can pick different relays dependent on your location
• relaying is done using the TURN protocol, which is also standardized and probably more reliable

DO NOT PORT FORWARD WITH ICE

You do no longer need to forward any ports for ICE and won't improve anything by doing so.

TL;DR: DO NOT PORT FORWARD FOR FAF ANYMORE

How does ICE work:

You've made it this far, awesome! Welcome to the interesting part, how the heck does ICE actually work.

For each player you connect to, the ice adapter will run ICE to establish a connection. The process can be broken down into the following steps:

1. Gather candidates
2. Send candidates to the other player's adapter via the FAF server
3. Receive candidates from the other player's adapter via the FAF server
4. Form pairs of candidates (one local candidate, one remote candidate for the other player)
5. Try to reach the other player on all pairs (using the local address, sending to the remote candidate)
6. Pick best pair to succeeded, nominate, confirm (if failed, disable non relay candidates, go to step 1)
7. Start communication
8. (monitor connection, echo requests, once per second, restart on connection loss)

The following types of candidates exist (ignore the last one):

• HOST: a candidate obtained from a local interface, this is an IP address someone in your own network can reach you at (for local connections)
• SRFLX: server reflexive - a candidate obtained by asking a STUN server, where do you see this request coming from - usually your “public IP”
• RELAY: a relay candidate, this is basically a TURN server borrowing you its public IP address and port, you can use it by sending data there through a tunnel
• ( PRFLX**:** peer reflexive - a candidate obtained by asking another player, already connected to, where they see the request coming from - allows connection e.g. within autonomous systems, other WANs, without using the internet )

Side note: STUN - session traversal utilities for NAT, TURN - traversal around NAT using relay networks

So in step 1, your adapter wil gather all possible candidates, e.g.
host 192.168.0.10:6120 (your local IPv4)
host [fe80::2d5d:1a01:9e2b:4ac1]:6121 (your local IPv6)
srflx 1.2.3.4:6122 (your public IP)
relay 116.202.155.226:12345 (faforever.com relay)

It will then send those to the other peer and receive their candidate list (analogous, step 2 and 3).

It will then open a ton of sockets and start talking to the other side (4-7). When it reaches someone, it will attempt to communicate, and then establish connection on the preferred pair, an example list of pairs:
host<->host
host<->host
srflx<->host
host<->srlfx
relay<->srlfx
relay<->relay

A relay connection will ALWAYS succeed, therefore in theory the adapter should always be able to connect.

Side note for developers: One adapter is in offerer mode, the other in answerer mode. Game host is always offerer. Offerer sends candidates first, decides on the chosen candidate pair and monitors the connection.

Development

In theory all of this sounds quite simple. I wrote 95% of the current ICE adapter's code within a single day. The issue is with debugging, testing, figuring out what's wrong.

Every machine is different. Every operating system is different. Every network setup is different. Every ISP is different.

There were five attempts at building an ice adapter for FAF. The second adapter was written in nodeJS using WebRTC (by muellni, before I even joined the ICE project). After doing quite some testing, we couldn't figure out why the CPU load was unbearable.

Therefore the next adapter was written using C++ and a native binding of WebRTC. Debugging that took quite some time. That was when I joined the project, building a test client and server that simulated a running FA game for the adapter and took measurements as well as shipped logs to the sever automatically. From that point on we did weekly ice adapter tests with a dozen of volunteers (I want to thank everyone who helped out on that here , if you didn't get the ICE avatar yet, message a moderator) trying to figure out what was wrong while muellni and vk tried to fix the adapter each week.

After multiple months of that (during which I also wrote a 4th adapter as an exercise using the ice4j library), we finally arrived at a point where we had a stable, lightweight, reliable adapter ready for deployment. A deployment date was scheduled (September 2018), we were doing some final tests.

Then the day came when someone told us their game was lagging. After some investigation, we figured out that their upload bandwidth was too low. But it had worked before. As it turned out, the WebRTC ICE implementation was encrypting all of our packets. Some certain huge internet giants (cough) decided that it's a good idea to enforce packet encryption in WebRTC (ice supports multiple encryption types as well as none). In general I agree with that sentiment. For us, it meant 12 bytes of encryption header added to on average 15 byte long game data packets.

That doesn't sound like much. But in practice it meant, everyone able to play 6v6 might now be stuck with 4v4, everyone who was capable of 4v4 would now be having trouble with 2v2.

That was unacceptable. So we searched for a solution, some hacky way to disable encryption. We found none.

During that investigation I had also written a 5th adapter (somehow I accidentally completely lost the code for the 4th one, it was synced to my cloud and committed into git, I still don't get how that happened). So, back to the start. 6 more months of debugging and testing and countless bugs, issues, and (sometimes self-resolving) problems later and we got ready for deployment.

In total, ICE development took 2.5 years, 5 adapters, a test client and server software for simulation, multiple people joining / quitting the project, weekly tests and lots of volunteers.

Side note: I'm not going to go into detail on the python client vs java client issue which was co-dependent with ICE and caused a huge disaster on the deployment weekend.

Where are we today

There's an unresolved issue with the adapter that occurs in <1/100 cases. It is likely due to the OS not being happy with the adapter spawning hundreds of sockets. It happens in native Windows code and I got no clue what's going on, the reason for socket close supplied by Windows is incorrect. If you are experiencing this (lots of errors about sockets in log), try the following in order (see also the Wiki page on troubleshooting network issues): Deactivate virtual network adapters, use a VPN, reinstall Windows, use Linux.

Relay servers

The ice adapter takes into account mutiple relay servers.

Oceanian and especially australian players are often put behind CGNAT by their ISPs, preventing them from establishing a direct connection. Sometimes this can be resolved by calling your ISP and asking them if it's possible to get you off CGNAT. In this case, the relay via coturn (a STUN+TURN server) is needed. FAF's coturn is located in Nuremberg, Germany.

So two neighbors A and B living next door will be relayed via Europe, if you take into account round trip that means: A -> Europe -> B -> Europe -> A
One round trip is 300 ms at best, so two round trips is 600 ms, above FAF's maximum threshold of 500ms. This made quite a lot of players unable to play.

Therefore one community member offered to host another coturn server in ?Sydney? which is taken into account by the ice adapter. This should (and has, when it worked) allowed nearly all oceanian players to player with everyone else around the world completely fine. (as I said above, as long as the latency is below 500ms)

Current issue

There has been an ongoing issue with the oceanian relay over the past months. The FAF server has been under a denial of service attack for quite some time. This forced the server admins to disable ICMP echo on the server. The ICE adapter uses ICMP echo to determine the nearest relay server.

Therefore all users used the european relay on faforever.com, causing troubles as described in the previous section. We have been trying to fix this but couldn't find a permanent solution.

Server restructuring

To improve the infrastructure, the server admins have been moving the coturn server (relay) from faforever.com over to multiple separate machines. This allows enabling ICMP echo again and building a more robust infrastructure. You might experience some issues in game when playing at the moment, although the ice adapter should reconnect quite quickly. Ideally, we're aiming for multiple TURN servers around the world for the best connectivitiy. At the moment there are 2 in Germany, 1 in Australia and there are investigations about adding one for America.

PS: there is no ping in FA, there is no ICMP echo in FA, it's called a latency or round-trip time (RTT) ;). Otherwise that's like saying “I'm staying in bed today cause I have thermometer”.

Can you send me an ice adapter log?

@Krieggs You can give this a read: https://forum.faforever.com/topic/2264/moved-to-australia-is-it-still-possible-to-play/7
I get asked this question about the inner workings of the ice adapter quite a lot, I think I should write that down in the Wiki in more detail.

I'll see what I can do, it should work out, but hey, this is FAF, so soon.