Troubleshooting FAF client latency

So I've been at this for a few hour and have run out of ideas. I'm trying to figure out why one of our group of players has a super bad latency when we play FAF. The facts so far.

We've confirmed it's not his internet. We are all located in Australia and all other players are 25-50ms average. While the ping for Omer80 is 500-600ms. We've checked with multiple "speed test" sites and his ping is normal (~30ms). We've also started a 2 player SC Forged Alliance game (via Steam matchmaking) and the ping was ~20-30ms.

The FAF multiplayer games are playable and we can complete a match, but sometimes it's like playing in slow motion.

I've spent the morning comparing logs and reading articles I can find in the forums and internet and have found no cause. He's already done a full reinstall and is fully up-to-date with the client.

If I was guessing I'd assume he's going to the Europe Coturn Server, but I can't tell if that's the issue. And I don't know if you can "hard code" the server in?

Log attached. Any help would be super useful.

Side note: Loving FAF after just starting with it a month or so ago. Have now dragged another 10 players to play with us

omer_advanced-ice-adapter.log

I think he could try using the pre release client because I saw that it bumps the ice adapter version and I think that includes a fix for ANZ players getting the Europe relay.

https://forum.faforever.com/topic/2635/client-2021-10-0-upcoming-features

I used to have bad latency when I was on Wifi, but usually only in games with 6+ players

Now I'm on a landline connection and pretty much the only time I get lag is if other people in the house are using the internet

Cool, will report bad post testing with the pre-release client.

So results are in and the the 2021.10.0 client RC is different, but worse. When using the new client we can't connect to join a game. We've tried swapping the host to see and that doesn't help. As the host you do get a notification they are joining, but they never make it into to the game lobby to start a match.

Latest log of me trying to join attached.

My current guess is that the new client is better a directing to the local (Australian) coturn server, but there is some network issue stopping that from working.

advanced-ice-adapter.log

I had an issue last night: 10 player game, and several times a minute my pings started spiking over 2000ms. I ran ping bbc.co.uk in a console alongside and nearly all results were under 140ms (not great but clearly not the same problem) with zero packet loss — but 3 pings in sequence were over 1000ms.

My guess is that a small number of packets had high ping time, and FAF is very sensitive to this (continuous transmission and any single delay causes an issue in game) where the ping command only sends one per second. Also we had stormy weather which may have been the cause (the local cables are overhead and bandwidth very low to start with at 2-3 Mb/s).

I really need to get in contact with Brutus and get some more information on this. I was trying to write some software to diagnose these issues.

So here is my understanding of what could cause the issues you are experiencing:

• The pipe is too small. If someone is playing with a connection that is too slow (Mbs per second), the data will basically queue up until the other data is transmitted. Think of it like waiting it a queue of traffic. In my own testing a connection of 6Mb up and down the is minimum needed. Other people here that made the adapters say it will work with less but in my own testing 6Mb is the minimum. Even with 6Mb, if the person is doing something else like watching youtube, youtube will send micro bursts, which coupled with a poor router will increase ping times until the data is sent. In the UK I had a 300Mb connection and had no issues, but when trying with a 4Mb connection I could not play gap. Even if the speed test works great, well some ISPs do a thing called "traffic shaping". They degrade certain connections to prevent people using up too much of their network. I need to emphasis that the upload and download speed are both important. In Australia the connection speeds are awful. I'm currently limited to a 40Mb connection.

• Ports. Everyone connects using a thing called NAT. NAT means a public IP address is shared by multiple PCs. Think of it like having an apartment block. Your public IP address is the street address, then you will have an address in your building. Some cheaper ISPs (and Russians) use a thing called Global Carrier NAT. It means that the ISP is also using NAT. When you connect using FAF, the ports should get opened, but that is not always the case. Re FAF'ing means the ports are normally opened from the previous attempt.

It is also possible that another service is using those ports, hence why the relay is used. Check to see whether the ISP uses GC NAT. I may be wrong but if GC NAT is used, the connection will go via Nuremberg as GC NAT IP addresses are not strictly Australian ones, so it doesn't register the IP address as being Australian. I believe that if you speak to your ISP you may be able to get GC-NAT disabled.

https://www.aussiebroadband.com.au/help-centre/nbn/tech-support/cg-nat/

• Firewalls. There are loads of things that can block the ports. Antivirus comes with firewalls, as does windows. Either one of those can interfere. This is similar to the above NAT. It is possible to try disabling the firewall completely, then try. If it fixes the problem, you can add exceptions to your firewall.

• Wifi. If the signal gets interfered with, then the device will need to resend the information. Depending on what is around the device will mean that it might take a few hundred ms to get the information through. Make sure you only play games on ethernet.

• To the chap that said his BBC ping was okay but others were peaking, well a thing called "Quality Of Service" will prioritise the ICMP packet (ping request), so if you connection is maxed out as you download your strange porn, the Ping should come back without the delay.

Thanks that's some useful info. The GC NAT seems like a strong contender for being the issue here. IP4 address keep getting more expensive, so I can understand why ISPs would be trying to reduce the amount they need.

Our local lockdown has just ended, so we should be able to do a test with the same PC at a different location soon.

Update. Tested today and there was no issues when running the same PC from a different house. At this stage it seems to be ISP related. Will do a test with the new client just released, given we've tested with the RC that probably won't help. Would be good to know what exactly is going wrong with the network traffic, as the ISP support was wasting time talking about WiFi version support on the router (not even using wifi, ethernet only).

No offence to the guys at the ISP, but they just go through a checklist. They literally, don't have a clue. Anyone with any real knowledge of networking probably wouldn't be working in a call centre for an ISP. A good networking engineer could earn $800(US) a day, so no way are they going to sit in a call centre.

Things I would do:

• Step one: work out if the ISP is using Global Carrier NAT. Check your router IP address to the ISP. You can get this by login into the router. To log into the router, go into command prompt and type "ipconfig". The default gateway is the address of your router on your network. Load up a browser and go to http://<the IP address> (might be https). Check to see what your public IP address is. Using that you should be able to see if you are using Global Carrier NAT.

• Check the ISPs website to see if GC-NAT is used. It might not be.

• Go to https://whatismyipaddress.com/ and get your public IP address. See if you can ping the address. it might be that your ISP doesn't respond to ICMP (ping requests) for security. Go into your router and try and enable ICMP requests under security settings. I'm not sure how the FAF adapter decides whether or not to use the relay.

• Configure the DMZ of your router. This is a security risk. ALL TRAFFIC THAT GOES TO YOUR EXTERNAL IP ADDRESS WILL BE SENT TO THE DMZ ADDRESS! A DMZ stands for Demilitarized Zone. Any traffic that reaches your public IP address will be sent to this IP address. This means any vulnerabilities on your PC will be vulnerable, as it will be public-facing. Worth trying as a quick check though. It will guarantee that the router isn't the issue.

• Check to see if your router has any built-in firewalls that are stopping services. I once had to disable a setting to allow VPNs.

• Try pinging each others home IP addresses. There is a (small) chance that the ISP has some really bad routes to get from one Australian address to another. A few months ago, I was staying in a hotel and all the data was going via their HQ in Asia before it went a few km down the road. It's possible that the ISP has bad BGP routes. if you go into command prompt (cmd) type "tracert <destination IP address>" and it will show the route your traffic is taking.

These are my thoughts off the top of my head.

Thanks YouTheNoob. That's a really useful list. I suspect that an ISP change could be in order, but we'll see if we can confirm the actual issue first.

We noticed issues with several ISPs even in Europe. Best is probably raise a ticket so many times until they see it as a serious issue