RTP Session drops. Potential mitigation.

beebob

Hello there, maybe its the wrong Subforum, please move it to the preferred location.

FYI: This post ignores the DDOS attacks and faulty/shitty internet connections.

I have read through a few technical issue posts concerning disconnects and read through the logs.

A Few things that came to my attention

the vast majority are drops of the UDP/RTP Session to a single or a number of players
the ICE Client seems to utilize the full range of the High-Port spectrum
The ice advanced log is not detailed enough.

Now, I don’t pretend to understand the ICE Client, but as someone handling complex PBX (Telephony / SIP) Systems which also are based on RTP Sessions I can see a lot of similar issues here.

Since we can clearly see a successfully established RTP Session, which fails later on, we can assume that the firewall configuration is ok on all involved sides.

At one point or another we all have experienced the issue where we called some hotline, and got kicked out mid-call, or specialy when we are put on hold without waiting music being played.
This is considered inactivity for the RTP Session. Which can result in a timeout. I tend to believe that this is very similar to what happens to the RTP Sessions between players.
RTP Sessions drop by design when there is not enough noise on the line for defined period of time.

The PBX Admin can mitigate this issue by allowing much longer timeout periods or adding waiting music ← No shit.
I think this is something worth looking into from DEV side.

RTP-Sessions are a XXX, a nightmare for anyone with a well tuned Firewall.
To avoid health issues of the security admin, the PBX Admin usualy would limit the port range that RTP is allowed to utilize. 2.5X the factor of active sessions. E.G. lets assume we have a top of 2000 parallel players being active: We would allow 5000 RTP Ports. E.G. a predefined port range from 15000-20000. That's easier to manage and to document.
The ICE Log and the advanced ICE log only tell me that sessions are dropped. But I don’t see any reasons/error codes. Might be my eyesight though.

o/

deletethis

I have absolutely no clue if whatever you said is relevant but @Brutus5000 is your man.

Did you know work on the ice adapter is ongoing?

Brutus5000

@beebob said in RTP Session drops. Potential mitigation.:

the vast majority are drops of the UDP/RTP Session to a single or a number of players

I think it's not one player dropping all other players, but one player dropping one connection to another player.

@beebob said in RTP Session drops. Potential mitigation.:

Since we can clearly see a successfully established RTP Session, which fails later on, we can assume that the firewall configuration is ok on all involved sides.

That is a fallacy. Just because you were allowed to open connections does not mean, that the firewall doesn't decide to kill it later. Even the firewall configuration could change dynamically somewhere in the internet in between and you wouldn't even notice. (Simple example would be if you are on 4g and you are handed over from one station to the next).
When all players are connected, all we know that it works right now.

The big differene to telephony or other multimedia streams where ICE is used is: packet loss does not matter. If you switch over to a different connection and dropped a few packets, voice or image get blurry for a few seconds is fine. But in FAF it kills a game.

@beebob said in RTP Session drops. Potential mitigation.:

The ICE Log and the advanced ICE log only tell me that sessions are dropped.

The operating doesn't know why a connection is dropped, thus it can't tell us. Hell, with UDP and TCP you don't even know WHEN exactly the connection dropped.

@beebob said in RTP Session drops. Potential mitigation.:

the ICE Client seems to utilize the full range of the High-Port spectrum

I think we are using 40000-60000. We can reduce this, but what that help? I think the port range is mostly relevant for a system admin on a backend server trying to reduce the amount of open ports.

@beebob said in RTP Session drops. Potential mitigation.:

The ice advanced log is not detailed enough.

Most of the ICE logic is not implemented by us, but by the library called ice4j. It's a blackbox for us. We don't know what it exactly it does and due to connection timeouts etc. it's also impossible to debug. We can't add any logging there. Apart from that feel free to make suggestions what we could log.