Private coturn server for my games.

Brutus5000 · 16 Oct 2024, 18:32

The Forged Alliance engine will alway open one udp port per player in the game and send data multiple times.

In theory you could try to reverse engineer the binary network protocol and remux the streams that some intermediate software merges outgoing traffic into 1 connection and splits up incoming traffic into "per player" traffic.

The is actually a developer on Discord trying to do exactly that. I'm not sure if this really solves network connection issues. What I am sure is that it will cause latency issues for all users geographically far away from the central server.

2 players from australia don't take the direct route but sent over a gateway in europe adding 500ms+ latency.

We don't know much about the binary data protocol

IamFromRussia · 16 Oct 2024, 22:29

@brutus5000 If we know so little about the data protocol, how do you get information about the game starting/ending? How do you get messages from the chat?

If it's not too much trouble, send the discord of the modder who is trying to do this )

Surfer · 17 Oct 2024, 00:58

Game using peer-to-peer (P2P) architecture where every player connected to each other.
Thats good for LAN party and terrible for modern networks with all those firewalls and VPNs.
When two players can't establish direct connection, coturn/stun server help them to find the route using some smart tricks.
If you experience connectivity problems consider buying static IP address from your ISP.

As for single game server, that should improve connectivity, theoretically.
For example in 8v8 the total connection count is 49 (7 per player), with single server its only 7 (1 per player).
Chance of connection lost is much lower with less connection count.
But again, if host act as a server and unlucky with ISP all that doesn't matter.

If server running by FAF org or privately another problem arise:

hardware and traffic cost
server require maitance
increased latency depending on player location (nogo for Oceanic players)

I made few tests but noting real, the code is available on github

https://github.com/anykey111/fa-mp-test

Brutus5000 · 17 Oct 2024, 07:11

@iamfromrussia said in Private coturn server for my games.:

@brutus5000 If we know so little about the data protocol, how do you get information about the game starting/ending? How do you get messages from the chat?

Because there are different datastreams

There is a more "high-level" protocol where the game gives information about the game state. This is called GPGNet protocol and it's only sent between Game <-> FAF Client <-> FAF lobby server. This is well known.

Then we have the game datastream (which is basically the replay format). This is except for few unknown bits well known by now.

The part you are asking for is the network metadata between the UDP connections of the games and this was never in the focus. If you look at @Surfer's git repository, he reverse engineered a few pieces. But that is basically new work from the last few months.

If it's not too much trouble, send the discord of the modder who is trying to do this )

Its @Surfer or anykey111 in Discord.

IamFromRussia · 19 Oct 2024, 12:02

@brutus5000 said in Private coturn server for my games.:

In theory you could try to reverse engineer the binary network protocol and remux the streams that some intermediate software merges outgoing traffic into 1 connection and splits up incoming traffic into "per player" traffic.

@Surfer @Brutus5000 Why do we need to reverse engineer the binary protocol? Do we need to know what's inside the binary data stream? Why can't we work with binary stream packets like black boxes? Just passing them on without understanding what's inside.

Surfer · 19 Oct 2024, 14:32

@iamfromrussia You don't need deep protocol knowledge for such task.

See the patch fa-mp-test/blob/main/0001-add-proxy-server-draft.patch that completely remove P2P exchange from the ice-adapter and use well known server (main.c)

Please, state cleary what is the point?
If you need private server, just host multiplayer game and tell firends your IP, they will join and thats it.

IamFromRussia · 19 Oct 2024, 17:07

@surfer said in Private coturn server for my games.:

Please, state cleary what is the point?

We usually play in a small group of 4-5 people. When I say: "Let's try to play 8vs8." I usually get the answer: "We'll wait a long time for the game, and then we'll still leave because of the lags."
If I could say something like: "I have my own private server and there will be no lag" That would be a strong argument.

If FAF has the ability to register your own server in the client through which traffic would pass instead of a P2P connection.

If a person who has registered his own relay server creates a game, then everyone who connects to this game will redirect their traffic to the relay server.

If this is a high-quality VPS with a static IP, then the connection will be more stable.

As an option, you can send an excessive number of UDP packets to the server, for example, send two duplicates for each packet, which will increase the probability of reception on the player's side and remove duplicates in ICE. But here I am not sure about the possibility of implementing this. https://gafferongames.com/post/deterministic_lockstep/ (Yes, this is much more complicated than a simple Relay server)

This is a very brief description of the feature, so please don't beat me up XD

Brutus5000 · 19 Oct 2024, 20:23

You assume that running your own stable coturn solves all connection issues. Our reports from ICE adapters tell otherwise.
The problem is not necessarily unstable coturns, but with issues making a connection even with coturn.

Surfer · 19 Oct 2024, 20:46

Keep in mind with your own coturn server all 16 players should agree beforehand to use same settings (FAF Client -> "Preferred coturns for ICE").
With centralized game server they must also use custom FAF client modification.

IamFromRussia · 19 Oct 2024, 23:10

@surfer said in Private coturn server for my games.:

As for single game server, that should improve connectivity, theoretically.
For example in 8v8 the total connection count is 49 (7 per player), with single server its only 7 (1 per player).

I still can't understand, does it make sense to combine binary traffic? If it's such a difficult task, is it worth spending time on? I'm not an expert in network programming, but isn't UDP a connectionless protocol https://docs.oracle.com/cd/E19620-01/805-4041/6j3r8iu2f/index.html ?
Is there a difference whether the client will receive information on one port, on 16 or on 16,000 ports that the server listens on in case of UDP connection?

BanthaFodder · 20 Oct 2024, 09:36

FAF has been operational for over a decade, and has yet to establish reliable consistent connections between players.

You rarely see this issue occur in steam, and the best part in steam is that, if someone does disconnect, you can simply wait for them to reconnect. 9/10 times they successfully reconnect too. I remember when you could do this in faf many moons ago, but sadly this is no longer the case. And you can't reconnect in the client either to try salvage it.

Idk what the problem is with faf and players not being able to reliably connect to eachother in game. It's very painful going through 32 minutes of video gameplay only to have some random player randomly disconnect for no apparent reason.

I doubt faf will ever fully resolve this issue, I don't think they even know how.

Brutus5000 · 20 Oct 2024, 10:28

@yew said in Private coturn server for my games.:

I doubt faf will ever fully resolve this issue, I don't think they even know how.

This is correct. We don't know. None of us are professional game and/or network engineers. We can not afford to pay for a company to fix it for us. And making ourselves dependent of Steam is also not an option (even all the legal issues aside).

Surfer · 20 Oct 2024, 14:04

@iamfromrussia said in Private coturn server for my games.:

I still can't understand, does it make sense to combine binary traffic? If it's such a difficult task, is it worth spending time on? I'm not an expert in network programming, but isn't UDP a connectionless protocol https://docs.oracle.com/cd/E19620-01/805-4041/6j3r8iu2f/index.html ?
Is there a difference whether the client will receive information on one port, on 16 or on 16,000 ports that the server listens on in case of UDP connection?

Just for optimization to reduce the total number of connections and the numer of individual packets flying through. Every UDP packet is independent, for example 8v8, its about 30 packets on every tick (lets say ~250 milliseconds). The total number of packets is 30164=1920 per second, for each player server should open 15 ports, 15*16=240.

For now the biggest problem for me is testing, i need atleast 3 players. using my main PC as host, remote desktop on another PC and old PC under the table.

snoog · 20 Oct 2024, 14:52

If you need help testing feel free to ping me. I work from home so I'm available most of the day anyway.

Surfer · 23 Oct 2024, 18:32

Ok, now we can mythbust the proxy server approach.
I changed the faf-ice-adapter to use a single TCP socket for all communications.

faf-ice-adapter repo
proxy-test repo

After serveral mock tests i didn't see any difference. On weekend i plan to rent a basic VPS and run test in the wild.

Brutus5000 · 23 Oct 2024, 20:28

In mock tests ICE also always works perfectly

Nobody ever questioned whether one large plain proxy would work. The ICE adapter laid foundations for rerouting the game traffic anywhere.
However it has drawbacks in operational costs and in latency.
And in theory plain ICE connections should always work (with relay as a fallback which basically is single-connection proxy). As such in theory it is also superior to a proxy-solution that does not traffic deduplication.
But in practice (especially Windows and/or security software interference) and for the given software libraries available we see, that it does not hold up the promises.

However under current DDOS situation a proxy server becomes more interesting if it were tunneled through cloudflare websockets...

Surfer · 28 Oct 2024, 00:44

I have first feedback from 2v1 game, testers say it didn't feel better.

How to join testing?

backup your FAF Client/natives/faf-ice-adapter.jar
replace with https://github.com/user-attachments/files/17532274/faf-ice-adapter.zip or build from the source (https://github.com/anykey111/faf-star-adapter)
host game, invite firends (they must use the same version of faf-ice-adapter.jar as you're using)

Why should i participate?

For the sciene, even if we know that this solution is inferior there is no proofs and people keep returning to this idea.

Salz · 16 Nov 2024, 09:09

Can I see in the logs/debugger which player is in my lobby and has coturn server connections?
Can I prevent this by not allowing coturn server connections (so, I could have the IPs and block them as hosts in my firewall, then no more players can get into my lobby - there are only 4 Germany 1 / Finland / USA East / Global of them - or are there several and only the key regions are listed)?
When you switch from the lobby to the actual game, are all connections renegotiated, right or wrong? And what happens in the lobby is just a rudimentary check to see if it works at all with the connections?
What about the option “Allow the ICE adapter to use IPv6”? It says that it causes connection problems for some players, but what advantage does it have in general (more stable connections when a connection is established?)?
What actually causes the lag in a DDoS when players are in the game, if it's just P2P connections? If players are connected to coturn servers and these are then attacked (the coturn servers - are they attacked at all?), or are there other services that limit the game (live replay?)? Does the player still have connections to FaF service servers during the game, which can cause lag (in-game behind issues)?

Salz · 18 Nov 2024, 19:29

If you block the faf coturn server selected in the settings, you cannot connect to other players.
So a coturn server is always used?

Germany 1 = Hetzner (Dedicated Server?) always same IP
Finland = Hetzner (Dedicated Server?) always same IP
US East = Hetzner (Cloud?) always same IP although cloud?
Global = xirsys / DigitalOcean (Cloud?) mostly always a different IP

As the only coturn server, what does Global actually want with port 80?

Example:

Germany 1:
INFO Using all viable ice servers: [94.130.225.99:3478/tcp, 94.130.225.99:3478/udp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [94.130.225.99:3478/tcp, 94.130.225.99:3478/udp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [94.130.225.99:3478/tcp, 94.130.225.99:3478/udp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [94.130.225.99:3478/tcp, 94.130.225.99:3478/udp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [94.130.225.99:3478/tcp, 94.130.225.99:3478/udp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [94.130.225.99:3478/tcp, 94.130.225.99:3478/udp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [94.130.225.99:3478/tcp, 94.130.225.99:3478/udp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [94.130.225.99:3478/tcp, 94.130.225.99:3478/udp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [94.130.225.99:3478/tcp, 94.130.225.99:3478/udp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [94.130.225.99:3478/tcp, 94.130.225.99:3478/udp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [94.130.225.99:3478/tcp, 94.130.225.99:3478/udp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [94.130.225.99:3478/tcp, 94.130.225.99:3478/udp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [94.130.225.99:3478/tcp, 94.130.225.99:3478/udp] (c.f.iceadapter.ice.PeerIceModule:177)

Global:
INFO Using all viable ice servers: [207.154.234.214:3478/tcp, 207.154.234.214:3478/udp, 207.154.234.214:80/udp, 207.154.234.214:80/tcp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [207.154.234.214:3478/tcp, 207.154.234.214:3478/udp, 207.154.234.214:80/udp, 207.154.234.214:80/tcp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [207.154.234.214:3478/tcp, 207.154.234.214:3478/udp, 207.154.234.214:80/udp, 207.154.234.214:80/tcp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [207.154.234.214:3478/tcp, 207.154.234.214:3478/udp, 207.154.234.214:80/udp, 207.154.234.214:80/tcp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [207.154.234.214:3478/tcp, 207.154.234.214:3478/udp, 207.154.234.214:80/udp, 207.154.234.214:80/tcp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [207.154.234.214:3478/tcp, 207.154.234.214:3478/udp, 207.154.234.214:80/udp, 207.154.234.214:80/tcp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [207.154.234.214:3478/tcp, 207.154.234.214:3478/udp, 207.154.234.214:80/udp, 207.154.234.214:80/tcp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [207.154.234.214:3478/tcp, 207.154.234.214:3478/udp, 207.154.234.214:80/udp, 207.154.234.214:80/tcp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [207.154.234.214:3478/tcp, 207.154.234.214:3478/udp, 207.154.234.214:80/udp, 207.154.234.214:80/tcp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [207.154.234.214:3478/tcp, 207.154.234.214:3478/udp, 207.154.234.214:80/udp, 207.154.234.214:80/tcp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [207.154.234.214:3478/tcp, 207.154.234.214:3478/udp, 207.154.234.214:80/udp, 207.154.234.214:80/tcp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [207.154.234.214:3478/tcp, 207.154.234.214:3478/udp, 207.154.234.214:80/udp, 207.154.234.214:80/tcp] (c.f.iceadapter.ice.PeerIceModule:177)
INFO Using all viable ice servers: [207.154.234.214:3478/tcp, 207.154.234.214:3478/udp, 207.154.234.214:80/udp, 207.154.234.214:80/tcp] (c.f.iceadapter.ice.PeerIceModule:177)

Askaholic · 18 Nov 2024, 21:49

Can’t say for sure but using a common web port like 80 or 443 is often an easy way to get through a network firewall since those ports are often allowed in order to allow web traffic. Could be because of that.