Private coturn server for my games.
-
Hello, hello everyone!
I don't know the FAF ecosystem very well. But why can't a central server be used for games? Why is the concept of "Coturn Servers" used with connections to each player instead of a single game server?
I am experienced Java Developer with 6+ years of Java-based enterprise application. I'm interested to know what problems prevent us from moving to a client-server architecture.Please take a look at the diagram below and indicate the problems with this implementation:
The diagram is in better quality: https://drive.google.com/file/d/17jnYO3a58oMZonGRF6Q7PKkCtLwqhsxq/view?usp=drive_link
-
The game itself does not require a server, so having a server is pointless. Unless you rewrite the game engine to use a client-server-architecture.
So the only useful thing that a server can do is forward connections, which is exactly what Coturn does.
Replace "Game Server" with "Coturn Server" in your picture and you have almost exactly what FAF is using for people that have direct connection problems.
-
The Forged Alliance engine will alway open one udp port per player in the game and send data multiple times.
In theory you could try to reverse engineer the binary network protocol and remux the streams that some intermediate software merges outgoing traffic into 1 connection and splits up incoming traffic into "per player" traffic.
The is actually a developer on Discord trying to do exactly that. I'm not sure if this really solves network connection issues. What I am sure is that it will cause latency issues for all users geographically far away from the central server.
2 players from australia don't take the direct route but sent over a gateway in europe adding 500ms+ latency.
We don't know much about the binary data protocol
-
@brutus5000 If we know so little about the data protocol, how do you get information about the game starting/ending? How do you get messages from the chat?
If it's not too much trouble, send the discord of the modder who is trying to do this )
-
Game using peer-to-peer (P2P) architecture where every player connected to each other.
Thats good for LAN party and terrible for modern networks with all those firewalls and VPNs.
When two players can't establish direct connection, coturn/stun server help them to find the route using some smart tricks.
If you experience connectivity problems consider buying static IP address from your ISP.As for single game server, that should improve connectivity, theoretically.
For example in 8v8 the total connection count is 49 (7 per player), with single server its only 7 (1 per player).
Chance of connection lost is much lower with less connection count.
But again, if host act as a server and unlucky with ISP all that doesn't matter.If server running by FAF org or privately another problem arise:
- hardware and traffic cost
- server require maitance
- increased latency depending on player location (nogo for Oceanic players)
I made few tests but noting real, the code is available on github
-
@iamfromrussia said in Private coturn server for my games.:
@brutus5000 If we know so little about the data protocol, how do you get information about the game starting/ending? How do you get messages from the chat?
Because there are different datastreams
There is a more "high-level" protocol where the game gives information about the game state. This is called GPGNet protocol and it's only sent between Game <-> FAF Client <-> FAF lobby server. This is well known.
Then we have the game datastream (which is basically the replay format). This is except for few unknown bits well known by now.
The part you are asking for is the network metadata between the UDP connections of the games and this was never in the focus. If you look at @Surfer's git repository, he reverse engineered a few pieces. But that is basically new work from the last few months.
If it's not too much trouble, send the discord of the modder who is trying to do this )
Its @Surfer or anykey111 in Discord.
-
@brutus5000 said in Private coturn server for my games.:
In theory you could try to reverse engineer the binary network protocol and remux the streams that some intermediate software merges outgoing traffic into 1 connection and splits up incoming traffic into "per player" traffic.
@Surfer @Brutus5000 Why do we need to reverse engineer the binary protocol? Do we need to know what's inside the binary data stream? Why can't we work with binary stream packets like black boxes? Just passing them on without understanding what's inside.
-
@iamfromrussia You don't need deep protocol knowledge for such task.
See the patch
fa-mp-test/blob/main/0001-add-proxy-server-draft.patchthat completely remove P2P exchange from the ice-adapter and use well known server (main.c)Please, state cleary what is the point?
If you need private server, just host multiplayer game and tell firends your IP, they will join and thats it. -
@surfer said in Private coturn server for my games.:
Please, state cleary what is the point?
We usually play in a small group of 4-5 people. When I say: "Let's try to play 8vs8." I usually get the answer: "We'll wait a long time for the game, and then we'll still leave because of the lags."
If I could say something like: "I have my own private server and there will be no lag" That would be a strong argument.If FAF has the ability to register your own server in the client through which traffic would pass instead of a P2P connection.
If a person who has registered his own relay server creates a game, then everyone who connects to this game will redirect their traffic to the relay server.
If this is a high-quality VPS with a static IP, then the connection will be more stable.
As an option, you can send an excessive number of UDP packets to the server, for example, send two duplicates for each packet, which will increase the probability of reception on the player's side and remove duplicates in ICE. But here I am not sure about the possibility of implementing this. https://gafferongames.com/post/deterministic_lockstep/ (Yes, this is much more complicated than a simple Relay server)
This is a very brief description of the feature, so please don't beat me up XD
-
You assume that running your own stable coturn solves all connection issues. Our reports from ICE adapters tell otherwise.
The problem is not necessarily unstable coturns, but with issues making a connection even with coturn. -
Keep in mind with your own coturn server all 16 players should agree beforehand to use same settings (FAF Client -> "Preferred coturns for ICE").
With centralized game server they must also use custom FAF client modification. -
@surfer said in Private coturn server for my games.:
As for single game server, that should improve connectivity, theoretically.
For example in 8v8 the total connection count is 49 (7 per player), with single server its only 7 (1 per player).I still can't understand, does it make sense to combine binary traffic? If it's such a difficult task, is it worth spending time on? I'm not an expert in network programming, but isn't UDP a connectionless protocol https://docs.oracle.com/cd/E19620-01/805-4041/6j3r8iu2f/index.html ?
Is there a difference whether the client will receive information on one port, on 16 or on 16,000 ports that the server listens on in case of UDP connection? -
FAF has been operational for over a decade, and has yet to establish reliable consistent connections between players.
You rarely see this issue occur in steam, and the best part in steam is that, if someone does disconnect, you can simply wait for them to reconnect. 9/10 times they successfully reconnect too. I remember when you could do this in faf many moons ago, but sadly this is no longer the case. And you can't reconnect in the client either to try salvage it.
Idk what the problem is with faf and players not being able to reliably connect to eachother in game. It's very painful going through 32 minutes of video gameplay only to have some random player randomly disconnect for no apparent reason.
I doubt faf will ever fully resolve this issue, I don't think they even know how.
-
@yew said in Private coturn server for my games.:
I doubt faf will ever fully resolve this issue, I don't think they even know how.
This is correct. We don't know. None of us are professional game and/or network engineers. We can not afford to pay for a company to fix it for us. And making ourselves dependent of Steam is also not an option (even all the legal issues aside).
-
@iamfromrussia said in Private coturn server for my games.:
I still can't understand, does it make sense to combine binary traffic? If it's such a difficult task, is it worth spending time on? I'm not an expert in network programming, but isn't UDP a connectionless protocol https://docs.oracle.com/cd/E19620-01/805-4041/6j3r8iu2f/index.html ?
Is there a difference whether the client will receive information on one port, on 16 or on 16,000 ports that the server listens on in case of UDP connection?Just for optimization to reduce the total number of connections and the numer of individual packets flying through. Every UDP packet is independent, for example 8v8, its about 30 packets on every tick (lets say ~250 milliseconds). The total number of packets is 30164=1920 per second, for each player server should open 15 ports, 15*16=240.
For now the biggest problem for me is testing, i need atleast 3 players. using my main PC as host, remote desktop on another PC and old PC under the table.
-
If you need help testing feel free to ping me. I work from home so I'm available most of the day anyway.
-
Ok, now we can mythbust the proxy server approach.
I changed the faf-ice-adapter to use a single TCP socket for all communications.
faf-ice-adapter repo
proxy-test repoAfter serveral mock tests i didn't see any difference. On weekend i plan to rent a basic VPS and run test in the wild.
-
In mock tests ICE also always works perfectly
Nobody ever questioned whether one large plain proxy would work. The ICE adapter laid foundations for rerouting the game traffic anywhere.
However it has drawbacks in operational costs and in latency.
And in theory plain ICE connections should always work (with relay as a fallback which basically is single-connection proxy). As such in theory it is also superior to a proxy-solution that does not traffic deduplication.
But in practice (especially Windows and/or security software interference) and for the given software libraries available we see, that it does not hold up the promises.However under current DDOS situation a proxy server becomes more interesting if it were tunneled through cloudflare websockets...
-
I have first feedback from 2v1 game, testers say it didn't feel better.
How to join testing?
- backup your
FAF Client/natives/faf-ice-adapter.jar - replace with https://github.com/user-attachments/files/17532274/faf-ice-adapter.zip or build from the source (https://github.com/anykey111/faf-star-adapter)
- host game, invite firends (they must use the same version of faf-ice-adapter.jar as you're using)
Why should i participate?
For the sciene, even if we know that this solution is inferior there is no proofs and people keep returning to this idea.
- backup your
-
- Can I see in the logs/debugger which player is in my lobby and has coturn server connections?
- Can I prevent this by not allowing coturn server connections (so, I could have the IPs and block them as hosts in my firewall, then no more players can get into my lobby - there are only 4 Germany 1 / Finland / USA East / Global of them - or are there several and only the key regions are listed)?
- When you switch from the lobby to the actual game, are all connections renegotiated, right or wrong? And what happens in the lobby is just a rudimentary check to see if it works at all with the connections?
- What about the option “Allow the ICE adapter to use IPv6”? It says that it causes connection problems for some players, but what advantage does it have in general (more stable connections when a connection is established?)?
- What actually causes the lag in a DDoS when players are in the game, if it's just P2P connections? If players are connected to coturn servers and these are then attacked (the coturn servers - are they attacked at all?), or are there other services that limit the game (live replay?)? Does the player still have connections to FaF service servers during the game, which can cause lag (in-game behind issues)?
