Open discussion on testing and rollout of the new ice adapter
-
For quite a while we have been working on a new ice adapter implementation (server side, client side and the adapter itself which is now called faf-pioneer).
It has a few pros and cons compared to the old ice-adapter solution:
Pros
- Auto-update in the FAF client. So far we needed to rollout a new client to make a version change. Updates are now possible without a client update, just by releasing a new version on Github.
- It no longer requires a lobby server for ICE candidate exchange. We can run updates on the lobby server or the FAF client connection (or the client itself) might crash and it should still be able to reconnect.
- It supports Cloudflare TURN services, which should make connections more stable and hopefully less expensive than on our current provider Xirsys.
- (Unless you opt out in client settings) You share your connection logs centrally with FAF servers, so we have the accumulated logs of all players of a single game to debug issues.
Cons
- No more debug connection window. The builtin UI is vanished and we also have no alternative WebUI so far. A reimplementation would take too much effort right now.
- Probably more bandwidth required. We use the WebRTC protocol now, which does a lot of work for us, but it enforces encryption. According to the old ice adapter developer this idea was dismissed in 2017 because it required more bandwith (in worst case doubling it). But now its 2025 and even mobile connections offer more bandwith. And better slow connectivity than no connectivity I guess.
So what is the state?
It is now in a state that games are stable on 1v1 and we need to get more people on board to broaden the test experience.There is a big problem however for both testing an final release. It's not backwards compatible. We deploy a new FAF client and people on that version can no longer join old client version games or vice versa. So testing only works if all players use the modified client version. And for that time they should probably use password protected games, and TMM should be avoided completely.
So once we go live, there is no turning back really. If it makes problem for some players, they cannot downgrade. They can't play anymore until the issue is fixed.
I know that this is undesirable, but we evaluated options and there basically no way to offer a fallback strategy and/or backwards compatibility.
.
So before I do anything big, I am asking you, the community, for feedback. How much time do we want to spend on testing? Who is willing to test? How much risk do we want to take?
-
Put me down for testing.
Is testing done on the test server or the main server?
-
My opinion is: being unable to play FAF would be very annoying.
With that said, i'm not sure i understood this correctly (i think i missread this the first time). Will people be able to choose wether to opt into the new patch (the one for testing) or everyone will be forced in? If only testers are "sacrificed" then i think this will be mostly fine (and you should ignore my next paragraph), else, here i go with the considerations.
I would be willing to act as a guinea pig for the testing, but i would suggest to minimize, if possible, the potential negative impact of the tests:
- is there a way to drop the update on some separate instance like some "closed circuit" FAF beta? (this is me being optimistic, i'm guessing there isn't since you didn't say anything about it)
- if the only way is big mandatory update on the main client and shutting down normal games and people happiness in favor of essential testing, i suggest we make some schedules to have test windows that won't make people hate life more than they already do. Something like a week every 3 or so, or maybe we might test on the first half of the week and patch back on thursday or something (don't know if patching back and forth so much is a problem). The majority of games is in the weekend anyway so maybe the second option might lessen the discontent while giving frequent testing (or lower the testing personnel
).
What did you have in mind @Brutus5000 ? How many hours were you hoping to be dedicated to this, and how (low or high frequency, short or long duration for each instance)? Would a list of volunteers and their weekly availability help?
(Also, stuff like this makes me wish we could involve more of the community to get their feedback, because i know it will be at best the usual 200 or so guys who will see this post, BUT HOW TO DO IT? T_T)
Oh and btw, amazing job, didn't expect to see a prototype this quickly
Edit: i think i heavily misinterpreted some stuff from brutus initial post and reworked a bit for clarity
-
I also don't really get what testing means in this case. Can I still switch back to the normal server and play with everyone else or how does this exactly work?
-
Brutus, I would say test for a few weeks to see the outcome. I understand once released as mentioned there is no backwards compatibility. If it were to be released say tomorrow, we probably wouldn't know the problems it would cause. Perhaps a select pool of people should participate until it is clear if it will not cause further problems. Otherwise, it could turn into a disaster.
-
I haven't been playing much lately, so I'm down to potentially brick my game for the sake of testing a new adapter. Should we schedule a day & time for people to get on and test?
-
The sooner we start to test it, the faster we can identify and resolve any bugs. Even if there are some short-term issues, it is absolutely worth in the long run when we get a more stable connection system. Count me in for helping to test it out.
-
So you need a new test-client version. That only works with other test clients. You can go back to the normal client (but I don't know if you can install them in parallel or need to uninstall and reinstall each time).
You can play on the production server.
But a test-client can only connect to a test client and a normal client can only connect to a normal client. That's why you should only play custom games, in the matchmaker you will be matched with people using the normal client.If you encounter issues you describe the problem and tell us the game id and we can look at all the logs.
At some point we have to decide to roll it out for everyone. At this point there is no turning back for single users where it doesn't work (in worst case we need to do a global rollback to the old version)
-
@Brutus5000 said in Open discussion on testing and rollout of the new ice adapter:
So you need a new test-client version. That only works with other test clients. You can go back to the normal client (but I don't know if you can install them in parallel or need to uninstall and reinstall each time).
People should be able to have two versions of the client installed at the same time.
The only part that is really shared that might get wonky is the client prefs between them.
-
Drop announcement in the discord with testing date and we should get enough people to stress test it. Hell, hit up @Strydxr or @Razana and we might even get CNN hosted on the new client for even better test.
I also don't think that the bandwidth requirements should matter at all, unless it were to really baloon. We are no longer in copper era of internet, so I would assume everyone would have that 1-2Mb/s of upload available for themselves. And considering how bandwidth intensive FAF is, I really don't think that doubling it will matter at all.
And if available I will also show up for some testing games.
-
@TheVVheelboy said in Open discussion on testing and rollout of the new ice adapter:
Drop announcement in the discord with testing date and we should get enough people to stress test it. Hell, hit up @Strydxr or @Razana and we might even get CNN hosted on the new client for even better test.
I also don't think that the bandwidth requirements should matter at all, unless it were to really baloon. We are no longer in copper era of internet, so I would assume everyone would have that 1-2Mb/s of upload available for themselves. And considering how bandwidth intensive FAF is, I really don't think that doubling it will matter at all.
And if available I will also show up for some testing games.
Sounds good, if details are confirmed for this testing please forward them to me so i can get the Trainers ready to discuss a possible test CGN on top of the normal scheduled ones!
-
@Brutus5000 Consider adding an idiot proof notification system of some kind. People who find they can't join games may not be aware of Discord, these forums, etc.
-
Is this supposed to help against DDoS?
-
@Kilatamoro said in Open discussion on testing and rollout of the new ice adapter:
Is this supposed to help against DDoS?
Yes and no.
Due to DDoS we can no longer run our own TURN servers, so we have to use 3rd party ones. Our current provider Xirsys seems to work, but people have still connectivity issues and it could be related due to issues in our current ice adapter. We know this, because it doesn't work together with Cloudflare TURN service at all.With the new solution we can use Cloudflare, which is also cheaper than Xirsys.
So it does not directly impacts DDoS related problem, but it is related to it.
-
@Brutus5000 said in Open discussion on testing and rollout of the new ice adapter:
No more debug connection window.
There is an old trick serving a minimal html page from the application.
Setup setInterval(1000,function(){location.reload()}) in the page header so the page refresh itself after timeout.
