Team Lead group

Nomander

@JaggedAppliance said:

Yes these desyncs and crashes affected replays as well as live games. Actually they were really common in replays which helped a lot with testing.

The reason why this fix was so difficult is because it's a memory corruption writing into random space. With the way your computer set up the game's memory it caused regular desyncs and crashes in replays. For a lot of other people it rarely did anything. So for Brutus's idea, this specific bug would be caught by luck in a simulation CI.

Here's how I see potential improvements compared to my experience:

Desync tracking would automatically and seriously tell devs of deysncs. I only noticed desyncs from constantly checking forums and chats, and eventually I had to go through a specific replay search to even have any data about desyncs, and even then there wasn't a really big response. Automation would've been a more serious alert and it could've even caught it much earlier on.
The reason this specific patch wasn't caught early was because there was 0 testing in prod. It's a tricky bug, so fair enough that some testing in dev wouldn't catch it, but since we released it without any feature using it, it took all the way up until the painting feature for its effects to show, which made the change that introduced the bug disconnected from the updates exhibiting the bug.
With AI we can do much better reviews. This bug was a very human error that is easily picked up by AI or I guess some linter. I don't think we do very complex patches most of the time, so static analysis should work quite easily. In this sense I think we could actually speed up engine patches because of their more reliable code, but of course it's best if we just have the backup testing in prod with the two points above.

Giebmasse

Welcome back Jip, you know you want to tinker with the engine a bit

hybrid_

General Information

Dates:
Signups Close: 14th of November, 14 UTC
Drafting From: 14th of November, 15 UTC until 20th of November
Tournament Start: 21st of November, 14 UTC
Bracket: https://tournaments.faforever.com/t/6c60000c42
Chat: KoS OR PoS, FAF Tournaments website, And the appropriate discord channel
Live coverage:
Official: https://www.twitch.tv/faflive
Tournament Organiser: hybrid_
Tournament Director: TBA

Due to chat bugs and network issues, all official communication on the days of the tournament will be via discord in the appropriate text channel in the #tournaments section. Additional communication may take place on the tournament website.

Captains Draft

Kings Draft

This tournament will be your typical captains draft, with the 6-8 highest rated players being captains for the King's side. If you have been selected as a captain, please make sure we are able to reach out to you on discord.

Princes Draft

Once the players in the Kings bracket have been selected, we will proceed with the Princes Draft. Once again the 6-8 highest rated remaining players will become captains, and will start the pick process for the participants that weren't picked.

Draft Information

The draft will take place on the 14th of November, 15 UTC until the 20th of November. It will take place in the #KingofSetons channel on the Official FAF Discord in the Tournaments section.

I would prefer to do it in one go and if captains would like to arrange a call date so that we could potentially stream the draft that would be great, I will contact all of the captains to see if this is possible once they have been selected.

Captains may pick any player, and no player is able to reject being drafted. If the player does not wish to play with who they have been drafted with, they will withdraw from the tournament.

The draft orders will be determined once the signups have closed.

Format

A simple double elimination bracket or swiss with a final, with all the games being BO3. There will be 6-8 teams in each bracket.

Tournament Specific Rules

Players may not be used in the same slot consecutively.

Mappool

Seton's Clutch - scmp_009

Rules

The official FAF Tournament Rules apply.
Some of the most important of these rules are:

Disconnects within 5 minutes may result in a rehost. The Tournament Director has the last say afterwards.
Players will replay the same map if they draw.
If the incorrect map is hosted, the correct map will be hosted and the incorrect map will not count towards the score
Players who do not appear without valid reasoning will be given a warning that may result in being temporarily suspended from tournaments.
If a player does not show up in Round 1, they may be replaced. For all subsequent rounds, it will be treated as a forfeit and they will be eliminated and not replaced.
All participants will be allowed a 15-minute wait at the start of a tournament to show up however after this 15 minute wait the Tournament Direct has the right to disqualify you from the tournament. In the case where you will be late but do intend to arrive you need to message the Tournament Director preferably via Discord to inform them you will be late but will show up. When doing this please provide an approximate time delay until you arrive, it will then be the Tournament Directors decision whether than delay is too long or acceptable. If your team loses a member, you may find a substitution within the rank requirements.
The Tournament Director's decision is final.
A "substitute" is a person who replaces a member of a team participating in a team tournament. The substitute may only replace a single person, and the replaced member is considered withdrawn from the tournament.
A person may only participate as a substitute in case they were/are not already an active participant of the tournament - either organizer, participating player, a substitute, or a replaced player.
The replaced player, relinquishes any and all prize eligibility to the substitute but only once the substitute has won at least 1 series.
Please make sure all lobbies are passworded and inform your opponents the password and casters if applicable.
You can read the rest here: https://forum.faforever.com/assets/uploads/files/1685891393985-official-faf-tournament-rules.pdf

Prizes

Kings

1st: King of Setons Avatar + $200
2nd: Faction Face Avatar + $100
3rd: Faction Logo Avatar

Princes

1st: Prince of Setons Avatar + $150
2nd: Faction Face Avatar + $50
3rd: Faction Logo Avatar

The previous winners will receive a "Tournament Winner" avatar once the tournament concludes.

Prizes are from the FAF Tournament fund.

If you would like to donate towards the tournament and receive a cool avatar, please contact the Tournament Team Lead on discord (hyb/tbounce).

Reminder that all payments of official FAF fund prizes are subject to the official FAF tournament payout guidelines.
https://forum.faforever.com/topic/7161/faf-official-tournament-payout-guidelines

Signup

To sign up, use the link below, login and register on the tournament page.

Tournament Page: https://tournaments.faforever.com/t/6c60000c42

hybrid_

Qualifier #1 Pool has been released. You can find it on the website.

Brutus5000

@blackyps Would any of these desync issues appear in a replay out did it only happen in live games? Can we build a CI-pipeline on old repos? @draiget does this for the reverse engineering development somehow.

We now have desync tracking in the lobby server, so this could be used as alarm tracking after patches. But that would basically require more releases / smaller changes per patch.

Giebmasse

Rewrites and RE's ofcourse interesting and probably relevant soon.

Until then:
Some years ago we had a "forced" testing period for more significant changes (maybe it was ICE). For instance test thing X for 3 days, more activity on feedback gathering/analytics, then revert and digest results and decide from that. New releases are kinda that, but this would maybe emphasize more knowingly it is a test and data gathering for the decision making later on.

Another avenue which has been used is the fafdevelop, if a binary patch could used that and more users pushed/incentivized to test that and report feedback, or again analytics in place.

These could always be coordinated with promo team to try pushing people to test and report specific things we are on the lookout.

BlackYps

Let's start with the good news first: Due to relentless work by Nomander, supported through testing by JaggedAppliance, they managed to pinpoint the cause of the desync issues that plagued FAF recently. As I suspected, it was an engine patch, which brings me to the main topic I want to talk about: How do we deal with engine patches?

We are in a peculiar position, because the people doing engine patches and opening Pull Requests for them are frustrated that getting them actually deployed in the game takes so long. On the other hand, these changes are extremely hard to review, because they look like this:
´´´
0x007A7858:
jmp asm__ShouldAddUnit
nop

0x007EF12B:
push esp
mov ecx, esi
call RenderRange__Moho__UserUnit__UserUnit
nop

0x005BF011:
mov ecx, edi
mov edx, eax
call Hooked_SyncVisionRange
jmp 0x005BF02C
´´´
Good luck checking the code for logical errors. And as we can see with the latest debacle, an engine patch can work on the surface. When you test it, it does what it is supposed to do, but it can bite us later when random desyncs pop up because the patch also can corrupt memory under special conditions.
So for me as a team lead, merging in a barely tested engine patch feels like chugging an improvised bomb into my basement for storage. Probably nothing bad happens, but it definitely doesn't feel comfortable.

Recently we got a lot more engine patches by new contributors, which is nice in principle, but it makes the problem even bigger, because with the old contributors I at least know that they have some experience. With the new guys it could be some AI-assisted reverse engineering which can be a good or a bad thing, but I can't tell at all how diligenty they worked on that.

And the thing that annoys me the most is that I as a team lead am supposed to provide a good contribution environment, where people don't feel bogged down by bureaucracy and long approval processes. But if things go south, and nobody knows where the mysterious bugs that have appeared originated from, then I feel like all eyes go to the team lead to fix it. And frankly, I would rather stop storing IEDs in my basement than constantly rebuild the room. In this case, thankfully Nomander took the initiative, but he is also the Balance Team Lead since recently, so once again it is a team lead investing time cleaning up the mess other people made.

I personally think that it is not a good look on FAForever as a whole, if the gaming experience is tainted by random desyncs and crashes. And I think we need to change something about how we apporach this whole engine patch topic. I realize that the position of "just stop it entirely" is too extreme, but just carring on is also unbearable. God, this whole thing makes me angry, not at anyone in particular, but at this whole situation. We can do better than this. We need to do better.

I am looking forward to constructive suggestions.

Giebmasse

Awesome work!

Brutus5000

We had rented a global 3rd party TURN relay service. It got maliciously abused and we had to disable. The costs went from 100$ to 600$ per month.

Brutus5000

Since you referenced my pull request: it was never done to be able to actually play with it. I just need it to run the client and test FAF things while I mostly use a macbook

Team Lead

Posts