Desync on all replays and games with Windows 11 Arm on M1 Max Apple

malu5531 · 1 Nov 2021, 22:11

Hi,

I'm running SupCom FA & FAF on Windows 11 ARM on Apple MacbookPro M1 Max in Parallels 17.1. The game runs good (298 cpu score, ~90 fps), but I've noticed that all replays I've tried desync the first second after game starts. This happens also if I join a PVP game. The full game can be viewed on FAF replay, I'm user "malu5531": https://replay.faforever.com/15627000 (needles to say, the game probably looks quite different when others view it compared to when I played it).

I've tried to adjust emulation settings, parallel settings, etc, but not closer to a solution. I'm running Windows 11 Arm release 22489 with FAF Arm emulation setting set to "Safe Emulation".

I've posted a video of the game running on youtube:
https://www.youtube.com/watch?v=ukKOy20BwBk

I attach client.log and game_11612043.log with details on the error when I try to view the following replay: https://replay.faforever.com/11612043

I guess the issue may be some patch / mod not loading correctly since the desync happens immediately. Any ideas what I can try to fix this and/or is there any more information or detailed logs needed for troubleshooting?

Attachments: game_11612043.log client.log

JazzFunkNoob · 2 Nov 2021, 14:22

If you are lucky this is a fixable issue. But I remember the original lead engineer saying in an interview that they had some issues with some hardware not being deterministic with certain operations (hardware accelerated sine/cosine calculation was the issue if I remember correctly). If the emulator is partially non deterministic then there is nothing FAF can do.

Uveso · 2 Nov 2021, 23:00

@malu5531,

the game has a checksum that is calculated based on the Sim state.
If this checksum is not equal to host/client/replay then you will get a desync.

That means your game does not execute the same commands or has not the same data then other games.
This happens when you have different game or modfiles than your host/client or replay.

Have in mind there can be several mod folders.
Mods can also be installed as *.SCD archive inside the gamedata folder.

After viewing your log, i would suggest:

Go to steam and make a filecheck to validate your base game files.
go to C:\ProgramData\FAForever\gamedata\ and delete everything inside the gamedata folder.
Update the FAF client and downlaod the latest FAF game version (host a game alone to update the files)

Then try again.
(without mods please)

Corsaire · 19 Dec 2021, 12:45

I confirm the problem too, game is impossible to play without having the error message popping up.

It seems to run fine if you ignore it but it may crash in the long term, I didn't push very far.
I don't think the file versions or mod installed have any influence on this matter.

Corsaire · 20 Dec 2021, 01:20

But I remember the original lead engineer saying in an interview that they had some issues with some hardware not being deterministic with certain operations (hardware accelerated sine/cosine calculation was the issue if I remember correctly). If the emulator is partially non deterministic then there is nothing FAF can do.

This page mentions this https://www.applegamingwiki.com/wiki/Supreme_Commander_Forged_Alliance
"Multiplayer desync may be due to slight different values for hardware sine/cosine calculations causing checksum differences. "

Would be nice to have some more infos on this, pretty please?

Corsaire · 20 Dec 2021, 12:55

I have made a report to Parallels on the matter, their answer is interesting:

"Please contact Microsoft support as we don't have any expertise as the issue you have reported is not directly related to Parallels Desktop for Mac.

This is a Windows OS issue which is also possible to occur on real Windows PCs. You may either contact your local administrator or request assistance from the OS vendor directly. We advise you to check with the OS vendor/other 3rd parties providing support for Windows OS issues in advance if they charge any additional expenses for their support services. Unfortunately, Parallels will not be able to provide you with further support for this out-of-scope issue."

It brings the hint that the issue is in fact in Windows11 (for arm64 or also in x64 version ?).

JazzFunkNoob · 20 Dec 2021, 23:02

@corsaire apparently I am quoted now for super low level debugging B).
If it is indeed due to some non deterministic part of the emulation microsoft might be able to fix it. But good luck getting them to trouble shoot it.

Corsaire · 21 Dec 2021, 13:39

@harzernoob said in Desync on all replays and games with Windows 11 Arm on M1 Max Apple:

@corsaire apparently I am quoted now for super low level debugging B).
If it is indeed due to some non deterministic part of the emulation microsoft might be able to fix it. But good luck getting them to trouble shoot it.

Well I just did that.
Chances are slim yeah, but hey, no harm in pointing out the discrepancy between x86-64 and emulated on arm64... Maybe with a bit of luck someone at Redmond will be interested.

One can hope

enc0re · 19 Mar 2022, 23:10

Sorry to resurrect an older thread, but I'm wondering if anyone's had or heard of any progress in this front? I just got FA on a crazy discount on steam because I used to love this game... and I'd love to get it running smoothly if possible!

malu5531 · 12 Jun 2022, 12:11

Tried again now with latest parallels and windows updates:

Edition Windows 11 Pro Insider Preview
Version 22H2
Installed on ‎6/‎12/‎2022
OS build 25136.1000
Experience Windows Feature Experience Pack 1000.25136.1000.0

Same issue still, i.e. immediate desync on replay playback (but the replay continues fine). Single player works.

Maybe with Metal 3 with MacOS Ventura and new DX12 support in Parallels later this fall it will be improved? Will try again in a few months.

Geosearchef · 7 Jan 2022, 20:55

Just to add my two cents to this issue, this sounds a lot like you're running into a numerical issue. This basically means your system makes an error during the calculation. This is completely normal and taken into account by most software.

The IEEE754 standard defines the minimum precision for floating point calculations. It doesn't define a maximum precision. Therefore, different CPUs may produce slightly different results for the same computation / code. Adding numerical instability, those errors can get large, fast. FA deals with this using strict / stable floating point calculations. (I think they don't do floating point, they do fixed point math)

The issue now arises when there is some aspect of the way your non-x86 CPU works that wasn't anticipated by the developers of FA (e.g. when computing a sine/cosine as mentioned above), that causes your system to make a different calculation error than everyone else. Then you get a different result and a desync.

If those speculations are correct, that would make it nearly impossible to run FA on a non x86 chip to produce the same deterministic results as on an x86 chip.

Nex · 1 Jul 2022, 23:04

@geosearchef Well if your Arm PC emulates a X86 (perfectly ofc.), then you could do it.

But yeah, playing a synchronized simulation on two different architectures would need special software to make sure the two softwares generate the same output.
Since FA developers didn't expect people to run it on arm, you would need to implement that step your self (in the form of an X86 emulator for arm).

Geosearchef · 7 Mar 2022, 20:29

@nex Exactly, you would need an emulator. The M1 is not powerful enough to emulate FA though.

Corsaire · 14 Aug 2022, 14:05

@geosearchef Not sure what you're talking about here, you can perfectly play solo campaign, so yeah the M1 is perfectly fine. Problem is to do a multiplayer game or run a replay.

Not sure if there will ever be a way to mitigate that calculation discrepancy.

Corsaire · 14 Aug 2022, 14:08

On that note, did someone had the possibility to try running 2 FAF setups on M1 in a multiplayer game ?

We do know that between a Windows PC and a windows with emulation on M1 it does not work... but how about 2 identical emulated setups ?

p4block · 14 Aug 2022, 16:12

In addition to FPU shenanigans, lightly quoting Marcan, the cores in the M1 speculate execution extremely hard, and that is making pre-existing unknown bugs surface. It's possible FA itself is broken in a way that never shows up in Intel/AMD cpus, and if that were the case it's probably pretty much unfixable.

It would also mean that the game would likely stop working at some future point when x86 cpus get more complex and trigger the bug. (you can talk about retro compatibility being x86's only reason to exist though and argue that last point may never happen)

Nex · 14 Aug 2022, 22:29

@corsaire said in Desync on all replays and games with Windows 11 Arm on M1 Max Apple:

Not sure what you're talking about here, you can perfectly play solo campaign,

Yes, what he/we meant, was that you would need to perfectly emulate an X86 to play with other X86.
Singleplayer and playing with other M1 should/might be fine?

But i also heard from someone on discord, that he successfully plays ladder with an M1, but gets desyncs in lobbies, which is quite strange.

Eyvind · 7 Apr 2023, 20:34

I've checked some sources. In https://gafferongames.com/post/floating_point_determinism/, a developer of SupCom confirms that they already use the IEEE754 float standard to avoid desync problems with different x86 processors (AMD and Intel). They use the _controlfp command to determine how floating-point calculations work. The problem is that on ARM Windows, _controlfp has a slightly different effect - it affects a different set of registers (https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/control87-controlfp-control87-2?view=msvc-170).
This is only a guess, but it seems like we can just rewrite some code and change the used float registers.
I'm not sure that this will solve everything, but it looks like a good first attempt.

Jip · 7 Apr 2023, 21:05

@Eyvind that is an interesting find , I'm not sure how trivial it would be to fix what you describe. We don't have access to the source code of the engine, for example

malu5531 · 5 May 2024, 21:26

@Eyvind Great find! Even without source code, perhaps it could be possible patch? (or if Microsoft needs to do something?)

It would be interesting to compare with Qualcomm ARM based Windows 11 laptop, if anyone have access to one of those?