Humans vs Sorian/Uveso AIx - Game Crashes on AI ACU Kill since FAF Client 1.4.2

jchamlin · 3 Jun 2021, 00:36

We've been playing humans vs AI games, with mods, for a long time without issue. We usually take a long time to get out from under the assault of the AI and gather up a large enough attack force to go attack and kill the AI ACUs, so a lot of game time elapses between when we start and when we kill our first AI ACU. However, since about the time we updated to the FAF Client 1.4.2, when we kill an AI ACU, the game will crash for some (or all) players. It happens pretty reliably on the first AI ACU kill. The first time, 2 of the 4 players crashed, and the second time, 2 of 2 players crashed. We launch with the FAF Client lobby, and we use a few mods (and have for many years without issue):
AI-Uveso
BlackOps FAF: ACUs
BlackOps FAF: ExUnits
BlackOps FAF: Unleashed
Flying engineers - Slow Speed
Hive Engineering Stations for All
Ninja Reclaim Drones
Resources x2

The error we are seeing when it crashes:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x0067aa5f
attempted to read memory at 0x00000000

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe
Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_14001564.log /gpgnet 127.0.0.1:17552 /mean 671.853 /deviation 191.512 /savereplay gpgnet://127.0.0.1:5839/14001564/jchamlin.SCFAreplay /country US /clan CKA /numgames 34

Callstack:
Unknown symbol (address 0x0067aa5f)

I've attached a copy of the full game log from my computer:
game_14001564.log

Uveso · 6 Mar 2021, 04:30

Hello Jchamlin,

Is there a reason why you installed most of your mods in 2 different locations ?

'c:\users\jchamlin\documents\my games\gas powered games\supreme commander forged alliance\mods\ai-uveso', mounted as '/mods/ai-uveso/'
'c:\programdata\faforever\user\my games\gas powered games\supreme commander forged alliance\mods\ai-uveso', mounted as '/mods/ai-uveso/'

Beside this i see many BuilderConditionTest failures after an AI dies.
I will check this after the weekend.
Maybe one of the mods is changing the defeat function from the AI.

Dragun101 · 6 Mar 2021, 04:39

Uveso that is due then having clicked vault fallback location

jchamlin · 3 Jun 2021, 07:34

@Uveso thanks for looking into the problem I reported.

The dual mod locations is because I'm using the fallback location for the FAF Client so it stores them in C:\Users\jchamlin\Documents\My Games to match what my friends use. I wasn't using it at first, so there was copies of mods in the C:\ProgramData\FAForever\user directory as well, but I compared them with Beyond Compare folder compare and what was in there was binary identical to my Users directory. I just removed C:\ProgramData\FAForever\user to just make sure it is clean. I'm pretty sure this isn't the problem though, the other players I play with also all crash at the same moment. I don't have their game logs to upload though. Next time it happens, I'll try to gather the Error screen and game logs from all players.

Uveso · 3 Jun 2021, 19:17

Hello jchamlin,

we are aware of this crash since 2015, so no further logs are needed.
see:
https://github.com/FAForever/fa/issues/318
https://github.com/FAForever/fa/issues/3235
https://forum.faforever.com/topic/342/abort-program-error

Kionx stated to this error :

Crash occurred when trying to read the type of collision of the unit.
When trigger_g1pos is defeated, the ura0001 collision set to "None".
RevertCollisionShape has not been called.
Similar problems occur with other air units.

So we need to check your mods to see if there are some changed to the TransferUnfinishedUnitsAfterDeath function where SetCollisionShape() is called.

Uveso · 3 Aug 2021, 18:19

@jchamlin

I have now checked the endgame functions and they are not causing any crash.

I also tested these mods without any crashes:

AI-Uveso (62e2j64a-AIUV-0910-89465-146as555a8u3)
BlackOps FAF: ACUs (9e8ea941-c306-4751-b367-a00000018000)
BlackOps FAF: EXUnits (9e8ee941-c406-4751-b367-e10000018000)
BlackOps FAF: Unleashed (9e8ea941-c306-aaaf-b367-u00000018000)
Common Mod Tools (zcbf6277-24e3-437a-b968-Common-v1)
Ninja Reclaim Drones (FEC17324-7392-3652-0180-119A739294E0)

Please disable the following mods and do some test games:

Supreme Score Board (HUSSAR-PL-a1e2-c4t4-scfa-ssbmod-v0060)
NoShake (06E0D4F2-2231-11E3-A9FB-92EE6088709B)
Rescources x2 (d5c7af75-6944-490b-b647-47dc1efffdc8)
Flying engineers- Slow Speed (b5bcfd65-57fc-405e-83ab-a124eab3c27faa)
Hive Engineering Stations For All (755F8830-FD79-11E0-8691-CBAF4724019D)

jchamlin · 13 Mar 2021, 01:51

@Uveso thank you again for looking into this.

Is it possible for a UI mod to cause other player's games to crash? I'll run without UI mods for now just in case, but nobody else in my group uses them.

Also, we've been playing with a few non-standard game options. AI cheat/build multiplier at 1.2, Share Unit Cap at Death: Allies, and Share Conditions: Full Share (full sharing of resources and units at death). Full Share is something we turned on recently, could that possibly be the culprit? Sounds like the function TransferUnfinishedUnitsAfterDeath might be related to that option being turned on. Did you play/test with multiple humans and AIs and with that game option turned on?

We tried a few more games tonight and we played with all our normal mods, except we changed Share Conditions: Full Share back to the default which is Share Until Death. We didn't see any crashes in this game when we killed the AI ACUs. So, my guess is that the crashing has something to do with Share Conditions: Full Share.

Uveso · 14 Mar 2021, 14:57

Ui mods have no effect on other players game.

i will test the original TransferUnfinishedUnitsAfterDeath function for a while.
In case its working, we need to check every single mod you are using.

jchamlin · 18 Mar 2021, 15:55

@Uveso thank you again for looking into it.
Would you like to join us for one of our games against the AI? Would you being in a game when it crashes help identify the problem? We normally play in the evenings CDT (UTC-5).

Uveso · 19 Mar 2021, 17:21

I am not playing much online because i am running the game in a development environment.

This weekend is my "Girlfriend" weekend, so i will start with testing next week.

jchamlin · 23 Mar 2021, 02:32

@Uveso just confirmed that Full Share on with AIx: Uveso Rush and AIx: Uveso Adaptive, killing one enemy ACU does cause the game to crash. With Full Share off, it doesn't crash. So. Full Share seems to be the culprit.

Uveso · 23 Mar 2021, 08:38

@jchamlin

Thank you for testing and your report!

i will do my best to fix it this week.

Uveso · 24 Mar 2021, 22:33

well this bug is really nasty.

i am testing with 4 PCs since 8 hours and only got 1 crash.
But, i can reproduce it

At least i can say the crash is caused inside the function TransferUnfinishedUnitsAfterDeath
I added some debug lines to the function and i am now waiting for the next crash...

Debug fun:

INFO: * AI-Uveso: CRASHTRACE TransferUnfinishedUnitsAfterDeath START
INFO: * AI-Uveso: CRASHTRACE TransferUnfinishedUnitsAfterDeath unfinishedUnits count...
INFO: * AI-Uveso: CRASHTRACE 393 reclaim:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 400 GetUnitsInRect loop
INFO: * AI-Uveso: CRASHTRACE 400 u:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 400 GetUnitsInRect loop
INFO: * AI-Uveso: CRASHTRACE 400 u:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 400 GetUnitsInRect loop
INFO: * AI-Uveso: CRASHTRACE 400 u:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 400 GetUnitsInRect loop
INFO: * AI-Uveso: CRASHTRACE 400 u:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 400 GetUnitsInRect loop
INFO: * AI-Uveso: CRASHTRACE 400 u:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 400 GetUnitsInRect loop
INFO: * AI-Uveso: CRASHTRACE 400 u:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 400 GetUnitsInRect loop
INFO: * AI-Uveso: CRASHTRACE 400 u:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 410 unit:Destroy()
INFO: * AI-Uveso: CRASHTRACE 410 IssueBuildMobile
INFO: * AI-Uveso: CRASHTRACE 400 GetUnitsInRect loop
INFO: * AI-Uveso: CRASHTRACE 400 u:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 400 GetUnitsInRect loop
INFO: * AI-Uveso: CRASHTRACE 400 u:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 400 GetUnitsInRect loop
INFO: * AI-Uveso: CRASHTRACE 400 u:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 400 GetUnitsInRect loop
INFO: * AI-Uveso: CRASHTRACE 400 u:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 400 GetUnitsInRect loop
INFO: * AI-Uveso: CRASHTRACE 400 u:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 400 GetUnitsInRect loop
INFO: * AI-Uveso: CRASHTRACE 400 u:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 400 GetUnitsInRect loop
INFO: * AI-Uveso: CRASHTRACE 400 u:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 400 GetUnitsInRect loop
INFO: * AI-Uveso: CRASHTRACE 400 u:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 410 unit:Destroy()
INFO: * AI-Uveso: CRASHTRACE 410 IssueBuildMobile
INFO: * AI-Uveso: CRASHTRACE 412 WaitTicks(3)
INFO: * AI-Uveso: CRASHTRACE 413 WaitTicks(4)
INFO: * AI-Uveso: CRASHTRACE 414 builder in builders START
INFO: * AI-Uveso: CRASHTRACE 414 builder in builders (buildrate: 193881.796875 ... LOOP
INFO: * AI-Uveso: CRASHTRACE 414 builder in builders... END
INFO: * AI-Uveso: CRASHTRACE 414 builder in builders (buildrate: 795.31726074219 ... LOOP
INFO: * AI-Uveso: CRASHTRACE 414 builder in builders... END
INFO: * AI-Uveso: CRASHTRACE 415 WaitTicks(1)
INFO: * AI-Uveso: CRASHTRACE 415 builder:Destroy()
INFO: * AI-Uveso: CRASHTRACE 415 builder:Destroy()
INFO: * AI-Uveso: CRASHTRACE 416 builder finished
INFO: * AI-Uveso: CRASHTRACE TransferUnfinishedUnitsAfterDeath CreateWreckage...
INFO: * AI-Uveso: CRASHTRACE TransferUnfinishedUnitsAfterDeath SetCollisionShape...
INFO: * AI-Uveso: CRASHTRACE 514 wreck:SetCollisionShape
INFO: * AI-Uveso: CRASHTRACE 514 Shape Data: shape "Box" - centerx 0 - centery 0 - centerz 0 - sizex 0.40000000596046 - sizey 0.125 - sizez 0.69999998807907
INFO: * AI-Uveso: CRASHTRACE TransferUnfinishedUnitsAfterDeath RevertCollisionShape...
INFO: * AI-Uveso: CRASHTRACE TransferUnfinishedUnitsAfterDeath END 2

I just need one crash ^^

Uveso · 29 Mar 2021, 18:07

just for Info:

i got some crshes now, and sadly the function TransferUnfinishedUnitsAfterDeath is unstable.
It's all propper coded in LUA but at some point the c-engine will trigger an error.

So i disabled the function and i am now continuing testing fullshare.

I already included it as patch to my AI and will release it today or tomorrow.
We will lose the transfer of unfinished units to the new owner, but get hopefully a crash-free gameend.

Uveso · 31 Mar 2021, 09:14

AI-Uveso V92 is out and has a patch to exclude the TransferUnfinishedUnitsAfterDeath function.

Please test it, and report if the error at address 0x0067aa5f is gone.

Uveso · 2 Jun 2021, 21:16

Well, i tested the gameend without the function TransferUnfinishedUnitsAfterDeath() for 2 month now and got not a single crash.

Wow its working on your side @jchamlin ?

left_boy · 4 Jun 2021, 16:41

Client version has zero effect on the game