Replays of Games Stuck in Infinite Loop

Jip

After an initial investigation I think it is the same bug that we experienced (and fixed) previously, but now occurring somewhere else too. I have not yet found a fix. I'll need to keep investigating.

Please keep the replays coming, especially hoping for one that shows the freeze a little sooner

Rama

I've been able to find replays that freeze by comparing the Game Time versus the actual Play Time. Replays that show a Game Time that is around 20 mins to 30 mins, but have a Play Time closer to an hour are usually a replay that has the freeze.

Rama

This replay is shorter, freezes at 21 mins and 34 sec. One player desyncs at start.

https://replay.faforever.com/23060971

So far, it seems all the replays I've come across with freezes are when players are using big unit packs, like Total Mayhem.

Jip

@Rama the last replay you shared with me desyncs by default. Is it representative of what actually happened?

Jip

@Rama I've made a small pull request that definitively fixes the previous issue that caused simulation freezes, see also #6398.

I'll update FAF Develop shortly. Best to encourage people to play on FAF Develop, see if the issue still occurs.

Rama

Great work, Jip! I'll have to add an over-powered unit named after you, that levels peoples' bases. Then they can yell at you about it!

I haven't had much time for hosting, so don't know when I'll be able to try. Will let other players know when I see them. Do the vanilla units not use "SetBallisticAcceleration"? I've only seen the freeze with playthroughs with unit packs.

LightWire

Thanks for the research and work put in.

Been testing with FAF Develop solo with M28AI. Found 1 situation where no Game freeze, and 2 with game freeze.

All game have the exact same settings and set up.

Freeze (2 games) - Clues: happens at the end, maybe due to something spawning in near the end game.. Maybe when dooms are spawning. Always happens where there seem to be a lot of units on the map, seems sudden and unexpected, but there doesn't seem to be anything in particular that is noticeably different about the game around the time it occurs.
https://replay.faforever.com/23107974
https://replay.faforever.com/23107054

Fully Completed Game, No Freeze happened -
https://replay.faforever.com/23107205

Main Mods:
AI Wave Survival
M28AI
Total Mayhem Light
BlackOps units and Commanders

LightWire

Posting a replay where the game freezes at a point when there isn't a lot going on on the map.Thought this might make it easier to spot something. The point of the freeze happens when there's basically no air, only after Boss Doom Colossus: Stand Tall death and it starts spawning units.

1 Human Aeon, 2 Ally Aeon (M28AI), 1 AI Wave Survival

https://replay.faforever.com/23108256

LightWire

Replay of a game where the Freeze restarted after 1 player dropped.

AI wave survival. 2 human players. Game froze near the end of the game. Purple player disconnected and then the game continued.

https://replay.faforever.com/23136485

LightWire

Been testing, and it seems that the freeze is happening a lot less frequently. Many games are operating unexpectedly

This game is a recent one that froze. I think I enabled the enhanced FAF logger recently based on the responses from another thread, but not sure if that helps.

https://replay.faforever.com/23141485

LightWire

Posting another example, where the game restarts after a freeze, and some people leave the game.

https://replay.faforever.com/23162621

Rama

Here is a replay using Blackops and Total Mayhem, that freezes at 45 mins in.

https://replay.faforever.com/23191245

LightWire

The last error in a game log. Not sure if this is a clue. Game froze right after a Rift landed.

\defaultunits.lua(100): access to nonexistent global variable "ACUUnit"
stack traceback:
[C]: in function `error'

https://replay.faforever.com/23198584

Rama

So only had seen the infinite loop bug in playbacks with lots of unit packs, but just occurred in a game we were playing with only a few mods. Mod list for the game is:

2x Resources,Storage,BuildRate,BuildRange
ACU Boost 1.5x
AI Wave Survival
25% Air Crash Damage
All Faction Quantum Gate

https://replay.faforever.com/23203884

Rama

@Jip

Do you know if this issue has occurred in other games not using the AI Wave Survival mod, or is the issue exclusive AI Wave Survival mod game?

Around the same time as the last FAF Patch, I increased the use of the following code that attaches orbs to a unit's bones. Now all units have a chance at having orbs attached to increase their DPS, depending on player strength. I've also noticed the freeze is most likely to occur during the endgame phase of the game, when many bosses are spawning and they will all have multiple orbs attached.

Here is the code for attaching orbs to units. Note that the script first checks to make sure units have more than 1 bone before calling the function to attach the orbs.
For Land Units:

AttachedLandUnitSpawn = function(self, selfId, attachedUnitId, selfBoneId)
    local platOrient = self:GetOrientation()
    local location = self:GetPosition()
    local StellarCore = CreateUnit(attachedUnitId, self:GetArmy(), location[1], location[2], location[3], platOrient[1], platOrient[2], platOrient[3], platOrient[4], 'Land')
    WaitTicks(2)
    StellarCore:AttachTo(self, selfBoneId)
    StellarCore:SetCreator(self)
    self.Trash:Add(StellarCore)
end

For Air Units:

AttachedAirUnitSpawn = function(self, selfId, attachedUnitId, selfBoneId)
    local platOrient = self:GetOrientation()
    local location = self:GetPosition()
    local StellarCore = CreateUnit(attachedUnitId, self:GetArmy(), location[1], location[2], location[3], platOrient[1], platOrient[2], platOrient[3], platOrient[4], 'Air')
    WaitTicks(2)
    StellarCore:AttachTo(self, selfBoneId)
    StellarCore:SetCreator(self)
    self.Trash:Add(StellarCore)
end

For Navy Units:

AttachedNavyUnitSpawn = function(self, selfId, attachedUnitId, selfBoneId)
    local platOrient = self:GetOrientation()
    local location = self:GetPosition()
    local StellarCore = CreateUnit(attachedUnitId, self:GetArmy(), location[1], location[2], location[3], platOrient[1], platOrient[2], platOrient[3], platOrient[4], 'Naval')
    WaitTicks(2)
    StellarCore:AttachTo(self, selfBoneId)
    StellarCore:SetCreator(self)
    self.Trash:Add(StellarCore)
end