Questions about performance

Jip

Hi everyone. Recently I've taken on the task to optimize the FAF base game repository. Since LOUD shows it is possible - why shouldn't we do that too.

As it stands I've made a profiler to help figure out what functions are called often and how to write code that is more efficient. One approach to this is through benchmarks. See also this folder:

https://github.com/Garanas/fa/tree/optimize-effects/lua/profiler/benchmarks

All benchmarks (with a similar name) perform the same operations, some faster than others. To give a few examples:

synthetic benchmarks

table-insert.lua / AddInsertGlobal = 0.046 seconds
table-insert.lua / AddCountAlt= 0.00219 seconds (the regular approach is 500% - 2000% slower (!))
table-hash.lua / Hash01 = 0.00488
table-hash.lua / HashCached01 = 0.00341 (the regular approach is about 10% - 30% slower)
table-loops.lua / ForGetn = 0.2136
table-loops.lua / ForPairs = 0.3704 (the regular approach is about 40% - 50% slower)

rewriting of existing functions

table-hash.lua / HashCross1 = 0.08498
table-hash.lua / HashCrossCached1 = 0.05175 (the regular approach is 30% - 40% slower)

This shows that it is possible from a theoretical perspective. I've been doing work on optimizing projectiles that you can find here:

https://github.com/Garanas/fa/commit/2cc658ed293097ac09a3af5a0a8250e63b268d5f

When 400 Zthuee are firing at one location it takes 12 - 15 ms on the base branch, assuming you are completely zoomed out. On the optimized branch it takes 8 - 9 ms. That is a significant difference on its own - and this is just on projectiles. @CheeseBerry can confirm this - on his computer it runs about 18 - 20 ms on the base branch, where as it runs 14 - 15 ms on the optimized branch.

Up to this point I've only optimized while trying to keep the original logic alive. However, there are some things that I feel are not required for the overall game, are expensive, and can not be made cheaper. I'll use this topic to discuss these issues to get a bearing as to whether people think it is worth changing or taking out of the game.

In order to facilitate discussion each problem will have its own topic on the forum. Please remain on topic, whatever you are writing for. As it is easier for me to identify how people feel about certain changes.

Jip

PRs / topics that change aesthetics for performance:

PRs / topics that change balance for performance:

(Topic) Tactical missiles

PRs / topics that just refactor for performance:

(PR) Engi mod (2021-09-19:: merged - available on FAF)
(PR) Structure rotating on build (2021-09-19:: merged - available on FAF develop)
(PR) Optimize trashbag (2021-09-19:: merged - available on FAF)
(PR) Prefetching during lobby (2021-09-19:: merged - available on FAF)
(PR) Removing unused rocking behavior calls (2021-09-19:: merged - available on FAF)
(PR) Cache engine calls (2021-09-19:: closed, another approach is taken)
(PR) Cybran build drones (2021-09-19:: merged - available on FAF)
(PR) Prevent unnecessary table allocations
(PR) Remove schook'ing
(PR) AI - Rework Factory Builder Manager
(PR) Layer optimizations (2021-09-19:: merged - available on FAF)
(PR) Optimize AI Managers
(PR) Reworking of AI Utilities
(PR) Optimize / fix Aeon build animations
(PR) Optimize / fix default explosions (2021-09-19: merged - available on FAF Develop)
(Issue) Discussion about caching engine calls
(PR) Optimize hover effects

PRs / topics related to performance:

(PR) Benchmarks (2021-09-19:: merged - available on FAF)

Vanifica

I would be totally into this. I guess I assumed LOUD worked by graphical changes, but I never tested it extensively since it didn't have an online client as far as I could tell.

They claim 5000 units without any simspeed issues IIRC. That does imply there's some "magic bullet" somewhere in the code they figured out?

Jip

@vanifica said in Questions about performance:

I would be totally into this. I guess I assumed LOUD worked by graphical changes, but I never tested it extensively since it didn't have an online client as far as I could tell.

They claim 5000 units without any simspeed issues IIRC. That does imply there's some "magic bullet" somewhere in the code they figured out?

Graphics changes do help, but are quite limiting. There is a separate render thread that takes care of most of the rendering. Only emitters have a (relative small) impact on the sim.

LOUD can run a significant larger number of units than FAF while staying at +0 or higher. They do this by applying the same techniques, everywhere. If you look at their repository you'll see this. Everything is written with maximum performance in the back of their head.

That means there is no magic bullet - as I did for the projectiles, you have to go over all the files (including the base classes such as seraphimprojectiles.lua, defaultprojectiles.lua, and projectiles.lua) and apply the same techniques everywhere. And that just takes time.

In the end 95% of the work is 'grunt work', in terms of having to go over hundreds of units / weapon files and looking at the logic, making small improvements performance-wise. Only very little of the work is difficult and abstract - there are some examples like specializing code (moving it down a class so that not all units call StopRocking() by default, instead only units that actually do Rocking (hover units) will call that in their own super class). Another example is the impact of the score(board) / hotstats.

If you're interested in helping, I can help get you started.

Dragun101

The more important thing for us not is not hreaking the various often used sim mods. So I’d also test various changes with:
Total Mayham
BlackOps (All Variants)
BrewLan
Marlo’s Mod Compedium

And then a few other sim mod.

Jip

@dragun101
I agree. I'm not changing the API or the outcome of a function. I am only changing how that came to be. I'll keep it in mind, but I do not expect any issues.

Dragun101

It might be worthwhile if and when you change a how certain variables are defined or when. To make sure the big mods don’t rely on that or made use of those variables.

Sheikah

From looking at how Jip has implemented this it should have no impact on the other big mods as all the original variables are all still accessible. They just will not be optimized in the mods

Sprouto

@vanifica said in Questions about performance:

I would be totally into this. I guess I assumed LOUD worked by graphical changes, but I never tested it extensively since it didn't have an online client as far as I could tell.

They claim 5000 units without any simspeed issues IIRC. That does imply there's some "magic bullet" somewhere in the code they figured out?

This has long been a myth - LOUD never gained any real performance boost from suppression of graphics. We did toy with this several years ago, and it stuck for some reason. There are certainly some graphical things we simplified or tied to the SIM speed, but over time, most of the pretty was returned to normal.

As Jip rightly points out - there was never a 'magic bullet' - just a lot of patient work going thru almost the entire code base, implementing the techniques that Jip is encouraging FAF to adopt now. The gains are real, but the work is extensive, but not difficult.

Jip

I've updated the 2nd post of this topic to keep track of everything related to performance.

Jip

Note - two additional performance-related PRs have been merged.

Jip

We're aiming for a patch near the start of September. However, we need more people to play on the develop branch to test whether everything works. This patch is larger than typical because the last patch is from September 2020.

Therefore my question is for you all to host games on FAFDevelop and report back!

Report back any strange behavior afterwards. You can find the log of the game here.

C:\Users\%USERNAME%\AppData\Local\Gas Powered Games\Supreme Commander Forged Alliance

If the bug is visual, make sure to add a screenshot.

Always attach the log - we can't work on the issues without it. You can send it to me through the forums or through discord where you can find me as Jip#4301.

edit: note that the branch is ranked (!).

Zeldafanboy

And this is just performance improvements, or are there other changes like bugfixes or balance changes?

Sheikah

Fafdevelop contains all the bugfixes and updates and optimizations since September 12 2020 which was the day of the last game patch.

It shouldn't include any balance changes.

Jip might best know the full change log

Jip

It includes no balance changes. Keyser and I will be working on the full changelog in a few days.

Aulex

Nice work Jip and the rest of the folks working on it!

Azraaa

To key in on this performance and development large phase that faf is currently going through. I am working on really improving functionality and performance of ALL core AI code on faf so helpfully for you AI Players you'll get even more benefit once these patches come out.

Very exciting stuff that jip has allowed us to do

Utopian

@jip game_15187368.log
Hey, I just played this game and got a memory access violation crash after one of the AI players died. This is the full log which FA gave me afterwards. The only sim mods active were AI-Uveso and AI-RNG. All else is just UI stuff like SSB.

Jip

@utopian Thanks for your report, looking into it now.

Do you happen to have the exact exception code that occured?

edit: after inspecting the log I suspect this crash is not related to the develop branch, but to something else. If you happen to have the exact exception code then I can compare it with exception codes that we've found previously.

Utopian

@jip

Here's the specific error message. Don't know if it's related to develop after all, but might be worth a gander.

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x0050dfd8
attempted to read memory at 0x0000005c

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe
Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15187368.log /gpgnet 127.0.0.1:34600 /mean 1333.36 /deviation 222.314 /savereplay gpgnet://127.0.0.1:52527/15187368/Utopian.SCFAreplay /country HU /numgames 28

Callstack:
Unknown symbol (address 0x0050dfd8)
Unknown symbol (address 0x0068af40)
Unknown symbol (address 0x1d1ed9fb)
Unknown symbol (address 0x1d1ed9fb)

EDIT: if you need system specs or anything, just tell me.