Supcom/Moho engine rewrite

SlightlyGrilled

So this is a really big update from my side, tons of things, internally I use issues to track what I have and haven't done, so this time around I've put the issue numbers on what I've done

Open source

First off I've finally released the source code, it took a ton of effort to make this ready for other people, and I'm sure it still needs more, so please be patient with what you find

I have made it as a fresh repo, as understandably when you work on something yourself which you're never sure may see the light of day, you can get sloppy, and I didn't want to risk passwords or what ever being public.

I'm happy for people to raise issues for bugs and such

and I hope people have fun playing around

source code
https://github.com/ThomasMonkman/rts-core-engine
exes:
https://github.com/ThomasMonkman/rts-core-engine/releases/tag/v0.1.0

#367 Replay keyframes: fast seeking, played ahead in the background

Replays now support quick rewinding! and skip forward (to degree)

when starting a replay, a second process now starts up in the background and plays the replay at max speed saving out "keyframes" of the game. You can see this in the video below as a buffer bar.

You can now skip to anywhere in this buffer bar. a brief delay as the game reloads in at that time.

But you can now go back and forth as you please.

and thanks to differential compression it uses very little storage.
These keyframes stay even after closing the game, so you can quickly look through a replay or preload it, and then scan through the whole replay later with no background processing

Video:
https://youtu.be/0MWgoXI4nQE

this can all be controlled from settings.

#386 The lobby downloads a missing map from FAF's vault

The lobby will now download maps from faf that are missing from clients, this happens as they are selected by the host, so no reloading needed.

Testing

An important part of any modern software development is testing.
As such this project has a few different types.

unit test (very isolated, does the function add(2, 3) equal 5)
benchmarking
end to end, or real scenarios tests

the scenarios tests are what I want to show below, it's massively sped up so you can actually see it somewhat in a lifetime.

they each test something fairly specific, like "can this tank attack this tank", "is this tanks shot blocked by terrain"

i did spot a few errors during this recording, but that's all part of the process

https://youtu.be/edDiLLQhGL4

Naval

#351 Torpedoes run under the water, naval units start on the surface
#355 Submarines start submerged
#357/#358 Submarines surface on their own for carrier loads, unloads and AutoSurfaceToAttack
#356 Torpedoes fired from a still sub start at rest
#457 Submarines dive and surface bow first
#455 Carriers launch aircraft from their launch points, one a tick, without moving
#418 The cheat menu spawns ships on water

Movement and pathing

#475 Units ease into a stop and coast to a halt on arrival
#327 Formations run at 0.85× their slowest unit, and members steer for their final slots
#329 Units only push moving units on other orders, as fading impulses
#406 Land units no longer path over cliffs into or out of the water
#469 Hover units lean a tenth as far and level out after stopping (the seraphim where just bobbing around all over the place )
#467 Finished ground units stand on the ground under them
#361 Walk animations follow the unit's speed and turns
Issue* orders skip units that can't do the order, such as an shield generator being told to attack, or tanks being told to reclaim

Combat and vision

#347 We now draw nice fog of war instead of pixelated from before
#345 Weapon target searches skip targets the terrain blocks
#348 Weapons on hold fire search for no targets
#342 Aggressive moves and patrols engage a task step late
#420 Turret pitch is aimed from the barrel bone's own pose
#370 Persistent blips (FAF's Blip class, destroy hooks, real-game lifetime)
#468 Fog of war's clear ground is drawn as vision circles
#499 Czar and some other beams had effects sideways
Overcharge works
Weapons in general work better and more faithfully to the base game
Energy drains are actually paid, so T2 artillery charge-up, teleport and overcharge wait for energy
Weapons fire the beat after they take a target, as in the real game (i feel like this is something maybe should be changed about the game?)
Homing missiles steer by their facing. The Corsair's AA darts now fly level and hit, where before they flew into the ground
Solace torpedo bombers now actaully do damage
TMDs now actaully work correctly
Fire Beetle detonates on attack orders and no longer blows up at game start (this was more when doing testing, as you never start the game with fire beetles, but they also had other bugs)
HARMS sits on the sub layer
Weapon/damage callbacks now get the real tick number it happened on, instead of the tick n - 1
GiveStorage / GiveResource apis behave as in the real game
Anti torpedo weapons now work

Networking, replays, lobby

#449 Lockstep desync fixes (local-only input, ejected turns, command text, observer sources) and a desync dialog
(this was a nasty little bug, i was double reading local player input once, before the network tick and once after, so it was possible to input commands in to your local sim without sending to others!!)
#454 Desync detection: a checksum every tick, in parts, covering motion, orders and economy
While i've made efforts to build the sim in such a way desyncs should be hard to get, its always still good to have logging for it
#380 One broken map no longer empties the lobby

Building, engineers, factories

#403 Build sites snap to the cell grid
#464 Assisting factories share the assisted queue and keep building their own
#465 An engineer's assist ends when an order is queued behind it
#451 Engineers parking to assist slide off each other instead of stacking
#456 Assisting engineers show the progress of what they help build or repair
#452 Build beams go away when an unfinished structure is destroyed
#463 Factories keep a queue of rally orders, shown with Shift and kept on upgrade
#416 The construction panel queue works for mobile factories and ACU enhancements
#417 One waypoint ETA per structure, at its builders' combined rate
#459 The ACU upgrade ETA shows without selecting the ACU
#466 Reclaiming a unit takes its health down with its fraction
#453 SetMaxHealth leaves health alone
#458 Excess resources overflow to allies
#365 Enhancement tabs are in the real game's order
#507 engineers assisting one of themselves leave its own build queue alone

Camera and interface

#390 Camera spin (Space + mouse, Insert / Delete, cam_Free)
#389 Camera field of view and zoom works as in moho
#394 You can now scroll in and out when tracking a unit with "T"
#450 Camera SnapTo / MoveTo / RestoreSettings honour heading and pitch
#360 Queued waypoints and build placements can be dragged to move them
#414 A test that dragging the build queue reorders the factory queue
#382 Unit stats for the end-of-game screen
#385 The stats bar pause button no longer shifts
#404 The in-game exit says "Exit" on platforms other than Windows
Minimap shows an earlier game's map, if you play a second game without restarting
Wrong army in later games, if you play a second game without restarting
Game freezes when the other player leaves (no clickable buttons, black screen after the score view)
Process stays running after the window closes → exits on its own after 3s
Map download "builder error" on Linux, i've added more logging, and a failed download now blocks that player's launch

Visuals

#470 Seraphim build mesh and Seraphim / Aeon build beams match the real game
#474 Cybran structures under construction hide what's behind them
#461 Mesh LOD fixes
#492 Collision beam's far end starts at its target when enabled (Monkeylord laser)
#493 Ythotha death: energy being draws, wreck keeps its death pose, no white beam slabs

Platform

#383 The join code now copies to the clipboard on sway / Wayland
#384 FAF and Steam are found where faf-linux looks for them
#413 A hidden window no longer stops the other window drawing, plus you can now window the other window
and in general multi monitor should be much better supported, although their are limitations with wgpu and different platform support
Wayland terminal flooded with clipboard log lines
No vsync cap on Wayland (1000 FPS in the menu), wayland wasn't giving a framerate limit at startup, we now find the highest refresh rate screen and use that until we get the correct refresh rate

Performance and internals

#338 Map decals skip lighting their transparent pixels
#339 Terrain shadow is worked out once a pixel

So one of the only graphics changes i have made to the project is using SMAA for AA and PCF (percentage closer filtering) for shadows, along with up to 2k shadow maps instead of 1k.

the downside to that shadow improvement is the shadow calculations becomes more expensive on the gpu.

While playing around I noticed with just looking at the map was dropping my frame rate which absolutely shouldn't be happening on a game from 2007!

well time to fire up the graphics profilers and have a look.

got to say im very impressed with xcodes metal debugger.

and its profiling instantly shows almost all the gpu time is spent drawing the decals on the map!

you can see here in this image up at the top left hand side that vs_decal/fs_decal_albedo is at 68% of the world stage which its self is 90% of the frames work!

well it turns out now calculating shadows is much more expesnive and its done for all the decals across the whole map.

well the fix is easy, most of the decals texture is transparent.

so right at the start of the shader, if the pixel is transparent stop.

if (alpha <= 0.0) { discard; }

this took my frame time from 15.46ms to 11.10ms!

next still for every decal we are calculating the shadows again, as decals can stack on top of each other, this is a lot of extra work.

so we now precalculate the shadows and share between the terrian and decals

this takes 11.10ms to 7.96ms!

~twice as fast as before!

But we can go further! I noticed at large zoomed out views my fps again was dropping. Unacceptable!

Most of the map drawing is dominated by reading lots of textures, but for when the texture is far away, you could use smaller textures right? you see less of it so who cares if its smaller?\

well thats a well known thing in graphics already, called mipmaps. you store with the texture a version of it halfed and halfed until it gets to 4x4.
like so

512
256
128
64
32
16
8
4

then the shader uses the size appropriate in the moment, this saves a lot of memory bandwidth.

this gives me 5.6ms to 3.88ms! (on a different map to the one above, so just look at the change)

In fairness, the base game also does this, i just assumed all textures shipped with mipmaps...

These mipmap textures are saved out to a cached folder, which can be configured both in size and expire date in the settings.
it will make first startup slower

#374 Faster weapon target searches (this really only effects the 64k stress test, but weapons are now twice as fast)
#376 Effect expiry uses an index instead of scanning every effect
#368 Landing for aircraft searches are much faster now
#407 Mouse moves are coalesced, so hovering hit-tests once a frame (especially useful with super high hz mice)
#421 The interface keeps control layouts between frames (this is very much targeted at the restrictions page which is very heavy on the ui, now its atleast twice as fast)
maps can now be saved loaded when compressed with zstandard (god I love a good compression algo, this shrinks a 81km map to like 5-24x smaller)

here is a grid of all the inbuilt maps which really shows you how good compression is, and to think they shipped the game on dvd!

Base game map sizes, raw .scmap vs zstd level 19

SCMP_001 Burial Mounds (20 km) 9.4 MB -> 2.5 MB (3.7x)
SCMP_002 Concord Lake (20 km) 7.9 MB -> 2.5 MB (3.2x)
SCMP_003 Drake's Ravine (20 km) 8.8 MB -> 1.7 MB (5.2x)
SCMP_004 Emerald Crater (20 km) 10.8 MB -> 3.0 MB (3.6x)
SCMP_005 Gentleman's Reef (40 km) 37.9 MB -> 5.9 MB (6.4x)
SCMP_006 Ian's Cross (20 km) 10.8 MB -> 2.9 MB (3.7x)
SCMP_007 Open Palms (10 km) 2.7 MB -> 0.7 MB (3.7x)
SCMP_008 Seraphim Glaciers (20 km) 9.9 MB -> 2.0 MB (5.0x)
SCMP_009 Seton's Clutch (20 km) 9.6 MB -> 2.6 MB (3.7x)
SCMP_010 Sung Island (20 km) 9.9 MB -> 2.3 MB (4.4x)
SCMP_011 The Great Void (40 km) 31.2 MB -> 1.9 MB (16.1x)
SCMP_012 Theta Passage (5 km) 1.0 MB -> 0.2 MB (4.0x)
SCMP_013 Winter Duel (5 km) 0.9 MB -> 0.3 MB (3.5x)
SCMP_014 The Bermuda Locket (20 km) 11.1 MB -> 2.2 MB (5.0x)
SCMP_015 Fields of Isis (10 km) 3.9 MB -> 1.2 MB (3.4x)
SCMP_016 Canis River (5 km) 1.4 MB -> 0.4 MB (3.2x)
SCMP_017 Syrtis Major (10 km) 3.0 MB -> 1.0 MB (2.9x)
SCMP_018 Sentry Point (5 km) 2.0 MB -> 0.6 MB (3.5x)
SCMP_019 Finn's Revenge (10 km) 3.5 MB -> 0.8 MB (4.2x)
SCMP_020 Roanoke Abyss (20 km) 10.3 MB -> 2.3 MB (4.4x)
SCMP_021 Alpha 7 Quarantine (40 km) 32.2 MB -> 7.6 MB (4.2x)
SCMP_022 Arctic Refuge (10 km) 2.5 MB -> 0.8 MB (3.0x)
SCMP_023 Varga Pass (10 km) 2.5 MB -> 0.8 MB (3.1x)
SCMP_024 Crossfire Canal (20 km) 12.0 MB -> 3.5 MB (3.4x)
SCMP_025 Saltrock Colony (10 km) 2.7 MB -> 0.6 MB (4.3x)
SCMP_026 Vya-3 Protectorate (10 km) 4.0 MB -> 1.0 MB (4.0x)
SCMP_027 The Scar (20 km) 9.2 MB -> 2.8 MB (3.3x)
SCMP_028 Hanna Oasis (40 km) 32.5 MB -> 7.7 MB (4.2x)
SCMP_029 Betrayal Ocean (81 km) 122.0 MB -> 20.3 MB (6.0x)
SCMP_030 Frostmill Ruins (81 km) 126.8 MB -> 16.6 MB (7.6x)
SCMP_031 Four-Leaf Clover (10 km) 2.6 MB -> 0.9 MB (3.0x)
SCMP_032 The Wilderness (10 km) 3.6 MB -> 1.0 MB (3.5x)
SCMP_033 White Fire (10 km) 3.6 MB -> 0.7 MB (5.0x)
SCMP_034 High Noon (10 km) 2.9 MB -> 0.7 MB (4.2x)
SCMP_035 Paradise (10 km) 3.0 MB -> 0.8 MB (3.6x)
SCMP_036 Blasted Rock (5 km) 1.0 MB -> 0.3 MB (3.0x)
SCMP_037 Sludge (5 km) 0.8 MB -> 0.3 MB (2.8x)
SCMP_038 Ambush Pass (5 km) 1.3 MB -> 0.4 MB (3.3x)
SCMP_039 Four-Corners (5 km) 1.2 MB -> 0.3 MB (3.6x)
SCMP_040 The Ditch (20 km) 9.7 MB -> 2.4 MB (4.1x)
X1CA_001 campaign (20 km) 8.7 MB -> 2.3 MB (3.7x)
X1CA_002 campaign (20 km) 8.5 MB -> 2.7 MB (3.2x)
X1CA_003 campaign (40 x 20 km) 17.2 MB -> 4.5 MB (3.8x)
X1CA_004 campaign (20 km) 7.9 MB -> 1.5 MB (5.4x)
X1CA_005 campaign (20 km) 8.5 MB -> 2.1 MB (4.0x)
X1CA_006 campaign (20 km) 8.4 MB -> 2.1 MB (3.9x)
X1MP_001 Crag Dunes (5 km) 1.2 MB -> 0.4 MB (2.7x)
X1MP_002 Williamson's Bridge (5 km) 1.0 MB -> 0.4 MB (2.5x)
X1MP_003 Snoey Triangle (10 km) 2.4 MB -> 0.7 MB (3.3x)
X1MP_004 Haven Reef (10 km) 3.1 MB -> 0.8 MB (3.7x)
X1MP_005 The Dark Heart (10 km) 2.8 MB -> 1.0 MB (2.9x)
X1MP_006 Daroza's Sanctuary (10 km) 2.2 MB -> 0.7 MB (3.0x)
X1MP_007 Strip Mine (20 km) 8.6 MB -> 1.7 MB (5.0x)
X1MP_008 Thawing Glacier (20 km) 9.1 MB -> 1.8 MB (5.0x)
X1MP_009 Liberiam Battles (20 km) 8.5 MB -> 2.2 MB (3.8x)
X1MP_010 Shards (40 km) 33.9 MB -> 7.8 MB (4.3x)
X1MP_011 Shuriken Island (40 km) 32.4 MB -> 4.6 MB (7.0x)
X1MP_012 Debris (81 km) 122.2 MB -> 5.0 MB (24.5x)
X1MP_014 Flooded Strip Mine (20 km) 10.4 MB -> 2.5 MB (4.2x)
X1MP_017 Eye of the Storm (10 km) 3.0 MB -> 1.0 MB (3.1x)

Total, 60 maps: 862.1 MB -> 152.8 MB (5.6x)

Saver

The_Janitor

What can I do If I face this error message "ERROR rts_core_engine: engine init failed: invalid Zip archive: Could not find EOCD" Verified the game files and they all checked out ok, If there is something I am maybe missing please let me know, really wanna try it out.

IndexLibrorum

@slightlygrilled It might be fun to do an AMA presentation thing in the FAF discord sometime, including an explanation of your project and how you're working on it to people that are not as technically inclined. It would both improve visibility of your project, and be good content for the FAF youtube channel/community, I think.

If you're not in the FAF discord yet, come join us and reach out to me or someone from the news team.

Separate question: saw your testing video. Would your tests be a little bit faster if you used a map that was smaller and had fewer/no assets? I wonder if not loading setons clutch every time might be a (probably minor) significant positive effect on your testing speed.

hobosapien

God DAMN this is AWESOME

Xiaomi

@slightlygrilled Hi. I'm very interested in the technical details:

Is the simulation single-threaded?
Did you use ECS?
How much slower is fixed-point math?

SlightlyGrilled

@Xiaomi Always happy to talk tech

is the simulation single-threaded?
Hell no! I want every core i paid for to be used!
However to be a bit more technical, there are parts that are threaded, and there are parts that are single threaded.

So the tick runs in stages, which you can see listed here:
https://github.com/ThomasMonkman/rts-core-engine/blob/develop/docs/architecture/sim.md#the-tick
normally the way it works is, each stage has calculations done in rust in a threadpool, then lua callbacks happen

so you end up with a spiky pattern of cpu usage,

rust multicore --> lua single thread -> rust multicore -> lua single thread

this is just the nature of how the game was designed, and i am staying combatible to supreme commanders lua.

I do have ideas for a threadpool lua api, that modders and such could use at will to distribute work to as many cores as they please.

steps that are expensive enough to justify giving work to a threadpool have that work split up across how ever many cores you have.

you can see this talked about in this post
https://forum.faforever.com/post/75138

with the image below you can see the thread usage

each row is a thread
workers 0-17 is the general thread pool (i have an 18 core machine)
there is a dedicated sim thread
dedicated threads for each window/screen
6 render worker threads (these and the window threads have higher OS priority so the sim can't stave out the graphics from actually appearing...)

you can see in the screenshot below (this is a 64k airfight so its extremely taxing on the lua)
also its super pixerlated...
the movement, intel, weapons, projectiles, and transfer sim state to the renderer are all threaded

Not to say this couldn't be improved though! always want more core usage!

Did you use ECS?
No, right at the very start i looked at it, and as much as i think its a fantastic data structure, it just doesn't fit supcoms design, as remember i wanted it to be a drop in replacement.
And in the lua phases there is a ton of random access.

I do however take in to account the ideas of ECS in terms of cpu cache awareness, and all the "hot" data (data that is accessed in tight loops) is all packed in to dense arrays, with "cold" data stored elsewhere.

Fixed point vs floating point

I just ran a quick benchmark for you, of course this is a mirco benchmark, so take with a massive massive grain of salt, as normal code isn't like this, but its the raw numbers.

│ Operation │ Fx (I32F32) │ f32 │ f64 │ Fx vs f32 │
│ add │ 0.48 ns │ 0.49 ns │ 0.50 ns │ ~1× │
│ mul │ 0.50 ns │ 0.49 ns │ 0.49 ns │ ~1× │
│ div │ 3.9 ns │ ≤0.49 ns │ ≤0.49 ns │ ≥8× │
│ sqrt │ 5.4 ns │ ≤0.48 ns │ ≤0.47 ns │ ≥11× │
│ atan2 │ 12.9 ns │ 2.1 ns │ 4.6 ns │ ~6× │
│ sin_cos │ 10.4 ns │ 2.3 ns │ 3.5 ns │ ~5× │
│ vec3 normalise │ 15.5 ns │ 1.0 ns │ 1.1 ns │ ~16× │
│ quat multiply │ 5.5 ns │ 1.1 ns │ 1.6 ns │ ~5× │

as you can see it depends what you are doing.

But what i would say, from my posts above, i don't really believe it matters even though floating point is faster, my engine is still super fast, and im almost fully limited by the single thread nature of supcoms lua.

SlightlyGrilled

@hobosapien Thank you

SlightlyGrilled

@The_Janitor Can you send me the log file please? it should be right next to the exe

SlightlyGrilled

@IndexLibrorum Hello! I think i have actually spoke to you a bit on the discord, when I've posted links, i think you had ideas on the keyframe stuff with the replays right? well know there in!

I'm always happy to chat tech, or anything like that

On the speeding up tests, that's actually a great idea and didn't occur to me! thank you
I spent a bit of effort to load in the maps as fast as possible before, a main bottleneck for me is some of fafs start up lua is quite slow, although i did some work to speed that up, and the tests do run faster now.
In reality i run the tests in parallel without the window, that i just measured, completes in 9.49 seconds on my machine.

I did the screen recording just for visuals

But faster is always better!

The_Janitor

@SlightlyGrilled
https://limewire.com/d/DWWf2#9gbKJxH9UW
game.log is uploaded on the link, text size too big to place here, sorry.

SlightlyGrilled

@IndexLibrorum you peaked my interest on the map loading, so i went and actually timed it.

Both the total time with graphics running in a window.
and if i run it headless without the graphics.

Theta Passage (SCMP_012), 5 km
total 0.89 s | map textures ready 0.71 s | sim running 0.75 s
(headless total 0.47 s)

Open Palms (SCMP_007), 10 km
total 0.93 s | map textures ready 0.80 s | sim running 0.78 s
(headless total 0.50 s)

Seton's Clutch (SCMP_009), 20 km
total 1.04 s | map textures ready 0.88 s | sim running 0.87 s
(headless total 0.58 s)

Gentleman's Reef (SCMP_005), 40 km
total 1.38 s | map textures ready 0.87 s | sim running 1.20 s
(headless total 0.83 s)

Betrayal Ocean (SCMP_029), 81 km
total 1.51 s | map textures ready 1.06 s | sim running
(headless total 0.91 s)

Frostmill Ruins (SCMP_030), 81 km
total 1.51 s | map textures ready 1.02 s | sim running 1.34 s
(headless total 0.94 s)

Debris (X1MP_012), 81 km
total 1.45 s | map textures ready 1.21 s | sim running
(headless total 0.82 s)

SlightlyGrilled

@The_Janitor Thanks I will have a look!

SlightlyGrilled

@Dim-A-miD yeah its super cool!

In case three isn’t enough and you’d like to bring the total to five...

I think this, and your link goes way over my head?

SlightlyGrilled

@The_Janitor ok found it, i will release a new version that fixes this, but in the mean time, in your mod folder you have a file mods/auto-upgrading mass extractors.7z

my engine is picking this up and thinking its a zip file. if you move that file it should work

but i will also fix this in the next version.