Supcom/Moho engine rewrite

The_Janitor

@SlightlyGrilled
https://limewire.com/d/DWWf2#9gbKJxH9UW
game.log is uploaded on the link, text size too big to place here, sorry.

SlightlyGrilled

@IndexLibrorum you peaked my interest on the map loading, so i went and actually timed it.

Both the total time with graphics running in a window.
and if i run it headless without the graphics.

Theta Passage (SCMP_012), 5 km
total 0.89 s | map textures ready 0.71 s | sim running 0.75 s
(headless total 0.47 s)

Open Palms (SCMP_007), 10 km
total 0.93 s | map textures ready 0.80 s | sim running 0.78 s
(headless total 0.50 s)

Seton's Clutch (SCMP_009), 20 km
total 1.04 s | map textures ready 0.88 s | sim running 0.87 s
(headless total 0.58 s)

Gentleman's Reef (SCMP_005), 40 km
total 1.38 s | map textures ready 0.87 s | sim running 1.20 s
(headless total 0.83 s)

Betrayal Ocean (SCMP_029), 81 km
total 1.51 s | map textures ready 1.06 s | sim running
(headless total 0.91 s)

Frostmill Ruins (SCMP_030), 81 km
total 1.51 s | map textures ready 1.02 s | sim running 1.34 s
(headless total 0.94 s)

Debris (X1MP_012), 81 km
total 1.45 s | map textures ready 1.21 s | sim running
(headless total 0.82 s)

SlightlyGrilled

@The_Janitor Thanks I will have a look!

SlightlyGrilled

@Dim-A-miD yeah its super cool!

In case three isn’t enough and you’d like to bring the total to five...

I think this, and your link goes way over my head?

SlightlyGrilled

@The_Janitor ok found it, i will release a new version that fixes this, but in the mean time, in your mod folder you have a file mods/auto-upgrading mass extractors.7z

my engine is picking this up and thinking its a zip file. if you move that file it should work

but i will also fix this in the next version.

The_Janitor

@SlightlyGrilled said:

@The_Janitor ok found it, i will release a new version that fixes this, but in the mean time, in your mod folder you have a file mods/auto-upgrading mass extractors.7z

my engine is picking this up and thinking its a zip file. if you move that file it should work

but i will also fix this in the next version.

Confirmed, that did it! Game lunches now, thanks!

Xiaomi

@slightlygrilled I see that the lua calls are grouped. I thought there would be a lot of small callback calls and everything would look like "Swiss cheese" XD. So does it queue up the lua callback calls and execute them together?

@SlightlyGrilled said:

No, right at the very start i looked at it, and as much as i think its a fantastic data structure, it just doesn't fit supcoms design

Could you provide more details? Why did you abandon ecs?

SlightlyGrilled

@Xiaomi you can see under lua weapon events, there is the number 13k, that's the individual lua callbacks, I'm just not zoomed in enough to see.
And again under lua projectile impacts another ~3k

So yes the general setup is rust systems calculate everything, then apply the effects.

Actually now I'm looking I need to check that weapon step, and whether its unmarked lua, or native code. Thanks for the curiosity!

Ok an update on that, i went and had a look, and it could totally disappear! here you go thank you!

This brought the 64k air fight from 307ms a tick to 270ms! (although not in that screenshot, but as an average, Tracy does also slow down the game alittle)

On ECS, so from my understanding, ecs is really great when you have tight data you want to iterate very fast, moving as little data as possible from ram to cpu cache.

Like if you want to move over every entities health and add regen.

It's also great as an abstraction allowing separate systems to just deal with the data they want to deal with, they don't need to know about everything else, they just want to know health, and health regen etc...

However in supcom, there is a huge amount of lua, and it's structured around being able to access any of the state at any point.

So to this effect, i just store entities (units etc..) in a dense array, and lookup almost instantly by id/index.

Another issue is the sim is deterministic, which means i need fixed iteration order, and when looking at the ecs around, they didn't guaranty fixed order, when components where added or removed.

This was ages ago i made this choice though, so i could be miss remembering it, or it just straight wrong!

And if i do get cache bottle necks on tight systems, i can always move to separate arrays for different parts of the data.

Good questions though! happy to discuss more if you have more questions/ideas

Saver

Hi, do you have any installation guides explaining how I can set up and test your engine? I'm not very familiar with this area.

Thanks for all your great work!

Xiaomi

@slightlygrilled Am I imagining it, or have we already hit the "ceiling" of the old engine architecture?

To get a noticeable boost, there needs to be the ability to call lua callbacks from workers. And an additional worker for Lua scripts wouldn't hurt either....
This is a big architectural challenge.
But for that the project needs to be rewritten. Although maybe ai agents could handle it XD.

MayeulC

@Saver this should probably be written in a document somewhere, but:

Grab the latest release from the GitHub releases page
a. The most recent one should be on top (biggest version number)
b. look at "assets" at the bottom. You may need to expand the list.
c. Grab the one for the correct platform (windows x64, Macos, etc)
Extract it somewhere
Launch the executable, it should auto detect the original game and FAF locations if they are installed (if auto detection fails, they can be specified manually, but it's a tad more complex).
Bonus: a game.log file is created next to it. Upload it of you encounter any issues.

SlightlyGrilled

A new version! (now working with faf 3841!)

https://github.com/ThomasMonkman/rts-core-engine/releases/tag/v0.2.1

Lua Debugging

Thinking of developers, both for the game and modding, i have added debugging! you can see in the video below

breakpoints
conditional breakpoints
seeing variables
running code live in the console, such as in the video with SetHealth on the scout

https://youtu.be/EZGhvC2t4Uk

this can even be used to debug mods! here is me stepping in to the hooked files of the RNGAI mod (credit to @relent0r)
https://youtu.be/3pvMWk2clw0

to try this your self, you can look here
https://github.com/ThomasMonkman/rts-core-engine/blob/main/docs/guides/debugging-lua.md

Improving the stats bar:

The stats I use to see at a glance the performance of the game, both sim and rendering, but it was messy, time for a face lift. Thanks to @MayeulC for prompting and giving ideas for this work.

hover on fps

hover on sim

and to show off the networking, a very cool networking simulator, now you can pretend to be in Australia (or further) even if your not!

Next a series of bug fixes

Units turn for factory roll off
Trees now correctly blown away by explosions
Functions implemented that RNGAI used, which means RNGAI now works :), thanks relent0r
Mods and maps are loaded with case insentivity, so they now load on linux
Tree correctly blow away from explosions!
Blast damage now impacts shields
Diskwatch now reloads lua,textures,models, etc...
A vault folder without maps/ or mods/ still mounts
Closing the second monitor's window closes only that window, instead of shutting the game
Selection brackets are centred on the mesh, and range rings scale with the map when zoomed out
Intel is now fully parallel
Follow weapons are also now fully parallel
Improvements in movement with turning and corners
Aircraft refuel while being repaired, and dock at the nearest air staging platform
Eco stall for both mass and energy now handle like the base game

Also thanks to:

MayeulC a PR for improving issue onboard, and for detailed bug reports
relent0r and an94 for bug reports

SlightlyGrilled

@Xiaomi So yes your right, i think I'm approaching the ceiling on what can be done, although I'm sure there are always clever tricks to go further.

Do bare in mind though that, that trace is a 64k air battle, the normal balance of the game simply doesn't run in to that kind of unit count, even if you played with 64 players.

I do think a lua thread pool api could be introduced similar to love2d , but that would really only help with big chunks of work like ai and such.

with this i would imagine some task api where you opt in to certain data (unit position, eco, intel, etc...) to be copied over to an isolated lua thread, then you would set a duration of ticks for the work to be completed in, such that it remains deterministic, if it hasn't completed in time the sim would wait.

something a bit like this in the sim (this is pseudo code just to illustrate the idea)



-- Plain-data arguments, deep-copied at post time
local taskArgs = {
    thing = 5,
}

local dueInTicks = 10

-- Which parts of the sim to copy over
local unitsView = {
    fields     = { 'Position', 'Army', 'BlueprintId', 'Health' },
    categories = categories.MOBILE,
    armies     = 'enemies',
}

local access = {
    Units    = unitsView,
    Terrain  = true, 
    PathGrid = { 'Land', 'Amphibious' }, 
    Economy  = 'own',
}

-- Which task to run: a module path and a function in it
local taskModule   = '/mods/myai/lua/tasks/threat.lua'
local taskFunction = 'Compute'

local pending = Pool.PostTask(taskModule, taskFunction, taskArgs, dueInTicks, access)

-- do more stuff

local ok, result = Pool.WaitForTask(pending)  -- resumes at exactly post tick + 10

then the worker script


function Compute(args, View)
    local units = View.Units
    local count = units.Count()

    for i = 1, count do
        local position = units.Position(i)
        local health   = units.Health(i)
        -- do stuff thats slow...
    end

    return args.thing * 42  -- plain data, handed back at the due tick
end

This has the advantage of allowing devs in the lua to break up work in to as many threads as they see fit, and with so many cores now available, more tasks are always better.

https://love2d.org/wiki/love.thread

There is also robloxs thread model where you can have read only access in parallel, and write access is then serialised.

https://create.roblox.com/docs/scripting/multithreading

but yes you are right, it is in conflict with how supcom was made, perhaps another possibility would be to allow some wasm runtime (still allowing scripting of a sorts, just compiled), but I'm not sure if those are actually that much faster than the luau vm.

of course you could move the whole game to native code, but then you lose modding and such...

interesting none the less