Large Unit Clumping and Unresponsiveness

@DragonStrike406 said in Large Unit Clumping and Unresponsiveness:

I found this out the hard way. While the DLL files are a mix of Lua and C++, and the executable itself is C++ only, that is still all I can say. After disassembling I did not find any names I could put to use such as function calls, function names, file names, etc. Everything was in deep x86 assembly, and even when translated to C++, it was basically the same useless code. I say useless because it is. You cannot get anything useful from it, no names, no function structures, no nothing. One big gibberish. It's exactly the same thing with the libraries stored in the DLL files. They too are a massive pile of organized mess with no names.

I can mostly agree with this (which just a few minutes of looking). So far I could get some useful class names, but not really useful function names or implemenations. The decompiled implementations in C++ where pretty much word-by-word translations of the assembly - aka useless.

As to swapping FAF libraries with that of SC2, that will not work either. While yes, the base engine is the same, the two versions vastly differ, which I can say from just looking at how the starting hexes of each executable are laid down. Both are the SpringRTS engine, yes, but one is from late 2007 while the other is from early 2010.

I wonder, what makes you conclude that these both are derivations of the SpringRTS engine?

But it doesn't take going into reverse engineering to notice that the two games are too different to swap any files with each other. If you go to SC:FA installation and check what's in it, then compare it to how SC2's structure is made you can come to a conclusion that they work very differently from each other if your IQ is higher than the room temperature in Celcius.

The initial file layout is not that insanely important. Their linking method might have changed, fooling you into thinking there is absolutely no commonality left - while there is - albeit very limited.
Additionally I never meant "trivial file swapping". I'm a software dev myself, I know things aren't this easy.

TLDR; I did reverse engineering on the FAF executable, no you cannot modify it to make it run better, no you cannot swap files from SC2 because the two games are structured in an entirely different way, it would be easier and faster to remake FAF in some new game engine.

Well, it was an idea. But apparently a flawed one. I did not expect supcom 1 and 2 foundations to diverge this insanely much.

I wonder, what makes you conclude that these both are derivations of the SpringRTS engine?

My bad. The engine is a homebrew. Just my janky memory connecting the dots that SpringRTS was released in 2007, is an RTS engine, suffered from lack of multithreading, and that FA has exactly same issues. Plus the fact that if you go to wikepedia, it says it's related in external links. https://en.wikipedia.org/wiki/Supreme_Commander_2#External_links Then again, I checked that like a year or two ago, and back then I still had my memory department still functioning correctly.

The initial file layout is not that insanely important. Their linking method might have changed, fooling you into thinking there is absolutely no commonality left - while there is - albeit very limited.

I also did say that after looking at the hexes of both games I noticed they are different. Maybe there is a way to put them together, although without any source code, or at least knowing what wants what, that's basically guessing.

Additionally I never meant "trivial file swapping". I'm a software dev myself, I know things aren't this easy.

Ah, a man of culture. Then I needn't explain more.

Well, it was an idea. But apparently a flawed one.

Ideas are flawed, mistakes are made, but there's always a lesson to be learnt. Have a good day, commander.

@WackelKontakt said in Large Unit Clumping and Unresponsiveness:

@Katharsas Given how insaenly fast CPUs nowadays are compared to 2007, even in single core, when used properly, I'd not even try to attempt any hacking to "multithreading" in (which wouldn't help much, when the core engine doesn't support it and you end up doing pointless tiny calculations multithreaded without any gain).

Instead, one should attempt to revive the work to modify luajit-GC to make it fit with supcom. This should (at least according o micro benchmarks of luajit) in some cases give up to 4x or even more performance.

I actually take a look at ghidra right now. It's an amazing tool, however making sense of all will take time (and stamina).

Initially I will take a look at both, supcom 1 and supcom 2 mohoengine.dll. If its not entirely easily possible to replace supcom 1 pathing implementation with an own one, maybe there is a way to give supcom 1 at least supcom 2 path finding.

Yeah it might be stupid to try to multithread anything. But its a complaint that comes up quite often, and it fits with the engine related complaints. The thing i fear about with LuaJIT is determinism. I have a feeling that might make it impossible to use a JIT in general.

I have fiddled with Ghidra, and i suggest to use Ghidra plugins like OOAnalyzer, because it apparently can auto-reverse typical C++ structures (vtables tec.), but havent tried that yet.

The bigger problem is collaboration. There is a "Ghidra Server" out of the box. However:
A) That server implements its own version control system for Ghidra project files (custom stuff, not Git)
B) You can only connect to that with Ghidra, however Ghidra is new and i guess a lot of people still use IDA, so they wounld have to switch to Ghidra to be able to contribute.
C) Somebody needs to look into the legal siuation and potential contributers need to know if/how they can stay clean.

Not unsurmountable though.

Edit:
Hadnt read DragonStrike406 posts yet, thanks for sharing sop much experience.

I fully expect a reverse engineering project to take multiple 1000 hours. Howver, i expect a full reimplementation of the engine to take just as much time if not longer, it it wants to surpass current engine and not just be an alternative with other problems than the current engine. Imo a new engine is pointless unless it works a drop-in replacement for the old engine. This means that physics need to behave exactly like in current engine. If not, FAF community will not switch or split, and then its worse than before.

It would be massive undertaking either way. Cant we actually start reversing at the points were calls into known DLLs are made? Graphics engine calls into DirectX, Audio calls, file system, engine Lua functions.

Just reversing the audio engine might be actually super usefull as a start, given how many problems the game has with Audio.

The more is reversed, the easier it gets, hardest is the start. And i dont suggest a single person to do this. It MUST be a group effort for there to be any chance of success, with makes the collaboration topic critical to success.

Maybe we should wait 10 years until Ghidra server-side projects work with Git and IDA has plugin for it.

A couple of things.

Let's not compare Supcom and Starcraft.

Or hey, maybe let's.

Starcraft 2 maps are max 256x256 grid units.
A 20x20 Setons is 1024x1024 grid units. That's 16 times as big, and it's the middle size.

Starcraft maps are laughably small by comparison, and the maximum you can bandbox is an absolute joke; you can easily run a pixel-perfect Dijkstra or A+ discovery for each selected unit in between frames without skipping a beat, and it's likely exactly what gets done. It's a one-banana problem.

FA's pathfinding algorithm is much more constrained simply due to scale, and the optimizations built into it are very apparent at scale. And is definitely NOT inefficient; it simply bogs down due to collision detection at the start of a mass move. Everything freezes in a dining philosophers' deadlock until free edges are opened. Starcraft's algo would suffer exactly the same fate were it even possible to select 200 units at a time to begin with.

Can it be optimized?

Of course it can. I'd like to see anyone here try, though.

So its not impossible just beyond range of most of us. Dragon is my understanding correct?

I’m a shitty 1k Global. Any balance or gameplay suggestions should be understood or taken as such.

Project Head and current Owner/Manager of SCTA Project

“middle-sized”

40x40 and up exist on FAF, but the game (on FAF) certainly isn’t designed for those sizes. Game balance quickly breaks beyond 20x20 simply due to the ability to scale.

The pathfinding in Starcraft 2 is based on flocking behavior, in many ways it is similar to the pathfinding in Supreme Commander 2 and therefore not comparable to the pathfinding in Supreme Commander.

Sending a group of 200 units across the map would cost about the same of sending a single unit: a path is computed for each bundle of units (which is generally 1 large bundle, or when a micro is in play maybe 3 - 4) and the units will try and follow the path while at the same time keeping their distance between other units. It is the algorithm that the engine developer suggests in the interview with Annihiline:

If you'd like to know more about flocking behaviors, steering behaviors are quite similar but more applicable to games. See also:

I haven't reverse-engineered anything myself and don't intent on doing so after looking at the output of Ghidra with the faf executable as input. Implementing an alternative pathfinding system is a challenging task, let alone designing it in such a way that it does not change the behavior of units too much from how they behave right now. Then of course make it bug-free. And then there is the reverse-engineering part on top of that to make it fit with the rest of the building blocks.

Taking DragunStrike and Katharsas on their word, it is an almost impossible task.

A work of art is never finished, merely abandoned

@FtXCommando said in Large Unit Clumping and Unresponsiveness:

“middle-sized”

40x40 and up exist on FAF, but the game (on FAF) certainly isn’t designed for those sizes. Game balance quickly breaks beyond 20x20 simply due to the ability to scale.

In my opinion that's an interplay between the sheer size of the map grid, and the fact that T1 is a thing. Case in point, the other day I played a 40x40 world dom for the first time, and mistakenly treated it like ladder. By the time my spam got anywhere, dudes were already at t2 and climbing. So I spammed more... and thus.

I mean, looking at how many people have been interested in improving the engine over time, there imo is a chance if we split up the task. Start with small parts of the engine that are as indepedent from the rest as possible. Therfore audio engine. Use such a small project to test the waters.

But there seems to be a culture of "lone wolfs" in RE. People often work alone? Probably because collaboration tools for RE suck or are not mature yet or cost too much. Maybe because there is somtheing inherent to REing that makes collaboration hard. Maybe because people are so afraid to even try doing it because of some notion of "REing is an ancient hacker skill that only the chosen ones are able to wield". Its weird. If you can undestand the mess that is Javascript or even CSS, you are intelligent enough to read typical x86 Assembly just fine.

All this talk of overhauling the pathfinding system and I find myself wondering - just because we can do it, does it mean we should? Supcom 2/StarCraft II are held up as paragons of efficient pathfinding, but is their pathfinding necessarily better, beyond a technical level, than what we currently have? Having played both a lot of Supcom FA and Supcom 2, I personally think the way the tanks move in Supcom FA have a higher degree of realism to them. The inefficiencies of the pathfinding system actually lend a weightiness to the units, and make them react in a way that feels more realistic. In Supcom 2, tanks keep their distance between one another as if controlled by a hive mind and move with instantaneous fluidity. Again, it's more of a design decision, but I don't think it makes a lot of sense for my platoon of tanks to move like a shoal of sardines that are schooling. Just my 2 cents.