Large Unit Clumping and Unresponsiveness
-
Id say 50/50 after say a correction of issue placed within SupCom 2, which was fine, but still seemed to be lacking something for the gain to say.
-
Besides the "typical" (and I mean that in a non-condescending way) "out of our reach response": How much reverse engineering work was acutally done on this? Was the exact DLL and maybe functions and data/objects that deal with or are affected with pathfinding identified? If so, one could maybe attempt an dll inject approach.
Additionally, pathing in Supcom 2 is mostly fixed and both supcom 1 and 2 use the same engine. Can we maybe get clues by diffing their mohoengine.dlls? (symbol table etc.)
-
@Katharsas Given how insaenly fast CPUs nowadays are compared to 2007, even in single core, when used properly, I'd not even try to attempt any hacking to "multithreading" in (which wouldn't help much, when the core engine doesn't support it and you end up doing pointless tiny calculations multithreaded without any gain).
Instead, one should attempt to revive the work to modify luajit-GC to make it fit with supcom. This should (at least according o micro benchmarks of luajit) in some cases give up to 4x or even more performance.
I actually take a look at ghidra right now. It's an amazing tool, however making sense of all will take time (and stamina).
Initially I will take a look at both, supcom 1 and supcom 2 mohoengine.dll. If its not entirely easily possible to replace supcom 1 pathing implementation with an own one, maybe there is a way to give supcom 1 at least supcom 2 path finding.
-
I recently picked up reverse engineering the FAF executable and libraries.
If you want credibility to my claims, I worked on two game reverse engineering projects in the past. One on Robocraft, where I had my part in deobfuscation of a massive C# assembly, and the other was the HAWaKENing project where I assembled a team which goal was to make Hawken run on PC again. This was no easy task because the game was only multiplayer, and was shut down on January 2nd 2018. We had to not only disassemble the executable and find out how it's inputs work, we also had to make a login and services server from absolute scratch. We did succeed to some extent, we are yet to get actual multiplayer working but we did manage to get the account server running and player can now even play against bots.
That's for my background, and as someone who works with IDA, Snowman Disassembler, and writes some of my own tools to help me in the process of making old software run again, I can attest that it would be much faster to remake the game using current-day technologies than it would be to improve the executable.
I found this out the hard way. While the DLL files are a mix of Lua and C++, and the executable itself is C++ only, that is still all I can say. After disassembling I did not find any names I could put to use such as function calls, function names, file names, etc. Everything was in deep x86 assembly, and even when translated to C++, it was basically the same useless code. I say useless because it is. You cannot get anything useful from it, no names, no function structures, no nothing. One big gibberish. It's exactly the same thing with the libraries stored in the DLL files. They too are a massive pile of organized mess with no names.
As to swapping FAF libraries with that of SC2, that will not work either. While yes, the base engine is the same, the two versions vastly differ, which I can say from just looking at how the starting hexes of each executable are laid down. Both are the SpringRTS engine, yes, but one is from late 2007 while the other is from early 2010. In that period of time, Square Enix could have easily implemented an entirely new compiler and change a lot of engine functions themselves to better suit their needs for the game. And they certainly did so, because one major difference that can be pointed out right out of the gate, is the fact SC2 benefits from running on multiple threads, while SC:FA's engine does not run the simulation this way.
But it doesn't take going into reverse engineering to notice that the two games are too different to swap any files with each other. If you go to SC:FA installation and check what's in it, then compare it to how SC2's structure is made you can come to a conclusion that they work very differently from each other if your IQ is higher than the room temperature in Celcius.
All this amounts to a one conclusion. Reverse engineering and recompiling the game with new technologies is not impossible, but one would have to spend roughly 5-6 thousand hours of work on it. What is affordable and could actually work, is remaking the game from scratch, based on what we know, and how it plays. Engines like Unity, Unreal Engine 4, or Godot have the potential to make this real. However that said, I personally believe Godot would be the most suitable engine to do it, simply because how simple it's structured, how easy it is to learn, and how full of content it is. With Unity you will need more than just unity, because the engine by itself does not have an IDE, and more often than not, you'll have to use a lot of middleware to achieve your goal. Unreal Engine 4 is a great tool for FPS games, and not so much for everything else, which shows in how the engine wants you to structure your game, and how many prefabs are oriented around FPS style of gamedev. Godot itself comes with it's own IDE, physics, 2D and 3D graphics, Visual Scripting for non-programmers to make code, and is very simple in how it handles game design. Not only that, but it's completely free. No royalties, no pay-up-fronts, no per-seat licences. Of course, if you want to you can make a game in whatever engine you want it to run, or even make your own engine from scratch too.
TLDR; I did reverse engineering on the FAF executable, no you cannot modify it to make it run better, no you cannot swap files from SC2 because the two games are structured in an entirely different way, it would be easier and faster to remake FAF in some new game engine.
-
@DragonStrike406 said in Large Unit Clumping and Unresponsiveness:
I found this out the hard way. While the DLL files are a mix of Lua and C++, and the executable itself is C++ only, that is still all I can say. After disassembling I did not find any names I could put to use such as function calls, function names, file names, etc. Everything was in deep x86 assembly, and even when translated to C++, it was basically the same useless code. I say useless because it is. You cannot get anything useful from it, no names, no function structures, no nothing. One big gibberish. It's exactly the same thing with the libraries stored in the DLL files. They too are a massive pile of organized mess with no names.
I can mostly agree with this (which just a few minutes of looking). So far I could get some useful class names, but not really useful function names or implemenations. The decompiled implementations in C++ where pretty much word-by-word translations of the assembly - aka useless.
As to swapping FAF libraries with that of SC2, that will not work either. While yes, the base engine is the same, the two versions vastly differ, which I can say from just looking at how the starting hexes of each executable are laid down. Both are the SpringRTS engine, yes, but one is from late 2007 while the other is from early 2010.
I wonder, what makes you conclude that these both are derivations of the SpringRTS engine?
But it doesn't take going into reverse engineering to notice that the two games are too different to swap any files with each other. If you go to SC:FA installation and check what's in it, then compare it to how SC2's structure is made you can come to a conclusion that they work very differently from each other if your IQ is higher than the room temperature in Celcius.
The initial file layout is not that insanely important. Their linking method might have changed, fooling you into thinking there is absolutely no commonality left - while there is - albeit very limited.
Additionally I never meant "trivial file swapping". I'm a software dev myself, I know things aren't this easy.TLDR; I did reverse engineering on the FAF executable, no you cannot modify it to make it run better, no you cannot swap files from SC2 because the two games are structured in an entirely different way, it would be easier and faster to remake FAF in some new game engine.
Well, it was an idea. But apparently a flawed one. I did not expect supcom 1 and 2 foundations to diverge this insanely much.
-
I wonder, what makes you conclude that these both are derivations of the SpringRTS engine?
My bad. The engine is a homebrew. Just my janky memory connecting the dots that SpringRTS was released in 2007, is an RTS engine, suffered from lack of multithreading, and that FA has exactly same issues. Plus the fact that if you go to wikepedia, it says it's related in external links. https://en.wikipedia.org/wiki/Supreme_Commander_2#External_links Then again, I checked that like a year or two ago, and back then I still had my memory department still functioning correctly.
The initial file layout is not that insanely important. Their linking method might have changed, fooling you into thinking there is absolutely no commonality left - while there is - albeit very limited.
I also did say that after looking at the hexes of both games I noticed they are different. Maybe there is a way to put them together, although without any source code, or at least knowing what wants what, that's basically guessing.
Additionally I never meant "trivial file swapping". I'm a software dev myself, I know things aren't this easy.
Ah, a man of culture. Then I needn't explain more.
Well, it was an idea. But apparently a flawed one.
Ideas are flawed, mistakes are made, but there's always a lesson to be learnt. Have a good day, commander.
-
@WackelKontakt said in Large Unit Clumping and Unresponsiveness:
@Katharsas Given how insaenly fast CPUs nowadays are compared to 2007, even in single core, when used properly, I'd not even try to attempt any hacking to "multithreading" in (which wouldn't help much, when the core engine doesn't support it and you end up doing pointless tiny calculations multithreaded without any gain).
Instead, one should attempt to revive the work to modify luajit-GC to make it fit with supcom. This should (at least according o micro benchmarks of luajit) in some cases give up to 4x or even more performance.
I actually take a look at ghidra right now. It's an amazing tool, however making sense of all will take time (and stamina).
Initially I will take a look at both, supcom 1 and supcom 2 mohoengine.dll. If its not entirely easily possible to replace supcom 1 pathing implementation with an own one, maybe there is a way to give supcom 1 at least supcom 2 path finding.
Yeah it might be stupid to try to multithread anything. But its a complaint that comes up quite often, and it fits with the engine related complaints. The thing i fear about with LuaJIT is determinism. I have a feeling that might make it impossible to use a JIT in general.
I have fiddled with Ghidra, and i suggest to use Ghidra plugins like OOAnalyzer, because it apparently can auto-reverse typical C++ structures (vtables tec.), but havent tried that yet.
The bigger problem is collaboration. There is a "Ghidra Server" out of the box. However:
A) That server implements its own version control system for Ghidra project files (custom stuff, not Git)
B) You can only connect to that with Ghidra, however Ghidra is new and i guess a lot of people still use IDA, so they wounld have to switch to Ghidra to be able to contribute.
C) Somebody needs to look into the legal siuation and potential contributers need to know if/how they can stay clean.Not unsurmountable though.
Edit:
Hadnt read DragonStrike406 posts yet, thanks for sharing sop much experience.I fully expect a reverse engineering project to take multiple 1000 hours. Howver, i expect a full reimplementation of the engine to take just as much time if not longer, it it wants to surpass current engine and not just be an alternative with other problems than the current engine. Imo a new engine is pointless unless it works a drop-in replacement for the old engine. This means that physics need to behave exactly like in current engine. If not, FAF community will not switch or split, and then its worse than before.
It would be massive undertaking either way. Cant we actually start reversing at the points were calls into known DLLs are made? Graphics engine calls into DirectX, Audio calls, file system, engine Lua functions.
Just reversing the audio engine might be actually super usefull as a start, given how many problems the game has with Audio.
The more is reversed, the easier it gets, hardest is the start. And i dont suggest a single person to do this. It MUST be a group effort for there to be any chance of success, with makes the collaboration topic critical to success.
Maybe we should wait 10 years until Ghidra server-side projects work with Git and IDA has plugin for it.
-
A couple of things.
Let's not compare Supcom and Starcraft.
Or hey, maybe let's.
Starcraft 2 maps are max 256x256 grid units.
A 20x20 Setons is 1024x1024 grid units. That's 16 times as big, and it's the middle size.Starcraft maps are laughably small by comparison, and the maximum you can bandbox is an absolute joke; you can easily run a pixel-perfect Dijkstra or A+ discovery for each selected unit in between frames without skipping a beat, and it's likely exactly what gets done. It's a one-banana problem.
FA's pathfinding algorithm is much more constrained simply due to scale, and the optimizations built into it are very apparent at scale. And is definitely NOT inefficient; it simply bogs down due to collision detection at the start of a mass move. Everything freezes in a dining philosophers' deadlock until free edges are opened. Starcraft's algo would suffer exactly the same fate were it even possible to select 200 units at a time to begin with.
Can it be optimized?
Of course it can. I'd like to see anyone here try, though.
-
So its not impossible just beyond range of most of us. Dragon is my understanding correct?
-
“middle-sized”
40x40 and up exist on FAF, but the game (on FAF) certainly isn’t designed for those sizes. Game balance quickly breaks beyond 20x20 simply due to the ability to scale.
-
The pathfinding in Starcraft 2 is based on flocking behavior, in many ways it is similar to the pathfinding in Supreme Commander 2 and therefore not comparable to the pathfinding in Supreme Commander.
Sending a group of 200 units across the map would cost about the same of sending a single unit: a path is computed for each bundle of units (which is generally 1 large bundle, or when a micro is in play maybe 3 - 4) and the units will try and follow the path while at the same time keeping their distance between other units. It is the algorithm that the engine developer suggests in the interview with Annihiline:
If you'd like to know more about flocking behaviors, steering behaviors are quite similar but more applicable to games. See also:
I haven't reverse-engineered anything myself and don't intent on doing so after looking at the output of Ghidra with the faf executable as input. Implementing an alternative pathfinding system is a challenging task, let alone designing it in such a way that it does not change the behavior of units too much from how they behave right now. Then of course make it bug-free. And then there is the reverse-engineering part on top of that to make it fit with the rest of the building blocks.
Taking DragunStrike and Katharsas on their word, it is an almost impossible task.
-
@FtXCommando said in Large Unit Clumping and Unresponsiveness:
“middle-sized”
40x40 and up exist on FAF, but the game (on FAF) certainly isn’t designed for those sizes. Game balance quickly breaks beyond 20x20 simply due to the ability to scale.
In my opinion that's an interplay between the sheer size of the map grid, and the fact that T1 is a thing. Case in point, the other day I played a 40x40 world dom for the first time, and mistakenly treated it like ladder. By the time my spam got anywhere, dudes were already at t2 and climbing. So I spammed more... and thus.
-
I mean, looking at how many people have been interested in improving the engine over time, there imo is a chance if we split up the task. Start with small parts of the engine that are as indepedent from the rest as possible. Therfore audio engine. Use such a small project to test the waters.
But there seems to be a culture of "lone wolfs" in RE. People often work alone? Probably because collaboration tools for RE suck or are not mature yet or cost too much. Maybe because there is somtheing inherent to REing that makes collaboration hard. Maybe because people are so afraid to even try doing it because of some notion of "REing is an ancient hacker skill that only the chosen ones are able to wield". Its weird. If you can undestand the mess that is Javascript or even CSS, you are intelligent enough to read typical x86 Assembly just fine.
-
All this talk of overhauling the pathfinding system and I find myself wondering - just because we can do it, does it mean we should? Supcom 2/StarCraft II are held up as paragons of efficient pathfinding, but is their pathfinding necessarily better, beyond a technical level, than what we currently have? Having played both a lot of Supcom FA and Supcom 2, I personally think the way the tanks move in Supcom FA have a higher degree of realism to them. The inefficiencies of the pathfinding system actually lend a weightiness to the units, and make them react in a way that feels more realistic. In Supcom 2, tanks keep their distance between one another as if controlled by a hive mind and move with instantaneous fluidity. Again, it's more of a design decision, but I don't think it makes a lot of sense for my platoon of tanks to move like a shoal of sardines that are schooling. Just my 2 cents.
