I recently picked up reverse engineering the FAF executable and libraries.
If you want credibility to my claims, I worked on two game reverse engineering projects in the past. One on Robocraft, where I had my part in deobfuscation of a massive C# assembly, and the other was the HAWaKENing project where I assembled a team which goal was to make Hawken run on PC again. This was no easy task because the game was only multiplayer, and was shut down on January 2nd 2018. We had to not only disassemble the executable and find out how it's inputs work, we also had to make a login and services server from absolute scratch. We did succeed to some extent, we are yet to get actual multiplayer working but we did manage to get the account server running and player can now even play against bots.
That's for my background, and as someone who works with IDA, Snowman Disassembler, and writes some of my own tools to help me in the process of making old software run again, I can attest that it would be much faster to remake the game using current-day technologies than it would be to improve the executable.
I found this out the hard way. While the DLL files are a mix of Lua and C++, and the executable itself is C++ only, that is still all I can say. After disassembling I did not find any names I could put to use such as function calls, function names, file names, etc. Everything was in deep x86 assembly, and even when translated to C++, it was basically the same useless code. I say useless because it is. You cannot get anything useful from it, no names, no function structures, no nothing. One big gibberish. It's exactly the same thing with the libraries stored in the DLL files. They too are a massive pile of organized mess with no names.
As to swapping FAF libraries with that of SC2, that will not work either. While yes, the base engine is the same, the two versions vastly differ, which I can say from just looking at how the starting hexes of each executable are laid down. Both are the SpringRTS engine, yes, but one is from late 2007 while the other is from early 2010. In that period of time, Square Enix could have easily implemented an entirely new compiler and change a lot of engine functions themselves to better suit their needs for the game. And they certainly did so, because one major difference that can be pointed out right out of the gate, is the fact SC2 benefits from running on multiple threads, while SC:FA's engine does not run the simulation this way.
But it doesn't take going into reverse engineering to notice that the two games are too different to swap any files with each other. If you go to SC:FA installation and check what's in it, then compare it to how SC2's structure is made you can come to a conclusion that they work very differently from each other if your IQ is higher than the room temperature in Celcius.
All this amounts to a one conclusion. Reverse engineering and recompiling the game with new technologies is not impossible, but one would have to spend roughly 5-6 thousand hours of work on it. What is affordable and could actually work, is remaking the game from scratch, based on what we know, and how it plays. Engines like Unity, Unreal Engine 4, or Godot have the potential to make this real. However that said, I personally believe Godot would be the most suitable engine to do it, simply because how simple it's structured, how easy it is to learn, and how full of content it is. With Unity you will need more than just unity, because the engine by itself does not have an IDE, and more often than not, you'll have to use a lot of middleware to achieve your goal. Unreal Engine 4 is a great tool for FPS games, and not so much for everything else, which shows in how the engine wants you to structure your game, and how many prefabs are oriented around FPS style of gamedev. Godot itself comes with it's own IDE, physics, 2D and 3D graphics, Visual Scripting for non-programmers to make code, and is very simple in how it handles game design. Not only that, but it's completely free. No royalties, no pay-up-fronts, no per-seat licences. Of course, if you want to you can make a game in whatever engine you want it to run, or even make your own engine from scratch too.
TLDR; I did reverse engineering on the FAF executable, no you cannot modify it to make it run better, no you cannot swap files from SC2 because the two games are structured in an entirely different way, it would be easier and faster to remake FAF in some new game engine.