Simspeed: Revive work on modifying luajit to work with FAF

WackelKontakt

Asking to revive a dead project again: Can we maybe ask Jeff Petkau on the changes they did on the lua vm clone and potentially replicate those?

splitice

This would be absolutely amazing. In our testing (non game related) we found luajit to perform at-least 2x faster. In many of our benchmarks it was 4x as fast (although this varied and was specific to areas that luajit did well).

x86 (32bit) luajit was slightly worse than x64 but the difference wasnt much (~10%)

Askaholic

You don’t need to ask anybody, I’ve already recreated the major changes: https://github.com/FAForever/lua-lang

splitice

@Askaholic

Is lua-lang in use for anything currently?

Looking through git the only thing I could find was the unit testing (syntax only?).

What method was used to detirmine the changes (from GPG) to lua. Was it just known language changes?

The reason I ask is because I've got some experience with binary patching and (more specifically) in-memory binary patching. It's been about 15 years but I did do some work in that area in the past.

Patching in luajit could begin first by patching memory patching in an in-house build of lua (interpreter) that is supposedly compatible (as a PoC).

Askaholic

It’s currently being used to run a small number of unit tests on the main game repo.

I tried to write it so that each git commit more or less corresponds to one language change to help document what is touched by each feature. Mostly I started with the language features that have been empirically discovered by the modding community but I also used some knowledge that we have about the binary itself. I know there are still some missing changes which I have documented in the GitHub issues, and as of yet I have not tested if the lua byte code produced by the FAF lua is identical to that produced by the binary. Mostly this is due to time constraints and I just got to a point where it was sufficient for running the unit tests I wanted to write.

If you want to talk about more technical details you can message me on discord as well.

splitice

Thank you @Askaholic those issues are great. I'll likely take you up on that when I get to that stage (or if something I do wouldnt be best shared with the community).

One way to look for differences between your Lua and the Games lua would be to compare the output of the globals.

    local seen={}

    function dump(t,i)
        seen[t]=true
        local s={}
        local n=0
        for k in pairs(t) do
            n=n+1 s[n]=k
        end
        table.sort(s)
        for k,v in ipairs(s) do
            print(i,v)
            v=t[v]
            if type(v)=="table" and not seen[v] then
                dump(v,i.."\t")
            end
        end
    end

    dump(_G,"")

My thoughts regarding a process to do this would be.

Phase 1. Move to in-house lua (PoC only)

Build a compatible (compare export list) LuaPlus + Lua dll as a drop in for LuaPlus_1081.dll

This will fail if there are any missing changes (and it sounds like there will be). Implementation of whats missing. Also need to look for any modifications that may have been made to LuaPlus.

Get "roughly playable" proving that the concept is possible.

This PoC wouldnt have to be perfect, the goals would be:

To identify any otherwise unknown changes by GPG
To verify the dll replacement method

Stage 2 (Luajit)

Swap the library to luajit and move the lua cfunctions to luajit. luajit is ABI compatible so this is actually pretty easy.

Deal with any ABI or language differences. There are some nuances for luajit applications none that I think will apply here (mostly with the calling of cfunctions from lua that call back into lua)

At this stage it's a suitable optional drop-in and we can focus on getting it properly playable and bug free

Stage 3 (further acceptance and performance)

Optimize the code for luajit. We now have a modern Lua variant

splitice

Also step 0, get LuaPlus compiling. After spending a fair portion of today just trying to get its build system to work (with both VS 2017 and VS 2019) it's not a simple problem.

Jip

KionX is still working on LuaJit on Zulip. He has no Discord and I don't think he's on the forums. He says he's nearly there - but before we can release it we'll need to make sure it is stable so it can still easily take a while before the general population gets to see it. Hard to pin point an exact date on it .

splitice

Ended up manually throwing together a testing bundle (expected a failure). And it worked without a problem. I looked into it and I was wrong it wouldnt be as simple as a binary compatible DLL. Neither MohoEngine.dll or LuaPlus_1081.dll are actually used by Supcom. Both are statically (lib) compiled into the exe.

Doh.