I now have 1400 ranked 1v1 and sentons games simulated of average ratings > 800
https://forum.faforever.com/post/69369
This is well structured dataset with an example of contents shown in above link and I am sure some good balance analysis can be done on some of this data now
Please pm me for the full dataset, and I will continue to grow it
I now have a few gigabytes from 1400 games of 1v1 ranked and sentons of structured data. If you would like this data just PM me. The data format is below:
An index file is created that shows all the replays, this is created from the replay file
replay_id scfa_version replay_version has_mods has_ai has_unit_restrictions cheats_enabled ranked game_start_epoch game_end_epoch no_players no_teams scenario_type victory_condition teams_locked unranked map_folder_name sim_ticks desync_tick share average_rating
21013072 3770 Replay v1.9 False False False False False 1697735541 1697737795 8 2 skirmish demoralization True False SCMP_009 19991 0 FullShare 1883
21013451 3770 Replay v1.9 False False False False True 1697737205 1697738254 2 2 skirmish demoralization True False frithen.v0004 715 0 FullShare 1853
21021738 3770 Replay v1.9 False False False False False 1697830981 1697834029 8 2 skirmish demoralization True False SCMP_009 24578 0 FullShare 977
21029660 3770 Replay v1.9 False False False False True 1697906403 1697906892 2 2 skirmish demoralization True False neroxis_map_generator_1.9.0_aj5umyuf4xuk6_aqea 4512 0 FullShare 1456
an index file is created for all the games that have been simulated
replay_id desync end_of_replay_reached sim_time_seconds
21013072 False True 3868
21013451 False True 162
21021738 False True 4460
21029660 False True 558
21029783 False True 6580
each game has an armies file generated directly from the replay file
army_name player_id player_name rating_mean rating_deviation faction team start_spot player_color human ai_personality
ARMY_2 413862 cZARJc 2048 95 4 3 2 6 True
ARMY_4 420129 HorohLoxMato 2123 94 4 3 4 19 True
ARMY_8 145 ZLO 2359 71 1 3 8 17 True
ARMY_3 393899 GULMO 2242 94 1 2 3 9 True
ARMY_7 372109 Terarii 2543 95 1 2 7 7 True
ARMY_6 304809 ZmeiGorinich 2099 94 1 3 6 1 True
ARMY_5 486930 el_PISTOLERO 2027 97 2 2 5 2 True
ARMY_1 278651 ERROR_4O4 1831 94 2 2 1 3 True
each game has an armies result file generated after simulation is complete
army_index army_name result
1 ARMY_1 won
2 ARMY_2 defeated
3 ARMY_3 won
4 ARMY_4 defeated
5 ARMY_5 defeated
6 ARMY_6 defeated
7 ARMY_7 won
8 ARMY_8 defeated
Each game has a economy.tsv file
GameTick Army lastUseRequested.MASS lastUseRequested.ENERGY lastUseActual.MASS lastUseActual.ENERGY stored.MASS stored.ENERGY income.MASS income.ENERGY maxStorage.MASS maxStorage.ENERGY reclaimed.MASS reclaimed.ENERGY
13 1 0.00 0.00 0.00 0.00 650.00 3924.00 0.10 2.00 650.00 4000.00 0.00 0.00
20 1 0.00 0.00 0.00 0.00 650.00 3938.00 0.10 2.00 650.00 4000.00 0.00 0.00
27 1 0.00 0.00 0.00 0.00 650.00 3952.00 0.10 2.00 650.00 4000.00 0.00 0.00
34 1 0.00 0.00 0.00 0.00 650.00 3966.00 0.10 2.00 650.00 4000.00 0.00 0.00
46 2 0.00 0.00 0.00 0.00 650.00 3990.00 0.10 2.00 650.00 4000.00 0.00 0.00
52 3 0.00 0.00 0.00 0.00 650.00 4000.00 0.10 2.00 650.00 4000.00 0.00 0.00
59 3 0.00 0.00 0.00 0.00 650.00 4000.00 0.10 2.00 650.00 4000.00 0.00 0.00
66 3 0.00 0.00 0.00 0.00 650.00 4000.00 0.10 2.00 650.00 4000.00 0.00 0.00
each game has a units.tsv file. FractionComplete is % built
GameTick Army EntityId UnitId FractionComplete Position.x Position.y Position.z MaxHealth Health ShieldRatio IsIdle MassKilled
13 1 0 uel0001 1 63.50 25.53 468.50 12000 12000 0 1 0
19 2 1048576 xsl0001 1 448.50 25.53 43.50 11500 11500 0 1 0
67 1 0 uel0001 1 63.50 25.53 468.50 12000 12000 0 1 0
67 1 1 ueb0101 0.04 67.50 25.56 454.50 4000 148 0 1 0
73 2 1048577 xsb0101 0.06 440.50 25.58 46.50 3500 199 0 1 0
73 2 1048576 xsl0001 1 448.50 25.53 43.50 11500 11500 0 1 0
109 1 0 uel0001 1 63.50 25.53 468.50 12000 12000 0 1 0
109 1 1 ueb0101 0.18 67.50 25.56 454.50 4000 708 0 1 0
I am not watching them in a normal way and am opening them up with script tools without the FAF client involved
I am trying to watch some older ranked replays and am missing the below list of map folders:
mystic_land.v0008
torn_desert.v0012
esais_vengeance_1v1.v0008
black_winter_night.v0006
black_winter_night.v0005
clarkes_retreat.v0005
the land wilderness
little_pass_x05.v0003
plana_mons_pugna.v0005
desertarena.v0002
distortion.v0017
sandstorm.v0002
cobalt_canyon_v2
cloudscape.v0008
mountaineous_hill.v0004
tamara_pass.v0002
hjemsokt.v0006
mountaineous_hill.v0003
adaptive_esais_vengeance.v0006
adaptive_tourmaline.v0003
Does anyone have them?
Data driven balance
What units, built when, actually win you the game when you automatically review hundreds of replays with a python CLI program running 5 games at once on a linux server?. See https://forum.faforever.com/topic/7746/unit-logger-mod/2?_=1756499882452
I now have a few gigabytes from 1400 games of 1v1 ranked and sentons of structured data. If you would like this data just PM me. The data format is below:
An index file is created that shows all the replays, this is created from the replay file
replay_id scfa_version replay_version has_mods has_ai has_unit_restrictions cheats_enabled ranked game_start_epoch game_end_epoch no_players no_teams scenario_type victory_condition teams_locked unranked map_folder_name sim_ticks desync_tick share average_rating
21013072 3770 Replay v1.9 False False False False False 1697735541 1697737795 8 2 skirmish demoralization True False SCMP_009 19991 0 FullShare 1883
21013451 3770 Replay v1.9 False False False False True 1697737205 1697738254 2 2 skirmish demoralization True False frithen.v0004 715 0 FullShare 1853
21021738 3770 Replay v1.9 False False False False False 1697830981 1697834029 8 2 skirmish demoralization True False SCMP_009 24578 0 FullShare 977
21029660 3770 Replay v1.9 False False False False True 1697906403 1697906892 2 2 skirmish demoralization True False neroxis_map_generator_1.9.0_aj5umyuf4xuk6_aqea 4512 0 FullShare 1456
an index file is created for all the games that have been simulated
replay_id desync end_of_replay_reached sim_time_seconds
21013072 False True 3868
21013451 False True 162
21021738 False True 4460
21029660 False True 558
21029783 False True 6580
each game has an armies file generated directly from the replay file
army_name player_id player_name rating_mean rating_deviation faction team start_spot player_color human ai_personality
ARMY_2 413862 cZARJc 2048 95 4 3 2 6 True
ARMY_4 420129 HorohLoxMato 2123 94 4 3 4 19 True
ARMY_8 145 ZLO 2359 71 1 3 8 17 True
ARMY_3 393899 GULMO 2242 94 1 2 3 9 True
ARMY_7 372109 Terarii 2543 95 1 2 7 7 True
ARMY_6 304809 ZmeiGorinich 2099 94 1 3 6 1 True
ARMY_5 486930 el_PISTOLERO 2027 97 2 2 5 2 True
ARMY_1 278651 ERROR_4O4 1831 94 2 2 1 3 True
each game has an armies result file generated after simulation is complete
army_index army_name result
1 ARMY_1 won
2 ARMY_2 defeated
3 ARMY_3 won
4 ARMY_4 defeated
5 ARMY_5 defeated
6 ARMY_6 defeated
7 ARMY_7 won
8 ARMY_8 defeated
Each game has a economy.tsv file generated after simulation is complete
GameTick Army lastUseRequested.MASS lastUseRequested.ENERGY lastUseActual.MASS lastUseActual.ENERGY stored.MASS stored.ENERGY income.MASS income.ENERGY maxStorage.MASS maxStorage.ENERGY reclaimed.MASS reclaimed.ENERGY
13 1 0.00 0.00 0.00 0.00 650.00 3924.00 0.10 2.00 650.00 4000.00 0.00 0.00
20 1 0.00 0.00 0.00 0.00 650.00 3938.00 0.10 2.00 650.00 4000.00 0.00 0.00
27 1 0.00 0.00 0.00 0.00 650.00 3952.00 0.10 2.00 650.00 4000.00 0.00 0.00
34 1 0.00 0.00 0.00 0.00 650.00 3966.00 0.10 2.00 650.00 4000.00 0.00 0.00
46 2 0.00 0.00 0.00 0.00 650.00 3990.00 0.10 2.00 650.00 4000.00 0.00 0.00
52 3 0.00 0.00 0.00 0.00 650.00 4000.00 0.10 2.00 650.00 4000.00 0.00 0.00
59 3 0.00 0.00 0.00 0.00 650.00 4000.00 0.10 2.00 650.00 4000.00 0.00 0.00
66 3 0.00 0.00 0.00 0.00 650.00 4000.00 0.10 2.00 650.00 4000.00 0.00 0.00
each game has a units.tsv file. FractionComplete is % built, generated after simulation is complete
GameTick Army EntityId UnitId FractionComplete Position.x Position.y Position.z MaxHealth Health ShieldRatio IsIdle MassKilled
13 1 0 uel0001 1 63.50 25.53 468.50 12000 12000 0 1 0
19 2 1048576 xsl0001 1 448.50 25.53 43.50 11500 11500 0 1 0
67 1 0 uel0001 1 63.50 25.53 468.50 12000 12000 0 1 0
67 1 1 ueb0101 0.04 67.50 25.56 454.50 4000 148 0 1 0
73 2 1048577 xsb0101 0.06 440.50 25.58 46.50 3500 199 0 1 0
73 2 1048576 xsl0001 1 448.50 25.53 43.50 11500 11500 0 1 0
109 1 0 uel0001 1 63.50 25.53 468.50 12000 12000 0 1 0
109 1 1 ueb0101 0.18 67.50 25.56 454.50 4000 708 0 1 0
I simulated 1436 sentons and raned 1v1 replays with this tool.
you can determine if the first team to make a stratiegic bomber actually has an increased chance of winning
You can determine the average mass killed of a novax after 10 mins of it being built
you can see if a t1 frigate rush is a good stratergy to win an ocean against players rated 200 less points than the oponent, and you can see what factions are best in the beech position and win that ocean, with unit position data.
You can see how long it takes each game ender, on average to end the game and chance of winning after it has been up for 5 mins.
If you would like the raw unit data please message me, its a few hundred mb. I would like some people to go over it and try and pick out these correlations
After creating this mod
I created some python code that runs multiple instances of FAF at once on a linux server with this mod enabled and goers through a folder full of replays at high gamespeed generating this data.
I created a seperate python program to parse out the log into an unitfile displaying
"GameTick,Army,EntityId,UnitId,FractionComplete,Position.x,Position.y,Position.z,MaxHealth,Health,ShieldRatio,IsIdle,MassKilled"
and an economy file displaying
GameTick,Army,lastUseRequested.MASS,lastUseRequested.ENERGY,lastUseActual.MASS,lastUseActual.ENERGY,stored.MASS,stored.ENERGY,income.MASS,income.ENERGY,maxStorage.MASS,maxStorage.ENERGY,reclaimed.MASS,reclaimed.ENERGY
This was done a year ago or so. I have processed 659 replays creating 12GB of CSV files (compressed to ~ 300mb) of unit positions with time, player economies, and if players won or lost. From this data you can absolutley get insight into what units built at what time correlate to a win. If you would like this data please message me.
Example data is attached.
I have created a mod that dumps unit information and economy information into the game log. It is on the vault called unit logger.
I created some python code that runs multiple instances of FAF at once on a linux server with this mod enabled and goers through a folder full of replays at high gamespeed generating this data.
I created a seperate python program to parse out the log into an unitfile displaying
"GameTick,Army,EntityId,UnitId,FractionComplete,Position.x,Position.y,Position.z,MaxHealth,Health,ShieldRatio,IsIdle,MassKilled"
and an economy file displaying
GameTick,Army,lastUseRequested.MASS,lastUseRequested.ENERGY,lastUseActual.MASS,lastUseActual.ENERGY,stored.MASS,stored.ENERGY,income.MASS,income.ENERGY,maxStorage.MASS,maxStorage.ENERGY,reclaimed.MASS,reclaimed.ENERGY
This was done a year ago or so. I have processed 659 replays creating 12GB of CSV files (compressed to ~ 300mb) of unit positions with time, player economies, and if players won or lost. From this data you can absolutley get insight into what units built at what time correlate to a win. If you would like this data please message me.
https://forum.faforever.com/topic/7746/unit-logger-mod?_=1756499882431
I have created the "Unit Logger" mod. It is in the mod vault. It is a UI mod.
It logs information on every unit in the game log of the focus army every 10 game ticks, configureable in main.lua
Information logged:
"UL02","GameTick","EntityId","Army","UnitId","FractionComplete","Position.x","Position.y","Position.z","MaxHealth","Health","ShieldRatio","IsIdle","MassKilled"
Log size for long games is normally a few MegaBytes
It works in replays if you select the army of the player you require logging.
It requires UMT / UI Mod Tools / ui-mod-tools-4z0t-v12
Thanks for @Ctrl-K / 4z0t for some of the code that makes this work
This data is not in an compact format. It will compress at around ~ 8>1 ratio
Build order can be extracted from this data - and not just the time units / buildings are started or commands given but when units are finished and where they actually are on the map.
using an external tool unit icons could be plotted onto a map with this data for a kind of "replay prieview" function that can be scrolled fowards and back in time, without having to launch the FAF executable.
Does anyone know about selecting all units while in observer? I would like to convert replay files to this unit position format, and currently I can only run one army at a time.
Example data:
info: UL02 GameTick EntityId Army UnitId FractionComplete Position.x Position.y Position.z MaxHealth Health ShieldRatio IsIdle MassKilled
info: UL02 1350 321 1 uea0303 1 171.915 35.555 717.439 2300 1550 0 0 180
info: UL02 1350 421 1 uea0303 1 231.838 34.695 652.384 2300 300 0 0 482
info: UL02 1350 344 1 uea0303 1 109.883 35.376 720.888 2300 2050 0 0 45
info: UL02 1350 369 1 uea0303 1 142.596 38.140 688.592 2300 1300 0 0 45
info: UL02 1360 736 1 uea0303 1 111.701 36.286 717.892 2300 1550 0 0 270
info: UL02 1360 0 1 ual0001 1 108.207 25.830 920.259 11000 11000 0 1 0
info: UL02 1360 262 1 uea0303 1 98.083 37.232 730.919 2300 1550 0 0 450
info: UL02 1360 280 1 uea0303 1 97.939 36.571 708.237 2300 1050 0 0 90
info: UL02 1360 364 1 uea0303 1 145.077 37.485 686.651 2300 1800 0 0 225
info: UL02 1360 318 1 uea0303 1 142.312 37.525 684.928 2300 1550 0 0 360
info: UL02 1360 438 1 uea0303 1 196.809 37.850 668.522 2300 1550 0 0 433
An example log from a high ranked setons game
replay ID 22654992
from the south air player from 0 mins to 20 mins in
20 mins of a setons game is now only 1.6MB log size with the floating point decimal trimming
test.log
I have created the "Unit Logger" mod. It is in the mod vault. It is a UI mod.
EDIT - V2 Released with reduced decimal places and unit masskilled added
Logs information on every unit of of focus army every 10 game ticks, configureable in main.lua
Information logged:
"UL02","GameTick","EntityId","Army","UnitId","FractionComplete","Position.x","Position.y","Position.z","MaxHealth","Health","ShieldRatio","IsIdle","MassKilled"
WARNING: If you have 1500 units and an hour game expect gigabyte size logs.
It works in replays if you select the army of the player you require logging.
It requires UMT / UI Mod Tools / ui-mod-tools-4z0t-v12
Thanks for 4z0t for some of the code that makes this work
This data is not in an eficcient format at all - that can be done after the game has ended.
I need to look at logging all units from all armies at once when observer. i dont know if this is possible with a UI mod.
Example data:
info: UL02 GameTick EntityId Army UnitId FractionComplete Position.x Position.y Position.z MaxHealth Health ShieldRatio IsIdle MassKilled
info: UL02 1350 321 1 uea0303 1 171.915 35.555 717.439 2300 1550 0 0 180
info: UL02 1350 421 1 uea0303 1 231.838 34.695 652.384 2300 300 0 0 482
info: UL02 1350 344 1 uea0303 1 109.883 35.376 720.888 2300 2050 0 0 45
info: UL02 1350 369 1 uea0303 1 142.596 38.140 688.592 2300 1300 0 0 45
info: UL02 1360 736 1 uea0303 1 111.701 36.286 717.892 2300 1550 0 0 270
info: UL02 1360 0 1 ual0001 1 108.207 25.830 920.259 11000 11000 0 1 0
info: UL02 1360 262 1 uea0303 1 98.083 37.232 730.919 2300 1550 0 0 450
info: UL02 1360 280 1 uea0303 1 97.939 36.571 708.237 2300 1050 0 0 90
info: UL02 1360 364 1 uea0303 1 145.077 37.485 686.651 2300 1800 0 0 225
info: UL02 1360 318 1 uea0303 1 142.312 37.525 684.928 2300 1550 0 0 360
info: UL02 1360 438 1 uea0303 1 196.809 37.850 668.522 2300 1550 0 0 433
I am trying to understand how to make a mod that captures information about units through the replay
Looking at the LUA documentation:
https://supcom.fandom.com/wiki/LUADOC_1.5.3599#User
https://github.com/FAForever/fa/blob/develop/engine/User/UserUnit.lua
does using User.UserUnit make it a UI only mod?
For the dump to file script
if unit is alive (using IsDead)
if it is dump a line to log that contains
GetGameTick
GetArmy
GetUnitId
GetPosition
GetOrientation
GetVelocity
GetMaxHealth
GetHealth
GetShieldRatio
IsIdle
UserUnit
repeat every X seconds.
Could dump some unit types info at a faster rate than others, depending on unit speed, or importance for instance.
Do sim mods have access to any files that are not the game log?
Could just add an identifier start charicter to each line of the game log however.
Data could then be put into a better datasctructure and compressed using other external tools / scripting
Is this much script to write? is anyone here able to do so? @maudlin27
Thanks for all the info @maudlin27
While I am sure I could figure out some python scripting around launching the faf.exe and replays automatically
The exporting information to a text file is beyond my skillset
Is it possible to run the FAF EXE in a windowless or headless graphics free mode?
Is there a documented list of command line options avalable to the executable?
@sainserow said in Add a latest replay URL:
@nooby The first part is probably possible, Kazbek faf analysis was doing something similar. You could scroll the entire game directly on the website in 15-30 seconds and units were displayed at yellow dots. The more units in the same area, the bigger the dots. You could basically see map control. But it didn’t distinguish t1 scout and gc in these dots.
I think it was analysing commands, as Askaholic stated, rather than where the units actually were.
I dont understand what you mean by creating a new replay from the old one?
The purpose is to generate unit position, identifcation and time data that can be used to get some insight into the way the game is played with other tools.
thanks for all this info - It was very easy to get the AI automation script running.
To use this script to anaylse replays could some LUA script get all the existing units and there locations every 10? seconds and dump it to a text file?
And then just modify the script to run replays instead of make new games?
I would like to automatically review some replays (units built, when they were built) in certian rating brackets on certian maps using askaholics python replay parser and get a win / loss from each one using the API. Is there no way to do this now?
@brutus5000 said in Add a latest replay URL:
No it would not be good to add logic to this endpoint. Nobody is supposed to filter on the replay.faforever.com domain.
Just (learn to) use our api instead.https://api.faforever.com/data/game?filter=replayAvailable==true&sort=-id&page[limit]=1 gives you the latest present replay in the
replayUrlfield.
Has the API been locked down? I am getting a permissions error on this link.
I dont like the teleport mechanic, but I think it should be left in the game. I think that a good way to balance it would be that enemy teleports within omni range it is pinged as a nuke ping would be, rather than just the visual icon. This would give a few more seconds of time to move units into position.
Teleport in higher rated teamgames does not improve the game, does not require much skill and does not carry much risk.
Has anyone got some hih level replays ID showing the power of this?
I would prefer if teleport required a large amount of instant energy in use - Like the eye of riahhne. This would make it easier to scout by the large ( think 20 + ) E storages required.
Making it more expensive use wise just means you can stall your eco for a few more seconds then forcing storage.