Monthly AI Tourney Series

AI biweekly/monthly tourney: 20km Combined arms
The theme of this map was 20km maps that allow use of land, navy and air. The maplist chosen was:

• Crossfire Canal - FAF version
• Norfair
• The Ditch - FAF version
• Setons Clutch - FAF version
• The Bermuda Locket - FAF version

Setup
Setup is similar to the last tourney - 12 AIs are split into 4 groups. The weakest 4 are eliminated in the first round, with the winners from one group fighting the runners up from the other in the quarter-finals in 3 matches.
As there were 5 maps, maps were chosen randomly for the group and quarters, and then for the semis and finals each map was played.
Starting positions are the default first and second positions set by the map (e.g. on Setons this means a 'mid vs mid' battle)

Changes to AI
5 of the featured AI have released new versions since the last tourney: M27, Swarm Terror, SCTA Arm, Sorian Edit, SCTA Core

The results
Group stages

Quarter finals

Semi finals

Finals

Total wins by AI

Detailed results
The full results can be seen in the 'DetailedResults' worksheet of the following file:
https://docs.google.com/spreadsheets/d/1i2ObsZ49S9gUTgWw0E68qZ8VrOkaP_Jk/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=true

Other Observations

• While some AI attempted to build navy, in all cases it proved ineffective with the mass invested in navy far exceeding the mass killed. Neither of the finalists built a single naval unit the entire tourney.
• The finals were closer than the scoreline suggests, with DilliDalli dominating land on most of the maps and winning map control (e.g. holding 5 out of the 6 islands at one point on Crossfire Canals, and effectively raiding and even claiming mexes behind M27's main base on Setons). The finals ended up being an air vs land contest on most maps.
• SCTA Arm managed to have 2 T3 tanks solo a GC (GC's shot was blocked by cliffs) but it wasn't enough to let it claim a win
• Heroic unit of the tourney: DilliDalli's lone striker vs M27 on Setons which managed 19 kills and 1,720 mass kills raiding mexes
• Notable changes in fortunes: Swarm managed a turnaround in speed, going from the slowest AI to claim wins in the last tourney to getting the second fastest win (11m21, beaten only by Dilli who got a win in 10m48)

AI biweekly/monthly tourney: MapGen
The idea of this tourney was to use MapGen to create a map for the AI to fight against for 5km, 7.5km, 10km, 15km and 20km. Due to the AI tourney still being run manually a single map was used for the map size for the tourney (rather than generating a new random map every time). The maplist chosen was:

• neroxis_map_generator_1.8.5_vgf5tqx2uxrcu_aicae_
• neroxis_map_generator_1.8.5_uydyqpk4utuuu_aidae_
• neroxis_map_generator_1.8.5_xaghpcvo2lnrk_aieae_
• neroxis_map_generator_1.8.5_bvwpdifwrols4_aigae_
• neroxis_map_generator_1.8.5_44sxnthefcek6_aiiae_

Setup
Setup is similar to the last tourney - 12 AIs are split into 4 groups. The weakest 4 are eliminated in the first round, with the winners from one group fighting the runners up from the other in the quarter-finals in 3 matches.
As there were 5 maps, maps were chosen randomly for the group and quarters, and then for the semis and finals each map was played.

Changes to AI
3 of the featured AI have released new versions since the last tourney: M27, SCTA Arm, SCTA Core

The results
Group stages

Quarter finals

Semi finals

Finals

(M27 and RNG drew 1 game)
Total wins by AI

Detailed results
The full results can be seen in the 'DetailedResults' worksheet of the following file:
https://docs.google.com/spreadsheets/d/1i2ObsZ49S9gUTgWw0E68qZ8VrOkaP_Jk/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=true

Other Observations

• Notable changes in fortunes: In the final, RNG had c.25 mass to M27's 621, yet still managed a draw (M27 attacked RNG's base with its ACU leading to both ACUs dying)
• The 7.5km map featured two land masses separated by water, which caused several AIs issues
• Although SCTA failed to get a win it managed to match Sorian Edit for map control and eco for much of the game, with both games going over an hour
• The group stages required two deciding matches to determine groupings, with an extra Dilli vs Adaptive and Sorian Edit vs DilliDalli match due to them getting the same number of wins as each other.

AI biweekly/monthly tourney: Sorian vs the world
The idea for this tournament was to see the highest AiX modifier that each AI could beat Sorian Adaptive at, on a selection of 5km, 10km and 20km maps from the current map pool, hopefully highlighting how much more comeptitive the FAF custom AIs are, as well as giving a better gauge of the distance between different AI than the previous tournaments win rate. The maplist chosen was:

• Mirage
• Stickleback Ridge
• Open Palms
• 8 - Badlands_v4
• The Bermuda Locket - FAF version
• Crossfire Canal

Setup
Each AI entered will face off against Sorian Adaptive AiX (with no omni). If they win, then they repeat the fight but with Sorian Adaptive getting +0.1 to its modifier, until the challenging AI no longer wins. This is done for each map separately. This resulted in a mammoth 245 games being run, so I doubt I'll be repeating this format due to the massive amount of time it ended up taking (for comparison, the last tournament took just 51 games)

Changes to AI
There were significant changes to the AI this time. In addition to a new version of M27, Sorian Edit, and SCTA, there was a major update for RNG (the first in many months), and we have a new AI entrant with Uveso Rush featuring for the first time!
The results
Summary

Results by map

Detailed results
The full results can be seen in the Data worksheet of the following file:

https://docs.google.com/spreadsheets/d/1i2ObsZ49S9gUTgWw0E68qZ8VrOkaP_Jk/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=true

Other Observations

• Uveso was the dominant AI on badlands (due to its ability to punish Sorian Adaptive for sending its ACU to attack on its own early on), managing a win against a 2.2 AiX (it lost against a 2.1 AiX that was running at the same time hence why 2.0 is shown in its results). In contrast as an AiX 1.2 it lost to a 1.0 Sorian Adaptive on Crossfire Canal.
• Crossfire Canal was by far Sorian Adaptive's best map, with 3 of the AI (Sorian Edit, SCTA Core and Uveso) needing AiX modifiers to beat it (although not strictly correct, if an AI needed an AiX modifier to win I've shown each 0.1 modifier as decreasing the Sorian AiX beaten by 0.1, so an AI needing a 1.5 modifier to win would show as 0.5)
• Surprisingly, Adaptive AI was better than Sorian Adaptive, beating it on every map with no modifier, and beating 1.2 on Mirage.
• There were compatibility issues between Sorian Edit and Sorian Adaptive on some maps, so results for these have been omitted
• 3 AI managed the achievement of beating Sorian Adaptive AiX 2.0 or better - M27 and RNG (2.0, Stickleback Ridge) and Uveso (2.2, Badlands)

Edit: Note that some of the AI aren't intended for 20km or water based maps, which will be a reason for poor results on Crossfire (e.g. SCTA isn't meant for such maps). Excluding crossfire from the results would give the following:

AI monthly tourney: Survival of the fittest
Suggested by Fearghal, this tournament's theme is to see which AI does best on a survival map. chp2001 helpfully identified the map Final Rush Pro 5 as a potential candidate, since it features 1 team against another (thus avoiding the problem caused by survival maps which lack an enemy brain/base location for the AI to fight against).

For those not familiar with it, it's a 4v4 map where teams face off against each other with a central chokepoint area where the AI spawns. The AI spawns will be delivered by transport, but air is disabled (so the transports are harder to shoot down).

The survival spawns are split allocated equally between two different survival 'brains', and wont attack each other, but will attack one (and only one) of the teams. This means that a team can choose to be aggressive by sending their units at the enemy base, to help support the survival AI's attack.

To avoid people exploiting the survival AI, ACUs are prevented from hiding in the water by taking damage over time, and prevented from hiding in the rear corner of the map behind cliffs by a group of galactic colossi. The map has a number of configurable settings, including difficulty, which affects the size of the surivival spawns, and how long it takes before it advances in waves (with a wave starting off with T1 units, before progressively getting harder, e.g. sending in bombers; t2 land; cybran destroyers; t3 land; experimental land; etc.). Players start with 9 hives, an ACU from each faction, and lots of mass points at their base.

Setup
In the first round, each AI will play against itself (i.e. all 8 players will be that AI), and see how long it takes for the last player to die. These times will then be used to give each AI an initial 'seeding', with the last seed playing the second last, the winner of that playing the 3rd last, and so-on until the last ranked AI has lost to the 2nd last ranked AI which has lost to the 3rd last ranked AI and so on.

The map settings still present some issues for AI, in particular the logic for preventing exploiting the survival AI by hiding - I turned this off for the initial 'seeding' round over fears most of the AI wouldn't be able to understand the 'ACUs die when entering water' approach or that 'those hostile GC near your base can be ignored' , but it led to its own problems, e.g. fully upgraded ACUs hiding in the water being very slowly whittled down by the torpedoes of a couple of bricks and didnt give a fair reflection of how well some AI did. I therefore settled on a compromise for the main event, where a team would be treated as having lost if all 4 land bases were wiped out (i.e. it had no more mexes or land factories).

The difficulty was also set to easiest, as all AI struggle compared to humans with such a setup, and this way there was more of a chance of seeing waves hit the experimental land stage.

Changes to AI
New versions of M27, RNG, and SCTA were reflected for the tournament.
The results
Initial seedings - Time lasted when fighting itself

Final rankings and head to heads

For example, DilliDalli was initially seeded 9th (out of 10 AI); it lost to M27 (the 10th seed), but beat the 5 AI that placed worse than it in the final results to reach 5th place. It then lost to RNG (4th place), so didnt advance any further. In contrast, as Swarm was seeded first it only fought the 2nd place initially (which ended up being M27), and after losing that it then fought the then 3rd place (Uveso Rush) which it won, resulting in ending as 2nd place.

Detailed results
The full results can be seen in the Data worksheet of the following file:

https://docs.google.com/spreadsheets/d/1i2ObsZ49S9gUTgWw0E68qZ8VrOkaP_Jk/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=true

Other Observations

• There were a wide variety of approaches used by the different AI, for example:
• M27 favoured a firebase appraoch, but chose to place this near the centre of the map meaning it'd get overwhelmed around the time of T3 land or experimentals
• DilliDalli preferred an all-out assault on the enemy team before spawns even started arriving, sticking to T1 mexes but often being able to kill the enemy team before their economic advantage could be brought to bear
• Swarm chose to have a massive swarm of mostly T1 units occupying the central area, meaning they could quickly ambush the survival AI drops as they arrived (up until the late T3 land/early experimental stage)
• Uveso made heavy use of T1 artillery, which when firing from behind T3 shields at the base proved an effective roadblock to the survival AI's T3 land waves
• There were dramatic transformations in fortunes for the AI from their initial seedings. In DilliDalli's case this was because it's aggressive approach would lead to short matches in all its games, while in M27's case it had chain reactions caused by the various starting ACU (16 per team) being near the centre and one of them dying. Sorian Edit's reversal in fortunes was likely caused by the stricter measuring of when a team had lost, as it was usually better at retreating its ACU behind cliffs/into water where the survival AI couldnt reach it
• A number of games came down to the wire with one team losing its final base just before the opposing team lost its final base, including the final between M27 and Swarm, and the semi-final between Uveso Rush and Swarm
• SCTA performed strongly, with a survival time in head to heads of more than 30m (bettered only by Uveso, Swarm and M27), but DilliDalli's early aggression proved its undoing
• It looked like RNG would pull back from the brink in its game against Uveso Rush when it completed a cloaked laser upgrade on its ACU at its final base. Sadly it turns out that the map gives omni vision meaning its ACU still died quickly to the T3 land spam.

"This week I also pulled performance stats for each AI against itself in the two round 1 maps, which produced the results below:"

I have been testing for the upcoming RC tourny and have been using "mostly" M27 as the standard to
measure against the other Game specific AI's,
I also start test, at game option speed = Fast, (which is = to +4 sim) to make sure the game speeds are "not likely" to drop below "0" game speed.
Game speed limits the size of the maps I can use -
(if games slow to "-1", after 45 minutes, I limit the AI's to 10x10 maps)
The faster AI's like DilliDalli and RNG are easily able to handle the 20x20 maps.
Since the test games are a combo of the the Game AI and M27 -
I was wondering if with all the recent updates to M27 if it is still as fast as the performance results shown above.

I would test vs Uveso, (since it needs to be loaded for all games anyway)
but it seems that after Uveso grows it's base large it also slows game down a little,
so I don't want to run tests with Uveso as the standard, as it might skew the test results.
Update: I just tested Uveso since post and It runs fast.
It was still +4 at end of 37 min game with 6 AI:Uveso and Lots of units!

Would it be possible to include another performance chart after you run your next tourny,
with the updated AI versions?
Thanks.

Per Fearghal's request, this AI tourney is a bit different, and focuses on how fast/slow the different AI run. The scenarios were as follows:

• Adaptive Tournament Decision (10km): Rainbow cup style setup, with 3 teams (3v3v2)
• Rebellion (20km): Simple 2 team head to head with all slots filled (so a 5vs5)

Due to how long games were taking, I ran 2 games on adaptive tournament decision and only one on rebellion.
Unfortunately I forgot to add tick freeze tracking to my profiling code until I was almost done with the games and couldnt face re-running them (as on Rebellion it was taking more than 1 hour for some of the games in real time).

Results are shown both by time, and by unit count. The main point for me is this shows the largest factor in how slow the game will run is how many units are built. In particular, just looking at results by time DilliDalli is by far the quickest AI. However, looking at it by unit count there's little difference between it, RNG, SCTA and Adaptive. That said, Swarm, Uveso and M27 performed noticeably slower even when adjusting for unit count.

For the below graphs, the x axis is either the unit count or the game time in seconds. The y axis in both cases is how long in real-life seconds it took to run 10 game time seconds. I.e. a value of above 10 means that the average game speed was below +0.

Adaptive tournament decision

The large blue spike was a brief period when swarm reached -8 game speed

Rebellion

Unfortunately the current version of Uveso doesnt work with Sorian Edit (which requires Uveso) so no stats are available for this.

I also reran M27 to see how it did after the v47 optimisation release (c. 9% faster) but there wasn’t a noticeable difference in the graph lines so the above reflects the v45 position.

AI monthly tourney: 10km Analytics
Per a request by Relent0r, this tournament was aimed both at providing detailed stats for each AI (so each AI developer gets hopefully useful information), and to give weaker AI a boost to make the results less predictable. The current 1v1 ladder map TAG_Craftius Maximus has been selected since it allows navy to play a role without it being required (so both naval and non-naval AI get to compete), while it's also 10km which will be useful for those AI that aren't designed to function on 20km+ maps.

Setup
Each AI will fight each other AI once. The AI will get an AiX modifier based on their average placement in the last 3 tournaments, ranging from 1.0 for first place to 1.5 for last place (where more than 8 AI featured; 1.35 where only 8 AI featured). The actual AiX modifier will then be based on the difference between the AiX modifiers of the two competing AI (so for example DilliDalli will use an AiX 1.3 modifier, while RNG Standard will use 1.1; when fighting each other that means RNG 1.0 will fight DilliDalli 1.2). The AiX modifiers used are shown in the results.

Changes to AI
Almost all of the AI have featured an update since the last tournament.

The results
Summary

Congratulations to DilliDalli/Softles, with DilliDalli beating every other AI with it's tank focused game style.

Detailed results
The full results including the cleansed data used for the above graphs can be seen in the Data worksheet of the following file:

https://docs.google.com/spreadsheets/d/1i2ObsZ49S9gUTgWw0E68qZ8VrOkaP_Jk/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=true

AI analysis/stats
The following summarise some of the stats obtained from the games (with the underlying data linked above if you wanted to analyse further). Feel free to suggest ways of displaying the data that you think would be interesting (currently stats are available relating to mass killed by unit, mass lost by unit, mass built by unit, the total mass split by active units at a particular point of time, and overall mass and energy income, with the stats recorded in 60 second intervals).

Since it's the first time I've done this it's likely there'll be errors/unintended consequences with the stats (e.g. BuildingDefence is meant to cover units like shields and SMD but has evidently picked up SCTA nukes). If you spot anything odd let me know and I can look at updating the stat tracking for future tournaments.

The stat tracking functionality will be incorporated into the next release of M27 to allow it to be run even if M27 itself isn't playing in a game (by making the variable M27StatTracking true in the M27Config.lua file)

Note some of the below pictures will need opening/viewing zoomed in!

• Average army composition for all AI over the course of the game
  (This only considers the shortest 75% of the games to avoid a handful of games resulting in a false impression of the AI unit composition)

• Where each AI invested their mass
  

• How effective each unit type was (mass kills as a percentage of mass built)
  

Great data. Is this last post still relevant ? I am kind of surprised about high Dillidali and M27 scores. I used both in probably over 30 games and at least on maps i use, M27 deals with Dillidali in a matter of minutes. I am guessing M27 has been improved a lot or maybe i broke Dillidali somehow ?

M27 will have improved slightly since then but I expect AiX DilliDalli with a 1.3 resource and build mod would still beat 1.0 M27 AI (i.e. no resource/build boost) on the majority of maps (which is the last tournament that I ran). Data on unit composition is likely to be largely unchanged for most AI, and with the exception of the Adaptive AI I wouldn't be expecting a significant change in rankings.

There have been a few changes since then though, most notably a new AI (M28), the Adaptive AI getting significant improvements, and some updates to M27, RNG and Uveso. I'll probably consider doing another AI tournament at some point to see how things stand - there's a good chance Swarm will be fixed in the not too distant future so it'll probably be after that. It takes a lot of time to run the tournaments though, hence why I'm no longer doing them regularly.

I am playing a lot on the dark heart so i can give some feedback, i am not sure though if those observations only apply to this map. I switched from M27 to M28 as ally because he stays in base with com and i avoid those early 5-15min deaths that ruin the game. 1v1 vs M27 is a different story though, as there he will be attacking u with com and it will be hard to defend + snipe him.

So as far as M27, if i was doing things i would set a hard time limit as to how long commander is allowed outside the base. Snipes from 10 onwards are to easy and he mostly has no air cover. If you wonder far enough from base, it gives enemies plenty of time to make enough snipe tools even if you start straight for your base right away. I noticed that M27 and RNG, even in minute 30, they will occasionaly venture to middle of map, collect some mass, extractors .. again, i think i would set a hard limit and put commander to the safest place possible. The fact that M27 wonders around area marked with yellow makes me belive he thinks he is safe in the water. I would prefer him moved to the water behind the island, with enough AA on main island to hopefuly prevent any over island torp bomber attempts.

M28 is as far as economy goes not far from M27, i even like the fact that he builds factories in remote mass extractor bases sometimes. But with that mass he seems to be less dangerous. But once base is built, you have a very reliable ally at home that will keep his base safe.

2 ideas that i had but i am not sure if its even possible to implement, i would have to look at the files myself. First is use of signals. For instance, even in best scenarios, something can go wrong and commander will maybe follow enemy engineer and wonder far away. It would be cool if you could use 1 of the 3 signals and that would move his com to that location. As an alternative, i see M27 and M28 greet eachother at the start of battle, maybe they could also listen to some of the ally advices, like, move com back to base, help me in my base, hold skies clear. That would simulate a real person ally to some degree.

Keep up the good work and if you have any ideas of interesting scenarios i can simulate, let me know. So far i am enjoying those AI's i use more than real players. And my playing has improved a lot more than it did before where in 6v6 i did the always same predictable things and enemy also always was completely predictable and either killed you before you could learn anything or was so bad that it offered no learning experience.

ps: i used M27, M28 and RNG in quite a few battles so far and had absolutely no tehnical problems, desync, lagging, etc. I did have issues with core game itself, mostly things like putting bug on patrol in the mid of base, only to see it follow a spy plane back to enemy base and get killed when i look elsewhere. Same with commander on patrol. Should be limited by how far from patrol area u can run after an unit before going back. Probably i am doing something wrong, using the wrong command.