Monthly AI Tourney Series

3rd December 2021 - Edition 2 - 2v2 double trouble
This week the AIs played 2v2s on the <1000 rating TMM pool, with full share active. A similar format was followed to last week, except that 2 AIs were eliminated each round (in order to reduce the total number of games that had to be run). 6 AIs were entered this week:

  • Adaptive AI
  • Sorian AI Adaptive
  • Dilli AI - last update: 2018/06/10
  • DilliDalli AI - last update: 2021/07/27
  • M27AI - last update: 2021/11/19
  • RNG AI -

As before, any game that reached 45 mins was ended and called as a draw.

The scores in each round were calculated as 3 points per win and 1 point per draw, with the round results as follows:

AI Round 1 Round 2 Round 3
RNG 103 53 13
M27 98 53 13
DilliDalli 110 46 X
Dilli 73 20 X
Adaptive 24 X X
Sorian Adaptive 18 X X

Giving us another tied tournament, with #1 spot shared between M27AI and RNG AI!

These results masked some interesting individual strengths and weaknesses for each of the AIs, including our two winners:

M27 AI:

  • Great ACU duelling, superb t3 land rushes, and effectively used gifted bases if teammates died.
  • Struggled on water maps (as well as serenity due to a bug), and risky plays with the ACU left it vulnerable.

RNG AI:

  • Great all round, particularly outclassing other AIs on the water maps. Solid base construction made it hard to beat.
  • Struggled for map control on land focused maps, and doesn't yet make use of gifted allied bases on full share.

Now for some extra stats for your viewing pleasure..
AI head to head results across all rounds (cells are from the perspective of the row AI):

RNG M27AI DilliDalli Dilli Adaptive Sorian Adaptive
RNG ------ 10W, 9D, 11L 5W, 3D, 12L 17W, 1D, 2L 10W, 0D, 0L 10W, 0D, 0L
M27AI 11W, 9D, 10L ------ 12W, 3D, 5L 14W, 2D, 4L 4W, 5D, 1L 7W, 1D, 2L
DilliDalli 12W, 3D, 5L 5W, 3D, 12L ------ 14W, 2D, 4L 8W, 1D, 1L 10W, 0D, 0L
Dilli 2W, 1D, 17L 4W, 2D, 14L 4W, 2D, 14L ------ 9W, 1D, 0L 10W, 0D, 0L
Adaptive 0W, 0D, 10L 1W, 5D, 4L 1W, 1D, 8L 0W, 1D, 9L ------ 1W, 8D, 1L
Sorian Adaptive 0W, 0D, 10L 2W, 1D, 7L 0W, 0D, 10L 0W, 0D, 10L 1W, 8D, 1L ------

Special credit to Dilli and DilliDalli for perfect records against Sorian Adaptive, as well as RNG for perfect records against both Adaptive and Sorian Adaptive.

Map stats for each AI:

Map RNG M27AI DilliDalli Dilli Adaptive Sorian Adaptive
Adaptive Meadow 6W, 2D, 1L 4W, 2D, 3L 3W, 1D, 4L 5W, 0D, 3L 0W, 2D, 3L 0W, 1D, 4L
Angel Lagoon 7W, 2D, 0L 2W, 4D, 3L 3W, 2D, 3L 1W, 3D, 4L 0W, 3D, 2L 2W, 0D, 3L
Charity 5W, 1D, 3L 6W, 1D, 2L 6W, 1D, 1L 2W, 0D, 6L 0W, 2D, 3L 0W, 1D, 4L
Desert Planet II v2 5W, 1D, 3L 7W, 2D, 0L 5W, 1D, 2L 2W, 0D, 6L 0W, 1D, 4L 0W, 1D, 4L
Nomadiah 4W, 1D, 4L 8W, 1D, 0L 5W, 1D, 2L 2W, 0D, 6L 0W, 2D, 3L 0W, 1D, 4L
Pelagial v2 7W, 2D, 0L 2W, 4D, 3L 1W, 2D, 5L 3W, 3D, 2L 1W, 2D, 2L 1W, 1D, 3L
Strife of Titan 3W, 2D, 4L 5W, 2D, 2L 7W, 0D, 1L 4W, 0D, 4L 0W, 1D, 4L 0W, 1D, 4L
Serenity Desert 7W, 0D, 2L 0W, 1D, 8L 8W, 0D, 0L 4W, 0D, 4L 2W, 0D, 3L 0W, 1D, 4L
Syrtis Major 4W, 0D, 5L 8W, 0D, 1L 5W, 0D, 3L 4W, 0D, 4L 0W, 1D, 4L 0W, 1D, 4L
Turtle Rocks 4W, 2D, 3L 6W, 3D, 0L 6W, 1D, 1L 2W, 0D, 6L 0W, 1D, 4L 0W, 1D, 4L

Special credit to DilliDalli for the 100% record on Serenity Desert.

As an extra special treat, I also recorded performance over time stats for each of the games, and averaged over all their games to get an idea of each AI's in game performance (i.e. how quickly does it run in game). This stat isn't perfect, since it doesn't control for how performant opponents were in their games, but it gives a rough idea.
tmp.png
(perf here is measured as #real seconds per 10 game seconds while speed is set to +10, with full +10 roughly meaning a perf of 1.5 and +0 roughly meaning a perf of 13)

Hope people find this interesting - tune in next week for 4v4s, 5v5s and 6v6s on classic team-game maps. As ever feel free to drop in ideas for future weeks or anything else 🙂

Also let me know how you want results tables formatted in the future - should I stick with {Wins, Draws, Losses}, swap to {points} or go back to {win %} in the head to head and map results tables? (or something else entirely??)

Points or win %, w/d/l is impossible to compare across a table like that.

I would definitely like for your continued insights each week on the different AI's strengths and weaknesses.
Very Interesting Stuff!
Thanks for testing, and posting these updates 🙂

11th December 2021 - Edition 3 - 4v4+: you can't spell teaim without AI.

This week the AIs were playing on a selection of team game maps to find out who you should bring to back you up in a com fight:

  • Round 1: Tabula Rasa v3 (4v4) and The Pyramid 5v5 (5v5)
  • Round 2: Hilly Plateau (4v4) and Diversity (4v4)
  • Round 3: Canis 4v4 spezial edition (4v4) and Adaptive Wonder Open (8v8)

Rules this week were share until death and a 2 hour game limit (which none of the games reached). I'll just include the win % in the tables below since there were no draws this week.

Each AI played every other AI on every map once in round 1, twice in round 2, and 4 times in round 3. Without further ado, here were the results:

AI Round 1 Round 2 Round 3
RNG Standard 60% 58% 87%
DilliDalli 90% 75% 13%
M27AI 70% 33% X
Dilli 60% 33% X
Sorian Edit Adaptive 10% X X
Swarm 10% X X

Congrats to RNG Standard for the first outright win of the series!

The head to head and per map results were as follows:

RNG Standard DilliDalli M27AI Dilli Sorian Edit Adaptive Swarm
RNG Standard - 64% 33% 83% 100% 100%
DilliDalli 36% - 100% 67% 100% 100%
M27AI 67% 0% - 50% 100% 100%
Dilli 17% 33% 50% - 100% 100%
Sorian Edit Adaptive 0% 0% 0% 0% - 50%
Swarm 0% 0% 0% 0% 50% -
RNG Standard DilliDalli M27AI Dilli Sorian Edit Adaptive Swarm
Tabula Rasa 60% 80% 60% 80% 0% 20%
The Pyramid 60% 100% 80% 40% 20% 0%
Hilly Plateau 33% 100% 67% 0% X X
Diversity 83% 50% 0% 67% X X
Canis 100% 0% X X X X
Wonder 75% 25% X X X X

This week I also pulled performance stats for each AI against itself in the two round 1 maps, which produced the results below:
tmp.png
Again for reference, a performance of X => X/10 real seconds per game second (trying to run at +10 speed). Results plotted against the number of units on the map.

Congrats again to RNG Standard for winning, and remember tune in next week for some Free For All chaos!

Great series of tournaments, thanks for conducting them 🙂

22nd January 2022 - Edition 4 - 1v1s: Cry me a river

After a festive break we are back with a bumper edition of the AI tourney - 11 AIs entered, fighting it out across 5 rounds (round robin) of 1v1s on the following maps:

  • Twin Rivers (round 1)
  • Twin Rivers (round 2)
  • Twin Rivers (round 3)
  • Twin Rivers (round 4)
  • Twin Rivers (round 5)

The AIs entered for this week are:
Adaptive AI, Dalli AI, Dilli AI, DilliDalli AI, M27 AI, RNG Standard AI, SCTA Arm, SCTA Core, Sorian Adaptive AI, Sorian Edit Adaptive AI, Swarm Terror AI.

Format
The twist is that each round AIs are granted bonus cheat multipliers based on the previous round's results. In brief, an AI gets a +0.1 boost to resources and build rate for every position down the leaderboard it was in the last round - and accumulates these bonuses as the tournament goes on.

To stop these boosts getting out of hand in the matches, the boosts applied in game are scaled down based on the lower of the boosts between two AIs. For example if AI Alpha with boost 2.0 is playing AI Bravo with boost 2.4, then the applied settings would be a 1.0x multiplier for Alpha (2.0/2.0) and a 1.2x multiplier for Bravo (2.4/2.0). Applied boosts are rounded to the nearest 0.1 due to UI limitations.

The aim is that as the rounds progress, we get a better and better idea of what boosts each AI needs to be an equal match for every other AI (kinda like the AI's handicap). The AI with the lowest boost at the end of the tourney wins!

Matches:
M: Accumulated Modifier going into the round
W: Wins that round
P: End of round placement (* joint)

Round 1 Round 2 Round 3 Round 4 Round 5
AI M W P M W P M W P M W P M W P
Adaptive 1.0 3 8 1.7 7 3* 1.9 0 11 2.9 6 2* 3.0 2 9*
Dalli 1.0 9 1* 1.0 2 10 1.9 9 2 2.0 5 7* 2.6 9 2
Dilli 1.0 8 3 1.2 3 8* 1.9 8 3 2.1 6 2* 2.2 2 9*
DilliDalli 1.0 6 4* 2.3 7 3* 1.5 4 6* 2.3 2 11 2.9 10 1
M27 1.0 5 6* 1.5 9 1 1.5 3 9 2.3 5 7* 2.9 6 4*
RNG Standard 1.0 9 1* 1.0 0 11 2.0 10 1 2.0 6 2* 2.1 4 6*
SCTA Arm 1.0 2 9 1.8 5 6* 2.3 4 6* 2.8 3 9* 3.6 7 3
SCTA Core 1.0 1 10* 2.9 8 2 2.0 2 10 2.9 6 2* 3.0 4 6*
Sorian Adaptive 1.0 1 10* 1.9 6 5 2.3 6 4 2.6 3 9* 3.4 6 4*
Sorian Edit 1.0 6 4* 1.3 3 8* 2.0 5 5 2.4 6 2* 2.5 3 8
Swarm Terror 1.0 5 6* 1.5 5 6* 2.0 4 6* 2.5 7 1 2.5 2 9*

Results:
For a final set of modifiers at (raw, normalised to nearest 0.1):
2.6, 1.0: RNG Standard
2.7, 1.0: Dalli
3.0, 1.2: Dilli, DilliDalli
3.2, 1.2: M27, Sorian Edit
3.3, 1.3: Swarm Terror
3.5, 1.3: SCTA Core
3.7, 1.4: Sorian Adaptive
3.8, 1.5: Adaptive, SCTA Arm

Congrats to RNG Standard AI for winning, commiserations to everyone else!

For fun, here's a view of the normalised modifiers (normalised compared to leading AI that round) each AI had as the tourney progressed:
tmp.png

I think the scores would have continued to settle down a little bit if I'd kept doing more rounds, but after manually running 275 games in a row I didn't feel like anymore...

--

The particularly sharp amongst you may have noticed that the weekly tourney hasn't exactly been weekly lately - and going forwards this will become a less regular thing (maybe monthly?) so that I can focus on developing AIs to enter.

See you all in the next one!

@softles Thanks again for running these! Re moving to once a month I’m happy to run one once a month to share the load if you wanted to have it as a biweekly tourney?

AI biweekly tourney - AstromAInia
The humans had their turn on this map in a recent tourney, now it's the AI's time to show how the map should really be played!

Format
This time there were 12 entrants - the 11 from the last tourney, and a new addition of the base game's AI: Turtle on what is arguably a turtler's dream map.

The setup of the tourney is as follows:

All matches: Random AI faction with a random start position

  • Group stage: The 12 AI are split into 4 groups of 3, based on their performance in the last tourney (twin rivers). Each AI plays the other AI in its group, with the worst AI being eliminated
  • Quarter finals: The winner from each group plays the 2nd best AI from another group, in a best of 3
  • Semi finals: The quarter final winners are paired up to fight in a best of 3, similar to the quarters
  • Finals: The two finalists duke it out in a best of 5

The results
Group stages

Group AI Wins
Group 1 RNG Standard 1
Group 1 Swarm Terror 2
Group 1 SCTA Arm 0
Group 2 Dalli 2
Group 2 SCTA Core 0
Group 2 AI: Turtle 1
Group 3 Dilli 1
Group 3 M27 2
Group 3 Sorian Adaptive 0
Group 4 DilliDalli 2
Group 4 Sorian Edit 1
Group 4 Adaptive 0

Quarter finals

AI 1 (group stage winner) AI1 wins AI2 (group stage runner up) AI2 wins
Swarm Terror 3 Dilli 0
Dalli 3 Sorian Edit 0
M27 3 RNG Standard 0
DilliDalli 1 AI: Turtle 2

Semi finals

AI 1 AI1 wins AI2 AI2 wins
Swarm Terror 2 Dalli 1
M27 3 AI: Turtle 0

Finals

AI 1 AI1 wins AI2 AI2 wins
Swarm Terror 0 M27 5

Winner: M27

Overall results

AI Wins Losses Win percent
M27 13 0 100%
Swarm Terror 7 6 54%
Dalli 6 2 75%
AI: Turtle 3 5 38%
DilliDalli 3 2 60%
Sorian Edit 1 4 20%
Dilli 1 4 20%
RNG Standard 1 4 20%
SCTA Arm 0 2 0%
SCTA Core 0 2 0%
Sorian Adaptive 0 2 0%
Adaptive 0 2 0%

Other random observations

  • First blood - Sorian Edit took just 8m09 to wrap up its winning game; in contrast, Swarm terror averaged 52m53 for the games it won
  • Sneak attack - RNG was the only AI to make use of the rear entrance to the base, sending a galactic colossus through it in one game
  • Best recovery - In one game Swarm lost 1/3 of its 18 base mexes, yet managed to repel the attack and eventually claim a win
  • In a case of the AI results imitating the human tourney, the winning AI's last update was created only a short while before the tourney started.

I'll probably do another mini-AI tourney in 1-2 weeks - does anyone have any particular scenarios they're interested in seeing and/or are people still interested in these?

I've noted some scenario examples below; whether I pick a scenario will depend in part on if it won't take too long to manually run (i.e. the number of rounds and games that would be required), and my preference is probably towards a variety of maps (since the tourneys up until now have shown how differently the same AIs can perform based on the map chosen).

Tourney ideas:

  • 1v1 5km / 10km / 20km Tourney - selection of just 5km, or just 10km or just 20km maps, based on some of the maps that have featured in recent 1v1 map pools
  • Matchmaker based tourney - e.g. <700 rank map pool for 1v1; 2v2; or 4v4 full share
  • Mimicking other recent human tourneys (e.g. the FFA tourney; King of Setons; King of Dual Gap)

This makes me depressive

I’m a shitty 1k Global. Any balance or gameplay suggestions should be understood or taken as such.

Project Head and current Owner/Manager of SCTA Project

Would love to see 20km map games with all 3 theaters. But I know we don't have enough coverage in the AI's for naval. But thats an area where we are weakest.

~-~-~

AI biweekly/monthly tourney: 20km Combined arms
The theme of this map was 20km maps that allow use of land, navy and air. The maplist chosen was:

  • Crossfire Canal - FAF version
  • Norfair
  • The Ditch - FAF version
  • Setons Clutch - FAF version
  • The Bermuda Locket - FAF version

Setup
Setup is similar to the last tourney - 12 AIs are split into 4 groups. The weakest 4 are eliminated in the first round, with the winners from one group fighting the runners up from the other in the quarter-finals in 3 matches.
As there were 5 maps, maps were chosen randomly for the group and quarters, and then for the semis and finals each map was played.
Starting positions are the default first and second positions set by the map (e.g. on Setons this means a 'mid vs mid' battle)

Changes to AI
5 of the featured AI have released new versions since the last tourney: M27, Swarm Terror, SCTA Arm, Sorian Edit, SCTA Core

The results
Group stages

Group AI name Wins
1 M27 4
1 DilliDalli 2
1 SCTA Core 0
2 Swarm Terror 4
2 Sorian Adaptive 2
2 Sorian Edit 0
3 Dalli 3
3 Dilli 2
3 Adaptive 0
4 RNG Standard 4
4 SCTA Arm 2
4 AI: Turtle 0

Quarter finals

AI Name Wins AI Name Wins
M27 3 Dilli 0
Swarm Terror 3 SCTA Arm 0
DilliDalli 3 Dalli 0
RNG Standard 3 Sorian Adaptive 0

Semi finals

AI Name Wins AI Name Wins
M27 5 Swarm Terror 0
DilliDalli 5 RNG Standard 0

Finals

AI Name Wins AI Name Wins
M27 5 DilliDalli 0

Total wins by AI

AI Name Wins Win %
M27 17 100%
DilliDalli 10 59%
RNG Standard 7 58%
Swarm Terror 7 58%
Dalli 3 43%
Dilli 2 29%
Sorian Adaptive 2 29%
SCTA Arm 2 29%
Sorian Edit 0 0%
SCTA Core 0 0%
Adaptive 0 0%
AI: Turtle 0 0%

Detailed results
The full results can be seen in the 'DetailedResults' worksheet of the following file:
https://docs.google.com/spreadsheets/d/1i2ObsZ49S9gUTgWw0E68qZ8VrOkaP_Jk/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=true

Other Observations

  • While some AI attempted to build navy, in all cases it proved ineffective with the mass invested in navy far exceeding the mass killed. Neither of the finalists built a single naval unit the entire tourney.
  • The finals were closer than the scoreline suggests, with DilliDalli dominating land on most of the maps and winning map control (e.g. holding 5 out of the 6 islands at one point on Crossfire Canals, and effectively raiding and even claiming mexes behind M27's main base on Setons). The finals ended up being an air vs land contest on most maps.
  • SCTA Arm managed to have 2 T3 tanks solo a GC (GC's shot was blocked by cliffs) but it wasn't enough to let it claim a win
  • Heroic unit of the tourney: DilliDalli's lone striker vs M27 on Setons which managed 19 kills and 1,720 mass kills raiding mexes
  • Notable changes in fortunes: Swarm managed a turnaround in speed, going from the slowest AI to claim wins in the last tourney to getting the second fastest win (11m21, beaten only by Dilli who got a win in 10m48)

AI biweekly/monthly tourney: MapGen
The idea of this tourney was to use MapGen to create a map for the AI to fight against for 5km, 7.5km, 10km, 15km and 20km. Due to the AI tourney still being run manually a single map was used for the map size for the tourney (rather than generating a new random map every time). The maplist chosen was:

  • neroxis_map_generator_1.8.5_vgf5tqx2uxrcu_aicae_
  • neroxis_map_generator_1.8.5_uydyqpk4utuuu_aidae_
  • neroxis_map_generator_1.8.5_xaghpcvo2lnrk_aieae_
  • neroxis_map_generator_1.8.5_bvwpdifwrols4_aigae_
  • neroxis_map_generator_1.8.5_44sxnthefcek6_aiiae_

Setup
Setup is similar to the last tourney - 12 AIs are split into 4 groups. The weakest 4 are eliminated in the first round, with the winners from one group fighting the runners up from the other in the quarter-finals in 3 matches.
As there were 5 maps, maps were chosen randomly for the group and quarters, and then for the semis and finals each map was played.

Changes to AI
3 of the featured AI have released new versions since the last tourney: M27, SCTA Arm, SCTA Core

The results
Group stages

Group AI name Wins
3 DilliDalli 4
3 Sorian Edit 3
3 SCTA Arm 0
2 M27 4
4 SCTA Core 0
4 Dalli 2
4 RNG Standard 3
1 Adaptive 1
2 AI: Turtle 2
1 Swarm Terror 4
1 Dilli 2
2 Sorian Adaptive 0

Quarter finals

AI Name Wins AI Name Wins
Swarm Terror 2 Sorian Edit 1
M27 3 Dalli 0
Dilli 2 DilliDalli 1
RNG Standard 3 AI: Turtle 0

Semi finals

AI Name Wins AI Name Wins
Swarm Terror 0 M27 5
Dilli 1 RNG Standard 4

Finals

AI Name Wins AI Name Wins
M27 4 RNG Standard 0

(M27 and RNG drew 1 game)
Total wins by AI

AI Name Wins Win %
M27 16 94%
RNG Standard 10 59%
Swarm Terror 6 50%
DilliDalli 5 63%
Dilli 5 38%
Sorian Edit 4 50%
Dalli 2 29%
AI: Turtle 2 29%
Adaptive 1 20%
Sorian Adaptive 0 0%
SCTA Arm 0 0%
SCTA Core 0 0%

Detailed results
The full results can be seen in the 'DetailedResults' worksheet of the following file:
https://docs.google.com/spreadsheets/d/1i2ObsZ49S9gUTgWw0E68qZ8VrOkaP_Jk/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=true

Other Observations

  • Notable changes in fortunes: In the final, RNG had c.25 mass to M27's 621, yet still managed a draw (M27 attacked RNG's base with its ACU leading to both ACUs dying)
  • The 7.5km map featured two land masses separated by water, which caused several AIs issues
  • Although SCTA failed to get a win it managed to match Sorian Edit for map control and eco for much of the game, with both games going over an hour
  • The group stages required two deciding matches to determine groupings, with an extra Dilli vs Adaptive and Sorian Edit vs DilliDalli match due to them getting the same number of wins as each other.

AI biweekly/monthly tourney: Sorian vs the world
The idea for this tournament was to see the highest AiX modifier that each AI could beat Sorian Adaptive at, on a selection of 5km, 10km and 20km maps from the current map pool, hopefully highlighting how much more comeptitive the FAF custom AIs are, as well as giving a better gauge of the distance between different AI than the previous tournaments win rate. The maplist chosen was:

  • Mirage
  • Stickleback Ridge
  • Open Palms
  • 8 - Badlands_v4
  • The Bermuda Locket - FAF version
  • Crossfire Canal

Setup
Each AI entered will face off against Sorian Adaptive AiX (with no omni). If they win, then they repeat the fight but with Sorian Adaptive getting +0.1 to its modifier, until the challenging AI no longer wins. This is done for each map separately. This resulted in a mammoth 245 games being run, so I doubt I'll be repeating this format due to the massive amount of time it ended up taking (for comparison, the last tournament took just 51 games)

Changes to AI
There were significant changes to the AI this time. In addition to a new version of M27, Sorian Edit, and SCTA, there was a major update for RNG (the first in many months), and we have a new AI entrant with Uveso Rush featuring for the first time!
The results
Summary

AI Name Average AiX Beaten
M27 1.63
RNG Standard 1.57
DilliDalli 1.43
Swarm Terror 1.27
Uveso Rush 1.25
SCTA Core 1.15
Adaptive 1.05
Sorian Edit 0.98

Results by map

8 - Badlands_v4 Adaptive 1.0
8 - Badlands_v4 DilliDalli 1.4
8 - Badlands_v4 M27 1.6
8 - Badlands_v4 RNG Standard 1.6
8 - Badlands_v4 SCTA Core 1.6
8 - Badlands_v4 Swarm Terror 1.3
8 - Badlands_v4 Uveso Rush 2.0
8 - Badlands_v4 Sorian Adaptive 0.4
Crossfire Canal Adaptive 1.0
Crossfire Canal DilliDalli 1.1
Crossfire Canal M27 1.3
Crossfire Canal RNG Standard 1.3
Crossfire Canal SCTA Core 0.2
Crossfire Canal Sorian Edit 0.8
Crossfire Canal Swarm Terror 1.1
Crossfire Canal Uveso Rush 0.7
Crossfire Canal Sorian Adaptive 1.0
Mirage Adaptive 1.2
Mirage DilliDalli 1.5
Mirage M27 1.9
Mirage RNG Standard 1.7
Mirage SCTA Core 1.4
Mirage Swarm Terror 1.5
Mirage Uveso Rush 1.2
Mirage Sorian Adaptive 0.4
Open Palms Adaptive 1.1
Open Palms DilliDalli 1.6
Open Palms M27 1.6
Open Palms RNG Standard 1.6
Open Palms SCTA Core 1.3
Open Palms Sorian Edit 1.0
Open Palms Swarm Terror 1.1
Open Palms Uveso Rush 1.1
Open Palms Sorian Adaptive 0.6
Stickleback Ridge Adaptive 1.0
Stickleback Ridge DilliDalli 1.6
Stickleback Ridge M27 2.0
Stickleback Ridge RNG Standard 2.0
Stickleback Ridge SCTA Core 1.3
Stickleback Ridge Sorian Edit 1.1
Stickleback Ridge Swarm Terror 1.2
Stickleback Ridge Uveso Rush 1.5
Stickleback Ridge Sorian Adaptive 0.4
The Bermuda Locket - FAF version Adaptive 1.0
The Bermuda Locket - FAF version DilliDalli 1.4
The Bermuda Locket - FAF version M27 1.4
The Bermuda Locket - FAF version RNG Standard 1.2
The Bermuda Locket - FAF version SCTA Core 1.1
The Bermuda Locket - FAF version Sorian Edit 1.0
The Bermuda Locket - FAF version Swarm Terror 1.4
The Bermuda Locket - FAF version Uveso Rush 1.0
The Bermuda Locket - FAF version Sorian Adaptive 0.7

Detailed results
The full results can be seen in the Data worksheet of the following file:

https://docs.google.com/spreadsheets/d/1i2ObsZ49S9gUTgWw0E68qZ8VrOkaP_Jk/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=true

Other Observations

  • Uveso was the dominant AI on badlands (due to its ability to punish Sorian Adaptive for sending its ACU to attack on its own early on), managing a win against a 2.2 AiX (it lost against a 2.1 AiX that was running at the same time hence why 2.0 is shown in its results). In contrast as an AiX 1.2 it lost to a 1.0 Sorian Adaptive on Crossfire Canal.
  • Crossfire Canal was by far Sorian Adaptive's best map, with 3 of the AI (Sorian Edit, SCTA Core and Uveso) needing AiX modifiers to beat it (although not strictly correct, if an AI needed an AiX modifier to win I've shown each 0.1 modifier as decreasing the Sorian AiX beaten by 0.1, so an AI needing a 1.5 modifier to win would show as 0.5)
  • Surprisingly, Adaptive AI was better than Sorian Adaptive, beating it on every map with no modifier, and beating 1.2 on Mirage.
  • There were compatibility issues between Sorian Edit and Sorian Adaptive on some maps, so results for these have been omitted
  • 3 AI managed the achievement of beating Sorian Adaptive AiX 2.0 or better - M27 and RNG (2.0, Stickleback Ridge) and Uveso (2.2, Badlands)

Edit: Note that some of the AI aren't intended for 20km or water based maps, which will be a reason for poor results on Crossfire (e.g. SCTA isn't meant for such maps). Excluding crossfire from the results would give the following:
26510667-0995-4b8c-955f-6ea6333eb573-image.png

AI monthly tourney: Survival of the fittest
Suggested by Fearghal, this tournament's theme is to see which AI does best on a survival map. chp2001 helpfully identified the map Final Rush Pro 5 as a potential candidate, since it features 1 team against another (thus avoiding the problem caused by survival maps which lack an enemy brain/base location for the AI to fight against).

For those not familiar with it, it's a 4v4 map where teams face off against each other with a central chokepoint area where the AI spawns. The AI spawns will be delivered by transport, but air is disabled (so the transports are harder to shoot down).

84a9e786-0137-446e-892a-a9b1f36ede14-image.png

The survival spawns are split allocated equally between two different survival 'brains', and wont attack each other, but will attack one (and only one) of the teams. This means that a team can choose to be aggressive by sending their units at the enemy base, to help support the survival AI's attack.

To avoid people exploiting the survival AI, ACUs are prevented from hiding in the water by taking damage over time, and prevented from hiding in the rear corner of the map behind cliffs by a group of galactic colossi. The map has a number of configurable settings, including difficulty, which affects the size of the surivival spawns, and how long it takes before it advances in waves (with a wave starting off with T1 units, before progressively getting harder, e.g. sending in bombers; t2 land; cybran destroyers; t3 land; experimental land; etc.). Players start with 9 hives, an ACU from each faction, and lots of mass points at their base.

Setup
In the first round, each AI will play against itself (i.e. all 8 players will be that AI), and see how long it takes for the last player to die. These times will then be used to give each AI an initial 'seeding', with the last seed playing the second last, the winner of that playing the 3rd last, and so-on until the last ranked AI has lost to the 2nd last ranked AI which has lost to the 3rd last ranked AI and so on.

The map settings still present some issues for AI, in particular the logic for preventing exploiting the survival AI by hiding - I turned this off for the initial 'seeding' round over fears most of the AI wouldn't be able to understand the 'ACUs die when entering water' approach or that 'those hostile GC near your base can be ignored' , but it led to its own problems, e.g. fully upgraded ACUs hiding in the water being very slowly whittled down by the torpedoes of a couple of bricks and didnt give a fair reflection of how well some AI did. I therefore settled on a compromise for the main event, where a team would be treated as having lost if all 4 land bases were wiped out (i.e. it had no more mexes or land factories).

The difficulty was also set to easiest, as all AI struggle compared to humans with such a setup, and this way there was more of a chance of seeing waves hit the experimental land stage.

Changes to AI
New versions of M27, RNG, and SCTA were reflected for the tournament.
The results
Initial seedings - Time lasted when fighting itself

AI Name Time survived
M27 11m25
AI Turtle 16m28
DilliDalli 17m27
Sorian adaptive 19m20
Adaptive 19m22
RNG Standard 29m06
Sorian Edit 31m41
Uveso Rush 34m02
SCTA Core 35m07
Swarm Terror 37m16

Final rankings and head to heads

AI Name Head to Heads won Head to Heads lost Longest time survived until a team died (in seconds)
Sorian Edit 0 6 992
Sorian adaptive 1 3 1122
AI Turtle 2 3 1029
Adaptive 2 3 1080
SCTA Core 1 4 1807
DilliDalli 5 2 1176
RNG Standard 3 2 1629
Uveso Rush 2 2 1907
Swarm Terror 1 1 1940
M27 9 0 1940

For example, DilliDalli was initially seeded 9th (out of 10 AI); it lost to M27 (the 10th seed), but beat the 5 AI that placed worse than it in the final results to reach 5th place. It then lost to RNG (4th place), so didnt advance any further. In contrast, as Swarm was seeded first it only fought the 2nd place initially (which ended up being M27), and after losing that it then fought the then 3rd place (Uveso Rush) which it won, resulting in ending as 2nd place.

Detailed results
The full results can be seen in the Data worksheet of the following file:

https://docs.google.com/spreadsheets/d/1i2ObsZ49S9gUTgWw0E68qZ8VrOkaP_Jk/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=true

Other Observations

  • There were a wide variety of approaches used by the different AI, for example:
  • M27 favoured a firebase appraoch, but chose to place this near the centre of the map meaning it'd get overwhelmed around the time of T3 land or experimentals
  • DilliDalli preferred an all-out assault on the enemy team before spawns even started arriving, sticking to T1 mexes but often being able to kill the enemy team before their economic advantage could be brought to bear
  • Swarm chose to have a massive swarm of mostly T1 units occupying the central area, meaning they could quickly ambush the survival AI drops as they arrived (up until the late T3 land/early experimental stage)
  • Uveso made heavy use of T1 artillery, which when firing from behind T3 shields at the base proved an effective roadblock to the survival AI's T3 land waves
  • There were dramatic transformations in fortunes for the AI from their initial seedings. In DilliDalli's case this was because it's aggressive approach would lead to short matches in all its games, while in M27's case it had chain reactions caused by the various starting ACU (16 per team) being near the centre and one of them dying. Sorian Edit's reversal in fortunes was likely caused by the stricter measuring of when a team had lost, as it was usually better at retreating its ACU behind cliffs/into water where the survival AI couldnt reach it
  • A number of games came down to the wire with one team losing its final base just before the opposing team lost its final base, including the final between M27 and Swarm, and the semi-final between Uveso Rush and Swarm
  • SCTA performed strongly, with a survival time in head to heads of more than 30m (bettered only by Uveso, Swarm and M27), but DilliDalli's early aggression proved its undoing
  • It looked like RNG would pull back from the brink in its game against Uveso Rush when it completed a cloaked laser upgrade on its ACU at its final base. Sadly it turns out that the map gives omni vision meaning its ACU still died quickly to the T3 land spam.

"This week I also pulled performance stats for each AI against itself in the two round 1 maps, which produced the results below:"
tmp.png

I have been testing for the upcoming RC tourny and have been using "mostly" M27 as the standard to
measure against the other Game specific AI's,
I also start test, at game option speed = Fast, (which is = to +4 sim) to make sure the game speeds are "not likely" to drop below "0" game speed.
Game speed limits the size of the maps I can use -
(if games slow to "-1", after 45 minutes, I limit the AI's to 10x10 maps)
The faster AI's like DilliDalli and RNG are easily able to handle the 20x20 maps.
Since the test games are a combo of the the Game AI and M27 -
I was wondering if with all the recent updates to M27 if it is still as fast as the performance results shown above.

I would test vs Uveso, (since it needs to be loaded for all games anyway)
but it seems that after Uveso grows it's base large it also slows game down a little,
so I don't want to run tests with Uveso as the standard, as it might skew the test results.
Update: I just tested Uveso since post and It runs fast.
It was still +4 at end of 37 min game with 6 AI:Uveso and Lots of units!

Would it be possible to include another performance chart after you run your next tourny,
with the updated AI versions?
Thanks.

Per Fearghal's request, this AI tourney is a bit different, and focuses on how fast/slow the different AI run. The scenarios were as follows:

  • Adaptive Tournament Decision (10km): Rainbow cup style setup, with 3 teams (3v3v2)
  • Rebellion (20km): Simple 2 team head to head with all slots filled (so a 5vs5)

Due to how long games were taking, I ran 2 games on adaptive tournament decision and only one on rebellion.
Unfortunately I forgot to add tick freeze tracking to my profiling code until I was almost done with the games and couldnt face re-running them (as on Rebellion it was taking more than 1 hour for some of the games in real time).

Results are shown both by time, and by unit count. The main point for me is this shows the largest factor in how slow the game will run is how many units are built. In particular, just looking at results by time DilliDalli is by far the quickest AI. However, looking at it by unit count there's little difference between it, RNG, SCTA and Adaptive. That said, Swarm, Uveso and M27 performed noticeably slower even when adjusting for unit count.

For the below graphs, the x axis is either the unit count or the game time in seconds. The y axis in both cases is how long in real-life seconds it took to run 10 game time seconds. I.e. a value of above 10 means that the average game speed was below +0.

Adaptive tournament decision
d6589c1d-e9a6-4605-924b-5e025f27dd8d-image.png
04f61a92-f2f2-4e41-b9ce-8ecb816d0139-image.png

The large blue spike was a brief period when swarm reached -8 game speed

Rebellion

5216ea05-885f-477f-9a0b-a743d57fb85d-image.png
665ef547-d7bb-4e08-a78c-5a132b3c6225-image.png

Unfortunately the current version of Uveso doesnt work with Sorian Edit (which requires Uveso) so no stats are available for this.

I also reran M27 to see how it did after the v47 optimisation release (c. 9% faster) but there wasn’t a noticeable difference in the graph lines so the above reflects the v45 position.

AI monthly tourney: 10km Analytics
Per a request by Relent0r, this tournament was aimed both at providing detailed stats for each AI (so each AI developer gets hopefully useful information), and to give weaker AI a boost to make the results less predictable. The current 1v1 ladder map TAG_Craftius Maximus has been selected since it allows navy to play a role without it being required (so both naval and non-naval AI get to compete), while it's also 10km which will be useful for those AI that aren't designed to function on 20km+ maps.

2754f903-697d-4835-8c65-d55c28e13f6a-image.png

Setup
Each AI will fight each other AI once. The AI will get an AiX modifier based on their average placement in the last 3 tournaments, ranging from 1.0 for first place to 1.5 for last place (where more than 8 AI featured; 1.35 where only 8 AI featured). The actual AiX modifier will then be based on the difference between the AiX modifiers of the two competing AI (so for example DilliDalli will use an AiX 1.3 modifier, while RNG Standard will use 1.1; when fighting each other that means RNG 1.0 will fight DilliDalli 1.2). The AiX modifiers used are shown in the results.

Changes to AI
Almost all of the AI have featured an update since the last tournament.

The results
Summary

AI Name and AiX mod Games won
DilliDalli 1.3 7
M27 1.0 6
Sorian Edit 1.4 5
RNG Standard 1.1 4
Swarm Terror 1.2 3
Uveso Rush 1.3 2
SCTA Core 1.4 1
Adaptive 1.4 0

Congratulations to DilliDalli/Softles, with DilliDalli beating every other AI with it's tank focused game style.

Detailed results
The full results including the cleansed data used for the above graphs can be seen in the Data worksheet of the following file:

https://docs.google.com/spreadsheets/d/1i2ObsZ49S9gUTgWw0E68qZ8VrOkaP_Jk/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=true

AI analysis/stats
The following summarise some of the stats obtained from the games (with the underlying data linked above if you wanted to analyse further). Feel free to suggest ways of displaying the data that you think would be interesting (currently stats are available relating to mass killed by unit, mass lost by unit, mass built by unit, the total mass split by active units at a particular point of time, and overall mass and energy income, with the stats recorded in 60 second intervals).

Since it's the first time I've done this it's likely there'll be errors/unintended consequences with the stats (e.g. BuildingDefence is meant to cover units like shields and SMD but has evidently picked up SCTA nukes). If you spot anything odd let me know and I can look at updating the stat tracking for future tournaments.

The stat tracking functionality will be incorporated into the next release of M27 to allow it to be run even if M27 itself isn't playing in a game (by making the variable M27StatTracking true in the M27Config.lua file)

Note some of the below pictures will need opening/viewing zoomed in!

  • Average army composition for all AI over the course of the game
    (This only considers the shortest 75% of the games to avoid a handful of games resulting in a false impression of the AI unit composition)

a3dd4fb7-883a-46eb-8965-ba7b5405b325-image.png

  • Where each AI invested their mass
    676a19de-7386-44e2-96a9-c7b3f7f5f9c0-image.png

  • How effective each unit type was (mass kills as a percentage of mass built)
    fe01b72c-3cde-482a-8116-8e9a47fc7cf5-image.png