Monthly AI Tourney Series

As of last week, I am running a weekly AI only tournament to put new and old AIs through their paces and see who comes out on top!

Each week I'm planning to change up the format; I'll be doing stuff like 1v1s, teamgames, FFAs, modded games, and no rush timers - let me know what you'd all like to see the AIs play

I'll be posting the results of each week in this thread, so stay tuned for updates.

PS: If anybody wants to enter an AI just let me know, all AIs are welcome!

26th November 2021 - Edition 1 - 1v1 fight club
In the first edition of the tourney, I ran a 1v1 competition on the current 1v1 ladder map pool for 200 - 700 rated players (not including the mapgen though). Rounds were played with each AI playing every other AI on every map in the pool each round. At the end of each round, the lowest scoring AI was eliminated from the next round, leading to 5 rounds in total for the 6 entered AIs:

• Adaptive AI
• Sorian AI Adaptive
• Dilli AI - last update: 2018/06/10
• DilliDalli AI - last update: 2021/07/27
• Swarm AI - last update: 2021/11/26
• M27AI - last update: 2021/11/19

The scores for each AI in each round were 0 points per loss, 1 point per draw, and 3 points per win.

In the final round, DilliDalli and M27 won 4/8 games each, leading to a tied competition!

• DilliDalli won on Eye of the Storm, Floralis, Forbidden Pass, and Open Palms.
• M27 won on Auburn Canyon, The Ganges Chasma, Pelagial, and Theta Passage.

To give an insight on where each AI was picking up points, I also worked out the win rate for each AI on each map (number of games varies per AI since they were eliminated at different times):

@softles Hey! if you would like to, I could make you a banner for this tourney!

I think this is really cool. Well done!

@Softles can I enter my AI?

@Femboy sure - but what would I do with a banner?

@Dragun101 no plans to include SCTA every week, but it would definitely make a good feature for one of the weeks

3rd December 2021 - Edition 2 - 2v2 double trouble
This week the AIs played 2v2s on the <1000 rating TMM pool, with full share active. A similar format was followed to last week, except that 2 AIs were eliminated each round (in order to reduce the total number of games that had to be run). 6 AIs were entered this week:

• Adaptive AI
• Sorian AI Adaptive
• Dilli AI - last update: 2018/06/10
• DilliDalli AI - last update: 2021/07/27
• M27AI - last update: 2021/11/19
• RNG AI -

As before, any game that reached 45 mins was ended and called as a draw.

The scores in each round were calculated as 3 points per win and 1 point per draw, with the round results as follows:

Giving us another tied tournament, with #1 spot shared between M27AI and RNG AI!

These results masked some interesting individual strengths and weaknesses for each of the AIs, including our two winners:

M27 AI:

• Great ACU duelling, superb t3 land rushes, and effectively used gifted bases if teammates died.
• Struggled on water maps (as well as serenity due to a bug), and risky plays with the ACU left it vulnerable.

RNG AI:

• Great all round, particularly outclassing other AIs on the water maps. Solid base construction made it hard to beat.
• Struggled for map control on land focused maps, and doesn't yet make use of gifted allied bases on full share.

Now for some extra stats for your viewing pleasure..
AI head to head results across all rounds (cells are from the perspective of the row AI):

Special credit to Dilli and DilliDalli for perfect records against Sorian Adaptive, as well as RNG for perfect records against both Adaptive and Sorian Adaptive.

Map stats for each AI:

Special credit to DilliDalli for the 100% record on Serenity Desert.

As an extra special treat, I also recorded performance over time stats for each of the games, and averaged over all their games to get an idea of each AI's in game performance (i.e. how quickly does it run in game). This stat isn't perfect, since it doesn't control for how performant opponents were in their games, but it gives a rough idea.

(perf here is measured as #real seconds per 10 game seconds while speed is set to +10, with full +10 roughly meaning a perf of 1.5 and +0 roughly meaning a perf of 13)

Hope people find this interesting - tune in next week for 4v4s, 5v5s and 6v6s on classic team-game maps. As ever feel free to drop in ideas for future weeks or anything else

Also let me know how you want results tables formatted in the future - should I stick with {Wins, Draws, Losses}, swap to {points} or go back to {win %} in the head to head and map results tables? (or something else entirely??)

Points or win %, w/d/l is impossible to compare across a table like that.

I would definitely like for your continued insights each week on the different AI's strengths and weaknesses.
Very Interesting Stuff!
Thanks for testing, and posting these updates

11th December 2021 - Edition 3 - 4v4+: you can't spell teaim without AI.

This week the AIs were playing on a selection of team game maps to find out who you should bring to back you up in a com fight:

• Round 1: Tabula Rasa v3 (4v4) and The Pyramid 5v5 (5v5)
• Round 2: Hilly Plateau (4v4) and Diversity (4v4)
• Round 3: Canis 4v4 spezial edition (4v4) and Adaptive Wonder Open (8v8)

Rules this week were share until death and a 2 hour game limit (which none of the games reached). I'll just include the win % in the tables below since there were no draws this week.

Each AI played every other AI on every map once in round 1, twice in round 2, and 4 times in round 3. Without further ado, here were the results:

Congrats to RNG Standard for the first outright win of the series!

The head to head and per map results were as follows:

This week I also pulled performance stats for each AI against itself in the two round 1 maps, which produced the results below:

Again for reference, a performance of X => X/10 real seconds per game second (trying to run at +10 speed). Results plotted against the number of units on the map.

Congrats again to RNG Standard for winning, and remember tune in next week for some Free For All chaos!

Great series of tournaments, thanks for conducting them

22nd January 2022 - Edition 4 - 1v1s: Cry me a river

After a festive break we are back with a bumper edition of the AI tourney - 11 AIs entered, fighting it out across 5 rounds (round robin) of 1v1s on the following maps:

• Twin Rivers (round 1)
• Twin Rivers (round 2)
• Twin Rivers (round 3)
• Twin Rivers (round 4)
• Twin Rivers (round 5)

The AIs entered for this week are:
Adaptive AI, Dalli AI, Dilli AI, DilliDalli AI, M27 AI, RNG Standard AI, SCTA Arm, SCTA Core, Sorian Adaptive AI, Sorian Edit Adaptive AI, Swarm Terror AI.

Format
The twist is that each round AIs are granted bonus cheat multipliers based on the previous round's results. In brief, an AI gets a +0.1 boost to resources and build rate for every position down the leaderboard it was in the last round - and accumulates these bonuses as the tournament goes on.

To stop these boosts getting out of hand in the matches, the boosts applied in game are scaled down based on the lower of the boosts between two AIs. For example if AI Alpha with boost 2.0 is playing AI Bravo with boost 2.4, then the applied settings would be a 1.0x multiplier for Alpha (2.0/2.0) and a 1.2x multiplier for Bravo (2.4/2.0). Applied boosts are rounded to the nearest 0.1 due to UI limitations.

The aim is that as the rounds progress, we get a better and better idea of what boosts each AI needs to be an equal match for every other AI (kinda like the AI's handicap). The AI with the lowest boost at the end of the tourney wins!

Matches:
M: Accumulated Modifier going into the round
W: Wins that round
P: End of round placement (* joint)

Results:
For a final set of modifiers at (raw, normalised to nearest 0.1):
2.6, 1.0: RNG Standard
2.7, 1.0: Dalli
3.0, 1.2: Dilli, DilliDalli
3.2, 1.2: M27, Sorian Edit
3.3, 1.3: Swarm Terror
3.5, 1.3: SCTA Core
3.7, 1.4: Sorian Adaptive
3.8, 1.5: Adaptive, SCTA Arm

Congrats to RNG Standard AI for winning, commiserations to everyone else!

For fun, here's a view of the normalised modifiers (normalised compared to leading AI that round) each AI had as the tourney progressed:

I think the scores would have continued to settle down a little bit if I'd kept doing more rounds, but after manually running 275 games in a row I didn't feel like anymore...

--

The particularly sharp amongst you may have noticed that the weekly tourney hasn't exactly been weekly lately - and going forwards this will become a less regular thing (maybe monthly?) so that I can focus on developing AIs to enter.

See you all in the next one!

@softles Thanks again for running these! Re moving to once a month I’m happy to run one once a month to share the load if you wanted to have it as a biweekly tourney?

AI biweekly tourney - AstromAInia
The humans had their turn on this map in a recent tourney, now it's the AI's time to show how the map should really be played!

Format
This time there were 12 entrants - the 11 from the last tourney, and a new addition of the base game's AI: Turtle on what is arguably a turtler's dream map.

The setup of the tourney is as follows:

All matches: Random AI faction with a random start position

• Group stage: The 12 AI are split into 4 groups of 3, based on their performance in the last tourney (twin rivers). Each AI plays the other AI in its group, with the worst AI being eliminated
• Quarter finals: The winner from each group plays the 2nd best AI from another group, in a best of 3
• Semi finals: The quarter final winners are paired up to fight in a best of 3, similar to the quarters
• Finals: The two finalists duke it out in a best of 5

The results
Group stages

Quarter finals

Semi finals

Finals

Winner: M27

Overall results

Other random observations

• First blood - Sorian Edit took just 8m09 to wrap up its winning game; in contrast, Swarm terror averaged 52m53 for the games it won
• Sneak attack - RNG was the only AI to make use of the rear entrance to the base, sending a galactic colossus through it in one game
• Best recovery - In one game Swarm lost 1/3 of its 18 base mexes, yet managed to repel the attack and eventually claim a win
• In a case of the AI results imitating the human tourney, the winning AI's last update was created only a short while before the tourney started.

I'll probably do another mini-AI tourney in 1-2 weeks - does anyone have any particular scenarios they're interested in seeing and/or are people still interested in these?

I've noted some scenario examples below; whether I pick a scenario will depend in part on if it won't take too long to manually run (i.e. the number of rounds and games that would be required), and my preference is probably towards a variety of maps (since the tourneys up until now have shown how differently the same AIs can perform based on the map chosen).

Tourney ideas:

• 1v1 5km / 10km / 20km Tourney - selection of just 5km, or just 10km or just 20km maps, based on some of the maps that have featured in recent 1v1 map pools
• Matchmaker based tourney - e.g. <700 rank map pool for 1v1; 2v2; or 4v4 full share
• Mimicking other recent human tourneys (e.g. the FFA tourney; King of Setons; King of Dual Gap)

This makes me depressive

Would love to see 20km map games with all 3 theaters. But I know we don't have enough coverage in the AI's for naval. But thats an area where we are weakest.

~-~-~

AI biweekly/monthly tourney: 20km Combined arms
The theme of this map was 20km maps that allow use of land, navy and air. The maplist chosen was:

• Crossfire Canal - FAF version
• Norfair
• The Ditch - FAF version
• Setons Clutch - FAF version
• The Bermuda Locket - FAF version

Setup
Setup is similar to the last tourney - 12 AIs are split into 4 groups. The weakest 4 are eliminated in the first round, with the winners from one group fighting the runners up from the other in the quarter-finals in 3 matches.
As there were 5 maps, maps were chosen randomly for the group and quarters, and then for the semis and finals each map was played.
Starting positions are the default first and second positions set by the map (e.g. on Setons this means a 'mid vs mid' battle)

Changes to AI
5 of the featured AI have released new versions since the last tourney: M27, Swarm Terror, SCTA Arm, Sorian Edit, SCTA Core

The results
Group stages

Quarter finals

Semi finals

Finals

Total wins by AI

Detailed results
The full results can be seen in the 'DetailedResults' worksheet of the following file:
https://docs.google.com/spreadsheets/d/1i2ObsZ49S9gUTgWw0E68qZ8VrOkaP_Jk/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=true

Other Observations

• While some AI attempted to build navy, in all cases it proved ineffective with the mass invested in navy far exceeding the mass killed. Neither of the finalists built a single naval unit the entire tourney.
• The finals were closer than the scoreline suggests, with DilliDalli dominating land on most of the maps and winning map control (e.g. holding 5 out of the 6 islands at one point on Crossfire Canals, and effectively raiding and even claiming mexes behind M27's main base on Setons). The finals ended up being an air vs land contest on most maps.
• SCTA Arm managed to have 2 T3 tanks solo a GC (GC's shot was blocked by cliffs) but it wasn't enough to let it claim a win
• Heroic unit of the tourney: DilliDalli's lone striker vs M27 on Setons which managed 19 kills and 1,720 mass kills raiding mexes
• Notable changes in fortunes: Swarm managed a turnaround in speed, going from the slowest AI to claim wins in the last tourney to getting the second fastest win (11m21, beaten only by Dilli who got a win in 10m48)