Monthly AI Tourney Series
-
Points or win %, w/d/l is impossible to compare across a table like that.
-
I would definitely like for your continued insights each week on the different AI's strengths and weaknesses.
Very Interesting Stuff!
Thanks for testing, and posting these updates
-
11th December 2021 - Edition 3 - 4v4+: you can't spell teaim without AI.
This week the AIs were playing on a selection of team game maps to find out who you should bring to back you up in a com fight:
- Round 1: Tabula Rasa v3 (4v4) and The Pyramid 5v5 (5v5)
- Round 2: Hilly Plateau (4v4) and Diversity (4v4)
- Round 3: Canis 4v4 spezial edition (4v4) and Adaptive Wonder Open (8v8)
Rules this week were share until death and a 2 hour game limit (which none of the games reached). I'll just include the win % in the tables below since there were no draws this week.
Each AI played every other AI on every map once in round 1, twice in round 2, and 4 times in round 3. Without further ado, here were the results:
AI Round 1 Round 2 Round 3 RNG Standard 60% 58% 87% DilliDalli 90% 75% 13% M27AI 70% 33% X Dilli 60% 33% X Sorian Edit Adaptive 10% X X Swarm 10% X X Congrats to RNG Standard for the first outright win of the series!
The head to head and per map results were as follows:
RNG Standard DilliDalli M27AI Dilli Sorian Edit Adaptive Swarm RNG Standard - 64% 33% 83% 100% 100% DilliDalli 36% - 100% 67% 100% 100% M27AI 67% 0% - 50% 100% 100% Dilli 17% 33% 50% - 100% 100% Sorian Edit Adaptive 0% 0% 0% 0% - 50% Swarm 0% 0% 0% 0% 50% - RNG Standard DilliDalli M27AI Dilli Sorian Edit Adaptive Swarm Tabula Rasa 60% 80% 60% 80% 0% 20% The Pyramid 60% 100% 80% 40% 20% 0% Hilly Plateau 33% 100% 67% 0% X X Diversity 83% 50% 0% 67% X X Canis 100% 0% X X X X Wonder 75% 25% X X X X This week I also pulled performance stats for each AI against itself in the two round 1 maps, which produced the results below:
Again for reference, a performance of X => X/10 real seconds per game second (trying to run at +10 speed). Results plotted against the number of units on the map.Congrats again to RNG Standard for winning, and remember tune in next week for some Free For All chaos!
-
Great series of tournaments, thanks for conducting them
-
22nd January 2022 - Edition 4 - 1v1s: Cry me a river
After a festive break we are back with a bumper edition of the AI tourney - 11 AIs entered, fighting it out across 5 rounds (round robin) of 1v1s on the following maps:
- Twin Rivers (round 1)
- Twin Rivers (round 2)
- Twin Rivers (round 3)
- Twin Rivers (round 4)
- Twin Rivers (round 5)
The AIs entered for this week are:
Adaptive AI, Dalli AI, Dilli AI, DilliDalli AI, M27 AI, RNG Standard AI, SCTA Arm, SCTA Core, Sorian Adaptive AI, Sorian Edit Adaptive AI, Swarm Terror AI.Format
The twist is that each round AIs are granted bonus cheat multipliers based on the previous round's results. In brief, an AI gets a +0.1 boost to resources and build rate for every position down the leaderboard it was in the last round - and accumulates these bonuses as the tournament goes on.To stop these boosts getting out of hand in the matches, the boosts applied in game are scaled down based on the lower of the boosts between two AIs. For example if AI Alpha with boost 2.0 is playing AI Bravo with boost 2.4, then the applied settings would be a 1.0x multiplier for Alpha (2.0/2.0) and a 1.2x multiplier for Bravo (2.4/2.0). Applied boosts are rounded to the nearest 0.1 due to UI limitations.
The aim is that as the rounds progress, we get a better and better idea of what boosts each AI needs to be an equal match for every other AI (kinda like the AI's handicap). The AI with the lowest boost at the end of the tourney wins!
Matches:
M: Accumulated Modifier going into the round
W: Wins that round
P: End of round placement (* joint)Round 1 Round 2 Round 3 Round 4 Round 5 AI M W P M W P M W P M W P M W P Adaptive 1.0 3 8 1.7 7 3* 1.9 0 11 2.9 6 2* 3.0 2 9* Dalli 1.0 9 1* 1.0 2 10 1.9 9 2 2.0 5 7* 2.6 9 2 Dilli 1.0 8 3 1.2 3 8* 1.9 8 3 2.1 6 2* 2.2 2 9* DilliDalli 1.0 6 4* 2.3 7 3* 1.5 4 6* 2.3 2 11 2.9 10 1 M27 1.0 5 6* 1.5 9 1 1.5 3 9 2.3 5 7* 2.9 6 4* RNG Standard 1.0 9 1* 1.0 0 11 2.0 10 1 2.0 6 2* 2.1 4 6* SCTA Arm 1.0 2 9 1.8 5 6* 2.3 4 6* 2.8 3 9* 3.6 7 3 SCTA Core 1.0 1 10* 2.9 8 2 2.0 2 10 2.9 6 2* 3.0 4 6* Sorian Adaptive 1.0 1 10* 1.9 6 5 2.3 6 4 2.6 3 9* 3.4 6 4* Sorian Edit 1.0 6 4* 1.3 3 8* 2.0 5 5 2.4 6 2* 2.5 3 8 Swarm Terror 1.0 5 6* 1.5 5 6* 2.0 4 6* 2.5 7 1 2.5 2 9* Results:
For a final set of modifiers at (raw, normalised to nearest 0.1):
2.6, 1.0: RNG Standard
2.7, 1.0: Dalli
3.0, 1.2: Dilli, DilliDalli
3.2, 1.2: M27, Sorian Edit
3.3, 1.3: Swarm Terror
3.5, 1.3: SCTA Core
3.7, 1.4: Sorian Adaptive
3.8, 1.5: Adaptive, SCTA ArmCongrats to RNG Standard AI for winning, commiserations to everyone else!
For fun, here's a view of the normalised modifiers (normalised compared to leading AI that round) each AI had as the tourney progressed:
I think the scores would have continued to settle down a little bit if I'd kept doing more rounds, but after manually running 275 games in a row I didn't feel like anymore...
--
The particularly sharp amongst you may have noticed that the weekly tourney hasn't exactly been weekly lately - and going forwards this will become a less regular thing (maybe monthly?) so that I can focus on developing AIs to enter.
See you all in the next one!
-
@softles Thanks again for running these! Re moving to once a month I’m happy to run one once a month to share the load if you wanted to have it as a biweekly tourney?
-
AI biweekly tourney - AstromAInia
The humans had their turn on this map in a recent tourney, now it's the AI's time to show how the map should really be played!Format
This time there were 12 entrants - the 11 from the last tourney, and a new addition of the base game's AI: Turtle on what is arguably a turtler's dream map.The setup of the tourney is as follows:
All matches: Random AI faction with a random start position
- Group stage: The 12 AI are split into 4 groups of 3, based on their performance in the last tourney (twin rivers). Each AI plays the other AI in its group, with the worst AI being eliminated
- Quarter finals: The winner from each group plays the 2nd best AI from another group, in a best of 3
- Semi finals: The quarter final winners are paired up to fight in a best of 3, similar to the quarters
- Finals: The two finalists duke it out in a best of 5
The results
Group stagesGroup AI Wins Group 1 RNG Standard 1 Group 1 Swarm Terror 2 Group 1 SCTA Arm 0 Group 2 Dalli 2 Group 2 SCTA Core 0 Group 2 AI: Turtle 1 Group 3 Dilli 1 Group 3 M27 2 Group 3 Sorian Adaptive 0 Group 4 DilliDalli 2 Group 4 Sorian Edit 1 Group 4 Adaptive 0 Quarter finals
AI 1 (group stage winner) AI1 wins AI2 (group stage runner up) AI2 wins Swarm Terror 3 Dilli 0 Dalli 3 Sorian Edit 0 M27 3 RNG Standard 0 DilliDalli 1 AI: Turtle 2 Semi finals
AI 1 AI1 wins AI2 AI2 wins Swarm Terror 2 Dalli 1 M27 3 AI: Turtle 0 Finals
AI 1 AI1 wins AI2 AI2 wins Swarm Terror 0 M27 5 Winner: M27
Overall results
AI Wins Losses Win percent M27 13 0 100% Swarm Terror 7 6 54% Dalli 6 2 75% AI: Turtle 3 5 38% DilliDalli 3 2 60% Sorian Edit 1 4 20% Dilli 1 4 20% RNG Standard 1 4 20% SCTA Arm 0 2 0% SCTA Core 0 2 0% Sorian Adaptive 0 2 0% Adaptive 0 2 0% Other random observations
- First blood - Sorian Edit took just 8m09 to wrap up its winning game; in contrast, Swarm terror averaged 52m53 for the games it won
- Sneak attack - RNG was the only AI to make use of the rear entrance to the base, sending a galactic colossus through it in one game
- Best recovery - In one game Swarm lost 1/3 of its 18 base mexes, yet managed to repel the attack and eventually claim a win
- In a case of the AI results imitating the human tourney, the winning AI's last update was created only a short while before the tourney started.
-
I'll probably do another mini-AI tourney in 1-2 weeks - does anyone have any particular scenarios they're interested in seeing and/or are people still interested in these?
I've noted some scenario examples below; whether I pick a scenario will depend in part on if it won't take too long to manually run (i.e. the number of rounds and games that would be required), and my preference is probably towards a variety of maps (since the tourneys up until now have shown how differently the same AIs can perform based on the map chosen).
Tourney ideas:
- 1v1 5km / 10km / 20km Tourney - selection of just 5km, or just 10km or just 20km maps, based on some of the maps that have featured in recent 1v1 map pools
- Matchmaker based tourney - e.g. <700 rank map pool for 1v1; 2v2; or 4v4 full share
- Mimicking other recent human tourneys (e.g. the FFA tourney; King of Setons; King of Dual Gap)
-
This makes me depressive
-
Would love to see 20km map games with all 3 theaters. But I know we don't have enough coverage in the AI's for naval. But thats an area where we are weakest.
-
~-~-~
-
AI biweekly/monthly tourney: 20km Combined arms
The theme of this map was 20km maps that allow use of land, navy and air. The maplist chosen was:- Crossfire Canal - FAF version
- Norfair
- The Ditch - FAF version
- Setons Clutch - FAF version
- The Bermuda Locket - FAF version
Setup
Setup is similar to the last tourney - 12 AIs are split into 4 groups. The weakest 4 are eliminated in the first round, with the winners from one group fighting the runners up from the other in the quarter-finals in 3 matches.
As there were 5 maps, maps were chosen randomly for the group and quarters, and then for the semis and finals each map was played.
Starting positions are the default first and second positions set by the map (e.g. on Setons this means a 'mid vs mid' battle)Changes to AI
5 of the featured AI have released new versions since the last tourney: M27, Swarm Terror, SCTA Arm, Sorian Edit, SCTA CoreThe results
Group stagesGroup AI name Wins 1 M27 4 1 DilliDalli 2 1 SCTA Core 0 2 Swarm Terror 4 2 Sorian Adaptive 2 2 Sorian Edit 0 3 Dalli 3 3 Dilli 2 3 Adaptive 0 4 RNG Standard 4 4 SCTA Arm 2 4 AI: Turtle 0 Quarter finals
AI Name Wins AI Name Wins M27 3 Dilli 0 Swarm Terror 3 SCTA Arm 0 DilliDalli 3 Dalli 0 RNG Standard 3 Sorian Adaptive 0 Semi finals
AI Name Wins AI Name Wins M27 5 Swarm Terror 0 DilliDalli 5 RNG Standard 0 Finals
AI Name Wins AI Name Wins M27 5 DilliDalli 0 Total wins by AI
AI Name Wins Win % M27 17 100% DilliDalli 10 59% RNG Standard 7 58% Swarm Terror 7 58% Dalli 3 43% Dilli 2 29% Sorian Adaptive 2 29% SCTA Arm 2 29% Sorian Edit 0 0% SCTA Core 0 0% Adaptive 0 0% AI: Turtle 0 0% Detailed results
The full results can be seen in the 'DetailedResults' worksheet of the following file:
https://docs.google.com/spreadsheets/d/1i2ObsZ49S9gUTgWw0E68qZ8VrOkaP_Jk/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=trueOther Observations
- While some AI attempted to build navy, in all cases it proved ineffective with the mass invested in navy far exceeding the mass killed. Neither of the finalists built a single naval unit the entire tourney.
- The finals were closer than the scoreline suggests, with DilliDalli dominating land on most of the maps and winning map control (e.g. holding 5 out of the 6 islands at one point on Crossfire Canals, and effectively raiding and even claiming mexes behind M27's main base on Setons). The finals ended up being an air vs land contest on most maps.
- SCTA Arm managed to have 2 T3 tanks solo a GC (GC's shot was blocked by cliffs) but it wasn't enough to let it claim a win
- Heroic unit of the tourney: DilliDalli's lone striker vs M27 on Setons which managed 19 kills and 1,720 mass kills raiding mexes
- Notable changes in fortunes: Swarm managed a turnaround in speed, going from the slowest AI to claim wins in the last tourney to getting the second fastest win (11m21, beaten only by Dilli who got a win in 10m48)
-
AI biweekly/monthly tourney: MapGen
The idea of this tourney was to use MapGen to create a map for the AI to fight against for 5km, 7.5km, 10km, 15km and 20km. Due to the AI tourney still being run manually a single map was used for the map size for the tourney (rather than generating a new random map every time). The maplist chosen was:- neroxis_map_generator_1.8.5_vgf5tqx2uxrcu_aicae_
- neroxis_map_generator_1.8.5_uydyqpk4utuuu_aidae_
- neroxis_map_generator_1.8.5_xaghpcvo2lnrk_aieae_
- neroxis_map_generator_1.8.5_bvwpdifwrols4_aigae_
- neroxis_map_generator_1.8.5_44sxnthefcek6_aiiae_
Setup
Setup is similar to the last tourney - 12 AIs are split into 4 groups. The weakest 4 are eliminated in the first round, with the winners from one group fighting the runners up from the other in the quarter-finals in 3 matches.
As there were 5 maps, maps were chosen randomly for the group and quarters, and then for the semis and finals each map was played.Changes to AI
3 of the featured AI have released new versions since the last tourney: M27, SCTA Arm, SCTA CoreThe results
Group stagesGroup AI name Wins 3 DilliDalli 4 3 Sorian Edit 3 3 SCTA Arm 0 2 M27 4 4 SCTA Core 0 4 Dalli 2 4 RNG Standard 3 1 Adaptive 1 2 AI: Turtle 2 1 Swarm Terror 4 1 Dilli 2 2 Sorian Adaptive 0 Quarter finals
AI Name Wins AI Name Wins Swarm Terror 2 Sorian Edit 1 M27 3 Dalli 0 Dilli 2 DilliDalli 1 RNG Standard 3 AI: Turtle 0 Semi finals
AI Name Wins AI Name Wins Swarm Terror 0 M27 5 Dilli 1 RNG Standard 4 Finals
AI Name Wins AI Name Wins M27 4 RNG Standard 0 (M27 and RNG drew 1 game)
Total wins by AIAI Name Wins Win % M27 16 94% RNG Standard 10 59% Swarm Terror 6 50% DilliDalli 5 63% Dilli 5 38% Sorian Edit 4 50% Dalli 2 29% AI: Turtle 2 29% Adaptive 1 20% Sorian Adaptive 0 0% SCTA Arm 0 0% SCTA Core 0 0% Detailed results
The full results can be seen in the 'DetailedResults' worksheet of the following file:
https://docs.google.com/spreadsheets/d/1i2ObsZ49S9gUTgWw0E68qZ8VrOkaP_Jk/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=trueOther Observations
- Notable changes in fortunes: In the final, RNG had c.25 mass to M27's 621, yet still managed a draw (M27 attacked RNG's base with its ACU leading to both ACUs dying)
- The 7.5km map featured two land masses separated by water, which caused several AIs issues
- Although SCTA failed to get a win it managed to match Sorian Edit for map control and eco for much of the game, with both games going over an hour
- The group stages required two deciding matches to determine groupings, with an extra Dilli vs Adaptive and Sorian Edit vs DilliDalli match due to them getting the same number of wins as each other.
-
AI biweekly/monthly tourney: Sorian vs the world
The idea for this tournament was to see the highest AiX modifier that each AI could beat Sorian Adaptive at, on a selection of 5km, 10km and 20km maps from the current map pool, hopefully highlighting how much more comeptitive the FAF custom AIs are, as well as giving a better gauge of the distance between different AI than the previous tournaments win rate. The maplist chosen was:- Mirage
- Stickleback Ridge
- Open Palms
- 8 - Badlands_v4
- The Bermuda Locket - FAF version
- Crossfire Canal
Setup
Each AI entered will face off against Sorian Adaptive AiX (with no omni). If they win, then they repeat the fight but with Sorian Adaptive getting +0.1 to its modifier, until the challenging AI no longer wins. This is done for each map separately. This resulted in a mammoth 245 games being run, so I doubt I'll be repeating this format due to the massive amount of time it ended up taking (for comparison, the last tournament took just 51 games)Changes to AI
There were significant changes to the AI this time. In addition to a new version of M27, Sorian Edit, and SCTA, there was a major update for RNG (the first in many months), and we have a new AI entrant with Uveso Rush featuring for the first time!
The results
SummaryAI Name Average AiX Beaten M27 1.63 RNG Standard 1.57 DilliDalli 1.43 Swarm Terror 1.27 Uveso Rush 1.25 SCTA Core 1.15 Adaptive 1.05 Sorian Edit 0.98 Results by map
8 - Badlands_v4 Adaptive 1.0 8 - Badlands_v4 DilliDalli 1.4 8 - Badlands_v4 M27 1.6 8 - Badlands_v4 RNG Standard 1.6 8 - Badlands_v4 SCTA Core 1.6 8 - Badlands_v4 Swarm Terror 1.3 8 - Badlands_v4 Uveso Rush 2.0 8 - Badlands_v4 Sorian Adaptive 0.4 Crossfire Canal Adaptive 1.0 Crossfire Canal DilliDalli 1.1 Crossfire Canal M27 1.3 Crossfire Canal RNG Standard 1.3 Crossfire Canal SCTA Core 0.2 Crossfire Canal Sorian Edit 0.8 Crossfire Canal Swarm Terror 1.1 Crossfire Canal Uveso Rush 0.7 Crossfire Canal Sorian Adaptive 1.0 Mirage Adaptive 1.2 Mirage DilliDalli 1.5 Mirage M27 1.9 Mirage RNG Standard 1.7 Mirage SCTA Core 1.4 Mirage Swarm Terror 1.5 Mirage Uveso Rush 1.2 Mirage Sorian Adaptive 0.4 Open Palms Adaptive 1.1 Open Palms DilliDalli 1.6 Open Palms M27 1.6 Open Palms RNG Standard 1.6 Open Palms SCTA Core 1.3 Open Palms Sorian Edit 1.0 Open Palms Swarm Terror 1.1 Open Palms Uveso Rush 1.1 Open Palms Sorian Adaptive 0.6 Stickleback Ridge Adaptive 1.0 Stickleback Ridge DilliDalli 1.6 Stickleback Ridge M27 2.0 Stickleback Ridge RNG Standard 2.0 Stickleback Ridge SCTA Core 1.3 Stickleback Ridge Sorian Edit 1.1 Stickleback Ridge Swarm Terror 1.2 Stickleback Ridge Uveso Rush 1.5 Stickleback Ridge Sorian Adaptive 0.4 The Bermuda Locket - FAF version Adaptive 1.0 The Bermuda Locket - FAF version DilliDalli 1.4 The Bermuda Locket - FAF version M27 1.4 The Bermuda Locket - FAF version RNG Standard 1.2 The Bermuda Locket - FAF version SCTA Core 1.1 The Bermuda Locket - FAF version Sorian Edit 1.0 The Bermuda Locket - FAF version Swarm Terror 1.4 The Bermuda Locket - FAF version Uveso Rush 1.0 The Bermuda Locket - FAF version Sorian Adaptive 0.7 Detailed results
The full results can be seen in the Data worksheet of the following file:Other Observations
- Uveso was the dominant AI on badlands (due to its ability to punish Sorian Adaptive for sending its ACU to attack on its own early on), managing a win against a 2.2 AiX (it lost against a 2.1 AiX that was running at the same time hence why 2.0 is shown in its results). In contrast as an AiX 1.2 it lost to a 1.0 Sorian Adaptive on Crossfire Canal.
- Crossfire Canal was by far Sorian Adaptive's best map, with 3 of the AI (Sorian Edit, SCTA Core and Uveso) needing AiX modifiers to beat it (although not strictly correct, if an AI needed an AiX modifier to win I've shown each 0.1 modifier as decreasing the Sorian AiX beaten by 0.1, so an AI needing a 1.5 modifier to win would show as 0.5)
- Surprisingly, Adaptive AI was better than Sorian Adaptive, beating it on every map with no modifier, and beating 1.2 on Mirage.
- There were compatibility issues between Sorian Edit and Sorian Adaptive on some maps, so results for these have been omitted
- 3 AI managed the achievement of beating Sorian Adaptive AiX 2.0 or better - M27 and RNG (2.0, Stickleback Ridge) and Uveso (2.2, Badlands)
Edit: Note that some of the AI aren't intended for 20km or water based maps, which will be a reason for poor results on Crossfire (e.g. SCTA isn't meant for such maps). Excluding crossfire from the results would give the following:
-
AI monthly tourney: Survival of the fittest
Suggested by Fearghal, this tournament's theme is to see which AI does best on a survival map. chp2001 helpfully identified the map Final Rush Pro 5 as a potential candidate, since it features 1 team against another (thus avoiding the problem caused by survival maps which lack an enemy brain/base location for the AI to fight against).For those not familiar with it, it's a 4v4 map where teams face off against each other with a central chokepoint area where the AI spawns. The AI spawns will be delivered by transport, but air is disabled (so the transports are harder to shoot down).
The survival spawns are split allocated equally between two different survival 'brains', and wont attack each other, but will attack one (and only one) of the teams. This means that a team can choose to be aggressive by sending their units at the enemy base, to help support the survival AI's attack.
To avoid people exploiting the survival AI, ACUs are prevented from hiding in the water by taking damage over time, and prevented from hiding in the rear corner of the map behind cliffs by a group of galactic colossi. The map has a number of configurable settings, including difficulty, which affects the size of the surivival spawns, and how long it takes before it advances in waves (with a wave starting off with T1 units, before progressively getting harder, e.g. sending in bombers; t2 land; cybran destroyers; t3 land; experimental land; etc.). Players start with 9 hives, an ACU from each faction, and lots of mass points at their base.
Setup
In the first round, each AI will play against itself (i.e. all 8 players will be that AI), and see how long it takes for the last player to die. These times will then be used to give each AI an initial 'seeding', with the last seed playing the second last, the winner of that playing the 3rd last, and so-on until the last ranked AI has lost to the 2nd last ranked AI which has lost to the 3rd last ranked AI and so on.The map settings still present some issues for AI, in particular the logic for preventing exploiting the survival AI by hiding - I turned this off for the initial 'seeding' round over fears most of the AI wouldn't be able to understand the 'ACUs die when entering water' approach or that 'those hostile GC near your base can be ignored' , but it led to its own problems, e.g. fully upgraded ACUs hiding in the water being very slowly whittled down by the torpedoes of a couple of bricks and didnt give a fair reflection of how well some AI did. I therefore settled on a compromise for the main event, where a team would be treated as having lost if all 4 land bases were wiped out (i.e. it had no more mexes or land factories).
The difficulty was also set to easiest, as all AI struggle compared to humans with such a setup, and this way there was more of a chance of seeing waves hit the experimental land stage.
Changes to AI
New versions of M27, RNG, and SCTA were reflected for the tournament.
The results
Initial seedings - Time lasted when fighting itselfAI Name Time survived M27 11m25 AI Turtle 16m28 DilliDalli 17m27 Sorian adaptive 19m20 Adaptive 19m22 RNG Standard 29m06 Sorian Edit 31m41 Uveso Rush 34m02 SCTA Core 35m07 Swarm Terror 37m16 Final rankings and head to heads
AI Name Head to Heads won Head to Heads lost Longest time survived until a team died (in seconds) Sorian Edit 0 6 992 Sorian adaptive 1 3 1122 AI Turtle 2 3 1029 Adaptive 2 3 1080 SCTA Core 1 4 1807 DilliDalli 5 2 1176 RNG Standard 3 2 1629 Uveso Rush 2 2 1907 Swarm Terror 1 1 1940 M27 9 0 1940 For example, DilliDalli was initially seeded 9th (out of 10 AI); it lost to M27 (the 10th seed), but beat the 5 AI that placed worse than it in the final results to reach 5th place. It then lost to RNG (4th place), so didnt advance any further. In contrast, as Swarm was seeded first it only fought the 2nd place initially (which ended up being M27), and after losing that it then fought the then 3rd place (Uveso Rush) which it won, resulting in ending as 2nd place.
Detailed results
The full results can be seen in the Data worksheet of the following file:Other Observations
- There were a wide variety of approaches used by the different AI, for example:
- M27 favoured a firebase appraoch, but chose to place this near the centre of the map meaning it'd get overwhelmed around the time of T3 land or experimentals
- DilliDalli preferred an all-out assault on the enemy team before spawns even started arriving, sticking to T1 mexes but often being able to kill the enemy team before their economic advantage could be brought to bear
- Swarm chose to have a massive swarm of mostly T1 units occupying the central area, meaning they could quickly ambush the survival AI drops as they arrived (up until the late T3 land/early experimental stage)
- Uveso made heavy use of T1 artillery, which when firing from behind T3 shields at the base proved an effective roadblock to the survival AI's T3 land waves
- There were dramatic transformations in fortunes for the AI from their initial seedings. In DilliDalli's case this was because it's aggressive approach would lead to short matches in all its games, while in M27's case it had chain reactions caused by the various starting ACU (16 per team) being near the centre and one of them dying. Sorian Edit's reversal in fortunes was likely caused by the stricter measuring of when a team had lost, as it was usually better at retreating its ACU behind cliffs/into water where the survival AI couldnt reach it
- A number of games came down to the wire with one team losing its final base just before the opposing team lost its final base, including the final between M27 and Swarm, and the semi-final between Uveso Rush and Swarm
- SCTA performed strongly, with a survival time in head to heads of more than 30m (bettered only by Uveso, Swarm and M27), but DilliDalli's early aggression proved its undoing
- It looked like RNG would pull back from the brink in its game against Uveso Rush when it completed a cloaked laser upgrade on its ACU at its final base. Sadly it turns out that the map gives omni vision meaning its ACU still died quickly to the T3 land spam.
-
"This week I also pulled performance stats for each AI against itself in the two round 1 maps, which produced the results below:"
I have been testing for the upcoming RC tourny and have been using "mostly" M27 as the standard to
measure against the other Game specific AI's,
I also start test, at game option speed = Fast, (which is = to +4 sim) to make sure the game speeds are "not likely" to drop below "0" game speed.
Game speed limits the size of the maps I can use -
(if games slow to "-1", after 45 minutes, I limit the AI's to 10x10 maps)
The faster AI's like DilliDalli and RNG are easily able to handle the 20x20 maps.
Since the test games are a combo of the the Game AI and M27 -
I was wondering if with all the recent updates to M27 if it is still as fast as the performance results shown above.I would test vs Uveso, (since it needs to be loaded for all games anyway)
butit seems that after Uveso grows it's base large it also slows game down a little,
so I don't want to run tests with Uveso as the standard, as it might skew the test results.
Update: I just tested Uveso since post and It runs fast.
It was still +4 at end of 37 min game with 6 AI:Uveso and Lots of units!Would it be possible to include another performance chart after you run your next tourny,
with the updated AI versions?
Thanks. -
Per Fearghal's request, this AI tourney is a bit different, and focuses on how fast/slow the different AI run. The scenarios were as follows:
- Adaptive Tournament Decision (10km): Rainbow cup style setup, with 3 teams (3v3v2)
- Rebellion (20km): Simple 2 team head to head with all slots filled (so a 5vs5)
Due to how long games were taking, I ran 2 games on adaptive tournament decision and only one on rebellion.
Unfortunately I forgot to add tick freeze tracking to my profiling code until I was almost done with the games and couldnt face re-running them (as on Rebellion it was taking more than 1 hour for some of the games in real time).Results are shown both by time, and by unit count. The main point for me is this shows the largest factor in how slow the game will run is how many units are built. In particular, just looking at results by time DilliDalli is by far the quickest AI. However, looking at it by unit count there's little difference between it, RNG, SCTA and Adaptive. That said, Swarm, Uveso and M27 performed noticeably slower even when adjusting for unit count.
For the below graphs, the x axis is either the unit count or the game time in seconds. The y axis in both cases is how long in real-life seconds it took to run 10 game time seconds. I.e. a value of above 10 means that the average game speed was below +0.
Adaptive tournament decision
The large blue spike was a brief period when swarm reached -8 game speed
Rebellion
Unfortunately the current version of Uveso doesnt work with Sorian Edit (which requires Uveso) so no stats are available for this.
I also reran M27 to see how it did after the v47 optimisation release (c. 9% faster) but there wasn’t a noticeable difference in the graph lines so the above reflects the v45 position.
-
AI monthly tourney: 10km Analytics
Per a request by Relent0r, this tournament was aimed both at providing detailed stats for each AI (so each AI developer gets hopefully useful information), and to give weaker AI a boost to make the results less predictable. The current 1v1 ladder map TAG_Craftius Maximus has been selected since it allows navy to play a role without it being required (so both naval and non-naval AI get to compete), while it's also 10km which will be useful for those AI that aren't designed to function on 20km+ maps.
Setup
Each AI will fight each other AI once. The AI will get an AiX modifier based on their average placement in the last 3 tournaments, ranging from 1.0 for first place to 1.5 for last place (where more than 8 AI featured; 1.35 where only 8 AI featured). The actual AiX modifier will then be based on the difference between the AiX modifiers of the two competing AI (so for example DilliDalli will use an AiX 1.3 modifier, while RNG Standard will use 1.1; when fighting each other that means RNG 1.0 will fight DilliDalli 1.2). The AiX modifiers used are shown in the results.Changes to AI
Almost all of the AI have featured an update since the last tournament.The results
SummaryAI Name and AiX mod Games won DilliDalli 1.3 7 M27 1.0 6 Sorian Edit 1.4 5 RNG Standard 1.1 4 Swarm Terror 1.2 3 Uveso Rush 1.3 2 SCTA Core 1.4 1 Adaptive 1.4 0 Congratulations to DilliDalli/Softles, with DilliDalli beating every other AI with it's tank focused game style.
Detailed results
The full results including the cleansed data used for the above graphs can be seen in the Data worksheet of the following file:AI analysis/stats
The following summarise some of the stats obtained from the games (with the underlying data linked above if you wanted to analyse further). Feel free to suggest ways of displaying the data that you think would be interesting (currently stats are available relating to mass killed by unit, mass lost by unit, mass built by unit, the total mass split by active units at a particular point of time, and overall mass and energy income, with the stats recorded in 60 second intervals).Since it's the first time I've done this it's likely there'll be errors/unintended consequences with the stats (e.g. BuildingDefence is meant to cover units like shields and SMD but has evidently picked up SCTA nukes). If you spot anything odd let me know and I can look at updating the stat tracking for future tournaments.
The stat tracking functionality will be incorporated into the next release of M27 to allow it to be run even if M27 itself isn't playing in a game (by making the variable M27StatTracking true in the M27Config.lua file)
Note some of the below pictures will need opening/viewing zoomed in!
- Average army composition for all AI over the course of the game
(This only considers the shortest 75% of the games to avoid a handful of games resulting in a false impression of the AI unit composition)
-
Where each AI invested their mass
-
How effective each unit type was (mass kills as a percentage of mass built)
- Average army composition for all AI over the course of the game
-
M maudlin27 referenced this topic on
-
M maudlin27 referenced this topic on
-
M maudlin27 referenced this topic on
-
Great data. Is this last post still relevant ? I am kind of surprised about high Dillidali and M27 scores. I used both in probably over 30 games and at least on maps i use, M27 deals with Dillidali in a matter of minutes. I am guessing M27 has been improved a lot or maybe i broke Dillidali somehow ?
-
M27 will have improved slightly since then but I expect AiX DilliDalli with a 1.3 resource and build mod would still beat 1.0 M27 AI (i.e. no resource/build boost) on the majority of maps (which is the last tournament that I ran). Data on unit composition is likely to be largely unchanged for most AI, and with the exception of the Adaptive AI I wouldn't be expecting a significant change in rankings.
There have been a few changes since then though, most notably a new AI (M28), the Adaptive AI getting significant improvements, and some updates to M27, RNG and Uveso. I'll probably consider doing another AI tournament at some point to see how things stand - there's a good chance Swarm will be fixed in the not too distant future so it'll probably be after that. It takes a lot of time to run the tournaments though, hence why I'm no longer doing them regularly.
