AI vs AI Tournament series

maudlin27

AI vs AI Tournament series

It's been a few years since the last of the AI vs AI tournaments was run, so I thought I'd work on refreshing this as and when I have time.

The idea for this tournament is simple - match every AI in a 1v1 against M28AI and see what AIx modifier it needs to win at least half its games on the map in question, to come up with a ranking of AI by difficulty.

There's no player participation for this tournament (for those interested in such things the rainbow cup is coming up in February, and I've pitched an idea for an AI vs player tournament to the tournament team separately but don't think it's going anywhere). However I thought it would be of interest both for people who play with/against AI, and for new players (as a relatively common question is what AI should they start with if they find M28 too hard and stock AI too easy, and don't want to use sub-1.0 AIx modifiers)

To start with, I'll be running the tournament on the 5km map Theta Passage. This is mainly because it takes far less time to run such games, but if/when I have time I'll be looking to progress to a 10km and possibly then a 20km map (as AI performance will vary significantly based on map size - for example I'd expect much higher modifiers to be required by some of the weaker AI to win on a 5km map where for example an ACU's combat potential has a much bigger impact than on a 20km+ map).

maudlin27

AI tourney: 5km Theta Passage
Each AI faces off against M28AI in a 1v1 on Theta Passage to see what AIx build and resource cheat multiplier it needs to win at least half of its games.

Setup
M28AI will fight against the enemy AIx using mirrored factions. Based on how heavy the win/loss the AI modifier gets adjusted to try and quickly arrive at the AIx modifier that means half of games get lost and half get won (with 1 game being played for each of the 4 factions at that modifier). In the event more/fewer games are won at that modifier, the next closest modifier will be checked for all 4 factions, and if appropriate this will result in a 'hybrid' AIx modifier (e.g. if an AI wins 75% at 1.2, and wins 25% at 1.1, it will be shown as needing a 1.15 modifier). The AI will then be ranked based on the AIx modifier needed.

Changes to AI
Compared to the AI tournament from more than 3 years ago the more notable changes are probably:

M28AI (new AI)
Various updates to RNGAI
Minor updates to M27AI
AI compatible with steam, LOUD and FAF (M28AI) now available which allows for a comparison between all Supcom AIs (not just FAF AI)
Default (adaptive) FAF AI has had some improvements made to it
These are in addition to any FAF balance changes over the last few years

The results
Summary

AI Name	AIx modifier needed to win half of games
M28 Adaptive	1.0
M28 Easy	1.075
M27	1.1
RNG Standard	1.25
DilliDalli	1.3
Swarm terror	1.7
Adaptive	1.8
Sorian Edit	1.8
Uveso Rush	1.9
Sorian adaptive (FAF)	2.4
Steam: Nutcracker	2.675
Steam: Adaptive	2.6
Steam: Sorian Adaptive	2.775
Vanguard	2.325
LOUD	N/A

Most AI achieved a 50% win rate at the given modifier, with only a few (RNG, M28Easy and the Steam AI) having more mixed results. LOUD is shown as N/A for this map as it crashed 4 minutes into playing (LOUD is designed for 20-40km maps, and while it has 1 10km map, it has no 5km maps and is not designed for maps of this size. On a 10km map I'd expect it to need around a 1.2-1.3 modifier to reach parity but would need more detailed testing - so hopefully it will feature if I do a re-run of this tournament on a 10km map)

Detailed results
The full results (i.e. showing games played by faction and modifier, and how long it took for the AI to win/lose) can be seen in the Data worksheet of the following file:

https://docs.google.com/spreadsheets/d/1aN41mAhv8-lfoR2RcEDL7NmG6vXoJug-/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=true

Other Observations

Updates made to the base AI appear to have had an impact - when the last AI tournaments were done the 'default' AI performed notably worse than 'mid-tier' AI such as Uveso
M28Easy looked to be cruising to a comfortable win at 1.1 in all its matchups but got completely destroyed by a single TML in one game.
Despite being a 5km map, many games were won at the experimental stage, with M28 achieving one of its wins with a T3 artillery
Against the steam AI a major vulnerability (leading to very high modifiers being required) was they would park their ACU in TML range with no TMD or shields, and wouldn't move even after a first missile landed.
Surprisingly (to me) despite more than 100 games being played, no games were tied
I'd wanted to include both the FAF Sorian AI version, and the steam version, in case Sorian was much less effective due to FAF balance changes (as it was written for steam). To my surprise, the FAF version performed (relatively) much better than the steam version. I therefore suspect the steam balance is such that it lends itself very well to M28's playstyle (despite M28's logic being designed for FAF), hence why such a high modifier was required for the steam AI to win.

maudlin27

In the event I find time to run a tournament on a 10km map, I've created a poll to get views on what map to choose in the AI dev discord. The FAF information resources channel includes links to various other servers including the AI development discord:
https://discord.com/channels/197033481883222026/433140724276527109/801914101324054609

With the post being in the tournaments channel of the AI dev discord:
https://discord.com/channels/619646141306503181/975157386614542377

Also if anyone has an AI not featured in this list that they want included let me know

waffelzNoob

Proxy signing up FtxCommando

Nuggets

signing up as AI: Random

Skrat

signing up @paralon

maudlin27

AI tourney: 10km Twin Rivers
Since the vote on the next map for the tournament was tied between twin rivers and white fire, I decided to go with Twin Rivers.

Each AI faces off against M28AI in a 1v1 on Twin Rivers to see what AIx build and resource cheat multiplier it needs to win half of its games.

Setup
M28AI will fight against the enemy AIx using mirrored factions. Based on how heavy the win/loss the AI modifier gets adjusted to try and quickly arrive at the AIx modifier that means half of games get lost and half get won; in some cases where significant factional differences are suspected or I have the patience to run more games, games will be run with each faction to identify the lowest modifier required with that faction for the AI to win. Results will then be displayed for the estimated modifier (e.g. if an AI wins with a 1.2 modifier with 3 factions, but a 1.1 modifier with 1, then it is assumed the 3 factions would draw with a 1.15 modifier, and the other faction would draw with a 1.05 modifier, meaning it's shown with a modifier of 1.125, i.e. (1.15*3+1.05)/4.

Changes to AI
Since the last tournament there have been minor updates to M28AI and RNGAI, and more interestingly a new AI mod is on the FAF vault, currently on v102, called Vanguard AI.
See the 5km tournament for details on other changes made in the previous 3 years
The results
Summary

AI Name	AIx modifier needed to win half of games
M28 Adaptive	1.0
M28 Easy	1.0625
M27	1.0625
RNG Standard	1.175
DilliDalli	1.25
LOUD	1.375
Sorian Edit	1.475
Adaptive	1.6
Uveso Rush	1.6
Steam: Sorian Adaptive	1.65
Vanguard	1.7
Sorian adaptive (FAF)	1.825
Steam: Nutcracker	1.8625
Steam: Adaptive	1.925
Swarm terror	2.375

Detailed results
The full results including the data used for the above graphs can be seen in the Data worksheet of the following file:

https://docs.google.com/spreadsheets/d/101PqVFKozMaWcHbaceIbabLGDsfdnoY-/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=true

Other Observations

M28 was generally hurt by going for navy on one of the ponds, with a significant mass investment for minimal benefit
AIs with high modifiers would typically win with nukes or an early experimental
M28 was able to win and hold both plateaus in almost every game it won
Land played a significant role with air generally not having much impact unless an early air experimental from a high AIx modifier, or M28 using gunships defensively
Swarm's poor performance (most AI would be expected to require a lower AIx modifier the larger the map) appeared due to not building/using anti-air units effectively until late in the game allowing M28's gunships to attack its tanks with impunity
Games on LOUD would generally go on far longer than on FAF (despite it being a 10km map which I thought might be enough to mean shorter games - I was curious whether LOUD's typically long game times are due primarily to its preference for 20-40km map sizes). My guess is a significant part of the reason for the long game times is the effectiveness of shielding (e.g. no shield damage overflow and rapid recharge times meant M28 in one game built more than 8 T3 artillery before it could break through T3 shields, in another game a mavor and 3 T3 arti were not sufficient for it to break through)
I've updated the 5km tournament results to include the new Vanguard AI, and also to re-run all the steam results in case they were affected by a subsequent bug discovered that could cause AIx overwhelm settings to be enabled and adjust resource multipliers (however the steam results only changed marginally so the bug didn't appear to have had a significant impact)
The average AIx modifier on twin rivers was 1.57, compared with 1.9 for Theta Passage. Such a decrease is expected since as the map size increases ACUs have less impact, it takes longer for land units to reach the enemy (so greater risk of being defended and/or out-teched by the units they meet), a greater proportion of mexes tend to be outside the core base (meaning greater benefit from the initial expansion boost provided by a build rate modifier), the economy (and hence resource modifier) generally plays a bigger role due to this, and high impact units that can end the game without significant investment to defend against them (like a land experimental or a nuke built with a build rate modifier) become much easier to build (more resources, and more time).