AI vs AI Tournament series

maudlin27

AI vs AI Tournament series

It's been a few years since the last of the AI vs AI tournaments was run, so I thought I'd work on refreshing this as and when I have time.

The idea for this tournament is simple - match every AI in a 1v1 against M28AI and see what AIx modifier it needs to win at least half its games on the map in question, to come up with a ranking of AI by difficulty.

There's no player participation for this tournament (for those interested in such things the rainbow cup is coming up in February, and I've pitched an idea for an AI vs player tournament to the tournament team separately but don't think it's going anywhere). However I thought it would be of interest both for people who play with/against AI, and for new players (as a relatively common question is what AI should they start with if they find M28 too hard and stock AI too easy, and don't want to use sub-1.0 AIx modifiers)

To start with, I'll be running the tournament on the 5km map Theta Passage. This is mainly because it takes far less time to run such games, but if/when I have time I'll be looking to progress to a 10km and possibly then a 20km map (as AI performance will vary significantly based on map size - for example I'd expect much higher modifiers to be required by some of the weaker AI to win on a 5km map where for example an ACU's combat potential has a much bigger impact than on a 20km+ map).

maudlin27

AI monthly tourney: 5km Theta Passage
Each AI faces off against M28AI in a 1v1 on Theta Passage to see what AIx build and resource cheat multiplier it needs to win at least half of its games.

Setup
M28AI will fight against the enemy AIx using mirrored factions. Based on how heavy the win/loss the AI modifier gets adjusted to try and quickly arrive at the AIx modifier that means half of games get lost and half get won (with 1 game being played for each of the 4 factions at that modifier). In the event more/fewer games are won at that modifier, the next closest modifier will be checked for all 4 factions, and if appropriate this will result in a 'hybrid' AIx modifier (e.g. if an AI wins 75% at 1.2, and wins 25% at 1.1, it will be shown as needing a 1.15 modifier). The AI will then be ranked based on the AIx modifier needed.

Changes to AI
Compared to the AI tournament from more than 3 years ago the more notable changes are probably:

M28AI (new AI)
Various updates to RNGAI
Minor updates to M27AI
AI compatible with steam, LOUD and FAF (M28AI) now available which allows for a comparison between all Supcom AIs (not just FAF AI)
Default (adaptive) FAF AI has had some improvements made to it
These are in addition to any FAF balance changes over the last few years

The results
Summary

AI Name	AIx modifier needed to win half of games
M28 Adaptive	1.0
M28Easy	1.075
M27	1.1
RNG Standard	1.25
DilliDalli	1.3
Swarm terror	1.7
Adaptive	1.8
Sorian Edit	1.8
Uveso Rush	1.9
Sorian adaptive (FAF)	2.4
Steam: Nutcracker	2.7
Steam: Adaptive	2.85
Steam: Sorian Adaptive	2.85
LOUD	N/A

Most AI achieved a 50% win rate at the given modifier, with only a few (RNG, M28Easy and the Steam AI) having more mixed results. LOUD is shown as N/A for this map as it crashed 4 minutes into playing (LOUD is designed for 20-40km maps, and while it has 1 10km map, it has no 5km maps and is not designed for maps of this size. On a 10km map I'd expect it to need around a 1.2-1.3 modifier to reach parity but would need more detailed testing - so hopefully it will feature if I do a re-run of this tournament on a 10km map)

Detailed results
The full results (i.e. showing games played by faction and modifier, and how long it took for the AI to win/lose) can be seen in the Data worksheet of the following file:

https://docs.google.com/spreadsheets/d/1aN41mAhv8-lfoR2RcEDL7NmG6vXoJug-/edit?usp=sharing&ouid=100973959280546778272&rtpof=true&sd=true

Other Observations

Updates made to the base AI appear to have had an impact - when the last AI tournaments were done the 'default' AI performed notably worse than 'mid-tier' AI such as Uveso
M28Easy looked to be cruising to a comfortable win at 1.1 in all its matchups but got completely destroyed by a single TML in one game.
Despite being a 5km map, many games were won at the experimental stage, with M28 achieving one of its wins with a T3 artillery
Against the steam AI a major vulnerability (leading to very high modifiers being required) was they would park their ACU in TML range with no TMD or shields, and wouldn't move even after a first missile landed.
Surprisingly (to me) despite more than 100 games being played, no games were tied
I'd wanted to include both the FAF Sorian AI version, and the steam version, in case Sorian was much less effective due to FAF balance changes (as it was written for steam). To my surprise, the FAF version performed (relatively) much better than the steam version. I therefore suspect the steam balance is such that it lends itself very well to M28's playstyle (despite M28's logic being designed for FAF), hence why such a high modifier was required for the steam AI to win.

maudlin27

In the event I find time to run a tournament on a 10km map, I've created a poll to get views on what map to choose in the AI dev discord. The FAF information resources channel includes links to various other servers including the AI development discord:
https://discord.com/channels/197033481883222026/433140724276527109/801914101324054609

With the post being in the tournaments channel of the AI dev discord:
https://discord.com/channels/619646141306503181/975157386614542377

Also if anyone has an AI not featured in this list that they want included let me know

waffelzNoob

Proxy signing up FtxCommando

Nuggets

signing up as AI: Random

Skrat

signing up @paralon