Setting up AI's to fight each other is ok to get a base number.
Good to have as side information side (another tab on your spreadsheet maybe? its easier to setup an AI game and walk away if your low on time). Small note here, recently the skirmish game mode was made to allow the human to become observer so you could run these on the offline executable without spamming replays to the replay server.
But I kinda agree with tatsu in that its not representative of how those same AI's will fair against humans and there needs to be a 'feel' indicator, especially when it comes to how they play with different human play styles, map preferences, cheat multipliers and even ai game settings. One might do ok against a turtle but garbage against a pressure player and vice versa. Its all 'eye of the beholder' sort of thing, but as someone whos stared at AI games far too much its worth something.