For anyone seeing in this in the future -

PLAY a custom game FIRST. It can just be a 1v1 versus an AI. The client does not create all necessary files and directories until you play your first game.

Replays should then work.