Thanks @BlackYps and @Cetea
To make it easy for people, I will put the instructions for how to get ASI working for your replays in just 1 post
This assumes that you already have ASI installed on your computer through the installer. How to get/install ASI is a separate thing that I won't put here.
Step 1: Go to your "gamedata" folder.
Step 1 A: go to the "hamburger button" in the client (what is a hamburger button?)
Step 1 B: select "show replay folder." The path should be something like "C:/ProgramData/FAForever"
Step 1 C : double click on the "gamedata" folder
Step 2: select the file "Advanced strategic icons.nxt" right click on it, and select "copy"
Step 3: go to the replaydata/gamedata folder
Step 3 A: go back one level (go back to the folder that came up when you chose the option to "show replay folder"
Step 3 B: double click on "replaydata" to open that folder
Step 3 C : double click on "gamedata" to open that folder. This is a different folder than you had open before. This one will have a path named something like "C:/ProgramData/FAForever/replaydata/gamedata"
Step 3 D : right click inside this folder and select "paste." Since you had the "Advanced strategic icons.nxt" file copied, a copy of that file will be pasted into this folder
And that's it, it should be working. At least, this is what worked for me.