I feel like the most likely reason people leave is because they have other games to play. Some people want to try RTS but inevitably decide they prefer fps or mmorpg, etc. Every game has a learning curve and every game has new players, good players, and great players.
I don't really feel that FAF has alot of abrasive players; most people who play a game as old as this one do so because it's great and have an appropriate sense of ettiquette as a result.
I do have some suggestions based on my time here that I think would greatly improve player retention both for new players and experienced players:
Ladder matching i think would work better if instead of matching closely related ladder ranking, there are also matched based on number of games played. Maybe create a new players bracket where they'll only be matched with other players in the same bracket, maybe 0-99 games or 0-199 games. This will give new players to play against other new players, learn from each other, add people to their friends list, etc.
Instead of having ladder games launching every 3 mintues, I recommend having them launch after X number of players are queued up to play. The matching system can then more closely match rated players. Possibly it would be nice for the client to show how many people are queued to play(EDIT: I see in 1.4 this has already been added) / how many people are required for the games to launch.
Upon matching, I think it would be absolutely awesome if each pair received a popup of 3 ladder maps and given the OPTION to veto one of them. The system could pull a 5k, 10k, and 20k map from the pool, each player can then accept all 3 maps or veto one of them not to play. if both players choose different maps to veto, the 3 map gets played. If 2 or more maps are left unvetoed, the system can then randomly pick. This will allow players at both low and high levels to play a wider variety of maps. This will also allow people who don't have time to play a large 20k map to veto it to play a quicker game and vice versa. This would be a big boon for people to enjoy ladder more. I personally didn't play any ladder in december because I really didn't like most of the pool. (BONUS: devs can also use the information received about map veto vs draws to determine what maps people like vs don't like, thereby allowing everyone to essentially have a vote as to which maps to remove from the pool everytime it changes)
As for custom games, Morax has a very good point about people getting kicked from all welcome games. I might suggest required hosters to retype the game title each time instead of the client saving the name from the last game, although I can see some players saying that would be annoying. Sadly I have no other idea on who to improve custom games simply because of the nature of custom games.