Friend can't join my custom games

My friend can't join any custom games I host, and vice versa. We are both able to join and play in other people's custom games though, at the same time as well. When setting up the lobby I just use default settings. Am I doing something wrong? Is there a reason he can't join my games?

Did you "foe" each other?