Just played a match, and, similar to the OP, I was forced to recall against my will in 3v3 TMM.
Though we were on the backfoot, the game wasn't nearly over - at least in my eyes, that is.
I saw the recall notification pop up, then bam, two votes agreed.
No point in my vote.
I get the feeling that Team Match Maker should have all players make a unanimous vote to make it fairer for those who wish to stay.
Other than that, the game is already fullshare, so if they don't want to stay after the vote, they can always leave.
At the very least, I believe 3v3 TMM should have all players vote and 4v4 should have all or 75% of your team vote since it is a larger group.
~ Stryker