There might be a auto-ban from custom/team games probably, You won't be toxic if you play ladder 1vs1)
I am avid dota player and i'm quite toxic then i see people doing dumb shit.
if you get reported for 'communication missconduct' you get banned from chat, but still can use 'pings' or predefined chat commands like 'retreat/attack/etc'. It works pretty nicely.
You also have 'behaviour score' and griefers/players with boosted rank/etc get it low and basically get matched with same low score, so where is like cesspit for all griefers who can grief each other but they get seprated from 'good behaving' players.
Also even if y get 3+ reports and get low behaviour score you can play nicely and get commended so your score jumps back. So it's sort of easy to fix by being nice and helping your team and playing properly.
Something like this requires a ton of coding, but maybe something simpler can be coded, even simple system like seeing public comments or negative downvote from someone i know like "zlo" or other 'decent' players, would make me consider playing AGAINST that person in team game. Currently i 'foe' players, but it works kinda bad cause you can't see foes in lobby, half the time i just forget i foed someone and later find out i already foed someone for example.