So I have finally found the issue and would like to share the cause:
I use a KVM switch for my 2 displays. Now I have bypassed my main display (have it directly connected) and the crashes are gone.
Sometimes my KVM switch would drop the displayport connection and I have to switch it back and forth. It seems that this dp drop caused the crashes