Your shown rating is the rating the system is 99.7% sure you can at least play at. When you beat players, the system is more and more confident that the rating can rise. If you lose to a 500 rated guy when the system thinks you are 99.7% likely to be no worse than a 300, then you don't lose much, because that data fits the distribution. You do lose uncertainty though, which in turn impacts how much the wins will give and also in turn actually results in you losing more rating via your median going down. Think of the game looking at your skill as a player as a statistical distribution. More data means a tighter expectation of performances, losses will shift the top of the bell curve but the shown rating won't show that.
The short of it is that the loss is impacting you in a variable you don't actually see, your median rating, which is what the system actually uses to determine balanced games by messing with the uncertainty it has about you. Wins impact your shown rating because it drastically increases the minimum expectation the system can have of you as a player. You only see that shown rating increase because that's what is used to maintain leaderboards in a TrueSkill system.