Pausing game memory leak lag

Pausing the game for prolonged periods will make the game lag, I thought this was caused by a mod on a survival map but I think it happens on the current build with no mods. I don't think this effects other clients just the one who paused the game.