The only "annoying" delay I encountered, is the lag delay from other players, which can result in several seconds until the fab reacts again after it has upgraded or changed its units queue.
I've not found out why this occurs yet . I think it is because the 'previous' factory occupies the build location, and therefore the factory waits a few seconds for it to clear. But I have no idea if that is correct