Some update from my friend, when we switched his windows power plan setting from Balanced to High Performance, the lag was vastly reduced. So might be related to how windows scheduler handles the loading of files from the OS or something like that (assuming this task is delegated to a background thread). Worth trying for anyone else with the issue