There is a specific option that enables mouse panning in windowed mode, have you enabled this?
Personally, I find zooming out and zooming back into my desired location is faster and more accurate than edge panning with the mouse or arrow keys but I suck so that's probably part of the reason why.