What is your NAT type?
Windows 10 - go to Settings > Gaming > Xbox Networking
https://support.xbox.com/en-AU/help/hardware-network/connect-network/troubleshoot-party-chat
Make sure you run check twice as there is a bug where the first check can fail.

Windows 11 - Install Xbox console companion - then go to settings - network

It may not be a NAT issues however it is a good place to start.

Also check https://www.windowscentral.com/how-get-open-nat-xbox-one-enabling-upnp

Also make sure your outbound NAT source port is set to static and not dynamic (this differs between routers)
https://wiki.faforever.com/en/FAQ/Connection-Issuses-and-Solutions

Here is an example of what having a NAT outbound static source port looks like in pfsense
Untitled.png