The network structure is:
Server -> NAT router1 -> switch <- NAT router2 <- Client
The part I cannot solve is that, after xserver connects to 177/udp of xdm, xdm connects back to xserver, say 6001/tcp. So the server network does not know how to connect to the IP inside another NAT.
The NAT routers are not rich functioned (as to routing), just exposes any ports necessary. For example, I exposed 22/tcp and 177/udp from Server.
I tried xqproxy, xdm just rejects the session id.