Bridged Xen domU with gateway in different subnet - linux

Question

I have a Xen dom0 running Debian Wheezy (7.8) and Xen 4.1, set up with bridged networking.
199.XXX.161.64 is the dom0 gateway.
199.XXX.161.65 is the dom0 address.
192.XXX.13.128/28 is the subnet for the domU's.
Configuration dom0:
root#dom0:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
auto xenbr0
iface xenbr0 inet static
address 199.XXX.161.65
netmask 255.255.255.254
network 199.XXX.161.64
broadcast 199.XXX.161.65
gateway 199.XXX.161.64
dns-nameservers 199.XXX.162.41 199.XXX.162.141
bridge_ports eth0
bridge_stp off # disable Spanning Tree Protocol
bridge_fd 0 # no forwarding delay
bridge_maxwait 0 # no delay before a port becomes available
allow-hotplug xenbr0 # start interface on hotplug event
root#dom0:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP qlen 1000
link/ether 00:25:90:d5:06:1a brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 00:25:90:d5:06:1b brd ff:ff:ff:ff:ff:ff
4: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 00:25:90:d5:06:1a brd ff:ff:ff:ff:ff:ff
inet 199.XXX.161.65/31 brd 199.XXX.161.65 scope global xenbr0
inet6 fe80::XXXX:90ff:fed5:61a/64 scope link
valid_lft forever preferred_lft forever
8: vif1.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master xenbr0 state UP qlen 32
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
inet6 fe80::fcff:ffff:feff:ffff/64 scope link
valid_lft forever preferred_lft forever
root#dom0:~# brctl show
bridge name bridge id STP enabled interfaces
xenbr0 8000.002590d5061a no eth0
vif1.0
root#dom0:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 199.XXX.161.64 0.0.0.0 UG 0 0 0 xenbr0
192.XXX.13.128 0.0.0.0 255.255.255.240 U 0 0 0 xenbr0
199.XXX.161.64 0.0.0.0 255.255.255.254 U 0 0 0 xenbr0
root#dom0:~# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif1.0 --physdev-is-bridged
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif1.0 --physdev-is-bridged udp spt:68 dpt:67
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif1.0 --physdev-is-bridged
ACCEPT all -- 192.XXX.13.129 0.0.0.0/0 PHYSDEV match --physdev-in vif1.0 --physdev-is-bridged
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
This host can reach its gateway and thus the internet.
root#dom0:~# ping -c 1 199.XXX.161.64
PING 199.XXX.161.64 (199.XXX.161.64) 56(84) bytes of data.
64 bytes from 199.XXX.161.64: icmp_req=1 ttl=64 time=0.459 ms
--- 199.XXX.161.64 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.459/0.459/0.459/0.000 ms
I also have a domU (with the same OS) which needs a primary IP address in a different subnet. There is no gateway on the network in this subnet. I want to keep my network setup bridged (no dom0 routing or NAT) so I added the dom0 gateway as the gateway for the domU as described in this blogpost.
Configuration domU:
root#domU:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:16:3e:b7:7e:cc brd ff:ff:ff:ff:ff:ff
inet 192.XXX.13.129/28 brd 192.XXX.13.143 scope global eth0
inet6 fe80::XXXX:3eff:feb7:7ecc/64 scope link
valid_lft forever preferred_lft forever
root#domU:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 199.XXX.161.64 0.0.0.0 UG 0 0 0 eth0
192.XXX.13.128 0.0.0.0 255.255.255.240 U 0 0 0 eth0
199.XXX.161.64 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
With this configuration the domU still has no network access. To test if the bridge was working I manually added a route to dom0.
root#domU:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 199.XXX.161.64 0.0.0.0 UG 0 0 0 eth0
192.XXX.13.128 0.0.0.0 255.255.255.240 U 0 0 0 eth0
199.XXX.161.64 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
199.XXX.161.65 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
Now the dom0 and domU can communicate through the bridge.
root#domU:~# ping -c 1 199.XXX.161.65
PING 199.XXX.161.65 (199.XXX.161.65) 56(84) bytes of data.
64 bytes from 199.XXX.161.65: icmp_req=1 ttl=64 time=0.037 ms
--- 199.XXX.161.65 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.037/0.037/0.037/0.000 ms
root#dom0:~# ping -c 1 192.XXX.13.129
PING 192.184.13.129 (192.XXX.13.129) 56(84) bytes of data.
64 bytes from 192.XXX.13.129: icmp_req=1 ttl=64 time=0.100 ms
--- 192.XXX.13.129 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.100/0.100/0.100/0.000 ms
However, the domU still can't reach the gateway.
root#domU:~# ping -c 1 199.XXX.161.64
PING 199.XXX.161.64 (199.XXX.161.64) 56(84) bytes of data.
From 192.XXX.13.129 icmp_seq=1 Destination Host Unreachable
--- 199.XXX.161.64 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
I have attempted to log establish if the traffic is actually being sent through the bridge by inserting a -j LOG rule at the top of the INPUT, OUTPUT and FORWARD iptables chains. When the domU attempts to ping the gateway the dom0 does not log a single packet. I also tried manually adding an entry for the gateway in domU's ARP table but the results were the same. The domU can't reach the gateway and thus has no network access other than being able to communicate with dom0 via static route.

Answer 1

So if I am understanding this correctly, the following is the network configuration for your DomU:
192.XXX.13.129/28 - DomU IP Address
199.XXX.161.64 - DomU GW Address
The problem is that your DomU does not have a route (layer 3) to allow it to talk to the GW address since the GW address is in a different subnet. So even though the router is on the same layer 2 network, the router (if it is processing your packets) does not know about your layer 3 network and is sending it's responses to it's default gateway.
That you are able to ping the Dom0 from the DomU is odd and probably the result of both the Dom0 and the DomU using the same Linux Bridge (which it not a true ethernet switch, more like a dumb hub).
The simple fix is to add an address from your DomU network to the LAN interface on your router.
The better fix would be to use VLANs to segment the different networks via layer 2 and replace Linux Bridges with Open vSwitch. This would completely isolate the Dom0 and DomU traffic so that they would be required to communication via a router and possibly a firewall.