Default gateway route gone missing

[me@…]

At some point while configuring Zentyal as an OpenVPN client My Zentyal stopped displaying a default gateway:

# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
link-local      *               255.255.0.0     U     1000   0        0 br1
192.168.1.0     *               255.255.255.0   U     0      0        0 br1
199.188.193.0   *               255.255.255.0   U     0      0        0 eth0
# ip route
169.254.0.0/16 dev br1  scope link  metric 1000
192.168.1.0/24 dev br1  proto kernel  scope link  src 192.168.1.1
199.188.193.0/24 dev eth0  proto kernel  scope link  src 199.188.193.186

Internet access still works, I assume because there's a default gateway route in the default table:

# ip route show table default
default via 199.188.193.1 dev eth0

This causes problems when using an OpenVPN client because it can't determine the default route in order to replace it. I imagine it could cause problems elsewhere as well.

Initially, it displayed a default route as well. The problem persists after disabling the OpenVPN client or the module and rebooting. I'm not sure how to reproduce this problem but now that it's happening it's very persistent. What can I do to try and get more information?

[me@…]

Here's a bit from the {{{zentyal.log}} that may be relevant:

/sbin/ip route del default table default
/sbin/ip route del default
2012/06/25 10:21:51 DEBUG> Sudo.pm:164 EBox::Sudo::_root - sudo commands: ip route add table default default nexthop via 199.188.193.1 dev eth0 weight 1
2012/06/25 10:21:51 DEBUG> Sudo.pm:164 EBox::Sudo::_root - sudo commands: set -e
/sbin/sysctl -q -w net.ipv4.conf.all.rp_filter=0
/sbin/sysctl -q -w net.ipv4.conf.eth0.rp_filter=0
2012/06/25 10:21:51 DEBUG> Sudo.pm:164 EBox::Sudo::_root - sudo commands: set -e
/sbin/iptables -t mangle -F
/sbin/iptables -t mangle -X
/sbin/iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
/sbin/iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
2012/06/25 10:21:51 DEBUG> Sudo.pm:164 EBox::Sudo::_root - sudo commands: set -e
/usr/share/zentyal-network/flush-fwmarks
/sbin/ip route flush table 101
/sbin/ip route add default via 199.188.193.1 dev eth0 src 199.188.193.186 table 101
/sbin/ip rule add from 199.188.193.1 table 101
/sbin/ip rule add fwmark 1/0xFF table 101
/sbin/ip rule add table main

[me@…]

I was able to restore the default gateway by doing aptitude purge zentyal-network, then reinstalling all the zentyal packages I'm using and reconfiguring my network settings. So it appears some configuration data is getting corrupted at some point.

[me@…]

I was able to reproduce this by just enabling the VPN module and then enabling the VPN client connection.

[me@…]

Much easier to reproduce, if I just disable the network module then the main table has a default route, if the network module is enabled then it doesn't and only the default table has a default route.

[jamor@…]

Hello R. Patterson,

the default route is the show with 'ip route show table default'. You can configure the default gateway in Network -> Gateways.

I don't see what is the problem here, this is the intended behaviour.

[me@…]

Under normal conditions, a default gateway is shown in the "main" table and hence is shown by "route". When, as it is with the way Zentyal configures it, the default gateway is only in the "default" table then things, like OpenVPN clients, that count on being able to find the default gateway throug "route" or in the "main" table, may fail. In this case, a Zentyal server running an OpenVPN client will take no effect because OpenVPN can't find the default gateway in order to replace it:

Mon Jun 25 12:23:56 2012 NOTE: unable to redirect default gateway -- Cannot read current default gateway from system

Regardless, the routing tables as configured by Zentyal are unnecessarily different from what can generally be expected for a simple NAT gateway.

[jamor@…]

We have this setup because of multigateway rules. I copy below a mail from Carlos Pérez-Aradros where e explains the issue:

There is a good reason to do it this way:

exekias@fry:~$ sudo ip rule ls
0:      from all lookup local                                                                                                                                                                                  
32759:  from all lookup main                                                                                                                                                                                   
32760:  from all fwmark 0x1/0xff lookup 101                                                                                                                                                                    
32761:  from all fwmark 0x2/0xff lookup 102                                                                                                                                                                    
32762:  from 192.168.1.254 lookup 101                                                                                                                                                                          
32763:  from 192.168.1.1 lookup 101                                                                                                                                                                            
32764:  from 192.168.2.254 lookup 102                                                                                                                                                                          
32765:  from 192.168.2.1 lookup 102                                                                                                                                                                            
32766:  from all lookup main                                                                                                                                                                                   
32767:  from all lookup default         

As you can see, main table is repeated in this list, it is the first table 
(after loopback) and the penultimate (before default).

exekias@fry:~$ sudo ip route ls table main
192.168.100.0/24 dev eth0  proto kernel  scope link  src 192.168.100.254                                                                                                                                       
192.168.160.0/24 dev tap1  proto kernel  scope link  src 192.168.160.1                                                                                                                                         
192.168.161.0/24 dev tap2  proto kernel  scope link  src 192.168.161.1                                                                                                                                         
192.168.2.0/24 dev eth2  proto kernel  scope link  src 192.168.2.254 
192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.254 
192.168.159.0/24 dev tap3  proto kernel  scope link  src 192.168.159.1 
10.183.192.0/19 via 192.168.161.5 dev tap2  proto zebra  metric 2 
10.202.128.0/18 via 192.168.161.5 dev tap2  proto zebra  metric 2 
10.202.64.0/18 dev tap0  proto kernel  scope link  src 10.202.64.12 
10.200.0.0/16 via 10.202.64.1 dev tap0 


That table is in charge of route traffic between local networks, but not to 
the internet, which is done in 10X (for balanced traffic already marked) and 
default table (for new output traffic).

exekias@fry:~$ sudo ip route ls table 101
default via 192.168.1.1 dev eth1  src 192.168.1.254 

exekias@fry:~$ sudo ip route ls table default
default 
        nexthop via 192.168.1.1  dev eth1 weight 1
        nexthop via 192.168.2.1  dev eth2 weight 1


So, wen can change the name of main table and use main for gateways, but will 
be also strange for a newbie.  I think there is nothing to change here :)

[me@…]

As I described before, this behavior will still break anything that depends on getting the default gateway from route such as an OpenVPN client.

[me@…]

I'm doing something slightly different so I haven't tested this exact approach, but something like the following should work around this issue. Firstly, create a new routing table for the VPN default gateway and override the default gateway using a rule.

Copy /usr/share/zentyal/stubs/network/rt_tables.mas to /etc/zentyal/stubs/network/rt_tables.mas and add these lines to the latter:

10      vpn

Then move some rule priorities around and add a rule for the vpn table in a /etc/zentyal/hooks/network.postservice hook script:

main=$(ip rule list | grep -m 1 "lookup main $")
main_rule=$(echo $main | cut -d ":" -f 2)
main_priority=$(echo $main | cut -d ":" -f 1)
ip rule add $main_rule priority 16384
ip rule delete $main_rule priority $main_priority
ip rule add priority 24576 iif lo lookup vpn

Copy /usr/share/zentyal/stubs/openvpn/openvpn-client.conf.mas to /etc/zentyal/stubs/openvpn/openvpn-client.conf.mas and append:

script-security 2
route-up "/bin/sh -x -c 'route_net_gateway=$(ip route ls table default | cut -d \" \" -f 3) && ip route add $trusted_ip via $route_net_gateway table vpn && /sbin/ip route add default via $ifconfig_remote table vpn' route-up"