POSTROUTING marks packets already marked the wrong way in multigateway config

[stevz3@…]

scenario: external interface eth1 (192.168.0.254/24) with 2 hardware gateway boxes that do NAT on their own (192.168.0.1 and 192.168.0.2) up saving, two iptables POSTROUTING for table mangle get created - each with the same condition "-o eth1" first rule with --setmark 1, second rule with --setmark 2 this kills incoming traffic on the first gateway (192.168.0.1) since sessions that came in through the mac of 192.168.0.1 get the right mark 1 and will pass outbound first with mark 1, but get rewritten to mark 2 even their is already a mark 1 on the packets during the POSTROUTING defect introduced in:  http://trac.zentyal.org/changeset/18808 fix: Network.pm # mark in postrouting (only if more than one output iface) if ( scalar keys %interfaces > 1 ) {

push(@cmds, '/sbin/iptables -t mangle -A POSTROUTING ' . "-o $iface -m mark --mark 0/0xff -j MARK --set-mark $mark");

} the missing check IMHO is "-m mark --mark 0/0xff" this will only mark outbound packets on the given interface if they have not been marked yet

[cperez@…]

Hi Steve,

As you can see in the code, first check to introduce that rules is for make sure that gateways are in different ifaces, if both gateways are in the same interface it does not introduce the rules.

if ( scalar keys %interfaces > 1 )

Can you explain your full scenario? do you have gateway in another interface?

Thank you for your report and for looking for a solution! We really appreciate?

[stevz3@…]

ok, here is a list of all the gateways:

ppp-gw-eth2 ppp0/93.200.xxx.xxx   	eth2   	1   	yes   	
fixedgw1 	192.168.101.204 	eth1 	1 	no 	
fixedgw2 	192.168.101.201 	eth1 	1       no

ppp-gw-eth2 is a pppoe dsl line that gets dialed by zentyal via pppoe modem (works fine)

fixedgw1 & 2 are hardware dsl routers which do their own natting and dialing

these are the interfaces and their respective macs:

eth0 00:0c:29:c8:da:d3 inet addr:192.168.100.254 Bcast:192.168.100.255  Mask:255.255.255.0
eth1 00:0c:29:c8:da:dd inet addr:192.168.101.254 Bcast:192.168.101.255  Mask:255.255.255.0
eth2 00:0c:29:c8:da:e7 (no associated ip)

hardware routers and other machines

arp -a
? (192.168.100.2) at 00:0c:29:52:be:9a [ether] on eth0
? (192.168.101.201) at 00:1c:4a:bb:aa:ba [ether] on eth1
? (192.168.101.204) at 00:1c:4a:b1:a3:1c [ether] on eth1

original output:

iptables-save | grep mark
-A PREROUTING -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A PREROUTING -i ppp0 -m mark --mark 0x0/0xff -j MARK --set-xmark 0x1/0xffffffff
-A PREROUTING -m mark --mark 0x0/0xff -m mac --mac-source 00:1C:4A:B1:A3:1C -j MARK --set-xmark 0x2/0xffffffff
-A PREROUTING -m mark --mark 0x0/0xff -m mac --mac-source 00:1C:4A:BB:AA:BA -j MARK --set-xmark 0x3/0xffffffff
-A PREROUTING -m mark --mark 0x0/0xff -j MARK --set-xmark 0x1/0xffffffff
-A PREROUTING -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A OUTPUT -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A OUTPUT -m mark --mark 0x0/0xff -j MARK --set-xmark 0x1/0xffffffff
-A OUTPUT -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A POSTROUTING -o ppp0 -j MARK --set-xmark 0x1/0xffffffff
-A POSTROUTING -o eth1 -j MARK --set-xmark 0x2/0xffffffff
-A POSTROUTING -o eth1 -j MARK --set-xmark 0x3/0xffffffff
-A POSTROUTING -j CONNMARK --save-mark --nfmask 0xff --ctmask 0xffffffff

here is the log of iptables (introduced as last PREROUTING rule with LOG):

Nov  8 19:39:00 gateway kernel: [ 6806.793222] IN=eth1 OUT= MAC=00:0c:29:c8:da:dd:00:1c:4a:b1:a3:1c:08:00 SRC=externalhost DST=192.168.101.254 LEN=48 TOS=0x00 PREC=0x00 TTL=121 ID=15912 DF PROTO=TCP SPT=58416 DPT=3101 SEQ=1152755416 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 MARK=0x2
Nov  8 19:39:00 gateway kernel: [ 6806.793528] IN=eth0 OUT= MAC=00:0c:29:c8:da:d3:00:0c:29:52:be:9a:08:00 SRC=192.168.100.2 DST=externalhost LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=5872 DF PROTO=TCP SPT=3101 DPT=58416 SEQ=3982169925 ACK=1152755417 WINDOW=32767 RES=0x00 ACK SYN URGP=0 MARK=0x2
Nov  8 19:39:00 gateway kernel: [ 6806.848233] IN=eth1 OUT= MAC=00:0c:29:c8:da:dd:00:1c:4a:b1:a3:1c:08:00 SRC=externalhost DST=192.168.101.254 LEN=40 TOS=0x00 PREC=0x00 TTL=121 ID=15913 DF PROTO=TCP SPT=58416 DPT=3101 SEQ=1152755417 ACK=3982169926 WINDOW=64860 RES=0x00 ACK URGP=0 MARK=0x3
Nov  8 19:39:00 gateway kernel: [ 6806.848295] IN=eth1 OUT= MAC=00:0c:29:c8:da:dd:00:1c:4a:b1:a3:1c:08:00 SRC=externalhost DST=192.168.101.254 LEN=59 TOS=0x00 PREC=0x00 TTL=121 ID=15914 DF PROTO=TCP SPT=58416 DPT=3101 SEQ=1152755417 ACK=3982169926 WINDOW=64860 RES=0x00 ACK PSH URGP=0 MARK=0x3
Nov  8 19:39:00 gateway kernel: [ 6806.972674] IN=eth0 OUT= MAC=00:0c:29:c8:da:d3:00:0c:29:52:be:9a:08:00 SRC=192.168.100.2 DST=externalhost LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=5885 DF PROTO=TCP SPT=3101 DPT=58416 SEQ=3982169926 ACK=1152755436 WINDOW=32748 RES=0x00 ACK URGP=0 MARK=0x3
Nov  8 19:39:00 gateway kernel: [ 6807.138302] IN=eth1 OUT= MAC=00:0c:29:c8:da:dd:00:1c:4a:b1:a3:1c:08:00 SRC=externalhost DST=192.168.101.254 LEN=59 TOS=0x00 PREC=0x00 TTL=121 ID=15919 DF PROTO=TCP SPT=58416 DPT=3101 SEQ=1152755417 ACK=3982169926 WINDOW=64860 RES=0x00 ACK PSH URGP=0 MARK=0x3
Nov  8 19:39:00 gateway kernel: [ 6807.138484] IN=eth0 OUT= MAC=00:0c:29:c8:da:d3:00:0c:29:52:be:9a:08:00 SRC=192.168.100.2 DST=externalhost LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=5895 DF PROTO=TCP SPT=3101 DPT=58416 SEQ=3982169926 ACK=1152755436 WINDOW=32748 RES=0x00 ACK URGP=0 MARK=0x3
Nov  8 19:39:01 gateway kernel: [ 6807.526790] IN=eth0 OUT= MAC=00:0c:29:c8:da:d3:00:0c:29:52:be:9a:08:00 SRC=192.168.100.2 DST=externalhost LEN=51 TOS=0x00 PREC=0x00 TTL=128 ID=5903 DF PROTO=TCP SPT=3101 DPT=58416 SEQ=3982169926 ACK=1152755436 WINDOW=32748 RES=0x00 ACK PSH URGP=0 MARK=0x3
Nov  8 19:39:01 gateway kernel: [ 6807.737768] IN=eth1 OUT= MAC=00:0c:29:c8:da:dd:00:1c:4a:b1:a3:1c:08:00 SRC=externalhost DST=192.168.101.254 LEN=59 TOS=0x00 PREC=0x00 TTL=121 ID=15931 DF PROTO=TCP SPT=58416 DPT=3101 SEQ=1152755417 ACK=3982169926 WINDOW=64860 RES=0x00 ACK PSH URGP=0 MARK=0x3

... the packet from the externalhost flows in and gets marked as "2" since it stems from mac ..a3:1c (first line)

the reply packet from the internal host (192.168.100.2) comes in with the correct mark, but due to POSTROUTING it will get a mark 3 and thus be delivered out to the wrong gateway

subsequent trial packages from the external host are then marked as 3 due to the CONNMARK save-mark

... and this is the reason why the packets continue to be delivered to the wrong gateway...

after fix:

iptables-save | grep mark
-A PREROUTING -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A PREROUTING -i ppp0 -m mark --mark 0x0/0xff -j MARK --set-xmark 0x1/0xffffffff
-A PREROUTING -m mark --mark 0x0/0xff -m mac --mac-source 00:1C:4A:B1:A3:1C -j MARK --set-xmark 0x2/0xffffffff
-A PREROUTING -m mark --mark 0x0/0xff -m mac --mac-source 00:1C:4A:BB:AA:BA -j MARK --set-xmark 0x3/0xffffffff
-A PREROUTING -m mark --mark 0x0/0xff -j MARK --set-xmark 0x1/0xffffffff
-A PREROUTING -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A OUTPUT -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A OUTPUT -m mark --mark 0x0/0xff -j MARK --set-xmark 0x1/0xffffffff
-A OUTPUT -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A POSTROUTING -o ppp0 -m mark --mark 0x0/0xff -j MARK --set-xmark 0x1/0xffffffff
-A POSTROUTING -o eth1 -m mark --mark 0x0/0xff -j MARK --set-xmark 0x2/0xffffffff
-A POSTROUTING -o eth1 -m mark --mark 0x0/0xff -j MARK --set-xmark 0x3/0xffffffff
-A POSTROUTING -j CONNMARK --save-mark --nfmask 0xff --ctmask 0xffffffff

thx for the fast answer, and if you need any additional information i would be glad to provide it

[cperez@…]

I'm not sure this is a fix, we had a lot of problems because of this and we decided not to support this kind of mixed setup (multiple gw with some in an iface and some in different ones).

I'm going to do some test with your propose, stay tunned :)

Thank you again for your research and support