iptables: Fix incorrect SNAT bypass with endpoint routes and tunneling
iptables: Fix incorrect SNAT bypass with endpoint routes and tunneling

--- Analysis ---

In tunneling mode, our CILIUM_POST_nat chain is currently as follows.

1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE
2. -A CILIUM_POST_nat ! -o cilium_host -m comment --comment "exclude non-cilium_host traffic from masquerade" -j RETURN
3. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquarade" -j ACCEPT
4. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6
5. -A CILIUM_POST_nat -s 127.0.0.1/32 -o cilium_host -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6

The second rule implements an early exit from the chain, as none of the
subsequent rules match on output interfaces other than cilium_host.

Once per-endpoint routes are enabled in addition to tunneling, the chain
changes. The second and fifth rules now match on lxc+ as the output
interface:

1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE
2. -A CILIUM_POST_nat ! -o lxc+ -m comment --comment "exclude non-cilium_host traffic from masquerade" -j RETURN
3. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquarade" -j ACCEPT
4. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6
5. -A CILIUM_POST_nat -s 127.0.0.1/32 -o lxc+ -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6

Commit c496e25 ("eni: Support masquerading") implemented that change,
based on the fact that with per-endpoint routes, packets are routed
directly to lxc devices without going through cilium_host.

Nevertheless, the fourth rule still matches on cilium_host and therefore
becomes noop. At the time c496e25 was implemented, this change was
correct because the fourth rule is only present when tunneling is
enabled and per-endpoint routes were not compatible with tunneling.
Commit 3179a47 ("datapath: Support enable-endpoint-routes with
encapsulation") however made those options compatible and the above
chain possible.

--- Fix ---

Ideally, we would update the second rule when running with tunneling and
per-endpoint routes, to be '! -o lxc+ ! -o cilium_host'. Iptables
however doesn't support multiple output interface matchers. This commit
implements a different fix and drops the second rule. Since subsequent
SNATing rules already match on an output interface, the second rule is
unnecessary. With tunneling and per-endpoint routes, the table now looks
like:

1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE
2. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquerade" -j ACCEPT
3. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6
4. -A CILIUM_POST_nat -s 127.0.0.1/32 -o lxc+ -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6

--- Bug Impact ---

This lack of masquerading can cause issues for example when trying to
connect to a VIP with a remote backend from the hostns in our test VMs:

1. DNS request is made to VIP 10.96.0.10.
2. 10.0.2.15, the IP of enp0s3 (default route) is assigned as source IP.
3. kube-proxy translates VIP to backend IP on different node, e.g.
   10.0.0.87.
3. Packet is sent to cilium_host as per the ip routes.
4. The packet is not masqueraded because it matches rule 2 in the bogus
   iptables chain (i.e., cilium_host != lxc+).
5. The packet arrives as 10.0.2.15 -> 10.0.0.87 on the second node.
6. Second node tries to answer to 10.0.2.15 unsuccessfully (all nodes
   have the same IP 10.0.2.15 for enp0s3; that IP isn't routable across
   nodes).

This bug is described in https://github.com/cilium/cilium/issues/13774.

Fixes: https://github.com/cilium/cilium/issues/13774
Co-authored-by: Gilberto Bertin <e45dca12016e9fd4d68d2c7903502c5e1a1f4f1b@isovalent.com>
Signed-off-by: Paul Chaignon <a027184a55211cd23e3f3094f1fdc728df5e0500@cilium.io>

tgraf/cilium,michi-covalent/cilium,tgraf/cilium,tgraf/cilium,tklauser/cilium,tklauser/cilium,michi-covalent/cilium,tklauser/cilium,tgraf/cilium,tklauser/cilium,cilium/cilium,tgraf/cilium,cilium/cilium,tklauser/cilium,cilium/cilium,michi-covalent/cilium,tgraf/cilium,cilium/cilium,michi-covalent/cilium,michi-covalent/cilium,cilium/cilium