ipsec: Fix L7 with endpoint routes
ipsec: Fix L7 with endpoint routes

With the previous patch, when IPsec and endpoint routes are enabled,
packets flow directly from bpf_network to bpf_lxc via the Linux stack,
instead of going through bpf_host. However, we noticed that, when L7
policies are applied, connectivity fails between the proxy and the
destination:

    43.808: cilium-test/client2-6dd75b74c6-9nxhx:45704 -> cilium-test/echo-other-node-697d5d69b7-lglxf:8080 from-endpoint FORWARDED (TCP Flags: SYN)
    43.808: cilium-test/client2-6dd75b74c6-9nxhx:45704 -> cilium-test/echo-other-node-697d5d69b7-lglxf:8080 UNKNOWN 5 (TCP Flags: SYN)
    43.808: cilium-test/client2-6dd75b74c6-9nxhx:45704 -> cilium-test/echo-other-node-697d5d69b7-lglxf:8080 to-proxy FORWARDED (TCP Flags: SYN)
    43.808: cilium-test/echo-other-node-697d5d69b7-lglxf:8080 -> cilium-test/client2-6dd75b74c6-9nxhx:45704 from-proxy FORWARDED (TCP Flags: SYN, ACK)
    43.808: cilium-test/echo-other-node-697d5d69b7-lglxf:8080 -> cilium-test/client2-6dd75b74c6-9nxhx:45704 to-endpoint FORWARDED (TCP Flags: SYN, ACK)
    43.808: cilium-test/client2-6dd75b74c6-9nxhx:45704 -> cilium-test/echo-other-node-697d5d69b7-lglxf:8080 from-endpoint FORWARDED (TCP Flags: ACK)
    43.808: cilium-test/client2-6dd75b74c6-9nxhx:45704 -> cilium-test/echo-other-node-697d5d69b7-lglxf:8080 to-proxy FORWARDED (TCP Flags: ACK)
    43.810: cilium-test/client2-6dd75b74c6-9nxhx:45704 -> cilium-test/echo-other-node-697d5d69b7-lglxf:8080 from-endpoint FORWARDED (TCP Flags: ACK, PSH)
    43.810: cilium-test/client2-6dd75b74c6-9nxhx:45704 -> cilium-test/echo-other-node-697d5d69b7-lglxf:8080 to-proxy FORWARDED (TCP Flags: ACK, PSH)
    43.810: cilium-test/echo-other-node-697d5d69b7-lglxf:8080 -> cilium-test/client2-6dd75b74c6-9nxhx:45704 from-proxy FORWARDED (TCP Flags: ACK)
    43.810: cilium-test/echo-other-node-697d5d69b7-lglxf:8080 -> cilium-test/client2-6dd75b74c6-9nxhx:45704 to-endpoint FORWARDED (TCP Flags: ACK)
    43.810: cilium-test/client2-6dd75b74c6-9nxhx:45704 -> cilium-test/echo-other-node-697d5d69b7-lglxf:8080 http-request FORWARDED (HTTP/1.1 GET http://10.240.0.55:8080/)
    43.812: cilium-test/client2-6dd75b74c6-9nxhx:45704 -> cilium-test/echo-other-node-697d5d69b7-lglxf:8080 from-network FORWARDED (TCP Flags: SYN)
    43.812: cilium-test/client2-6dd75b74c6-9nxhx:45704 -> cilium-test/echo-other-node-697d5d69b7-lglxf:8080 from-stack FORWARDED (TCP Flags: SYN)
    43.812: cilium-test/client2-6dd75b74c6-9nxhx:45704 -> cilium-test/echo-other-node-697d5d69b7-lglxf:8080 to-endpoint FORWARDED (TCP Flags: SYN)
    43.812: cilium-test/echo-other-node-697d5d69b7-lglxf:8080 -> cilium-test/client2-6dd75b74c6-9nxhx:45704 from-endpoint FORWARDED (TCP Flags: SYN, ACK)
    43.812: cilium-test/echo-other-node-697d5d69b7-lglxf:8080 -> cilium-test/client2-6dd75b74c6-9nxhx:45704 to-stack FORWARDED (TCP Flags: SYN, ACK)
    43.813: cilium-test/echo-other-node-697d5d69b7-lglxf:8080 -> cilium-test/client2-6dd75b74c6-9nxhx:45704 from-network FORWARDED (TCP Flags: SYN, ACK)
    44.827: cilium-test/client2-6dd75b74c6-9nxhx:45704 -> cilium-test/echo-other-node-697d5d69b7-lglxf:8080 from-network FORWARDED (TCP Flags: SYN)
    44.827: cilium-test/client2-6dd75b74c6-9nxhx:45704 -> cilium-test/echo-other-node-697d5d69b7-lglxf:8080 from-stack FORWARDED (TCP Flags: SYN)
    44.827: cilium-test/client2-6dd75b74c6-9nxhx:45704 -> cilium-test/echo-other-node-697d5d69b7-lglxf:8080 to-endpoint FORWARDED (TCP Flags: SYN)

We can see the TCP handshake between the client and proxy, followed by an
attempt to perform the TCP handshake between the proxy and server. That
second part fails, as the SYN+ACK packets sent by the server never seem to
reach the proxy; they are dropped somewhere after the from-network
observation point.

At the same time, we can see the IPsec error counter XfrmInNoPols
increasing on the client node. This indicates that SYN+ACK packets were
dropped after decryption, because the XFRM state used for decryption
doesn't match any XFRM policy. The XFRM state used for decryption is:

    src 0.0.0.0 dst 10.240.0.18
    proto esp spi 0x00000003 reqid 1 mode tunnel
    replay-window 0
    mark 0xd00/0xf00 output-mark 0xd00/0xf00
    aead rfc4106(gcm(aes)) 0x3ec3e84ac118c3baf335392e7a4ea24ee3aecb2b 128
    anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
    sel src 0.0.0.0/0 dst 0.0.0.0/0

And it should match the following XFRM policy:

    src 0.0.0.0/0 dst 10.240.0.0/16
    dir in priority 0
    mark 0xd00/0xf00
    tmpl src 0.0.0.0 dst 10.240.0.18
        proto esp reqid 1 mode tunnel

After the packet is decrypted however, we hit the following rule in
iptables because we're going to the proxy.

    -A CILIUM_PRE_mangle -m socket --transparent -m comment --comment "cilium: any->pod redirect proxied traffic to host proxy" -j MARK --set-xmark 0x200/0xffffffff

As a result, the packet mark is set to 0x200 and we don't match the 0xd00
packet mark of the XFRM policy anymore. The packet is therefore dropped
with XfrmInNoPols.

To avoid this, we can simply mark the XFRM policy optional when endpoint
routes are enabled, in the same way we do for tunneling.

Fixes: 287f49c2 ("cilium: encryption, fix redirect when endpoint routes enabled")
Signed-off-by: Paul Chaignon <a027184a55211cd23e3f3094f1fdc728df5e0500@cilium.io>

cilium/cilium,michi-covalent/cilium,michi-covalent/cilium,michi-covalent/cilium,cilium/cilium,michi-covalent/cilium,cilium/cilium,cilium/cilium,cilium/cilium,michi-covalent/cilium