CI:  L3-Dependent L7 Egress fail

https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/2303/
Stacktrace

/home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:312
"app3" expects curl public URL on httpd2 ([f00d::a0f:0:0:29bf]) to succeed
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/Policies.go:520

Standard Output

�[1mSTEP�[0m: Setting PolicyEnforcement=default
time="2018-05-02T15:19:50Z" level=info msg="setting PolicyEnforcement=default"
time="2018-05-02T15:19:50Z" level=debug msg="running command: sudo cilium config -o json | jq -r '.status.realized[\"policy-enforcement\"]'"
time="2018-05-02T15:19:50Z" level=debug msg="running command: sudo cilium config PolicyEnforcement=default"
time="2018-05-02T15:19:55Z" level=debug msg="running command: cilium endpoint list -o jsonpath='{range [*]}{<EMAIL_ADDRESS>time="2018-05-02T15:19:56Z" level=info msg="'7' containers are in a 'ready' state of a total of '7' containers." functionName=WaitEndpointsReady status="map[ready:7]" test=RunPolicies
�[1mSTEP�[0m: Setting up policy: /home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json
time="2018-05-02T15:19:56Z" level=debug msg="running command: sudo cilium policy get -o json | jq '.revision'"
time="2018-05-02T15:19:56Z" level=info msg="before importing policy" file-path=/home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json policyRevision=39 test=RunPolicies
time="2018-05-02T15:19:56Z" level=info msg="validating policy before importing" file-path=/home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json test=RunPolicies
time="2018-05-02T15:19:56Z" level=debug msg="running command: sudo cilium policy validate /home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json"
time="2018-05-02T15:19:57Z" level=debug msg="running command: sudo cilium policy import /home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json"
time="2018-05-02T15:20:02Z" level=debug msg="running command: sudo cilium policy get -o json | jq '.revision'"
time="2018-05-02T15:20:03Z" level=debug msg="running command: sudo cilium policy wait 40"
time="2018-05-02T15:20:03Z" level=debug msg="running command: sudo cilium policy get -o json | jq '.revision'"
time="2018-05-02T15:20:03Z" level=info msg="policy import finished and revision increased" file-path=/home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json policyRevision=40 test=RunPolicies
time="2018-05-02T15:20:03Z" level=debug msg="running command: sudo cilium endpoint list -o jsonpath='{range [?(@.status.labels.security-relevant[0]!=\"reserved:health\")]}{<EMAIL_ADDRESS>time="2018-05-02T15:20:03Z" level=debug msg="running command: sudo docker inspect app3"
time="2018-05-02T15:20:04Z" level=debug msg="running command: sudo docker inspect httpd1"
�[1mSTEP�[0m: Client "app3" attempting to curl public URL on httpd1
time="2018-05-02T15:20:04Z" level=debug msg="running command: docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://<IP_ADDRESS>:80/public"
�[1mSTEP�[0m: Client "app3" attempting to curl public URL on httpd1
time="2018-05-02T15:20:04Z" level=debug msg="running command: docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://[f00d::a0f:0:0:10da]:80/public"
time="2018-05-02T15:20:04Z" level=debug msg="running command: sudo cilium endpoint get 3978 -o json"
time="2018-05-02T15:20:04Z" level=debug msg="running command: sudo docker inspect app3"
time="2018-05-02T15:20:04Z" level=debug msg="running command: sudo docker inspect httpd1"
�[1mSTEP�[0m: Client "app3" attempting to curl private URL on httpd1
time="2018-05-02T15:20:05Z" level=debug msg="running command: docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://<IP_ADDRESS>:80/private"
�[1mSTEP�[0m: Client "app3" attempting to curl private URL on httpd1
time="2018-05-02T15:20:05Z" level=debug msg="running command: docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://[f00d::a0f:0:0:10da]:80/private"
time="2018-05-02T15:20:05Z" level=debug msg="running command: sudo cilium endpoint get 3978 -o json"
time="2018-05-02T15:20:05Z" level=debug msg="running command: sudo docker inspect app3"
time="2018-05-02T15:20:05Z" level=debug msg="running command: sudo docker inspect httpd2"
�[1mSTEP�[0m: Client "app3" attempting to curl public URL on httpd2
time="2018-05-02T15:20:06Z" level=debug msg="running command: docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://<IP_ADDRESS>:80/public"
�[1mSTEP�[0m: Client "app3" attempting to curl public URL on httpd2
time="2018-05-02T15:20:06Z" level=debug msg="running command: docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://[f00d::a0f:0:0:29bf]:80/public"

Endpoint config looks good:
[
  {
    "id": 3978,
    "spec": {
      "label-configuration": {
        "user": []
      },
      "options": {
        "Conntrack": "Enabled",
        "ConntrackAccounting": "Enabled",
        "ConntrackLocal": "Disabled",
        "Debug": "Enabled",
        "DebugLB": "Disabled",
        "DropNotification": "Enabled",
        "EgressPolicy": "Enabled",
        "IngressPolicy": "Disabled",
        "NAT46": "Disabled",
        "TraceNotification": "Enabled"
      }
    },
    "status": {
      "controllers": [
        {
          "configuration": {
            "error-retry": true,
            "interval": "5m0s"
          },
          "name": "resolve-identity-3978",
          "status": {
            "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
            "last-success-timestamp": "2018-05-02T15:15:53.000Z",
            "success-count": 2
          },
          "uuid": "016e8ec4-4e1b-11e8-ac1a-0800277280d1"
        },
        {
          "configuration": {
            "error-retry": true,
            "interval": "5m0s"
          },
          "name": "sync-IPv4-identity-mapping (3978)",
          "status": {
            "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
            "last-success-timestamp": "2018-05-02T15:15:53.024Z",
            "success-count": 2
          },
          "uuid": "016fb513-4e1b-11e8-ac1a-0800277280d1"
        },
        {
          "configuration": {
            "error-retry": true,
            "interval": "5m0s"
          },
          "name": "sync-IPv6-identity-mapping (3978)",
          "status": {
            "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
            "last-success-timestamp": "2018-05-02T15:15:53.024Z",
            "success-count": 2
          },
          "uuid": "016fb5c4-4e1b-11e8-ac1a-0800277280d1"
        },
        {
          "configuration": {
            "error-retry": true,
            "interval": "1m0s"
          },
          "name": "sync-identity-to-k8s-pod (3978)",
          "status": {
            "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
            "last-success-timestamp": "2018-05-02T15:19:53.021Z",
            "success-count": 10
          },
          "uuid": "016fb3f5-4e1b-11e8-ac1a-0800277280d1"
        }
      ],
      "external-identifiers": {
        "container-id": "6de96faf9fa1be32456ceb6fb518a2bb41aafb5335edca035b08c31a8d4fdb1e",
        "container-name": "app3",
        "docker-endpoint-id": "3fc18ee406c6feb02bfdd00da8730ec856335b8202208940be967226439e08db",
        "docker-network-id": "0a42519156d0f48299b98978e895c5c52203f0c5b8f70dd6d3e98fb899c593e7",
        "pod-name": ":"
      },
      "health": {
        "bpf": "OK",
        "connected": true,
        "overallHealth": "OK",
        "policy": "OK"
      },
      "identity": {
        "id": 51492,
        "labels": [
          "container:id.app3"
        ],
        "labelsSHA256": "2186c66cd5d8fb631adf86d9ca4ce027fcb02f3de2e5de8868f83250f203240e"
      },
      "labels": {
        "derived": [],
        "disabled": [],
        "realized": {
          "user": []
        },
        "security-relevant": [
          "container:id.app3"
        ]
      },
      "log": [
        {
          "code": "OK",
          "message": "Successfully regenerated endpoint program due to endpoint policy updated \u0026 changes were needed",
          "state": "ready",
          "timestamp": "2018-05-02T15:20:01Z"
        }
      ],
      "networking": {
        "addressing": [
          {
            "ipv4": "<IP_ADDRESS>",
            "ipv6": "f00d::a0f:0:0:f8a"
          }
        ],
        "host-mac": "02:41:b2:63:f2:8b",
        "interface-index": 169,
        "interface-name": "lxc3fc18",
        "mac": "9e:b7:7c:7f:8a:16"
      },
      "policy": {
        "proxy-statistics": [
          {
            "allocated-proxy-port": 12908,
            "location": "egress",
            "port": 80,
            "protocol": "http",
            "statistics": {
              "requests": {
                "denied": 2,
                "forwarded": 2,
                "received": 4
              },
              "responses": {
                "forwarded": 2,
                "received": 2
              }
            }
          }
        ],
        "realized": {
          "allowed-egress-identities": [
            5572
          ],
          "allowed-ingress-identities": [
            1
          ],
          "build": 40,
          "cidr-policy": {
            "egress": [],
            "ingress": []
          },
          "id": 51492,
          "l4": {
            "egress": [
              {
                "derived-from-rules": [
                  [],
                  []
                ],
                "rule": "{\n  \"port\": 80,\n  \"protocol\": \"TCP\",\n  \"l7-rules\": [\n    {\n      \"any.id.httpd1=\": {\n        \"http\": [\n          {\n            \"path\": \"/public\",\n            \"method\": \"GET\"\n          }\n        ]\n      }\n    },\n    {\n      \"any.id.httpd2=\": {\n        \"http\": [\n          {}\n        ]\n      }\n    }\n  ]\n}"
              }
            ],
            "ingress": []
          },
          "policy-enabled": "egress",
          "policy-revision": 40
        },
        "spec": {
          "allowed-egress-identities": [
            5572
          ],
          "allowed-ingress-identities": [
            1
          ],
          "build": 40,
          "cidr-policy": {
            "egress": [],
            "ingress": []
          },
          "id": 51492,
          "l4": {
            "egress": [
              {
                "derived-from-rules": [
                  [],
                  []
                ],
                "rule": "{\n  \"port\": 80,\n  \"protocol\": \"TCP\",\n  \"l7-rules\": [\n    {\n      \"any.id.httpd1=\": {\n        \"http\": [\n          {\n            \"path\": \"/public\",\n            \"method\": \"GET\"\n          }\n        ]\n      }\n    },\n    {\n      \"any.id.httpd2=\": {\n        \"http\": [\n          {}\n        ]\n      }\n    }\n  ]\n}"
              }
            ],
            "ingress": []
          },
          "policy-enabled": "egress",
          "policy-revision": 40
        }
      },
      "realized": {
        "label-configuration": {
          "user": []
        },
        "options": {
          "Conntrack": "Enabled",
          "ConntrackAccounting": "Enabled",
          "ConntrackLocal": "Disabled",
          "Debug": "Enabled",
          "DebugLB": "Disabled",
          "DropNotification": "Enabled",
          "EgressPolicy": "Enabled",
          "IngressPolicy": "Disabled",
          "NAT46": "Disabled",
          "TraceNotification": "Enabled"
        }
      },
      "state": "ready"
    }
  }
]

Connection using IPv4 works, it didn't work using Ipv6:
time="2018-05-02T15:20:06Z" level=debug msg="running command: docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://<IP_ADDRESS>:80/public"
cmd: "docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://<IP_ADDRESS>:80/public" exitCode: 0
 { 'val': 'this is public' }

time="2018-05-02T15:20:06Z" level=debug msg="running command: docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://[f00d::a0f:0:0:29bf]:80/public"
cmd: "docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://[f00d::a0f:0:0:29bf]:80/public" exitCode: 22

test_results_Ginkgo-CI-Tests-Pipeline_2303.zip

Other build with same problem in different place:
Policy is loaded, wait until all endpoints but has not been applied:
https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/2312/testReport/junit/(root)/runtime/RuntimeValidatedPolicies_L3_Dependent_L7_Egress/
Stacktrace

/home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:312
"app3" expects curl public URL on httpd1 (<IP_ADDRESS>) to succeed
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/Policies.go:499

time="2018-05-03T08:59:32Z" level=info msg="'7' containers are in a 'ready' state of a total of '7' containers." functionName=WaitEndpointsReady status="map[ready:7]" test=RunPolicies
time="2018-05-03T08:59:32Z" level=debug msg="running command: sudo cilium policy get -o json | jq '.revision'"
cmd: "sudo cilium policy get -o json | jq '.revision'" exitCode: 0
 31

time="2018-05-03T08:59:32Z" level=info msg="before importing policy" file-path=/home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json policyRevision=31 test=RunPolicies
time="2018-05-03T08:59:32Z" level=info msg="validating policy before importing" file-path=/home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json test=RunPolicies
time="2018-05-03T08:59:32Z" level=debug msg="running command: sudo cilium policy validate /home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json"
cmd: "sudo cilium policy validate /home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json" exitCode: 0
 All policy elements are valid.

time="2018-05-03T08:59:33Z" level=debug msg="running command: sudo cilium policy import /home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json"
cmd: "sudo cilium policy import /home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json" exitCode: 0
 Revision: 32

time="2018-05-03T08:59:38Z" level=debug msg="running command: sudo cilium policy get -o json | jq '.revision'"
cmd: "sudo cilium policy get -o json | jq '.revision'" exitCode: 0
 32

time="2018-05-03T08:59:38Z" level=debug msg="running command: sudo cilium policy wait 32"
cmd: "sudo cilium policy wait 32" exitCode: 0

time="2018-05-03T08:59:38Z" level=debug msg="running command: sudo cilium policy get -o json | jq '.revision'"
cmd: "sudo cilium policy get -o json | jq '.revision'" exitCode: 0
 32

time="2018-05-03T08:59:38Z" level=info msg="policy import finished and revision increased" file-path=/home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json policyRevision=32 test=RunPolicies
time="2018-05-03T08:59:38Z" level=debug msg="running command: sudo cilium endpoint list -o jsonpath='{range [?(@.status.labels.security-relevant[0]!=\"reserved:health\")]}{<EMAIL_ADDRESS>cmd: "sudo cilium endpoint list -o jsonpath='{range [?(@.status.labels.security-relevant[0]!=\"reserved:health\")]}{<EMAIL_ADDRESS>exitCode: 0
 app2=3978
app3=4314
httpd1=10687
httpd3=15124
app1=25729
httpd2=60670

time="2018-05-03T08:59:38Z" level=debug msg="running command: sudo docker inspect app3"
cmd: "sudo docker inspect app3" exitCode: 0
 [
    {
        "Id": "60b6f0a4af0d1eb8df5af12d1caf66763ea21178f9540aa142da15159ca2bbf4",
        "Created": "2018-05-03T08:51:10.431379734Z",
        "Path": "/usr/bin/netserver",
        "Args": [
            "-D"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 18965,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2018-05-03T08:51:12.662913704Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:a8708b9070de1002ece7c05ae3f11712a93f8275612a2f16f64351a60bba5117",
        "ResolvConfPath": "/var/lib/docker/containers/60b6f0a4af0d1eb8df5af12d1caf66763ea21178f9540aa142da15159ca2bbf4/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/60b6f0a4af0d1eb8df5af12d1caf66763ea21178f9540aa142da15159ca2bbf4/hostname",
        "HostsPath": "/var/lib/docker/containers/60b6f0a4af0d1eb8df5af12d1caf66763ea21178f9540aa142da15159ca2bbf4/hosts",
        "LogPath": "/var/lib/docker/containers/60b6f0a4af0d1eb8df5af12d1caf66763ea21178f9540aa142da15159ca2bbf4/60b6f0a4af0d1eb8df5af12d1caf66763ea21178f9540aa142da15159ca2bbf4-json.log",
        "Name": "/app3",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "cilium-net",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/c1ed097dcac2611af278f1b0b087ab265e270e0a5b402ea3d134f8625dab808e-init/diff:/var/lib/docker/overlay2/21f87a73d9f96fbb6b919493e67810d11b0f0a8faca50d0da15bc6d41308c04e/diff:/var/lib/docker/overlay2/740f018cdd07ee5a28e9ac342f40a95d47cf97b10b31db07bee978bdb9d7c4b4/diff:/var/lib/docker/overlay2/2119e185e72eab51f5fe788664dc26a07dcb3479f1c66d0239c79ea8b949f47c/diff",
                "MergedDir": "/var/lib/docker/overlay2/c1ed097dcac2611af278f1b0b087ab265e270e0a5b402ea3d134f8625dab808e/merged",
                "UpperDir": "/var/lib/docker/overlay2/c1ed097dcac2611af278f1b0b087ab265e270e0a5b402ea3d134f8625dab808e/diff",
                "WorkDir": "/var/lib/docker/overlay2/c1ed097dcac2611af278f1b0b087ab265e270e0a5b402ea3d134f8625dab808e/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "60b6f0a4af0d",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/usr/bin/netserver",
                "-D"
            ],
            "ArgsEscaped": true,
            "Image": "tgraf/netperf",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "id.app3": ""
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "157b9f8bc5185ffb2a98ff59ffc1ae1e023a8eafe86b4cb531d242ff4c7d5992",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/157b9f8bc518",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "cilium-net": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "60b6f0a4af0d"
                    ],
                    "NetworkID": "480670d37b4056405ef8703ffc073dc8dda113279b5f14360aad1292f06b6dde",
                    "EndpointID": "34524f29bf780d3383c3a8fefd2c6672afe65602e5e0fcbeb5a780e26a4beaf0",
                    "Gateway": "",
                    "IPAddress": "<IP_ADDRESS>",
                    "IPPrefixLen": 32,
                    "IPv6Gateway": "f00d::a0f:0:0:8ad6",
                    "GlobalIPv6Address": "f00d::a0f:0:0:10da",
                    "GlobalIPv6PrefixLen": 128,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

time="2018-05-03T08:59:38Z" level=debug msg="running command: sudo docker inspect httpd1"
cmd: "sudo docker inspect httpd1" exitCode: 0
 [
    {
        "Id": "3cfe5c46743099b9e85abc991e668a38c2961ccedb7b423cd45cc76ba5552ef6",
        "Created": "2018-05-03T08:51:12.803129454Z",
        "Path": "httpd-foreground",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 19142,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2018-05-03T08:51:14.67764505Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:f95bd435901653695d908df0f820b7c4fd6ea9d8145f07a63a90d3678588bd0e",
        "ResolvConfPath": "/var/lib/docker/containers/3cfe5c46743099b9e85abc991e668a38c2961ccedb7b423cd45cc76ba5552ef6/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/3cfe5c46743099b9e85abc991e668a38c2961ccedb7b423cd45cc76ba5552ef6/hostname",
        "HostsPath": "/var/lib/docker/containers/3cfe5c46743099b9e85abc991e668a38c2961ccedb7b423cd45cc76ba5552ef6/hosts",
        "LogPath": "/var/lib/docker/containers/3cfe5c46743099b9e85abc991e668a38c2961ccedb7b423cd45cc76ba5552ef6/3cfe5c46743099b9e85abc991e668a38c2961ccedb7b423cd45cc76ba5552ef6-json.log",
        "Name": "/httpd1",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "cilium-net",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/1cc863808541ef5934285d8674eae43ff6fd7992ed99d96b97bd72a347e0fbcf-init/diff:/var/lib/docker/overlay2/c90807ee25e49462398fe9025ec0ab477796437d74d85f885f3064ee7a52a5b7/diff:/var/lib/docker/overlay2/953c7005d1c2fc15a4bfcfbad80a743a016550d202a1cf502d32db73b90a7539/diff:/var/lib/docker/overlay2/aec46362ba8b0b53401ee585d6c6da0b6e3ee4888809660662ee10222e2cf892/diff:/var/lib/docker/overlay2/776b34687ab74cbcf9ced0f1d97823799d435127589263f3dffeec51b448149c/diff:/var/lib/docker/overlay2/26e91333477da497a6b9feed41f857ef096f742bf5f8d485d183c18ec1e18891/diff:/var/lib/docker/overlay2/25cbaa052c8b4931c0d779f93aed035337bf4ab62bb5bd12747cf6189e63729d/diff:/var/lib/docker/overlay2/a41d2ab3f6c398cde11467fdc88bcbaecb21eb40cd8fa2419b3dd196596c9285/diff:/var/lib/docker/overlay2/4f48ad1a28c0fd89d7899023e6547e8ec30fab3b796d616f959923a1f866f5cf/diff",
                "MergedDir": "/var/lib/docker/overlay2/1cc863808541ef5934285d8674eae43ff6fd7992ed99d96b97bd72a347e0fbcf/merged",
                "UpperDir": "/var/lib/docker/overlay2/1cc863808541ef5934285d8674eae43ff6fd7992ed99d96b97bd72a347e0fbcf/diff",
                "WorkDir": "/var/lib/docker/overlay2/1cc863808541ef5934285d8674eae43ff6fd7992ed99d96b97bd72a347e0fbcf/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "3cfe5c467430",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "HTTPD_PREFIX=/usr/local/apache2",
                "NGHTTP2_VERSION=1.18.1-1",
                "OPENSSL_VERSION=1.0.2k-1~bpo8+1",
                "HTTPD_VERSION=2.4.25",
                "HTTPD_SHA1=bd6d138c31c109297da2346c6e7b93b9283993d2",
                "HTTPD_BZ2_URL=https://www.apache.org/dyn/closer.cgi?action=download&filename=httpd/httpd-2.4.25.tar.bz2",
                "HTTPD_ASC_URL=https://www.apache.org/dist/httpd/httpd-2.4.25.tar.bz2.asc"
            ],
            "Cmd": [
                "httpd-foreground"
            ],
            "Image": "cilium/demo-httpd",
            "Volumes": null,
            "WorkingDir": "/usr/local/apache2",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "id.httpd1": "",
                "id.service1": ""
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "e3b9a3c00169a8c7946f947d4374b8a914eddbb3f7315a4ff3b157bcc80fcb51",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/e3b9a3c00169",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "cilium-net": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "3cfe5c467430"
                    ],
                    "NetworkID": "480670d37b4056405ef8703ffc073dc8dda113279b5f14360aad1292f06b6dde",
                    "EndpointID": "3c4773a3bdbfb42f0b4edb40987a41cc346cf5c090efef75436aea29adff1260",
                    "Gateway": "",
                    "IPAddress": "<IP_ADDRESS>",
                    "IPPrefixLen": 32,
                    "IPv6Gateway": "f00d::a0f:0:0:8ad6",
                    "GlobalIPv6Address": "f00d::a0f:0:0:29bf",
                    "GlobalIPv6PrefixLen": 128,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

time="2018-05-03T08:59:39Z" level=debug msg="running command: docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://<IP_ADDRESS>:80/public"
cmd: "docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://<IP_ADDRESS>:80/public" exitCode: 22

The monitor log:
[2018-05-03 08:59:38] CPU 01: MARK 0x14fb6441 FROM 0 DEBUG: 197 bytes, Packet to proxy port 0 (Post)
[2018-05-03 08:59:38] Ethernet	{Contents=[..14..] Payload=[..118..] SrcMAC=8e:c9:b7:e7:e9:eb DstMAC=8e:c9:b7:e7:e9:eb EthernetType=IPv4 Length=0}
[2018-05-03 08:59:38] IPv4	{Contents=[..20..] Payload=[..98..] Version=4 IHL=5 TOS=0 Length=183 Id=47302 Flags=DF FragOffset=0 TTL=64 Protocol=TCP Checksum=53831 SrcIP=<IP_ADDRESS> DstIP=<IP_ADDRESS> Options=[] Padding=[]}
[2018-05-03 08:59:38] TCP	{Contents=[..32..] Payload=[..66..]  The original description state that the problem is that IPv6 connectivity is not working when IPv4 connectivity is working. How do you know that these two are the same problem?

My fault here :-(

Looking at this as I saw a similar failure in this build: https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated/3057/
Logs:
test_results_Cilium-PR-Ginkgo-Tests-Validated_3057.zip
Failure output: /home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:312 "app3" expects curl public URL on httpd1 ([f00d::a0f:0:0:29bf]) to succeed Expected     <bool>: false to be true /home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/src/github.com/cilium/cilium/test/runtime/Policies.go:499
Jenkins console output:
time="2018-05-09T18:28:08Z" level=info msg="setting PolicyEnforcement=default"
time="2018-05-09T18:28:08Z" level=debug msg="running command: sudo cilium config -o json | jq -r '.status.realized[\"policy-enforcement\"]'"
time="2018-05-09T18:28:08Z" level=debug msg="running command: sudo cilium config PolicyEnforcement=default"
time="2018-05-09T18:28:13Z" level=debug msg="running command: cilium endpoint list -o jsonpath='{range [*]}{<EMAIL_ADDRESS>time="2018-05-09T18:28:13Z" level=info msg="'7' containers are in a 'ready' state of a total of '7' containers." functionName=WaitEndpointsReady status="map[ready:7]" test=RunPolicies
STEP: Setting up policy: /home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json
time="2018-05-09T18:28:13Z" level=debug msg="running command: sudo cilium policy get -o json | jq '.revision'"
time="2018-05-09T18:28:13Z" level=info msg="before importing policy" file-path=/home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json policyRevision=31 test=RunPolicies
time="2018-05-09T18:28:13Z" level=info msg="validating policy before importing" file-path=/home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json test=RunPolicies
time="2018-05-09T18:28:13Z" level=debug msg="running command: sudo cilium policy validate /home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json"
time="2018-05-09T18:28:14Z" level=debug msg="running command: sudo cilium policy import /home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json"
time="2018-05-09T18:28:19Z" level=debug msg="running command: sudo cilium policy get -o json | jq '.revision'"
time="2018-05-09T18:28:19Z" level=debug msg="running command: sudo cilium policy wait 32"
time="2018-05-09T18:28:19Z" level=debug msg="running command: sudo cilium policy get -o json | jq '.revision'"
time="2018-05-09T18:28:19Z" level=info msg="policy import finished and revision increased" file-path=/home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json policyRevision=32 test=RunPolicies
time="2018-05-09T18:28:19Z" level=debug msg="running command: sudo cilium endpoint list -o jsonpath='{range [?(@.status.labels.security-relevant[0]!=\"reserved:health\")]}{<EMAIL_ADDRESS>time="2018-05-09T18:28:19Z" level=debug msg="running command: sudo docker inspect app3"
time="2018-05-09T18:28:19Z" level=debug msg="running command: sudo docker inspect httpd1"
STEP: Client "app3" attempting to curl public URL on httpd1
time="2018-05-09T18:28:20Z" level=debug msg="running command: docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://<IP_ADDRESS>:80/public"
STEP: Client "app3" attempting to curl public URL on httpd1
time="2018-05-09T18:28:20Z" level=debug msg="running command: docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://[f00d::a0f:0:0:29bf]:80/public"
time="2018-05-09T18:28:20Z" level=debug msg="running command: sudo journalctl -au cilium --since '12.064338201 seconds ago'"

The test is testing the following:
"app3" expects curl public URL on httpd2 ([f00d::a0f:0:0:29bf]) to succeed

First, check that the IP to which egress is occurring (f00d::a0f:0:0:29bf) is in the BPF ipcache output, which is stored in the cilium-bugtool archive in the attached logs in the initial comment under cilium-bpf-ipcache-list.md.
cilium bpf ipcache list
IP PREFIX/ADDRESS    IDENTITY
fc00::10ca:1         1
f00d::a0f:0:0:3b14   34969
<IP_ADDRESS>         1
<IP_ADDRESS>            1
f00d::a0f:0:0:8ad6   1
<IP_ADDRESS>         9750
f00d::a0f:0:0:ecfe   56980
f00d::a0f:0:0:29bf   5572
f00d::a0f:0:0:10da   54590
<IP_ADDRESS>        5572
f00d::a0f:0:0:74ca   61249
f00d::a0f:0:0:6481   9750
<IP_ADDRESS>          54590
<IP_ADDRESS>         61249
f00d::a0f:0:0:f8a    51492
<IP_ADDRESS>        56980
<IP_ADDRESS>        34969
<IP_ADDRESS>         51492

The IP is indeed there, and corresponds to an identity 5572.
From cilium-debuginfo.md found in the aforementioned bugtool archive, this identity indeed corresponds to the httpd2 endpoint:
Endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4            STATUS
           ENFORCEMENT        ENFORCEMENT
3978       Disabled           Enabled           51492      container:id.app3             f00d::a0f:0:0:f8a    <IP_ADDRESS>    ready
4314       Disabled           Disabled          54590      container:id.httpd1           f00d::a0f:0:0:10da   <IP_ADDRESS>     ready
                                                           container:id.service1
10687      Disabled           Disabled          5572       container:id.httpd2           f00d::a0f:0:0:29bf   <IP_ADDRESS>   ready
                                                           container:id.service1
15124      Disabled           Disabled          34969      container:id.app1             f00d::a0f:0:0:3b14   <IP_ADDRESS>   ready
25729      Disabled           Disabled          9750       container:id.app2             f00d::a0f:0:0:6481   <IP_ADDRESS>    ready
29898      Disabled           Disabled          61249      reserved:health               f00d::a0f:0:0:74ca   <IP_ADDRESS>    ready
60670      Disabled           Disabled          56980      container:id.httpd3           f00d::a0f:0:0:ecfe   <IP_ADDRESS>   ready
                                                           container:id.service1

The endpoint has ID 10687.

app3 has endpoint ID 3978. Let's make sure policy is configured correctly for the endpoint.
BPF Policy Get 3978
DIRECTION   LABELS (source:key[=value])   PORT/PROTO   BYTES   PACKETS
Egress      container:id.httpd1           80/TCP       2200    24
            container:id.service1
Egress      container:id.httpd2           ANY          0       0
            container:id.service1
Ingress     reserved:host                 ANY          0       0
Egress      container:id.httpd2           80/TCP       1101    12
            container:id.service1

Egress is allowed to httpd2 on 80/TCP.
Endpoint health is ok:
Endpoint Health 3978
Overall Health:   OK
BPF Health:       OK
Policy Health:    OK
Connected:        yes

Per the Jenkins console output in the initial comment, app3 queried httpd2 at the following timestamp:
time="2018-05-02T15:20:06Z
The latest entry in the endpoint log shows that the endpoint program was successfully regenerated at 2018-05-02T15:20:01Z, before the query was ran. For listeners and policies, the proxy is synchronously updated (i.e., proxy policy for an endpoint is updated before an endpoint is marked as ready). However, the ipcache updates for envoy are done asynchronously, which means that even though the endpoint is marked as ready, envoy might not have received its IP to identity mapping.
Endpoint Log 3978
Timestamp              Status    State                   Message
2018-05-02T15:20:01Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-05-02T15:20:01Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-05-02T15:19:57Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-05-02T15:19:57Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-05-02T15:19:49Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-05-02T15:19:49Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-05-02T15:19:45Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-05-02T15:19:45Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-05-02T15:19:16Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-05-02T15:19:16Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-05-02T15:19:11Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-05-02T15:19:11Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-05-02T15:18:57Z   OK        ready                   Endpoint policy update skipped because no changes were needed
2018-05-02T15:18:55Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-05-02T15:18:55Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-05-02T15:18:51Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-05-02T15:18:51Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-05-02T15:18:25Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-05-02T15:18:25Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-05-02T15:18:20Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-05-02T15:18:20Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-05-02T15:18:14Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-05-02T15:18:14Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-05-02T15:18:09Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-05-02T15:18:09Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-05-02T15:18:01Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-05-02T15:18:01Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-05-02T15:17:59Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-05-02T15:17:59Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-05-02T15:17:13Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-05-02T15:17:13Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-05-02T15:17:08Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-05-02T15:17:08Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-05-02T15:16:53Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-05-02T15:16:53Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-05-02T15:16:47Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-05-02T15:16:47Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-05-02T15:14:37Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-05-02T15:14:37Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-05-02T15:14:32Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-05-02T15:14:32Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-05-02T15:14:22Z   OK        ready                   Endpoint policy update skipped because no changes were needed
2018-05-02T15:13:57Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-05-02T15:13:57Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-05-02T15:13:54Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-05-02T15:13:49Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-05-02T15:11:25Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-05-02T15:11:25Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-05-02T15:11:20Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-05-02T15:11:20Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-05-02T15:11:03Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-05-02T15:11:03Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-05-02T15:11:00Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-05-02T15:11:00Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-05-02T15:10:55Z   OK        ready                   Successfully regenerated endpoint program due to updated security labels
2018-05-02T15:10:55Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-05-02T15:10:52Z   OK        regenerating            Regenerating Endpoint BPF: updated security labels
2018-05-02T15:10:52Z   OK        waiting-to-regenerate   Triggering regeneration due to new identity
2018-05-02T15:10:52Z   OK        ready                   Set identity for this endpoint
2018-05-02T15:10:51Z   OK        waiting-for-identity    Successfully regenerated endpoint program due to Waiting on endpoint initial program regeneration while handling API PATCH
2018-05-02T15:10:51Z   Warning   waiting-for-identity    Skipped invalid state transition to ready due to: Completed endpoint regeneration with no pending regeneration requests
2018-05-02T15:10:49Z   OK        waiting-for-identity    Update endpoint from API PATCH

Endpoint get output for app3:
Endpoint Get 3978
[
  {
    "id": 3978,
    "spec": {
      "label-configuration": {
        "user": []
      },
      "options": {
        "Conntrack": "Enabled",
        "ConntrackAccounting": "Enabled",
        "ConntrackLocal": "Disabled",
        "Debug": "Enabled",
        "DebugLB": "Disabled",
        "DropNotification": "Enabled",
        "EgressPolicy": "Enabled",
        "IngressPolicy": "Disabled",
        "NAT46": "Disabled",
        "TraceNotification": "Enabled"
      }
    },
    "status": {
      "controllers": [
        {
          "configuration": {
            "error-retry": true,
            "interval": "5m0s"
          },
          "name": "resolve-identity-3978",
          "status": {
            "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
            "last-success-timestamp": "2018-05-02T15:15:53.000Z",
            "success-count": 2
          },
          "uuid": "016e8ec4-4e1b-11e8-ac1a-0800277280d1"
        },
        {
          "configuration": {
            "error-retry": true,
            "interval": "5m0s"
          },
          "name": "sync-IPv4-identity-mapping (3978)",
          "status": {
            "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
            "last-success-timestamp": "2018-05-02T15:15:53.024Z",
            "success-count": 2
          },
          "uuid": "016fb513-4e1b-11e8-ac1a-0800277280d1"
        },
        {
          "configuration": {
            "error-retry": true,
            "interval": "5m0s"
          },
          "name": "sync-IPv6-identity-mapping (3978)",
          "status": {
            "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
            "last-success-timestamp": "2018-05-02T15:15:53.024Z",
            "success-count": 2
          },
          "uuid": "016fb5c4-4e1b-11e8-ac1a-0800277280d1"
        },
        {
          "configuration": {
            "error-retry": true,
            "interval": "1m0s"
          },
          "name": "sync-identity-to-k8s-pod (3978)",
          "status": {
            "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
            "last-success-timestamp": "2018-05-02T15:19:53.021Z",
            "success-count": 10
          },
          "uuid": "016fb3f5-4e1b-11e8-ac1a-0800277280d1"
        }
      ],
      "external-identifiers": {
        "container-id": "6de96faf9fa1be32456ceb6fb518a2bb41aafb5335edca035b08c31a8d4fdb1e",
        "container-name": "app3",
        "docker-endpoint-id": "3fc18ee406c6feb02bfdd00da8730ec856335b8202208940be967226439e08db",
        "docker-network-id": "0a42519156d0f48299b98978e895c5c52203f0c5b8f70dd6d3e98fb899c593e7",
        "pod-name": ":"
      },
      "health": {
        "bpf": "OK",
        "connected": true,
        "overallHealth": "OK",
        "policy": "OK"
      },
      "identity": {
        "id": 51492,
        "labels": [
          "container:id.app3"
        ],
        "labelsSHA256": "2186c66cd5d8fb631adf86d9ca4ce027fcb02f3de2e5de8868f83250f203240e"
      },
      "labels": {
        "derived": [],
        "disabled": [],
        "realized": {
          "user": []
        },
        "security-relevant": [
          "container:id.app3"
        ]
      },
      "log": [
        {
          "code": "OK",
          "message": "Successfully regenerated endpoint program due to endpoint policy updated & changes were needed",
          "state": "ready",
          "timestamp": "2018-05-02T15:20:01Z"
        }
      ],
      "networking": {
        "addressing": [
          {
            "ipv4": "<IP_ADDRESS>",
            "ipv6": "f00d::a0f:0:0:f8a"
          }
        ],
        "host-mac": "02:41:b2:63:f2:8b",
        "interface-index": 169,
        "interface-name": "lxc3fc18",
        "mac": "9e:b7:7c:7f:8a:16"
      },
      "policy": {
        "proxy-statistics": [
          {
            "allocated-proxy-port": 12908,
            "location": "egress",
            "port": 80,
            "protocol": "http",
            "statistics": {
              "requests": {
                "denied": 3,
                "forwarded": 3,
                "received": 6
              },
              "responses": {
                "forwarded": 3,
                "received": 3
              }
            }
          }
        ],
        "realized": {
          "allowed-egress-identities": [
            5572
          ],
          "allowed-ingress-identities": [
            1
          ],
          "build": 40,
          "cidr-policy": {
            "egress": [],
            "ingress": []
          },
          "id": 51492,
          "l4": {
            "egress": [
              {
                "derived-from-rules": [
                  [],
                  []
                ],
                "rule": {
  "port": 80,
  "protocol": "TCP",
  "l7-rules": [
    {
      "any.id.httpd1=": {
        "http": [
          {
            "path": "/public",
            "method": "GET"
          }
        ]
      }
    },
    {
      "any.id.httpd2=": {
        "http": [
          {}
        ]
      }
    }
  ]
}
              }
            ],
            "ingress": []
          },
          "policy-enabled": "egress",
          "policy-revision": 40
        },
        "spec": {
          "allowed-egress-identities": [
            5572
          ],
          "allowed-ingress-identities": [
            1
          ],
          "build": 40,
          "cidr-policy": {
            "egress": [],
            "ingress": []
          },
          "id": 51492,
          "l4": {
            "egress": [
              {
                "derived-from-rules": [
                  [],
                  []
                ],
                "rule": {
  "port": 80,
  "protocol": "TCP",
  "l7-rules": [
    {
      "any.id.httpd1=": {
        "http": [
          {
            "path": "/public",
            "method": "GET"
          }
        ]
      }
    },
    {
      "any.id.httpd2=": {
        "http": [
          {}
        ]
      }
    }
  ]
}
              }
            ],
            "ingress": []
          },
          "policy-enabled": "egress",
          "policy-revision": 40
        }
      },
      "realized": {
        "label-configuration": {
          "user": []
        },
        "options": {
          "Conntrack": "Enabled",
          "ConntrackAccounting": "Enabled",
          "ConntrackLocal": "Disabled",
          "Debug": "Enabled",
          "DebugLB": "Disabled",
          "DropNotification": "Enabled",
          "EgressPolicy": "Enabled",
          "IngressPolicy": "Disabled",
          "NAT46": "Disabled",
          "TraceNotification": "Enabled"
        }
      },
      "state": "ready"
    }
  }
]

Of note in the above endpoint get output:
      "policy": {
        "proxy-statistics": [
          {
            "allocated-proxy-port": 12908,
            "location": "egress",
            "port": 80,
            "protocol": "http",
            "statistics": {
              "requests": {
                "denied": 3,
                "forwarded": 3,
                "received": 6
              },
              "responses": {
                "forwarded": 3,
                "received": 3
              }
            }
          }
        ],
        "realized": {
          "allowed-egress-identities": [
            5572
          ],
          "allowed-ingress-identities": [
            1
          ],
          "build": 40,
          "cidr-policy": {
            "egress": [],
            "ingress": []
          },
          "id": 51492,
          "l4": {
            "egress": [
              {
                "derived-from-rules": [
                  [],
                  []
                ],
                "rule": {
  "port": 80,
  "protocol": "TCP",
  "l7-rules": [
    {
      "any.id.httpd1=": {
        "http": [
          {
            "path": "/public",
            "method": "GET"
          }
        ]
      }
    },
    {
      "any.id.httpd2=": {
        "http": [
          {}
        ]
      }
    }
  ]
}
              }
            ],
            "ingress": []
          },
          "policy-enabled": "egress",
          "policy-revision": 40
        },

the proxy configuration for this endpoint is up-to-date. This indicates that there may be an issue with the syncing of the IP to identity mappings in Envoy for the endpoint to which egress is being performed (httpd2). This goes in line with the initial description, in which @eloycoto said:

Connection using IPv4 works, it didn't work using Ipv6:

IP-identity mappings are pushed to Envoy asynchronously when an event is received by the key-value store for the ip-identity mapping. Let's see if f00d::a0f:0:0:29bf mapping to identity 5572 is present in the logs.
NOTE: When debugging this issue for https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated/3057/, we noticed loss of logs. This may be issue https://github.com/cilium/cilium/issues/3860 .

To see what identity maps to which IPs in Envoy, I checked the logs for the following string: NetworkPolicyHosts accepted with

May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="level=debug msg="bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:59] gRPC config for type.googleapis.com/cilium.NetworkPolicyHosts accepted with 7 resources: [policy: 1"
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "<IP_ADDRESS>""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "<IP_ADDRESS>""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "f00d::a0f:0:0:8ad6""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "fc00::10ca:1""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg=", policy: 54590"
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "<IP_ADDRESS>""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "f00d::a0f:0:0:dc06""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg=", policy: 5572"
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "<IP_ADDRESS>""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "f00d::a0f:0:0:981b""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg=", policy: 61249"
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "<IP_ADDRESS>""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "f00d::a0f:0:0:74ca""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg=", policy: 56980"
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "<IP_ADDRESS>""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "f00d::a0f:0:0:ad""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg=", policy: 34969"
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "<IP_ADDRESS>""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "f00d::a0f:0:0:367d""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg=", policy: 9750"
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "<IP_ADDRESS>""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: "f00d::a0f:0:0:72c5""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="]" subsys=envoy-config thread=15350"
It looks like we are indeed being rate-limited by journald:
May 02 15:10:27 runtime cilium-agent[11937]: level=debug msg="level=debug msg=\"bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:59] gRPC config for type.googleapis.com/cilium.NetworkPolicyHosts accepted with 6 resources: [policy: 1"
May 02 15:10:27 runtime cilium-agent[11937]: level=debug msg="host_addresses: \"<IP_ADDRESS>\""
May 02 15:10:27 runtime cilium-agent[11937]: level=debug msg="host_addresses: \"<IP_ADDRESS>\""
May 02 15:10:27 runtime cilium-agent[11937]: level=debug msg="host_addresses: \"f00d::a0f:0:0:8ad6\""
May 02 15:10:49 runtime cilium-agent[11937]: level=debug msg="level=debug msg=\"external/envoy/source/common/upstream/original_dst_cluster.cc:126] Cleaning up stale original dst hosts.\" subsys=envoy-upstream thread=15350"

So we can't see what IP-identity mappings got sent to Envoy :(

The problem is that Envoy only got notified of IPs for this identity (5572) that do not correspond to the httpd2 endpoint:
$ grep -A3 "policy: 5572" cilium.log | grep -B1 "host_addresses"
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg=", policy: 5572"
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: \"<IP_ADDRESS>\""
May 02 15:10:26 runtime cilium-agent[11937]: level=debug msg="host_addresses: \"f00d::a0f:0:0:981b\""

Messaged @jrajahalme and @rlenglet to confirm my analysis here. We need to figure out why these IP -> identity mappings aren't being updated in Envoy.

Looks like your analysis is sound. Were the related xDS cache update logs missing?

@jrajahalme  this is what is in the logs:
$ grep "preparing new cache transaction: upserting\|updating resource in cache\|inserting resource into cache\|xdsResourceName=f00d::a0f:0:0:29bf" cilium.log | grep "f00d::a0f:0:0:29bf"
May 02 14:59:28 runtime cilium-agent[22991]: level=debug msg="inserting resource into cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=167
May 02 14:59:32 runtime cilium-agent[22991]: level=debug msg="updating resource in cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=169
May 02 14:59:36 runtime cilium-agent[22991]: level=debug msg="updating resource in cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=177
May 02 14:59:42 runtime cilium-agent[22991]: level=debug msg="updating resource in cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=187
May 02 14:59:50 runtime cilium-agent[22991]: level=debug msg="updating resource in cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=193
May 02 15:00:17 runtime cilium-agent[22991]: level=debug msg="updating resource in cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=237
May 02 15:01:27 runtime cilium-agent[22991]: level=debug msg="updating resource in cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=243
May 02 15:03:19 runtime cilium-agent[22991]: level=debug msg="updating resource in cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=261
May 02 15:04:59 runtime cilium-agent[22991]: level=debug msg="updating resource in cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=271
May 02 15:10:59 runtime cilium-agent[11937]: level=debug msg="inserting resource into cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=205
May 02 15:13:50 runtime cilium-agent[11937]: level=debug msg="updating resource in cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=239
May 02 15:16:53 runtime cilium-agent[11937]: level=debug msg="updating resource in cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=273
May 02 15:17:08 runtime cilium-agent[11937]: level=debug msg="updating resource in cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=279
May 02 15:18:20 runtime cilium-agent[11937]: level=debug msg="updating resource in cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=303
May 02 15:18:51 runtime cilium-agent[11937]: level=debug msg="updating resource in cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=307
May 02 15:19:57 runtime cilium-agent[11937]: level=debug msg="updating resource in cache" xdsResourceName="f00d::a0f:0:0:29bf" xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=343

Given that there is a gap in the logs which might contain critical information for debugging (which has been fixed in master by https://github.com/cilium/cilium/pull/4054 ), I think we can park our work on this for now as we cannot definitively say what information was sent to Envoy from the logs. If this issue occurs again, we can debug like I did above and triage further. Thus, I am reducing priority of this.

I might have hit this again in a PR build which only edits code comments (and thus is basically a build of master).
Build link: https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated/3268/
Logs:
test_results_Cilium-PR-Ginkgo-Tests-Validated_3268.zip
I'll investigate locally and report back here.

Step which failed:
STEP: Client "app3" attempting to "curl public URL on" "httpd1"

Policy import finished:
time="2018-05-16T23:18:15Z" level=info msg="policy import finished and revision increased" file-path=/home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/Policies-l3-dependent-l7-egress.json policyRevision=36 test=RunPolicies

time when traffic test was ran:
time="2018-05-16T23:18:16Z" level=debug msg="running command: docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://<IP_ADDRESS>:80/public"
cmd: "docker exec -i  app3 curl -s --fail --connect-timeout 5 --max-time 5 http://<IP_ADDRESS>:80/public" exitCode: 22

Endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4            STATUS
           ENFORCEMENT        ENFORCEMENT
3978       Disabled           Disabled          58772      container:id.app2             f00d::a0f:0:0:f8a    <IP_ADDRESS>    ready
4314       Disabled           Enabled           36673      container:id.app3             f00d::a0f:0:0:10da   <IP_ADDRESS>     ready
10687      Disabled           Disabled          53409      container:id.httpd1           f00d::a0f:0:0:29bf   <IP_ADDRESS>   ready
                                                           container:id.service1
15124      Disabled           Disabled          13127      container:id.httpd3           f00d::a0f:0:0:3b14   <IP_ADDRESS>   ready
                                                           container:id.service1
25729      Disabled           Disabled          21718      container:id.app1             f00d::a0f:0:0:6481   <IP_ADDRESS>    ready
29898      Disabled           Disabled          35293      reserved:health               f00d::a0f:0:0:74ca   <IP_ADDRESS>    ready
60670      Disabled           Disabled          24958      container:id.httpd2           f00d::a0f:0:0:ecfe   <IP_ADDRESS>   ready
                                                           container:id.service1

app3 endpoint ID: 4314
httpd1 endpoint ID: 10687
The policy for this test:
[{
    "endpointSelector":
        {"matchLabels":{"id.app3":""}},
    "egress": [{
	"toEndpoints": [
	    {"matchLabels":{"id.httpd1":""}}
	],
        "toPorts": [{
            "ports": [
                  {"port": "80",   "protocol": "tcp"}],
            "rules": {
                "HTTP": [
                    {"method": "GET", "path": "/public"}]
            }
        }]
    }]
},{
    "endpointSelector":
        {"matchLabels":{"id.app3":""}},
    "egress": [{
        "toEndpoints": [
            {"matchLabels":{"id.httpd2":""}}
        ]
    }]
}]

Thus, app3 should have entries in its BPF PolicyMap allowing all traffic to httpd2 , and traffic to httpd1 on port 80 / TCP, and it does. The packet counter is nonzero indicating that packets have indeed gone through the datapath with this entry in 4314's policy map:
BPF Policy Get 4314
DIRECTION   LABELS (source:key[=value])   PORT/PROTO   BYTES   PACKETS
Egress      container:id.httpd1           80/TCP       487     6
            container:id.service1
Egress      container:id.httpd2           80/TCP       0       0
            container:id.service1
Ingress     reserved:host                 ANY          0       0
Egress      container:id.httpd2           ANY          0       0
            container:id.service1

4314 endpoint get policy:
      "policy": {
        "proxy-statistics": [
          {
            "allocated-proxy-port": 19870,
            "location": "egress",
            "port": 80,
            "protocol": "http",
            "statistics": {
              "requests": {
                "denied": 1,
                "received": 1
              },
              "responses": {}
            }
          }
        ],
        "realized": {
          "allowed-egress-identities": [
            24958
          ],
          "allowed-ingress-identities": [
            1
          ],
          "build": 36,
          "cidr-policy": {
            "egress": [],
            "ingress": []
          },
          "id": 36673,
          "l4": {
            "egress": [
              {
                "derived-from-rules": [
                  [],
                  []
                ],
                "rule": {
  "port": 80,
  "protocol": "TCP",
  "l7-rules": [
    {
      "any.id.httpd1=": {
        "http": [
          {
            "path": "/public",
            "method": "GET"
          }
        ]
      }
    },
    {
      "any.id.httpd2=": {
        "http": [
          {}
        ]
      }
    }
  ]
}
              }
            ],
            "ingress": []
          },
          "policy-enabled": "egress",
          "policy-revision": 36
        },
        "spec": {
          "allowed-egress-identities": [
            24958
          ],
          "allowed-ingress-identities": [
            1
          ],
          "build": 36,
          "cidr-policy": {
            "egress": [],
            "ingress": []
          },
          "id": 36673,
          "l4": {
            "egress": [
              {
                "derived-from-rules": [
                  [],
                  []
                ],
                "rule": {
  "port": 80,
  "protocol": "TCP",
  "l7-rules": [
    {
      "any.id.httpd1=": {
        "http": [
          {
            "path": "/public",
            "method": "GET"
          }
        ]
      }
    },
    {
      "any.id.httpd2=": {
        "http": [
          {}
        ]
      }
    }
  ]
}
              }
            ],
            "ingress": []
          },
          "policy-enabled": "egress",
          "policy-revision": 36
        }
      },

There is a proxy port configured for this endpoint, but it has denied a request.

For httpd1's IP, am checking logs to see its lifecycle:
$ grep "<IP_ADDRESS>.*modification" cilium.log
May 16 23:10:18 runtime cilium-agent[15780]: level=debug msg="upserting IP->ID mapping to kvstore" identity=53409 ipAddr=<IP_ADDRESS> ipMask="<nil>" modification=Upsert
May 16 23:10:18 runtime cilium-agent[15780]: level=debug msg="endpoint IP cache state change" cached-identity=0 identity=53409 ipAddr=<IP_ADDRESS> ipMask="<nil>" modification=Upsert
May 16 23:10:18 runtime cilium-agent[15780]: level=debug msg="daemon notified of IP-Identity cache state change" identity=53409 ipAddr=<IP_ADDRESS> ipMask="<nil>" modification=Upsert
May 16 23:15:18 runtime cilium-agent[15780]: level=debug msg="upserting IP->ID mapping to kvstore" identity=53409 ipAddr=<IP_ADDRESS> ipMask="<nil>" modification=Upsert

I checked to see what the NetworkPolicyHosts that are sent to Envoy for this endpoint are:
$ grep -A 150 "NetworkPolicyHosts accepted with" cilium.log | grep -A2 "policy: 53409"
May 16 23:08:20 runtime cilium-agent[15780]: level=debug msg=", policy: 53409"
May 16 23:08:20 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"<IP_ADDRESS>\""
May 16 23:08:20 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"<IP_ADDRESS>\""
--
May 16 23:10:01 runtime cilium-agent[15780]: level=debug msg=", policy: 53409"
May 16 23:10:01 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"<IP_ADDRESS>\""
May 16 23:10:01 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"<IP_ADDRESS>\""
--
May 16 23:10:02 runtime cilium-agent[15780]: level=debug msg="policy: 53409"
May 16 23:10:02 runtime cilium-agent[15780]: level=debug msg="ingress_per_port_policies {"
May 16 23:10:02 runtime cilium-agent[15780]: level=debug msg="}"
--
May 16 23:10:02 runtime cilium-agent[15780]: level=debug msg=", policy: 53409"
May 16 23:10:02 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"<IP_ADDRESS>\""
May 16 23:10:02 runtime cilium-agent[15780]: level=debug msg="ACK received but no observers are waiting for ACKs" xdsClientNode="id:\"host~<IP_ADDRESS>~no-id~localdomain\" cluster:\"cluster1\" build_version:\"16ccd368b5d8358560c766c0deaba656139506c7/1.7.0-dev/Modified/RELEASE\" " xdsNonce=3 xdsStreamID=2 xdsTypeURL=type.googleapis.com/cilium.NetworkPolicyHosts xdsVersionInfo=13
--
May 16 23:10:02 runtime cilium-agent[15780]: level=debug msg="policy: 53409"
May 16 23:10:02 runtime cilium-agent[15780]: level=debug msg="ingress_per_port_policies {"
May 16 23:10:02 runtime cilium-agent[15780]: level=debug msg="}"
--
May 16 23:10:02 runtime cilium-agent[15780]: level=debug msg=", policy: 53409"
May 16 23:10:02 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"<IP_ADDRESS>\""
May 16 23:10:02 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"<IP_ADDRESS>\""
--
May 16 23:10:03 runtime cilium-agent[15780]: level=debug msg="level=debug msg=\"bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:59] gRPC config for type.googleapis.com/cilium.NetworkPolicyHosts accepted with 7 resources: [policy: 53409"
May 16 23:10:03 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"<IP_ADDRESS>\""
May 16 23:10:03 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"f00d::a0f:0:0:44c7\""
--
May 16 23:10:12 runtime cilium-agent[15780]: level=debug msg=", policy: 53409"
May 16 23:10:12 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"<IP_ADDRESS>\""
May 16 23:10:12 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"f00d::a0f:0:0:44c7\""
--
May 16 23:10:14 runtime cilium-agent[15780]: level=debug msg=", policy: 53409"
May 16 23:10:14 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"<IP_ADDRESS>\""
May 16 23:10:14 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"f00d::a0f:0:0:44c7\""
--
May 16 23:10:15 runtime cilium-agent[15780]: level=debug msg=", policy: 53409"
May 16 23:10:15 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"<IP_ADDRESS>\""
May 16 23:10:15 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"f00d::a0f:0:0:44c7\""
--
May 16 23:10:17 runtime cilium-agent[15780]: level=debug msg="level=debug msg=\"bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:59] gRPC config for type.googleapis.com/cilium.NetworkPolicyHosts accepted with 11 resources: [policy: 53409"
May 16 23:10:17 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"<IP_ADDRESS>\""
May 16 23:10:17 runtime cilium-agent[15780]: level=debug msg="host_addresses: \"f00d::a0f:0:0:44c7\""

There is no host address for httpd1's IP?
$ grep -B3 "host_addresses: \"<IP_ADDRESS>\""  cilium.log
$

This would explain why Envoy denied it.

It looks like the XDS cache is updated:
$ grep "preparing new cache transaction: upserting\|updating resource in cache\|inserting resource into cache\|xdsResourceName=<IP_ADDRESS>" cilium.log | grep "<IP_ADDRESS>"
May 16 23:10:18 runtime cilium-agent[15780]: level=debug msg="inserting resource into cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=134
May 16 23:10:31 runtime cilium-agent[15780]: level=debug msg="updating resource in cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=142
May 16 23:12:54 runtime cilium-agent[15780]: level=debug msg="updating resource in cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=148
May 16 23:13:27 runtime cilium-agent[15780]: level=debug msg="updating resource in cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=154
May 16 23:15:58 runtime cilium-agent[15780]: level=debug msg="updating resource in cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=164
May 16 23:16:50 runtime cilium-agent[15780]: level=debug msg="updating resource in cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=178
May 16 23:17:00 runtime cilium-agent[15780]: level=debug msg="updating resource in cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=186
May 16 23:17:27 runtime cilium-agent[15780]: level=debug msg="updating resource in cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=192

@rlenglet @jrajahalme where can we go from here?

It looks like the XDS cache is updated:
Have you looked at deletions from the cache?

I could not find "app3 curl" from cilium.log, nor any evidence that there was a proxy DENY at that time or right after?

@jrajahalme check test-output.log for the test logs; cilium.log only contains the Cilium logs.
@rlenglet Doesn't look like anything is deleted here, I searched the logs again looking for deletions as well:
$ grep "preparing new cache transaction: upserting\|updating resource in cache\|inserting resource into cache\|deleting resource from cache\|xdsResourceName=<IP_ADDRESS>" cilium.log | grep "<IP_ADDRESS>"
May 16 23:10:18 runtime cilium-agent[15780]: level=debug msg="inserting resource into cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=134
May 16 23:10:31 runtime cilium-agent[15780]: level=debug msg="updating resource in cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=142
May 16 23:12:54 runtime cilium-agent[15780]: level=debug msg="updating resource in cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=148
May 16 23:13:27 runtime cilium-agent[15780]: level=debug msg="updating resource in cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=154
May 16 23:15:58 runtime cilium-agent[15780]: level=debug msg="updating resource in cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=164
May 16 23:16:50 runtime cilium-agent[15780]: level=debug msg="updating resource in cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=178
May 16 23:17:00 runtime cilium-agent[15780]: level=debug msg="updating resource in cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=186
May 16 23:17:27 runtime cilium-agent[15780]: level=debug msg="updating resource in cache" xdsResourceName=<IP_ADDRESS> xdsTypeURL=type.googleapis.com/cilium.NetworkPolicy xdsVersionInfo=192

Last hostmap update Envoy has received is at 23:10:17:
May 16 23:10:17 runtime cilium-agent[15780]: level=debug msg="level=debug msg=\"bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:59] gRPC config for type.googleapis.com/cilium.NetworkPolicyHosts accepted with 11 resources: [policy: 53409"

Destination endpoint's IP is assigned also at 23:10:17:
May 16 23:10:17 runtime cilium-agent[15780]: level=debug msg="Update endpoint from API PATCH" code=OK containerID= endpointID=10687 endpointState=waiting-for-identity ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:29bf" k8sPodName=":" policyRevision=0 type=0

Reading the full logs for timestamp 23:10:17 reveal that the endpoint is created and assigned it's IP address AFTER the last NetworkPolicyHosts update to Envoy is made!

$ grep 'upserting IP->ID mapping to kvstore' cilium.log | grep '<IP_ADDRESS>'
May 16 23:10:18 runtime cilium-agent[15780]: level=debug msg="upserting IP->ID mapping to kvstore" identity=53409 ipAddr=<IP_ADDRESS> ipMask="<nil>" modification=Upsert
May 16 23:15:18 runtime cilium-agent[15780]: level=debug msg="upserting IP->ID mapping to kvstore" identity=53409 ipAddr=<IP_ADDRESS> ipMask="<nil>" modification=Upsert

$ grep 'received event' cilium.log | grep <IP_ADDRESS>
May 16 23:10:18 runtime cilium-agent[15780]: level=debug msg="received event" key=cilium/state/ip/v1/default/<IP_ADDRESS> kvstore-event=create
May 16 23:15:18 runtime cilium-agent[15780]: level=debug msg="received event" key=cilium/state/ip/v1/default/<IP_ADDRESS> kvstore-event=modify

No problem with writing and reading from etcd.

$ grep 'Upserting into ipcache layer' cilium.log  | grep <IP_ADDRESS>
May 16 23:10:18 runtime cilium-agent[15780]: level=debug msg="Upserting into ipcache layer" identity=53409 ipAddr=<IP_ADDRESS>

The IP-to-ID mapping that we see in xDS comes from the first Cilium instance, before the Cilium crash:
May 16 23:07:06 runtime cilium-agent[21134]: level=info msg="Identity of endpoint changed" containerID=450502724c endpointID=17607 identity=53409 identityLabels="container:id.httpd1,container:id.service1" ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:44c7" k8sPodName=":" oldIdentity="no identity" policyRevision=0

This mapping survives the crash, in etcd, and gets restored into the ipcache and xDS cache when Cilium restarts.
It seems like the calculation of the new list of IP addresses for identity 53409 is incorrect, when the new endpoint is allocated that IP.


Missing pages generation workflow

@mikelikesrobots could I please ask you to set up the correct workflow to build the site?
I've never done it and I have no visibility on how it was done in the other repo

Solved


Didn't get my BTC.

Hi there. The address that you gave me to send my BTC  to you was bc1q2hzf3axtj2jcmpsvtg3u3llg7706hvep5yn5sn.. The address that I gave you to send  my BTC was  bc1qsffhh4amnm2c8hgfd3cjekln3nfez4yqm5vc8q And. I didn't still get it. If it's something wrong -send my BTC to address from which I deposited to

I did not give you anything.
You have been scammed, but I am not the scammer. I'm sorry but there is very little likehood that you will see your coins again. See the README or "About" section on this repo.

how did i get scammed? i went on coinjoin.io.

That is a scam website which encourages users to send funds, but does not return them.

but firt time i got 100$ ))

now understood ,thanks. got an experience)

Yikes! Glad you caught it early then.

Здравствуйте! Помогите пожалуйста разобраться с транзакцией:  bc1q7kgzl4glnzp42wgzz20n0dvwwad6w2j89z0nk9
0.05921691 BTC

Это мошеннический сайт, который призывает пользователей отправлять средства, но не возвращает их.

Помогите разобраться с транзакцией

Какой Ваш официальный сайт?


Add dat-polyfill

Fritter is a great application for showing off the power of the P2P web.
That's why I think it would be a great starting point to get people onboarded from the regular web.
I've managed to get fritter to work on the web using dat-polyfill. From the look of it, pretty much everything is working.

Creating new accounts
Following users
Viewing images / clicking links

Would you be open to having the dat-polyfill added into the main fritter repo? With that in place, people can view content from fritter on their phones, or on their desktops without having to install Beaker.
My goal with this is to give people a taste of what it's like and to make the technology more accessible. Ideally, people using the polyfill version of fritter should have something popping up prompting them to install Beaker (or Bunsen once that's out on mobile), along with a description of why it'd be better (privacy, offline-first, etc).
Some of the limitations of I can thing of when using this approach:

Gateways can discover which dats a user is accessing, so users should be aware that they might have their dat URLs seen by third parties (my gateway doesn't track any of this, but it's a technical limitation)
Currently there's just my gateway that's up which has this websocket sync thing, and it doesn't have TLS support yet.

I'll need to figure out getting letsencrypt wildcard certs up, or you'll need to set up a different one elsewhere. Maybe a reason for adding WS support to Hashbase

My gateway doesn't support the 404 fallback yet so URLs on fritter won't work on it after refreshing

This won't matter too much since people will be accessing it through the hashbase version of fritter which works properly

I don't have the versions of dat-polyfill or dat-archive-web that work tagged yet, but that'll be done soon.

I'd definitely be interested in seeing this! It's exactly the use-case that @flpvsk is asking about in https://github.com/beakerbrowser/beaker/issues/1018

Awesome. I think the only blocker would be to have a gateway with HTTPS that people can use. I'll send a PR when it's ready.
By the way, would you mind linking me to whatever you followed to get the Letsencrypt wildcard DNS cert working for hashbase? Do you have it automated by any chance?

I dont recall what I used, but I had to google around a bit. I maybe found it in an issue

K, I'll just look around, then. I don't think the docker image is compatible with my DNS registrar, sadly. :(


throughput degradation in load test

Environment

OS: Debian 8
Erlang/OTP:  Cannot found it
EMQ: 2.2.0

Hardware
AWS EC2 machine, type M4.XLARGE  for the subscriber
AWS EC2 machine, type C4.2XLARGE for the publisher
Throughput in my use case
1 device can publish 100 msg / seconds with a payload of a 1,3KB.
What I expect from the load test
Find the max throughput (in /out) that can support only one node and then choose the appropriate number of node composing my cluster. We're planning to scale about 10 000 devices publishing data to MQTT.
MQTT Informations
We're using the qos 1 and the topics in which data are published is like : topicName/deviceName/uniqId
Description
Hi,
I did a simple load test using emqtt_benchmark like this :
/emqtt_bench_pub -h myHost -c 75 -I 10 -t streaming_junk/%i/%i -s 1316 -q 1
When I'm running this, it say that the rate is about 7500 msg / sec, which seems good because it corresponding to the options set in the previous command line.
sent(1003): total=4411, rate=4411(msg/sec)
sent(2001): total=11897, rate=7486(msg/sec)
sent(3001): total=19395, rate=7498(msg/sec)
sent(4001): total=26897, rate=7502(msg/sec)
sent(5001): total=34392, rate=7495(msg/sec)
sent(6000): total=41893, rate=7501(msg/sec)
sent(7001): total=49397, rate=7504(msg/sec)
sent(8000): total=56880, rate=7483(msg/sec)
sent(9001): total=64400, rate=7520(msg/sec)
sent(10002): total=71900, rate=7500(msg/sec)
sent(11001): total=79397, rate=7497(msg/sec)
sent(12001): total=86891, rate=7494(msg/sec)
sent(13001): total=94390, rate=7499(msg/sec)
sent(14001): total=101889, rate=7499(msg/sec)

My first question is about the rate : it is the real rate of publishing message or some log for the user ?
Then about my consumer (it's classic java paho library), I observed my consumer rate decreased in the time when the publisher rate is quite stable:
2017-08-03 17:18:40.501 - Fetch 7007 messages from mqtt [Total : 9164 ]
2017-08-03 17:18:41.502 - Fetch 7142 messages from mqtt [Total : 16306 ]
2017-08-03 17:18:42.502 - Fetch 7186 messages from mqtt [Total : 23492 ]
2017-08-03 17:18:43.502 - Fetch 7159 messages from mqtt [Total : 30651 ]
2017-08-03 17:18:44.502 - Fetch 7097 messages from mqtt [Total : 37748 ]
2017-08-03 17:18:45.502 - Fetch 7142 messages from mqtt [Total : 44890 ]
2017-08-03 17:18:46.502 - Fetch 7097 messages from mqtt [Total : 51987 ]
2017-08-03 17:18:47.502 - Fetch 7042 messages from mqtt [Total : 59029 ]
2017-08-03 17:18:48.502 - Fetch 7104 messages from mqtt [Total : 66133 ]
2017-08-03 17:18:49.502 - Fetch 7014 messages from mqtt [Total : 73147 ]
2017-08-03 17:18:50.502 - Fetch 7003 messages from mqtt [Total : 80150 ]
2017-08-03 17:18:51.502 - Fetch 7008 messages from mqtt [Total : 87158 ]
2017-08-03 17:18:52.502 - Fetch 7006 messages from mqtt [Total : 94164 ]
2017-08-03 17:18:53.502 - Fetch 6851 messages from mqtt [Total : 101015 ]
2017-08-03 17:18:54.502 - Fetch 6808 messages from mqtt [Total : 107823 ]
2017-08-03 17:18:55.502 - Fetch 6689 messages from mqtt [Total : 114512 ]
2017-08-03 17:18:56.502 - Fetch 6556 messages from mqtt [Total : 121068 ]
2017-08-03 17:18:57.502 - Fetch 6785 messages from mqtt [Total : 127853 ]
2017-08-03 17:18:58.502 - Fetch 6722 messages from mqtt [Total : 134575 ]
2017-08-03 17:18:59.502 - Fetch 6732 messages from mqtt [Total : 141307 ]
2017-08-03 17:19:00.502 - Fetch 6819 messages from mqtt [Total : 148126 ]
2017-08-03 17:19:01.502 - Fetch 6742 messages from mqtt [Total : 154868 ]
2017-08-03 17:19:02.502 - Fetch 6671 messages from mqtt [Total : 161539 ]
2017-08-03 17:19:03.502 - Fetch 6664 messages from mqtt [Total : 168203 ]
2017-08-03 17:19:04.502 - Fetch 6530 messages from mqtt [Total : 174733 ]
2017-08-03 17:19:05.502 - Fetch 6547 messages from mqtt [Total : 181280 ]
2017-08-03 17:19:06.502 - Fetch 6440 messages from mqtt [Total : 187720 ]
2017-08-03 17:19:07.502 - Fetch 6506 messages from mqtt [Total : 194226 ]
2017-08-03 17:19:08.502 - Fetch 6538 messages from mqtt [Total : 200764 ]
2017-08-03 17:19:09.502 - Fetch 6426 messages from mqtt [Total : 207190 ]
2017-08-03 17:19:10.502 - Fetch 6424 messages from mqtt [Total : 213614 ]
2017-08-03 17:19:11.502 - Fetch 6309 messages from mqtt [Total : 219923 ]
2017-08-03 17:19:12.502 - Fetch 6238 messages from mqtt [Total : 226161 ]
2017-08-03 17:19:13.502 - Fetch 6282 messages from mqtt [Total : 232443 ]
2017-08-03 17:19:14.502 - Fetch 6229 messages from mqtt [Total : 238672 ]
2017-08-03 17:19:15.502 - Fetch 6239 messages from mqtt [Total : 244911 ]
2017-08-03 17:19:16.502 - Fetch 6176 messages from mqtt [Total : 251087 ]
2017-08-03 17:19:17.502 - Fetch 6020 messages from mqtt [Total : 257107 ]
2017-08-03 17:19:18.502 - Fetch 6053 messages from mqtt [Total : 263160 ]
2017-08-03 17:19:19.502 - Fetch 5895 messages from mqtt [Total : 269055 ]
2017-08-03 17:19:20.502 - Fetch 5727 messages from mqtt [Total : 274782 ]
2017-08-03 17:19:21.502 - Fetch 5663 messages from mqtt [Total : 280445 ]
2017-08-03 17:19:22.502 - Fetch 5562 messages from mqtt [Total : 286007 ]
2017-08-03 17:19:23.502 - Fetch 5614 messages from mqtt [Total : 291621 ]
2017-08-03 17:19:24.502 - Fetch 5629 messages from mqtt [Total : 297250 ]
2017-08-03 17:19:25.504 - Fetch 5566 messages from mqtt [Total : 302816 ]
2017-08-03 17:19:26.491 - Fetch 5508 messages from mqtt [Total : 308324 ]
2017-08-03 17:19:27.502 - Fetch 5631 messages from mqtt [Total : 313955 ]
2017-08-03 17:19:28.502 - Fetch 5562 messages from mqtt [Total : 319517 ]
2017-08-03 17:19:29.502 - Fetch 5457 messages from mqtt [Total : 324974 ]
2017-08-03 17:19:30.502 - Fetch 5411 messages from mqtt [Total : 330385 ]

The publisher and the subscriber are in the same AWS region and during the load tests. On the emq broker, I have 4 CPU but not bounded at all (sometimes goes up to 50 / 70 % but only for one CPU, others are quite calm).
I know that I'm using the qos 1 but the reach throughput seems to be very low for me regarding the AWS machine used for this test. I'm using the default parameters of emq.
I never drop messages during the test which is a good  for me but I want to know if there are some parameters to tune to reach a better throughput.
Thanking you guys,
Alifirat.

Hi guys,
I have some extra informations to give about my results.
When I did the same benchmark on 50 clients, I have no throughput issue and all my CPUs are not bounded. Below the screenshot after 2 minutes of publishing data :

I take the same screenshot for 100 clients publishing at the same rate and here the results ;

CPUs seems to not be used quite well which can explain my consumer throughput decreases along the time. After 5 minutes, I got less than 1000 msg / sec.

I have the same problem, I have a cluster including two 2-core 4G server, but I measured the client to accept the rate can only reach 1200msg / sec, and how my server only one thread(beam.smp), I use the emq2 .1.2, is my version of the problem? I used to test the client's configuration is 24-core 54G, I was using the order is
  ./emqtt_bench_pub -c 20 -k -I 10 -t bench /% i
./emqtt_bench_sub -c 2000 -k -i 1 -t bench /% i

Hi guys,
The issue comes from the publisher : instead to simulate 100 devices on 1 machine, I split it in two machines and I get the expected throughput.

Hi,
I reopen this issue because I got new problems in my load tests.
 Each node has the same hardware configuration that my first message.
In publisher side, I have 4 machines running the emqtt bencharming tool as below :
./emqtt_bench_pub -h myhost -c 50 -I 10 -t streaming_junk/%i/%i -s 1316 -q 1 -n myoffset

Each machine publish 100 msg / seconds (so 5000 msg / sec) and simulate about 50 devices. I limited the count to 50 because the machines used in the test didn't more support more than that.
In consumer side, I have two mqtt-clients (classical java paho) and each subscribe to the topic $local/streaming_junk/# in order to balance the load.
During 10 minutes, I got the expected throughput with 10 000 msg / second for each mqtt consumer but then my consumer's throughput decrease quickly :

You can see that the simulated devices are quite well balanced between the two nodes. My consumer throughputs decreases after 10 minutes, the used RAM increase due to qos 1. I changed two parameters of emq in each node :
mqtt.mqueue.max_lenght = 1000000 # this one is to avoid dropped messages
mqtt.session.max_inflight = 100

Can I have advices about the way to fix this issue ? Or maybe it's an existing problem in this release of emq ?

Hi,
I run the same test but on only one emq-node. I didn't change the number of publishers (200  devices streaming 100 msg / sec) and during 9 hours, all have been right then I got this message :
client 43 EXIT: {shutdown,tcp_closed}
client 47 EXIT: {shutdown,tcp_closed}
client 46 EXIT: {shutdown,tcp_closed}
client 41 EXIT: {shutdown,tcp_closed}
client 39 EXIT: {shutdown,tcp_closed}
client 22 EXIT: {shutdown,tcp_closed}
client 49 EXIT: {shutdown,tcp_closed}
client 40 EXIT: {shutdown,tcp_closed}
client 27 EXIT: {shutdown,tcp_closed}
client 42 EXIT: {shutdown,tcp_closed}
client 48 EXIT: {shutdown,tcp_closed}
client 36 EXIT: {shutdown,tcp_closed}
client 14 EXIT: {shutdown,tcp_closed}
client 18 EXIT: {shutdown,tcp_closed}
client 29 EXIT: {shutdown,tcp_closed}
client 4 EXIT: {shutdown,tcp_closed}
client 34 EXIT: {shutdown,tcp_closed}
client 19 EXIT: {shutdown,tcp_closed}
client 28 EXIT: {shutdown,tcp_closed}
client 8 EXIT: {shutdown,tcp_closed}
client 9 EXIT: {shutdown,tcp_closed}

So it means that the broker closed these connections.

Crash dump is being written to: /var/log/emqttd/crash.dump...done
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed

@alifirat Did find a resolution to your issue? I'm curious because I'm starting to experiment with this tool and other MQTT brokers.

@gihad Hi, I cannot found a solution for my issue, sorry.

@huangdan Please add a benchmark case for the issue

Hi, @alifirat , Sorry for late reply. EMQ X 3.0 has been released now, hope the latest emqx could help you solve this issue.


need tutorial

can you please provide a video tutorial for making our own Search bot for personal use. I loved your telegram seach bot but for sometime it is not working for me anymore. I am a complete noob when it comes to coding in general. I cannot understand half the things in your post.
Thanks.

It's possible that it's due to the very limited daily quota on Google Search API bring exceeded

I donot know how to code. But, i must say I absolutely love your bot. It is
a great tool for learning and keeping track of things.
On Fri, Jun 15, 2018 at 7:08 AM Ming<EMAIL_ADDRESS>wrote:

It's possible that it's due to the very limited daily quota on Google
Search API bring exceeded
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/nkming2/google-search-telegram-bot/issues/3#issuecomment-397487923,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AmO5TW_lBrk07ks-JffS8-ru4pNYxOrBks5t8xAPgaJpZM4UhNO4
.


deleted smui/tooltip

fix #872

after removing @smui/tooltip, dashboard/electricity has following error and cannot be opened at all.
Cannot read properties of undefined (reading 'mdc')
TypeError: Cannot read properties of undefined (reading 'mdc')
    at webpackUniversalModuleDefinition (webpack://mdc.[name]/webpack/universalModuleDefinition:16:21)
    at webpack://mdc.[name]/webpack/universalModuleDefinition:17:3
    at instantiateModule (file:///Users/j_igarashi/Documents/git/UNDP-Data/geohub/node_modules/vite/dist/node/chunks/dep-67e7f8ab.js:53058:15)

finally deleted SMUI!!!


Allow without redis

Please is there a way to use the package without redis?

No, this package only supports redis for locking, at least for now.


Allow link helpers to accept a block

To properly match the API of Rails's link_to, Kaminari's link_to_next_page et al should accept a block in place of a body.

Looked into implementing this. I guess it's not possible since link_to_{next,prev}_page is actually a wrapper around link_to_if, not link_to as I had thought - so the block is already taken by the default (ie, no more pages) text.


Is possible to add custom field types from server side?

Is it possible to add custom field types from the server-side. I'm asking because I'm looking to improve one existing form builder and I need to extend the field types.

It is on the roadmap, but no not possible yet. Thanks for asking!

and about the server-side rendering

I haven't tried it with SSR, but since it uses all Quasar components I see no reason why it couldn't be used that way.