bump testng to 7.9.0

https://github.com/testng-team/testng/releases/tag/7.9.0

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 0.00%. Comparing base (59551e4) to head (9cd21f5).
Report is 8 commits behind head on master.

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #12492       +/-   ##
=============================================
- Coverage     61.75%    0.00%   -61.76%
=============================================
  Files          2436     2360       -76
  Lines        133233   129489     -3744
  Branches      20636    20081      -555
=============================================
- Hits          82274        0    -82274
- Misses        44911   129489    +84578
+ Partials       6048        0     -6048

Flag
Coverage Δ

custom-integration1
?

integration
0.00% <ø> (-0.01%)
:arrow_down:

integration1
?

integration2
0.00% <ø> (ø)

java-11
?

java-21
0.00% <ø> (-61.63%)
:arrow_down:

skip-bytebuffers-false
0.00% <ø> (-61.75%)
:arrow_down:

skip-bytebuffers-true
?

temurin
0.00% <ø> (-61.76%)
:arrow_down:

unittests
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Swift 3 compatibility

Hey, I'd love to see this lib available in Swift 3 so I can implement it in my project.
Thanks 😄

Hi @lfarah, thanks for the reminder. I have updated the library to Swift 3.0. Let me know how it works for you.

Thanks! 🎉

No worries, closing the issue, feel free to reopen if necessary.


Empty compression messages

Description
We've identified an issue with compression, that the trailing empty ops added to the batch to preserve the batch shape and reserve sequence numbers are not really empty, as the content is still present in the deserializedContent property. This leads to compression creating a large payload which does not end up getting chunked.
Example of a compressed batch:

with actual content in the messages which are supposed to be empty:

This made the batch exceed 2.5 MB

It would be great to also have end-to-end test, that validates that we are sending substantially fewer bytes over the wire (compared to initial DDS payload) when dealing with large non-random payloads. Including hitting cases of compression, compression + chunking.

Ok, we need to keep this for back-compat reasons (old loader submit function requires an object passed as opposed to string contents). We could work around it, but it would add a performance hit as we'd be doing JSON.parse twice for the same message when compression happens. We can close on this as it's not a real issue. I'll add a note to enhance some of the end-to-end tests to verify the resulting payload size and add more confidence..


Unable to push existing image to resgistry

Unable to push existing image to registry through error "unauthorized: authentication required"
Portainer v1.11.1
docker info: Server Version: 1.12.6
OS: Ubuntu 14.04.5 LTS
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: overlay null bridge host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-74-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64

Portainer does not allow a user to push to Dockerhub or a private registry at the moment, see #147 and #56 for more info.

Portainer already allows a user to push to the Dockerhub or a private record at the moment?

@camanducci yes, this was introduced in the release 1.13.3


Realtime user activity msg subscribing

WebSocketing between online users clients
Sending of messages on three events:

Login ---> triggered after login into application or page has been successfully loaded
Logout ---> triggered after logout from application or page has been successfully unloaded (clossing browser)
Participants ---> once page is loaded user is asking for list of all online users

Ok, dneska po práci na to mrknu a pofixuju co půjde. Potom to mergnu.


Types for JSON webhook data?

It would be super helpful to have types for JSON webhook data...I don't suppose this exists somewhere and I missed it?

hu @tsongas
at the moment we don't have webhook data model within the library.

@ibalosh would you be open to accept a PR for this?

Hi @arosemena
yes, we are open for improvements of the library. Once we review the PR we can approve it add it to the codebase.

👍🏻 for this one as well!

we have released types for web hooks today, it's available as of version 3.0.19.
https://github.com/ActiveCampaign/postmark.js/wiki/Handling-Web-hooks


feat: [CHAOS-6152]: Add the docs for the pod-fs-fill fault

Thanks for contributing to the Harness Developer Hub! Our code owners will review your submission.
Description

Please describe your changes: __________________________________
Jira/GitHub Issue numbers (if any): ______________________________
Preview links/images (Internal contributors only): __________________

PR lifecycle
We aim to merge PRs within one week or less, but delays happen sometimes.
If your PR is open longer than two weeks without any human activity, please tag a code owner in a comment.
PRs must meet these requirements to be merged:

[ ] Successful preview build.
[ ] Code owner review.
[ ] No merge conflicts.
[ ] Release notes/new features docs: Feature/version released to at least one prod environment.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://6698ce93539f4d6bc7ea9b70--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://669a43cabbeee5c6990ba482--harness-developer.netlify.app


update llvm v14

Hey mate, I'm happy to merge this as is. But I thought that I'd mention that llvm support in this repo, leaves a lot to be desired. I have been slowly working on an llvm replacement for the gcc based toolchains for embedded devices here;
https://github.com/bazelembedded/rules_cc_toolchain/pull/35
This will eventually replace the toolchain definitions here. The goal being to have a more modular and unified toolchain that works for multiple platforms including the host.


Isolate main is already scheduled on mutator

Got a strange error recently, here are logs:
../../third_party/dart/runtime/vm/dart_api_impl.cc: 1708: error: Isolate main is already scheduled on mutator thread 0xa8f9f400, failed to schedule from os thread 0x5b27
pthread_create failed: couldn't allocate 528384-bytes mapped space: Out of memory
W/libc    ( 9741): pthread_create failed: couldn't allocate 528384-bytes mapped space: Out of memory
Original issue: https://github.com/Renesanse/worker_manager/issues/19#issue-759347845

This might be the same bug as https://github.com/flutter/flutter/issues/72144


Can't use tslint-loader: TypeError: this.addDependency is not a function

Trying to use tslint-loader, happypack fails with TypeError: this.addDependency is not a function

Oh, sorry, this feature seems not merged yet (#15)

Looking into it, need to figure out the deal with preLoaders / postLoaders. Any sample configs / sources would help a lot!


Shared generics work

Enough of shared generics support that after commenting out the reflection initialization injection code, we can compile and run:
internal class Program
{
    class Gen<T> where T : class
    {
        public T Do(object o)
        {
            return o as T;
        }
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static T Do<T>(object o) where T : class
    {
        return o as T;
    }

    private static void Main(string[] args)
    {
        Do<Object>(new object());
        Do<string>(new object());

        var gen1 = new Gen<string>();
        gen1.Do(new object());

        var gen2 = new Gen<object>();
        gen2.Do(new object());
    }
}

Not much else works and there's still a lot of work ahead.

@jkotas @davidwrighton PTAL
I guess we can either review this and merge it, or I can keep iterating on it until it becomes too big to review. ("This review is huge. LGTM?")

:clock1:


Fail HelidonTests by defaul when pinning jfr event is detected

We should monitor JFR event jdk.VirtualThreadPinned in @HelidonTest tests and fail the test causing the pinning

I think JFR requires Oracle JDK. I guess the way to approach this is to include one script, with Oracle JDK, that executes the tests with the JVM args:
-XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:StartFlightRecording=duration=200s,filename=flight.jfr
And then find events on the generated JRF file:
jfr print --events jdk.VirtualThreadPinned flight.jfr
If found, then error.

It seems from JDK 9 it is part of JDK, so we can use it in @HelidonTest

I am trying to make a simple reproducer, but for some reason I don't catch the jdk.VirtualThreadPinned event:
import static org.junit.Assert.assertTrue;

import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;

import jdk.jfr.Event;
import jdk.jfr.Name;
import jdk.jfr.consumer.RecordingStream;
import org.junit.Test;

public class JFRTest {

    @Test
    public void pinnedVirtualThreads() throws InterruptedException {
        CountDownLatch stopListening = new CountDownLatch(1);
        CountDownLatch startListening = new CountDownLatch(1);
        try (RecordingStream recordingStream = new RecordingStream()) {
            // Create and start the recording stream
            startRecordingStream(recordingStream, startListening, stopListening);

            // Make sure JFR listener is running before continue
            while (true) {
                new StartEvent().commit();
                if (startListening.await(1, TimeUnit.SECONDS)) {
                    break;
                }
            }

            // Simulate blocking operation
            Thread vt = Thread.ofVirtual().start(() -> {
                synchronized (this) {
                    System.out.println("Virtual Thread is pinned");
                }
            });
            vt.join();
        }
        // Wait till the jdk.VirtualThreadPinned triggers
        assertTrue("jdk.VirtualThreadPinned was not sent", stopListening.await(5, TimeUnit.SECONDS));
    }

    private void startRecordingStream(RecordingStream recordingStream, CountDownLatch startListening, CountDownLatch stopListening) {
        // Enable the jdk.VirtualThreadPinned event
        recordingStream.enable("jdk.VirtualThreadPinned").withStackTrace();

        // Notify listener is running after receiving the StartEvent
        recordingStream.onEvent("StartEvent", event -> {
            System.out.println("Received " + event);
            startListening.countDown();
        });

        // Set an event handler
        recordingStream.onEvent("jdk.VirtualThreadPinned", event -> {
            System.out.println("VirtualThreadPinned event detected!");
            System.out.println("Timestamp: " + event);
            stopListening.countDown();
        });

        // Start the recording stream
        recordingStream.startAsync();
    }

    @Name("StartEvent")
    private static class StartEvent extends Event {
    }
}

It fails with:
[INFO] Running com.example.general.JFRTest
Received StartEvent {
  startTime = 09:38:59.825 (2024-06-12)
  eventThread = "main" (javaThreadId = 1)
  stackTrace = [
    com.example.general.JFRTest.pinnedVirtualThreads() line: 25
    jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Object, Object[]) line: 103
    java.lang.reflect.Method.invoke(Object, Object[]) line: 580
    org.junit.runners.model.FrameworkMethod$1.runReflectiveCall() line: 59
    org.junit.internal.runners.model.ReflectiveCallable.run() line: 12
  ]
}

Virtual Thread is pinned
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.208 s <<< FAILURE! - in com.example.general.JFRTest
[ERROR] com.example.general.JFRTest.pinnedVirtualThreads  Time elapsed: 8.183 s  <<< FAILURE!
java.lang.AssertionError: jdk.VirtualThreadPinned was not sent
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at com.example.general.JFRTest.pinnedVirtualThreads(JFRTest.java:46)

You have to block the carrier thread for more than 20 millis(that is the default)
            Thread vt = Thread.ofVirtual().start(() -> {
                synchronized (this) {
                    Thread.sleep(50);
                }
            });

You have to block the carrier thread for more than 20 millis(that is the default)
            Thread vt = Thread.ofVirtual().start(() -> {
                synchronized (this) {
                    Thread.sleep(50);
                }
            });

I read that too, but I thought it means that be default jdk.VirtualThreadPinned is disabled, unless it takes more than 20ms that becomes enabled. Anyway, I added 2 seconds of sleep and still nothing.

The issue was the event was triggered but not flushed immediately. Closing the RecordingStream doesn't flush.
So I had to add before closing:
recordingStream.stop();


Add Gitpod configuration file

Hi, thank you for your contribution but we are not going to provide recommended deployment configurations at this time.


fixed a typo

What this PR does / why we need it:
Fix typo
Which issue(s) this PR fixes:
out --> our
Release note:
NONE

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks! :) Can you sign the CLA?


Failed to evaluate networking: failed to find plugin "rancher-bridge" due to metadata returning inconsistent results for /self

Rancher Versions:
Server: 1.5.2
healthcheck: 0.2.3
ipsec: rancher/net:holder
network-services: rancher/network-manager:v0.5.3
scheduler: rancher/scheduler:v0.7.5
kubernetes (if applicable):
Docker Version:
1.12.6
OS and where are the hosts located? (cloud, bare metal, etc):
Ubuntu 16.04/Bare Metal/Multiple locations
Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)
Single node rancher, external DB
Environment Type: (Cattle/Kubernetes/Swarm/Mesos)
Cattle
Steps to Reproduce:
I have about 100 hosts running 4 containers. On 5-6 of them ipsec runs ok, on every other host the ipsec stack fails (also the healthcheck) with Timeout getting ip. Inside the network-manager logs I can see:
nsenter: cannot open /proc/26034/ns/ipc: No such file or directory
time="2017-04-01T02:31:01Z" level=error msg="Failed to evaluate network state for 059fae587888cf4a80bc7020e0ab684927783790703d73591edf07fdf0f6e769: netplugin failed but error parsing its diagnostic message \"\": unexpected end of JSON input"
time="2017-04-01T02:31:01Z" level=error msg="Error processing event &docker.APIEvents{Action:\"start\", Type:\"container\", Actor:docker.APIActor{ID:\"059fae587888cf4a80bc7020e0ab684927783790703d73591edf07fdf0f6e769\", Attributes:map[string]string{\"io.rancher.container.agent_id\":\"58465\", \"io.rancher.project_service.name\":\"healthcheck/healthcheck\", \"io.rancher.service.hash\":\"d0a8fd4061d3b2a8c5782f5563db6df0b25655cb\", \"io.rancher.stack_service.name\":\"healthcheck/healthcheck\", \"io.rancher.cni.network\":\"ipsec\", \"io.rancher.scheduler.global\":\"true\", \"io.rancher.service.requested.host.id\":\"209\", \"io.rancher.stack.name\":\"healthcheck\", \"name\":\"r-healthcheck-healthcheck-28-2576d08e\", \"image\":\"rancher/healthcheck:v0.2.3\", \"io.rancher.container.create_agent\":\"true\", \"io.rancher.container.ip\":\"<IP_ADDRESS>/16\", \"io.rancher.container.mac_address\":\"02:4f:71:44:58:1a\", \"io.rancher.service.launch.config\":\"io.rancher.service.primary.launch.config\", \"io.rancher.cni.wait\":\"true\", \"io.rancher.container.system\":\"true\", \"io.rancher.container.uuid\":\"2576d08e-3100-4935-aa34-3ecb73da302a\", \"io.rancher.project.name\":\"healthcheck\", \"io.rancher.service.deployment.unit\":\"8a360a36-5ff6-497f-a7db-92193ecf2415\", \"io.rancher.container.name\":\"healthcheck-healthcheck-28\"}}, Status:\"start\", ID:\"059fae587888cf4a80bc7020e0ab684927783790703d73591edf07fdf0f6e769\", From:\"rancher/healthcheck:v0.2.3\", Time:1491013861, TimeNano:1491013861288487613}. Error: netplugin failed but error parsing its diagnostic message \"\": unexpected end of JSON input"

Can you provide the ipsec log or can you try http://rancher-metadata/2015-12-19/self in a container which use Managed network and ipsec failed on the same host.
Have you tried delete the network stack and re-run them, then restart the failed ipsec container.
Maybe it's related to #8368

No container uses managed network. All 4 are running with network_mode: host.  I just deactivated all my hosts in rancher because it was causing one host (where the scheduler is apparently running) to go at 400Mb/s and also my rancher server to stay 100% at all times with 8 cores and 16gb of RAM. One of the 4 containers is a healthchecker of mine designed to keep the other 3 alive so my services are running even with hosts deactivated in rancher.
I just wish I could get rid of this ipsec thing and clustering features as I'm not interested in them.
I only picked rancher because it was the only one supporting the run one container on each host feature and allowed you to NOT treat all your machines as a single block. Things are moving in the other direction it seems and it's giving me all sorts of issues. I can't even disable ipsec at all.
ipsec does not produce any logs, it doesn't even start.

Tried adding a dummy ubuntu (with the managed network) container that just sleeps and prints something to the console. It fails to start, with the same error saying it failed to bring up the network.

I have the same issue, reproducible with 1.5.2, 1.5.3 and 1.5.4 Error (Couldn't bring up network: failed to find plugin "rancher-bridge" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin])

I had the same issue because I used a virtualbox host rancher within a host rancher :-S. When I removed the virtualbox machine from the cluster the error disappear.
Are you using a similar config?

same issue happens to us on physical hardware  :-(

@asg1612 you can read the ticket and see it's not the same config - at all.

do someone managed some sort of workaround? this is really annoying issue as we cannot use more than 1 host for now (out of 4 which we are using normally)

me too

do someone managed some sort of workaround? this is really annoying issue as we cannot use more than 1 host for now (out of 4 which we are using normally)

Nope...and noone is responding from the devs either. I upgraded to 1.5.4 and 1.5.5 without luck. Bug is still present and it affects 90% of all our servers. I can't pinpoint the fault at all. I checked, re-installed, deleted, upgraded, downgraded ..you name it, nothing pops out.
We may have to roll our own custom solution as this is not a production ready setup for us unfortunately.

I've reproduce and solve a similar issue #8276 . Hope this helps.

@snipking I believe that's required when you run the agent on the same host with the server. Can you confirm?

same issue

@snipking
i'm investigating a bit here
there is no binaries in /opt/cni/bin dir in rancher/network-manager:v0.6.6
And all of my servers today show such messages like in first message.

Ok this is not funny anymore. Looks like healthcheck and ipsec are constantly failing
http://i.share.pho.to/dda8108c_o.png
Any docker container executed from rancher fails. Doing same thing from console works without issues.

Same issue on fresh, clean machine with Ubuntu 16

@ciokan Actually, I've reproduced the #8276 in two env which means it happen both when agent and server on the same host or not. Even one host in a rancher env with the wrong IP will cause this issue on other hosts in this env. This may not happen immediately because you may add a host with the wrong IP to an env which already have hosts working fine. But it may happen after you reboot some of your working hosts.
I'm not sure your problem same as me. But it worth a try.

@fedya I'm using rancher/network-manager:v0.6.6 and there's only rancher-bridge in /opt/cni/bin. In my case I didn't see any error message relevant with it.

if bad instance of rancher agent is up
i can't run containers until flush firewall rules commited by rancher.
Issue still exist and very annoying me

Still not working
docker 17.03.1
time="2017-04-20T11:12:02Z" level=info msg="Evaluating state from retry" cid=eac88e13f505ba0362c49601f269e758a8cf4bb4d2cec434eccdbce0ca908d6e count=14
time="2017-04-20T11:12:02Z" level=info msg="CNI up" cid=eac88e13f505ba0362c49601f269e758a8cf4bb4d2cec434eccdbce0ca908d6e networkMode=ipsec
time="2017-04-20T11:12:02Z" level=error msg="Failed to evaluate networking: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"

Something similar is happening to me after updating to 1.5.5. I've even tried creating a while new environment to try and get the ipsec stack to start, but it still exhibits the same error. "Failed to evaluate networking: netplugin failed but error parsing its diagnostic message \"\": unexpected end of JSON input"

@penguinxr2 yes, same issue, there is no network interface in container.

Same issue here, anyone have any updates?

My issues went away once I switched to Ubuntu 16.04 instead of Debian

We've been struggling all day with ipsec issues on one of the nodes after upgrading a rancher environment running 1.4 to 1.5.6 (in other words, we have several nodes where ipsec was working).
After a lot of headscratching and cussing, we got ipsec working following these simple steps:

copy /var/lib/docker/volumes/rancher-cni-driver/_data/bin/rancher-bridge from one of the working nodes (we were missing this file on the node with ipsec issues).
the file is tiny, a one line script calling nsenter, and needs the PID of the target process it should enter. this needs to be adjusted for each host and time.
so, replace the pid for the -t option to nsenter with the pid of ps ax | grep rancher-cni (it should be the sidekick tailing the log file:

[root@xxx bin]# ps ax | grep rancher-cni
2754 ?        Ss     0:00 tail ---disable-inotify -F /var/log/rancher-cni.log

contents of /var/lib/docker/volumes/rancher-cni-driver/_data/bin/rancher-bridge after patching:

[root@xxx bin]# cat rancher-bridge
#!/bin/sh
exec /usr/bin/nsenter -m -u -i -n -p -t 2754 -- $0 "$@"

now, restart network-manager and things starts rolling!! ;)

So, in conclusion, it seems like the process responsible for updating this file is failing, for some to us yet unknown reason (haven't had time to look into that yet).

tagging @leodotcloud into this ticket. Expert sleuthing @kaos

@kaos that seems to work. I have to play "catch" since rancher is constantly changing the pid since it restarts the cni service and, once I'm fast enough with setting the pid, ipsec starts rolling. With so many servers though (and with many of them being restarted through the day) that's quite a lot to manage so I'll try to deploy a container that constantly monitors the pids and does just that...at lest until rancher devs fix this.

Indeed, if I was digging in the right place, the plugin-manager updates stuff every 5 minutes, or when triggered by a change in the metadata.. I'm guess it's in there some where something's broken.

Here's my "working" code to fix this:
import time
import subprocess

bridge_tpl = """#!/bin/sh
exec /usr/bin/nsenter -m -u -i -n -p -t {} -- \$0 \"\$@\"
"""

def cmd(command):
    try:
        return subprocess.check_output(command, shell=True).decode('utf-8').strip()
    except subprocess.CalledProcessError:
        return ''

def watch():
    ipsec_pid = cmd("docker ps --filter \"name=ipsec-cni\" --format {{.ID}} | xargs docker inspect  --format '{{.State.Pid}}'").strip()

    if ipsec_pid:
        bridge_contents = cmd("cat /var/lib/docker/volumes/rancher-cni-driver/_data/bin/rancher-bridge")

        if ipsec_pid not in bridge_contents:
            cmd('truncate -s0 /var/lib/docker/volumes/rancher-cni-driver/_data/bin/rancher-bridge')

            code = bridge_tpl.format(int(ipsec_pid))

            cmd('echo "%s" >> /var/lib/docker/volumes/rancher-cni-driver/_data/bin/rancher-bridge' % code)

if __name__ == "__main__":
    while True:
        watch()
        time.sleep(1)

This one runs inside a container that has the following volumes:
/usr/bin/docker:/usr/bin/docker
/var/run/docker.sock:/var/run/docker.sock
/var/log/:/var/log/

It constantly monitors the pid and nsenter file for a mismatch. Not the prettiest one but it works for now.

@ciokan can you ping me on our users slack? I'd gladly troubleshoot this with you

Let me explain how things work so maybe people can troubleshoot this better.  The "network-manager" container looks at metadata and looks for services with kind "networkDriverService" and then look for local container for that service.  Once it find the container it writes a small nsenter shell script that works as the CNI plugin.   The code is here, it's pretty straight forward https://github.com/rancher/plugin-manager/blob/master/binexec/watcher.go.   So a couple things that could go wrong

Metadata isn't working: Look at the docker logs output of metadata and see if anything is fishy
Metadata has the wrong info in it: Go into the network-manager container and run curl -H "Accept: application/json " http://<IP_ADDRESS>/latest.  Then send me the output<EMAIL_ADDRESS>
I'm guessing there could be some issue if there is some crufty data around that is causing issues.  Sorry this issue went unnoticed for so long, the couple weeks before and after DockerCon are pretty hectic for us.  Thanks!

Any eta on this? We're seeing this in production too, post upgrade the entire environment goes down. Nothing to see in the metadata logs, but healthcheck seems to not work either.

@lkm please try the image I linked above. I have it working in production with 100+ servers. Until an official fix, this is the only thing that worked. It incorporates what @kaos provided earlier.

@ciokan @kaos @lkm @spawnthink @asg1612 @OneB1t @maltyxx @snipking @fedya @tpgxyz @Emil983
Can you please ping me on https://slack.rancher.io so that I can take a look at your setup and debug what's going on?
Also could you please collect the logs and share them with me?
https://github.com/leodotcloud/rancher-logs-collector ==> this will make it easier to collect the logs.

We have the same issue and will help with logs if wanted

@ingardm Thank you. The logs might be more than the allowed limit of email attachment size. Please share them with me on slack.

@leodotcloud which logs are you after specifically? The logs of the r-ipsec-ipsec containers are empty

@ingardm Please use https://github.com/leodotcloud/rancher-logs-collector to collect the logs :)

@leodotcloud @superseb Aight. I've got the logs collected from the environment having problems. Please contact me on slack for details

Thanks for all the digging and fixes, we need to get to the bottom of this. Does anyone have the steps to reproduce please, I think there have been some log digging and troubleshooting on/offline but there is no reproducible plan yet. I've tried multiple big clusters running Debian/Ubuntu, rebooting/resetting nodes, host networked/managed, killing/restarting infra containers.

@superseb I can't provide any steps to reproduce it (if I knew exactly what's the cause of it I would probably avoid it entirely) but I can tell you that it's affecting 95% of all our hosts/servers. I can provide a share screen or even ssh into a non working host if you want to take a look at it.

@superseb  Any news on this ? Whole orchestration on rancher is not working.
healthcheck, ipsec are failing.
healthcheck (Expected state running but got error: Couldn't bring up network: failed to find plugin "rancher-bridge" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin])

@tpgxyz Not yet, working to reproduce. Will update soon.
@ciokan Ping me on Rancher Users Slack, would love to check at the hosts to locate the problem and get this out of the way for a lot of users.

@superseb Can you please provide step-by-step procedure how to get all the needed logs ?

@tpgxyz Logging can be collected using: https://github.com/leodotcloud/rancher-logs-collector
As we are trying to reproduce, I would love to find the common denominator in this case. So if you could post the following:
Current Rancher version:
Upgraded from Rancher version:
Infrastructure rancher/network-manager version:
Docker version: (`docker version`,`docker info` preferred)
Operating system and kernel: (`cat /etc/os-release`, `uname -r` preferred)
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)

Basically we are running rancher to manage de-centralized RPM build system around globe. Each machine is running different operating system (fedora, centos, openmandriva etc.)
We had to disable rancher, due this bug, on all machines and run all the needed docker by hand.
Below you can find information about system where rancher server is running:
Current Rancher version: 1.6.0
Upgraded from Rancher version: 1.5.5
Infrastructure rancher/network-manager version:
Docker version: (docker version,docker info preferred)
abf.openmandriva.org@root[0]:[~] # docker version
Client:
Version:      1.10.3
API version:  1.22
Go version:   go1.5.1
Git commit:   c3959b1
Built:
OS/Arch:      linux/amd64
Server:
Version:      1.10.3
API version:  1.22
Go version:   go1.5.1
Git commit:   c3959b1
Built:
OS/Arch:      linux/amd64
Operating system and kernel: (cat /etc/os-release, uname -r preferred)
abf.openmandriva.org@root[0]:[~] # cat /etc/os-release
NAME="OpenMandriva Lx"
VERSION="2014.0 (Phosphorus)"
ID="openmandriva_association"
VERSION_ID=2014.0
BUILD_ID=20160709.21
PRETTY_NAME="OpenMandriva Lx 2014.0 (Phosphorus)"
ANSI_COLOR="1;43"
CPE_NAME="cpe:/o:openmandriva_association:openmandriva_lx:2014.0"
HOME_URL="http://openmandriva.org/"
BUG_REPORT_URL="https://issues.openmandriva.org/
abf.openmandriva.org@root[0]:[~] # uname -r
4.9.0-server-1omv
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
Bare-metal
I've set up CLI and run that script, well there's nothing interesing inside:
rancher_host.log and rancher_ps_c_a.log are empty - wtf ?
[tpg@lazur rancher.Qz1X]$ cat rancher_ps_s_a.log
ID        TYPE                   NAME                               IMAGE                            STATE       SCALE     SYSTEM    ENDPOINTS   DETAIL
1s21      service                healthcheck/healthcheck            rancher/healthcheck:v0.2.3       inactive    0/1       true
1s22      service                network-services/network-manager   rancher/network-manager:v0.7.0   unhealthy   0/1       true
1s23      service                network-services/metadata          rancher/metadata:v0.9.1          unhealthy   0/1       true
1s23      sidekick               network-services/metadata/dns      rancher/dns:v0.15.0              unhealthy   0/1       true
1s24      networkDriverService   ipsec/ipsec                        rancher/net:holder               inactive    0/1       true
1s24      sidekick               ipsec/ipsec/router                 rancher/net:v0.11.2              inactive    0/1       true
1s24      sidekick               ipsec/ipsec/cni-driver             rancher/net:v0.11.2              inactive    0/1       true
1s25      service                scheduler/scheduler                rancher/scheduler:v0.7.5         inactive    0/1       true

@superseb I have a fresh environment with this problem, and I can provide full access for debug. 3 host nodes in HA, each running a MySQL container (1 master, 2 slave), each running a Rancher/Server container, each added as a host within Rancher.

Current Rancher version: 1.6.0
Upgraded from Rancher version: none (fresh install)
Infrastructure rancher/network-manager version: v0.7.0
Docker version: 17.03.1-ce, build c6d412e
Operating system and kernel: (cat /etc/os-release, uname -r preferred)[rancher@rancher-ha1 ~]$ cat /etc/os-release
NAME="RancherOS"
VERSION=v1.0.1
ID=rancheros
ID_LIKE=
VERSION_ID=v1.0.1
PRETTY_NAME="RancherOS v1.0.1"
HOME_URL="http://rancher.com/rancher-os/"
SUPPORT_URL="https://forums.rancher.com/c/rancher-os"
BUG_REPORT_URL="https://github.com/rancher/os/issues"
BUILD_ID=
[rancher@rancher-ha1 ~]$ uname -r
4.9.24-rancher

Type/provider of hosts: Vultr (DO)

Steps to reproduce:

Provision 3 nodes with iPXE script
On one node, set MySQL master via /home/rancher/mysql/config/master.cnf[mysqld]
bind-address = <IP_ADDRESS>
server-id = 1

log_bin = /var/log/mysql/mysql-bin.log
binlog_do_db = rancher

On remaining 2 nodes, set MySQL slave via /home/rancher/mysql/config/slave.cnf[mysqld]
bind-address = <IP_ADDRESS>
server-id = 2 # or 3

log_bin = /var/log/mysql/mysql-bin.log
binlog_do_db = rancher

Start MySQL containers on each node with the following script:docker run -td --restart=unless-stopped \
  -v /home/rancher/mysql/config:/etc/mysql/conf.d \
  -v /home/rancher/mysql/data:/var/lib/mysql \
  -p 3306:3306 \
  -e MYSQL_DATABASE=rancher \
  -e MYSQL_USER=rancher \
  -e MYSQL_PASSWORD=userpass \
  -e MYSQL_INITDB_SKIP_TZINFO=yes \
  -e MYSQL_ROOT_PASSWORD=rootpass \
  --name rancher-mysql \
  mysql:5.7

On MySQL master node: set master modedocker exec -it rancher-mysql mysql -u root -prootpass
GRANT REPLICATION SLAVE ON *.* TO 'slave_user'@'%' IDENTIFIED BY 'userpass';
FLUSH PRIVILEGES;
USE rancher;
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;

+------------------+----------+--------------+------------------+-------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000004 |      595 | rancher      |                  |                   |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

UNLOCK TABLES;
QUIT;

On MySQL slave nodes: set slave mode (using master IP address)docker exec -it rancher-mysql mysql -u root -prootpass
CHANGE MASTER TO MASTER_HOST='12.34.567.890',MASTER_USER='slave_user', MASTER_PASSWORD='userpass', MASTER_LOG_FILE='mysql-bin.000004', MASTER_LOG_POS=595;
START SLAVE;
SHOW SLAVE STATUS\G

Start Rancher on first node (MySQL master)# Ensure db-host is IP of master mysql node
docker run -td --restart=unless-stopped \
  -p 8080:8080 \
  -p 9345:9345 \
  --name rancher-server \
  rancher/server:v1.6.0 \
  --db-host 12.34.567.890 --db-port 3306 --db-user rancher --db-name rancher --db-pass userpass

Start Rancher on nodes 2 & 3 (repeat last step)
Watch IPSEC/Healthcheck chaos 🍿

Just finished a debug session with @superseb (super indeed!) and he managed to identify that one of the host nodes was reporting the docker0 bridge network as its IP address. This is obviously incorrect, as the docker0 bridge is internal and does not represent the external IP address.
We analyzed the HA 4-host stack as it sat on Rancher 1.5.1, then tore it down and built it back up on Rancher 1.6.0. This time, each host was added with a specified external IP address. Stability has been achieved.
I'm not convinced that this issue thread is related to my problem, but I did experience the symptoms described here. Either way, it might not hurt to confirm that your hosts are reporting the correct IP address.
Many thanks for your time and effort, @superseb!

This issue started showing up in our environment since last week. We recently upgraded from Rancher 1.5.3 to 1.5.9. Luckily, most environments are created correctly, however some experience the Timeout getting IP address issue.
We are running Rancheros 1.0.1 on VMware. I will try to find some time later today to debug this further.

@superseb  or anyone can resolve this issue ?

Found this
id": "1i33627",
"type": "container",
"links": {
"self": "…/v2-beta/projects/1a5/containers/1i33627",
"account": "…/v2-beta/projects/1a5/containers/1i33627/account",
"credentials": "…/v2-beta/projects/1a5/containers/1i33627/credentials",
"healthcheckInstanceHostMaps": "…/v2-beta/projects/1a5/containers/1i33627/healthcheckinstancehostmaps",
"hosts": "…/v2-beta/projects/1a5/containers/1i33627/hosts",
"instanceLabels": "…/v2-beta/projects/1a5/containers/1i33627/instancelabels",
"instanceLinks": "…/v2-beta/projects/1a5/containers/1i33627/instancelinks",
"instances": "…/v2-beta/projects/1a5/containers/1i33627/instances",
"mounts": "…/v2-beta/projects/1a5/containers/1i33627/mounts",
"ports": "…/v2-beta/projects/1a5/containers/1i33627/ports",
"serviceEvents": "…/v2-beta/projects/1a5/containers/1i33627/serviceevents",
"serviceExposeMaps": "…/v2-beta/projects/1a5/containers/1i33627/serviceexposemaps",
"serviceLogs": "…/v2-beta/projects/1a5/containers/1i33627/servicelogs",
"services": "…/v2-beta/projects/1a5/containers/1i33627/services",
"targetInstanceLinks": "…/v2-beta/projects/1a5/containers/1i33627/targetinstancelinks",
"volumes": "…/v2-beta/projects/1a5/containers/1i33627/volumes",
"stats": "…/v2-beta/projects/1a5/containers/1i33627/stats",
"containerStats": "…/v2-beta/projects/1a5/containers/1i33627/containerstats"
},
"actions": {
"remove": "…/v2-beta/projects/1a5/containers/1i33627/?action=remove"
},
"baseType": "instance",
"name": "network-services-network-manager-3",
"state": "error",
"accountId": "1a5",
"blkioDeviceOptions": null,
"blkioWeight": null,
"build": null,
"capAdd": null,
"capDrop": null,
"cgroupParent": null,
"command": [ 3 items
"plugin-manager",
"--metadata-url",
"http://<IP_ADDRESS>/2016-07-29"
],
"cpuCount": null,
"cpuPercent": null,
"cpuPeriod": null,
"cpuQuota": null,
"cpuSet": null,
"cpuSetMems": null,
"cpuShares": null,
"createIndex": 5,
"created": "2017-05-24T19:05:46Z",
"createdTS":<PHONE_NUMBER>000,
"dataVolumeMounts": { },
"dataVolumes": [ 8 items
"/var/run/docker.sock:/var/run/docker.sock",
"/var/lib/docker:/var/lib/docker",
"/var/lib/rancher/state:/var/lib/rancher/state",
"/lib/modules:/lib/modules:ro",
"/run:/run:ro",
"/var/run:/var/run:ro",
"rancher-cni-driver:/etc/cni",
"rancher-cni-driver:/opt/cni"
],
"dataVolumesFrom": null,
"deploymentUnitUuid": "ae43589e-e1cd-4c88-bc83-e346a3b2bad5",
"description": null,
"devices": null,
"diskQuota": null,
"dns": null,
"dnsOpt": null,
"dnsSearch": [ 2 items
"network-services.rancher.internal",
"network-manager.network-services.rancher.internal"
],
"domainName": null,
"entryPoint": null,
"environment": {
"DOCKER_BRIDGE": "docker0",
"METADATA_IP": "<IP_ADDRESS>"
},
"expose": null,
"externalId": "",
"extraHosts": null,
"firstRunning": null,
"groupAdd": null,
"healthCheck": null,
"healthCmd": null,
"healthInterval": null,
"healthRetries": null,
"healthState": null,
"healthTimeout": null,
"hostId": "1h4",
"hostname": null,
"imageUuid": "docker:rancher/network-manager:v0.7.0",
"instanceTriggeredStop": "stop",
"ioMaximumBandwidth": null,
"ioMaximumIOps": null,
"ip": null,
"ip6": null,
"ipcMode": null,
"isolation": null,
"kernelMemory": null,
"kind": "container",
"labels": {
"io.rancher.service.deployment.unit": "ae43589e-e1cd-4c88-bc83-e346a3b2bad5",
"io.rancher.service.requested.host.id": "4",
"io.rancher.stack_service.name": "network-services/network-manager",
"io.rancher.service.launch.config": "io.rancher.service.primary.launch.config",
"io.rancher.project.name": "network-services",
"io.rancher.project_service.name": "network-services/network-manager",
"io.rancher.stack.name": "network-services",
"io.rancher.service.hash": "f3bf3137ed558867efb7108874cd42cf5baea9a3",
"io.rancher.scheduler.global": "true",
"io.rancher.container.system": "true"
},
"logConfig": {
"type": "logConfig",
"config": {
"max-file": "2",
"max-size": "25m"
},
"driver": "json-file"
},
"lxcConf": null,
"memory": null,
"memoryReservation": null,
"memorySwap": null,
"memorySwappiness": null,
"milliCpuReservation": null,
"mounts": null,
"nativeContainer": false,
"netAlias": null,
"networkContainerId": null,
"networkMode": "host",
"oomKillDisable": null,
"oomScoreAdj": null,
"pidMode": "host",
"pidsLimit": null,
"ports": null,
"primaryIpAddress": null,
"primaryNetworkId": "1n1",
"privileged": true,
"publishAllPorts": false,
"readOnly": false,
"registryCredentialId": null,
"removed": null,
"requestedHostId": "1h4",
"restartPolicy": null,
"secrets": null,
"securityOpt": null,
"serviceId": null,
"serviceIds": [
"1s30"
],
"shmSize": null,
"stackId": null,
"startCount": 0,
"startOnCreate": true,
"stdinOpen": false,
"stopSignal": null,
"storageOpt": null,
"sysctls": null,
"system": true,
"tmpfs": null,
"transitioning": "error",
"transitioningMessage": "Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/var/lib/docker/containers/f2e9fb35fdcd937a51839746cae0542a80632d120c8892061863e41c344a1814/secrets\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/overlay/50978fadaf02d2e29421737ee02290920d62e56904c240bab5e824f44b360077/merged\\\\\\\" at \\\\\\\"/var/lib/docker/overlay/50978fadaf02d2e29421737ee02290920d62e56904c240bab5e824f44b360077/merged/run/secrets\\\\\\\" caused \\\\\\\"mkdir /var/lib/docker/overlay/50978fadaf02d2e29421737ee02290920d62e56904c240bab5e824f44b360077/merged/run/secrets: read-only file system\\\\\\\"\\\"\"\n"",
"transitioningProgress": 84,
"tty": false,
"ulimits": null,
"user": null,
"userPorts": null,
"usernsMode": null,
"uts": null,
"uuid": "850e5929-05ad-449d-b27e-f385c5f9d62d",
"version": "0",
"volumeDriver": null,
"workingDir": null
}

Again output form healtcheck
{
"id": "1i36522",
"type": "container",
"links": {
    "self": "…/v2-beta/projects/1a5/containers/1i36522",
    "account": "…/v2-beta/projects/1a5/containers/1i36522/account",
    "credentials": "…/v2-beta/projects/1a5/containers/1i36522/credentials",
    "healthcheckInstanceHostMaps": "…/v2-beta/projects/1a5/containers/1i36522/healthcheckinstancehostmaps",
    "hosts": "…/v2-beta/projects/1a5/containers/1i36522/hosts",
    "instanceLabels": "…/v2-beta/projects/1a5/containers/1i36522/instancelabels",
    "instanceLinks": "…/v2-beta/projects/1a5/containers/1i36522/instancelinks",
    "instances": "…/v2-beta/projects/1a5/containers/1i36522/instances",
    "mounts": "…/v2-beta/projects/1a5/containers/1i36522/mounts",
    "ports": "…/v2-beta/projects/1a5/containers/1i36522/ports",
    "serviceEvents": "…/v2-beta/projects/1a5/containers/1i36522/serviceevents",
    "serviceExposeMaps": "…/v2-beta/projects/1a5/containers/1i36522/serviceexposemaps",
    "serviceLogs": "…/v2-beta/projects/1a5/containers/1i36522/servicelogs",
    "services": "…/v2-beta/projects/1a5/containers/1i36522/services",
    "targetInstanceLinks": "…/v2-beta/projects/1a5/containers/1i36522/targetinstancelinks",
    "volumes": "…/v2-beta/projects/1a5/containers/1i36522/volumes",
    "stats": "…/v2-beta/projects/1a5/containers/1i36522/stats",
    "containerStats": "…/v2-beta/projects/1a5/containers/1i36522/containerstats"
},
"actions": {
    "remove": "…/v2-beta/projects/1a5/containers/1i36522/?action=remove"
},
"baseType": "instance",
"name": "healthcheck-healthcheck-2",
"state": "error",
"accountId": "1a5",
"blkioDeviceOptions": null,
"blkioWeight": null,
"build": null,
"capAdd": null,
"capDrop": null,
"cgroupParent": null,
"command": null,
"cpuCount": null,
"cpuPercent": null,
"cpuPeriod": null,
"cpuQuota": null,
"cpuSet": null,
"cpuSetMems": null,
"cpuShares": null,
"createIndex": 700,
"created": "2017-05-29T08:08:00Z",
"createdTS":<PHONE_NUMBER>000,
"dataVolumeMounts": { },
"dataVolumes": [
    "/var/lib/rancher/etc:/var/lib/rancher/etc:ro"
],
"dataVolumesFrom": null,
"deploymentUnitUuid": "9ebfbcef-5316-48cd-a544-e0755259f9f1",
"description": null,
"devices": null,
"diskQuota": null,
"dns": [
    "<IP_ADDRESS>"
],
"dnsOpt": null,
"dnsSearch": [ 3 items
    "healthcheck.rancher.internal",
    "healthcheck.healthcheck.rancher.internal",
    "rancher.internal"
],
"dockerPorts": [ ],
"domainName": null,
"entryPoint": null,
"environment": null,
"expose": null,
"externalId": "97d2206b6778fe1d71ae0eb64deabf15b2ed289f67293f09ea2acd48afd6a004",
"extraHosts": null,
"firstRunning": null,
"groupAdd": null,
"healthCheck": {
    "type": "instanceHealthCheck",
    "healthyThreshold": 2,
    "interval": 2000,
    "port": 42,
    "responseTimeout": 2000,
    "strategy": "recreate",
    "unhealthyThreshold": 3
},
"healthCmd": null,
"healthInterval": null,
"healthRetries": null,
"healthState": null,
"healthTimeout": null,
"hostId": "1h7",
"hostname": null,
"imageUuid": "docker:rancher/healthcheck:v0.3.1",
"instanceTriggeredStop": "stop",
"internalVolumes": [ ],
"ioMaximumBandwidth": null,
"ioMaximumIOps": null,
"ip": null,
"ip6": null,
"ipcMode": null,
"isolation": null,
"kernelMemory": null,
"kind": "container",
"labels": {
    "io.rancher.service.deployment.unit": "9ebfbcef-5316-48cd-a544-e0755259f9f1",
    "io.rancher.service.requested.host.id": "7",
    "io.rancher.stack_service.name": "healthcheck/healthcheck",
    "io.rancher.service.launch.config": "io.rancher.service.primary.launch.config",
    "io.rancher.project.name": "healthcheck",
    "io.rancher.project_service.name": "healthcheck/healthcheck",
    "io.rancher.container.create_agent": "true",
    "io.rancher.stack.name": "healthcheck",
    "io.rancher.service.hash": "7f8d78bdc5e48b52d1821bdfa68b774c0b74d456",
    "io.rancher.scheduler.global": "true",
    "io.rancher.container.system": "true"
},
"logConfig": {
    "type": "logConfig",
    "config": {
        "max-file": "2",
        "max-size": "25m"
    },
    "driver": "json-file"
},
"lxcConf": null,
"memory": null,
"memoryReservation": null,
"memorySwap": null,
"memorySwappiness": null,
"milliCpuReservation": null,
"mounts": null,
"nativeContainer": false,
"netAlias": null,
"networkContainerId": null,
"networkMode": "managed",
"oomKillDisable": null,
"oomScoreAdj": null,
"pidMode": null,
"pidsLimit": null,
"ports": null,
"primaryIpAddress": null,
"primaryNetworkId": "1n5",
"privileged": false,
"publishAllPorts": false,
"readOnly": false,
"registryCredentialId": null,
"removed": null,
"requestedHostId": "1h7",
"restartPolicy": null,
"secrets": null,
"securityOpt": null,
"serviceId": null,
"serviceIds": [
    "1s26"
],
"shmSize": null,
"stackId": null,
"startCount": 0,
"startOnCreate": true,
"stdinOpen": false,
"stopSignal": null,
"storageOpt": null,
"sysctls": null,
"system": true,
"tmpfs": null,
"transitioning": "error",
"transitioningMessage": "Couldn't bring up network: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]",
"transitioningProgress": 84,
"tty": false,
"ulimits": null,
"user": null,
"userPorts": null,
"usernsMode": null,
"uts": null,
"uuid": "ef09b71e-7326-4e7a-b1ca-089015765387",
"version": "0",
"volumeDriver": null,
"workingDir": null

}

time="2017-05-31T11:44:48Z" level=error msg="Failed to evaluate networking: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"
Issue still not resolved.

Had the time to look around @ciokan systems, thanks a lot for that. Great to get an insight into the actual running problem. So as all the logs hint to a network problem, the root cause seems to be metadata returning inconsistent results. So quering <IP_ADDRESS>/latest/self/host/uuid would return different results, and this output is actually used to setup networking. We have a lead and are investigating why this is happening.

@superseb do you have any plans to fix it? Is it fixed in 1.6.1?

@superseb @maciekciolek Hi, looks like 1.6.1 did not fixed this issue.

As long as this is open we are working on it, if a fix is there we will definitely share it here to let people try it. Let me get you an update today on our investigation. In the meantime, if you can confirm you are seeing the same issue as described in https://github.com/rancher/rancher/issues/8383#issuecomment-305400907 that would be great.

@superseb  How to confirm #8383 (comment) ? Can you please share with me a step-by-step procedure ?

I face the same issue

@maciekciolek @tpgxyz On the host experiencing issues, you can Execute shell on a container (i.e. network-manager or healthcheck) and execute curl http://<IP_ADDRESS>/latest/self/host/uuid repeatedly. This would result in random results.

@superseb I am getting same results while executing this repeatedly. The only fix which works for me is this script: https://github.com/rancher/rancher/issues/8383#issuecomment-298100234

@superseb @tpgxyz I have just solved the problem on our environment.
In our case the problem was, that we have 3 NICs per host and when I was starting the agent, it was auto-detecting wrong agent host IP (IP address of local docker network) which was not accessible from server and the other hosts. The solution which works for me is just providing IP address in 4th step of "Add host" which is accessible from server and the other hosts.
In other words, you need to manually provide CATTLE_AGENT_IP env variable while you are starting an agent. It also means that you need to do it event for host (agent) which will be run on the same host as the server.
sudo docker run -e CATTLE_AGENT_IP="<IP_ADDRESS>"  --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.2 http://<IP_ADDRESS>:7000/v1/scripts/***
After this fix ipsec, healthcheck, cni erros disappear.
Tested on: 1.5.1 and 1.5.10

@maciekciolek Many thanks! It works now.
@fedya \o/

It works on 1.6.2 as well :1st_place_medal:

@maciekciolek Where is that CATTLE_AGENT_IP supposed to point when having multiple servers? The rancher ip address?

It is already stable branch?

@ciokan as far as I understand you have one (or more) rancher servers S{x} and many instances on which you want to run rancher agents (worker) - let`s say  I1, I2, I3. If you are adding new host (rancher agent / worker), which will run on instance I1 you should provided the value of CATTLE_AGENT_IP which is the IP address of the instance I1. If you are adding host which runs on I2, then use the IP address of I2.
The most important thing is, that value of CATTLE_AGENT_IP should be IP of an interface, which is accessible (eg. same network) by other instances and servers.

@maciekciolek i'm right understood it that CATTLE_AGENT_IP must be server IP not docker0 interface?

@fedya CATTLE_AGENT_IP must be the IP of agent which you are starting. The value will be propagated by this agent to the server and other agents and will be used by them to communicate with this agent. So, if you have multiple host in the local network, the value of  should be the IP of host on which you are starting an agent.
CATTLE_AGENT_IP is not server IP. The server IP is attached in starting command, eg.:
sudo docker run -e CATTLE_AGENT_IP="<IP_ADDRESS>" --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.2 http://<IP_ADDRESS>:7000/v1/scripts/**
<IP_ADDRESS> - IP of host on which I am starting the agent
<IP_ADDRESS>:7000 - IP and port of Rancher server

Well.
how i run node
docker run --rm -e CATTLE_AGENT_IP="<IP_ADDRESS>" --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.2 http://abf.openmandriva.org:8080/v1/scripts/xxxxxxx
CATTLE_AGENT_IP = ip from my node
And i got same results as above in this thread
nsenter: failed to execute /opt/cni/bin/rancher-bridge: No such file or directory
time="2017-06-26T14:05:46Z" level=error msg="Failed to evaluate networking: netplugin failed but error parsing its diagnostic message \"\": unexpected end of JSON input"
time="2017-06-26T14:05:47Z" level=info msg="Evaluating state from retry" cid=63e6008665a6b764541593ecfa9fba465674b8c420714e39752d83fe7392de75 count=11
time="2017-06-26T14:05:47Z" level=info msg="CNI up" cid=63e6008665a6b764541593ecfa9fba465674b8c420714e39752d83fe7392de75 networkMode=ipsec
nsenter: failed to execute /opt/cni/bin/rancher-bridge: No such file or directory
time="2017-06-26T14:05:47Z" level=error msg="Failed to evaluate networking: netplugin failed but error parsing its diagnostic message \"\": unexpected end of JSON input"

Metadata returning inconsistent results (exposed ports of container).
Rancher v1.6.2
Network Services v0.2.2
root@92e5bd7ca9cf:/# curl rancher-metadata/2015-12-19/stacks/lb-test/serv
containers/0/ports
root@92e5bd7ca9cf:/# curl rancher-metadata/2015-12-19/stacks/lb-test/services/1/
containers/0/ports
0

yes, still not working
2017-07-10 16:17 GMT+03:00 Jan B<EMAIL_ADDRESS>
Metadata returning inconsistent results (exposed ports of container).
Rancher v1.6.2
Network Services v0.2.2
root@92e5bd7ca9cf:/# curl rancher-metadata/2015-12-19/stacks/lb-test/serv
containers/0/ports
root@92e5bd7ca9cf:/# curl rancher-metadata/2015-12-19/stacks/lb-test/services/1/
containers/0/ports
0
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rancher/rancher/issues/8383#issuecomment-314102316,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAkhzQOxEIfWQidxL5Uom4gBrFKRRf10ks5sMiRygaJpZM4MwWTE
.

Is there any progress from the dev team regarding this issue? It has almost 4 months and 71 comments so far. I keep delaying my team from going fully with rancher due to this bug.

We are too. Very annoying issue.
Btw i tested it with stable CoreOS, same bug.
2017-07-10 21:52 GMT+03:00 Romeo Mihalcea<EMAIL_ADDRESS>
Is there any progress from the dev team regarding this issue? It has
almost 4 months and 71 comments so far. I keep delaying my team from going
fully with rancher due to this bug.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rancher/rancher/issues/8383#issuecomment-314200405,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAkhzV1bGpAytoTkvM6ewgQ5pxsqM1tpks5sMnL2gaJpZM4MwWTE
.

Setting CATTLE_AGENT_IP = ip from my node works for me. I've tested on the cloud not local.

i tried with CATTLE_AGENT_IP no success
2017-07-11 1:01 GMT+03:00 Bruno<EMAIL_ADDRESS>
Setting CATTLE_AGENT_IP = ip from my node works for me. I've tested on
the cloud not local.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rancher/rancher/issues/8383#issuecomment-314262602,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAkhzWo1rDjD6Wf0-MpSq_iqg2wNbYjeks5sMp9OgaJpZM4MwWTE
.

@fedya
Try with the latest version of the agent: rancher/agent:v1.2.5

still no success
docker run --rm --privileged -e CATTLE_AGENT_IP=rancher_server_ip_here -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.5 http://rancher_server_ip_here:8080/v1/scripts/KEY
 Time:1499874158, TimeNano:1499874158521952432}. Error: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"
time="2017-07-12T15:42:40Z" level=info msg="Evaluating state from retry" cid=eadbd8bf8bf74677e15e2d7b6d04d744e841ec3cc94ee421fa8291820cfba211 count=10
time="2017-07-12T15:42:40Z" level=info msg="CNI up" cid=eadbd8bf8bf74677e15e2d7b6d04d744e841ec3cc94ee421fa8291820cfba211 networkMode=ipsec
time="2017-07-12T15:42:40Z" level=error msg="Failed to evaluate networking: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"

Which docker version are you using on your rancher server and agents?

1.12.6
On Wed, Jul 12, 2017 at 6:56 PM, Bruno<EMAIL_ADDRESS>wrote:

Which docker version are you using on your rancher server and agents?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rancher/rancher/issues/8383#issuecomment-314814679,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAc_RHZbTWtUaaHQdokLidLzB_R41CAfks5sNOzIgaJpZM4MwWTE
.

Are agents able to ping the rancher server? Are you using a firewall?

@fedya: in your commannd.. "CATTLE_AGENT_IP=rancher_server_ip_here" .. that doesn't sound right. CATTLE_AGENT_IP should be IP of the docker host, not the IP of the rancher server .. or did I miss something?

@pasikarkkainen yes I tried with the ip of the client running the agent. Same error: Couldn't bring up network: failed to find plugin "rancher-bridge" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]. Docker version doesn't matter as I see the same error on different hosts running the same engine version. Rancher server picks up the correct agent ip address so I don't believe CATTLE_AGENT_IP has anything to do here (tried and tested). I managed to repair 80% of the hosts using my above image but the rest of them refuse to work properly even with it.
I wouldn't be bothered with this bug as it's no affecting my services but, with it present, my rancher server stays at 100% on all cores (8 of them). If I deactivate the clients with the bug present my rancher server CPU goes down to normal 10% - 20%. With the clients deactivated in rancher server services work ok but, if a client gets restarted, they don't start any more.
This is the second "most discussed" ticket on this project and we still don't have a clue, roadmap, timeline or official answer regarding this bug. Everyone involved here provided everything requested, I even collaborated with the dev team and provided SSH access to multiple boxes including rancher server. The culprit seems to be tracked but after that...silence.
We need an official answer as I don't have a single clue regarding how much we're supposed to keep waiting. If it's 1 year I can re-route my team and build something custom. If it's 1-2 months we can probably wait but the lack of feedback generates a lot of frustration within our office and, I'm sure, other teams affected by this thing.
I can see over 1700 tickets being open which must be overwhelming. I'm happy to pay for support knowing this will get the bug fixed. I'm sure others are holding cash in hand for this.
TL; DR: Give us something. We're barking at trees here.

@ciokan @fedya and everyone who are having this problem, can you please ping me on https://slack.rancher.io? I would like to schedule a debug session with you guys. I work in PST timezone so please give me a couple of time slots that work for both us. @superseb @janeczku Can you please coordinate too?
We have not been able to reproduce this issue on our end in a reliable way, hence the delay in root causing the issue. Our QA team puts in a lot of effort testing various scenarios on different operating systems and also different rancher versions. Our team also test rancher/server upgrade scenarios as well. Along with these efforts, we also have a bunch on tag automation runs triggered on a daily basis. In the past few months we have never been able to hit this issue.
TL;DR: We are not ignoring anyone or this issue and would like to fix it if we can find a way to reproduce it.

To avoid confusion regarding CATTLE_AGENT_IP, this has to be set to the IP address of the host where the command is being run. And also this IP address should be reachable by all other nodes in the cluster.
For example, in AWS if you have two hosts in two different VPCs and you have used the private IP addresses of these hosts to register them with rancher/server you might have a problem if inter VPC communication is not configured properly.
Setting this variable is not needed for all environments but only on setups/clouds where the auto IP detection doesn't work due to the cloud provider/network topology limitations.

Okay, i will do.
My timezone is MSK, but usually i'm online for 14-17 hours per day
12 июля 2017 г. 21:22 пользователь "leodotcloud"<EMAIL_ADDRESS>написал:

To avoid confusion regarding CATTLE_AGENT_IP, this has to be set to the IP
address of the host where the command is being run. And also this IP
address should be reachable by all other nodes in the cluster.
For example, in AWS if you have two hosts in two different VPCs and you
have used the private IP addresses of these hosts to register them with
rancher/server you might have a problem if inter VPC communication is not
configured properly.
Setting this variable is not needed for all environments but only on
setups/clouds where the auto IP detection doesn't work due to the cloud
provider/network topology limitations.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rancher/rancher/issues/8383#issuecomment-314854952,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAkhzRGprRte_v_PifwDlcMBxxyCuovMks5sNQ7wgaJpZM4MwWTE
.

woah...after all this time...my stack is green. I'll have to thank @leodotcloud and @superseb for taking the time to get to the bottom of it. From the looks of it (leo might step in with more updates) the docker daemon was running on a different ip address than expected, an old one actually. They managed to fix my setup, I can't give a fix right now because I have bits of it but they fixed about 12 - 15 machines and I'm sure the team now has an idea and can push a fix soon. It' def a hard thing to reproduce in their testing setup that's why the long wait over here.
Thank you guys!

@ciokan Thank you for taking time to share your setup for debugging.
The error message: Failed to evaluate networking: failed to find plugin \"rancher-bridge\" could be due to a variety of problems. One of the reasons is as below:
Summary:
TL; DR version: If the IP address of the docker0 bridge on any host is same as the IP address assigned to any container, especially metadata container, in the cluster this problem surfaces.
Long version:
All of Rancher's infra microservices/containers running on any host of the cluster rely on metadata to fetch the information needed to do their jobs. rancher-metadata answers the queries in an intelligent fashion, giving out information based on the source IP address of the request. So <IP_ADDRESS>/latest/self/container/ will yield different answers for various containers. If the source IP address lookup fails in rancher-metadata only parts of the information which are common to all the containers on that host are available while querying. Ex: Only <IP_ADDRESS>/latest/self/host is available but not <IP_ADDRESS>/latest/self/container as we cannot determine from where the request is coming from.
When a new host is added to rancher, the first microservice to start would be rancher-metadata along with it's sidekick rancher-dns. This microservice is on docker's bridge network and gets an IP address from the subnet configured for docker. Example: If the bridge subnet is <IP_ADDRESS>/16, an IP address in the range <IP_ADDRESS> to <IP_ADDRESS> is allocated for this container. In general the first IP address of this container, i.e., <IP_ADDRESS> is configured on the bridge, rancher-metadata generally/most of the times gets <IP_ADDRESS>. While debugging @ciokan's setup, we noticed that some of the rancher-metadata containers in the cluster had an IP address of <IP_ADDRESS>, which was quite unusual and upon further investigation we found docker0 bridge had an IP: <IP_ADDRESS>.
Now let's understand how network-manager works. The microservice network-manager is responsible for setting up the CNI executable, CNI config so that IPSec stack can start properly. network-manager queries <IP_ADDRESS>/latest/containers, <IP_ADDRESS>/latest/services, <IP_ADDRESS>/latest/stacks to figure out what containers, services, stacks are running in the cluster. Now to figure out local containers, it relies on the information returned by <IP_ADDRESS>/latest/self/host. Since network-manager runs on the host network (--net=host) all the requests reaching the metadata container have the IP address of docker0 bridge as the source IP. This means the source IP seen by metadata is <IP_ADDRESS> in most of the use cases and there is no hit for this IP while looking up metadata's internal info. But in @ciokan's setup there are few containers with this IP address, the lookup didn't miss and metadata was returning information which was incorrect in this context. Since there are multiple entries with the same lookup key, the information from metadata was flapping as observed/mentioned in a previous debugging session with a different user.
Workaround:
On hosts where the IP address of the docker0 bridge is not <IP_ADDRESS>, change the docker config and restart docker.
Here are steps for a Ubuntu:
cat > /etc/docker/daemon.json << EOF
{
    "bip": "<IP_ADDRESS>/16",
    "fixed-cidr": "<IP_ADDRESS>/16"
}
EOF

service docker stop
rm -rf /var/run/docker.sock
ps -ef | grep docker
# make sure none of the old docker processes are running.
# if yes do kill -9 ____
service docker start
docker ps
ip addr show docker0

Why docker0 bridge has this IP address?
May be the host was running an older version of docker when the default IP used for the bridge used to be <IP_ADDRESS> and later the docker version was upgraded? Not sure.

Just got off the call with @fedya ... The reason for his problem was different than above.
Summary:
Host 1: Running both rancher/server and rancher/agent ... And this caused the IP detected for this host in Cattle as <IP_ADDRESS>.
Host 2: This is the host having problem "Error .... "
When running both server and agent, it's needed to pass CATTLE_AGENT_IP as the public IP address of the host while registering the host.
Fix:

Delete rancher-agent container on Host1
Start rancher-agent again on Host1, but this time pass CATTLE_AGENT_IP=PUBLIC_IP_OF_HOST_1
Things started fine.

Tonight we were fixed this issue on my servers.
Thanks to @leodotcloud for his time.
Main culrpit was rancher agent that running on rancher server, in such case need to pass CATTLE_AGENT_IP when you deploy agent on the server.

One more reason: https://github.com/rancher/rancher/issues/9367

My devops team approached me with this behavior yesterday when applying an upgrade to rancher-server 1.6.5
I've double checked CATTLE_AGENT_IP is properly set when adding all of the hosts.
After doing some digging, I found that two of the hosts had docker0 as the 'old' <IP_ADDRESS>
Applying this fix from @leodotcloud seemed to clear up the problems for them.
But there is still one other host that is in a bad state where its IPsec containers will not start up and complains about not finding rancher-bridge on the path.
I have tried the workaround mentioned by @kaos where we try to write the file ourselves.  I just wrote a script to to get it to pull the right PID in for me as the stack appears.
Here's what the script looks like if anyone else is interested:
`
root@o1:/var/lib/docker/volumes/rancher-cni-driver/_data/bin # ls .
patch*  rancher-bridge*
root@o1:/var/lib/docker/volumes/rancher-cni-driver/_data/bin # cat patch
echo "#!/bin/sh" > rancher-bridge
echo "exec /usr/bin/nsenter -m -u -i -n -p -t $(pgrep -f rancher-cni.log) -- $0 "$@"" >> rancher-bridge
chmod +x rancher-bridge
`
This seems to change the message away from any mention of rancher-bridge - but it still fails to start up.
I tried the whole process a few times, and then tried to restart network-manager right after running my patch script as well, but I still can't get IPsec to start on this host.
What I can see in the logs leads me to believe there is something screwed up more deeply with the networking (like maybe iptables) on this box.
7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="routesync: DOCKER_BRIDGE=docker0, METADATA_IP=<IP_ADDRESS>" 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="routesync: starting monitoring on bridge: docker0, for metadataIP: <IP_ADDRESS> every 60 seconds" 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Waiting for metadata" 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Inspecting on start" cid=c7ef46e9e9460071d743df6e537f281b322bc163918a7a7c50cd180dbbfaaa2b running=true startedAt="2017-07-22T23:17:57.908299925Z" 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Recording previously started" cid=c7ef46e9e9460071d743df6e537f281b322bc163918a7a7c50cd180dbbfaaa2b startedAt="2017-07-22T23:17:57.908299925Z" 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Inspecting on start" cid=f941b2e1dd4583705e96b6d152f3fa3524e1444831f476301a28828dc673f04b running=true startedAt="2017-07-22T23:17:57.489328691Z" 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Recording previously started" cid=f941b2e1dd4583705e96b6d152f3fa3524e1444831f476301a28828dc673f04b startedAt="2017-07-22T23:17:57.489328691Z" 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Inspecting on start" cid=09cc4d7f1ee925431cb750e75109f60f76a300c65dcf1b1481990414ae66cf1c running=true startedAt="2017-07-22T23:16:40.734314734Z" 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Recording previously started" cid=09cc4d7f1ee925431cb750e75109f60f76a300c65dcf1b1481990414ae66cf1c startedAt="2017-07-22T23:16:40.734314734Z" 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Inspecting on start" cid=3ba8a7fdd11b0d4cac3a0f832a1ca51d54b6ce35171f9066810ac3a0e4b11942 running=true startedAt="2017-07-22T23:16:28.621030996Z" 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Recording previously started" cid=3ba8a7fdd11b0d4cac3a0f832a1ca51d54b6ce35171f9066810ac3a0e4b11942 startedAt="2017-07-22T23:16:28.621030996Z" 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Inspecting on start" cid=5a449b793e2964413b560cedacc13e294a4761b5d15a0c5664b024d651c30e76 running=true startedAt="2017-07-22T23:18:08.507012556Z" 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Recording previously started" cid=5a449b793e2964413b560cedacc13e294a4761b5d15a0c5664b024d651c30e76 startedAt="2017-07-22T23:18:08.507012556Z" 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Inspecting on start" cid=1225ef04618c2219494cdf8f12220f7a18c3cdba799a2a691c4f8b98431c5045 running=true startedAt="2017-07-22T22:41:02.63298098Z" 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Recording previously started" cid=1225ef04618c2219494cdf8f12220f7a18c3cdba799a2a691c4f8b98431c5045 startedAt="2017-07-22T22:41:02.63298098Z" 7/22/2017 7:18:08 PMnet.bridge.bridge-nf-call-iptables = 1 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Applying new nat rules" 7/22/2017 7:18:08 PMnet.ipv4.conf.docker0.route_localnet = 1 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Setting up binaries for: map[rancher-bridge:6d8147bf17e4581110a250e22fd7c3419c8ef85b36a06808b55ec14a3f9ef71f]" 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Starting event router." 7/22/2017 7:18:08 PMtime="2017-07-22T23:18:08Z" level=info msg="Applying new port rules" 7/22/2017 7:18:09 PMtime="2017-07-22T23:18:09Z" level=error msg="macsync: error syncing MAC addresses for the first tiime: inspecting container: 4c65b32cb3ee2a4f1d7c0fd9bc27103e9bd6955765db713d084057df5751ad36: Error: No such container: 4c65b32cb3ee2a4f1d7c0fd9bc27103e9bd6955765db713d084057df5751ad36" 7/22/2017 7:18:09 PMtime="2017-07-22T23:18:09Z" level=info msg="Setting up binaries for: map[rancher-bridge:6d8147bf17e4581110a250e22fd7c3419c8ef85b36a06808b55ec14a3f9ef71f]" 7/22/2017 7:18:09 PMtime="2017-07-22T23:18:09Z" level=error msg="Failed to apply cni conf: Error: No such container: 6d8147bf17e4581110a250e22fd7c3419c8ef85b36a06808b55ec14a3f9ef71f" 7/22/2017 7:18:09 PMtime="2017-07-22T23:18:09Z" level=info msg="Setting up resolv.conf for ContainerId [c7ef46e9e9460071d743df6e537f281b322bc163918a7a7c50cd180dbbfaaa2b]" 7/22/2017 7:18:09 PMtime="2017-07-22T23:18:09Z" level=info msg="Setting up resolv.conf for ContainerId [f941b2e1dd4583705e96b6d152f3fa3524e1444831f476301a28828dc673f04b]" 7/22/2017 7:18:09 PMtime="2017-07-22T23:18:09Z" level=info msg="arpsync: (host) wrong ARP entry found={LinkIndex:3 Family:2 State:32 Type:1 Flags:0 IP:<IP_ADDRESS> HardwareAddr:}(expected: 02:73:10:e6:78:a8) for local container, fixing it" 7/22/2017 7:18:09 PMtime="2017-07-22T23:18:09Z" level=info msg="Network router changed, syncing ARP tables 1/10 in containers, new MAC: 02:73:10:e6:78:a8" 7/22/2017 7:18:09 PMtime="2017-07-22T23:18:09Z" level=error msg="arpsync: while syncing, got error: inspecting container: 4c65b32cb3ee2a4f1d7c0fd9bc27103e9bd6955765db713d084057df5751ad36: Error: No such container: 4c65b32cb3ee2a4f1d7c0fd9bc27103e9bd6955765db713d084057df5751ad36" 7/22/2017 7:18:24 PMtime="2017-07-22T23:18:24Z" level=error msg="macsync: error syncing MAC addresses for the first tiime: inspecting container: 4c65b32cb3ee2a4f1d7c0fd9bc27103e9bd6955765db713d084057df5751ad36: Error: No such container: 4c65b32cb3ee2a4f1d7c0fd9bc27103e9bd6955765db713d084057df5751ad36"
I also think there's that in combination with having crufty values somewhere in the metadata as mentioned in @ibuildthecloud described.  I can see that there is a message still getting spammed in the network-manager log for that host:
level=error msg="macsync: error syncing MAC addresses for the first tiime: inspecting container: 4c65b32cb3ee2a4f1d7c0fd9bc27103e9bd6955765db713d084057df5751ad36: Error: No such container: 4c65b32cb3ee2a4f1d7c0fd9bc27103e9bd6955765db713d084057df5751ad36"

@leodotcloud helped me troubleshoot our setup.  We discovered a veth that was showing up on the host in question that was being returned if you did an ip addr
After verifying this particular one looked off and should not be there, we tried removing it.  It seems that it was not being cleaned up for whatever reason, and had some values for the host from the "old" setup before the upgrade.
Removing manually via an ip link delete veth... seemed to fix things on the next go 'round, IPSec started up correctly, then healthcheck, etc.
Thanks, @leodotcloud and other users on this thread.

I could pitch in another case of netw interface issues that relates to this that we found out.
It was something with the docker0 interface that apparently was causing issues for us on some occasions, as it got resolved by stopping docker, wiping all docker state (/var/lib/docker, /var/run/docker, and recreating storage volumes) in addition to actually removing the docker0 interface. Then start docker again. It was that last piece with removing the docker0 interface that fixed it for us.

@kaos do you happen to know what the docker bridge link address was? Just ran into this today where metadata was stealing docker's bridge ip of '<IP_ADDRESS>', possibly because post upgrade of docker it came up with a '172.42.x.x' address...

@aemneina IIRC it was <IP_ADDRESS>. It had just incremented one octet by one.

Guys I'm being overwhelmed with this error.
We started the upgrade from version 1.5.5 to rancher 1.6.10.
We followed the recommendations to upgrade the network-services before ipsec. After successfully upgrading network-services we started to upgrade the ipsec stack...it started failing on the first host with the following error:
Rancher managed network not work with error 'Couldn't bring up network: netplugin failed but error parsing its diagnostic message "": unexpected end of JSON input'

@calexandre rancher server and rancher node is same pc?

Nope, separate hosts.
I've created a forum post with more detail here: https://forums.rancher.com/t/ipsec-stack-upgrade-problems-when-upgrading-from-rancher-1-5-5-to-1-6-10/7476
Let me know if you want me to bring the issue to to github.

I had the same issue yesterday. I have two hosts, and one of them in the same machine of my Rancher. The node I've created in the same PC of the server worked well but the other one was giving me the rancher-bridge error.
I resolved this removing both hosts and reinstalling them. But first I've installed the host from another machine and then the other one (the one in the same machine of the server).

I do the same steps as @guilhermefernandes1 said above and solve the issue.

With the release of Rancher 2.0, development on v1.6 is only limited to critical bug fixes and security patches.


Add support for HTTP/2

I just did a test with Lighthouse (Desktop) and it showed incredibely nice results on the first try. This is really awesome.

For the Performance section the major penalty was missing HTTP\2

Describe the solution you'd like
I'd like to have an option to activate HTTP\2 via config.

I'm adding the pkg:adapter-node label to this, because I don't think the question of supporting HTTP/2 really makes sense in other contexts.

This is currently not possible as polka (current server in use) does not support http2.
Maybe we can get a polka update with an http2 option....

I created a feature request there: https://github.com/lukeed/polka/issues/162
Since http2 is already supported by Node.js, it's more a framework and design decision for polka.

With  this doc: https://github.com/sveltejs/kit/tree/master/packages/adapter-node#custom-server
The issue is outdated no?

The pull request was denied, one of the reasons was TLS. Maybe it would be possible to start without TLS?
Enable HTTP/2.0 (h2c) without TLS in SvelteKit, so users can benefit from decreased latencies and improved page load speeds. With this the usual reverse proxy can handle TLS, and we can still benefit from the http/2 speed improvements on the internal network.
Reference: HTTP/2.0 h2c on Wikipedia

@bluepuma77 Your best bet is just using a custom server, they're not gonna add built-in support for this.

http/2 in unencrypted cleartext (h2c) works on Node 20.12.0 and earlier with a custom polka server. Support broke between v20.12.0 to v20.12.1, which was discussed over here: https://github.com/sveltejs/kit/issues/11365#issuecomment-2066804017
Anyway, for the record, here is how I run h2c on node v20.12.0:

Create a server.js file at the root of your sveltekit project
Use the following contents:

import polka from 'polka';
import http2 from 'node:http2';
import { handler } from './build/handler.js';
import { env } from './build/env.js';

const path = env('SOCKET_PATH', false);
const host = env('HOST', '<IP_ADDRESS>');
const port = env('PORT', !path && '3000');

// Note: http2.createServer() is unencrypted and will not work in any browser
// This server should be behind a reverse proxy (eg. NGINX) with SSL termination.
const server = polka({ server: http2.createServer() })
  .use(handler)
  .listen({ path, host, port }, () => {
    console.log(`Listening on ${path ? path : host + ':' + port}`);
  });

const sessions = new Set();

server.server.on('session', (session) => {
  sessions.add(session);
  session.on('close', () => {
    sessions.delete(session);
  });
});

function shutdown() {
  server.server.close(() => {
    console.log('Server has closed.');
  });

  for (const session of sessions) {
    session.close();
  }
}

process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);
process.on('exit', () => {
  console.log('Process is exiting.');
});

To test locally

Run with e.g. PORT=1234 node server.js
Use curl command curl -i --http2-prior-knowledge http://localhost:1234

Again, the above will only work in Node 20.12.0 and below, unfortunately. Running an old version of Node is probably not the best idea long-term, so it would be nice with a better solution for h2c http/2.
(PS. I'm using Google Cloud Run for end-to-end http/2 which requires h2c for it to work)


Unexpected token operator «=», expected punc «,» [~/phpunserialize/phpUnserialize.js:80,0]

While running this along with symfony app i get this error:
Unexpected token operator «=», expected punc «,» [~/phpunserialize/phpUnserialize.js:80,0]

[Symfony\Component\Process\Exception\ProcessFailedException]
  The command "'node_modules/.bin/webpack' '--config' '~/var/cache/prod/webpack.config.js'" failed.

  Exit Code: 2(Misuse of shell builtins)

By looking at line 80, i see default parameter expect = '"'
When removing = '"' it works again. Issue appeared after upgrading from v1.0.1 to v1.1.0

I think its related with: https://github.com/bd808/php-unserialize-js/commit/6aedd57e3b6d7a59edffe5a9c1d6ac3b0101c79c
@bd808 @bd808

Default arguments are an ES2015 (sometimes known as ES6) feature. My brain skipped over that when working on  6aedd57. In this case it is a thing that is pretty easy to fix to push the default down into the method body.


Change TableElementsAnalyzer to only require parameters

Current coverage is 81.56% (diff: 100%)

Merging #4750 into master will decrease coverage by 0.16%

@@             master      #4750   diff @@
==========================================
  Files          1314       1314
  Lines         45078      45079     +1
  Methods           0          0
  Messages          0          0
  Branches       7062       7062
==========================================
- Hits          36842      36769    -73
- Misses         5565       5634    +69
- Partials       2671       2676     +5

Powered by Codecov. Last update acd3aa8...99623dd


HTML in multipleSeparator

Hello,
I try to customize my select with multipleSeparator: I need to display each selected option on a new line.
This is what I need:

And this is what I get when setting multipleSeparator = "<br>":

Do you know how I can do that ?
Thanks

A workaround:
<select class="selectpicker" multiple data-multiple-separator="
">
</select>

.bootstrap-select .dropdown-toggle .filter-option-inner-inner {
    white-space: pre-line;
}

Thank you !


RCTImageLoader error

When I load a image marker a yellow box display the error [RCTImageLoader loadImageWithTag:size:scale:resize Mode:progressBlock:completionBlock:] is deprecated. Instead use [RCTImageLoader loadImageWithURLRequest:size:scale:clipped:resizeMode:progressBlock:completionBlock:]
Whats the best way to update the library? I can update and create a pr but I dont find where is happening the bug.

Ohh cool, the PR isnt merged, whats the plan?

I recommend you to fork the repo and manually merge this pr.
It will be much faster then waiting for maintainer to merge it into this repo.

So what are you planning for the update?

I really don't know. There are a lot of open PR's and nobody merges them or even reviews.

Does anyone have a stable react-native-map fork to share that works with both iOS and Android and a recent RN version?
Or can recommend the latest version of react-native that is stable with a version of react-native-maps?
I have spent days trying to find a solution.  I wish I would have stayed with react-native-maps 0.4.2.  Or started with Xamarin months ago.
After much effort upgrading from:
    "react": "15.0.2",
    "react-native": "0.26.2",
    "react-native-maps": "^0.4.2",

to:
    "react": "^15.3.1",
    "react-native": "^0.32.0",
    "react-native-maps": "^0.7.1",

The result is:

Now map callouts no longer display correctly anymore on Android.
On iOS rendering map just red-box crashes with:

2016-08-30 11:43:16.361 [error][tid:main][RCTConvert.m:57] Error setting property 'source' of RCTImageView with tag #369: JSON value '<null>' of type NSNull cannot be converted to NSString
dyld_image_header_containing_address(0x1019f6201)

Oh yes, Im revert the change becouse the solution have a collision bug bilding for ios.
I dont know how repare this bug, bit its a warning and my app build without problems.

The fix for this has already been merged: https://github.com/lelandrichardson/react-native-maps/pull/456
I'll be releasing 0.8.0 soon.


Provide a display for the OWL property restriction representation of a phyloreference

In order to implement reasoning (#3), we will need to produce an OWL property restriction corresponding to each phyloreference in a PHYX file. For the purpose of explaining how phyloreferencing works, it would be useful to allow users to display this OWL expression through the interface. This is also a chance to find the best way to display these expressions: in a previous experiment, we found that italicizing OWL Manchester terms worked well.

An alternative would be to use the same formatting at Uberon use in their paper, which is to render OWL terms in fixed-width fonts, to italicize properties and to surround classes and , e.g. 'pulmonary alveolus' SubClassOf part_of some 'lung'.
We will also want to make the properties into links so that users can look up their definitions.

Sounds like some kind of "expert mode"?

Note that the JSON-LD file produced by the Curation Tool can be opened in Protege and the OWL property restriction viewed there. Protege's display of the definition uses labels instead of term IDs and allows the user to click on individual specifiers. So, implementing this issue only makes sense if we can provide something here that is missing from Protege.

Agreed!


Manage two or more Ldap Servers with one osixia/phpldapadmin

Hi,
First, thanks for your ldap containers! We are using them in production with success since 03/2016 first in kubernetes and now in Rancher/Cattle.
We have a question, sorry if it is newbie.
In phpldapadmin when we set PHPLDAPADMIN_LDAP_HOSTS=ldap-master, we manage our master.
Are there any way I pass another ldap server, so we can manage two or more servers with the same phpldap admin?
Something like: PHPLDAPADMIN_LDAP_HOSTS=ldap-master/ldap-slave...

Hi Marlin. You will need to define the PHPLDAPADMIN_LDAP_HOSTS like this in your RC/Deployment/etc:
        - name: PHPLDAPADMIN_LDAP_HOSTS
          value: "#PYTHON2BASH:[{'ldap1.example.com': [{'server': [{'tls': True}]},{'login': [{'bind_id': 'cn=admin,dc=example,dc=com'}]}]}, 'ldap2.example.com', 'ldap3.example.com', 'ldap4.example.com']"

Thank you @d4lk0s

Hi @BertrandGouny, is this still the only acceptable approach, or would it be possible to define multiple servers in a config.php which then gets mounted as described here? Many thanks in advance.


make ocr work with multi-framed gif files

👍 so this just pulls out the first frame of the gif?

@pastudan correct

Coverage remained the same at 35.36% when pulling 9b5abdcc35c6e591a9378f2c527ae8e199202b71 on dev-ocr-gifs into 8540b2e1833b968aa58be61a179aa22a1ad98659 on master.


DnsServerAddress: improve dhcp checks

Having submitted my own PR (#480) for this resource and reviewed the most recent PR (#482) for this resource, I found a missing component. When choosing to -ResetServerAddresses there is not equivalent logic to see if the interface is obtaining an address via DHCP.
When a DNS Server is statically assigned, it will reside in the following registry key:
HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\{<Interface GUID>}\NameServer

When a DNS Server is assigned via DHCP, it will reside in the following registry key:
HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\{<Interface GUID>}\DhcpNameServer

However Get-DnsClientServerStaticAddress (NetworkingDsc.Common.psm1) only retrieves the value for NameServer.
Ideally the new workflow should first identify if an interface is DHCP enabled and then perform the following check on these registry keys.
if (DhcpEnabled)
{
    DhcpNameServer != Empty
    NameServer == Empty
} else {
    DhcpNameServer == Empty
    NameServer != Empty
}

This would be a significant change and greatly impact my current PR for this resource. Would it make sense to include these changes in my PR or close it and open a new one?

After delving into this more, taking the proposed approach doesn't make sense. The cmdlet doesn't modify DhcpNameServer and thus shouldn't be added as additional functionality.


allow unreffing hotplug

little feature request, very low priority, just here to remind myself in case I have time someday.
it would be cool if usb had ref() and unref() methods (like all Node.js event sources) to allow not holding the event loop alive. some projects may need it.

Would that be similar to the hotplug ref/unref but for the usb_context?

sorry, I can't make sense of this issue... as you mentioned, there's ref/unref for the hotplug events.
according to the history, it already existed when I opened this issue. really weird. anyway, closing.


User friendly list definitions

Allow lists to be defined inline for suitable component types
Default list name to the component name?
Remove the need to specify a json name for the list

#121


No link on the labels in the "Initiative view"

There should not be any links on the labels that exists in the "Inititaive view". So in other words, you should not be able to click on the "iOS" "Android" "Website". But they should still be there.

fixed


Do not cleanup headers on received mails

https://github.com/deltachat/chatmail/pull/156 just got merged and it removes Received headers for sent mails to avoid leaking MUA IP address.
These settings however apply to both incoming and outgoing mails currently, so when we receive an incoming message with a lot of Received headers, we drop all of them. I sent a mail from testrun.org (running mailcow) to c20.testrun.org and got only a single Received header:
Return-Path<EMAIL_ADDRESS>Delivered-To<EMAIL_ADDRESS>Received: from c20.testrun.org
	by c20 with LMTP
	id Z5BuAQW7mGWcQQEAysZ61w
	(envelope-from<EMAIL_ADDRESS>	for<EMAIL_ADDRESS>Sat, 06 Jan 2024 02:29:25 +0000

We generally want to have Received headers to display hop info in the message info in Delta Chat, it is like a traceroute that can show you where the message was delayed if you receive it late.

I understand your point, but it is angainst a strictly data save stadard. Only data is save, which is not stored in any way. The bigest critic to deltachat is the mass of meta-data in emails.
Maybe a compromiss is a flag into chatmai.ini to choose the strength of headercleaning (nad the strength of pipeling-stopp). But think of , that it is a harder point to programm for the cmdeploy programmers


Delay in showing Ajaxcart content on first time opening.

I've noticed there is a delay when you first open the Ajax quick cart, when the quick cart slides into view it takes a moment for the cart content to load in.
Is there a way to fix this so that the content is immediately visible at the first time of revealing the quick cart?

Unfortunately not*. With any ajax request, there will be some time needed to load the contents. In our case, we grab the cart contents via ajax and only then can we show the contents.

If you want, you can do some custom JS to load the cart contents without opening the drawer on page load. We haven't done this because it increases the page weight and load times.


with provided market_953.pth.tar, unable to get 95.3 rank1 results

Hi,
Thanks a lot for your provided codes and models.
But with your provided with provided model “market_953.pth.tar”, and your code main_reid.py, I cannot get the result of rank1 95.3%.
Is there any tricks to play with inference? Thanks in advance!
I run the following command:
python3 main_reid.py train --save_dir='./pytorch-ckpt/market_bfe' --model_name=bfe --train_batch=32 --test_batch=32 --dataset=market1501 --pretrained_model='./pytorch-ckpt/market_bfe/market_953.pth.tar' --evaluate
And I get the market1501 dataset from the below link:
https://jingdongwang2017.github.io/Projects/ReID/Datasets/Market-1501.html
resuls:
=> Market1501 loaded
Dataset statistics:
subset   | # ids | # images
train    |   751 |    12936
query    |   750 |     3368
gallery  |   751 |    15913
total    |  1501 |    32217
initializing model ...
load pretrained model ./pytorch-ckpt/market_bfe/market_953.pth.tar
model size: 32.27600M
Extracted features for query set: 3368 x 1536
Extracted features for gallery set: 15913 x 1536
Computing distance matrix
Computing CMC and mAP
Results ----------
mAP: 85.5%
CMC curve
Rank-1  : 94.6%
Rank-2  : 96.5%
Rank-4  : 97.4%
Rank-5  : 97.8%
Rank-8  : 98.4%
Rank-10 : 98.8%
Rank-16 : 99.3%
Rank-20 : 99.3%
hardwares and softwares:
GPU : 2080TI     CUDA:10.0     pytorch:1.0.0

I meet the same problem. Did you solve it? @sijun-zhou


[Feat] Add support user-defined serve service ports

Why are these changes needed?
Users may only need to customize the serve service to use the NodePort mode for external access, but currently, the implementation requires setting all ports in the cluster to NodePort mode.
Related issue number
Checks

[x] I've made sure the tests are passing.
Testing Strategy

[x] Unit tests
[x] Manual tests
[ ] This PR is not tested :(

@kevin85421 @MortalHappiness PTAL


Deal with multiple addrs in get-ticket command

The ticket can now contain multiple addresses on which the provider is listening if the provider is bound to the wildcard address.  However the get-ticket command only uses the first address.
It should instead try and connect to all the addresses and use the first connection which works.

Done in x-hp


Issue about issues

Just checking if Sylvain is still around...
Probably he is fixing issues now @huggingface
Congratz on your new job, are you bringing fastai2 to the table?

I'm still looking when I have a bit of time, but I won't be a main maintainer either. As for HuggingFace, no plan for more integration in fastai v2 on my side no. They do provide data and a model that you can pass to a Learner and train with fastai.

Super, are you working remotely or with the team at NY?

Remotely because of the pandemic, but I'm based in NY so will join the NY office when it's safe.


Google download error in China

Because Google is not in service somewhere, I suggest the following change in the 'data_load.py'.
`
try:

    paths = self.download_google_drive_file()

except:

    paths = ''

`

This project is temporally stopped after I read the sentence of the paper that authors will release the code with TensorFlow.


When I segment a 3D Volume, how could I mask different frames of image in one initialization

Now I segment one frame of the 3D image, I must re-initialization then put prompt points in next frame. Is it possible to mask all frames of the 3D image in one initialization.

The initialization should be a one-time thing. Once initialized, all embeddings throughout the volume are cached. The new changes https://github.com/bingogome/samm/commit/5b63c7dfa120c39066215fb766f8b246be3e2414 made some improvements over system glitches and should be easier to use.


Shorter width for new GitHub UI

After the GitHub UI update with more padding in featured repos/gists, my musicbox looks like this:

Maybe we can decrease the width of the text, perhaps have it configurable?

i can't seem to reproduce this? are you sure it's not a matter of screen size/resolution?
as for having it be configurable, i'm going to eventually rewrite the entire project in js/ts and have it have different configurable options :)-

A ts rewrite would be awesome!

@aabbccsmith soon ;)

i can't seem to reproduce this? are you sure it's not a matter of screen size/resolution?

Could be. Do you think you can see my profile and show me what it looks like on your device? https://github.com/AnandChowdhary

seems to be all good on my end

Thanks, @jacc! It's probably just me: I'm using Safari on macOS and it's been like this since the GitHub UI update.

@AnandChowdhary all good! last question, do you happen to be on big sur? wondering if that's the issuer

No, just Safari 13.1.1 (156<IP_ADDRESS>.2) on macOS Catalina 10.15.5 (19F101).


Fix katana consuming all cpu

consider a scenario of url is some icon or css url
it does not have any more urls to traverse, at that time
it was looping and checking if queue is nil or not for 10s.
if we are passing multiple urls, then it consume all the CPU
coz, infinite loop and using time.Now() in for loop.
now, it will check after 1s.

Using dev branch

using this PR


Update API.md

Added canceled to relay documentation (as it was missing)

Thanks for the contribution


how to support remote ip is a domain?

how to support remote ip is a domain? like this:
/usr/local/bin/udp2raw --log-level 3 -c -l <IP_ADDRESS>:5888 -r xxx.com:5883
there is a question:
It seems not support the xxx.com domain.

how to support remote ip is a domain? like this:
/usr/local/bin/udp2raw --log-level 3 -c -l <IP_ADDRESS>:5888 -r xxx.com:5883 -f 20:10
there is a question:
It seems not support the xxx.com domain.

and the same question with udpspeeder

There is no native function.
You can use dig to get the IP address of the domain name and assign it.


Uncaught TypeError: Cannot read property 'version' of undefined

[Enter steps to reproduce:]

...
...

Atom: 1.46.0 x64
Electron: 4.2.7
OS: Unknown Windows version
Thrown From: Atom Core
Stack Trace
Uncaught TypeError: Cannot read property 'version' of undefined
At C:\Users\miguelangel\AppData\Local\atom\app-1.46.0\resources\app\static\<embedded>:14

TypeError: Cannot read property 'version' of undefined
    at ~/AppData/Local/atom/app-1.46.0/resources/app/static/<embedded>:14:2635690
    at ~/AppData/Local/atom/app-1.46.0/resources/app/static/<embedded>:11:941157
    at r (~/AppData/Local/atom/app-1.46.0/resources/app/static/<embedded>:11:938163)
    at o (~/AppData/Local/atom/app-1.46.0/resources/app/static/<embedded>:14:1122275)
    at t.bufferStream (~/AppData/Local/atom/app-1.46.0/resources/app/static/<embedded>:14:1122429)
    at Socket.e.on (~/AppData/Local/atom/app-1.46.0/resources/app/static/<embedded>:14:1121031)
    at Socket.emit (events.js:187:15)
    at Pipe._handle.close (net.js:606:12)

Commands
     -0:20.1.0 core:confirm (input.hidden-input)

Non-Core Packages
atom-console 0.4.6
atom-live-server 2.3.0
emmet 2.4.3
file-icons 2.1.41
minimap 4.29.9

Duplicate of https://github.com/atom/atom/issues/20723


Add hierarchy / explorer view

Since parent / child relations  #42 are important in navigating Atomic Data, I should make it easier to navigate these. I think adding a special Hierarchy view for resources might do the trick. This might also temporarily replace #75

Duplicate of #96