0.13-rc1 to 0.13-rc2 upgrade faild

etcd did not start, and the cfn-init service did never send OK signal. I would imagine that the updated etcd version of rc2 might be the issue, failing to upgrade?
+00:17:55       ip-10-1-43-193.eu-west-1.compute.internal: "etcdadm-reconfigure.service: Failed with result 'exit-code'."
+00:17:55       ip-10-1-43-193.eu-west-1.compute.internal: "Dependency failed for etcd (System Application Container)."
+00:17:55       ip-10-1-43-193.eu-west-1.compute.internal: "Dependency failed for etcdadm update status."
+00:17:55       ip-10-1-43-193.eu-west-1.compute.internal: "cfn-signal.service: Failed with result 'exit-code'."
+00:19:05       ip-10-1-43-193.eu-west-1.compute.internal: "etcdadm-reconfigure.service: Failed with result 'exit-code'."
+00:19:05       ip-10-1-43-193.eu-west-1.compute.internal: "Dependency failed for etcd (System Application Container)."
+00:19:05       ip-10-1-43-193.eu-west-1.compute.internal: "Dependency failed for etcdadm update status."
+00:19:05       ip-10-1-43-193.eu-west-1.compute.internal: "cfn-signal.service: Failed with result 'exit-code'."

Rollback did also fail, never sending cfn-init.
Why did this happen, and what can be done to recover?

So it's impossible to recover this stack now.
I'm not allowed to continue a rollback on a nested stack. And I cannot skip a nested stack that is in rollback failed status when trying to continue rollback on the parent stack. So a catch 22...

Hi, sorry to hear that - I haven’t experienced this issue with rc2.  Perhaps you could take a look at the etcd-member journalctl logs to see if ectd is starting? Or what it is erroring with?  If you can manually get etcd to start then you can tell cloud formation to continue the rollback.

Thanks. First of all, this is only a test cluster. So no harm done. Second, the etcd node that failed was terminated by CloudFormation when the rollback started. So I have no way examining that failed instance. The replacement/rollback etcd instance is of course running, but etcd has probably not started because cfn-init never executed.
How to I proceed to debug and get etcd started manually?

This is the log from the rollback instance
-- Logs begin at Fri 2019-06-21 05:30:25 UTC, end at Fri 2019-06-21 07:01:49 UTC. --
Jun 21 07:01:45 ip-10-1-43-240.eu-west-1.compute.internal systemd[1]: Stopped etcd (System Application Container).
Jun 21 07:01:45 ip-10-1-43-240.eu-west-1.compute.internal systemd[1]: etcd-member.service: Scheduled restart job, restart counter is at 2502.
Jun 21 07:01:45 ip-10-1-43-240.eu-west-1.compute.internal systemd[1]: etcd-member.service: Service RestartSec=10s expired, scheduling restart.
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal systemd[1]: Failed to start etcd (System Application Container).
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal systemd[1]: etcd-member.service: Failed with result 'exit-code'.
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal systemd[1]: etcd-member.service: Main process exited, code=exited, status=1/FAILURE
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.372226 C | etcdmain: member dab84c2963a2830f has already been bootstrapped
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.361388 I | pkg/netutil: resolving ip-10-1-43-133.eu-west-1.compute.internal:2380 to <IP_ADDRESS>:2380
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.361082 I | pkg/netutil: resolving ip-10-1-43-133.eu-west-1.compute.internal:2380 to <IP_ADDRESS>:2380
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.349844 I | embed: listening for client requests on <IP_ADDRESS>:2379
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.349792 I | embed: listening for peers on https://<IP_ADDRESS>:2380
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.348982 I | embed: peerTLS: cert = /etc/ssl/certs/etcd.pem, key = /etc/ssl/certs/etcd-key.pem, ca = , trusted-ca>
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.348961 N | etcdmain: the server is already initialized as member before, starting as etcd member...
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.348914 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.348901 I | etcdmain: Go OS/Arch: linux/amd64
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.348893 I | etcdmain: Go Version: go1.8.5
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.348889 I | etcdmain: Git SHA: 95a726a
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.348884 I | etcdmain: etcd Version: 3.2.13
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.348844 W | pkg/flags: unrecognized environment variable ETCD_IMAGE_TAG=v3.2.13
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.348838 W | pkg/flags: unrecognized environment variable ETCD_USER=etcd
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.348818 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd-trusted>
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.348802 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd-tr>
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.348795 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd-key.pem
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.348783 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd.pem
Jun 21 07:01:35 ip-10-1-43-240.eu-west-1.compute.internal etcd-wrapper[15974]: 2019-06-21 07:01:35.348775 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd0
-- Logs begin at Fri 2019-06-21 05:30:25 UTC, end at Fri 2019-06-21 07:01:49 UTC. --

I managed to get the rollback to proceed. But the etcd0 node is now in a bad state.
I imagine that the upgrade service tried to upgrade the etcd0 node to a new version of etcd (part of 0.13-rc2), then failed execute cfn-init (maybe because of underlying etcd problems, I don't know). The etcd data was probably upgraded. A rollback was executed by CloudFormation due to the failed cfn-init service. Then the etcd0 node created by the rollback refuses to start because the data has already been upgraded to a new etcd version. Is it possible to wipe all etcd data on a node, and let the node catch up by reading data from the two other etcd nodes in the cluster?
This is the log when I tried the upgrade again.
core@ip-10-1-43-119 ~ $ journalctl -xe
-- The job identifier is 6110 and the job result is dependency.
Jun 21 07:24:32 ip-10-1-43-119.eu-west-1.compute.internal systemd[1]: etcdadm-update-status.service: Job etcdadm-update-status.service/start failed with result 'depende>
Jun 21 07:24:32 ip-10-1-43-119.eu-west-1.compute.internal systemd[1]: etcd-member.service: Job etcd-member.service/start failed with result 'dependency'.
Jun 21 07:24:32 ip-10-1-43-119.eu-west-1.compute.internal systemd[1]: Starting cfn-signal.service...
-- Subject: A start job for unit cfn-signal.service has begun execution
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- A start job for unit cfn-signal.service has begun execution.
--
-- The job identifier is 6020.
Jun 21 07:24:32 ip-10-1-43-119.eu-west-1.compute.internal systemctl[6570]: inactive
Jun 21 07:24:32 ip-10-1-43-119.eu-west-1.compute.internal systemd[1]: cfn-signal.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- An ExecStartPre= process belonging to unit cfn-signal.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 3.
Jun 21 07:24:32 ip-10-1-43-119.eu-west-1.compute.internal systemd[1]: cfn-signal.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The unit cfn-signal.service has entered the 'failed' state with result 'exit-code'.
Jun 21 07:24:32 ip-10-1-43-119.eu-west-1.compute.internal systemd[1]: Failed to start cfn-signal.service.
-- Subject: A start job for unit cfn-signal.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- A start job for unit cfn-signal.service has finished with a failure.
--
-- The job identifier is 6020 and the job result is failed.

The upgrade looks pretty inconspicuous (fc29a9f, but this feels like a blocker if we're going to include the etcd upgrade in the v0.13.x release. Especially if it's reproducible.

I agree. Etcd updates that is not backwards compatible is always dangerous, given the way CloudFormation handles rollbacks. I think etcd upgrades is something that a user should have to specifically initiate by specifying the new version in the cluster.yaml file.
Any tips on how to recover the etcd instance?

Is there only one out of the three etcd instances that have failed?

Yes. The stack update failed on etcd0, so etcd1 and etcd2 are untouched and still reports as healthy.

I thought with the upgrade being within the 3.2.x etcd branch that it should remain compatible with the previous 3.2.x version.  I had been told that etcd runs in compatibility mode until it sees all members have been updated then it enables the new features.  You can safely reclone etcd0 from the others by deleting the data a rejoining it to the cluster.  I will keep an eye out for issue, so far I’ve not seen it upgrading legacy versions to 0.13.0-rc3.

On 21 Jun 2019, at 10:44, Dominic Gunn<EMAIL_ADDRESS>wrote:
The upgrade looks pretty inconspicuous (fc29a9f, but this feels like a blocker if we're going to include the etcd upgrade in the v0.13.x release. Especially if it's reproducible.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

The following should probably fix etcd0.
$ set -a; source /var/run/coreos/etcdadm-environment; source /etc/etcd-environment; source /var/run/coreos/etcdadm-environment; set +a
$ sudo systemctl stop etcd-member.service
$ /opt/bin/etcdadm replace
$ sudo systemctl start etcd-member.service

I thought with the upgrade being within the 3.2.x etcd branch that it should remain compatible with the previous 3.2.x version. I had been told that etcd runs in compatibility mode until it sees all members have been updated then it enables the new features. You can safely reclone etcd0 from the others by deleting the data a rejoining it to the cluster. I will keep an eye out for issue, so far I’ve not seen it upgrading legacy versions to 0.13.0-rc3.
…
On 21 Jun 2019, at 10:44, Dominic Gunn @.***> wrote: The upgrade looks pretty inconspicuous (fc29a9f, but this feels like a blocker if we're going to include the etcd upgrade in the v0.13.x release. Especially if it's reproducible. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Yeah, I thought the same @davidmccormick, which is why I thought it was inconspicuous. It looks like it shouldn't cause an issue, but I also trust @paalkr's reports. I'd be interested in finding out what exactly caused this.
You've had successful upgrades from v0.13.0 to v<IP_ADDRESS>-RC3 right? What version have you tried to update from @paalkr?

This issue happened when trying to upgrade from 0.13.0-rc1 to 0.13.0-rc2. I college if mine will explain the actions he has tried to recover the etcd node.
@dominicgunn , you are referring to 0.13.0-rc3? I may have missed something, but I can't seem to find that release anywhere.
I will also try to upgrade another dev cluster from 0.13.0-rc1, to see if the same thing happens to this cluster as well.

Apologies, I referred to 0.13.0-rc3 which I haven’t released - I have been testing by building the HEAD of the v0.13.x branch.  I’m just waiting for a couple of changes before making it a release.

Let’s see if it happens on any other clusters

@dominicgunn I'm a colleague of @paalkr that has been looking into this. I've tried the commands you've listed above (basically the same as those listed here: https://kubernetes-incubator.github.io/kube-aws/advanced-topics/etcd-backup-and-restore.html#manually-restoring-a-permanently-failed-etcd-node-from-etcd-snapshot), and it does not fix the issue.
The log from etcd-member.service after doing those commads are:
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal systemd[1]: Stopped etcd (System Application Container).
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal systemd[1]: etcd-member.service: Succeeded.
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal rkt[15547]: pod "b52ab21d-4c61-4842-8bf2-6d519681dc7d" is already stopped
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.055355 E | etcdmain: forgot to set Type=notify in systemd service file?
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053696 I | rafthttp: stopped peer c8a02416947a5853
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053691 I | rafthttp: stopped streaming with peer c8a02416947a5853 (stream Message reader)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053677 I | rafthttp: stopped streaming with peer c8a02416947a5853 (stream MsgApp v2 reader)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053645 I | rafthttp: stopped HTTP pipelining with peer c8a02416947a5853
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053628 I | rafthttp: stopped streaming with peer c8a02416947a5853 (writer)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053619 I | rafthttp: stopped streaming with peer c8a02416947a5853 (writer)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053609 I | rafthttp: stopping peer c8a02416947a5853...
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053604 I | rafthttp: stopped peer 268d5cbd6382d8f2
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053573 I | rafthttp: stopped streaming with peer 268d5cbd6382d8f2 (stream Message reader)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053565 I | rafthttp: stopped streaming with peer 268d5cbd6382d8f2 (stream MsgApp v2 reader)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053556 I | rafthttp: stopped HTTP pipelining with peer 268d5cbd6382d8f2
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053539 I | rafthttp: stopped streaming with peer 268d5cbd6382d8f2 (writer)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053531 I | rafthttp: stopped streaming with peer 268d5cbd6382d8f2 (writer)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053496 I | rafthttp: stopping peer 268d5cbd6382d8f2...
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053469 I | etcdserver: aborting publish because server is stopped
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053454 E | etcdserver: publish error: etcdserver: request cancelled
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053396 I | etcdserver: the data-dir used by this member must be removed.
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.053373 E | etcdserver: the member has been permanently removed from the cluster
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.049609 I | etcdserver/membership: added member dab84c2963a2830f [https://ip-10-1-43-133.eu-west-1.compute.internal:2380] to cluster e146e62db33b7661
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.049527 I | etcdserver/membership: added member c8a02416947a5853 [https://ip-10-1-45-117.eu-west-1.compute.internal:2380] to cluster e146e62db33b7661
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.049368 I | etcdserver/membership: added member 268d5cbd6382d8f2 [https://ip-10-1-44-127.eu-west-1.compute.internal:2380] to cluster e146e62db33b7661
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.042771 I | rafthttp: started streaming with peer c8a02416947a5853 (stream MsgApp v2 reader)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.035834 I | rafthttp: started streaming with peer c8a02416947a5853 (stream Message reader)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.035651 I | rafthttp: started streaming with peer c8a02416947a5853 (writer)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.032264 I | embed: ClientTLS: cert = /etc/ssl/certs/etcd.pem, key = /etc/ssl/certs/etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/etcd-trusted-ca.pem, client-cert-auth = true
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.027408 I | etcdserver: starting server... [version: 3.2.13, cluster version: to_be_decided]
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.027316 I | rafthttp: added peer c8a02416947a5853
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.027214 I | rafthttp: started peer c8a02416947a5853
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.027001 I | rafthttp: started streaming with peer c8a02416947a5853 (writer)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.026600 I | rafthttp: started streaming with peer 268d5cbd6382d8f2 (stream MsgApp v2 reader)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.020326 I | rafthttp: started streaming with peer 268d5cbd6382d8f2 (stream Message reader)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.007598 I | rafthttp: started HTTP pipelining with peer c8a02416947a5853
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.007467 I | rafthttp: starting peer c8a02416947a5853...
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.007316 I | rafthttp: added peer 268d5cbd6382d8f2
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.006977 I | rafthttp: started peer 268d5cbd6382d8f2
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.005147 I | rafthttp: started streaming with peer 268d5cbd6382d8f2 (writer)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.000964 I | rafthttp: started streaming with peer 268d5cbd6382d8f2 (writer)
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:01.000618 I | rafthttp: started HTTP pipelining with peer 268d5cbd6382d8f2
Jun 26 07:36:01 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.999942 I | rafthttp: starting peer 268d5cbd6382d8f2...
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.989195 W | auth: simple token is not cryptographically signed
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.981813 I | raft: dab84c2963a2830f became follower at term 1
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.981806 I | raft: newRaft dab84c2963a2830f [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.981793 I | raft: dab84c2963a2830f became follower at term 0
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.981754 I | etcdserver: starting member dab84c2963a2830f in cluster e146e62db33b7661
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.977731 I | etcdserver: initial cluster = etcd0=https://ip-10-1-43-133.eu-west-1.compute.internal:2380,etcd1=https://ip-10-1-44-127.eu-west-1.compute.internal:2380,etcd2=https://ip-10-1-45-117.eu-west-1.compute.internal:2380
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.977716 I | etcdserver: initial advertise peer URLs = https://ip-10-1-43-133.eu-west-1.compute.internal:2380
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.977690 I | etcdserver: advertise client URLs = https://ip-10-1-43-133.eu-west-1.compute.internal:2379
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.977679 I | etcdserver: snapshot count = 100000
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.977674 I | etcdserver: election = 1000ms
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.977670 I | etcdserver: heartbeat = 100ms
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.977665 I | etcdserver: member dir = /var/lib/etcd/member
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.977658 I | etcdserver: data dir = /var/lib/etcd
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.977630 I | etcdserver: name = etcd0
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.969205 I | pkg/netutil: resolving ip-10-1-43-133.eu-west-1.compute.internal:2380 to <IP_ADDRESS>:2380
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.968280 I | pkg/netutil: resolving ip-10-1-43-133.eu-west-1.compute.internal:2380 to <IP_ADDRESS>:2380
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.960243 I | embed: listening for client requests on <IP_ADDRESS>:2379
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.960189 I | embed: listening for peers on https://<IP_ADDRESS>:2380
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959354 I | embed: peerTLS: cert = /etc/ssl/certs/etcd.pem, key = /etc/ssl/certs/etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/etcd-trusted-ca.pem, client-cert-auth = false
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959283 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959277 I | etcdmain: Go OS/Arch: linux/amd64
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959273 I | etcdmain: Go Version: go1.8.5
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959268 I | etcdmain: Git SHA: 95a726a
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959263 I | etcdmain: etcd Version: 3.2.13
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959237 W | pkg/flags: unrecognized environment variable ETCD_IMAGE_TAG=v3.2.13
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959229 W | pkg/flags: unrecognized environment variable ETCD_USER=etcd
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959205 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd-trusted-ca.pem
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959184 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd-trusted-ca.pem
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959177 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd-key.pem
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959164 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd.pem
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959155 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd0
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959140 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://<IP_ADDRESS>:2380
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959132 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://<IP_ADDRESS>:2379
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959122 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/etc/ssl/certs/etcd-key.pem
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959110 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959098 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd0=https://ip-10-1-43-133.eu-west-1.compute.internal:2380,etcd1=https://ip-10-1-44-127.eu-west-1.compute.internal:2380,etcd2=https://ip-10-1-45-117.eu-west-1.compute.internal:2380
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959037 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://ip-10-1-43-133.eu-west-1.compute.internal:2380
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.959006 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.958991 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.958983 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/etc/ssl/certs/etcd.pem
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.958973 I | pkg/flags: recognized and used environment variable ETCD_AUTO_COMPACTION_RETENTION=1
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: 2019-06-26 07:36:00.958883 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://ip-10-1-43-133.eu-west-1.compute.internal:2379
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: + exec /usr/bin/rkt run --uuid-file-save=/var/lib/coreos/etcd-member-wrapper.uuid --trust-keys-from-https --mount volume=coreos-systemd-dir,target=/run/systemd/system --volume coreos-systemd-dir,kind=host,source=/run/systemd/system,readOnly=true --volume coreos-data-dir,kind=host,source=/var/lib/etcd2,readOnly=false --volume coreos-etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true --volume coreos-usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true --volume coreos-etc-hosts,kind=host,source=/etc/hosts,readOnly=true --volume coreos-etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true --mount volume=coreos-data-dir,target=/var/lib/etcd --mount volume=coreos-etc-ssl-certs,target=/etc/ssl/certs --mount volume=coreos-usr-share-certs,target=/usr/share/ca-certificates --mount volume=coreos-etc-hosts,target=/etc/hosts --mount volume=coreos-etc-resolv,target=/etc/resolv.conf --inherit-env --stage1-from-dir=stage1-fly.aci quay.io/coreos/etcd:v3.2.13 --user=232 --
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[15517]: ++ id -u etcd
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal systemd[1]: Started etcd (System Application Container).
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal systemctl[15513]: active
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal systemctl[15511]: active
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal rkt[15504]: rm: failed to remove one or more pods
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal rkt[15504]: rm: cannot get pod: no matches found for "55fef84a-b746-4e04-bfe1-8157513251fd"
Jun 26 07:36:00 ip-10-1-43-48.eu-west-1.compute.internal systemd[1]: Starting etcd (System Application Container)...

What I notice from the logs above, is that this node (the failing one), still lists it's old etcd cluster id (dab84c2963a2830f), even after running the replace command. I would assume it should get a new id?
Also, I did some searching around for etcd cluster recovery, and as far as I could tell, when adding a node into an existing cluster, we should set the ETCD_INITIAL_CLUSTER_STATE to existing. I've also tried to do that, but I've been unable to pass that in to the command, since the cfn scripts pulls down the config on service and sets the env variables, overwriting any changes I've done.
I guess one way to buypass this would be to just remove the cfn config part from the etcd-member.service start temporarily, but I haven't looked more into the ETCD_INITIAL_CLUSTER_STATE=existing yet.

Hey @andersosthus.
Yes, it should definitely get a new id. I'm not massively concerned about the initial cluster state flag, but what you could try doing is removing the bad id from the cluster completely, something along these lines perhaps.
On the broken node, source all the env configs, and follow the steps up until stopping etcd. After that, remove the broken node from the cluster using its old id, $ etcdctl member remove e6e3c5cbabef648a, then follow the replace & start procedure.
I've replaced a handful of broken nodes with these steps, so i'd be surprised if it didn't work.
Fwiw, the commands I gave you differ a little from the doc you shared, there's some additional source'ing going on to pull in some env vars there are otherwise missed.

@dominicgunn Yeah, I noticed that there was an extra env file that you included in your command :)
So, following the steps, and running etcdctl member remove dab84c2963a2830f gives the output: Couldn't find a member in the cluster with an ID of dab84c2963a2830f.
Then continuing with etcdadm replace and starting etcd-member.service, the logs show the same id for this broken node
...
Jun 26 09:01:07 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[17055]: 2019-06-26 09:01:07.108451 I | etcdserver/membership: added member dab84c2963a2830f [https://ip-10-1-43-133.eu-west-1.compute.internal:2380] to cluster e146e62db33b7661
Jun 26 09:01:07 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[17055]: 2019-06-26 09:01:07.108368 I | etcdserver/membership: added member c8a02416947a5853 [https://ip-10-1-45-117.eu-west-1.compute.internal:2380] to cluster e146e62db33b7661
Jun 26 09:01:07 ip-10-1-43-48.eu-west-1.compute.internal etcd-wrapper[17055]: 2019-06-26 09:01:07.108245 I | etcdserver/membership: added member 268d5cbd6382d8f2 [https://ip-10-1-44-127.eu-west-1.compute.internal:2380] to cluster e146e62db33b7661
...

I'm not that familiar with etcd, but to me, it kinda looks like the id is cached on the node so the etcd-member.service thinks it's still the old node trying to join.
But then again, I'm just guessing here.

I created a fresh 0.13-rc1 cluster, the cluster did launch without errors.
The tried to upgrade to 0.13-rc2 by
Replacing the kube-aws binary
run ./kube-aws render stack
run ./kube-aws apply --pretty-print
Same problem. etcd0 fails to execute cfn-signal and start a rollback. The rollback etcd0 instances fail to start.

I've tried the replace procedure you listed above on the new cluster that also failed @dominicgunn, and the output is as follows:
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal systemd[1]: Stopped etcd (System Application Container).
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal systemd[1]: etcd-member.service: Succeeded.
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal rkt[31888]: pod "4352a8f3-3476-4d55-acf5-ebc5438273c5" is already stopped
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal systemd[1]: Started etcd (System Application Container).
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.016861 I | rafthttp: stopped peer 70f28abc02f5566f
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.016855 I | rafthttp: stopped streaming with peer 70f28abc02f5566f (stream Message reader)
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.016848 I | rafthttp: stopped streaming with peer 70f28abc02f5566f (stream MsgApp v2 reader)
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.016839 I | rafthttp: stopped HTTP pipelining with peer 70f28abc02f5566f
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.016793 I | rafthttp: stopped streaming with peer 70f28abc02f5566f (writer)
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.016785 I | rafthttp: stopped streaming with peer 70f28abc02f5566f (writer)
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.016776 I | rafthttp: stopping peer 70f28abc02f5566f...
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.016770 I | rafthttp: stopped peer 123c3adfe7b1a0db
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.016764 I | rafthttp: stopped streaming with peer 123c3adfe7b1a0db (stream Message reader)
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.016753 I | rafthttp: stopped streaming with peer 123c3adfe7b1a0db (stream MsgApp v2 reader)
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.016689 I | rafthttp: stopped HTTP pipelining with peer 123c3adfe7b1a0db
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.013897 I | rafthttp: stopped streaming with peer 123c3adfe7b1a0db (writer)
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.013870 I | rafthttp: stopped streaming with peer 123c3adfe7b1a0db (writer)
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.013851 I | rafthttp: stopping peer 123c3adfe7b1a0db...
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.013807 I | etcdserver: aborting publish because server is stopped
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.013762 I | etcdserver: the data-dir used by this member must be removed.
Jun 26 12:28:48 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:48.013727 E | etcdserver: the member has been permanently removed from the cluster
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.989592 E | rafthttp: streaming request ignored (ID mismatch got 1a62be873a3c997c want 34082a3956203c13)
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.989347 E | rafthttp: streaming request ignored (ID mismatch got 1a62be873a3c997c want 34082a3956203c13)
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.989134 E | rafthttp: streaming request ignored (ID mismatch got 1a62be873a3c997c want 34082a3956203c13)
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.988684 I | raft: raft.node: 34082a3956203c13 elected leader 123c3adfe7b1a0db at term 9
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.988675 I | raft: 34082a3956203c13 became follower at term 9
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.988646 I | raft: 34082a3956203c13 [term: 1] received a MsgHeartbeat message with higher term from 123c3adfe7b1a0db [term: 9]
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.988358 E | rafthttp: streaming request ignored (ID mismatch got 1a62be873a3c997c want 34082a3956203c13)
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.987546 I | rafthttp: started streaming with peer 70f28abc02f5566f (stream Message reader)
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.987330 I | rafthttp: started streaming with peer 70f28abc02f5566f (stream MsgApp v2 reader)
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.983359 I | embed: ClientTLS: cert = /etc/ssl/certs/etcd.pem, key = /etc/ssl/certs/etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/etcd-trusted-ca.pem, client-cert-auth = true
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.982484 I | rafthttp: started streaming with peer 70f28abc02f5566f (writer)
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.982372 I | etcdserver: starting server... [version: 3.2.13, cluster version: to_be_decided]
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.982283 I | rafthttp: added peer 70f28abc02f5566f
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.982188 I | rafthttp: started peer 70f28abc02f5566f
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.982051 I | rafthttp: started streaming with peer 70f28abc02f5566f (writer)
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.981874 I | rafthttp: started streaming with peer 123c3adfe7b1a0db (stream MsgApp v2 reader)
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.981446 I | rafthttp: started streaming with peer 123c3adfe7b1a0db (stream Message reader)
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.981085 I | rafthttp: started HTTP pipelining with peer 70f28abc02f5566f
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.981002 I | rafthttp: starting peer 70f28abc02f5566f...
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.980865 I | rafthttp: added peer 123c3adfe7b1a0db
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.977715 I | rafthttp: started peer 123c3adfe7b1a0db
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.960965 I | rafthttp: started streaming with peer 123c3adfe7b1a0db (writer)
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.954896 I | rafthttp: started streaming with peer 123c3adfe7b1a0db (writer)
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.942874 I | rafthttp: started HTTP pipelining with peer 123c3adfe7b1a0db
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.942675 I | rafthttp: starting peer 123c3adfe7b1a0db...
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.922510 W | auth: simple token is not cryptographically signed
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.907041 I | mvcc: restore compact to 95296
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.900823 I | etcdserver/membership: added member 123c3adfe7b1a0db [https://ip-10-1-45-203.eu-west-1.compute.internal:2380] to cluster 766b2c59c152b9ce from store
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.900808 I | etcdserver/membership: added member 70f28abc02f5566f [https://ip-10-1-44-92.eu-west-1.compute.internal:2380] to cluster 766b2c59c152b9ce from store
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.900786 I | etcdserver/membership: added member 34082a3956203c13 [https://ip-10-1-43-74.eu-west-1.compute.internal:2380] to cluster 766b2c59c152b9ce from store
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.900597 I | raft: newRaft 34082a3956203c13 [peers: [123c3adfe7b1a0db,34082a3956203c13,70f28abc02f5566f], term: 1, commit: 3, applied: 3, lastindex: 3, lastterm: 1]
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.900569 I | raft: 34082a3956203c13 became follower at term 1
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.900504 I | etcdserver: restarting member 34082a3956203c13 in cluster 766b2c59c152b9ce at commit index 3
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.839875 I | etcdserver: advertise client URLs = https://ip-10-1-43-74.eu-west-1.compute.internal:2379
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.839783 I | etcdserver: snapshot count = 100000
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.839717 I | etcdserver: election = 1000ms
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.839650 I | etcdserver: heartbeat = 100ms
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.839573 I | etcdserver: member dir = /var/lib/etcd/member
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.839494 I | etcdserver: data dir = /var/lib/etcd
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.839315 I | etcdserver: name = etcd0
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.828274 I | mvcc: restore compact to 95296
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.812532 I | etcdserver: recovered store from snapshot at index 3
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.799192 I | embed: listening for client requests on <IP_ADDRESS>:2379
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.799009 I | embed: listening for peers on https://<IP_ADDRESS>:2380
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.794017 I | embed: peerTLS: cert = /etc/ssl/certs/etcd.pem, key = /etc/ssl/certs/etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/etcd-trusted-ca.pem, client-cert-auth = false
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.793859 N | etcdmain: the server is already initialized as member before, starting as etcd member...
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.793738 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.793669 I | etcdmain: Go OS/Arch: linux/amd64
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.793602 I | etcdmain: Go Version: go1.8.5
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.793529 I | etcdmain: Git SHA: 95a726a
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.793455 I | etcdmain: etcd Version: 3.2.13
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.793358 W | pkg/flags: unrecognized environment variable ETCD_IMAGE_TAG=v3.2.13
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.793289 W | pkg/flags: unrecognized environment variable ETCD_USER=etcd
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.793191 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd-trusted-ca.pem
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.793097 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd-trusted-ca.pem
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.793015 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd-key.pem
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.792922 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd.pem
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.792847 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd0
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.792760 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://<IP_ADDRESS>:2380
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.792673 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://<IP_ADDRESS>:2379
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.792590 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/etc/ssl/certs/etcd-key.pem
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.791386 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.791319 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd0=https://ip-10-1-43-74.eu-west-1.compute.internal:2380,etcd1=https://ip-10-1-44-92.eu-west-1.compute.internal:2380,etcd2=https://ip-10-1-45-203.eu-west-1.compute.internal:2380
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.791248 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://ip-10-1-43-74.eu-west-1.compute.internal:2380
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.791161 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.791074 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.791002 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/etc/ssl/certs/etcd.pem
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.790919 I | pkg/flags: recognized and used environment variable ETCD_AUTO_COMPACTION_RETENTION=1
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: 2019-06-26 12:28:47.790675 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://ip-10-1-43-74.eu-west-1.compute.internal:2379
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: + exec /usr/bin/rkt run --uuid-file-save=/var/lib/coreos/etcd-member-wrapper.uuid --trust-keys-from-https --mount volume=coreos-systemd-dir,target=/run/systemd/system --volume coreos-systemd-dir,kind=host,source=/run/systemd/system,readOnly=true --mount volume=coreos-notify,target=/run/systemd/notify --volume coreos-notify,kind=host,source=/run/systemd/notify --set-env=NOTIFY_SOCKET=/run/systemd/notify --volume coreos-data-dir,kind=host,source=/var/lib/etcd2,readOnly=false --volume coreos-etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true --volume coreos-usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true --volume coreos-etc-hosts,kind=host,source=/etc/hosts,readOnly=true --volume coreos-etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true --mount volume=coreos-data-dir,target=/var/lib/etcd --mount volume=coreos-etc-ssl-certs,target=/etc/ssl/certs --mount volume=coreos-usr-share-certs,target=/usr/share/ca-certificates --mount volume=coreos-etc-hosts,target=/etc/hosts --mount volume=coreos-etc-resolv,target=/etc/resolv.conf --inherit-env --stage1-from-dir=stage1-fly.aci quay.io/coreos/etcd:v3.2.13 --user=232 --
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal etcd-wrapper[31864]: ++ id -u etcd
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal systemctl[31860]: active
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal systemctl[31858]: active
Jun 26 12:28:47 ip-10-1-43-221.eu-west-1.compute.internal rkt[31847]: "f13fba88-4a37-4e99-94e9-9c3e9ccd98ba"
Jun 26 12:28:46 ip-10-1-43-221.eu-west-1.compute.internal systemd[1]: Starting etcd (System Application Container)...

From the logs, etcd is adding the node with it's old id, 34082a3956203c13. Then it gets the new id 1a62be873a3c997c, but rejects it since it thinks it's looking for the old id 34082a3956203c13.

I think I can finally throw some light on this issue
Jun 26 15:01:10 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[20903]: Unable to find image 'quay.io/coreos/etcd:vv3.2.26' locally
This is from the node that fails to start during update, and that eventually will trigger a rollback. Look at the etcd image. the double v in the version tag. This does only happen during upgrade, because launching a new cluster with kube-aws 0.13-rc2 does work.

Jun 26 15:10:37 ip-10-1-43-147.eu-west-1.compute.internal systemd[1]: Failed to start etcdadm reconfigure runner.
Jun 26 15:10:37 ip-10-1-43-147.eu-west-1.compute.internal systemd[1]: etcdadm-reconfigure.service: Failed with result 'exit-code'.
Jun 26 15:10:37 ip-10-1-43-147.eu-west-1.compute.internal systemd[1]: etcdadm-reconfigure.service: Main process exited, code=exited, status=125/n/a
Jun 26 15:10:37 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: See '/run/torcx/bin/docker run --help'.
Jun 26 15:10:37 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /run/torcx/bin/docker: Error response from daemon: manifest for quay.io/coreos/etcd:vv3.2.26 not found.
Jun 26 15:10:36 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: Unable to find image 'quay.io/coreos/etcd:vv3.2.26' locally
Jun 26 15:10:36 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: info: member_restore_from_local_snapshot: restoring etcd0
Jun 26 15:10:36 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: info: member_clean_data_dir: no files found in /var/lib/etcd2
Jun 26 15:10:36 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: info: member_clean_data_dir: data dir /var/lib/etcd2 exists. finding files to remove
Jun 26 15:10:36 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: info: member_clean_data_dir: cleaning data dir of etcd0
Jun 26 15:10:36 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: info: member_bootstrap: backup found. restoring etcd0...
Jun 26 15:10:36 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: info: member_local_snapshot_exists: checking existence of file /var/run/coreos/etcdadm/snapshots/etcd0.db
Jun 26 15:10:36 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: info: member_local_snapshot_exists: checking existence of file /var/run/coreos/etcdadm/snapshots/etcd0.db
Jun 26 15:10:36 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: [1.0K blob data]
Jun 26 15:10:35 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: info: member_download_snapshot: downloading /var/run/coreos/etcdadm/snapshots/etcd0.db from s3://gdo-kubernetes/k8s7/kube-aws/clusters/k8s7/instances/ca3b0cf0-981a-11e9-9b11-0ae78d106d3e/etcd-snapshots/snapshot.db
Jun 26 15:10:35 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: info: member_remote_snapshot_exists: s3://gdo-kubernetes/k8s7/kube-aws/clusters/k8s7/instances/ca3b0cf0-981a-11e9-9b11-0ae78d106d3e/etcd-snapshots/snapshot.db exists
Jun 26 15:10:34 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: 2019-06-26 15:09:26    4382752 snapshot.db
Jun 26 15:10:34 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: info: member_remote_snapshot_exists: checking existence of s3://gdo-kubernetes/k8s7/kube-aws/clusters/k8s7/instances/ca3b0cf0-981a-11e9-9b11-0ae78d106d3e/etcd-snapshots/snapshot.db
Jun 26 15:10:34 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: info: member_reconfigure: only 0 of 3 nodes for etcd members are running, which means cluster is still in bootstrap process. searching for a etcd snapshot to recover this member
Jun 26 15:10:34 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: info: member_set_unit_type: setting etcd unit type to "simple". `systemctl daemon-reload` required afterwards
Jun 26 15:10:34 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: info: member_reconfigure: 3 more nodes are required until the quorum is met
Jun 26 15:10:34 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: error: cluster_num_running_nodes:  not found
Jun 26 15:10:34 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: line 153: tester_num_running_nodes_file: command not found
Jun 26 15:10:34 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /opt/bin/etcdadm: info: member_reconfigure: observing cluster state: quorum=2 healthy=0
Jun 26 15:10:34 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: See '/run/torcx/bin/docker run --help'.
Jun 26 15:10:34 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /run/torcx/bin/docker: Error response from daemon: manifest for quay.io/coreos/etcd:vv3.2.26 not found.
Jun 26 15:10:33 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: Unable to find image 'quay.io/coreos/etcd:vv3.2.26' locally
Jun 26 15:10:33 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: See '/run/torcx/bin/docker run --help'.
Jun 26 15:10:33 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /run/torcx/bin/docker: Error response from daemon: manifest for quay.io/coreos/etcd:vv3.2.26 not found.
Jun 26 15:10:32 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: Unable to find image 'quay.io/coreos/etcd:vv3.2.26' locally
Jun 26 15:10:31 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: See '/run/torcx/bin/docker run --help'.
Jun 26 15:10:31 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: /run/torcx/bin/docker: Error response from daemon: manifest for quay.io/coreos/etcd:vv3.2.26 not found.
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: Unable to find image 'quay.io/coreos/etcd:vv3.2.26' locally
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal sudo[3638]: pam_unix(sudo:session): session closed for user root
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal sudo[3638]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal sudo[3638]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/[ -w /var/lib/etcd2 ]
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal sudo[3633]: pam_unix(sudo:session): session closed for user root
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal sudo[3633]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal sudo[3633]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/[ -w /var/run/coreos/etcdadm/snapshots ]
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal sudo[3616]: pam_unix(sudo:session): session closed for user root
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal sudo[3616]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal sudo[3616]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/[ -w /var/run/coreos/etcdadm ]
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: declare -x ETCDCTL_KEY_FILE="/etc/ssl/certs/etcd-client-key.pem"
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: declare -x ETCDCTL_KEY="/etc/ssl/certs/etcd-client-key.pem"
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: declare -x ETCDCTL_CERT_FILE="/etc/ssl/certs/etcd-client.pem"
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: declare -x ETCDCTL_CERT="/etc/ssl/certs/etcd-client.pem"
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: declare -x ETCDCTL_CA_FILE="/etc/ssl/certs/etcd-trusted-ca.pem"
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal etcdadm[3610]: declare -x ETCDCTL_CACERT="/etc/ssl/certs/etcd-trusted-ca.pem"
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal systemctl[3606]: active
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal systemctl[3604]: active
Jun 26 15:10:30 ip-10-1-43-147.eu-west-1.compute.internal systemd[1]: Starting etcdadm reconfigure runner...

This line is the source of the problem i think.
https://github.com/HotelsDotCom/kube-aws/blob/fc29a9f5f46872cdb72ccab9f9b767807ac8eadc/build#L9
should be
ETCD_VERSION="3.2.26"
not
ETCD_VERSION="v3.2.26"
The etcd version in the cluster.yaml under etcd.version is specified without a v
https://github.com/HotelsDotCom/kube-aws/blob/fc29a9f5f46872cdb72ccab9f9b767807ac8eadc/builtin/files/cluster.yaml.tmpl#L765
This does explain why the etcdadm-reconfigure.service fails to execute during a cluster upgrade, but it does not explain why the rollback operations fails.

This is how etcd version is specified on line 88 of etcdadm
etcd_version=${ETCD_VERSION:-3.2.13}
The member_restore_from_local_snapshot() function in etcdadm eventually will be executed by the /opt/bin/etcdadm reconfigure command in the etcdadm-reconfigure.service. This function tries to download the etcd docker image defined on line 514 as quay.io/coreos/etcd:v$etcd_version.

Should be fixed in https://github.com/kubernetes-incubator/kube-aws/pull/1645

@davidmccormick , is this more in line with your desired way of handling etcd versions?
https://github.com/kubernetes-incubator/kube-aws/pull/1646

Closing this issue as resolved, following these merged PRs, thanks for the work @paalkr and @andersosthus!
https://github.com/kubernetes-incubator/kube-aws/pull/1646
https://github.com/kubernetes-incubator/kube-aws/pull/1658
https://github.com/kubernetes-incubator/kube-aws/pull/1657

There is still the issue that when the upgrade failed, it did not auto-revert etcd, that this issue was originally about. Also, we haven't been able to get that etcd node that's failing back in the etcd cluster.

@dominicgunn I managed to fix the broken etcd node a few weeks ago.
I basically did the same as the etcdctl replace, but manually doing each step in the script instead of running the script. That worked, and the node recovered. Based on this, I think there might be a subtle bug in etcdctl replace, but I haven't looked for it.
I could try to create a new cluster using 0.13 rc1, upgrade with 0.13 rc2 to reproduce this issue and see if I can figure out what the issue with the script is.

I appreciate you continuing to look into this @andersosthus, is there is a bug in the script getting it fixed would be great!


loss function matching between the paper and the project implementation

I have a mismatch between the theoretical loss function to the way it's implemented in the project.
my questions are:

where is the loss function separation to layers?
where can I see the complete loss process (cos function, lambda parameter, the l_1 loss and the weighted sum.
attaching screenshots.

Thanks!
Almog.

Hello!
I believe you might be tracing the irrelevant code, the loss function in the paper is about DistilHubert and you will need to go to the s3prl/pretrain/distiller folder. For more specific usage, @vectominist is the author of DistilHubert and he might give you more details.

You can find them here:
https://github.com/s3prl/s3prl/blob/26113e9febbaaee6311f8c2cdd9475dc495acce9/s3prl/pretrain/distiller/pretrain_expert.py#L330-L337


Serialize built-in reflecting properties

Description
lit-ssr needs to include a whitelist of built-in reflecting properties on built-in nodes (including those on HTMLElement) and automatically serialize property bindings to those properties to their corresponding attributes.
Acceptance Criteria

Tests that demonstrate correct serialization of built-in properties to attributes are checked in and passing.

This is fixed now.


Support of new Raml 1.0 ?

Will the last version of Raml 1.0 will be supported soon ?

@xGouley We plan to add support for it in future but not in next couple weeks.
I can't provide any estimates since I didn't research 1.0 yet.

Any news on this? 😄

Not so far. If you have an immediate requirement, you could try https://apimatic.io/transformer

@bobby-brennan what do you think to stripping out the current RAML 0.8 support and using https://github.com/mulesoft/oas-raml-converter ? Or is it too heavy-weight to include a whole converter framework?
I think it would work well with your recent transitive conversion changes and give us another output format.

Using mulesoft/oas-raml-converter sounds like the way to go. I think Mulesoft is more or less the owner of RAML, so we might as well use their solution.

First pass using oas-raml-converter here: https://github.com/LucyBot-Inc/api-spec-converter/compare/raml_1?expand=1
It works for the most part, but fails in the browser due to an fs.readFileSync. Not sure what the right solution is.

It works for the most part, but fails in the browser due to an fs.readFileSync. Not sure what the right solution is.

Are you stubbing out fs with something like browserify-fs and it's just the sync variant that's the problem? Maybe it can be fixed upstream?

Update - waiting on this PR to add URL support:
https://github.com/mulesoft/oas-raml-converter/pull/19


os: windows processes started with foreign token inherit the wrong environment [CVE-2019-11888]

Passing a nil environment but a non-nil token to os.CreateProcess results in the new potentially unprivileged process inheriting the parent potentially privileged environment, or results in the new potentially privileged process inheriting the parent potentially unprivileged environment. In the former case, it's an infoleak. In the latter case, it's a possible EoP, since things like PATH could be overwritten.

/cc @alexbrainman @ianlancetaylor

Should probably backport this.

@gopherbot Please open a backport to 1.12.


Corrected definition for set full_refresh_mode

resolves #260
Description
The current definition of the file dbt-spark\dbt\include\spark\macros\materializations\incremental\incremental.sql has an errouneous implementation of set full_refresh_mode that does not take into consideration the definition that is set in the config.
This PR corrects this issue by replacing it with the implementation in dbt-core/dbt-snowflake/etc.
As this PR only brings dbt-spark in sync with the other repos, and it's only one line of code, I figured tests are not relevant for this PR.
Checklist

[X] I have signed the CLA
[X] I have run this code in development and it appears to resolve the stated issue
[X] This PR includes tests, or tests are not required/relevant for this PR
[X] I have updated the CHANGELOG.md and added information about my change to the "dbt-spark next" section.

@cla-bot check:


groff: add conflicts_with mdocml

Please note that this change should go along with homebrew #35790.

The mandoc that was merged renames the one conflicting manpage, so I don't think we need this after all. Thanks!


Adding Witness

I didnt add the part about the balance between the attention and witness part of the score as we don't have the numbers yet. Also deleted the proof of life part since that comes from periodical requirement to re-submit POI.

attention and witness == redundant concept. removing this.


Fix issue of job continuing although 'Run Tests' step failed

Proposed changes:

...

Status (please check what you already did):

[ ] added some tests for the functionality
[ ] updated the documentation
[ ] updated the changelog (please check changelog for instructions)
[ ] reformat files using black (please check Readme for instructions)

/modeltest
include:
 - dataset: ["financial-demo"]
   config: ["Sparse + BERT + DIET(seq) + ResponseSelector(t2t)"]


OSS Console - Router wallet not returned, impossible to burn DCs

Since update to version 2022.01.27 It not anymore possible to burn DCs for me
The api does not returns the router wallet ID and the the transaction can't be generated. (memo is correctly generated)
https://console.helium-iot.eu/api/data_credits/router_address return code 200
but body is:
{"address":""}

hi, team is continuing to investigate..

Try kicking router container while console container is up.

It has no effect

I also update to the last release. No effect

The team is working on a patch to address this issue. Thanks for raising.

@disk91 Do you have problem that it:

never returns address
or
sometimes returns (it can join and send some uplink), and later it does not return?

Never returns ... at least all the time I have tested
You can do test on console.helium-iot.eu if you want ... it's open.


cocodataset

After coping the project how do I prepare for coco dataset?

You can follow the instruction in Readme to prepare your dataset.


Does flutter expandable recycle views?

I building an app that have a screen with ExpandablePanel inside another ExpandablePanel with big lists of Strings (Images (small sizes) and texts), I would like to know if expandable recycle views or my app will probably crash, but in that screen I'm building I needed to show all data.
ExpandablePanel -> ExpandablePanel -> Header -> Expanded
Rows (Images and texts) ->  Rows (Images and texts) -> Description (big and small texts)
Thank you.

Expandable accepts two widgets: expanded and collapsed. These widgets are reused between the states of Expandable.

Thank you.


fix: trying to fix test segfaults

ohhhh, numpy 2.0 just hit pypi!  :joy:  that's almost certainly what's going on here

yep, it's tensorstore, which has been fixed, but not yet released:
https://github.com/google/tensorstore/issues/165
https://github.com/google/tensorstore/commit/3c9b9c0d516b92d60f6ac2ac05a11ff2a1502eee
pinning to numpy<2 for now

I just tagged v0.1.62 which will be built with numpy2 support.  So you should be able to test this soon.

Thank you stopping by @laramiel!  Much appreciated


Add option to trace metrics messages

Closing this in favour of enhancing spring-metrics (which will overtake the current functionality in Stream).


Support viewing .scr files

One way to do that could be converting it internally to a snapshot running an endless loop.

Supported in e5e7be5.


Support for enqueue/enqueue-bundle ^0.9

Is there any specific reason why there isn't support for enqueue/enqueue-bundle[0.9.x-dev]?

The bundle has not been released yet. There are some BC breaks since 0.8
So it might not as easy as allowing new version in composer.json


On Screen QR Code follow

Description of the new feature / enhancement
It would be really nice to have the ability to click on-screen QR codes (e.g., from a pdf or a picture) and be able to follow them without using a 3rd party app.
Scenario when this would be used?
I have a pdf that contains a QR code and want to follow the link to the page without scanning it with my phone.
Supporting information
No response

/dup #4859


adds a description; license; link to c4c

Getting the readme going.. addressing issue #2
Signed-off-by: Mike Brown<EMAIL_ADDRESS>

/lgtm
/approve


Fixes for security reviews script

Fixes to keep security reviews section of the dashboard in sync. Changes include minor bug fixes, improved parsing (included more fields/metadata from security review repository), and added abstraction for readability.

The JSON Model needs a couple of minor tweaks Ito support this. I'd be happy to send the JSON Model generated from my dashboard if necessary.

Already fixed, closing this PR.


multi_get, multi_set not working with ssdb 1.9.2

Hi,
Have you tried to use this library with ssdb 1.9.2?
When you lookup multiple keys together the multi_get function fails, returning two undefined,
conn.multi_get ([
  '11832126', '11832149'], function (err, data) {
    console.log (err, data);
  })

But it works with [11832126] or [11832149]
Can you check that?

Please ignore this. The official documents says SSDB is compatible with redis protocol. Yes, node-redis works great.


In Windows 10, it didn't find the ssh command, cannot run ssh<EMAIL_ADDRESS>Does it only run in linux?

In Windows 10, it didn't find the ssh command, cannot run ssh<EMAIL_ADDRESS>Does it only run in linux?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3b81a147-db1c-b92f-88d1-6dd0991a72fe
Version Independent ID: f5a2c949-498f-5848-d89c-031d1a757120
Content: SSH into Azure Container Service (AKS) cluster nodes
Content Source: articles/aks/aks-ssh.md
Service: container-service
GitHub Login: @neilpeterson
Microsoft Alias: nepeters

@jianhuatian - in Windows you will either need to use an SSH client like putty, or use the Windows Subsystem for Linux.
We are working on some docs platform changes that will allow us to include both Linux and Windows experiences in the same doc. Once these changes have occurred this doc will be updated to include some differentiation and Windows specific instruction.
Does this help?

@jianhuatian Thanks for the feedback! I have assigned the issue to the content author to investigate further and update the document as appropriate.

az vm list-ip-addresses --resource-group MC_myAKSCluster_myAKSCluster_eastus -o table always gives no o/p . I tried with a lot of options. Nothing worked. Can you help?


Add env support to Dockerfile

In cases where Molecule runs behind a http proxy, the
image needs proxies set.  Taken from Renaud Martinet's
blog post.
Signed-off-by: John Dewey<EMAIL_ADDRESS>PR Type

Feature Pull Request

Hi @retr0h, sorry but we will need the commits with --signoff for DCO matter.


isNumeric implementation seems overkill

In the past I've used isNumeric impls like:
function isNumeric(val) {
  return !isNaN(parseFloat(val)) && isFinite(val);
}

what we currently have is here:
var is_array = Array.isArray;

export default function isNumeric(val) {
  return !is_array(val) && (val - parseFloat(val) + 1) >= 0;
};

I think the is_array check and the adding 1 bit is dubious. @trxcllnt do you have any reasoning/examples of why this is necessary?
I don't think this is something that is necessarily impacting performance meaningfully, but wanted to hash out what it was doing and why, so we could make sure the code is doing things that make sense.

@blesh this function was lifted from jQuery, and afaik is the best way to check if a value can be cast to a Number.

@trxcllnt strange, it makes me wonder if this is for supporting some crazy old IE implementations or something?

This isn't really an issue, right now. Closing.


When running the script, the following error occurs

The following error occurred: DGL backend not selected or invalid. Assuming PyTorch for now.
Setting the default backend to "pytorch". You can change it in the ~/.dgl/config.json file or export the DGLBACKEND environment variable. Valid options are: pytorch, mxnet, tensorflow (all lowercase).
/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/torchdata/datapipes/init.py:18: UserWarning:
################################################################################
WARNING!
The 'datapipes', 'dataloader2' modules are deprecated and will be removed in a
future torchdata release! Please see https://github.com/pytorch/data/issues/1196
to learn more and leave feedback.
################################################################################
deprecation_warning()
Traceback (most recent call last):
File "scripts/map_reaction_smiles_using_indigo.py", line 8, in
from atom_to_atom_mapping.utility import IndigoAtomToAtomMappingUtility
File "/home/top/atom-to-atom-mapping/atom_to_atom_mapping/utility/init.py", line 9, in
from atom_to_atom_mapping.utility.micc_snu.local_mapper import LocalMapperAtomToAtomMappingUtility
File "/home/top/atom-to-atom-mapping/atom_to_atom_mapping/utility/micc_snu/init.py", line 3, in
from atom_to_atom_mapping.utility.micc_snu.local_mapper import LocalMapperAtomToAtomMappingUtility
File "/home/top/atom-to-atom-mapping/atom_to_atom_mapping/utility/micc_snu/local_mapper.py", line 7, in
from localmapper import localmapper
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/localmapper/init.py", line 6, in
from .localmapper import localmapper
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/localmapper/localmapper.py", line 11, in
from .utils import *
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/localmapper/utils.py", line 3, in
import dgl
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/dgl/init.py", line 16, in
from . import (
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/dgl/dataloading/init.py", line 13, in
from .dataloader import *
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/dgl/dataloading/dataloader.py", line 27, in
from ..distributed import DistGraph
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/dgl/distributed/init.py", line 5, in
from .dist_graph import DistGraph, DistGraphServer, edge_split, node_split
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/dgl/distributed/dist_graph.py", line 11, in
from .. import backend as F, graphbolt as gb, heterograph_index
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/dgl/graphbolt/init.py", line 55, in
load_graphbolt()
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/dgl/graphbolt/init.py", line 45, in load_graphbolt
raise FileNotFoundError(
FileNotFoundError: Cannot find DGL C++ graphbolt library at /home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/dgl/graphbolt/libgraphbolt_pytorch_2.4.1.so

And I downgraded PyTorch to version 2.2.0, and encountered the following error when running the command:
python scripts/map_reaction_smiles_using_indigo.py
--reaction_smiles "[O-]N+c1ccc(Br)cn1.CC(=O)Nc1ccc(O)cc1>>CC(=O)Nc1ccc(Oc2ccc(nc2)N+=O)cc1"
DGL backend not selected or invalid. Assuming PyTorch for now.
Setting the default backend to "pytorch". You can change it in the ~/.dgl/config.json file or export the DGLBACKEND environment variable. Valid options are: pytorch, mxnet, tensorflow (all lowercase)
Traceback (most recent call last):
File "scripts/map_reaction_smiles_using_indigo.py", line 8, in
from atom_to_atom_mapping.utility import IndigoAtomToAtomMappingUtility
File "/home/top/atom-to-atom-mapping/atom_to_atom_mapping/utility/init.py", line 9, in
from atom_to_atom_mapping.utility.micc_snu.local_mapper import LocalMapperAtomToAtomMappingUtility
File "/home/top/atom-to-atom-mapping/atom_to_atom_mapping/utility/micc_snu/init.py", line 3, in
from atom_to_atom_mapping.utility.micc_snu.local_mapper import LocalMapperAtomToAtomMappingUtility
File "/home/top/atom-to-atom-mapping/atom_to_atom_mapping/utility/micc_snu/local_mapper.py", line 7, in
from localmapper import localmapper
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/localmapper/init.py", line 6, in
from .localmapper import localmapper
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/localmapper/localmapper.py", line 11, in
from .utils import *
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/localmapper/utils.py", line 3, in
import dgl
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/dgl/init.py", line 16, in
from . import (
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/dgl/dataloading/init.py", line 13, in
from .dataloader import *
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/dgl/dataloading/dataloader.py", line 27, in
from ..distributed import DistGraph
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/dgl/distributed/init.py", line 5, in
from .dist_graph import DistGraph, DistGraphServer, edge_split, node_split
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/dgl/distributed/dist_graph.py", line 11, in
from .. import backend as F, graphbolt as gb, heterograph_index
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/dgl/graphbolt/init.py", line 8, in
from .base import *
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/dgl/graphbolt/base.py", line 8, in
from torchdata.datapipes.iter import IterDataPipe
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/torchdata/datapipes/init.py", line 11, in
from . import iter, map, utils
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/torchdata/datapipes/iter/init.py", line 79, in
from torchdata.datapipes.iter.util.cacheholder import (
File "/home/top/anaconda3/envs/atom-to-atom-mapping-env/lib/python3.8/site-packages/torchdata/datapipes/iter/util/cacheholder.py", line 24, in
from torch.utils._import_utils import dill_available
ModuleNotFoundError: No module named 'torch.utils._import_utils'

Thank you for the interest in the repository. The error you are encountering has to do with the DGL library (it has trouble detecting the backend) which is required for the localmapper tool. As we are simply installing localmapper using pip, there is nothing to be done when it comes to this repository. I suggest going to the original repository (https://github.com/snu-micc/LocalMapper) and figuring out how to install DGL correctly on your local machine, and then simply adjusting the versions accordingly for this environment as well. Hope this helps.


Update gfx

Fixes the screen resolution and mouse coordinates on macOS
Fixes the blurry font by using linear filtering on the texture sampler
Fixes the default color scheme when using gfx

The second patch addresses #54

Thanks! :smile:


Object isn't selceted after cancel save process

I remember on an older version while the save process it was possible to hit one of the changed/added objects (a potential hidden object is coming to visual area) and after stopping the save function the object was selected.
Now with the current version selecting and bringing the object to the screen area is still working, but after quitting the save procedure the object isn't selected anymore.
This was quite comfortable to do an easy double check for any matter.
See movie, at second 21 the traffig_sign poi was moved to center, but isn't active/selected (so it is not possible to immediately edit the traffig_sign tags on the left window side ).
If there is a view of many poi's, it isn't easy to find the choosen object.

I'm not sure if/when this was changed or if it was intentional, but it seems to me that neither workflow is great. The purpose of listing the changes in the save screen is so the mapper can review them before uploading. But this is hard since the map is greyed out and selecting a feature would exit the list.
I'd like to look into a less rigid review/save workflow that maybe isn't its own mode. #6671 is also about improving the save screen.


Add sync adapter access

Fixes the issues found in #13
Created a new pull request because github doesn't allow pull requests to be updated unless the user has write access to that person's repository.

I think this can be merged. Could you please provide a small example that shows how to use this feature from a SyncAdapter? Something we could add to the readme?


Socket subscribe events do not call toJSON

I'm using sails v0.10.5, and subscribing on model events that have a toJSON seem to indicate that the toJSON is only called for blueprint update actions (see https://github.com/balderdashy/sails/blob/master/lib/hooks/blueprints/actions/update.js#L72 - this is the only spot I could find to do with pubsub that calls toJSON).
I'm pretty sure this isn't expected behavior, but maybe I'm wrong?  We can work with you more easily from there to get some tests in.

Submitted the PR for you @tjwebb. Let me know if you need anything else.


Allow platform option for NodeJS transpilation

Simple change so esbuild-loader can also be used for NodeJS (but with browser as default target)

Sorry, I didn't see the default branch was not develop


WHERE IS MY NOTTE??

Add regexp to notte, Buonanotte, bouna notte
With answers thats comes with "ночи".
And... You already know shen_mue with his "привет жопы"
Add like "привет жопик" or something sumilar. Ehehehehe

Reject issue due to lack of importance.


Fix micromamba tar command for macOS

This has been merged into development and is now proposed as a hotfix to main
Moved the -O from after the file to after the tar command for compatibility with macOS
Signed-off-by: Kevin Coakley<EMAIL_ADDRESS>

This closes #1500

I think the best plan is to let me rebase my incoming installer hotfix hotfixes (😦) onto this, then make a hotfix release (2.1.4) of the whole thing. This would take care of all known issues in the Unified Installer - I haven't touched anything outside of that.
The only things I'm waiting on are

from @damian0815

M1 hashes for torch==1.12.0
signoff on a MacOS change to the invoke script
test the whole business on M1


Fix: FlatFileDriver does not notice new messages

FlatFileDriver does not notice new messages until the polling duration ends.
So we should update the $files. 💡

I think we should check if the problem still persists after merge of #308 and merge this after!

@acrobat #308 seems it is not active. 👀 Would you please merge this as before #308 ?

@ackintosh I will take a look at this PR later today!

Sorry for the delay! Changes look good to me, but can you add a test for this behavior (messages added after the initial collect)?

@acrobat @sagikazarmark Thank you for your review! I replaced Snidel with pcntl_fork.

I fixed it!

After going trough this PR I realized that it probably affects performance as well since between the loops it reads the whole file list again.
I would love to see some benchmarks comparing this version to the previous one, just to make sure it does not introduce a performance regression.
But I don't think it blocks merging @acrobat !

I will try to do some benchmarks before merging, just to be sure we don't have (huge) performance regressions or if you are able to do some tests (or blackfire profiles) @ackintosh would be nice to! (Not sure if I will find time before mid next week or so)

I would not say that this could cause a performance bug because it is only reading the directory again if there are no files.
The other solution would be to not enter the while loop because it makes no sense to run when there are no files. But this would cause the job to quit and it must be restarted manually or automatically.
So for my opinion this is the better solution as the runtime is used to find messages and not to always ask the driver again and again. But I think there is not the right way for it.

I've tried to benchmark with blackfire and it seems there's no critical performance problems.
Here is the sample code used on benchmark and the results of profile. Please let me know if there's something wrong on the result or my profiling.

test.php

<?php
require_once(__DIR__ . '/vendor/autoload.php');

$driver = new \Bernard\Driver\FlatFile\Driver(__DIR__ . '/queue');

$driver->createQueue('test');

list($message,) = $driver->popMessage('test', 30);

var_dump($message);

before(master)

https://blackfire.io/profiles/eb65f6df-c03f-43be-8fa4-be953c471c95/graph

after(this PR)

https://blackfire.io/profiles/bd1aec80-1ee9-406e-bf62-8492a307c4b1/graph

Resolved the conflicts between master branch. 😉
https://github.com/bernardphp/bernard/pull/330/commits/0923aa3b81dd0f1ed9123661a10f454f101a87cc


Old-style plist parser error during iOS compile

I recently installed FastImage as a plugin for my project and now come across this error during react-native run-ios
xcodebuild[30434:12026280] CFPropertyListCreateFromXMLData(): Old-style plist parser: missing semicolon in dictionary on line 1167. Parsing will be abandoned. Break on _CFPropertyListMissingSemicolon to debug.

2017-12-20 07:52:33.428 xcodebuild[30434:12026280] CFPropertyListCreateFromXMLData(): Old-style plist parser: missing semicolon in dictionary on line 1167. Parsing will be abandoned. Break on _CFPropertyListMissingSemicolon to debug.

2017-12-20 07:52:33.432 xcodebuild[30434:12026280] Error Domain=NSCocoaErrorDomain Code=3840 "Unexpected character / at line 1" UserInfo={NSDebugDescription=Unexpected character / at line 1, kCFPropertyListOldStyleParsingError=Error Domain=NSCocoaErrorDomain Code=3840 "Missing ';' on line 1167" UserInfo={NSDebugDescription=Missing ';' on line 1167}}

Line 1167 in .pbxproj points to
- HEADER_SEARCH_PATHS = "$(inherited)";
+ HEADER_SEARCH_PATHS = "$(inherited)""$(SRCROOT)/../node_modules/react-native-fast-image/ios/FastImage/**";

Was able to resolve the issue by making this update
Old
HEADER_SEARCH_PATHS = "$(inherited)""$(SRCROOT)/../node_modules/react-native-fast-image/ios/FastImage/**";

New
HEADER_SEARCH_PATHS = "$(inherited) $(SRCROOT)/../node_modules/react-native-fast-image/ios/FastImage/**";


Allow configuring specific paths on domains for trusted domains

Currently it's impossible to configure rules that support paths. For example: foo.com/bar should only allow foo.com/bar and foo.com, but not any other paths on foo.com.
We should make this possible, so we can trust specific URLs. This allows us to whtelist specific URL shortened links.

Actually duplicate of #83731.