[BUG] Multiple nodes are upgrading simultaneously in an RKE2 upgrade

Describe the bug

Upgrade to:

RKE2: v1.24.6+rke2r1
Rancher: v2.6.9-rc3

In a multi-node setup, the upgrade might be stuck in the middle. Behind the scenes, it is because there are multiple nodes (>= 2) upgrading simultaneously while upgrading RKE2, which implies they are in an unschedulable mode. This could result in the following while a node is in the pre-drain state:

The Longhorn volumes cannot go back from "degraded" to "healthy" state due to the replicas scheduling failure
The pre-drain job cannot live-migrate VMs out of the node they run on

Most of the time, the pre-drain Job will be stuck in the first situation because we only start to live-migrate VMs after all the volumes are healthy.
Below is the crime scene:
node-0:~ # k -n harvester-system get upgrades
NAME                 AGE
hvst-upgrade-t9mbq   4h29m
node-0:~ # k -n harvester-system get upgrades hvst-upgrade-t9mbq -o jsonpath='{.status.conditions}' | jq .
[
  {
    "status": "Unknown",
    "type": "Completed"
  },
  {
    "lastUpdateTime": "2022-10-11T09:46:52Z",
    "status": "True",
    "type": "ImageReady"
  },
  {
    "lastUpdateTime": "2022-10-11T09:49:11Z",
    "status": "True",
    "type": "RepoReady"
  },
  {
    "lastUpdateTime": "2022-10-11T10:36:19Z",
    "status": "True",
    "type": "NodesPrepared"
  },
  {
    "lastUpdateTime": "2022-10-11T10:59:49Z",
    "status": "True",
    "type": "SystemServicesUpgraded"
  },
  {
    "status": "Unknown",
    "type": "NodesUpgraded"
  }
]
node-0:~ # k -n harvester-system get upgrades hvst-upgrade-t9mbq -o jsonpath='{.status.nodeStatuses}' | jq .
{
  "node-0": {
    "state": "Succeeded"
  },
  "node-1": {
    "state": "Pre-drained"
  },
  "node-2": {
    "state": "Pre-draining"
  },
  "node-3": {
    "state": "Images preloaded"
  }
}

The first node node-0 is upgraded successfully. But node-1 and node-2 are being upgraded at the same time (though not exactly "start" at the "same time", it violates what we expected, i.e. upgrade one node at a time). Both node-1 and node-2 are marked with SchedulingDisabled.
node-0:~ # k get no
NAME     STATUS                     ROLES                       AGE     VERSION
node-0   Ready                      control-plane,etcd,master   5d11h   v1.24.6+rke2r1
node-1   Ready,SchedulingDisabled   control-plane,etcd,master   5d10h   v1.24.6+rke2r1
node-2   Ready,SchedulingDisabled   control-plane,etcd,master   5d10h   v1.22.12+rke2r1
node-3   Ready                      <none>                      5d10h   v1.22.12+rke2r1

Go check the upgrade-related Pods. Here we can see the post-drain Job on the first upgraded node node-0 has been completed, which means node-0 is fully upgraded without issues. There is a pre-drain Job left on node-2 running. This implies that node-2 is in the pre-draining state, and VMs should be evacuated to other nodes. How about node-1? From the output of k get no shown above, we know that RKE2 on node-1 has been upgraded to v1.24.6+rke2r1. The uptime of node-1 is the same as node-2 and node-3 (given that all 4 nodes are booted up at the same time). We can infer that node-1 has never entered the post-drain state. But somehow node-2 is triggered to upgrade before the upgrade is completed on node-1.
node-0:~ # k -n harvester-system get po -l harvesterhci.io/upgradeComponent=node
NAME                                         READY   STATUS      RESTARTS   AGE
hvst-upgrade-t9mbq-post-drain-node-0-fbrwk   0/1     Completed   0          3h10m
hvst-upgrade-t9mbq-pre-drain-node-2-rbj2w    1/1     Running     0          160m

From the log of the pre-drain Job on node-2, we can see that it keeps waiting for the Longhorn volumes to become healthy.
node-0:~ # k -n harvester-system logs hvst-upgrade-t9mbq-pre-drain-node-2-rbj2w --since=1m
+ '[' true ']'
+ '[' 4 -gt 2 ']'
++ kubectl get volumes.longhorn.io/pvc-306f865e-5bfa-4d12-8779-fe3371425305 -n longhorn-system -o 'jsonpath={.status.robustness}'
+ robustness=degraded
+ '[' degraded = healthy ']'
+ '[' -f /tmp/skip-pvc-306f865e-5bfa-4d12-8779-fe3371425305 ']'
+ echo 'Waiting for volume pvc-306f865e-5bfa-4d12-8779-fe3371425305 to be healthy...'
+ sleep 10
Waiting for volume pvc-306f865e-5bfa-4d12-8779-fe3371425305 to be healthy...
+ '[' true ']'
+ '[' 4 -gt 2 ']'
++ kubectl get volumes.longhorn.io/pvc-306f865e-5bfa-4d12-8779-fe3371425305 -n longhorn-system -o 'jsonpath={.status.robustness}'
Waiting for volume pvc-306f865e-5bfa-4d12-8779-fe3371425305 to be healthy...
+ robustness=degraded
+ '[' degraded = healthy ']'
+ '[' -f /tmp/skip-pvc-306f865e-5bfa-4d12-8779-fe3371425305 ']'
+ echo 'Waiting for volume pvc-306f865e-5bfa-4d12-8779-fe3371425305 to be healthy...'
+ sleep 10
+ '[' true ']'
+ '[' 4 -gt 2 ']'
++ kubectl get volumes.longhorn.io/pvc-306f865e-5bfa-4d12-8779-fe3371425305 -n longhorn-system -o 'jsonpath={.status.robustness}'
Waiting for volume pvc-306f865e-5bfa-4d12-8779-fe3371425305 to be healthy...
+ robustness=degraded
+ '[' degraded = healthy ']'
+ '[' -f /tmp/skip-pvc-306f865e-5bfa-4d12-8779-fe3371425305 ']'
+ echo 'Waiting for volume pvc-306f865e-5bfa-4d12-8779-fe3371425305 to be healthy...'

That's impossible for the current situation because there are only 2 schedulable nodes (node-1 and node-3) for all the 3 replicas to run on. We can prove this by examining the condition of the volume, it shows there's a ReplicaSchedulingFailure:
node-0:~ # k -n longhorn-system get volumes pvc-306f865e-5bfa-4d12-8779-fe3371425305 -o jsonpath='{.status.conditions}' | jq .
[
  {
    "lastProbeTime": "",
    "lastTransitionTime": "2022-10-11T09:46:57Z",
    "message": "",
    "reason": "",
    "status": "False",
    "type": "restore"
  },
  {
    "lastProbeTime": "",
    "lastTransitionTime": "2022-10-11T11:30:29Z",
    "message": "",
    "reason": "ReplicaSchedulingFailure",
    "status": "False",
    "type": "scheduled"
  },
  {
    "lastProbeTime": "",
    "lastTransitionTime": "2022-10-11T09:46:57Z",
    "message": "",
    "reason": "",
    "status": "False",
    "type": "toomanysnapshots"
  }
]

And the running replicas of the volume only exist on node-0 and node-3
node-0:~ # k -n longhorn-system get lhr -l longhornvolume=pvc-306f865e-5bfa-4d12-8779-fe3371425305
NAME                                                  STATE     NODE     DISK                                   INSTANCEMANAGER               IMAGE                               AGE
pvc-306f865e-5bfa-4d12-8779-fe3371425305-r-4477ad68   running   node-0   24e133ac-90bf-47ea-bf3f-0a519adda3ec   instance-manager-r-7cffeda1   longhornio/longhorn-engine:v1.3.1   5h58m
pvc-306f865e-5bfa-4d12-8779-fe3371425305-r-8f915cdf   stopped                                                                                4h14m
pvc-306f865e-5bfa-4d12-8779-fe3371425305-r-f902b787   running   node-3   1e8daca0-9a5b-47fc-b17a-e87da5c49edf   instance-manager-r-3bf011a4   longhornio/longhorn-engine:v1.3.1   5h58m

The whole upgrade is stuck in the pre-draining state on node-2 almost forever because the volume it waits for can never become a healthy one.
It's time to take a look at the other side. The upgrade controller instructs Rancher to upgrade RKE2 on each node by updating the manifest of clusters.provisioning.cattle.io/local CR. Here we specified the upgrade concurrency of both the control plane and worker to 1 to make sure the upgrade of Harvester goes smoothly.
node-0:~ # k -n fleet-local get clusters local -o jsonpath='{.spec.rkeConfig}' | jq .
{
  "chartValues": null,
  "machineGlobalConfig": null,
  "provisionGeneration": 1,
  "upgradeStrategy": {
    "controlPlaneConcurrency": "1",
    "controlPlaneDrainOptions": {
      "deleteEmptyDirData": true,
      "enabled": true,
      "force": true,
      "ignoreDaemonSets": true,
      "postDrainHooks": [
        {
          "annotation": "harvesterhci.io/post-hook"
        }
      ],
      "preDrainHooks": [
        {
          "annotation": "harvesterhci.io/pre-hook"
        }
      ],
      "timeout": 0
    },
    "workerConcurrency": "1",
    "workerDrainOptions": {
      "deleteEmptyDirData": true,
      "enabled": true,
      "force": true,
      "ignoreDaemonSets": true,
      "postDrainHooks": [
        {
          "annotation": "harvesterhci.io/post-hook"
        }
      ],
      "preDrainHooks": [
        {
          "annotation": "harvesterhci.io/pre-hook"
        }
      ],
      "timeout": 0
    }
  }
}

The pre-drain and post-drain annotations are set on the corresponding secrets when the RKE2 of the node/machine is upgrading.
node-0:~ # k -n fleet-local get machines
NAME                  CLUSTER   NODENAME   PROVIDERID      PHASE     AGE     VERSION
custom-2d94d5d682dc   local     node-3     rke2://node-3   Running   5d13h
custom-7c1afab6e79d   local     node-0     rke2://node-0   Running   5d14h
custom-929d403d1670   local     node-1     rke2://node-1   Running   5d14h
custom-c05d0d11190c   local     node-2     rke2://node-2   Running   5d13h

Check the annotations of the secret custom-929d403d1670-machine-plan (node-1):
node-0:~ # k -n fleet-local get secrets custom-929d403d1670-machine-plan -o jsonpath='{.metadata.annotations}' | jq .
{
  "harvesterhci.io/pre-hook": "{\"IgnoreErrors\":false,\"deleteEmptyDirData\":true,\"disableEviction\":false,\"enabled\":true,\"force\":true,\"gracePeriod\":0,\"ignoreDaemonSets\":true,\"postDrainHooks\":[{\"annotation\":\"harvesterhci.io/post-hook\"}],\"
preDrainHooks\":[{\"annotation\":\"harvesterhci.io/pre-hook\"}],\"skipWaitForDeleteTimeoutSeconds\":0,\"timeout\":0}",
  "objectset.rio.cattle.io/applied": "H4sIAAAAAAAA/4xTwW7bOhD8lYc9i36iLMuUgF6K9pRDgaDope5htVzGqihSIOmkhaF/L6TEaO3Ebo8iZ4azo9kjDJxQY0JojoDO+YSp8y7On779zpQip1Xo/IowJcurzv/faWgg9CwGpH3nGLKrUP/kOIiHx/6Z8cfNo8z+u+ucfnd/9/G99ymmgONflRwODA3QISY/iLqodZmvtay2+T9R4
4g0841lTsJ6QgvZhTGLLdsIDRx3sMfwyDFx2FM33w3o8IH1DpodpHDgHUwwZUCBl9Q+dwPHhMMIjTtYm8FJ6whkD7PO6ofoVZylXg5OE528XHkRmuW9G0PuMe6hgZJVyVpLs1FbluWm1kySVIFG57piaTA3bZHjq7Gv+LkAeZeCt2K06FgEb/m3sXMkJ9I3AS/dEUuZyKDKDVY5ttsNb3K5xpwKhRuqeFOqqjV1JZlbY1BxhbxeV0pSqZTmbV7L
q+I363JOefKh53BmecrgusCp/EsW8Ix8u15L/e7ZcGBHHKH5egQcuy8cYufdG4sBGbTWU/9pJn5gy2nBzZ4yePkFlsPppO/cnOHFHt0c/bCkLg1TgYUSZNZSlKRroYoqF8xUV4hI27KG6duUQfo58iujZwFMvwIAAP//B0mT1koEAAA",
  "objectset.rio.cattle.io/id": "rke-machine",
  "objectset.rio.cattle.io/owner-gvk": "rke.cattle.io/v1, Kind=RKEBootstrap",
  "objectset.rio.cattle.io/owner-name": "custom-929d403d1670",
  "objectset.rio.cattle.io/owner-namespace": "fleet-local",
  "rke.cattle.io/drain-done": "{\"IgnoreErrors\":false,\"deleteEmptyDirData\":true,\"disableEviction\":false,\"enabled\":true,\"force\":true,\"gracePeriod\":0,\"ignoreDaemonSets\":true,\"postDrainHooks\":[{\"annotation\":\"harvesterhci.io/post-hook\"}],\"
preDrainHooks\":[{\"annotation\":\"harvesterhci.io/pre-hook\"}],\"skipWaitForDeleteTimeoutSeconds\":0,\"timeout\":0}",
  "rke.cattle.io/drain-options": "{\"IgnoreErrors\":false,\"deleteEmptyDirData\":true,\"disableEviction\":false,\"enabled\":true,\"force\":true,\"gracePeriod\":0,\"ignoreDaemonSets\":true,\"postDrainHooks\":[{\"annotation\":\"harvesterhci.io/post-hook\"}]
,\"preDrainHooks\":[{\"annotation\":\"harvesterhci.io/pre-hook\"}],\"skipWaitForDeleteTimeoutSeconds\":0,\"timeout\":0}",
  "rke.cattle.io/join-url": "https://<IP_ADDRESS>:9345",
  "rke.cattle.io/labels": "{\"harvesterhci.io/managed\":\"true\"}",
  "rke.cattle.io/post-drain": "{\"IgnoreErrors\":false,\"deleteEmptyDirData\":true,\"disableEviction\":false,\"enabled\":true,\"force\":true,\"gracePeriod\":0,\"ignoreDaemonSets\":true,\"postDrainHooks\":[{\"annotation\":\"harvesterhci.io/post-hook\"}],\"
preDrainHooks\":[{\"annotation\":\"harvesterhci.io/pre-hook\"}],\"skipWaitForDeleteTimeoutSeconds\":0,\"timeout\":0}",
  "rke.cattle.io/pre-drain": "{\"IgnoreErrors\":false,\"deleteEmptyDirData\":true,\"disableEviction\":false,\"enabled\":true,\"force\":true,\"gracePeriod\":0,\"ignoreDaemonSets\":true,\"postDrainHooks\":[{\"annotation\":\"harvesterhci.io/post-hook\"}],\"p
reDrainHooks\":[{\"annotation\":\"harvesterhci.io/pre-hook\"}],\"skipWaitForDeleteTimeoutSeconds\":0,\"timeout\":0}",
  "rke.cattle.io/uncordon": "{\"IgnoreErrors\":false,\"deleteEmptyDirData\":true,\"disableEviction\":false,\"enabled\":true,\"force\":true,\"gracePeriod\":0,\"ignoreDaemonSets\":true,\"postDrainHooks\":[{\"annotation\":\"harvesterhci.io/post-hook\"}],\"pr
eDrainHooks\":[{\"annotation\":\"harvesterhci.io/pre-hook\"}],\"skipWaitForDeleteTimeoutSeconds\":0,\"timeout\":0}"
}

Check the annotations of the secret custom-c05d0d11190c-machine-plan (node-2):
node-0:~ # k -n fleet-local get secrets custom-c05d0d11190c-machine-plan -o jsonpath='{.metadata.annotations}' | jq .
{
  "objectset.rio.cattle.io/applied": "H4sIAAAAAAAA/4xTzW7bPBB8lQ97Fv3xRz+WgF6K9pRDgaDope5hSa1iVRQpkHTSwtC7F1JitHZit0eRM8PZ0ewRRkrYYkJojoDO+YSp9y4un15/J5MipU3o/cZgSpY2vf+/b6GBMBAb0ex7R5BdhfonR4E9PA7PjD9uHkX2313v2nf3dx/fe59iCjj9VcnhSNCAOcTkR2Z40fJWCFFz80/UO
KFZ+J0lSsx6gxayC2MWNdkIDRx3sMfwSDFR2Jt+uRvR4QO1O2h2kMKBdjDDnIEJtKb2uR8pJhwnaNzB2gxOWkcw9rDobH6wYRsXqZeD00QnL1dehGZ978aQe4x7aEDUXFVVVZpC867OqdNFZbiSnSwqIkmC2iLXong19hU/FyDvUvCWTRYdseAt/TZ2jqRk2puAl+6wtUwSS8GraivzUnAjSCkjtFZc5LKotULTKSOR1FbVeVFKIlV32miqJKdc
mKviN+tyTnnyYaBwZnnO4LrAqfxrFvCMfLtea/3uqaNAzlCE5usRcOq/UIi9d28sBmSgrTfDp4X4gSylFbd4yuDlF1gKp5Ohd0uGF3t0c/TDmnpVtCi0RFbKHFle5i3TUhKrW56rnKtyW3cwf5szSD8nemX0LID5VwAAAP//98YwqkoEAAA",
  "objectset.rio.cattle.io/id": "rke-machine",
  "objectset.rio.cattle.io/owner-gvk": "rke.cattle.io/v1, Kind=RKEBootstrap",
  "objectset.rio.cattle.io/owner-name": "custom-c05d0d11190c",
  "objectset.rio.cattle.io/owner-namespace": "fleet-local",
  "rke.cattle.io/drain-options": "{\"IgnoreErrors\":false,\"deleteEmptyDirData\":true,\"disableEviction\":false,\"enabled\":true,\"force\":true,\"gracePeriod\":0,\"ignoreDaemonSets\":true,\"postDrainHooks\":[{\"annotation\":\"harvesterhci.io/post-hook\"}]
,\"preDrainHooks\":[{\"annotation\":\"harvesterhci.io/pre-hook\"}],\"skipWaitForDeleteTimeoutSeconds\":0,\"timeout\":0}",
  "rke.cattle.io/join-url": "https://<IP_ADDRESS>:9345",
  "rke.cattle.io/labels": "{\"harvesterhci.io/managed\":\"true\"}",
  "rke.cattle.io/pre-drain": "{\"IgnoreErrors\":false,\"deleteEmptyDirData\":true,\"disableEviction\":false,\"enabled\":true,\"force\":true,\"gracePeriod\":0,\"ignoreDaemonSets\":true,\"postDrainHooks\":[{\"annotation\":\"harvesterhci.io/post-hook\"}],\"p
reDrainHooks\":[{\"annotation\":\"harvesterhci.io/pre-hook\"}],\"skipWaitForDeleteTimeoutSeconds\":0,\"timeout\":0}"
}

node-0:~ # k -n cattle-system logs -l app=rancher
W1012 02:39:20.056363      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1012 02:45:04.057107      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1012 02:47:50.962077      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1012 02:52:42.058995      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1012 02:53:17.964514      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1012 03:00:00.096962      33 transport.go:288] Unable to cancel request for *client.addQuery
W1012 03:00:25.077424      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1012 03:00:36.965184      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1012 03:05:52.966654      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1012 03:08:43.085276      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2022/10/12 03:00:05 [INFO] [planner] rkecluster fleet-local/local: waiting: uncordoning etcd node(s) custom-929d403d1670: waiting for uncordon to finish
2022/10/12 03:03:13 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022/10/12 03:03:14 [INFO] Downloading repo index from https://releases.rancher.com/server-charts/stable/index.yaml
2022/10/12 03:08:13 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022/10/12 03:08:14 [INFO] Downloading repo index from https://releases.rancher.com/server-charts/stable/index.yaml
2022/10/12 03:08:23 [INFO] [planner] rkecluster fleet-local/local: waiting: uncordoning etcd node(s) custom-929d403d1670: waiting for uncordon to finish
2022/10/12 03:08:23 [INFO] [planner] rkecluster fleet-local/local: waiting: uncordoning etcd node(s) custom-929d403d1670: waiting for uncordon to finish
2022/10/12 03:09:05 [INFO] [planner] rkecluster fleet-local/local: waiting: uncordoning etcd node(s) custom-929d403d1670: waiting for uncordon to finish
2022/10/12 03:09:08 [INFO] [planner] rkecluster fleet-local/local: waiting: uncordoning etcd node(s) custom-929d403d1670: waiting for uncordon to finish
2022/10/12 03:10:06 [INFO] [planner] rkecluster fleet-local/local: waiting: uncordoning etcd node(s) custom-929d403d1670: waiting for uncordon to finish
W1012 02:34:06.943613      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1012 02:39:19.762602      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1012 02:40:24.941360      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1012 02:47:05.943248      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1012 02:47:15.766388      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1012 02:52:50.945579      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1012 02:54:48.767367      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1012 02:59:23.961176      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1012 03:04:26.770103      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1012 03:05:31.964012      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+

To Reproduce
Steps to reproduce the behavior:

Prepare a multi-node (>= 3) Harvester cluster in v1.0.3
Trigger an upgrade to v1.1.0-rc2
Wait for the upgrade stuck in the pre-draining state (not always reproducible), and check the nodes' state to confirm the issue has happened

Expected behavior

The RKE2 upgrade should be kicked off one node at a time and the upgrade of Harvester should end up successfully.
Support bundle

supportbundle_ebe4b217-6355-4356-823a-f9e5f09e28b4_2022-10-11T16-46-36Z.zip
Environment

Harvester ISO version: v1.1.0-rc2
Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630):

Additional context
Related issue: rancher/rancher#39167
Related issues in previous versions: rancher/rancher#35999, rancher/rancher#37502
The issue could be work around temporarily with uncordoning the node manually, say kubectl uncordon node-1 from the above case. Then the upgrade will proceed. But there's high chance encountering other RKE2 issues like #2893, restarting the rke2-server on the corresponding node might be necessary.

Sometimes the situation can be coped with a simple uncordon, sometimes don't. To make the Harvester upgrade more stable, we tend to prevent the situation from happening. The main idea is to place some checks in the upgrade controller when it is about to create the pre-drain hook job. If the checks failed, it will log down the error and try again until the checks are passed, then the pre-drain hook job can be created.
Currently, the check will look for the following:

Is any pre-drain job running, if so, error out
Is any post-drain job running, if so, error out

cc @bk201

@starbops I was curious is it possible to leverage a version.yaml to upgrade to harvester-master-head with something like:
ubuntu@caddy-file-server:~/harvester-isos$ cat version_master_head.yaml
apiVersion: harvesterhci.io/v1beta1
kind: Version
metadata:
  name: master-head-10-13
  namespace: harvester-system
spec:
  isoChecksum: e0b7ac9eb47e5201be8ca087009dd5c25a7cb962095c83b91306d445f3f9a0e2174dd728115a90b31b51d90e17433988bbf16d5cc73ea66440ddc2fa86ae3dee
  isoURL: http://<IP_ADDRESS>/harvester-master-amd64-10-13.iso
  releaseDate: '20221013'

@starbops I was curious is it possible to leverage a version.yaml to upgrade to harvester-master-head with something like:
ubuntu@caddy-file-server:~/harvester-isos$ cat version_master_head.yaml
apiVersion: harvesterhci.io/v1beta1
kind: Version
metadata:
  name: master-head-10-13
  namespace: harvester-system
spec:
  isoChecksum: e0b7ac9eb47e5201be8ca087009dd5c25a7cb962095c83b91306d445f3f9a0e2174dd728115a90b31b51d90e17433988bbf16d5cc73ea66440ddc2fa86ae3dee
  isoURL: http://<IP_ADDRESS>/harvester-master-amd64-10-13.iso
  releaseDate: '20221013'

Yes, I do the upgrade tests like this every time when I build a new ISO image. Just swap the image name in the isoURL. For testing, you can solely provide the isoURL under spec, it will work.

@starbops thanks for looking at that :smile:
I think I was just needing to wait a bit.
From the UI it looks somewhat upgraded.
But I'm noticing that on "Upgrading System Service" it seems to be hung at 50%.
All the nodes have "Images preloaded".
From:
echo "" && kubectl -n fleet-local get secrets custom-c45b04e1f4bb-machine-plan -o jsonpath='{.metadata.annotations}' | jq && echo "" && kubectl -n fleet-local get secrets custom-9db392239b4b-machine-plan -o jsonpath='{.metadata.annotations}' | jq && echo "" && kubectl -n fleet-local get secrets custom-310756e7903e-machine-plan -o jsonpath='{.metadata.annotations}' | jq && echo ""

I notice:

{
  "objectset.rio.cattle.io/applied": "H4sIAAAAAAAA/4xTzW7bPBB8lQ97Fv3ph5IlAb0U7SmHAkHRS93DklzGrChSIOmkhaF3L6TEaO3Ebo8iZ4azo9kjjJRQYULoj4DO+YTJeBeXTy++k0yR0iYYv5GYkqWN8f8bBT2EgdiIcm8cQXYV6p8cBfbwODwz/rh5LLL/7oxT7+7vPr73PsUUcPqrksORoAd5iMmPTPJa5JwKzYX4J2qcUC58bYkSs16ihezCmEVBNkIPxx3sMTxSTBT20ix3Izp8ILWDfgcpHGgHM8wZyEBrap/NSDHhOEHvDtZmcNI6grSHRWfzgw1tXKReDk4TnbxceRH69b0bQ+4x7qEH3HY5p7ZpVVeJettVXG67QnJd6LzGUnDVKpRN+2rsK34uQN6l4C2bLDpiwVv6bewcSUmqm4CX7rC1THVVtm3bNI1Y3CqpOTbVVm+5KNq8I2y7QldFWeo8L8tGcq6qssqpUDWvSfOr4jfrck558mGgcGZ5zuC6wKn8axbwjHy7Xmv97klTICcpQv/1CDiZLxSi8e6NxYAMhPVy+LQQP5CltOIWTxm8/AJL4XQyGLdkeLFHN0c/rKlLqcquUAXDssgZ50qzTlPFmhJFlXe6E5WA+ducQfo50SujZwHMvwIAAP//RMCG9UoEAAA",
  "objectset.rio.cattle.io/id": "rke-machine",
  "objectset.rio.cattle.io/owner-gvk": "rke.cattle.io/v1, Kind=RKEBootstrap",
  "objectset.rio.cattle.io/owner-name": "custom-c45b04e1f4bb",
  "objectset.rio.cattle.io/owner-namespace": "fleet-local",
  "rke.cattle.io/join-url": "https://<IP_ADDRESS>:9345",
  "rke.cattle.io/labels": "{\"harvesterhci.io/managed\":\"true\"}"
}

{
  "objectset.rio.cattle.io/applied": "H4sIAAAAAAAA/4yTz67aPBTEX+XTWdt8BJxwHambql3dRaWrqpuqC8c+Jrlx7NQ+/CmId68SQAIKtMs4M6PxL5M9dEjKKFJQ7kF5H0hRE3waHkP1jpoS0iQ2YaIVkcNJE/5vDJQQW+Sd0nXjEdhDadh4jHy5bo+OizfrjP332njz4e3188cQKFFU/V+TvOoQStCrRKHj0lRzOZvNZSWqf7KmXunBbx0icRe0cnBgoCOOt/7adJhIdT2UfuUcA6cqdCML7VaJME62vH1JQ+jp4NzomMXu6IKnGBzvnfKDkOLqGbBapRpKEFNt86yaalQap7KYC2MKO5MSZ9JOlcpEbvK8MMBusD6odSO67MRjcBfFrpVI2jwVnCbAx03k/Y5M/55L23bJC22MWK/jptd+29fbLi/Wy1rv6sXy58aJxbbY2RezsA8zn37sa8smxBbjVdMDg8cB5+mOCOCovDcOBuN43tBiRK8xQfl9D6pvvmFMTfB3Zg0MKhd0+2UwfkKHNOqGTgxO5B3G80nb+AHdzV/w9OqrEfY0s9baheQzWWVcVEXGVZYLnuVCIJoMZbGAw48DA/rV4x9FrwAcfgcAAP//AMqU6ggEAAA",
  "objectset.rio.cattle.io/id": "rke-machine",
  "objectset.rio.cattle.io/owner-gvk": "rke.cattle.io/v1, Kind=RKEBootstrap",
  "objectset.rio.cattle.io/owner-name": "custom-9db392239b4b",
  "objectset.rio.cattle.io/owner-namespace": "fleet-local",
  "rke.cattle.io/join-url": "https://<IP_ADDRESS>:9345"
}

{
  "objectset.rio.cattle.io/applied": "H4sIAAAAAAAA/4RTzW7bPBB8lQ97Fv3phxQlAb0U7SmHAkHRS93DilzFqihSIGmnhaF3L6TYaO3EzlHkzHB2NHuEkSJqjAjNEdBaFzH2zobl07U/ScVAceN7t1EYo6FN7/7vNTTgB2Ijql1vCZKbUPdsybOnw/DC+OfmkCX/PfRWf3h8+PzRuRiix+ldJYsjQQNqH6IbWZGlUpQk67R438RCDROqhd8ZosiMU2gguTJmsCUToIHjFnboDxQi+Z3ql7sRLT6R3kKzhej3tIUZ5gSUpzW1r/1IIeI4QWP3xiRw1jqCMvtFZ/OLDVVYpE4H54nOXm68CM363p0hdxh20EDJVdFlHNsKZSdQ1LrsRFt2IhfES5lzmfGaNL4a+4afK5Cz0TvDJoOWmHeG/hq7RFJU+i7g1B22lomUaoUWlai5rNOO15prkdZdXVSY60JrQp1RysusTVslOC9JpTlVIi9lpoub4nfrckl5dn4gf2F5TuC2wLn8axbwgny7Xmv9HqkjT1ZRgOb7EXDqv5EPvbNvLAYk0Bqnhi8L8RMZiitu8ZTA6RcY8ueTobdLhld7dHf0/Zp6XadSpFqzOstKxotWMJRasbxSkiSXLeUZzD/mBOLviV4ZvQhg/hMAAP//SkIg9UoEAAA",
  "objectset.rio.cattle.io/id": "rke-machine",
  "objectset.rio.cattle.io/owner-gvk": "rke.cattle.io/v1, Kind=RKEBootstrap",
  "objectset.rio.cattle.io/owner-name": "custom-310756e7903e",
  "objectset.rio.cattle.io/owner-namespace": "fleet-local",
  "rke.cattle.io/join-url": "https://<IP_ADDRESS>:9345",
  "rke.cattle.io/labels": "{\"harvesterhci.io/managed\":\"true\"}"
}

Prior to upgrade for v1.0.3, I did:
kubectl patch managedcharts.management.cattle.io local-managed-system-upgrade-controller --namespace fleet-local --patch-file=/tmp/fix.yaml --type merge

Where /tmp/fix.yaml was:
spec:
  values:
    systemUpgradeJobActiveDeadlineSeconds: "3600"

And did:
kubectl -n cattle-system rollout restart deploy/system-upgrade-controller

It seems that there are lots of index's seemingly trying to be downloaded from different rancher-uuid nodes within like:
2022-10-14T02:05:15.352452404Z rancher-7fcffb89db-brgl8 2022/10/14 02:05:15 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022-10-14T02:10:15.352744796Z rancher-7fcffb89db-brgl8 2022/10/14 02:10:15 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022-10-14T02:10:15.352813216Z rancher-7fcffb89db-brgl8 2022/10/14 02:10:15 [INFO] Downloading repo index from https://releases.rancher.com/server-charts/stable/index.yaml
2022-10-14T02:15:15.353076120Z rancher-7fcffb89db-brgl8 2022/10/14 02:15:15 [INFO] Downloading repo index from https://releases.rancher.com/server-charts/stable/index.yaml
2022-10-14T02:15:15.353153583Z rancher-7fcffb89db-brgl8 2022/10/14 02:15:15 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022-10-14T02:20:15.353767670Z rancher-7fcffb89db-brgl8 2022/10/14 02:20:15 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022-10-14T02:20:15.353886448Z rancher-7fcffb89db-brgl8 2022/10/14 02:20:15 [INFO] Downloading repo index from https://releases.rancher.com/server-charts/stable/index.yaml
2022-10-14T02:25:15.354520990Z rancher-7fcffb89db-brgl8 2022/10/14 02:25:15 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022-10-14T01:54:56.288098961Z rancher-7fcffb89db-qfpx2 2022/10/14 01:54:56 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Setting
2022-10-14T01:54:56.288103975Z rancher-7fcffb89db-qfpx2 2022/10/14 01:54:56 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Orphan
2022-10-14T01:54:56.288108929Z rancher-7fcffb89db-qfpx2 2022/10/14 01:54:56 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Volume
2022-10-14T01:54:56.288113918Z rancher-7fcffb89db-qfpx2 2022/10/14 01:54:56 [INFO] Stopping metadata watch on longhorn.io/v1beta1, Kind=Setting
2022-10-14T01:54:56.288118846Z rancher-7fcffb89db-qfpx2 2022/10/14 01:54:56 [INFO] Stopping metadata watch on longhorn.io/v1beta1, Kind=RecurringJob
2022-10-14T01:54:56.288124037Z rancher-7fcffb89db-qfpx2 2022/10/14 01:54:56 [INFO] Stopping metadata watch on longhorn.io/v1beta1, Kind=Node
2022-10-14T01:54:56.288128933Z rancher-7fcffb89db-qfpx2 2022/10/14 01:54:56 [INFO] Stopping metadata watch on longhorn.io/v1beta1, Kind=Replica
2022-10-14T01:54:56.288134353Z rancher-7fcffb89db-qfpx2 2022/10/14 01:54:56 [INFO] Stopping metadata watch on longhorn.io/v1beta1, Kind=Volume
2022-10-14T01:54:56.288139538Z rancher-7fcffb89db-qfpx2 2022/10/14 01:54:56 [INFO] Stopping metadata watch on longhorn.io/v1beta1, Kind=ShareManager
2022-10-14T01:54:59.392036466Z rancher-7fcffb89db-qfpx2 W1014 01:54:59.379809      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2022-10-14T01:55:03.314132180Z rancher-7fcffb89db-qfpx2 W1014 01:55:03.274214      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2022-10-14T01:55:50.372182839Z rancher-7fcffb89db-qfpx2 W1014 01:55:50.368538      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2022-10-14T01:55:56.720184870Z rancher-7fcffb89db-qfpx2 W1014 01:55:56.718536      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2022-10-14T01:55:57.035500177Z rancher-7fcffb89db-qfpx2 2022/10/14 01:55:57 [INFO] Watching metadata for flavor.kubevirt.io/v1alpha1, Kind=VirtualMachineClusterPreference
2022-10-14T02:25:15.354582407Z rancher-7fcffb89db-brgl8 2022/10/14 02:25:15 [INFO] Downloading repo index from https://releases.rancher.com/server-charts/stable/index.yaml
2022-10-14T01:55:57.035557259Z rancher-7fcffb89db-qfpx2 2022/10/14 01:55:57 [INFO] Watching metadata for flavor.kubevirt.io/v1alpha1, Kind=VirtualMachinePreference
2022-10-14T02:30:15.356126963Z rancher-7fcffb89db-brgl8 2022/10/14 02:30:15 [INFO] Downloading repo index from https://releases.rancher.com/server-charts/stable/index.yaml
2022-10-14T02:30:15.356196084Z rancher-7fcffb89db-brgl8 2022/10/14 02:30:15 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022-10-14T02:35:15.355954709Z rancher-7fcffb89db-brgl8 2022/10/14 02:35:15 [INFO] Downloading repo index from https://releases.rancher.com/server-charts/stable/index.yaml
2022-10-14T02:35:15.356216298Z rancher-7fcffb89db-brgl8 2022/10/14 02:35:15 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022-10-14T01:56:03.924811142Z rancher-7fcffb89db-qfpx2 W1014 01:56:03.924436      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2022-10-14T01:56:07.314852987Z rancher-7fcffb89db-qfpx2 W1014 01:56:07.314566      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2022-10-14T01:58:55.188675536Z rancher-7fcffb89db-qfpx2 W1014 01:58:55.188475      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2022-10-14T02:00:58.590926642Z rancher-7fcffb89db-qfpx2 W1014 02:00:58.590600      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
2022-10-14T02:08:01.190885782Z rancher-7fcffb89db-qfpx2 W1014 02:08:01.190608      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2022-10-14T02:10:19.586356117Z rancher-7fcffb89db-qfpx2 W1014 02:10:19.586133      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
2022-10-14T02:15:32.591246087Z rancher-7fcffb89db-qfpx2 W1014 02:15:32.590979      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
2022-10-14T02:40:15.356585715Z rancher-7fcffb89db-brgl8 2022/10/14 02:40:15 [INFO] Downloading repo index from https://releases.rancher.com/server-charts/stable/index.yaml
2022-10-14T02:40:15.356709216Z rancher-7fcffb89db-brgl8 2022/10/14 02:40:15 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022-10-14T02:45:15.357170447Z rancher-7fcffb89db-brgl8 2022/10/14 02:45:15 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022-10-14T02:45:15.357347596Z rancher-7fcffb89db-brgl8 2022/10/14 02:45:15 [INFO] Downloading repo index from https://releases.rancher.com/server-charts/stable/index.yaml
2022-10-14T02:50:15.358140515Z rancher-7fcffb89db-brgl8 2022/10/14 02:50:15 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml

My version.yaml was as I had mentioned earlier.
And it was upgrading from v1.0.3 to Harvester Version: master-61a6124a-head
On 3 node qemu/kvm:
8c
16Gib mem
supportbundle_a0823d59-1dae-4339-83b7-0f831510063c_2022-10-14T02-46-05Z.zip

@irishgordo From the log, it's waiting for longhorn to upgrade instance-manager-r pod. Can you see if there are any abnormal volumes in Longhorn GUI?
2022-10-14T02:44:14.811352577Z instance-manager-r pod count is not 1 on node v103-node-0, will retry...
2022-10-14T02:44:19.998382167Z instance-manager-r pod count is not 1 on node v103-node-0, will retry...

Suspect this issue may only happen in 4-node cluster, with 3 control-plane and 1 worker node
According to upgrade setting, control-plane has at most 1 node in upgrade; worker is same.
For a cluster with >=5 nodes, say 3 control-plane and 2 worker, there are always >=3 nodes not in upgrade state, LH default 3 replicas will be met.
Wait for the confirmation from Rancher about how the concurrency inbetween Control-plane and worker is controlled.

@w13915984028 @bk201
End of day yesterday, I had powered off the nodes, but upon booting them up this morning what I was seeing with the upgrade for 10/13/22's master-head - master-61a6124a-head
Had all disappeared:

supportbundle_a0823d59-1dae-4339-83b7-0f831510063c_2022-10-14T19-33-36Z.zip
With:
suse-workstation-team-harvester➜  test_2907  ᐅ  echo "" && kubectl -n fleet-local get secrets custom-c45b04e1f4bb-machine-plan -o jsonpath='{.metadata.annotations}' | jq && echo "" && kubectl -n fleet-local get secrets custom-9db392239b4b-machine-plan -o jsonpath='{.metadata.annotations}' | jq && echo "" && kubectl -n fleet-local get secrets custom-310756e7903e-machine-plan -o jsonpath='{.metadata.annotations}' | jq && echo ""

Yields:
{
  "objectset.rio.cattle.io/applied": "H4sIAAAAAAAA/4xTzW7bPBB8lQ97Fv3ph5IlAb0U7SmHAkHRS93DklzGrChSIOmkhaF3L6TEaO3Ebo8iZ4azo9kjjJRQYULoj4DO+YTJeBeXTy++k0yR0iYYv5GYkqWN8f8bBT2EgdiIcm8cQXYV6p8cBfbwODwz/rh5LLL/7oxT7+7vPr73PsUUcPqrksORoAd5iMmPTPJa5JwKzYX4J2qcUC58bYkSs16ihezCmEVBNkIPxx3sMTxSTBT20ix3Izp8ILWDfgcpHGgHM8wZyEBrap/NSDHhOEHvDtZmcNI6grSHRWfzgw1tXKReDk4TnbxceRH69b0bQ+4x7qEH3HY5p7ZpVVeJettVXG67QnJd6LzGUnDVKpRN+2rsK34uQN6l4C2bLDpiwVv6bewcSUmqm4CX7rC1THVVtm3bNI1Y3CqpOTbVVm+5KNq8I2y7QldFWeo8L8tGcq6qssqpUDWvSfOr4jfrck558mGgcGZ5zuC6wKn8axbwjHy7Xmv97klTICcpQv/1CDiZLxSi8e6NxYAMhPVy+LQQP5CltOIWTxm8/AJL4XQyGLdkeLFHN0c/rKlLqcquUAXDssgZ50qzTlPFmhJFlXe6E5WA+ducQfo50SujZwHMvwIAAP//RMCG9UoEAAA",
  "objectset.rio.cattle.io/id": "rke-machine",
  "objectset.rio.cattle.io/owner-gvk": "rke.cattle.io/v1, Kind=RKEBootstrap",
  "objectset.rio.cattle.io/owner-name": "custom-c45b04e1f4bb",
  "objectset.rio.cattle.io/owner-namespace": "fleet-local",
  "rke.cattle.io/join-url": "https://<IP_ADDRESS>:9345",
  "rke.cattle.io/labels": "{\"harvesterhci.io/managed\":\"true\"}"
}

{
  "objectset.rio.cattle.io/applied": "H4sIAAAAAAAA/4yTz67aPBTEX+XTWdt8BJxwHambql3dRaWrqpuqC8c+Jrlx7NQ+/CmId68SQAIKtMs4M6PxL5M9dEjKKFJQ7kF5H0hRE3waHkP1jpoS0iQ2YaIVkcNJE/5vDJQQW+Sd0nXjEdhDadh4jHy5bo+OizfrjP332njz4e3188cQKFFU/V+TvOoQStCrRKHj0lRzOZvNZSWqf7KmXunBbx0icRe0cnBgoCOOt/7adJhIdT2UfuUcA6cqdCML7VaJME62vH1JQ+jp4NzomMXu6IKnGBzvnfKDkOLqGbBapRpKEFNt86yaalQap7KYC2MKO5MSZ9JOlcpEbvK8MMBusD6odSO67MRjcBfFrpVI2jwVnCbAx03k/Y5M/55L23bJC22MWK/jptd+29fbLi/Wy1rv6sXy58aJxbbY2RezsA8zn37sa8smxBbjVdMDg8cB5+mOCOCovDcOBuN43tBiRK8xQfl9D6pvvmFMTfB3Zg0MKhd0+2UwfkKHNOqGTgxO5B3G80nb+AHdzV/w9OqrEfY0s9baheQzWWVcVEXGVZYLnuVCIJoMZbGAw48DA/rV4x9FrwAcfgcAAP//AMqU6ggEAAA",
  "objectset.rio.cattle.io/id": "rke-machine",
  "objectset.rio.cattle.io/owner-gvk": "rke.cattle.io/v1, Kind=RKEBootstrap",
  "objectset.rio.cattle.io/owner-name": "custom-9db392239b4b",
  "objectset.rio.cattle.io/owner-namespace": "fleet-local",
  "rke.cattle.io/join-url": "https://<IP_ADDRESS>:9345"
}

{
  "objectset.rio.cattle.io/applied": "H4sIAAAAAAAA/4RTzW7bPBB8lQ97Fv3phxQlAb0U7SmHAkHRS93DilzFqihSIGmnhaF3L6TYaO3EzlHkzHB2NHuEkSJqjAjNEdBaFzH2zobl07U/ScVAceN7t1EYo6FN7/7vNTTgB2Ijql1vCZKbUPdsybOnw/DC+OfmkCX/PfRWf3h8+PzRuRiix+ldJYsjQQNqH6IbWZGlUpQk67R438RCDROqhd8ZosiMU2gguTJmsCUToIHjFnboDxQi+Z3ql7sRLT6R3kKzhej3tIUZ5gSUpzW1r/1IIeI4QWP3xiRw1jqCMvtFZ/OLDVVYpE4H54nOXm68CM363p0hdxh20EDJVdFlHNsKZSdQ1LrsRFt2IhfES5lzmfGaNL4a+4afK5Cz0TvDJoOWmHeG/hq7RFJU+i7g1B22lomUaoUWlai5rNOO15prkdZdXVSY60JrQp1RysusTVslOC9JpTlVIi9lpoub4nfrckl5dn4gf2F5TuC2wLn8axbwgny7Xmv9HqkjT1ZRgOb7EXDqv5EPvbNvLAYk0Bqnhi8L8RMZiitu8ZTA6RcY8ueTobdLhld7dHf0/Zp6XadSpFqzOstKxotWMJRasbxSkiSXLeUZzD/mBOLviV4ZvQhg/hMAAP//SkIg9UoEAAA",
  "objectset.rio.cattle.io/id": "rke-machine",
  "objectset.rio.cattle.io/owner-gvk": "rke.cattle.io/v1, Kind=RKEBootstrap",
  "objectset.rio.cattle.io/owner-name": "custom-310756e7903e",
  "objectset.rio.cattle.io/owner-namespace": "fleet-local",
  "rke.cattle.io/join-url": "https://<IP_ADDRESS>:9345",
  "rke.cattle.io/labels": "{\"harvesterhci.io/managed\":\"true\"}"
}

All I'm able to see, since start of nodes this morning then with label of app=rancher is:
suse-workstation-team-harvester➜  test_2907  ᐅ  kubectl -n cattle-system logs -l app=rancher --all-containers=true --since=26h
W1014 19:03:35.250264      34 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1014 19:04:20.180915      34 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1014 19:10:16.248878      34 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1014 19:13:58.175360      34 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1014 19:18:27.249111      34 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1014 19:19:53.185499      34 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1014 19:23:29.271959      34 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1014 19:27:42.201027      34 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1014 19:29:37.277673      34 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1014 19:35:28.209398      34 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1014 19:17:34.172538      34 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: got short buffer with n=0, base=4092, cap=9472") has prevented the request from succeeding
2022/10/14 19:21:31 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022/10/14 19:21:31 [INFO] Downloading repo index from https://releases.rancher.com/server-charts/stable/index.yaml
2022/10/14 19:26:31 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022/10/14 19:26:31 [INFO] Downloading repo index from https://releases.rancher.com/server-charts/stable/index.yaml
2022/10/14 19:31:31 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022/10/14 19:31:31 [INFO] Downloading repo index from https://releases.rancher.com/server-charts/stable/index.yaml
2022/10/14 19:34:28 [ERROR] Error during subscribe websocket: close sent
2022/10/14 19:36:31 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2022/10/14 19:36:31 [INFO] Downloading repo index from https://releases.rancher.com/server-charts/stable/index.yaml
W1014 19:03:02.847463      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1014 19:06:05.392444      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1014 19:10:15.850407      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1014 19:14:08.395518      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1014 19:18:46.858603      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1014 19:21:24.411967      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1014 19:25:18.862335      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1014 19:27:08.417431      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated
W1014 19:32:29.872754      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1014 19:35:21.422571      33 warnings.go:80] network.harvesterhci.io/v1beta1 NodeNetwork is deprecated

I'm not sure :sweat_smile:  - why having the nodes just restart seems to have fixed everything?

@irishgordo
I haven't tried rebooting nodes while stuck in the upgrade process yet, so I'm not quite sure if the upgrade will proceed or fail.
Assume the upgrade will proceed after rebooting the nodes. From @bk201's observation, the number of instance-manager-r Pods is more than 1 on v103-node-0. This is because there are both old and new versions of instance-manager-r Pod running simultaneously. Somehow the old one did not get killed/terminated. Rebooting the node might aid in this behavior, just my guess. I will try to reproduce it and do the same thing (reboot) on the cluster.

@w13915984028
Yes, #2923 only partially fixes, or more precisely, prevents the situation of subsequent pre/post-draining on multiple nodes from happening. Still finding ways to cope with this if the issue does not solve from the upstream.

@irishgordo
I just did the same thing as you did: reboot the cluster. The originally stuck upgrade seems to start working again.
I noticed that some of the old instance-manager-r Pods are being deleted, but some still remain on the system. In that case, we have to find out and shut down, i.e. set replicas to 0, the workload which is using the volume that the old instance-manager-r is managing. That way, the old instance-manager-r gets deleted, and the upgrade can continue.
The above-mentioned workaround and root cause are detailed here.

@starbops  thanks for looking into this! :smile:
I spent about 3.5hrs working through another run of this- setting up the env and then running the updates on 3 8C 16GiB 200GiB Disk QEMU/KVM VMs on a supermicro 1u.
Testing upgrading from v1.0.3 to Harvester version v1.1.0-rc3.
I wasn't able to see it be hung like it was in the past.
In order to prepare the 3 node cluster I needed to run through:
cat > /tmp/fix.yaml <<EOF
spec:
  values:
    systemUpgradeJobActiveDeadlineSeconds: "3600"
EOF

then:
kubectl patch managedcharts.management.cattle.io local-managed-system-upgrade-controller --namespace fleet-local --patch-file=/tmp/fix.yaml --type merge

then:
kubectl -n cattle-system rollout restart deploy/system-upgrade-controller

As the first time I did encounter the "Job was active longer than specified deadline" - https://github.com/harvester/harvester/issues/2894
Interesting Things I Saw:

"rancher-uiid"  CAPI Reconciler error

rancher-59b5c9cf89-vfsls 2022/10/18 05:31:15 [ERROR] [CAPI] Reconciler error: timed out waiting for the condition  rancher-59b5c9cf89-vfsls 2022/10/18 05:31:15 [ERROR] [planner] rkecluster fleet-local/local: error encountered during plan processing was Operation cannot be fulfilled on machines.cluster.x-k8s.rancher-59b5c9cf89-vfsls 2022/10/18 05:31:15 [ERROR] [planner] rkecluster fleet-local/local: error encountered during plan processing was Operation cannot be fulfilled on machines.cluster.x-k8s.rancher-59b5c9cf89-vfsls 2022/10/18 05:31:15 [ERROR] [planner] rkecluster fleet-local/local: error encountered during plan processing was Operation cannot be fulfilled on machines.cluster.x-k8s.rancher-59b5c9cf89-vfsls 2022/10/18 05:31:16 [ERROR] [planner] rkecluster fleet-local/local: error encountered during plan processing was Operation cannot be fulfilled on machines.cluster.x-k8s.rancher-59b5c9cf89-vfsls 2022/10/18 05:31:16 [ERROR] [planner] rkecluster fleet-local/local: error encountered during plan processing was Operation cannot be fulfilled on machines.cluster.x-k8s.rancher-59b5c9cf89-vfsls 2022/10/18 05:31:16 [ERROR] [CAPI] Reconciler error: timed out waiting for the condition

Harvester webhook requing often for:

harvester-webhook-5d59686d54-mzplz E1018 05:23:50.779850       7 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: autoscaling/v2: the server could not find the requested resource, flo

Fleet agent, mentioning that that harvester was progressing but with error message:

fleet-agent-58fd796595-w8tdk time="2022-10-18T05:28:16Z" level=error msg="bundle mcc-harvester: deployment.apps harvester-system/harvester [progressing] Available: 2/3"

Fleet agent error deployment doesn't have minimum availability

fleet-agent-58fd796595-w8tdk time="2022-10-18T05:27:52Z" level=error msg="bundle mcc-harvester: deployment.apps longhorn-system/longhorn-conversion-webhook [progressing] Deployment does not have minimum availability., Avail

Occasional helm-install-rke2-coredns error:

FROM
kube-system                    helm-install-rke2-coredns-5jcxf
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  4m5s                  default-scheduler  Successfully assigned kube-system/helm-install-rke2-coredns-5jcxf to vm-v103-node-c
  Normal   Pulling    4m5s                  kubelet            Pulling image "rancher/klipper-helm:v0.7.3-build20220613"
  Normal   Pulled     3m47s                 kubelet            Successfully pulled image "rancher/klipper-helm:v0.7.3-build20220613" in 17.63559317s
  Normal   Created    116s (x5 over 3m47s)  kubelet            Created container helm
  Normal   Started    116s (x5 over 3m47s)  kubelet            Started container helm
  Normal   Pulled     116s (x4 over 3m34s)  kubelet            Container image "rancher/klipper-helm:v0.7.3-build20220613" already present on machine
  Warning  BackOff    98s (x9 over 3m29s)   kubelet            Back-off restarting failed container

a helm-install-rke2-multus error

FROM
kube-system/helm-install-rke2-multus-2tgk9
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  4m55s                  default-scheduler  Successfully assigned kube-system/helm-install-rke2-multus-2tgk9 to vm-v103-node-b
  Normal   Pulled     2m54s (x5 over 4m52s)  kubelet            Container image "rancher/klipper-helm:v0.7.3-build20220613" already present on machine
  Normal   Created    2m54s (x5 over 4m52s)  kubelet            Created container helm
  Normal   Started    2m54s (x5 over 4m52s)  kubelet            Started container helm
  Warning  BackOff    2m8s (x10 over 4m23s)  kubelet            Back-off restarting failed container
#### some of the log####
+ helm_v3 mapkubeapis rke2-multus --namespace kube-system
Already installed rke2-multus
2022/10/18 05:18:22 Release 'rke2-multus' will be checked for deprecated or removed Kubernetes APIs and will be updated if necessary to supported API versions.
2022/10/18 05:18:22 Get release 'rke2-multus' latest version.
2022/10/18 05:18:22 Check release 'rke2-multus' for deprecated or removed APIs...
2022/10/18 05:18:22 Finished checking release 'rke2-multus' for deprecated or removed APIs.
2022/10/18 05:18:22 Release 'rke2-multus' has no deprecated or removed APIs.
2022/10/18 05:18:22 Map of release 'rke2-multus' deprecated or removed APIs to supported versions, completed successfully.
+ echo 'Upgrading helm_v3 chart'
+ echo 'Upgrading rke2-multus'
+ shift 1
+ helm_v3 upgrade --set-string global.cattle.systemDefaultRegistry= --set-string global.clusterCIDR=<IP_ADDRESS>/16 --set-string global.clusterCIDRv4=<IP_ADDRESS>/16 --set-string global.clusterCIDRv6= --set-string global.clusterD
Upgrading rke2-multus
Error: UPGRADE FAILED: chart requires kubeVersion: >= v1.24.6 which is incompatible with Kubernetes v1.22.12+rke2r1
+ exit
Stream closed EOF for kube-system/helm-install-rke2-multus-2tgk9 (helm)

something from hvst-upgrade-uuid-apply-manifests, about fluentbit root daemonsets.apps

2022-10-18T04:51:06.913504088Z Apply resource file
2022-10-18T04:51:11.129036311Z namespace/cattle-logging-system created
2022-10-18T04:51:11.187194291Z managedchart.management.cattle.io/rancher-logging-crd created
2022-10-18T04:51:11.263464611Z managedchart.management.cattle.io/rancher-logging created
2022-10-18T04:52:01.290019726Z Wait for rollout of logging and audit
2022-10-18T04:52:02.033226752Z deployment "rancher-logging" successfully rolled out
2022-10-18T04:52:02.357286991Z Error from server (NotFound): daemonsets.apps "rancher-logging-root-fluentbit" not found
2022-10-18T04:55:28.352888094Z Stream closed EOF for harvester-system/hvst-upgrade-sdcw5-apply-manifests-bg6lm (apply)
### same sorta but displayed different ###
Apply resource file
namespace/cattle-logging-system created
managedchart.management.cattle.io/rancher-logging-crd created
managedchart.management.cattle.io/rancher-logging created
Wait for rollout of logging and audit
deployment "rancher-logging" successfully rolled out
Error from server (NotFound): daemonsets.apps "rancher-logging-root-fluentbit" not found
curl: (28) Operation timed out after 60001 milliseconds with 0 bytes received
[ERROR]  000 received while downloading Rancher connection information. Sleeping for 5 seconds and trying again

lots of logs surrounding RFC 1123 with harvester-containerd-registry:

time="2022-10-18T04:43:45Z" level=error msg="error syncing 'cattle-system/sync-containerd-registry': handler system-upgrade-controller: secrets \"harvester-containerd-registry\" not found, handler system-upgrade-controller: failed to create cattle-system/apply-sync-containerd-registry-on-vm-v103-node-c-with- batch/v1, Kind=Job for system-upgrade-controller cattle-system/sync-containerd-registry: Job.batch \"apply-sync-containerd-registry-on-vm-v103-node-c-with-\" is invalid: [metadata.name: Invalid value: \"apply-sync-containerd-registry-on-vm-v103-node-c-with-\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.template.labels: Invalid value: \"apply-sync-containerd-registry-on-vm-v103-node-c-with-\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')], requeuing"

For all interesting things I saw, they seemed to resolve themselves and did not impact the upgrade.
The upgrade itself did take about 2 hours, on 3 nodes.
The Process I Noticed

First to upgrade was the 'main' (original) node, entered pre-drain, then post-drain, then reboot, then succeeded
Second to upgrade was the node-c, entered pre-drain, then post-drain, then reboot, then succeeded
Third to upgrade was node-b, pre-drain...however, I didn't catch it really at any other states I don't believe, it just like then immediately went to reboot...I'm not sure

Test Artifacts

Additional Test Artifact
I have the full length video 1.4Gi available if desired.
Here is it sped up a lot.
https://user-images.githubusercontent.com/5370752/196349175-8d7fece7-b70f-4f6c-bb97-6a2969b713f3.mp4
@starbops  - since it succeeded and the things that I noticed that arose ended up seeming to resolve themselves, I feel like we could close this out, pending that we have workarounds for some of the old instance-manager pods and a few other things like rebooting  the node if it does get stuck as well.
What are your thoughts?

Updating some new observations here.

node-0 is fully upgraded
node-1 is pre-drained and entering the draining state
node-2 is somewhat mistakenly put in a SchedulingDisabled mode by Rancher after node-1 is pre-drained
node-3 is untouched yet

In this kind of situation, the fix in #2923 can actually guard the upgrade procedure. It will hold up the pre-drain Job that's going to be placed on node-2 and let the draining and post-drain Job finish on node-1 without any interference. After node-1 is fully upgraded, the pre-drain Job will be placed on node-2 and proceed with the upgrade.
But if node-2 is put in the SchedulingDisabled mode by Rancher before node-1 is pre-drained, there is a high chance that the pre-drain Job on node-1 will be stuck in the waiting for all LH volumes to become healthy. The workaround for this is to uncordon node-2 or set the degraded volumes' numberOfReplicas to 2 or 1 to make them healthy (prefer the former one).

@irishgordo It's normal to see some error logs during an upgrade since there are versions bump or node offlines. I left some comments here:

We need to check. If the cluster is ready at the end and it should be OK. Run kubectl get clusters.provisioning.cattle.io local -n fleet-local  -o yaml to check if it's ready at the end.
autoscaling/v2 is deprecated in v1.23. (We upgrade to v1.24.x).
3, 4: this should be proper behavior during an upgrade since we turn on and off nodes.
5,  6.: This is due to the Kubernetes version being mismatched during an upgrade: some nodes are v1.24.x and some are still v1.22. Eventually, those helm-install pods should succeed.
Known issue. The job will run again and succeed.
@FrankYang0529 Please take a look. It might be related to the hostname or something.

@bk201  thanks for helping me understand that some of those errors are normal :smile:
Running:
kubectl get clusters.provisioning.cattle.io local -n fleet-local -o yaml and it seems ready
Yields:
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"provisioning.cattle.io/v1","kind":"Cluster","metadata":{"annotations":{},"labels":{"rke.cattle.io/init-node-machine-id":"7zsvdjpmx7r9fcz7sdsfdzmmd85m8p2x4bhdcdh9ql7jfbxk7q6s8w"},"name":"local","namespace":"fleet-local"},"spec":{"kubernetesVersion":"v1.22.12+rke2r1","rkeConfig":{"controlPlaneConfig":null}}}
    objectset.rio.cattle.io/applied: H4sIAAAAAAAA/4yQzU7DMBCEXwXt2Slt079Y4oAQ4sCVF9jYS2Ow15G9CYfK746SVqJC4udo78xovjlBIEGLgqBPgMxRUFzkPD1j+0ZGMskiubgwKOJp4eKts6ChT3F02UV2fKyMH7JQqkwiFAL1ozV+MKXqOL6DhoCMRwrEciUYa3Xz7NjePZwj/8xiDAQafDTo/yXOPZrJAUXB3NdFfnGBsmDoQfPgvQKPLflfR+gwd6Bhu9ztt3XdUGNwc7Crdr9u6jW1y/pg91vb2LXdbHarA6jzYpbSVwho6DCNNIMWBd9Yrtu+eiKpzpeiIPdkpnbzx2Wq+0G6R7Z9dCygT2WSCcpwwciURrJPxJRmZtDLUj4DAAD//5CVWGcAAgAA
    objectset.rio.cattle.io/id: provisioning-cluster-create
    objectset.rio.cattle.io/owner-gvk: management.cattle.io/v3, Kind=Cluster
    objectset.rio.cattle.io/owner-name: local
    objectset.rio.cattle.io/owner-namespace: ""
  creationTimestamp: "2022-10-16T04:14:02Z"
  finalizers:
  - wrangler.cattle.io/provisioning-cluster-remove
  - wrangler.cattle.io/rke-cluster-remove
  generation: 4
  labels:
    objectset.rio.cattle.io/hash: 50675339e9ca48d1b72932eb038d75d9d2d44618
    provider.cattle.io: harvester
  name: local
  namespace: fleet-local
  resourceVersion: "558214"
  uid: e6c154af-ed85-4e2f-95aa-65f1e55b7798
spec:
  kubernetesVersion: v1.24.6+rke2r1
  localClusterAuthEndpoint: {}
  rkeConfig:
    chartValues: null
    machineGlobalConfig: null
    upgradeStrategy:
      controlPlaneDrainOptions:
        timeout: 0
      workerDrainOptions:
        timeout: 0
status:
  clientSecretName: local-kubeconfig
  clusterName: local
  conditions:
  - status: "True"
    type: Ready
  - status: Unknown
    type: DefaultProjectCreated
  - status: Unknown
    type: SystemProjectCreated
  - lastUpdateTime: "2022-10-16T04:14:03Z"
    status: "False"
    type: Reconciling
  - lastUpdateTime: "2022-10-16T04:14:03Z"
    status: "False"
    type: Stalled
  - lastUpdateTime: "2022-10-18T05:35:25Z"
    status: "True"
    type: Created
  - lastUpdateTime: "2022-10-18T06:53:17Z"
    status: "True"
    type: RKECluster
  - lastUpdateTime: "2022-10-18T04:37:43Z"
    status: "True"
    type: Provisioned
  - lastUpdateTime: "2022-10-18T06:53:17Z"
    status: "True"
    type: Updated
  - lastUpdateTime: "2022-10-18T04:40:23Z"
    status: "True"
    type: Connected
  observedGeneration: 4
  ready: true

For 8. -for a bit more info- I have these logs from my notes:
time="2022-10-18T04:43:45Z" level=error msg="error syncing 'cattle-system/sync-containerd-registry': handler system-upgrade-controller: secrets \"harvester-containerd-registry\" not found, handler system-upgrade-controller: failed to create cattle-system/apply-sync-containerd-registry-on-vm-v103-node-c-with- batch/v1, Kind=Job for system-upgrade-controller cattle-system/sync-containerd-registry: Job.batch \"apply-sync-containerd-registry-on-vm-v103-node-c-with-\" is invalid: [metadata.name: Invalid value: \"apply-sync-containerd-registry-on-vm-v103-node-c-with-\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.template.labels: Invalid value: \"apply-sync-containerd-registry-on-vm-v103-node-c-with-\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')], requeuing"
time="2022-10-18T04:43:45Z" level=error msg="error syncing 'cattle-system/sync-containerd-registry': handler system-upgrade-controller: secrets \"harvester-containerd-registry\" not found, handler system-upgrade-controller: failed to create cattle-system/apply-sync-containerd-registry-on-vm-v103-node-c-with- batch/v1, Kind=Job for system-upgrade-controller cattle-system/sync-containerd-registry: Job.batch \"apply-sync-containerd-registry-on-vm-v103-node-c-with-\" is invalid: [metadata.name: Invalid value: \"apply-sync-containerd-registry-on-vm-v103-node-c-with-\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.template.labels: Invalid value: \"apply-sync-containerd-registry-on-vm-v103-node-c-with-\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')], requeuing"
time="2022-10-18T04:43:48Z" level=error msg="error syncing 'cattle-system/sync-containerd-registry': handler system-upgrade-controller: secrets \"harvester-containerd-registry\" not found, handler system-upgrade-controller: failed to create cattle-system/apply-sync-containerd-registry-on-vm-v103-node-c-with- batch/v1, Kind=Job for system-upgrade-controller cattle-system/sync-containerd-registry: Job.batch \"apply-sync-containerd-registry-on-vm-v103-node-c-with-\" is invalid: [metadata.name: Invalid value: \"apply-sync-containerd-registry-on-vm-v103-node-c-with-\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.template.labels: Invalid value: \"apply-sync-containerd-registry-on-vm-v103-node-c-with-\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')], requeuing"
time="2022-10-18T04:52:45Z" level=error msg="error syncing 'cattle-system/apply-hvst-upgrade-sdcw5-prepare-on-vm-v103-node-b-with-d-b712c': handler system-upgrade-controller: jobs.batch \"apply-hvst-upgrade-sdcw5-prepare-on-vm-v103-node-b-with-d-b712c\" not found, requeuing"

Hi @irishgordo, thank you for spending so much time on this! Just curious what's the type of your disks? Is it HDD? Could be a slow speed causes that many interesting errors that eventually self-healed.
I think we can only wait for rancher/rancher#39167 to reproduce and fix it from the root. This issue is just for a record of what we can do to get around this at this critical moment. Since we have a certain degree of protection and workarounds on our side to overcome it, I think we can close this out as you suggested.
cc @bk201

@starbops you're welcome, I'm glad I could help! :smile:  -- I do think yeah perhaps the there is something funny with the I/O limit on the disk, but the strange thing is, that it is SATA3, a Kingston or Samsung SSD (I don't recall the brand :sweat_smile: )- that's is installed on my 1U server. ( https://www.supermicro.com/products/archive/motherboard/x9drd-it_ )
@bk201 I could close this out and open up another ticket for things found with number 8 -with the RFC-1123 for cattle-system/sync-containerd-registry, if that works

@irishgordo Yes, we can track the cattle-system/sync-containerd-registry issue in another ticket. Thanks!

Hi @irishgordo, for number 8, I can't reproduce it in my cluster. Do you have support-bundle for it? I can take a look. Thank you

@FrankYang0529  - sorry for the late reply :sweat_smile:  - here's the issue for number 8:
https://github.com/harvester/harvester/issues/3001

The root cause of this issue, may be related to Rancher Planner, which selects etcdTier, controlPlaneTier nodes in parallel.
https://github.com/rancher/rancher/issues/39167#issuecomment-1291791590
double-checking with Rancher.


test_search does not work with os=linux (default)

test_search() returns a High School Musical song when you search for a Drake song. When os=Windows in test_search(), the search completes correctly.

Closed with PR #11


GFM is enabled by default in Velite

https://github.com/boh5/Aero/blob/ebbaef3ca2f2b99d25fbd073c64120b2899c8d53/velite.config.js#L48

@zce Thanks! Fixed in https://github.com/boh5/Aero/pull/19.


fix: Add diskcache hard autoreload

Closes #887

To me this a good feature to have especially now that we have an overwrite flag.
LGTM


rely on atomic counter per context instead of time.Now()

hi @duongphuhiep
what do you think of this ?

Thanks for consulting my opinion, there is 2 problems with this implementation

when we wanted to create milion of transients, the current implementation will do atomic.add(1) and save it to the context (which means derive a new context) uselessly million times => we should at least filter out the transient creation for the counter.

atomic counter will force the object invocation to be (needlessly) sequential so that each invocation would be attached to a different counter value. It is not neccessary. The object creation can be concurrent. There's shouldn't be any problem if there are multiple invocations at the same nano-second and attached to the same counter value.

In fact, I had considered using "hard" atomic counter like this at first but decided to replaced it with time.Now as a "natural soft counter". ("soft" counter means that , it is not absolute, multiple invocations happening at the same nanosecond would be attached to the same counter value).
This counter is a "nice to have" information, we don't need to make it this "hard". The "invocation level" is the real important one. For eg:
When NewA calls NewB and NewC like this: NewA( NewB(), NewC() ) then it is more important to dispose B and C before A.
However, the order of B and C are usually not as important. If B is invoked before C then the counter would help us to  dispose C before B as a "nice to have" behavior, but evens without the counter and we dispose C before B, it usually not a problem.

thanks @duongphuhiep for your valuable time
now i can understand your reasoning well
when i decided to move away from relying on time.Now my argument was we cannot actually rely on the user's system clock
because of numerous issues, the biggest of them was the OS may adjust the clock automatically in some cases like in some countries where they have DST (daylight saving time)
but you've cleared out the mishaps of using atomic counters and after a thoughtful reading on Go's implementation of time.After
i came to the conclusion that Go (whenever possible) uses the system's monotonic clock to compare 2 Time objects
and did a little experiment where i called time.Now then sleep for about 30 seconds and then edit my system's clock to set it back to yesterday
and then again called time.Now and compared the 2 Time objects using func (t Time) After(u Time) bool
it reported that the second time is still AFTER the first time, even though the second one is a day behind!

so that being cleared up, i no longer see any use to abandon our dependence on time.Now
thank you for helping out, your contributions are always very helpful


Add a CI workflow to lint .editorconfig

If anyone is aware of a github action that will lint the repo files to comply with .editorconfig, please let us know here.
We've tried https://github.com/megalinter/megalinter and https://github.com/github/super-linter configured to run only https://github.com/editorconfig-checker/editorconfig-checker, but they're both very slow.

Apparently editorconfig-checker provides the official Docker images too: https://hub.docker.com/r/mstruebing/editorconfig-checker.  As GitHub Actions allows to run any existing Docker images in workflows, it might be worth a try.

@dahlia Works like a charm, thank you!


chat historial

Hello, it's a very good work, congratulations.!!!
I need save chat historial, u can help me with this or tell me, If i can do on oracle db or i don't know "log file or mongo"???

Hi @jfernal
the most appropriate solution would be to store chat in oracle db, because apex then have "easy" access to this information. But this is big upgrade that will take time. Feel free to fork this repo and start working on this feature.
Best regards
isabolic

Thank @isabolic , i'll try with this upgrade. I ask you for help if you don'd mind.
I hope that you continues working with amazing projects as this.
Best regards.
jfernal


feat: Add a new Ruler tool to measure distances on a map

This PR adds a very basic implementation of a new map tool to measure distance

Add RulerMapTool with a new SVG icon
Add grid to SharedMapToolState to have access to both grid color (and use it for ruler's line) and cell size (to measure distance in cells)
Adds toolbar to player view. This toolbar contains Drag, Mark and Ruler tools.

Any feedback about implementation / design / usage of GraphQL etc. very welcome!

@petertrr Are you on the discord server?


Add zsh completion for 'docker network create --ipam-opt'

+ Reordering for better maintainability
Ref. #17316
Signed-off-by: Steve Durrheimer<EMAIL_ADDRESS>

LGTM

LGTM
ignoring tests, because this is not tested by CI


Update Light Client to raiden develop

[x] raiden-contracts v0.33.3
[x] update messages fields, packing & signing
[x] IOU fixes
[ ] update transport

As requested, I'm going to split this PR in smaller ones.


[Feature Request] Windows on top

Would it be possible to add a toy to allow windows to be easily pinned in front of other windows?  e.g. through a new icon next to the standard three (Minimize/Maximize/Close).

Tracked by #13


Button to copy curl command has misleading label

Should be "Copy command" not "Copy link"

@hannes-ucsc, got it, thank you.

Related to #1532 and humancellatlas/data-portal#852.


Exec interface review part 3

@steiler would you be ok addding to this branch the following functionality:

When command fails to execute (rc is not 0 or stderr has some bytes), we should emit an error in the log with stderr and rc values present
the error doesn't stop the code exec, it is meant to provide a visual guidance that the intent is not fully implemented

I used the following lab
name: linux

topology:
  nodes:
    l1:
      kind: linux
      image: alpine
      exec:
        - ash -c 'cat /etc/os-release | grep NAME' # correct cmd
        - "fools -la" # incorrect cmd, a warning should be produced and stderr should contain the error
    l2:
      kind: linux
      image: alpine
      exec:
        - 'echo "l2 node"' # correct cmd
    test-bridge: # should exist for this lab to run
      kind: bridge
      exec:
        - "ip link" # should be a noop, since execs have no meaning

  links:
    - endpoints: ["l1:eth1", "l2:eth1"]

Node l1 has a second exec that uses a command that doesn't exist, and currently, the log doesn't catch this as an error:
INFO[0001] Executed command 'fools -la' on l1. stdout:
 CI runtime exec failed: exec failed: unable to start container process: exec: "fools": executable file not found in $PATH: unknown


Feature request: Tree Select

It would be fantastic if this fantastic ui library included a tree select. I have been looking for this component everywhere, and only a few have it, but their style does not give me the same modern look as yours. I really hope that you can implement this.

Give example of tree select you meant!

Oh, sure thing !
Something like antd would do, here is the example.


ecommerce slider dynamic group not showing

PHP Version
8.2
Shopware Version
6.6.3
Expected behaviour
show a slicer with my dynamic group prefferences, or in this case products that are in stock and products that are not in stock but has no clearance set
Actual behaviour
it shows nothing ,even when advanced view shows the correct one , working on shopware <IP_ADDRESS> with no problem , broken on shopware 6..6.2 ,and up.
How to reproduce
create a dynamic group and put any value to pull out
create now a template and use the ecomerce slider so you can assign a dynamic group
it wont show up while it should show up

Same here, after an update from <IP_ADDRESS> to <IP_ADDRESS>
Manual assignment works, Dynamic product group assignment does not work inside Product sliders for me.
maybe relevant, I get the following error message in console if I add the slider to my layout:
An error was captured in current module: TypeError: this.$refs.productHolder is undefined

Same here. Also Upgraded from <IP_ADDRESS> to the newest version <IP_ADDRESS> - Product sliders does not show anything anymore. Manual Product selection is not working as well.
Did not find anything in the logs.

There was an issue in this line, that seems to be fixed in trunk. Until that, there was only a check if (!$variantConfig) {, but it always existed.
https://github.com/shopware/shopware/blob/24cf40c4509c0c65a0f18779cdf61c314b57c565/src/Core/Content/Product/Cms/ProductSliderCmsElementResolver.php#L237

[public] Automated response: This issue is linked to the internal issue https://shopware.atlassian.net/browse/NEXT-37799.
[created from NEXT-37799, comment 488233]

[public] The issue is already covered by the following ticket: https://github.com/shopware/shopware/issues/3949"
[created from NEXT-37799, comment 491448]


Extra Address Operator

https://github.com/troglobit/pimd/blob/11673d27c22420054247f002f049d3af64a348d2/src/route.c#L721
calc_oifs() calls for a pointer to the oifs.  Since oifs was declared as an array, that address would be 'oifs' not '&oifs'.

Thanks, fixed on master now.


chore: remove Ruby Pinning from 0.70

Summary
Starting from 0.72, we won't be pin any specific ruby version. However, 0.70 and 0.71 are still doing that.
This PR against 0.70, removes the pinning of a specific version of Ruby
Changelog
[iOS][Changed] - Do not pin a specific version of Ruby.
Test Plan

CircleCI should stay green

/rebase


BlockingLoadBalancerClient does not work with KubernetesDiscoveryClient

Hi,
I'm trying to migrate from ribbon LB to newly introduced blocking LB.
Here is a test replicating the issue:
    @Test
    public void testReconstructURIForK8S() {
        ServiceInstance instance = new KubernetesServiceInstance("auth-server", "auth-server",
                new EndpointAddress("hostname", "<IP_ADDRESS>", "nodeName", null),
                new EndpointPort("http", 8091, "tcp"),
                Collections.emptyMap(),
                false);
        URI uri = LoadBalancerUriTools.reconstructURI(new TruncateSchemaServiceInstance(instance),
                URI.create("http://auth-server/oauth/token"));

        assertThat(uri.toString()).isEqualTo("http://<IP_ADDRESS>:8091/oauth/token");
    }

Fails with:
   001:  "java.lang.IllegalStateException: Could not create URI object: Expected hostname at index 7: http://://<IP_ADDRESS>:8091/oauth/token"
   002:  "at org.springframework.web.util.HierarchicalUriComponents.toUri(HierarchicalUriComponents.java:514)"
   003:  "at org.springframework.cloud.client.loadbalancer.LoadBalancerUriTools.doReconstructURI(LoadBalancerUriTools.java:111)"
   004:  "at org.springframework.cloud.client.loadbalancer.LoadBalancerUriTools.reconstructURI(LoadBalancerUriTools.java:95)"
   005:  "at org.springframework.cloud.loadbalancer.blocking.client.BlockingLoadBalancerClient.reconstructURI(BlockingLoadBalancerClient.java:75)"
   006:  "at org.springframework.cloud.client.loadbalancer.ServiceRequestWrapper.getURI(ServiceRequestWrapper.java:43)"
   007:  "at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:97)"
   008:  "at org.springframework.cloud.client.loadbalancer.LoadBalancerRequestFactory.lambda$createRequest$0(LoadBalancerRequestFactory.java:61)"
   009:  "at org.springframework.cloud.loadbalancer.blocking.client.BlockingLoadBalancerClient.execute(BlockingLoadBalancerClient.java:62)"

The issue is that KubernetesServiceInstance returns for schema either "http://" or "https://".
Current implementation of LoadBalancerUriTools except schema to be "http" or "https".
I've checked implementation of ServiceIntance for Eureka, it returns null when getSchema is called.
Should we fix issue in KubernetesServiceInstance or adjust LoadBalancerUriTools to trim "://" during URI reconstruction? (second one looks more safe).
Thanks,
Artyom

@OlgaMaciaszek , hi Olga
What do you think about this issue? Should I fix it in LoadBalancerUriTools?
Thanks

Should we fix issue in KubernetesServiceInstance or adjust LoadBalancerUriTools to trim "://" during URI reconstruction? (second one looks more safe).

my vote for removing :// from KubernetesServiceInstance side (scheme is a URI part without :// https://tools.ietf.org/html/rfc3986#section-3.1)

Unfortunately we can't change the returned value of getScheme at this moment since that introduces a breaking change. AFAIK the only viable option would be to adapt LoadBalancerUriTools to strip :// from the scheme like @ArtyomGabeev already proposed.
Please keep your first PR open so we can merge this once we start working on the next major release.

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

@ArtyomGabeev sure, feel free to provide a PR.

I'm having the same issue here. when will this PR be merged?

I've opened another PR to trim double slash in util class.
We are not closing this one, probably on next major release we will be able to merge it as well.

Wait, why can't we fix kubernetes service instance? I don't want work around in a common library for this one's bug.

Hi, @TYsewyn @spencergibb
Are we waiting for next major release to get it merged?
Thanks


Register

@JuliaRegistrator register

@JuliaRegistrator register

@JuliaRegistrator register

Registration pull request created: JuliaRegistries/General/34754
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.7 -m "<description of version>" e58d6697f271fa10d65f82940d2531ae0fecf727
git push origin v0.1.7

@JuliaRegistrator register()

Registration pull request created: JuliaRegistries/General/53003
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v1.0.0 -m "<description of version>" c8ce3cedb21b695ab062e081c54bef060ca85509
git push origin v1.0.0


Add The Index to README

Add the file tracking service The Index to README

Will this hash need to be updated with each update? Might be good to use IPNS to reference it.

The hash will not need to be updated with each update. There is a script on
the page that tries to load the ipns and asks the user if they want to
switch to the new site if it is available. However, since IPNS de-resolves
after 12-48 hours, IPNS is not guaranteed to always be present, but this
version of the webapp will always be able to load the db either from ipfs
or from the api server that it connects to.
On Mon, Feb 6, 2017 at 10:53 AM Richard Littauer<EMAIL_ADDRESS>wrote:

Will this hash need to be updated with each update? Might be good to use
IPNS to reference it.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ipfs/awesome-ipfs/pull/78#issuecomment-277723984, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABWEjQpPKoPrerFuTYTl7yjs7Mm1GECaks5rZ0HbgaJpZM4L4PxI
.

Hey @cakenggt - on a closer look, I do not think I can currently include this. The Index very clearly contains a variety of copyrighted materials. Please read our Code of Conduct, and, specifically, the section on Copyright Violations. We take this subject very seriously. Please respect the Code of Conduct.
That having been said, if you can clean the Index of items which violate the Code, then we can add it in. For now, I am going to say no. Feel free to message me when you're ready for a second pass over the Index.


FirewallRule objects fail to serialize

It is caused by an underlying failure of the IAddress [] to properly serialize.
Here's a minimum viable demo of the broken behavior:
using System;
using System.Linq;
using Newtonsoft.Json;
using WindowsFirewallHelper;

namespace FirewallHelperTest
{
    class Program
    {
        static void Main(string[] args)
        {
            foreach (IFirewallRule rule in FirewallManager.Instance.Rules.ToArray())
            {
                Console.WriteLine(JsonConvert.SerializeObject(rule));
            }
        }
    }
}

Which results in:
Newtonsoft.Json.JsonSerializationException
  HResult=0x80131500
  Message=Error getting value from 'ScopeId' on 'WindowsFirewallHelper.Addresses.SingleIP'.
  Source=Newtonsoft.Json
  StackTrace:
   at Newtonsoft.Json.Serialization.ExpressionValueProvider.GetValue(Object target)

Inner Exception 1:
SocketException: The attempted operation is not supported for the type of object referenced

You don't need to serialize the whole Rule for this behavior.  I isolated each field to determine it was the addresses.  For example, this also will not work:
JsonConvert.SerializeObject(rule.LocalAddresses)

Hey @gfs,
None of the classes in this library is serializable. Therefore this is expected. A lot of properties might throw NotSupportedException or InvalidOperationException.
Also, there is a 1-1 relationship for IRule classes and the underlying COM objects (in fact every property is requested and then translated from the COM object when used) and even if you serialize them it is not possible to deserialize that exact instance. I can make it more serialize friendly but the extent of these facts won't allow for full compatibility.
I suggest you use another class to hold the information you need and then try to serialize that one.
For your specific problem with the SingleIP class, however, since it is a child of the IPAddress class you might be interested in the following SO question:
https://stackoverflow.com/questions/18668617/json-net-error-getting-value-from-scopeid-on-system-net-ipaddress

Thanks for the response! Makes sense to me.


VSCode extensions folder searched

Issue Type: Bug

Install GitLens, or run the Gitlens: Welcome command
Search setTimeout

A result is included from .vscode-insiders/extensions, the welcome.html from the GitLens webview. Also repros if the welcome webview is closed before searching. I'm not sure if this is an issue with what GitLens itself is doing, but I wouldn't expect files within the vscode extensions folder to be indexed and searched. This also repros with stable.

VS Code version: Code - Insiders 1.25.0-insider (923e9ccb3fc44bfaaf4b4049015c84e42bc40f95, 2018-06-26T13:00:22.932Z)
OS version: Darwin x64 17.4.0
Extensions (6)

Extension
Author (truncated)
Version

vscode-eslint
dba
1.4.12

gitlens
eam
8.4.1

vscode-rust
kal
0.4.2

vscode-pull-request-github
Mic
0.0.1

vscode-language-pack-hu
MS-
1.25.0

debugger-for-chrome
msj
4.7.0

(1 theme extensions excluded)

Fixed with another issue in 1.28


Suggestion in loading binaries from FFMpeg

This suggestion for a Pyglet change was made to Arcade:
https://github.com/pvcraven/arcade/issues/485
Bug Report
This bug is specific to ARM Raspberry Pi 4 devices.
A fix is proposed in Enhancement request section.
When trying to import arcade for the first time after installation in Python 3, an IndexError is thrown when the script pyglet_ffmpeg2/loader.py calls the function _ensure_linux_symlinks
Actual behavior:
load.py fails to create symlinks because the path defined in the bin_folder variable defined by the function _locate_binaries uses
path = os.path.join(this_dir, 'linux_x86_64')

instead of:
path = os.path.join(this_dir, 'RPi')

because sys.platform returns 'linux' instead of the expected 'armv7l' in the conditional block:
if 'armv7l' in sys.platform:
    path = os.path.join(this_dir, 'RPi')
else:
    path = os.path.join(this_dir, 'linux_x86_64')

Expected behavior:
sys.platform  should return 'armv7l' and symlinks can be created
Steps to reproduce/example code:

Install Raspbian GNU/Linux 10 (buster) on Raspberry Pi 4
Upgrade pip3
Run pip3 install arcade
Start python 3 and import arcade

Enhancement request:
What should be added/changed?
In /usr/local/lib/python3.7/dist-packages/pyglet_ffmpeg2/loader.py, change the condition on line 50 from this:
if 'armv7l' in sys.platform

to this:
if 'armv7l' in sys.platform or 'armv7l' in os.uname()

What would it help with?
os.uname() returns more comprehensive and useful info on the OS and hardware
In addition, the os module is already imported in the load.py file

Hi Paul,
Isn't this package under the pythonarcade org?  https://github.com/pythonarcade/pyglet-ffmpeg/
I believe that Dan wrote this originally, but it seems to live there.
In that case I don't think it should be a problem to update :)

The suggested change was in the pyglet part? I haven't looked too closely.
On Mon, Sep 23, 2019, 9:34 PM Benjamin<EMAIL_ADDRESS>wrote:

Hi Paul,
Isn't this package under the pythonarcade org?
https://github.com/pythonarcade/pyglet-ffmpeg/
I believe that Dan wrote this originally, but it seems to live there.
In that case I don't think it should be a problem to update :)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/pyglet/pyglet/issues/64?email_source=notifications&email_token=AALFGTCGVGERSPG6Y6QKXIDQLF4CVA5CNFSM4IZKUSK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7M3QUA#issuecomment-534362192,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AALFGTFPJ5ATRM353BIQVCDQLF4CVANCNFSM4IZKUSKQ
.

Looks like it's in the pyglet_ffmpeg package:  https://github.com/pythonarcade/pyglet-ffmpeg/blob/master/pyglet_ffmpeg/loader.py#L50
It looks like a fairly simple fix, so I went ahead and did an online pull request with the Github editor.

Oh man, I goofed. Sorry to open this under the wrong project.


NotFoundError: undefined symbol: _ZNK10tensorflow8OpKernel11TraceStringERKNS_15OpKernelContextEb

Environment
Anaconda virtual python environment with TF 2.6.0 installed.
$ conda list | grep tensorflow
tensorflow                2.6.0            py38h1abaa86_1    conda-forge
tensorflow-base           2.6.0            py38he1e5d52_1    conda-forge
tensorflow-datasets       4.3.0              pyhd8ed1ab_0    conda-forge
tensorflow-decision-forests 0.1.9                    pypi_0    pypi    # <---- Installed via PIP as conda is not available
tensorflow-estimator      2.6.0            py38h45e38c2_1    conda-forge
tensorflow-metadata       1.2.0              pyhd8ed1ab_0    conda-forge

$ conda --version
conda 4.10.3

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"

Problem
Importing TF, TFDS are OK.
import os
import math
import numpy as np
import pandas as pd
import tensorflow as tf
import tensorflow_datasets as tfds
from wurlitzer import sys_pipes

tf.__version__
---
2.6.0

Importing the tensorflow_decision_forests causes the issue. The same symbol issues was reported in https://github.com/tensorflow/text/issues/385 as well.
import tensorflow_decision_forests as tfdf
---
---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
/tmp/ipykernel_120273/401816866.py in <module>
----> 1 import tensorflow_decision_forests as tfdf

~/conda/envs/python_38_ml/lib/python3.8/site-packages/tensorflow_decision_forests/__init__.py in <module>
     49 __author__ = "Mathieu Guillame-Bert"
     50
---> 51 from tensorflow_decision_forests import keras
     52 from tensorflow_decision_forests.component import py_tree
     53 from tensorflow_decision_forests.component.builder import builder

~/conda/envs/python_38_ml/lib/python3.8/site-packages/tensorflow_decision_forests/keras/__init__.py in <module>
     47 from typing import Callable, List
     48
---> 49 from tensorflow_decision_forests.keras import core
     50 from tensorflow_decision_forests.keras import wrappers
     51

~/conda/envs/python_38_ml/lib/python3.8/site-packages/tensorflow_decision_forests/keras/core.py in <module>
     58 from tensorflow.python.training.tracking import base as base_tracking  # pylint: disable=g-direct-tensorflow-import
     59 from tensorflow_decision_forests.component.inspector import inspector as inspector_lib
---> 60 from tensorflow_decision_forests.tensorflow import core as tf_core
     61 from tensorflow_decision_forests.tensorflow.ops.inference import api as tf_op
     62 from tensorflow_decision_forests.tensorflow.ops.training import op as training_op

~/conda/envs/python_38_ml/lib/python3.8/site-packages/tensorflow_decision_forests/tensorflow/core.py in <module>
     29 import tensorflow as tf
     30
---> 31 from tensorflow_decision_forests.tensorflow.ops.training import api as training_op
     32 from yggdrasil_decision_forests.dataset import data_spec_pb2
     33 from yggdrasil_decision_forests.learner import abstract_learner_pb2

~/conda/envs/python_38_ml/lib/python3.8/site-packages/tensorflow_decision_forests/tensorflow/ops/training/api.py in <module>
     22 from tensorflow.python.framework import load_library
     23 from tensorflow.python.platform import resource_loader
---> 24 tf.load_op_library(resource_loader.get_path_to_datafile("training.so"))

~/conda/envs/python_38_ml/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py in load_op_library(library_filename)
     56     RuntimeError: when unable to load the library or get the python wrappers.
     57   """
---> 58   lib_handle = py_tf.TF_LoadLibrary(library_filename)
     59   try:
     60     wrappers = _pywrap_python_op_gen.GetPythonWrappers(

NotFoundError: /home/oonisim/conda/envs/python_38_ml/lib/python3.8/site-packages/tensorflow_decision_forests/tensorflow/ops/training/training.so: undefined symbol: _ZNK10tensorflow8OpKernel11TraceStringERKNS_15OpKernelContextEb

Related
There is a suggestion that this could be TF version issue but not certain.

Tensorflow: undefined symbol: _ZN10tensorflow11GetNodeAttrERKNS_9AttrSliceEN4absl14lts_2020_09_2311string_viewEPSs

Try to uninstall tensorflow (in mine colab it was 2.6.0) install tf 2.5.0 and then install tensorflow_decision_forests.

tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python3.6/dist-packages/tensorflow_text/python/metrics/_text_similarity_metric_ops.so: undefined symbol: _ZN10tensorflow8OpKernel11TraceStringEPNS_15OpKernelContextEb #385

Hi @oonisim,
Your message falls right around the release of "tensorflow-decision-forests" 0.2.0, so wonder if this could be related to it.
Would you mind:

Updating to the last version of TF and TF-DF.
Restarting your notebook instance (if you use notebooke/colabs).
Trying again to import TF-DF.

Hi @achoum, please refer to the result below. This time I switched to Python venv (not virtualenv, etc) and pip only.
Basically the same error.
Python environment
$ $ python --version
Python 3.8.10

$ pip list | grep tensorflow
tensorflow                   2.7.0
tensorflow-datasets          4.4.0
tensorflow-decision-forests  0.2.0
tensorflow-estimator         2.7.0
tensorflow-io-gcs-filesystem 0.21.0
tensorflow-metadata          1.4.0

$ pip list | grep jupyter
jupyter                      1.0.0
jupyter-client               7.0.6
jupyter-console              6.4.0
jupyter-core                 4.9.1
jupyterlab-pygments          0.1.2
jupyterlab-widgets           1.0.2

Jupyter Notebook
import tensorflow as tf
import tensorflow_datasets as tfds
from wurlitzer import sys_pipes

tf.__version__
---
`2.7.0'     # <--- PyPi tensorflow repository is updated to 2.7.0

tf.__path__
---
['/home/oonisim/venv/ml/lib/python3.8/site-packages/keras/api/_v2',
 '/home/oonisim/venv/ml/lib/python3.8/site-packages/tensorflow_estimator/python/estimator/api/_v2',
 '/home/oonisim/venv/ml/lib/python3.8/site-packages/tensorboard/summary/_tf',
 '/home/oonisim/venv/ml/lib/python3.8/site-packages/tensorflow',
 '/home/oonisim/venv/ml/lib/python3.8/site-packages/tensorflow/_api/v2']

import tensorflow_decision_forests as tfdf
---
otFoundError                             Traceback (most recent call last)
/tmp/ipykernel_48567/401816866.py in <module>
----> 1 import tensorflow_decision_forests as tfdf

~/venv/ml/lib/python3.8/site-packages/tensorflow_decision_forests/__init__.py in <module>
     49 __author__ = "Mathieu Guillame-Bert"
     50
---> 51 from tensorflow_decision_forests import keras
     52 from tensorflow_decision_forests.component import py_tree
     53 from tensorflow_decision_forests.component.builder import builder

~/venv/ml/lib/python3.8/site-packages/tensorflow_decision_forests/keras/__init__.py in <module>
     47 from typing import Callable, List
     48
---> 49 from tensorflow_decision_forests.keras import core
     50 from tensorflow_decision_forests.keras import wrappers
     51

~/venv/ml/lib/python3.8/site-packages/tensorflow_decision_forests/keras/core.py in <module>
     58 from tensorflow.python.training.tracking import base as base_tracking  # pylint: disable=g-direct-tensorflow-import
     59 from tensorflow_decision_forests.component.inspector import inspector as inspector_lib
---> 60 from tensorflow_decision_forests.tensorflow import core as tf_core
     61 from tensorflow_decision_forests.tensorflow.ops.inference import api as tf_op
     62 from tensorflow_decision_forests.tensorflow.ops.training import op as training_op

~/venv/ml/lib/python3.8/site-packages/tensorflow_decision_forests/tensorflow/core.py in <module>
     33 from tensorflow.python.distribute import parameter_server_strategy_v2
     34 from tensorflow_decision_forests.tensorflow.distribute import tf_distribution_pb2
---> 35 from tensorflow_decision_forests.tensorflow.ops.training import api as training_op
     36 from yggdrasil_decision_forests.dataset import data_spec_pb2
     37 from yggdrasil_decision_forests.learner import abstract_learner_pb2

~/venv/ml/lib/python3.8/site-packages/tensorflow_decision_forests/tensorflow/ops/training/api.py in <module>
     22 from tensorflow.python.framework import load_library
     23 from tensorflow.python.platform import resource_loader
---> 24 tf.load_op_library(resource_loader.get_path_to_datafile("training.so"))

~/venv/ml/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py in load_op_library(library_filename)
     56     RuntimeError: when unable to load the library or get the python wrappers.
     57   """
---> 58   lib_handle = py_tf.TF_LoadLibrary(library_filename)
     59   try:
     60     wrappers = _pywrap_python_op_gen.GetPythonWrappers(

NotFoundError: /home/oonisim/venv/ml/lib/python3.8/site-packages/tensorflow_decision_forests/tensorflow/ops/training/training.so: undefined symbol: _ZN10tensorflow11ResourceMgr8DoCreateERKSsNS_9TypeIndexES2_PNS_12ResourceBaseE

@oonisim
I tried with colab and I am able to import tfdf successfully, You can take a look at the gist here.
P.S : The versions that I used are TF 2.8.0 and TFDF 0.2.4

Closing this since it is most likely outdated now.

Closing this since it is most likely outdated now.

Same problem
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
explainable-ai-sdk 1.3.2 requires xai-image-widget, which is not installed.
tfx-bsl 1.5.0 requires absl-py<0.13,>=0.9, but you have absl-py 1.4.0 which is incompatible.
tfx-bsl 1.5.0 requires numpy<1.20,>=1.16, but you have numpy 1.20.3 which is incompatible.
tfx-bsl 1.5.0 requires pyarrow<6,>=1, but you have pyarrow 6.0.1 which is incompatible.
tensorflow-transform 1.5.0 requires absl-py<0.13,>=0.9, but you have absl-py 1.4.0 which is incompatible.
tensorflow-transform 1.5.0 requires numpy<1.20,>=1.16, but you have numpy 1.20.3 which is incompatible.
tensorflow-transform 1.5.0 requires pyarrow<6,>=1, but you have pyarrow 6.0.1 which is incompatible.
tensorflow-transform 1.5.0 requires tensorflow!=2.0.,!=2.1.,!=2.2.,!=2.3.,!=2.4.,!=2.5.,!=2.6.*,<2.8,>=1.15.2, but you have tensorflow 2.8.0 which is incompatible.
tensorflow-io 0.21.0 requires tensorflow<2.7.0,>=2.6.0, but you have tensorflow 2.8.0 which is incompatible.
tensorflow-io 0.21.0 requires tensorflow-io-gcs-filesystem==0.21.0, but you have tensorflow-io-gcs-filesystem 0.32.0 which is incompatible.
tensorflow-decision-forests 1.2.0 requires tensorflow~=2.11.0, but you have tensorflow 2.8.0 which is incompatible.

I believe that the issue you're seeing is the following: You are combining an outdated Tensorflow version (2.8.0) with an earlier TF-DF version (1.2.0). This is not possible since TF-DF and TF have to be in sync.
Possible solutions:

Force installation of TF-DF 0.2.4 pip install --force-reinstall -v "tensorflow-decision-forests==0.2.4"
Install the current Tensorflow version and TF-DF 1.3.0 pip install --force-reinstall tensorflow-decision-forests


Extract repeatable logic for generating Collector implementations for Seq-based types

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@91aaf03). Click here to learn what that means.
The diff coverage is 100%.

@@            Coverage Diff            @@
##             master    #2526   +/-   ##
=========================================
  Coverage          ?   94.95%
  Complexity        ?     5398
=========================================
  Files             ?       89
  Lines             ?    12767
  Branches          ?     1610
=========================================
  Hits              ?    12123
  Misses            ?      460
  Partials          ?      184

Impacted Files
Coverage Δ
Complexity Δ

src/main/java/io/vavr/collection/Collections.java
94.63% <100%> (ø)
136 <2> (?)

src/main/java/io/vavr/collection/Array.java
99.3% <100%> (ø)
245 <1> (?)

src/main/java/io/vavr/collection/Queue.java
96.45% <100%> (ø)
204 <1> (?)

src/main/java/io/vavr/collection/Vector.java
99.38% <100%> (ø)
215 <1> (?)

src/main/java/io/vavr/collection/Stream.java
98.65% <100%> (ø)
266 <1> (?)

src/main/java/io/vavr/collection/CharSeq.java
99.64% <100%> (ø)
351 <1> (?)

src/main/java/io/vavr/collection/List.java
99.43% <100%> (ø)
270 <1> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 91aaf03...58bee9d. Read the comment docs.


Fix/multiple-reloads

Thank you for your contribution to the KodaDot - One Stop Shop for Polkadot NFTs.
👇 __ Let's make a quick check before the contribution.
PR Type

[x] Bugfix

Needs QA check

@kodadot/qa-guild please review

Context

[x] Closes #9648

Links
monotifs
alteration
Notes

Usually it is better to use css to toggle visibilty, but not in this case. GenerativePreview is emitting messages, having 2 instances in the page is causing unexpcted behviours
yes, it is ugly to use setTimeout but in makes the first variation of the art to load consistently, i guess the code in the iframe needs a moment to load

Did your issue had any of the "$" label on it?

[x] My DOT address: Payout

Screenshot 📸

[ ] My fix has changed UI

Thanks!
pay 30 usd

😍 Perfect, I’ve sent the payout
💵 $30 @ 10.31 USD/DOT ~ 2.91 $DOT
🧗 1cAsKZYNRb8dkSCpn4eVkEn6ycNZTGoDo5dGDgB8J1UUWK8
🔗 0x95f8d068e744eee9109cb496b284800a58e37bef7a84803d751d731b031353fe
🪅 Let’s grab another issue and get rewarded!
🪄 github.com/kodadot/nft-gallery/issues