[BUG] Deleting the only pod in a single replica ReplicaSet causes an attachment failure of the replacement pods when the volume is RWX.

When you have a RWX volume attached to a pod in a single replica ReplicaSet, if you delete the pod, the replacement pod stays in CreatingContainer mode for some time and Terminates.
This cycle gets repeated a random number of times until one of the pods manages to start properly.
It can take from several seconds to 15 minutes to achieve a correctly connected pod.
If the ReplicaSet has multiple replicas this behavior does not appear.
To Reproduce
Apply a yaml like this:
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-test-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: longhorn
---
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: longhorn-test
  labels:
    app: longhorn-test
spec:
  # modify replicas according to your case
  replicas: 1
  selector:
    matchLabels:
      app: longhorn-test
  template:
    metadata:
      labels:
        app: longhorn-test
    spec:
      containers:
        - name: alpine
          image: docker.io/alpine:latest
          command: ["sleep", "infinity"]
          volumeMounts:
            - name: longhorn-test-volume
              mountPath: "/mnt"
      volumes:
        - name: longhorn-test-volume
          persistentVolumeClaim:
            claimName: longhorn-test-pvc

Verify the pod comes up.
$ kubectl get pods
NAME                  READY   STATUS    RESTARTS   AGE
longhorn-test-llncs   1/1     Running   0          18m

Delete the pod.
$ kubectl delete pod longhorn-test-llncs
pod "longhorn-test-llncs" deleted

Check the new pod and verify it fails to come up multiple times.
$ kubectl get pods
NAME                  READY   STATUS              RESTARTS   AGE
longhorn-test-6vl9s   0/1     Terminating         0          35s
longhorn-test-hxc24   0/1     ContainerCreating   0          3s
longhorn-test-llncs   0/1     Terminating         0          19m

$ kubectl get pods
NAME                  READY   STATUS              RESTARTS   AGE
longhorn-test-hxc24   0/1     ContainerCreating   0          20s

$ kubectl get pods
NAME                  READY   STATUS              RESTARTS   AGE
longhorn-test-hxc24   0/1     Terminating         0          31s
longhorn-test-x8wrp   0/1     ContainerCreating   0          1s

Eventually the pod comes up:
$ kubectl get pod
NAME                  READY   STATUS    RESTARTS   AGE
longhorn-test-x8wrp   1/1     Running   0          63s

Expected behavior
We expect the pod to come up and attach the volume on the first attempt.
Logs
This is an example of the events as seen from the volume side:
Events:
  Type    Reason    Age                    From                               Message
  ----    ------    ----                   ----                               -------
  Normal  Start     37m                    longhorn-kubernetes-pv-controller  Persistent Volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 started to use/reuse Longhorn volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03
  Normal  Remount   30m (x2 over 30m)      longhorn-volume-controller         Volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 requested remount at 2021-08-27T22:13:49Z
  Normal  Remount   30m                    longhorn-volume-controller         Volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 requested remount at 2021-08-27T22:13:50Z
  Normal  Detached  30m (x2 over 37m)      longhorn-volume-controller         volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 has been detached
  Normal  Remount   30m (x4 over 30m)      longhorn-volume-controller         Volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 requested remount at 2021-08-27T22:13:51Z
  Normal  Remount   30m                    longhorn-volume-controller         Volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 requested remount at 2021-08-27T22:13:52Z
  Normal  Remount   30m (x4 over 30m)      longhorn-volume-controller         Volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 requested remount at 2021-08-27T22:13:53Z
  Normal  Remount   30m (x2 over 30m)      longhorn-volume-controller         Volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 requested remount at 2021-08-27T22:13:54Z
  Normal  Attached  30m (x2 over 37m)      longhorn-volume-controller         volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 has been attached to worker-2.ocp.baremetal.casa.dev
  Normal  Remount   30m (x2 over 30m)      longhorn-volume-controller         Volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 requested remount at 2021-08-27T22:13:56Z
  Normal  Remount   30m                    longhorn-volume-controller         Volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 requested remount at 2021-08-27T22:14:03Z
  Normal  Remount   30m (x4 over 30m)      longhorn-volume-controller         Volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 requested remount at 2021-08-27T22:14:04Z
  Normal  Remount   22m (x127 over 30m)    longhorn-volume-controller         (combined from similar events): Volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 requested remount at 2021-08-27T22:21:24Z
  Normal  Remount   3m22s (x2 over 3m22s)  longhorn-volume-controller         Volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 requested remount at 2021-08-27T22:40:57Z
  Normal  Remount   3m21s                  longhorn-volume-controller         Volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 requested remount at 2021-08-27T22:40:58Z
  Normal  Remount   2m50s                  longhorn-volume-controller         Volume pvc-8cd3eff2-2a78-413d-aec7-84f27b274d03 requested remount at 2021-08-27T22:41:29Z

Environment:

Longhorn version: 1.1.2
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: Openshift 4.6,4.7,4.8

Number of management node in the cluster: 3
Number of worker node in the cluster: We've seen it with 3 dedicated workers, 2 dedicated workers and with 3 masters/workers.

Node config

OS type and version: Red Hat Enterprise Linux CoreOS 47.83.202102090044-0
CPU per node: 8
Memory per node: 32
Disk type(e.g. SSD/NVMe): HDD/SSD both dedicated and Shared with the OS.
Network bandwidth between the nodes: 1GB and 10GB,

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): KVM and Baremetal.
Number of Longhorn volumes in the cluster: From 1 to few tens.

Thank @rgomez-casa for reporting. I believe you hit a known bug https://github.com/longhorn/longhorn/issues/2874
It is in our radar to a future release. In the meantime, you can try the workaround https://github.com/longhorn/longhorn/issues/2874#issuecomment-897883677 (I know it is cumbersome)

Thanks a lot for the fast feedback @PhanLe1010,
While the problem does not seem identical to the bug you mention, the workaround works fine.
Do you know when is this targeted for a fix?

Can you give more detail as why you think they are not identical?
I am thinking they are the same because the example you mentioned in this issue is a ReplicaSet which essentially the same as the Deployment example in the issue https://github.com/longhorn/longhorn/issues/2874#issuecomment-897882978 . Underneath  of a Deployment is a ReplicaSet

Do you know when is this targeted for a fix?

cc @innobead

Can you give more detail as why you think they are not identical?
I am thinking they are the same because the example you mentioned in this issue is a ReplicaSet which essentially the same as the Deployment example in the issue #2874 (comment) . Underneath of a Deployment is a ReplicaSet

It is just an opinion, it could be exactly the same thing.
The bug you refer talks about an early release of the Sharemanager.
From my point of view (I could be wrong, don't trust me on this) in this case I think the Sharemanager should indeed be released, since the last pod using the volume is deleted.
When the new Pod comes up the volume should be attached and a new Sharemanager started. Looking at the logs this is what seems to happen, but the attachment fails anyway.
It is true that the Sharemanager seems to be gone while the pod is still terminating and keeping it might solve the issue.
The fact that the workaround works probably gives you the upper hand in the fact that it is the same bug.

BTW
Feel free to close the bug if you feel it is duplicated.

Thanks @rgomez-casa !
We can keep this issue open and make sure to test it together with https://github.com/longhorn/longhorn/issues/2874#issuecomment-897882978 once a fix available

tracking issue for v1.1.3 backports https://github.com/longhorn/longhorn/issues/3003

Verified with v1.2.1-rc1 and Longhorn-master `09/20/2021'
Validation - Pass
The pod gets re-created on first attempt after deleting it.
kubectl get pods -w
NAME                  READY   STATUS    RESTARTS   AGE
longhorn-test-kqhsn   1/1     Running   0          8m42s
longhorn-test-kqhsn   1/1     Terminating   0          9m
longhorn-test-mxfz4   0/1     Pending       0          0s
longhorn-test-mxfz4   0/1     Pending       0          0s
longhorn-test-mxfz4   0/1     ContainerCreating   0          0s
longhorn-test-mxfz4   0/1     ContainerCreating   0          1s
longhorn-test-mxfz4   1/1     Running             0          2s
longhorn-test-kqhsn   1/1     Terminating         0          9m30s
longhorn-test-kqhsn   0/1     Terminating         0          9m31s
longhorn-test-kqhsn   0/1     Terminating         0          9m40s
longhorn-test-kqhsn   0/1     Terminating         0          9m40s


Add partial extractors for HLists, Coproducts, tuples, records and unions.

Doesn't seem necessary


Add Normative Mineralogy feature

Composition is "the way in which a whole or mixture is made up."  This can be in terms of chemistry or minerals.
The early development of the package has focused purely on chemistry, as provided by the user as an input.
It is intended that mineral content can be provided in the same way, but also by inference from the chemistry.  Estimating mineral composition from chemical assay is known as Normative Mineralogy.
This package depends on a package called periodictable which has knowledge of elements, standard elemental codes, elemental mass and compounds.  The idea is to create a similar reference for minerals and leverage the periodtable information to create a routine that can solve for minerals given a) chemistry, and b) selected minerals.
To do this we'll need a reference table of minerals.  This will bring with it the advantage of standard mineral codes, similar to chemistry.

Thanks to BenFGA for the mineral reference resource: https://rruff.info/ima/