failed plugging phase1 at nic 'net1': Link not found

i cannot start vms with a 2nd nic
virt handler
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance ceph/ceph-3","pos":"vm.go:1474","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-04-25T14:26:55.169646Z"}

vmipool
              interfaces:
                - bridge: {}
                  name: lan
                - bridge: {}
                  name: default
          networks:
            - name: default
              pod: {}
            - multus:
                networkName: ceph/bridge-network
              name: lan

network attachment definition
spec:
  config: |
    {
        "cniVersion": "0.3.1",
        "name": "br0-l2",
        "plugins": [{
            "type": "bridge",
            "bridge": "br0",
            "ipam": {}
        }]
    }

root@nuc01:~/manifests/ceph# ip a show br0
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ae:e1:8c:24:b9:b1 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::ace1:8cff:fe24:b9b1/64 scope link
       valid_lft forever preferred_lft forever

@hijak , we need more information to understand what is the problem.
You need to fill all the details as specified in the template of the issue.
Specifically, we (also) need:

The version used.
The full manifests of: VMI, The VMI Pod, NAD.
The full log of the specific VMI Pod and the virt-handler pod which ran the VMI in question.

/sig network
/assign EdDev

im using the hyper converged operator on latest k3s its reporting
kubevirt.io/install-strategy-version=v0.59.0
the vmipool definition:
apiVersion: pool.kubevirt.io/v1alpha1
kind: VirtualMachinePool
metadata:
  labels:
    kubevirt-manager.io/managed: 'true'
    kubevirt.io/vmpool: ceph
  name: ceph
  namespace: ceph
spec:
  replicas: 3
  selector:
    matchLabels:
      kubevirt-manager.io/managed: 'true'
      kubevirt.io/vmpool: ceph
  virtualMachineTemplate:
    metadata:
      labels:
        kubevirt-manager.io/managed: 'true'
        kubevirt.io/vmpool: ceph
    spec:
      dataVolumeTemplates:
        - apiVersion: cdi.kubevirt.io/v1beta1
          kind: DataVolume
          metadata:
            annotations:
              cdi.kubevirt.io/storage.deleteAfterCompletion: 'false'
            name: ceph-ceph-disk1
            namespace: ceph
          spec:
            pvc:
              accessModes:
                - ReadWriteOnce
              resources:
                requests:
                  storage: 32Gi
              storageClassName: longhorn
            source:
              http:
                url: >-
                  http://<IP_ADDRESS>:8000/jammy-server-cloudimg-amd64.img
        - apiVersion: cdi.kubevirt.io/v1beta1
          kind: DataVolume
          metadata:
            annotations:
              cdi.kubevirt.io/storage.deleteAfterCompletion: 'false'
            name: ceph-ceph-disk2
            namespace: ceph
          spec:
            pvc:
              accessModes:
                - ReadWriteOnce
              resources:
                requests:
                  storage: 100Gi
              storageClassName: longhorn
            source:
              blank: {}
        - apiVersion: cdi.kubevirt.io/v1beta1
          kind: DataVolume
          metadata:
            annotations:
              cdi.kubevirt.io/storage.deleteAfterCompletion: 'false'
            name: ceph-ceph-disk3
            namespace: ceph
          spec:
            pvc:
              accessModes:
                - ReadWriteOnce
              resources:
                requests:
                  storage: 100Gi
              storageClassName: longhorn
            source:
              blank: {}
      running: true
      template:
        metadata:
          labels:
            kubevirt-manager.io/managed: 'true'
            kubevirt.io/vmpool: ceph
        spec:
          domain:
            cpu:
              cores: 4
              sockets: 1
              threads: 1
            devices:
              disks:
                - disk: {}
                  name: disk1
                - disk: {}
                  name: disk2
                - disk: {}
                  name: disk3
                - disk:
                    bus: virtio
                  name: disk4
              interfaces:
                - bridge: {}
                  name: default
                - bridge: {}
                  name: net1
            resources:
              requests:
                memory: 4Gi
          networks:
            - pod: {}
              name: default
            - multus:
#                default: true
                networkName: ceph/bridge-network
              name: net1
          #priorityClassName: vm-standard
          volumes:
            - dataVolume:
                name: ceph-ceph-disk1
              name: disk1
            - dataVolume:
                name: ceph-ceph-disk2
              name: disk2
            - dataVolume:
                name: ceph-ceph-disk3
              name: disk3
            - cloudInitNoCloud:
                userData: |
                  #cloud-config
                  manage_etc_hosts: false
                  runcmd:
                    - apt update && apt upgrade -y
                    - apt install ceph cephadm qemu-guest-agent -y
                    - groupmod -g 167 ceph && usermod -u 167 -g 167 ceph
                    - curl https://get.docker.com | sh
                  users:
                    - name: ubuntu
                      passwd: XXXX
                      shell: /bin/bash
                      lock-passwd: false
                      ssh_pwauth: True
                      chpasswd: { expire: False }
                      sudo: ALL=(ALL) NOPASSWD:ALL
                      groups: users, admin
                      ssh_authorized_keys:
                       - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6fxvW8VLHJrEj8v7WHLjNkootX+WGquz8ARqTWcnkLYueJZq4i9YsVf1wUIyIt2dqA1bfLPc3B+ulOs7MACICvayumCOzz5NVW5xEWG8tTbQGTN0RD9uXDEx1EmcCvsZ9gKwibLNGxcwiHzVPmOmKdbp3CT4615JJHdmLtdhHcEhiJhxNc8D+Etv929VOIH95Or9cpzn+aSKs4BX6f+R5TWY6cjuLgvR5f5i9hVxwmNp1zbzhMPijKpbUH56dyl2NPlPlcMU1FLwlBWZCx5hG+tBsY2wFyyp5JZATQsJQ+t7uGzmhZbqxa1VaAJivgfWRQglP8iTz3MhlKy4g4yYd
              name: disk4

virt-launcher pod logs
{"component":"virt-launcher","level":"info","msg":"Collected all requested hook sidecar sockets","pos":"manager.go:86","timestamp":"2023-04-27T06:54:42.657837Z"}
{"component":"virt-launcher","level":"info","msg":"Sorted all collected sidecar sockets per hook point based on their priority and name: map[]","pos":"manager.go:89","timestamp":"2023-04-27T06:54:42.657871Z"}
{"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon: qemu+unix:///session?socket=/var/run/libvirt/virtqemud-sock","pos":"libvirt.go:496","timestamp":"2023-04-27T06:54:42.658381Z"}
{"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon failed: virError(Code=38, Domain=7, Message='Failed to connect socket to '/var/run/libvirt/virtqemud-sock': No such file or directory')","pos":"libvirt.go:504","timestamp":"2023-04-27T06:54:42.658666Z"}
{"component":"virt-launcher","level":"info","msg":"libvirt version: 9.0.0, package: 3.el9<EMAIL_ADDRESS>2023-02-01-12:58:56, )","subcomponent":"libvirt","thread":"43","timestamp":"2023-04-27T06:54:42.669000Z"}
{"component":"virt-launcher","level":"info","msg":"hostname: ceph-3","subcomponent":"libvirt","thread":"43","timestamp":"2023-04-27T06:54:42.669000Z"}
{"component":"virt-launcher","level":"error","msg":"internal error: Unable to get session bus connection: Cannot spawn a message bus without a machine-id: Unable to load /var/lib/dbus/machine-id or /etc/machine-id: Failed to open file “/var/lib/dbus/machine-id”: No such file or directory","pos":"virGDBusGetSessionBus:126","subcomponent":"libvirt","thread":"43","timestamp":"2023-04-27T06:54:42.669000Z"}
{"component":"virt-launcher","level":"error","msg":"internal error: Unable to get system bus connection: Could not connect: No such file or directory","pos":"virGDBusGetSystemBus:99","subcomponent":"libvirt","thread":"43","timestamp":"2023-04-27T06:54:42.669000Z"}
{"component":"virt-launcher","level":"info","msg":"Connected to libvirt daemon","pos":"libvirt.go:512","timestamp":"2023-04-27T06:54:43.164266Z"}
{"component":"virt-launcher","level":"info","msg":"Registered libvirt event notify callback","pos":"client.go:537","timestamp":"2023-04-27T06:54:43.172772Z"}
{"component":"virt-launcher","level":"info","msg":"Marked as ready","pos":"virt-launcher.go:75","timestamp":"2023-04-27T06:54:43.173474Z"}
panic: timed out waiting for domain to be defined

goroutine 1 [running]:
main.waitForDomainUUID(0xc000127080?, 0xc000126f60, 0xc000329200, {0x1ce9698, 0xc000222410})
	cmd/virt-launcher/virt-launcher.go:246 +0x434
main.main()
	cmd/virt-launcher/virt-launcher.go:466 +0x1155
{"component":"virt-launcher-monitor","level":"info","msg":"Reaped pid 12 with status 512","pos":"virt-launcher-monitor.go:124","timestamp":"2023-04-27T06:59:16.178272Z"}
{"component":"virt-launcher-monitor","level":"error","msg":"dirty virt-launcher shutdown: exit-code 2","pos":"virt-launcher-monitor.go:142","timestamp":"2023-04-27T06:59:16.178372Z"}

virt handler snippet
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"ceph-2","namespace":"ceph","pos":"mount.go:405","timestamp":"2023-04-27T07:03:00.631789Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"ceph-2","namespace":"ceph","pos":"mount.go:408","timestamp":"2023-04-27T07:03:00.631879Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:272","reason":"Link not found","timestamp":"2023-04-27T07:03:00.633108Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"ceph-2","namespace":"ceph","pos":"vm.go:1830","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-04-27T07:03:00.633274Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance ceph/ceph-2","pos":"vm.go:1474","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-04-27T07:03:00.648863Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"ceph-2","namespace":"ceph","pos":"vm.go:1684","timestamp":"2023-04-27T07:03:01.929295Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"ceph-2","namespace":"ceph","pos":"mount.go:405","timestamp":"2023-04-27T07:03:01.930398Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"ceph-2","namespace":"ceph","pos":"mount.go:408","timestamp":"2023-04-27T07:03:01.930513Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:272","reason":"Link not found","timestamp":"2023-04-27T07:03:01.932174Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"ceph-2","namespace":"ceph","pos":"vm.go:1830","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-04-27T07:03:01.932372Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance ceph/ceph-2","pos":"vm.go:1474","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-04-27T07:03:01.951650Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"ceph-2","namespace":"ceph","pos":"vm.go:1684","timestamp":"2023-04-27T07:03:04.512369Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"ceph-2","namespace":"ceph","pos":"mount.go:405","timestamp":"2023-04-27T07:03:04.513232Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"ceph-2","namespace":"ceph","pos":"mount.go:408","timestamp":"2023-04-27T07:03:04.513328Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:272","reason":"Link not found","timestamp":"2023-04-27T07:03:04.514719Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"ceph-2","namespace":"ceph","pos":"vm.go:1830","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-04-27T07:03:04.514992Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance ceph/ceph-2","pos":"vm.go:1474","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-04-27T07:03:04.530174Z"}
{"component":"virt-handler","level":"error","msg":"failed to remove mdev types: failed to read the content of /sys/bus/mdev/devices directory","pos":"mediated_devices_types.go:263","reason":"open /sys/bus/mdev/devices: no such file or directory","timestamp":"2023-04-27T07:03:07.481243Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:321","timestamp":"2023-04-27T07:03:07.481300Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-04-27T07:03:07.481323Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:323","timestamp":"2023-04-27T07:03:07.481344Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"ceph-2","namespace":"ceph","pos":"vm.go:1684","timestamp":"2023-04-27T07:03:09.650761Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"ceph-2","namespace":"ceph","pos":"mount.go:405","timestamp":"2023-04-27T07:03:09.651052Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"ceph-2","namespace":"ceph","pos":"mount.go:408","timestamp":"2023-04-27T07:03:09.651076Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:272","reason":"Link not found","timestamp":"2023-04-27T07:03:09.651448Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"ceph-2","namespace":"ceph","pos":"vm.go:1830","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-04-27T07:03:09.651496Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance ceph/ceph-2","pos":"vm.go:1474","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-04-27T07:03:09.665024Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"ceph-2","namespace":"ceph","pos":"vm.go:1684","timestamp":"2023-04-27T07:03:19.905931Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"ceph-2","namespace":"ceph","pos":"mount.go:405","timestamp":"2023-04-27T07:03:19.906589Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"ceph-2","namespace":"ceph","pos":"mount.go:408","timestamp":"2023-04-27T07:03:19.906714Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:272","reason":"Link not found","timestamp":"2023-04-27T07:03:19.908027Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"ceph-2","namespace":"ceph","pos":"vm.go:1830","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-04-27T07:03:19.908180Z","uid":"5964f5c2-900b-482d-b63a-15256d13713d"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance ceph/ceph-2","pos":"vm.go:1474","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-04-27T07:03:19.923693Z"}

i have another vmi that just crashes but thats not the issue im interested in

vmi pod
apiVersion: v1
kind: Pod
metadata:
  name: virt-launcher-ceph-3-vld5r
  generateName: virt-launcher-ceph-3-
  namespace: ceph
  uid: bc1f3fbe-b908-4697-a3cb-e0dc83f16d51
  resourceVersion: '1978543'
  creationTimestamp: '2023-04-27T07:04:58Z'
  labels:
    kubevirt-manager.io/managed: 'true'
    kubevirt.io: virt-launcher
    kubevirt.io/created-by: 85170b71-dcc8-4ea9-9531-927cc0a19ad1
    kubevirt.io/vm-pool-revision-name: ceph-1
    kubevirt.io/vmpool: ceph
    vm.kubevirt.io/name: ceph-3
  annotations:
    k8s.v1.cni.cncf.io/networks: >-
      [{"interface":"net1","mac":"02:d8:01:00:00:08","name":"bridge-network","namespace":"ceph"}]
    kubectl.kubernetes.io/default-container: compute
    kubevirt.io/domain: ceph-3
    kubevirt.io/migrationTransportUnix: 'true'
    post.hook.backup.velero.io/command: >-
      ["/usr/bin/virt-freezer", "--unfreeze", "--name", "ceph-3", "--namespace",
      "ceph"]
    post.hook.backup.velero.io/container: compute
    pre.hook.backup.velero.io/command: >-
      ["/usr/bin/virt-freezer", "--freeze", "--name", "ceph-3", "--namespace",
      "ceph"]
    pre.hook.backup.velero.io/container: compute
  ownerReferences:
    - apiVersion: kubevirt.io/v1
      kind: VirtualMachineInstance
      name: ceph-3
      uid: 85170b71-dcc8-4ea9-9531-927cc0a19ad1
      controller: true
      blockOwnerDeletion: true
  managedFields:
    - manager: Go-http-client
      operation: Update
      apiVersion: v1
      time: '2023-04-27T07:04:58Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:k8s.v1.cni.cncf.io/networks: {}
            f:kubectl.kubernetes.io/default-container: {}
            f:kubevirt.io/domain: {}
            f:kubevirt.io/migrationTransportUnix: {}
            f:post.hook.backup.velero.io/command: {}
            f:post.hook.backup.velero.io/container: {}
            f:pre.hook.backup.velero.io/command: {}
            f:pre.hook.backup.velero.io/container: {}
          f:generateName: {}
          f:labels:
            .: {}
            f:kubevirt-manager.io/managed: {}
            f:kubevirt.io: {}
            f:kubevirt.io/created-by: {}
            f:kubevirt.io/vm-pool-revision-name: {}
            f:kubevirt.io/vmpool: {}
            f:vm.kubevirt.io/name: {}
          f:ownerReferences:
            .: {}
            k:{"uid":"85170b71-dcc8-4ea9-9531-927cc0a19ad1"}: {}
        f:spec:
          f:affinity:
            .: {}
            f:nodeAffinity:
              .: {}
              f:requiredDuringSchedulingIgnoredDuringExecution: {}
          f:automountServiceAccountToken: {}
          f:containers:
            k:{"name":"compute"}:
              .: {}
              f:command: {}
              f:env:
                .: {}
                k:{"name":"KUBEVIRT_RESOURCE_NAME_net1"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"POD_NAME"}:
                  .: {}
                  f:name: {}
                  f:valueFrom:
                    .: {}
                    f:fieldRef: {}
                k:{"name":"XDG_CACHE_HOME"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"XDG_CONFIG_HOME"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"XDG_RUNTIME_DIR"}:
                  .: {}
                  f:name: {}
                  f:value: {}
              f:image: {}
              f:imagePullPolicy: {}
              f:name: {}
              f:resources:
                .: {}
                f:limits:
                  .: {}
                  f:bridge.network.kubevirt.io/br0: {}
                  f:devices.kubevirt.io/kvm: {}
                  f:devices.kubevirt.io/tun: {}
                  f:devices.kubevirt.io/vhost-net: {}
                f:requests:
                  .: {}
                  f:bridge.network.kubevirt.io/br0: {}
                  f:cpu: {}
                  f:devices.kubevirt.io/kvm: {}
                  f:devices.kubevirt.io/tun: {}
                  f:devices.kubevirt.io/vhost-net: {}
                  f:ephemeral-storage: {}
                  f:memory: {}
              f:securityContext:
                .: {}
                f:allowPrivilegeEscalation: {}
                f:capabilities:
                  .: {}
                  f:add: {}
                  f:drop: {}
                f:privileged: {}
                f:runAsGroup: {}
                f:runAsNonRoot: {}
                f:runAsUser: {}
              f:terminationMessagePath: {}
              f:terminationMessagePolicy: {}
              f:volumeMounts:
                .: {}
                k:{"mountPath":"/var/run/kubevirt"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/var/run/kubevirt-ephemeral-disks"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/var/run/kubevirt-private"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/var/run/kubevirt-private/vmi-disks/disk1"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/var/run/kubevirt-private/vmi-disks/disk2"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/var/run/kubevirt-private/vmi-disks/disk3"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/var/run/kubevirt/container-disks"}:
                  .: {}
                  f:mountPath: {}
                  f:mountPropagation: {}
                  f:name: {}
                k:{"mountPath":"/var/run/kubevirt/hotplug-disks"}:
                  .: {}
                  f:mountPath: {}
                  f:mountPropagation: {}
                  f:name: {}
                k:{"mountPath":"/var/run/kubevirt/sockets"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/var/run/libvirt"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
          f:dnsPolicy: {}
          f:enableServiceLinks: {}
          f:hostname: {}
          f:nodeSelector: {}
          f:readinessGates: {}
          f:restartPolicy: {}
          f:schedulerName: {}
          f:securityContext:
            .: {}
            f:runAsGroup: {}
            f:runAsNonRoot: {}
            f:runAsUser: {}
            f:seccompProfile:
              .: {}
              f:localhostProfile: {}
              f:type: {}
          f:terminationGracePeriodSeconds: {}
          f:volumes:
            .: {}
            k:{"name":"container-disks"}:
              .: {}
              f:emptyDir: {}
              f:name: {}
            k:{"name":"disk1"}:
              .: {}
              f:name: {}
              f:persistentVolumeClaim:
                .: {}
                f:claimName: {}
            k:{"name":"disk2"}:
              .: {}
              f:name: {}
              f:persistentVolumeClaim:
                .: {}
                f:claimName: {}
            k:{"name":"disk3"}:
              .: {}
              f:name: {}
              f:persistentVolumeClaim:
                .: {}
                f:claimName: {}
            k:{"name":"ephemeral-disks"}:
              .: {}
              f:emptyDir: {}
              f:name: {}
            k:{"name":"hotplug-disks"}:
              .: {}
              f:emptyDir: {}
              f:name: {}
            k:{"name":"libvirt-runtime"}:
              .: {}
              f:emptyDir: {}
              f:name: {}
            k:{"name":"private"}:
              .: {}
              f:emptyDir: {}
              f:name: {}
            k:{"name":"public"}:
              .: {}
              f:emptyDir: {}
              f:name: {}
            k:{"name":"sockets"}:
              .: {}
              f:emptyDir: {}
              f:name: {}
            k:{"name":"virt-bin-share-dir"}:
              .: {}
              f:emptyDir: {}
              f:name: {}
    - manager: Go-http-client
      operation: Update
      apiVersion: v1
      time: '2023-04-27T07:04:58Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:conditions:
            k:{"type":"kubevirt.io/virtual-machine-unpaused"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
      subresource: status
    - manager: k3s
      operation: Update
      apiVersion: v1
      time: '2023-04-27T07:05:10Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:conditions:
            k:{"type":"ContainersReady"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
            k:{"type":"Initialized"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
            k:{"type":"Ready"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
          f:containerStatuses: {}
          f:hostIP: {}
          f:phase: {}
          f:podIP: {}
          f:podIPs:
            .: {}
            k:{"ip":"<IP_ADDRESS>"}:
              .: {}
              f:ip: {}
          f:startTime: {}
      subresource: status
  selfLink: /api/v1/namespaces/ceph/pods/virt-launcher-ceph-3-vld5r
status:
  phase: Running
  conditions:
    - type: kubevirt.io/virtual-machine-unpaused
      status: 'True'
      lastProbeTime: '2023-04-27T07:04:58Z'
      lastTransitionTime: '2023-04-27T07:04:58Z'
      reason: NotPaused
      message: the virtual machine is not paused
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-04-27T07:04:58Z'
    - type: Ready
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-04-27T07:05:10Z'
    - type: ContainersReady
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-04-27T07:05:10Z'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-04-27T07:04:58Z'
  hostIP: <IP_ADDRESS>
  podIP: <IP_ADDRESS>
  podIPs:
    - ip: <IP_ADDRESS>
  startTime: '2023-04-27T07:04:58Z'
  containerStatuses:
    - name: compute
      state:
        running:
          startedAt: '2023-04-27T07:05:10Z'
      lastState: {}
      ready: true
      restartCount: 0
      image: sha256:7b175446633ac195b8e0826cdc8e7ac231af57040278061917c6873d10f36607
      imageID: >-
        quay.io/kubevirt/virt-launcher@sha256:20a72f7f5349f1f4035863b491b9a857c7dbbe5c7d8368bb6d7384f4b279c57c
      containerID: >-
        containerd://9196a213101e9b7feeef57c86f2c1f54f1ad68244fa4d9b15fb020bdf83d1c9c
      started: true
  qosClass: Burstable
spec:
  volumes:
    - name: private
      emptyDir: {}
    - name: public
      emptyDir: {}
    - name: sockets
      emptyDir: {}
    - name: virt-bin-share-dir
      emptyDir: {}
    - name: libvirt-runtime
      emptyDir: {}
    - name: ephemeral-disks
      emptyDir: {}
    - name: container-disks
      emptyDir: {}
    - name: disk1
      persistentVolumeClaim:
        claimName: ceph-ceph-disk1-3
    - name: disk2
      persistentVolumeClaim:
        claimName: ceph-ceph-disk2-3
    - name: disk3
      persistentVolumeClaim:
        claimName: ceph-ceph-disk3-3
    - name: hotplug-disks
      emptyDir: {}
  containers:
    - name: compute
      image: >-
        quay.io/kubevirt/virt-launcher@sha256:20a72f7f5349f1f4035863b491b9a857c7dbbe5c7d8368bb6d7384f4b279c57c
      command:
        - /usr/bin/virt-launcher-monitor
        - '--qemu-timeout'
        - 304s
        - '--name'
        - ceph-3
        - '--uid'
        - 85170b71-dcc8-4ea9-9531-927cc0a19ad1
        - '--namespace'
        - ceph
        - '--kubevirt-share-dir'
        - /var/run/kubevirt
        - '--ephemeral-disk-dir'
        - /var/run/kubevirt-ephemeral-disks
        - '--container-disk-dir'
        - /var/run/kubevirt/container-disks
        - '--grace-period-seconds'
        - '45'
        - '--hook-sidecars'
        - '0'
        - '--ovmf-path'
        - /usr/share/OVMF
        - '--run-as-nonroot'
      env:
        - name: XDG_CACHE_HOME
          value: /var/run/kubevirt-private
        - name: XDG_CONFIG_HOME
          value: /var/run/kubevirt-private
        - name: XDG_RUNTIME_DIR
          value: /var/run
        - name: KUBEVIRT_RESOURCE_NAME_net1
          value: bridge.network.kubevirt.io/br0
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
      resources:
        limits:
          bridge.network.kubevirt.io/br0: '1'
          devices.kubevirt.io/kvm: '1'
          devices.kubevirt.io/tun: '1'
          devices.kubevirt.io/vhost-net: '1'
        requests:
          bridge.network.kubevirt.io/br0: '1'
          cpu: 400m
          devices.kubevirt.io/kvm: '1'
          devices.kubevirt.io/tun: '1'
          devices.kubevirt.io/vhost-net: '1'
          ephemeral-storage: 50M
          memory: '4582277121'
      volumeMounts:
        - name: private
          mountPath: /var/run/kubevirt-private
        - name: public
          mountPath: /var/run/kubevirt
        - name: ephemeral-disks
          mountPath: /var/run/kubevirt-ephemeral-disks
        - name: container-disks
          mountPath: /var/run/kubevirt/container-disks
          mountPropagation: HostToContainer
        - name: libvirt-runtime
          mountPath: /var/run/libvirt
        - name: sockets
          mountPath: /var/run/kubevirt/sockets
        - name: disk1
          mountPath: /var/run/kubevirt-private/vmi-disks/disk1
        - name: disk2
          mountPath: /var/run/kubevirt-private/vmi-disks/disk2
        - name: disk3
          mountPath: /var/run/kubevirt-private/vmi-disks/disk3
        - name: hotplug-disks
          mountPath: /var/run/kubevirt/hotplug-disks
          mountPropagation: HostToContainer
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        capabilities:
          add:
            - NET_BIND_SERVICE
          drop:
            - ALL
        privileged: false
        runAsUser: 107
        runAsGroup: 107
        runAsNonRoot: true
        allowPrivilegeEscalation: false
  restartPolicy: Never
  terminationGracePeriodSeconds: 60
  dnsPolicy: ClusterFirst
  nodeSelector:
    kubevirt.io/schedulable: 'true'
  serviceAccountName: default
  serviceAccount: default
  automountServiceAccountToken: false
  nodeName: nuc01
  securityContext:
    runAsUser: 107
    runAsGroup: 107
    runAsNonRoot: true
    seccompProfile:
      type: Localhost
      localhostProfile: kubevirt/kubevirt.json
  hostname: ceph-3
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-labeller.kubevirt.io/obsolete-host-model
                operator: DoesNotExist
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
  priority: 0
  readinessGates:
    - conditionType: kubevirt.io/virtual-machine-unpaused
  enableServiceLinks: false
  preemptionPolicy: PreemptLowerPriority

Based on the information so far, it seems the virt-launcher panics, terminating the VMI pod. Then the virt-handler does not see the net1 interface.
The interface will be missing if the CNI removed it already, or the container itself is in the middle of being removed.
The fact the virt-launcher pod is running, means that the CNI succeeded adding the net1 interface (otherwise it would just fail the pod creation).
Why the definition of the domain fails (through libvirt) I do not understand.
Can you please provide the following details:

The VMI manifest (VirtualMachineInstance).
Given the same specification but without the net1 network/interface, does everything works? You have a working VM?
Do you see by any chance the error of the missing net1 link prior to the domain failure (panic)? The provided log snippet only show it after, if we examine the timestamp.

@xpivarc , maybe you can help with this odd panic seen in the virt-launcher.

@hijak , in addition to the last info request, please also add a link to the virt-handler full logs.
@xpivarc has mentioned the panic seen in the virt-launcher is due to the virt-handler not sending a command to the virt-launcher. This means we need to investigate the reason in the virt-handler logs prior to the time it panics in the virt-launcher.

if i set the multus to default true and leave it as the only interface i am able to run the vm
if i just have pod networking it also works. its only when im trying to run from a vmipool definition with 2 network interfaces one being pod and the other multus

@EdDev I'm running into this issue too. I will try to add logs if I can catch it again.
When I exec -it into the virt-launcher pod, it had only 1 nic, even though I am specifying multiple, I also saw no taps

kubectl logs pod/virt-launcher-storshelladmindal0989-k8wrd -n file-block
{"component":"virt-launcher","level":"info","msg":"Collected all requested hook sidecar sockets","pos":"manager.go:86","timestamp":"2023-05-21T22:49:21.574086Z"}
{"component":"virt-launcher","level":"info","msg":"Sorted all collected sidecar sockets per hook point based on their priority and name: map[]","pos":"manager.go:89","timestamp":"2023-05-21T22:49:21.574163Z"}
{"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon: qemu:///system","pos":"libvirt.go:497","timestamp":"2023-05-21T22:49:21.574734Z"}
{"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon failed: virError(Code=38, Domain=7, Message='Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory')","pos":"libvirt.go:505","timestamp":"2023-05-21T22:49:21.575164Z"}
{"component":"virt-launcher","level":"info","msg":"libvirt version: 8.0.0, package: 2.module_el8.6.0+1087+b42c8331 (CentOS Buildsys<EMAIL_ADDRESS>2022-02-08-22:20:52, )","subcomponent":"libvirt","thread":"44","timestamp":"2023-05-21T22:49:21.600000Z"}
{"component":"virt-launcher","level":"info","msg":"hostname: storshelladmindal0989","subcomponent":"libvirt","thread":"44","timestamp":"2023-05-21T22:49:21.600000Z"}
{"component":"virt-launcher","level":"error","msg":"internal error: Child process (dmidecode -q -t 0,1,2,3,4,11,17) unexpected exit status 1: /dev/mem: No such file or directory","pos":"virCommandWait:2752","subcomponent":"libvirt","thread":"44","timestamp":"2023-05-21T22:49:21.600000Z"}
{"component":"virt-launcher","level":"info","msg":"Connected to libvirt daemon","pos":"libvirt.go:513","timestamp":"2023-05-21T22:49:22.078327Z"}
{"component":"virt-launcher","level":"info","msg":"Registered libvirt event notify callback","pos":"client.go:510","timestamp":"2023-05-21T22:49:22.083125Z"}
{"component":"virt-launcher","level":"info","msg":"Marked as ready","pos":"virt-launcher.go:74","timestamp":"2023-05-21T22:49:22.083462Z"}

kubectl exec -it pod/virt-launcher-storshelladmindal0989-k8wrd -n file-block -- bash
bash-4.4# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet <IP_ADDRESS>/8 scope host lo
       valid_lft forever preferred_lft forever
292: eth0@if293: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether fa:89:34:e2:10:a7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet <IP_ADDRESS>/32 scope global eth0
       valid_lft forever preferred_lft forever

kubectl describe virtualmachineinstance.kubevirt.io/storshelladmindal0989 -n file-block
Name:         storshelladmindal0989
Namespace:    file-block
Labels:       kubevirt.io/nodeName=worker-node1
              kubevirt.io/vm=storshelladmindal0989
Annotations:  kubevirt.io/latest-observed-api-version: v1
              kubevirt.io/storage-observed-api-version: v1alpha3
API Version:  kubevirt.io/v1
Kind:         VirtualMachineInstance
Metadata:
  Creation Timestamp:  2023-05-21T22:49:12Z
  Finalizers:
    kubevirt.io/virtualMachineControllerFinalize
    foregroundDeleteVirtualMachine
  Generation:  6
  Managed Fields:
    API Version:  kubevirt.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubevirt.io/latest-observed-api-version:
          f:kubevirt.io/storage-observed-api-version:
        f:finalizers:
          .:
          v:"kubevirt.io/virtualMachineControllerFinalize":
        f:labels:
          .:
          f:kubevirt.io/nodeName:
          f:kubevirt.io/vm:
        f:ownerReferences:
          .:
          k:{"uid":"02e230a8-c0eb-492d-8888-1d94fcc9a7f5"}:
      f:spec:
        .:
        f:domain:
          .:
          f:cpu:
            .:
            f:cores:
          f:devices:
            .:
            f:blockMultiQueue:
            f:disks:
            f:interfaces:
            f:networkInterfaceMultiqueue:
          f:firmware:
            .:
            f:uuid:
          f:ioThreadsPolicy:
          f:machine:
            .:
            f:type:
          f:resources:
            .:
            f:limits:
              .:
              f:cpu:
            f:requests:
              .:
              f:memory:
        f:networks:
        f:nodeSelector:
          .:
          f:networkmanager:
        f:terminationGracePeriodSeconds:
        f:volumes:
      f:status:
        .:
        f:activePods:
          .:
          f:9cbdd266-396f-471b-bd7d-30608149675e:
        f:conditions:
        f:guestOSInfo:
        f:migrationMethod:
        f:migrationTransport:
        f:nodeName:
        f:phase:
        f:phaseTransitionTimestamps:
        f:qosClass:
        f:runtimeUser:
        f:virtualMachineRevisionName:
        f:volumeStatus:
    Manager:    Go-http-client
    Operation:  Update
    Time:       2023-05-21T22:49:22Z
  Owner References:
    API Version:           kubevirt.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  VirtualMachine
    Name:                  storshelladmindal0989
    UID:                   02e230a8-c0eb-492d-8888-1d94fcc9a7f5
  Resource Version:        515588
  UID:                     b8ff6d55-ef97-4492-8d3e-0c0d9ebba084
Spec:
  Domain:
    Cpu:
      Cores:  3
      Model:  host-model
    Devices:
      Block Multi Queue:  true
      Disks:
        Disk:
          Bus:  virtio
        Name:   os-pvc
        Disk:
          Bus:  virtio
        Name:   drive1
        Disk:
          Bus:  virtio
        Name:   drive2
        Disk:
          Bus:  virtio
        Name:   cloudinitdisk
      Interfaces:
        Bridge:
        Mac Address:  02:00:b1:98:ae:9a
        Name:         usb-int
        Bridge:
        Mac Address:                 02:01:b1:98:ae:9a
        Name:                        vpc-mgmt
      Network Interface Multiqueue:  true
    Features:
      Acpi:
        Enabled:  true
    Firmware:
      Uuid:             85bcd2f8-2f7b-5963-b4b7-3fb2054d71be
    Io Threads Policy:  auto
    Machine:
      Type:  q35
    Resources:
      Limits:
        Cpu:  3
      Requests:
        Cpu:     3
        Memory:  3G
  Networks:
    Multus:
      Network Name:  usb-int-br50-nad
    Name:            usb-int
    Multus:
      Network Name:  vpc-mgmt-br4-nad
    Name:            vpc-mgmt
  Node Selector:
    Networkmanager:                  true
  Termination Grace Period Seconds:  0
  Volumes:
    Name:  os-pvc
    Persistent Volume Claim:
      Claim Name:  file-block-storshelladmindal0989-os-ubuntu22
    Name:          drive1
    Persistent Volume Claim:
      Claim Name:  file-block-storshelladmindal0989-drive1
    Name:          drive2
    Persistent Volume Claim:
      Claim Name:  file-block-storshelladmindal0989-drive2
    Cloud Init No Cloud:
      Network Data:  version: 2
ethernets:
  usb-int:
    match:
      macaddress: 02:00:b1:98:ae:9a
    set-name: usb-int
    dhcp4: false
    addresses:
      - <IP_ADDRESS>/28
    routes:
      - to: default
        via: <IP_ADDRESS>
      - to: <IP_ADDRESS>/25
        via: <IP_ADDRESS>
  vpc-mgmt:
    match:
      macaddress: 02:01:b1:98:ae:9a
    set-name: vpc-mgmt
    dhcp4: false
    addresses:
      - <IP_ADDRESS>/16
      User Data:  #cloud-config
chpasswd:
  list: |
    <redacted>
  expire: False
users:
    - default
    - name: tmp_filenblock
      ssh-authorized-keys:
        - ssh-ed25519 A<redacted>R
      sudo: ['ALL=(ALL) NOPASSWD:ALL']
      groups: sudo
      shell: /bin/bash
write_files:
  - content: |
      PubkeyAcceptedKeyTypes=+ssh-ed25519
    path: /etc/ssh/sshd_config.d/00-cloud-init
runcmd:
  - service sshd restart

    Name:  cloudinitdisk
Status:
  Active Pods:
    9cbdd266-396f-471b-bd7d-30608149675e:  worker-node1
  Conditions:
    Last Probe Time:       2023-05-21T22:49:12Z
    Last Transition Time:  2023-05-21T22:49:12Z
    Message:               Guest VM is not reported as running
    Reason:                GuestNotRunning
    Status:                False
    Type:                  Ready
    Last Probe Time:       <nil>
    Last Transition Time:  <nil>
    Message:               cannot migrate VMI: PVC file-block-storshelladmindal0989-os-ubuntu22 is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode)
    Reason:                DisksNotLiveMigratable
    Status:                False
    Type:                  LiveMigratable
    Last Probe Time:       <nil>
    Last Transition Time:  2023-05-21T22:49:22Z
    Message:               failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found
    Reason:                Synchronizing with the Domain failed.
    Status:                False
    Type:                  Synchronized
  Guest OS Info:
  Migration Method:     BlockMigration
  Migration Transport:  Unix
  Node Name:            worker-node1
  Phase:                Scheduled
  Phase Transition Timestamps:
    Phase:                        Pending
    Phase Transition Timestamp:   2023-05-21T22:49:12Z
    Phase:                        Scheduling
    Phase Transition Timestamp:   2023-05-21T22:49:12Z
    Phase:                        Scheduled
    Phase Transition Timestamp:   2023-05-21T22:49:22Z
  Qos Class:                      Burstable
  Runtime User:                   0
  Virtual Machine Revision Name:  revision-start-vm-02e230a8-c0eb-492d-8888-1d94fcc9a7f5-1
  Volume Status:
    Name:    cloudinitdisk
    Target:
    Name:    drive1
    Persistent Volume Claim Info:
      Access Modes:
        ReadWriteOnce
      Capacity:
        Storage:            19Gi
      Filesystem Overhead:  0
      Requests:
        Storage:    20G
      Volume Mode:  Block
    Target:
    Name:           drive2
    Persistent Volume Claim Info:
      Access Modes:
        ReadWriteOnce
      Capacity:
        Storage:            19Gi
      Filesystem Overhead:  0
      Requests:
        Storage:    20G
      Volume Mode:  Block
    Target:
    Name:           os-pvc
    Persistent Volume Claim Info:
      Access Modes:
        ReadWriteOnce
      Capacity:
        Storage:            10Gi
      Filesystem Overhead:  0
      Requests:
        Storage:    10Gi
      Volume Mode:  Block
    Target:
Events:
  Type     Reason            Age                   From                       Message
  ----     ------            ----                  ----                       -------
  Normal   SuccessfulCreate  3m7s                  virtualmachine-controller  Created virtual machine pod virt-launcher-storshelladmindal0989-k8wrd
  Warning  SyncFailed        13s (x16 over 2m57s)  virt-handler               failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found

kubectl describe vm/storshelladmindal0989 -n file-block
Name:         storshelladmindal0989
Namespace:    file-block
Labels:       kubevirt.io/vm=storshelladmindal0989
Annotations:  kubevirt.io/latest-observed-api-version: v1
              kubevirt.io/storage-observed-api-version: v1alpha3
API Version:  kubevirt.io/v1
Kind:         VirtualMachine
Metadata:
  Creation Timestamp:  2023-05-21T22:49:12Z
  Generation:          1
  Managed Fields:
    API Version:  kubevirt.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubevirt.io/latest-observed-api-version:
          f:kubevirt.io/storage-observed-api-version:
    Manager:      Go-http-client
    Operation:    Update
    Time:         2023-05-21T22:49:12Z
    API Version:  kubevirt.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:created:
        f:printableStatus:
        f:volumeSnapshotStatuses:
    Manager:      Go-http-client
    Operation:    Update
    Subresource:  status
    Time:         2023-05-21T22:49:12Z
    API Version:  kubevirt.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .:
          f:kubevirt.io/vm:
      f:spec:
        .:
        f:running:
        f:template:
          .:
          f:metadata:
            .:
            f:labels:
              .:
              f:kubevirt.io/vm:
          f:spec:
            .:
            f:domain:
              .:
              f:cpu:
                .:
                f:cores:
              f:devices:
                .:
                f:blockMultiQueue:
                f:disks:
                f:interfaces:
                f:networkInterfaceMultiqueue:
              f:ioThreadsPolicy:
              f:machine:
                .:
                f:type:
              f:resources:
                .:
                f:limits:
                  .:
                  f:cpu:
                f:requests:
                  .:
                  f:memory:
            f:networks:
            f:nodeSelector:
              .:
              f:networkmanager:
            f:terminationGracePeriodSeconds:
            f:volumes:
    Manager:         OpenAPI-Generator
    Operation:       Update
    Time:            2023-05-21T22:49:12Z
  Resource Version:  515590
  UID:               02e230a8-c0eb-492d-8888-1d94fcc9a7f5
Spec:
  Running:  true
  Template:
    Metadata:
      Creation Timestamp:  <nil>
      Labels:
        kubevirt.io/vm:  storshelladmindal0989
    Spec:
      Domain:
        Cpu:
          Cores:  3
        Devices:
          Block Multi Queue:  true
          Disks:
            Disk:
              Bus:  virtio
            Name:   os-pvc
            Disk:
              Bus:  virtio
            Name:   drive1
            Disk:
              Bus:  virtio
            Name:   drive2
            Disk:
              Bus:  virtio
            Name:   cloudinitdisk
          Interfaces:
            Bridge:
            Mac Address:  02:00:b1:98:ae:9a
            Name:         usb-int
            Bridge:
            Mac Address:                 02:01:b1:98:ae:9a
            Name:                        vpc-mgmt
          Network Interface Multiqueue:  true
        Io Threads Policy:               auto
        Machine:
          Type:  q35
        Resources:
          Limits:
            Cpu:  3
          Requests:
            Memory:  3G
      Networks:
        Multus:
          Network Name:  usb-int-br50-nad
        Name:            usb-int
        Multus:
          Network Name:  vpc-mgmt-br4-nad
        Name:            vpc-mgmt
      Node Selector:
        Networkmanager:                  true
      Termination Grace Period Seconds:  0
      Volumes:
        Name:  os-pvc
        Persistent Volume Claim:
          Claim Name:  file-block-storshelladmindal0989-os-ubuntu22
        Name:          drive1
        Persistent Volume Claim:
          Claim Name:  file-block-storshelladmindal0989-drive1
        Name:          drive2
        Persistent Volume Claim:
          Claim Name:  file-block-storshelladmindal0989-drive2
        Cloud Init No Cloud:
          Network Data:  version: 2
ethernets:
  usb-int:
    match:
      macaddress: 02:00:b1:98:ae:9a
    set-name: usb-int
    dhcp4: false
    addresses:
      - <IP_ADDRESS>/28
    routes:
      - to: default
        via: <IP_ADDRESS>
      - to: <IP_ADDRESS>/25
        via: <IP_ADDRESS>
  vpc-mgmt:
    match:
      macaddress: 02:01:b1:98:ae:9a
    set-name: vpc-mgmt
    dhcp4: false
    addresses:
      - <IP_ADDRESS>/16
          User Data:  #cloud-config
chpasswd:
  list: |
    <redacted>
  expire: False
users:
    - default
    - name: tmp_filenblock
      ssh-authorized-keys:
        - ssh-ed25519 A<redacted>R
      sudo: ['ALL=(ALL) NOPASSWD:ALL']
      groups: sudo
      shell: /bin/bash
write_files:
  - content: |
      PubkeyAcceptedKeyTypes=+ssh-ed25519
    path: /etc/ssh/sshd_config.d/00-cloud-init
runcmd:
  - service sshd restart

        Name:  cloudinitdisk
Status:
  Conditions:
    Last Probe Time:       2023-05-21T22:49:12Z
    Last Transition Time:  2023-05-21T22:49:12Z
    Message:               Guest VM is not reported as running
    Reason:                GuestNotRunning
    Status:                False
    Type:                  Ready
    Last Probe Time:       <nil>
    Last Transition Time:  <nil>
    Message:               cannot migrate VMI: PVC file-block-storshelladmindal0989-os-ubuntu22 is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode)
    Reason:                DisksNotLiveMigratable
    Status:                False
    Type:                  LiveMigratable
    Last Probe Time:       <nil>
    Last Transition Time:  2023-05-21T22:49:22Z
    Message:               failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found
    Reason:                Synchronizing with the Domain failed.
    Status:                False
    Type:                  Synchronized
  Created:                 true
  Printable Status:        Starting
  Volume Snapshot Statuses:
    Enabled:  true
    Name:     os-pvc
    Enabled:  true
    Name:     drive1
    Enabled:  true
    Name:     drive2
    Enabled:  false
    Name:     cloudinitdisk
    Reason:   Snapshot is not supported for this volumeSource type [cloudinitdisk]
Events:
  Type    Reason            Age    From                       Message
  ----    ------            ----   ----                       -------
  Normal  SuccessfulCreate  3m35s  virtualmachine-controller  Started the virtual machine by creating the new virtual machine instance storshelladmindal0989

kubectl get all -n kubevirt -o wide | grep worker-node1
Warning: kubevirt.io/v1 VirtualMachineInstancePresets is now deprecated and will be removed in v2.
pod/virt-api-7577f6fb77-v5xfw          1/1     Running   0               4h52m   <IP_ADDRESS>   worker-node1   <none>           <none>
pod/virt-controller-746fcd8c57-jm9lg   1/1     Running   0               4h52m   <IP_ADDRESS>   worker-node1   <none>           <none>
pod/virt-handler-5bhf4                 1/1     Running   2 (6h9m ago)    8h      <IP_ADDRESS>   worker-node1   <none>           <none>
pod/virt-operator-8685ddff-xxfgh       1/1     Running   0               4h52m   <IP_ADDRESS>    worker-node1   <none>           <none>

kubectl logs pod/virt-handler-5bhf4 -n kubevirt
{"component":"virt-handler","hostname":"worker-node1","level":"info","pos":"virt-handler.go:198","timestamp":"2023-05-21T17:35:31.051971Z"}
W0521 17:35:31.053816 1086079 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T17:35:31.100834Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T17:35:31.100858Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T17:35:31.100933Z"}
{"component":"virt-handler","level":"warning","msg":"host model mode is expected to contain only one model","pos":"cpu_plugin.go:98","timestamp":"2023-05-21T17:35:31.117581Z"}
{"component":"virt-handler","level":"info","msg":"node-labeller is running","pos":"node_labeller.go:110","timestamp":"2023-05-21T17:35:31.120465Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T17:35:31.120565Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T17:35:31.120737Z"}
{"component":"virt-handler","level":"info","msg":"Starting domain stats collector: node name=worker-node1","pos":"prometheus.go:531","timestamp":"2023-05-21T17:35:31.121183Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer CRDInformer","pos":"virtinformers.go:330","timestamp":"2023-05-21T17:35:31.121286Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer kubeVirtInformer","pos":"virtinformers.go:330","timestamp":"2023-05-21T17:35:31.121309Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer extensionsKubeVirtCAConfigMapInformer","pos":"virtinformers.go:330","timestamp":"2023-05-21T17:35:31.121323Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer vmiInformer-sources","pos":"virtinformers.go:330","timestamp":"2023-05-21T17:35:31.121338Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer vmiInformer-targets","pos":"virtinformers.go:330","timestamp":"2023-05-21T17:35:31.121350Z"}
{"component":"virt-handler","level":"info","msg":"metrics: max concurrent requests=3","pos":"virt-handler.go:495","timestamp":"2023-05-21T17:35:31.121565Z"}
{"component":"virt-handler","level":"info","msg":"certificate with common name 'kubevirt.io:system:node:virt-handler' retrieved.","pos":"cert-manager.go:198","timestamp":"2023-05-21T17:35:31.121714Z"}
{"component":"virt-handler","level":"info","msg":"certificate with common name 'kubevirt.io:system:client:virt-handler' retrieved.","pos":"cert-manager.go:198","timestamp":"2023-05-21T17:35:31.121870Z"}
{"component":"virt-handler","level":"info","msg":"Updating cluster config from KubeVirt to resource version '363880'","pos":"configuration.go:320","timestamp":"2023-05-21T17:35:31.129440Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T17:35:31.129482Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T17:35:31.129501Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T17:35:31.602778Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T17:35:31.603090Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T17:35:31.603378Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T17:35:31.603361Z"}
{"component":"virt-handler","level":"info","msg":"Starting virt-handler controller.","pos":"vm.go:1387","timestamp":"2023-05-21T17:35:31.652714Z"}
{"component":"virt-handler","level":"info","msg":"Starting a device plugin for device: vhost-net","pos":"device_controller.go:57","timestamp":"2023-05-21T17:35:31.652886Z"}
{"component":"virt-handler","level":"info","msg":"Starting a device plugin for device: kvm","pos":"device_controller.go:57","timestamp":"2023-05-21T17:35:31.653012Z"}
{"component":"virt-handler","level":"info","msg":"Starting a device plugin for device: tun","pos":"device_controller.go:57","timestamp":"2023-05-21T17:35:31.653055Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T17:35:31.653121Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T17:35:31.653225Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T17:35:31.653340Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T17:35:31.653655Z"}
{"component":"virt-handler","level":"info","msg":"tun device plugin started","pos":"generic_device.go:161","timestamp":"2023-05-21T17:35:31.655285Z"}
{"component":"virt-handler","level":"info","msg":"device '/dev/net/tun' is present.","pos":"generic_device.go:310","timestamp":"2023-05-21T17:35:31.655433Z"}
{"component":"virt-handler","level":"info","msg":"kvm device plugin started","pos":"generic_device.go:161","timestamp":"2023-05-21T17:35:31.655459Z"}
{"component":"virt-handler","level":"info","msg":"device '/dev/kvm' is present.","pos":"generic_device.go:310","timestamp":"2023-05-21T17:35:31.655751Z"}
{"component":"virt-handler","level":"info","msg":"vhost-net device plugin started","pos":"generic_device.go:161","timestamp":"2023-05-21T17:35:31.688327Z"}
{"component":"virt-handler","level":"info","msg":"device '/dev/vhost-net' is present.","pos":"generic_device.go:310","timestamp":"2023-05-21T17:35:31.688572Z"}
{"component":"virt-handler","level":"info","msg":"Starting a device plugin for device: intel.com/82599EB","pos":"device_controller.go:57","timestamp":"2023-05-21T17:35:31.690500Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:320","timestamp":"2023-05-21T17:35:31.690587Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: [intel.com/82599EB]","pos":"device_controller.go:321","timestamp":"2023-05-21T17:35:31.690633Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-05-21T17:35:31.690658Z"}
{"component":"virt-handler","level":"info","msg":"intel.com/82599EB device plugin started","pos":"pci_device.go:158","timestamp":"2023-05-21T17:35:31.692015Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:320","timestamp":"2023-05-21T17:35:31.727469Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: []","pos":"device_controller.go:321","timestamp":"2023-05-21T17:35:31.727550Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-05-21T17:35:31.727577Z"}
W0521 17:35:50.978639 1086079 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
{"component":"virt-handler","level":"info","msg":"Updating cluster config from KubeVirt to resource version '387845'","pos":"configuration.go:320","timestamp":"2023-05-21T17:35:56.186276Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T17:35:56.186426Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T17:35:56.186613Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:320","timestamp":"2023-05-21T17:35:56.227281Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: []","pos":"device_controller.go:321","timestamp":"2023-05-21T17:35:56.227339Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-05-21T17:35:56.227367Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T17:40:31.123841Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T17:45:31.123994Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T17:50:31.123709Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T17:55:31.123593Z"}
{"component":"virt-handler","level":"info","msg":"Updating cluster config from KubeVirt to resource version '396963'","pos":"configuration.go:320","timestamp":"2023-05-21T17:56:12.633445Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T17:56:12.633630Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T17:56:12.633834Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:320","timestamp":"2023-05-21T17:56:12.670328Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: []","pos":"device_controller.go:321","timestamp":"2023-05-21T17:56:12.670389Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-05-21T17:56:12.670417Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T18:00:31.124318Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T18:05:31.124316Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T18:10:31.124354Z"}
{"component":"virt-handler","level":"info","msg":"Updating cluster config from KubeVirt to resource version '403582'","pos":"configuration.go:320","timestamp":"2023-05-21T18:10:43.356663Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T18:10:43.356834Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T18:10:43.357052Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:320","timestamp":"2023-05-21T18:10:43.397273Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: []","pos":"device_controller.go:321","timestamp":"2023-05-21T18:10:43.397330Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-05-21T18:10:43.397357Z"}
{"component":"virt-handler","level":"info","msg":"Updating cluster config from KubeVirt to resource version '403710'","pos":"configuration.go:320","timestamp":"2023-05-21T18:10:48.521961Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T18:10:48.522195Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T18:10:48.522276Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:320","timestamp":"2023-05-21T18:10:48.562768Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: []","pos":"device_controller.go:321","timestamp":"2023-05-21T18:10:48.562829Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-05-21T18:10:48.562859Z"}
{"component":"virt-handler","level":"info","msg":"Updating cluster config from KubeVirt to resource version '405449'","pos":"configuration.go:320","timestamp":"2023-05-21T18:12:57.225585Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T18:12:57.225796Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T18:12:57.225893Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:320","timestamp":"2023-05-21T18:12:57.259955Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: []","pos":"device_controller.go:321","timestamp":"2023-05-21T18:12:57.260011Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-05-21T18:12:57.260040Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T18:15:31.123805Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T18:20:31.124527Z"}
{"component":"virt-handler","level":"info","msg":"Updating cluster config from KubeVirt to resource version '410588'","pos":"configuration.go:320","timestamp":"2023-05-21T18:25:10.731077Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T18:25:10.731187Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T18:25:10.731411Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:320","timestamp":"2023-05-21T18:25:10.768625Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: []","pos":"device_controller.go:321","timestamp":"2023-05-21T18:25:10.768686Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-05-21T18:25:10.768717Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T18:25:31.124641Z"}
{"component":"virt-handler","level":"info","msg":"Updating cluster config from KubeVirt to resource version '412315'","pos":"configuration.go:320","timestamp":"2023-05-21T18:28:12.708419Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T18:28:12.708540Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T18:28:12.708702Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:320","timestamp":"2023-05-21T18:28:12.750533Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: []","pos":"device_controller.go:321","timestamp":"2023-05-21T18:28:12.750593Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-05-21T18:28:12.750607Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T18:30:31.124726Z"}
{"component":"virt-handler","level":"info","msg":"Updating cluster config from KubeVirt to resource version '413808'","pos":"configuration.go:320","timestamp":"2023-05-21T18:31:08.095793Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T18:31:08.095955Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T18:31:08.096056Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:320","timestamp":"2023-05-21T18:31:08.132463Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: []","pos":"device_controller.go:321","timestamp":"2023-05-21T18:31:08.132527Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-05-21T18:31:08.132556Z"}
{"component":"virt-handler","level":"info","msg":"Updating cluster config from KubeVirt to resource version '414655'","pos":"configuration.go:320","timestamp":"2023-05-21T18:33:14.274800Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T18:33:14.274998Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T18:33:14.275239Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:320","timestamp":"2023-05-21T18:33:14.310517Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: []","pos":"device_controller.go:321","timestamp":"2023-05-21T18:33:14.310653Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-05-21T18:33:14.310718Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T18:35:31.124718Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T18:40:31.123975Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T18:45:31.124443Z"}
{"component":"virt-handler","level":"info","msg":"Updating cluster config from KubeVirt to resource version '421385'","pos":"configuration.go:320","timestamp":"2023-05-21T18:49:20.608748Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T18:49:20.608977Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T18:49:20.609106Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:320","timestamp":"2023-05-21T18:49:20.643482Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: []","pos":"device_controller.go:321","timestamp":"2023-05-21T18:49:20.643544Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-05-21T18:49:20.643576Z"}
{"component":"virt-handler","level":"info","msg":"Updating cluster config from KubeVirt to resource version '421661'","pos":"configuration.go:320","timestamp":"2023-05-21T18:49:33.767504Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T18:49:33.767615Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T18:49:33.767781Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:320","timestamp":"2023-05-21T18:49:33.807622Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: []","pos":"device_controller.go:321","timestamp":"2023-05-21T18:49:33.807698Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-05-21T18:49:33.807726Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T18:50:31.123935Z"}
{"component":"virt-handler","level":"info","msg":"Updating cluster config from KubeVirt to resource version '423269'","pos":"configuration.go:320","timestamp":"2023-05-21T18:51:58.315836Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T18:51:58.315991Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T18:51:58.316106Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:320","timestamp":"2023-05-21T18:51:58.353112Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: []","pos":"device_controller.go:321","timestamp":"2023-05-21T18:51:58.353171Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-05-21T18:51:58.353200Z"}
{"component":"virt-handler","level":"info","msg":"Updating cluster config from KubeVirt to resource version '424084'","pos":"configuration.go:320","timestamp":"2023-05-21T18:53:46.770643Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:480","timestamp":"2023-05-21T18:53:46.770819Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:471","timestamp":"2023-05-21T18:53:46.770980Z"}
{"component":"virt-handler","level":"info","msg":"refreshed device plugins for permitted/forbidden host devices","pos":"device_controller.go:320","timestamp":"2023-05-21T18:53:46.807969Z"}
{"component":"virt-handler","level":"info","msg":"enabled device-plugins for: []","pos":"device_controller.go:321","timestamp":"2023-05-21T18:53:46.808036Z"}
{"component":"virt-handler","level":"info","msg":"disabled device-plugins for: []","pos":"device_controller.go:322","timestamp":"2023-05-21T18:53:46.808068Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T18:55:31.124288Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T19:00:31.124543Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T19:05:31.124245Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T19:10:31.123851Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T19:15:31.124408Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T19:20:31.124298Z"}
{"component":"virt-handler","level":"info","msg":"certificate with common name 'kubevirt.io:system:node:virt-handler' retrieved.","pos":"cert-manager.go:198","timestamp":"2023-05-21T19:25:23.554856Z"}
{"component":"virt-handler","level":"info","msg":"certificate with common name 'kubevirt.io:system:client:virt-handler' retrieved.","pos":"cert-manager.go:198","timestamp":"2023-05-21T19:25:23.555149Z"}
{"component":"virt-handler","level":"info","msg":"certificate with common name 'kubevirt.io:system:node:virt-handler' retrieved.","pos":"cert-manager.go:198","timestamp":"2023-05-21T19:25:23.555318Z"}
{"component":"virt-handler","level":"info","msg":"certificate with common name 'kubevirt.io:system:client:virt-handler' retrieved.","pos":"cert-manager.go:198","timestamp":"2023-05-21T19:25:23.555690Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T19:25:31.124447Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T19:30:31.124635Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T19:35:31.124376Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T19:40:31.124464Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T19:45:31.123887Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T19:50:31.124364Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T19:55:31.124516Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T20:00:31.124202Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T20:05:31.124618Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T20:10:31.124569Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T20:15:31.124400Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T20:20:31.123864Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T20:25:31.124617Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T20:30:31.124637Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T20:35:31.123832Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T20:40:31.124726Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T20:45:31.124025Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T20:50:31.124462Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T20:55:31.123946Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T21:00:31.124217Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T21:05:31.124851Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T21:10:31.124746Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T21:15:31.124658Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T21:20:31.124220Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T21:25:31.124521Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T21:30:31.124456Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T21:35:31.124099Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T21:40:31.124517Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T21:45:31.124411Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T21:50:31.124835Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T21:55:31.124250Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T22:00:31.124304Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T22:05:31.124187Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T22:10:31.124134Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T22:15:31.124120Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T22:20:31.124095Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T22:25:31.123738Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T22:30:31.124526Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T22:35:31.124304Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T22:40:31.124597Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T22:45:31.124491Z"}
{"component":"virt-handler","level":"info","msg":"Generic Allocate: resourceName: kvm","pos":"generic_device.go:247","timestamp":"2023-05-21T22:49:12.812787Z"}
{"component":"virt-handler","level":"info","msg":"Generic Allocate: request: [\u0026ContainerAllocateRequest{DevicesIDs:[kvm638],}]","pos":"generic_device.go:248","timestamp":"2023-05-21T22:49:12.812940Z"}
{"component":"virt-handler","level":"info","msg":"Generic Allocate: resourceName: tun","pos":"generic_device.go:247","timestamp":"2023-05-21T22:49:12.815216Z"}
{"component":"virt-handler","level":"info","msg":"Generic Allocate: request: [\u0026ContainerAllocateRequest{DevicesIDs:[tun82],}]","pos":"generic_device.go:248","timestamp":"2023-05-21T22:49:12.815308Z"}
{"component":"virt-handler","level":"info","msg":"Generic Allocate: resourceName: vhost-net","pos":"generic_device.go:247","timestamp":"2023-05-21T22:49:12.817472Z"}
{"component":"virt-handler","level":"info","msg":"Generic Allocate: request: [\u0026ContainerAllocateRequest{DevicesIDs:[vhost-net803],}]","pos":"generic_device.go:248","timestamp":"2023-05-21T22:49:12.817575Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:49:22.710656Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:49:22.713656Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:49:22.713734Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:49:22.715323Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:22.715464Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:22.768603Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:49:22.769218Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:49:22.769656Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:49:22.769707Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:49:22.770501Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:22.770621Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:22.771172Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:49:22.774583Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:49:22.775041Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:49:22.775098Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:49:22.775942Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:22.776038Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:22.776458Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:49:22.797775Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:49:22.798279Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:49:22.798333Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:49:22.799125Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:22.799241Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:22.799672Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:49:22.839941Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:49:22.840468Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:49:22.840523Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:49:22.863065Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:22.863185Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:22.863617Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:49:22.944542Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:49:22.944993Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:49:22.945045Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:49:22.945885Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:22.945989Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:22.946338Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:49:23.106914Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:49:23.107363Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:49:23.107418Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:49:23.108162Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:23.108273Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:23.108610Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:49:23.429861Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:49:23.430355Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:49:23.430412Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:49:23.431263Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:23.431396Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:23.431758Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:49:24.072850Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:49:24.073414Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:49:24.073475Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:49:24.074420Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:24.074536Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:24.075015Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:49:25.356151Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:49:25.356853Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:49:25.356926Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:49:25.357954Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:25.358073Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:25.358475Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:49:27.918825Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:49:27.919426Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:49:27.919487Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:49:27.920515Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:27.920630Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:27.921044Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:49:33.041486Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:49:33.042088Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:49:33.042148Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:49:33.043193Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:33.043319Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:33.043681Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:49:43.284849Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:49:43.286320Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:49:43.286866Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:49:43.290794Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:43.291860Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:49:43.292251Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:50:03.773024Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:50:03.773630Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:50:03.773687Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:50:03.774668Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:50:03.774785Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:50:03.775223Z"}
{"component":"virt-handler","level":"info","msg":"resyncing virt-launcher domains","pos":"cache.go:385","timestamp":"2023-05-21T22:50:31.124713Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:50:44.736353Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:50:44.737043Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:50:44.737106Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:50:44.738049Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:50:44.738160Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:50:44.738697Z"}
{"component":"virt-handler","kind":"","level":"info","msg":"VMI is in phase: Scheduled | Domain does not exist","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1643","timestamp":"2023-05-21T22:52:06.660043Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"mounting kernel artifacts","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:405","timestamp":"2023-05-21T22:52:06.660707Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","kind":"","level":"info","msg":"kernel boot not defined - nothing to mount","name":"storshelladmindal0989","namespace":"file-block","pos":"mount.go:408","timestamp":"2023-05-21T22:52:06.660770Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"error","msg":"failed to get a link for interface: net1","pos":"common.go:301","reason":"Link not found","timestamp":"2023-05-21T22:52:06.662097Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"storshelladmindal0989","namespace":"file-block","pos":"vm.go:1781","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:52:06.662275Z","uid":"b8ff6d55-ef97-4492-8d3e-0c0d9ebba084"}
{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance file-block/storshelladmindal0989","pos":"vm.go:1433","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'net1': Link not found","timestamp":"2023-05-21T22:52:06.662830Z"}

kubectl get net-attach-def -n file-block
NAME               AGE
usb-int-br50-nad   5m56s
vpc-mgmt-br4-nad   5m51s

kubectl get net-attach-def -n file-block -o yaml
apiVersion: v1
items:
- apiVersion: k8s.cni.cncf.io/v1
  kind: NetworkAttachmentDefinition
  metadata:
    creationTimestamp: "2023-05-21T22:49:02Z"
    generation: 1
    name: usb-int-br50-nad
    namespace: file-block
    resourceVersion: "515410"
    uid: e5b33b99-0130-4ef1-81fa-f33c36babe47
  spec:
    config: '{ "cniVersion": "0.3.1", "name": "usb-int-br50-nad", "type": "bridge",
      "bridge": "usb-int-br50", "isDefaultGateway": false, "forceAddress": false,
      "ipMasq": false, "hairpinMode": false, "ipam": { } }'
- apiVersion: k8s.cni.cncf.io/v1
  kind: NetworkAttachmentDefinition
  metadata:
    creationTimestamp: "2023-05-21T22:49:07Z"
    generation: 1
    name: vpc-mgmt-br4-nad
    namespace: file-block
    resourceVersion: "515446"
    uid: 7f2f6186-b634-4f2b-8f4e-2a0f388c46e2
  spec:
    config: '{ "cniVersion": "0.3.1", "name": "vpc-mgmt-br4-nad", "type": "bridge",
      "bridge": "vpc-mgmt-br4", "isDefaultGateway": false, "forceAddress": false,
      "ipMasq": false, "hairpinMode": false, "ipam": { } }'
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

@k8scoder192 , thank you for sharing the information.
I would expect to see multus logging net1 and to see the annotation on the pod with both the desired and actual (status) from multus, but I think you have not added the pod manifest dump.
In general, the flow is as follows:

...=>> VM ==>> VMI (here we have a VMI)
The VMI spec is processed to create a virt-launcher with CNI/MULTUS annotations.
When the pod is created, CNI plugins are called (through multus) and multus updates the pod annotation with the results (networks-status is the key I think).

Please try to follow this flow and see if you see each step occurring.
Make sure you have an updated Multus, if it is too old, we are not testing with old versions and maybe there is a problem.
I would suggest testing it with what we use in kubevirtci.

@EdDev thanks. Can you clarify what pod manifest you wanted?  As you mentioned there is a full flow and various components interact so I need clarity.
Please be explicit in specifying exactly what you would like to me to capture.

@EdDev thanks. Can you clarify what pod manifest or logs you wanted? As you mentioned there is a full flow and various components interact so I need clarity.
Please be explicit in specifying exactly what you would like to me to capture.

I tried to guide you to help troubleshoot this.
I think the logs have been provided, but I could not see anything special in them.
Specifically, not an answer on why net1 & net2 (the secondary interfaces) are not assigned to the VM pod (virt-launcher).
A pod manifest has been provided by @hijak earlier [1].
The pod should contain the multus CNI annotations about the secondary networks. Kubevirt adds them there and multus needs to process them (this is why I suggester you to make sure you are using an up-to-date one).
[1] https://github.com/kubevirt/kubevirt/issues/9658#issuecomment-1524930221

@EdDev see kubectl describe virtualmachineinstance.kubevirt.io/storshelladmindal0989 -n file-block section in my post, a manifest can be extrapolated.
I'll have to check which multus version used but it's fairly recent.
I'm not doubting you were trying to help, I asked explicit because you referenced the term pod but wasn't clear if you were referring to multus or the vmi pod. Reproducing this isn't easy so it's best to ask before proceeding.

@EdDev see kubectl describe virtualmachineinstance.kubevirt.io/storshelladmindal0989 -n file-block section in my post, a manifest can be extrapolated.

I am asking for the pod manifest, not the VMI one.

I'll have to check which multus version used but it's fairly recent.
I'm not doubting you were trying to help, I asked explicit because you referenced the term pod but wasn't clear if you were referring to multus or the vmi pod. Reproducing this isn't easy so it's best to ask before proceeding.

Then perhaps you should post here what information you intend to collect, i.e. your plan. And then we can try to review/comment about it (e.g. if we see something missing).
Generally speaking, I would suggest to collect everything once you recreate it, or just freeze it until we can find something.
In the e2e tests we collect there a lot to be able and debug it later, if you want to do the same collection perhaps review the e2e reporter code [1].
[1] https://github.com/kubevirt/kubevirt/blob/39dc57cad5ea6f882b847fd1ab312ee951d8cd9c/tests/reporter/kubernetes.go#L120

@k8scoder192 , could you please update if you managed to collect the information?

@EdDev it's difficult to replicate and I'm on vacation for a while. I will do my best when back to get it in the failed state to get the data

Thank you for all the information.
From what I can see, everything seems to be right in terms of pod definition but obviously something is really odd with how the setup works with multus.
Could you share (if I missed it above, sorry) which version of multus are you using?
@maiqueb , could you have a look at the logs above? Perhaps you may have some insights on this.
My understanding is that Multus is not delegating the call to the bridge CNI, but I do not understand why. Maybe I am just missing something.

@maiqueb , could you have a look at the logs above? Perhaps you may have some insights on this.

I don't think multus can even run with its config file gone (it only accepts .json, .conf, .conflist sufixed files).
https://github.com/k8snetworkplumbingwg/multus-cni/blob/753d18740abbad1ee81ccf18c8a2180dd6b10ecb/vendor/github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/utils/cniconfig.go#L66

@EdDev @maiqueb
I agree. From the output of multus pod, it's having trouble finding/connecting with cilium agent. When I inspected the problematic host the VM was scheduled for, 00-multus.conf was missing. Idk if that's multus' behavior when it eventually gives up trying to communicate with the CNI (Cilium here) or not.
One other clue, when I looked at Cilium pods, there have been multiple restarts (I don't know the reason, another team handles that plug-in). My best guess, and hopefully you can provide some feedback, the erratic behavior of cilium pods coming/in out, causes multus to freakout/timeout, and gives up. When this happens, multus is not longer functional and calls from Kubevirt fail to configure the required network.
Is this is indeed the case, it's more of a CNI problem than anything else. Let me know your thoughts. Thanks for your assistance.