After Upgrade to 4.14 - all Cluster members not reachable

After updating vom 4.06 to 4.14 my cluster completly broke.
only message in lxc cluster list is "No heartbeat for xxx" on all members, even itself.
Maybe because of new cluster certs? Didnt find a migration document. Need help how to get my cluster responsive again.
config:
  cluster.https_address: <IP_ADDRESS>:9443
  core.https_address: <IP_ADDRESS>:9443
  core.trust_password: true
  images.auto_update_interval: "15"
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - <IP_ADDRESS>:9443
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
XXXXXXXXXXX
    -----END CERTIFICATE-----
  certificate_fingerprint: xxx
  driver: lxc | qemu
  driver_version: 4.0.9 | 5.2.0
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 5.10.0-0.bpo.3-amd64
  lxc_features:
    cgroup2: "true"
    devpts_fd: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Debian GNU/Linux
  os_version: "10"
  project: default
  server: lxd
  server_clustered: true
  server_name: cluster-00
  server_pid: 165814
  server_version: "4.14"
  storage: ceph | dir
  storage_version: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e)
    luminous (stable) | 1

Please can you enable debug mode and see what is being logged in the LXD log?
sudo snap set lxd daemon.debug=true; sudo systemctl reload snap.lxd.daemon
sudo tail -f /var/snap/lxd/common/lxd/logs/lxd.log

I just tried upgrading a LXD 4.0.6 (pre-per-server-certificates) cluster to a LXD 4.14 cluster and it worked fine.

t=2021-06-01T18:39:25+0200 lvl=info msg="LXD 4.14 is starting in normal mode" path=/var/snap/lxd/common/lxd
t=2021-06-01T18:39:25+0200 lvl=info msg="Kernel uid/gid map:"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - u 0 0<PHONE_NUMBER>"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - g 0 0<PHONE_NUMBER>"
t=2021-06-01T18:39:25+0200 lvl=info msg="Configured LXD uid/gid map:"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - u 0 1000000<PHONE_NUMBER>"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - g 0 1000000<PHONE_NUMBER>"
t=2021-06-01T18:39:25+0200 lvl=info msg="Kernel features:"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - closing multiple file descriptors efficiently: yes"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - netnsid-based network retrieval: yes"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - pidfds: yes"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - uevent injection: yes"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - seccomp listener: yes"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - seccomp listener continue syscalls: yes"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - seccomp listener add file descriptors: yes"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - attach to namespaces via pidfds: yes"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - safe native terminal allocation : yes"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - unprivileged file capabilities: yes"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - cgroup layout: hybrid"
t=2021-06-01T18:39:25+0200 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - shiftfs support: disabled"
t=2021-06-01T18:39:25+0200 lvl=info msg="Initializing local database"
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Refreshing local trusted certificate cache"
t=2021-06-01T18:39:25+0200 lvl=info msg="Set client certificate to server certificate e19623e335ba49179d1840f9d7bd1dcaabfe116fad0b5ab9c3cbb2d9bdcf8bc9"
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Initializing database gateway"
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Start database node" address=<IP_ADDRESS>:9443 id=1 role=spare
t=2021-06-01T18:39:25+0200 lvl=info msg="Starting cluster handler:"
t=2021-06-01T18:39:25+0200 lvl=info msg="Starting /dev/lxd handler:"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2021-06-01T18:39:25+0200 lvl=info msg="REST API daemon:"
t=2021-06-01T18:39:25+0200 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2021-06-01T18:39:25+0200 lvl=info msg=" - binding TCP socket" socket=<IP_ADDRESS>:9443
t=2021-06-01T18:39:25+0200 lvl=info msg="Initializing global database"
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Matched trusted cert" fingerprint=e19623e335ba49179d1840f9d7bd1dcaabfe116fad0b5ab9c3cbb2d9bdcf8bc9 subject="CN=root@cluster-00,O=linuxcontainers.org"
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Triggering an out of schedule hearbeat" address=<IP_ADDRESS>:9443
t=2021-06-01T18:39:25+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Triggering an out of schedule hearbeat" address=<IP_ADDRESS>:9443
t=2021-06-01T18:39:25+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Triggering an out of schedule hearbeat" address=<IP_ADDRESS>:9443
t=2021-06-01T18:39:25+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Triggering an out of schedule hearbeat" address=<IP_ADDRESS>:9443
t=2021-06-01T18:39:25+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Triggering an out of schedule hearbeat" address=<IP_ADDRESS>:9443
t=2021-06-01T18:39:25+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Triggering an out of schedule hearbeat" address=<IP_ADDRESS>:9443
t=2021-06-01T18:39:25+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Triggering an out of schedule hearbeat" address=<IP_ADDRESS>:9443
t=2021-06-01T18:39:25+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Triggering an out of schedule hearbeat" address=<IP_ADDRESS>:9443
t=2021-06-01T18:39:25+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Triggering an out of schedule hearbeat" address=<IP_ADDRESS>:9443
t=2021-06-01T18:39:25+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Triggering an out of schedule hearbeat" address=<IP_ADDRESS>:9443
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: connect to reported leader <IP_ADDRESS>:9443"
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Triggering an out of schedule hearbeat" address=<IP_ADDRESS>:9443
t=2021-06-01T18:39:25+0200 lvl=dbug msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: connected"
t=2021-06-01T18:39:25+0200 lvl=info msg="Firewall loaded driver \"nftables\""
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Notify node <IP_ADDRESS>:9443 of state changes"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Notify node <IP_ADDRESS>:9443 of state changes"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Notify node <IP_ADDRESS>:9443 of state changes"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Notify node <IP_ADDRESS>:9443 of state changes"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Notify node <IP_ADDRESS>:9443 of state changes"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Notify node <IP_ADDRESS>:9443 of state changes"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Notify node <IP_ADDRESS>:9443 of state changes"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Notify node <IP_ADDRESS>:9443 of state changes"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Notify node <IP_ADDRESS>:9443 of state changes"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Notify node <IP_ADDRESS>:9443 of state changes"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Notify node <IP_ADDRESS>:9443 of state changes"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Notify node <IP_ADDRESS>:9443 of state changes"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Notify node <IP_ADDRESS>:9443 of state changes"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Notify node <IP_ADDRESS>:9443 of state changes"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Notify node <IP_ADDRESS>:9443 of state changes"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs"
t=2021-06-01T18:39:26+0200 lvl=info msg="Initializing storage pools"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Initializing and checking storage pool \"default\""
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Mount started" driver=dir pool=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Mount finished" driver=dir pool=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Initializing and checking storage pool \"ceph\""
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Mount started" driver=ceph pool=ceph
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Mount finished" driver=ceph pool=ceph
t=2021-06-01T18:39:26+0200 lvl=info msg="Initializing daemon storage mounts"
t=2021-06-01T18:39:26+0200 lvl=info msg="Initializing networks"
t=2021-06-01T18:39:26+0200 lvl=dbug msg=Start driver=bridge network=lxdbr0 project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Setting up network" driver=bridge network=lxdbr0 project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Stable MAC generated" driver=bridge hwAddr=00:16:3e:08:bf:77 network=lxdbr0 project=default seed=e19623e335ba49179d1840f9d7bd1dcaabfe116fad0b5ab9c3cbb2d9bdcf8bc9.0.1
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Clearing firewall" driver=bridge network=lxdbr0 project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Setting up firewall" driver=bridge network=lxdbr0 project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="New task Operation: 68bae430-5f94-4edf-8fb8-797d1dbac63f"
t=2021-06-01T18:39:26+0200 lvl=info msg="Pruning leftover image files"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Started task operation: 68bae430-5f94-4edf-8fb8-797d1dbac63f"
t=2021-06-01T18:39:26+0200 lvl=info msg="Done pruning leftover image files"
t=2021-06-01T18:39:26+0200 lvl=info msg="Loading daemon configuration"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Success for task operation: 68bae430-5f94-4edf-8fb8-797d1dbac63f"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Initialized inotify with file descriptor 21"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Initialising mounted disk ref counter" device=root driver=disk instance=monitor project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="MountInstance started" driver=ceph instance=monitor pool=ceph project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="MountInstance finished" driver=ceph instance=monitor pool=ceph project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Initialising mounted disk ref counter" device=root driver=disk instance=vpn project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="MountInstance started" driver=ceph instance=vpn pool=ceph project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="MountInstance finished" driver=ceph instance=vpn pool=ceph project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Initialising mounted disk ref counter" device=root driver=disk instance=webc00-lb project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="MountInstance started" driver=ceph instance=webc00-lb pool=ceph project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="MountInstance finished" driver=ceph instance=webc00-lb pool=ceph project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Initialising mounted disk ref counter" device=shared_storage driver=disk instance=webc00-lb project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Initialising mounted disk ref counter" device=root driver=disk instance=webc01-lb project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="MountInstance started" driver=ceph instance=webc01-lb pool=ceph project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="MountInstance finished" driver=ceph instance=webc01-lb pool=ceph project=default
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Initialising mounted disk ref counter" device=shared_storage driver=disk instance=webc01-lb project=default
t=2021-06-01T18:39:26+0200 lvl=info msg="Started seccomp handler" path=/var/snap/lxd/common/lxd/seccomp.socket
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Refreshing trusted certificate cache"
t=2021-06-01T18:39:26+0200 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/1.0 username=root
t=2021-06-01T18:39:26+0200 lvl=dbug msg="New task Operation: 16b886ca-2c39-4b67-8534-0da1afe5d52a"
t=2021-06-01T18:39:26+0200 lvl=info msg="Pruning expired images"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Started task operation: 16b886ca-2c39-4b67-8534-0da1afe5d52a"
t=2021-06-01T18:39:26+0200 lvl=info msg="Done pruning expired images"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Success for task operation: 16b886ca-2c39-4b67-8534-0da1afe5d52a"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="New task Operation: df94334f-b340-40fa-a461-df8d2ac36a01"
t=2021-06-01T18:39:26+0200 lvl=info msg="Pruning expired instance backups"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Started task operation: df94334f-b340-40fa-a461-df8d2ac36a01"
t=2021-06-01T18:39:26+0200 lvl=info msg="Done pruning expired instance backups"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Success for task operation: df94334f-b340-40fa-a461-df8d2ac36a01"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="New task Operation: 48fc40c2-d90a-4b8f-aace-fd1160d3f7ca"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="New task Operation: f6a6bb37-1b23-44e9-8d67-bebd4b135a57"
t=2021-06-01T18:39:26+0200 lvl=info msg="Pruning resolved warnings"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Started task operation: f6a6bb37-1b23-44e9-8d67-bebd4b135a57"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="New task Operation: da5a8494-221b-46c4-a113-e8b56e231d58"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Skipping image synchronization task since we're not leader"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="New task Operation: cc7cd39b-0215-4a75-9c14-111ec26dae27"
t=2021-06-01T18:39:26+0200 lvl=info msg="Expiring log files"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Started task operation: da5a8494-221b-46c4-a113-e8b56e231d58"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Success for task operation: f6a6bb37-1b23-44e9-8d67-bebd4b135a57"
t=2021-06-01T18:39:26+0200 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/internal/ready username=root
t=2021-06-01T18:39:26+0200 lvl=info msg="Updating instance types"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Started task operation: 48fc40c2-d90a-4b8f-aace-fd1160d3f7ca"
t=2021-06-01T18:39:26+0200 lvl=info msg="Done expiring log files"
t=2021-06-01T18:39:26+0200 lvl=info msg="Done updating instance types"
t=2021-06-01T18:39:26+0200 lvl=info msg="Updating images"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Started task operation: cc7cd39b-0215-4a75-9c14-111ec26dae27"
t=2021-06-01T18:39:26+0200 lvl=info msg="Done pruning resolved warnings"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Processing image" alias=debian/10 fingerprint=49ccee803693d1898dd1decf091e689bf887016316423d346008fe5a7f9bb00d protocol=simplestreams server=https://images.linuxcontainers.org
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Connecting to a remote simplestreams server"
t=2021-06-01T18:39:26+0200 lvl=info msg="Done updating images"
t=2021-06-01T18:39:26+0200 lvl=dbug msg="Success for task operation: da5a8494-221b-46c4-a113-e8b56e231d58"
t=2021-06-01T18:39:27+0200 lvl=dbug msg="Acquiring lock for image download of \"49ccee803693d1898dd1decf091e689bf887016316423d346008fe5a7f9bb00d\""
t=2021-06-01T18:39:27+0200 lvl=dbug msg="Lock acquired for image download of \"49ccee803693d1898dd1decf091e689bf887016316423d346008fe5a7f9bb00d\""
t=2021-06-01T18:39:27+0200 lvl=eror msg="Failed to update the image" err="Failed locating image \"49ccee803693d1898dd1decf091e689bf887016316423d346008fe5a7f9bb00d\" in the cluster: Image not available on any online node" fingerprint=49ccee803693d1898dd1decf091e689bf887016316423d346008fe5a7f9bb00d
t=2021-06-01T18:39:27+0200 lvl=dbug msg="Success for task operation: cc7cd39b-0215-4a75-9c14-111ec26dae27"
t=2021-06-01T18:39:27+0200 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/1.0 username=root
t=2021-06-01T18:39:27+0200 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/internal/ready username=root
t=2021-06-01T18:39:28+0200 lvl=dbug msg="Success for task operation: 48fc40c2-d90a-4b8f-aace-fd1160d3f7ca"
t=2021-06-01T18:39:56+0200 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/1.0 username=nagios
t=2021-06-01T18:39:56+0200 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/1.0/cluster username=nagios
t=2021-06-01T18:39:56+0200 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url="/1.0/cluster/members?recursion=1" username=nagios

+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
|   NAME   |            URL            | DATABASE | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION |  STATE  |                                   MESSAGE                                    |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-00 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        | master      | OFFLINE | No heartbeat for 45m12.527066262s (2021-06-01 17:56:43.723998555 +0200 CEST) |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-01 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | OFFLINE | No heartbeat for 45m12.539581647s (2021-06-01 17:56:43.715328926 +0200 CEST) |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-02 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | OFFLINE | No heartbeat for 45m12.547349112s (2021-06-01 17:56:43.713720051 +0200 CEST) |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-03 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | OFFLINE | No heartbeat for 45m12.539511094s (2021-06-01 17:56:43.718599352 +0200 CEST) |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-04 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | OFFLINE | No heartbeat for 45m12.555776444s (2021-06-01 17:56:43.714338752 +0200 CEST) |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-05 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | OFFLINE | No heartbeat for 45m12.544860531s (2021-06-01 17:56:43.728898528 +0200 CEST) |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-06 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | OFFLINE | No heartbeat for 45m12.54925345s (2021-06-01 17:56:43.717585919 +0200 CEST)  |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-07 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | OFFLINE | No heartbeat for 45m12.548838017s (2021-06-01 17:56:43.715003601 +0200 CEST) |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-08 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | OFFLINE | No heartbeat for 45m12.57922149s (2021-06-01 17:56:43.703314653 +0200 CEST)  |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-09 | https://<IP_ADDRESS>:9443 | YES      | x86_64       | default        |             | OFFLINE | No heartbeat for 45m12.573754598s (2021-06-01 17:56:43.705906306 +0200 CEST) |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-10 | https://<IP_ADDRESS>:9443 | YES      | x86_64       | default        |             | OFFLINE | No heartbeat for 45m42.551174001s (2021-06-01 17:56:13.725542873 +0200 CEST) |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-11 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | OFFLINE | No heartbeat for 45m12.571239116s (2021-06-01 17:56:43.713975724 +0200 CEST) |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-12 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | OFFLINE | No heartbeat for 45m12.578177361s (2021-06-01 17:56:43.709885186 +0200 CEST) |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-13 | https://<IP_ADDRESS>:9443 | YES      | x86_64       | default        |             | OFFLINE | No heartbeat for 45m12.575895633s (2021-06-01 17:56:43.720660963 +0200 CEST) |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-14 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | OFFLINE | No heartbeat for 45m12.569188462s (2021-06-01 17:56:43.722227038 +0200 CEST) |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+
| cluster-15 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | OFFLINE | No heartbeat for 45m12.578721321s (2021-06-01 17:56:43.715697972 +0200 CEST) |
+----------+---------------------------+----------+--------------+----------------+-------------+---------+------------------------------------------------------------------------------+

+--------------+---------------+------------------------------+------------------------------+
| FINGERPRINT  |  COMMON NAME  |          ISSUE DATE          |         EXPIRY DATE          |
+--------------+---------------+------------------------------+------------------------------+
| 7a0789d8f4ae | root@cluster-05 | Sep 3, 2020 at 9:22pm (UTC)  | Sep 1, 2030 at 9:22pm (UTC)  |
+--------------+---------------+------------------------------+------------------------------+
| 8f9b0f0e64f6 | root@cluster-07 | Sep 3, 2020 at 9:22pm (UTC)  | Sep 1, 2030 at 9:22pm (UTC)  |
+--------------+---------------+------------------------------+------------------------------+
| 54bc4aaf430f | root@cluster-03 | Sep 3, 2020 at 9:22pm (UTC)  | Sep 1, 2030 at 9:22pm (UTC)  |
+--------------+---------------+------------------------------+------------------------------+
| 63a8ef389a94 | root@cluster-13 | Oct 12, 2020 at 4:22pm (UTC) | Oct 10, 2030 at 4:22pm (UTC) |
+--------------+---------------+------------------------------+------------------------------+
| 93a417b54cc5 | root@cluster-08 | Sep 3, 2020 at 9:26pm (UTC)  | Sep 1, 2030 at 9:26pm (UTC)  |
+--------------+---------------+------------------------------+------------------------------+
| 99a74abb5605 | root@cluster-06 | Sep 3, 2020 at 9:22pm (UTC)  | Sep 1, 2030 at 9:22pm (UTC)  |
+--------------+---------------+------------------------------+------------------------------+
| 233dcb570a54 | root@cluster-10 | Sep 3, 2020 at 9:22pm (UTC)  | Sep 1, 2030 at 9:22pm (UTC)  |
+--------------+---------------+------------------------------+------------------------------+
| 927a98fcc52e | root@cluster-09 | Sep 3, 2020 at 9:26pm (UTC)  | Sep 1, 2030 at 9:26pm (UTC)  |
+--------------+---------------+------------------------------+------------------------------+
| bbb0ae40fb4d | root@cluster-01 | Sep 3, 2020 at 9:20pm (UTC)  | Sep 1, 2030 at 9:20pm (UTC)  |
+--------------+---------------+------------------------------+------------------------------+
| d6fbc0eaa5d8 | root@cluster-12 | Oct 12, 2020 at 4:22pm (UTC) | Oct 10, 2030 at 4:22pm (UTC) |
+--------------+---------------+------------------------------+------------------------------+
| e2ff1f231680 | root@cluster-11 | Sep 11, 2020 at 9:49pm (UTC) | Sep 9, 2030 at 9:49pm (UTC)  |
+--------------+---------------+------------------------------+------------------------------+
| e98fc9e8a62c | root@cluster-02 | Sep 3, 2020 at 9:22pm (UTC)  | Sep 1, 2030 at 9:22pm (UTC)  |
+--------------+---------------+------------------------------+------------------------------+
| e19623e335ba | root@cluster-00 | Sep 3, 2020 at 8:07pm (UTC)  | Sep 1, 2030 at 8:07pm (UTC)  |
+--------------+---------------+------------------------------+------------------------------+
| f096a7cbb6a3 | root@cluster-15 | Oct 12, 2020 at 4:22pm (UTC) | Oct 10, 2030 at 4:22pm (UTC) |
+--------------+---------------+------------------------------+------------------------------+
| fad95ad8c988 | root@cluster-14 | Oct 12, 2020 at 4:22pm (UTC) | Oct 10, 2030 at 4:22pm (UTC) |
+--------------+---------------+------------------------------+------------------------------+
| fc960c95e4ea | root@cluster-16 | Jun 1, 2021 at 3:53pm (UTC)  | May 30, 2031 at 3:53pm (UTC) |
+--------------+---------------+------------------------------+------------------------------+
| fcecccb555d2 | root@cluster-04 | Sep 3, 2020 at 9:22pm (UTC)  | Sep 1, 2030 at 9:22pm (UTC)  |
+--------------+---------------+------------------------------+------------------------------+

Thanks. There does appear to be an issue indeed. The cluster certificate upgrade completes successfully, but the heartbeat process doesn't seem to restart again (unlike when upgrading from 4.13 to 4.14). Perhaps there is some difference between the way the heartbeats are initiated in the LTS branch.

any idea how to fix this? as this is a productive cluster, this is pretty bad for me - i had to update because i added two new servers and snap removed the older images - i was only able to pick 4.13 or 4.14.

@ninjuan11 for future reference you can pick the LTS branch when installing LXD using sudo snap install lxd --channel=4.0/stable. See https://discuss.linuxcontainers.org/t/managing-the-lxd-snap/8178 for more info around that.
As for fixing it, let me see if reloading LXD fixes...one moment please.

@ninjuan11 also can you still run lxc ls btw?

yes lxc ls works, but only local vms are visible, rest is error.

@ninjuan11 ok good so the dqlite cluster (and by extension the certificate trust store) is working OK, so it appears to be just the lack of heartbeats that is the issue.

@ninjuan11 I just tried the upgrade from 4.0.6 to 4.14 again and it succeeded this time, suggesting the issue may be if the time between upgrading the members was far apart the cluster may have become out of sync. Can you have a look at the logs on your other members and see when they got upgrade to 4.14 please, and how long apart that happened?

one member took a little longer to upgrade - about 10 min - but all members are on 4.14 right now.

@ninjuan11 do you see any database is locked in the logs?

i dont see anything regarding locked database in the logfiles.
i remember seeing in a logfile an error like "unable to inform server that database has been upgraded" (nopt exact wording) but i cant find it anymroe

@ninjuan11 in my experiments I found that I could do sudo systemctl reload snap.lxd.daemon on all the nodes except one of them which hung in the logs with an error about locking. I think if you try and reload all of the members and then kill -9 the lxd process of any that won't reload cleanly then that should fix it (it has on my recreation), as getting lxd to restart allows the heartbeat process to begin again.
As for the cause I'll need to look into this in more detail.

i dumped all logfiles right now, seems like i misstyped the regular expression before - i have multiple ones with different locks
grep -e "database is locked" log*
log03:t=2021-06-01T19:20:35+0200 lvl=dbug msg="Database error: protocol.Error{Code:5, Message:\"database is locked\"}"
log03:t=2021-06-01T19:20:35+0200 lvl=dbug msg="Retry failed db interaction (database is locked)"
log04:t=2021-06-01T19:20:35+0200 lvl=dbug msg="Database error: protocol.Error{Code:5, Message:\"database is locked\"}"
log04:t=2021-06-01T19:20:35+0200 lvl=dbug msg="Retry failed db interaction (database is locked)"
log09:t=2021-06-01T19:20:35+0200 lvl=dbug msg="Database error: &errors.errorString{s:\"insert or replaced row: database is locked\"}"
log09:t=2021-06-01T19:20:35+0200 lvl=dbug msg="Retry failed db interaction (insert or replaced row: database is locked)"
log11:t=2021-06-01T19:20:34+0200 lvl=dbug msg="Database error: failed to update node version info: database is locked"
log11:t=2021-06-01T19:20:34+0200 lvl=dbug msg="Retry failed db interaction (failed to update node version info: database is locked)"

i reloaded all except machine 11 - didnt help :-(

@ninjuan11 and did you kill -9 lxd process and restart lxd on that machine 11?

@ninjuan11 basically you need to restart LXD, so you need to kill and restart any of the ones that wont reload due to be locked.

ah misread the instruction, thought i should kill only these who dont reload correctly. i killed it and issued a systemctl restart snap.lxd.daemon - but it hangs.

@ninjuan11 it looks like you have a few that are locked so need to kill and restart  it on each of them.

i did it on every machine, all are hanging on the restart command.

it seems like now i have another bug to report. because i restarted the machines, new leaders got elected on the last 3 in the cluster. the restarted machines gave up after checking the first 10.

after i tried to restart one of the leaders:
t=2021-06-01T18:56:47+0200 lvl=info msg="Done pruning expired instance backups"
t=2021-06-01T18:56:47+0200 lvl=info msg="Done updating images"
t=2021-06-01T19:20:33+0200 lvl=info msg="Received 'terminated signal', waiting for all operations to finish"
t=2021-06-01T19:20:34+0200 lvl=info msg="All running operations finished, shutting down"
t=2021-06-01T19:20:38+0200 lvl=eror msg="Error updating" err="Failed to check if this node is out-of-date: failed to begin transaction: call exec-sql (budget 0s): receive: header: EOF"
t=2021-06-01T19:20:43+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T19:20:43+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T19:20:43+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T19:20:43+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T19:20:43+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T19:20:43+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T19:20:43+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T19:20:43+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T19:20:43+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T19:20:43+0200 lvl=warn msg="Dqlite: attempt 0: server <IP_ADDRESS>:9443: no known leader"
t=2021-06-01T19:32:21+0200 lvl=warn msg="Could not handover member's responsibilities: Post \"https://<IP_ADDRESS>:9443/internal/cluster/assign\": EOF"
t=2021-06-01T19:32:21+0200 lvl=info msg="Starting shutdown sequence"
t=2021-06-01T19:32:23+0200 lvl=warn msg="Give up waiting to get number of running containers"
t=2021-06-01T19:32:23+0200 lvl=info msg="Closing the database"
t=2021-06-01T19:32:23+0200 lvl=warn msg="Failed to get number of running containers: failed to begin transaction: sql: database is closed"
t=2021-06-01T19:32:23+0200 lvl=eror msg="Error updating" err="Failed to check if this node is out-of-date: failed to begin transaction: sql: database is closed"
t=2021-06-01T19:32:23+0200 lvl=eror msg="Error updating" err="Failed to check if this node is out-of-date: failed to begin transaction: sql: database is closed"
t=2021-06-01T19:32:23+0200 lvl=eror msg="Error updating" err="Failed to check if this node is out-of-date: failed to begin transaction: sql: database is closed"
t=2021-06-01T19:32:23+0200 lvl=eror msg="Error updating" err="Failed to check if this node is out-of-date: failed to begin transaction: sql: database is closed"
t=2021-06-01T19:32:23+0200 lvl=eror msg="Error updating" err="Failed to check if this node is out-of-date: failed to begin transaction: sql: database is closed"
t=2021-06-01T19:32:23+0200 lvl=info msg="Stop database gateway"
t=2021-06-01T19:32:23+0200 lvl=eror msg="Error updating" err="Failed to check if this node is out-of-date: failed to begin transaction: sql: database is closed"
t=2021-06-01T19:32:23+0200 lvl=eror msg="Error updating" err="Failed to check if this node is out-of-date: failed to begin transaction: sql: database is closed"
t=2021-06-01T19:32:23+0200 lvl=eror msg="Error updating" err="Failed to check if this node is out-of-date: failed to begin transaction: sql: database is closed"
t=2021-06-01T19:32:23+0200 lvl=eror msg="Error updating" err="Failed to check if this node is out-of-date: failed to begin transaction: sql: database is closed"

this killed everything on all machines...

@ninjuan11 but have you killed that lxd process using kill -9 <process id> - its clear the DB is locked and you need to forceably kill the process so it can restart. Doing systemctl won't work.

yes, i killed them with kill -9.

@ninjuan11 can you try this, shutdown all of the lxd processes using lxd shutdown and then on one of the DB members run
lxd recover-from-quorum-loss

Then start that lxd member up and check it has started cleanly and that your instances are listed.
Then start each member one by one after that.

i just sent the command (lxd shutdown) to all members of the cluster, takes very long to complete... i'll post here when i finished the task.

@ninjuan11 ok @stgraber has just suggested that recover-from-quorum-loss isn't whats needed.

i'm very concerned that i'm having a data loss right now. last backup is a few weeks old, would be terrible.

@ninjuan11 so I think we need to be clear about this. Have you stopped entirely (i.e you can confirmed no lxd process running) on all nodes and then started a single one up and check it starts cleanly?

its still in the task of stopping. lxd shutdown takes horribly long.

@ninjuan11 you probably need to do a kill -9 as it'll be hanging trying to hand over the role. You need to ensure they have all been entirely stopped before starting one up.

i got zombies.
231336 ?         I'm not sure what to do now.
You needed to shutdown everything and restart because it doesn't look like you achieved that earlier

sorry i was a little bit puzzled what to do now, but it seems like the command was not triggered because it tried to attach to the zombie.

seems like snap startet lxd and ignored systemctl disable. the cluster found all nodes after a good old reboot, but lxc ls is not working anymore
lxc cluster list
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
|   NAME   |            URL            | DATABASE | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-00 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        | master      | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-01 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-02 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-03 | https://<IP_ADDRESS>:9443 | YES      | x86_64       | default        |             | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-04 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-05 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-06 | https://<IP_ADDRESS>:9443 | YES      | x86_64       | default        |             | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-07 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-08 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-09 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-10 | https://<IP_ADDRESS>:9443 | YES      | x86_64       | default        |             | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-11 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-12 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-13 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-14 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| cluster-15 | https://<IP_ADDRESS>:9443 | NO       | x86_64       | default        |             | ONLINE | Fully operational |
+----------+---------------------------+----------+--------------+----------------+-------------+--------+-------------------+

A few simple things to check without getting too deep into things:

Make sure each server is able to reach all the others (curl -k -s https://172.16.0.X:8443 to every one of the other 14 IPs)
Make sure all servers have an accurate clock (they should all be within a 2s margin)

If that's all good, then you can try:

lxc info --resources to confirm local API is functional (doesn't use DB)
lxc info --resources --target OTHER-SERVER to check API access to each of the other servers

Then on each server, run lxd sql global "SELECT COUNT(id) FROM nodes" to check whether the global database is working.
If that still looks fine, then run lxc list --fast, if some are in ERROR, then run lxc info NAME on one of those.

i restarted the whole cluster very very slowly.. it worked .. for a few minutes. everything was available with all machines in.
a few minutes later a lot of the virtual machines are on error - but only if i use lxc list. with lxc list --fast everything is fine.

How many instances do you have and how many snapshots do they typically have?
CEPH clusters that haven't been tweaked can be extremely slow at getting snapshot size data (rbd du) which could have you hit some timeouts.

i have about 30 vms on 16 hosts with 1-2 snapshots each. worked flawless before

Yeah but you upgraded from 4.0.6 to 4.14 which added a lot of features, including the fact that snapshot sizes are now retrieved from ceph.

If lxc list --fast works fine but lxc list doesn't and lxc info itself is slow when querying snapshots, the problem is almost certainly your ceph cluster being very slow at retrieving snapshot sizes.
You can likely confirm that by just looking for ceph du processes showing up on all your systems while running that lxc list.
You could try to get an idea of what's going on by running just lxc query /1.0/instances/NAME/snapshots/SNAPNAME (picking one of your instances and one of its snapshots). That should get you an idea of what it takes to get the size data of just one snapshot and if things fail, hopefully get you a clearer error.

i didnt have any performance issues before with ceph, so this is really a drawback for me. i can indeed see the rbd du process when trying to do a lxc query on snapshots. how can i disable this feature? i really dont need to see the sizes of each snapshot everytime i do a list command.

Glad to hear that the LXD cluster is back up and running after a fresh restart, I will look into why the upgrade doesn't always work properly. Although it sounds like there is a different ceph issue now unrelated to the original issue.

it cant be the ceph. when issuing the rbd du command by hand it willl return the value in less than 1 second. still issuing the lxc query snapshot command will never return a value.

@ninjuan11 it can be quite slow as it will issue one of those commands for each instance and each snapshot. Do you see any errors now in your debug log?

it is not issuing this command on each snapshot right now, i issued time lxc query /1.0/instances/con-00/snapshots/running --verbose --debug - so i doubt it has something to do with ceph right now.

@ninjuan11 but to confirm the output of lxc cluster ls - the original problem is resolved now right? The heartbeats are going again?

yes, its working again, but the bug is still a problem as this should never happen in the first place :-/
the problem was indeed that one of the 16 servers did not upgrade in a timely manner (high load), which resulted in this behaviour. i would suggest the script should retry after a few minutes instead of giving up completly and lock up.
i'll create a new ticket regarding the other problem.

@ninjuan11 thanks, yes I agree, I am actively working on fixing the original issue. Thanks for opening another issue regarding the ceph performance problem.

Can you check ps fauxww | grep rbd on the server running that instance while running the lxc query to see if LXD is running those queries?

@stgraber seems so: https://github.com/lxc/lxd/issues/8834#issuecomment-852434644
"i can indeed see the rbd du process when trying to do a lxc query on snapshots."

yes, the queries are spawned and will stay forever.

Are you using ceph.external=true with the snap? It could be some issue where your ceph cluster needs a more recent version of the client than the one in the snap.

no i'm using the internal, had massive problems with external, as it was unable to connect

i just switched to external - good news: external is now working (issue was in version 3 of lxd, so maybe this has been fixed)
still - the same problem

An explanation for this issue is here: https://github.com/lxc/lxd/pull/8839#discussion_r643810346


Post v0.16.0 release updates

Description of the change:
post v0.16.0 release updates
Motivation for the change:
part of release process

Please hold off on restarting the travis build for this as we're trying to make way for the release tag CI to run.


Too slow while fetching @llvm-raw repos while building tensorflow from Source

Issue type
Build/Install
Have you reproduced the bug with TensorFlow Nightly?
No
Source
source
TensorFlow version
2.16.1
Custom code
No
OS platform and distribution
ubuntu 22.04.1 on aarch64
Mobile device
No response
Python version
python 3.10.12
Bazel version
7.1.1
GCC/compiler version
clang 17.0.2
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
It is taking too much time to fetch repo llvm-raw while building tensorflow from source on aarch64 machine with 32 cores. I have tried using all possible number of cores, from low to high. using  ex: --local_cpu_resources=4. The process gets upto this point
Loading:      Fetching repository @llvm-raw; starting 964s     Fetching ...c40c12e4c7/external/llvm-raw; Extracting 8697bbe2d4aed109520e83c6beab52196ec5b702.tar.gz 962s
and continues like this.
Please help me to solve this issue.
Commands used :
export ONEDNN_VERBOSE=1
export TF_ENABLE_ONEDNN_OPTS=1
export CC="/usr/local/bin/clang"
export CXX="/usr/local/bin/clang++"
export TF_PYTHON_VERSION=3.10
bazel build -s --config=mkl_aarch64 --features=-layering_check --copt=-O3 --copt=-march=armv8-a+sve --copt=-msve-vector-bits=256  --local_cpu_resources=16 //tensorflow/tools/pip_package:build_pip_package --verbose_failures
Standalone code to reproduce the issue
Commands used :
`export ONEDNN_VERBOSE=1`
`export TF_ENABLE_ONEDNN_OPTS=1`
`export CC="/usr/local/bin/clang"`
`export CXX="/usr/local/bin/clang++"`
`export TF_PYTHON_VERSION=3.10`
` bazel build -s --config=mkl_aarch64 --features=-layering_check --copt=-O3 --copt=-march=armv8-a+sve --copt=-msve-vector-bits=256  --local_cpu_resources=16 //tensorflow/tools/pip_package:build_pip_package --verbose_failures `

Relevant log output
`Loading:
    Fetching repository @llvm-raw; starting 964s
    Fetching ...c40c12e4c7/external/llvm-raw; Extracting 8697bbe2d4aed109520e83c6beab52196ec5b702.tar.gz 962s`

@vishwascm,
Could you please confirm whether the build is failing due to the above issue or the build continued? and also it might be due to the vpn/network you are trying to use. Also try to clear the cache by calling bazel clean –expunge
Command prompt must be Run as Admin for the bazel build of TensorFlow to be successful.

bazel build --override_repository="llvm-raw=${LLVM_SRC}" \
  -c opt tensorflow/compiler/mlir:tf-opt

As per Bazel documentation, the target patterns starting with // are resolved relative to the current workspace. The target patterns that do not begin with // are resolved relative to the current working directory.
https://www.tensorflow.org/install/source#gpu
Thank you!

Hi @tilakrayal build will continue but only after around 25 mins of fetching llvm-raw repo. And later the build fails, the reason I think is it is downloading and building clang-17. Before building I have specifically mentioned path to used clang 16, even then it is not using clang16. How to solve this issue and use clang16 to build because clang 17 has given build fails for others also.

@vishwascm,
Could you please provide the complete error log which helps us to debug/analyse the issue in an effective way. Thank you!

@tilakrayal when I use clang 16,  it will still take lot of time fetching llvm-raw repo (around 30 mins), but after that build will be successful.
The error i got when using clang 17 was :
ERROR: /home/vishwas/tensorflow/tensorflow/python/platform/BUILD:388:17: Linking tensorflow/python/platform/_pywrap_tf2.so failed: (Exit 1): clang-17 failed: error executing command (from target //tensorflow/python/platform:_pywrap_tf2.so)
(cd /home/vishwas/.cache/bazel/_bazel_vishwas/e4f9045e13254a05e1721bc40c12e4c7/execroot/org_tensorflow &&
exec env -
CLANG_COMPILER_PATH=/usr/bin/clang-17
PATH=/home/vishwas/.cache/bazelisk/downloads/sha256/7f7d2fe44570403c5c0c221259b5afc45f0a4176eeb0440bbaffbce76872599d/bin:/home/vishwas/venv_dir/tf_py311_original/bin:/home/vishwas/.vscode-server/cli/servers/Stable-863d2581ecda6849923a2118d93a088b0745d9d6/server/bin/remote-cli:/home/vishwas/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
PWD=/proc/self/cwd
PYTHON_BIN_PATH=/home/vishwas/venv_dir/tf216_perf23/bin/python3
PYTHON_LIB_PATH=/home/vishwas/venv_dir/tf216_perf23/lib/python3.10/site-packages
TF2_BEHAVIOR=1
/usr/bin/clang-17 @bazel-out/aarch64-opt/bin/tensorflow/python/platform/_pywrap_tf2.so-2.params)
Configuration: b7a52e65e2dd57fe1977152f59b9f737f0a4f8ef69a6d625a69816bca0a9acef
Execution platform: @local_execution_config_platform//:platform
ld.lld: error: version script assignment of 'global' to symbol 'init_pywrap_tf2' failed: symbol not defined
ld.lld: error: version script assignment of 'global' to symbol 'init__pywrap_tf2' failed: symbol not defined
clang-17: error: linker command failed with exit code 1 (use -v to see invocation)
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 4290.793s, Critical Path: 160.70s

@tilakrayal when I use clang 16,  it will still take lot of time fetching llvm-raw repo (around 30 mins), but after that build will be successful.
The error i got when using clang 17 was :
ERROR: /home/vishwas/tensorflow/tensorflow/python/platform/BUILD:388:17: Linking tensorflow/python/platform/_pywrap_tf2.so failed: (Exit 1): clang-17 failed: error executing command (from target //tensorflow/python/platform:_pywrap_tf2.so)    (cd /home/vishwas/.cache/bazel/_bazel_vishwas/e4f9045e13254a05e1721bc40c12e4c7/execroot/org_tensorflow && \   exec env - \     CLANG_COMPILER_PATH=/usr/bin/clang-17 \     PATH=/home/vishwas/.cache/bazelisk/downloads/sha256/7f7d2fe44570403c5c0c221259b5afc45f0a4176eeb0440bbaffbce76872599d/bin:/home/vishwas/venv_dir/tf_py311_original/bin:/home/vishwas/.vscode-server/cli/servers/Stable-863d2581ecda6849923a2118d93a088b0745d9d6/server/bin/remote-cli:/home/vishwas/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \     PWD=/proc/self/cwd \     PYTHON_BIN_PATH=/home/vishwas/venv_dir/tf216_perf23/bin/python3 \     PYTHON_LIB_PATH=/home/vishwas/venv_dir/tf216_perf23/lib/python3.10/site-packages \     TF2_BEHAVIOR=1 \   /usr/bin/clang-17 @bazel-out/aarch64-opt/bin/tensorflow/python/platform/_pywrap_tf2.so-2.params)  Configuration: b7a52e65e2dd57fe1977152f59b9f737f0a4f8ef69a6d625a69816bca0a9acef  Execution platform: @local_execution_config_platform//:platform ld.lld: error: version script assignment of 'global' to symbol 'init_pywrap_tf2' failed: symbol not defined ld.lld: error: version script assignment of 'global' to symbol 'init__pywrap_tf2' failed: symbol not defined clang-17: error: linker command failed with exit code 1 (use -v to see invocation) Target //tensorflow/tools/pip_package:build_pip_package failed to build INFO: Elapsed time: 4290.793s, Critical Path: 160.70s

@vishwascm,
I suspect the issue is that your tests are looking for a python3 binary. Bazel runs the tests in isolation, you need to make sure the PATH variable used there contains the path to Python & the Bazel you are trying . Also could you test with the python3.11 and let us know if you are facing  the same. Thank you!


Update README

Included website in examples section. TY!

Added my website to the showcase manifest.Thanks!

wow, it looks terribly awesome!
Could be possible create a PR with your cover changes? I want to apply in the codebase 😄

I'd be happy to. It was a joy working in your codebase. Let me clean it up and I'll create the PR. Thanks again!

@jmiramant if you need any kind of help contact me using this issue or my email in my github account!


Does Not Run with Bun

Hi,
I use Bun for my project, and when I try to use "bun x eslint --inspect-config" or "bun x --bun eslint --inspect-config", it doesn't work and shows this error message:
$ eslint --inspect-config
You can also run this command directly using 'npx @eslint/config-inspector@latest' in the same directory as your configuration file.
'npx' is not recognized as an internal or external command,
operable program or batch file.

Please support Bun.

All the info you need is there in the message that’s printed?
As it say: eslint --inspect-config is simply a shortcut within eslint that runs npx @eslint/config-inspector
Since you’re on Bun the npx command fails. Do what the command recommends and do the Bun equivalent of npx @eslint/config-inspector directly instead of using the shortcut.
If you do want to file an issue about adding support for Bun in the shortcut, then you would have to file it towards the main ESLint project, not the config inspector.

I get this error with all package managers in my project managed with bun

@antfu but it's not the Bun bug. I'm getting the same issue without Bun on Node v23

@Razunter @antfu Node 23, 22, 20 should all be tested in the CI but currently only latest LTS is. I suggest a PR to add the other versions to the text matrix


Improve chart layout/display for hard-drive usage, particularly when lots of drives

As reported on the Discord:

More from Discord:

Improved in upcoming v0.8.0 release.  Example shown were I artificially limited to only 3 chart columns to demonstrate.  Actual limit will be higher, but not unlimited as before.

Machinaris version 0.8.0 has now been released with support for Chia 1.4.0.  Thanks!

Re-opening as other users have complained about the missing drives.  One suggested this:


flutter: Another exception was thrown: A VideoPlayerController was used after being disposed in iOS Platform

Is there an existing issue for this?

[X] I have searched the existing issues
[X] I have read the guide to filing a bug

Steps to reproduce
I got the above error while playing videos one by one in a horizontal list by adding a listener to get the progress of the current video.
Expected results
Videos should play smoothly without getting an error
Actual results
Getting exceptions regarding video controller while playing multiple videos one by one inside list.
Code sample
Code sample
import 'dart:async';

import 'package:cached_network_image/cached_network_image.dart';
import 'package:flutter/material.dart';
import 'package:get/get.dart';
import 'package:hotake_mobile_prod/controllers/sparks/latest_sparks_controller.dart';
import 'package:hotake_mobile_prod/utils/constants.dart';
import 'package:video_player/video_player.dart';

class LatestVideoPlayer extends StatefulWidget {
  final String url;
  final String thumbnailImg;
  final int index;
  final int totalLength;
  final Function scrollToLSItem;

  const LatestVideoPlayer(
      {Key? key,
      required this.url,
      required this.index,
      required this.thumbnailImg,
      required this.totalLength,
      required this.scrollToLSItem})
      : super(key: key);
  @override
  LatestVideoPlayerState createState() => LatestVideoPlayerState();
}

class LatestVideoPlayerState extends State<LatestVideoPlayer> {
  late VideoPlayerController videoController;
  late Future<void> _initializeVideoPlayerFuture;
  int secondsToPlay = 4;
  Timer? timer;
  LatestSparksController latestSparksController =
      Get.find<LatestSparksController>();

  @override
  void initState() {
    super.initState();
    if (mounted) {
      _clearPreviousVideoController();
    }
    printLog("VideoWidget index ${widget.index} ");
    videoController = VideoPlayerController.network(widget.url,
        videoPlayerOptions: VideoPlayerOptions(mixWithOthers: true));

    _initializeVideoPlayerFuture = videoController.initialize().then((_) {
      // Ensure the first frame is shown after the video is initialized, even before the play button has been pressed.
      timer = Timer(const Duration(milliseconds: 1500), () {
        latestSparksController.setVideoPlayNotifier(true);

        try {
          videoController.addListener(checkVideo);
        } catch (e) {
          printLog("addListener ${e.toString()}");
        }
      });
    });
    videoController.setVolume(0.0);
    videoController.play();
    videoController.setLooping(true);
  }

  void checkVideo() {

    if (videoController.value.duration.inSeconds < secondsToPlay) {
      if (videoController.value.isPlaying &&
          videoController.value.position.inSeconds ==
              videoController.value.duration.inSeconds) {

        videoController.pause();
        videoController.seekTo(const Duration(seconds: 0));

        widget.scrollToLSItem();
      }
    } else {
      if (videoController.value.isPlaying &&
          videoController.value.position.inSeconds >= secondsToPlay) {

        videoController.pause();
        videoController.seekTo(const Duration(seconds: 0));

        widget.scrollToLSItem();
      }
    }
  }

  @override
  void didUpdateWidget(LatestVideoPlayer oldWidget) {

    videoController.setVolume(0.0);
    videoController.play();
    videoController.setLooping(true);

    super.didUpdateWidget(oldWidget);
  }

  @override
  void dispose() {
    if (!mounted) {
      latestSparksController.setVideoPlayNotifier(false);
    }

    timer?.cancel();

    tempPrevLatestVideoControllerList.add(videoController);
    _clearPreviousVideoController();

    super.dispose();
  }

  void _clearPreviousVideoController() {

    if (tempPrevLatestVideoControllerList.isNotEmpty) {
      var tempVideoControllerList = [];
      tempVideoControllerList.addAll(tempPrevLatestVideoControllerList);
      for (var elementVideoController in tempVideoControllerList) {
        elementVideoController?.dispose().then((_) {
          elementVideoController.removeListener(checkVideo);
        });
        tempPrevLatestVideoControllerList.remove(elementVideoController);
      }
    }

  }

  @override
  Widget build(BuildContext context) {
    return FutureBuilder(
        future: _initializeVideoPlayerFuture,
        builder: (context, snapshot) {
          return Obx(() => latestSparksController.videoPlayNotifier.value &&
                  snapshot.connectionState == ConnectionState.done
              ? VideoPlayer(videoController)
              : CachedNetworkImage(
                  imageUrl: widget.thumbnailImg,
                  errorWidget: (context, url, error) => const Icon(Icons.error),
                  imageBuilder: (context, imageProvider) => Container(
                    width: double.infinity,
                    height: double.infinity,
                    decoration: BoxDecoration(
                      image: DecorationImage(
                        image: imageProvider,
                        fit: BoxFit.fill,
                      ),
                    ),
                  ),
                ));
        });
  }
}

Screenshots or Video

Screenshots / Video demonstration
[Upload media here]

Logs
Logs
flutter: scrollToItem LS selectedIndex 2
flutter: VideoWidget index 2
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 3
flutter: VideoWidget index 3
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 4
flutter: VideoWidget index 4
flutter: Another exception was thrown: A VideoPlayerController was used after being disposed.
flutter: Another exception was thrown: HttpException: Connection closed before full header was received, uri = https://firebasestorage.googleapis.com/v0/b/hotake-development.appspot.com/o/uploads%2Fusers%2FgKLZLVNB7LOlG1a2BLdZ%2Fprofile_pic_240X240gKLZLVNB7LOlG1a2BLdZ?alt=media&token=af08b906-a6f7-4e6f-a549-892ac89b1684
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 5
flutter: VideoWidget index 5
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 6
flutter: VideoWidget index 6
flutter: Another exception was thrown: A VideoPlayerController was used after being disposed.
flutter: fetchLatestSparksData sparksList 20
flutter: fetchLatestSparksData sparksList 20
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 7
flutter: VideoWidget index 7
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 8
flutter: VideoWidget index 8
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 9
flutter: VideoWidget index 9
flutter: Another exception was thrown: A VideoPlayerController was used after being disposed.
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 10
flutter: VideoWidget index 10
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 11
flutter: VideoWidget index 11
flutter: Another exception was thrown: A VideoPlayerController was used after being disposed.
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 12
flutter: VideoWidget index 12
flutter: Another exception was thrown: A VideoPlayerController was used after being disposed.
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 13
flutter: VideoWidget index 13
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 14
flutter: VideoWidget index 14
flutter: Another exception was thrown: A VideoPlayerController was used after being disposed.
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 15
flutter: VideoWidget index 15
flutter: Another exception was thrown: A VideoPlayerController was used after being disposed.
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 16
flutter: VideoWidget index 16
flutter: fetchLatestSparksData sparksList 30
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 17
flutter: VideoWidget index 17
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 18
flutter: VideoWidget index 18
flutter: Another exception was thrown: A VideoPlayerController was used after being disposed.
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 19
flutter: VideoWidget index 19
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 20
flutter: VideoWidget index 20
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 21
flutter: VideoWidget index 21
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 22
flutter: VideoWidget index 22
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 23
flutter: VideoWidget index 23
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 24
flutter: VideoWidget index 24
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 25
flutter: VideoWidget index 25
flutter: Another exception was thrown: A VideoPlayerController was used after being disposed.
flutter: Print after 1 seconds
flutter: VideoWidget index 19
flutter: VideoWidget index 28
flutter: VideoWidget index 16
flutter: VideoWidget index 21
flutter: VideoWidget index 18
flutter: Print after 1 seconds
flutter: VideoWidget index 24
flutter: Print after 1 seconds
flutter: Print after 1 seconds
flutter: scrollToItem LS selectedIndex 25
flutter: VideoWidget index 25
flutter: Another exception was thrown: A VideoPlayerController was used after being disposed.
flutter: Print after 1 seconds

Flutter Doctor output
Doctor output
[✓] Flutter (Channel stable, 3.13.4, on macOS 13.4.1 22F770820d darwin-x64, locale en-IN)
    • Flutter version 3.13.4 on channel stable at /Users/computer/Documents/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 367f9ea16b (12 days ago), 2023-09-12 23:27:53 -0500
    • Engine revision 9064459a8b
    • Dart version 3.1.2
    • DevTools version 2.25.0

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.2)
    • Android SDK at /Users/computer/Library/Android/sdk
    • Platform android-33, build-tools 33.0.2
    • Java binary at: /Applications/Android Studio 2022.1.1.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.15+0-b2043.56-8887301)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.3.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14E300c
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.1)
    • Android Studio at /Applications/Android Studio 2021.1.1.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.11+0-b60-7590822)

[✓] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio 4.0.1.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] Android Studio (version 2022.1)
    • Android Studio at /Applications/Android Studio 2022.1.1.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.15+0-b2043.56-8887301)

[✓] VS Code (version 1.82.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.72.0

[✓] Connected device (4 available)
    • Dipak (mobile)    • 00008020-000C10C90CF1002E            • ios            • iOS 16.4.1 20E252
    • iOS 16.4 (mobile) • D502DC9A-7628-4B50-8F0C-D707CA7D7400 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-16-4 (simulator)
    • macOS (desktop)   • macos                                • darwin-x64     • macOS 13.4.1 22F770820d darwin-x64
    • Chrome (web)      • chrome                               • web-javascript • Google Chrome 116.0.5845.187

[✓] Network resources
    • All expected network resources are available.

@BrindaRathodB
Can you provide complete error log in properly formatted text ?
Also, the code sample you provided contains third party package code implementation. In order to properly address the issue, we need a minimal runnable reproducible code sample without using any third party implementation.

@darshankawar I have updated the error log above, and we are facing this issue while auto-playing the videos in the horizontal list one by one after every 4 seconds for that we need that third-party package.

@BrindaRathodB
You'll need to narrow down the issue with respect to video_player only in order to know if the exception is triggered using it or coming from other package's implementation.
The error thrown Another exception was thrown: A VideoPlayerController was used after being disposed. generally indicates the implementation of dispose() method could be wrong in which you are passing the videoplayer controller, so you also need to check if you are disposing the controller properly or not.

@darshankawar Yes we are disposing of it correctly also it works completely fine with Android but in iOS only we are facing this issue, using the same third-party package.

@BrindaRathodB
I am still not convinced whether the issue is occuring due to Flutter's first party package video_player but could be occuring due to other community packages you are using.
Since you are using the videos in a list, try to use cacheExtent on the list builder and set it to a large value and check if it helps or not. You may need to explore a bit more on it since it involves other packages. If you are able to narrow down the issue just to video_player plugin, then we can make the issue actionable.


Short solution needed: "Get all values in Redis" (redis)

Please help us write most modern and shortest code solution for this issue:
Get all values in Redis (technology: redis)
Fast way
Just write the code solution in the comments.
Prefered way

Create pull request with a new code file inside inbox folder.
Don't forget to use comments to make solution explained.
Link to this issue in comments of pull request.

Get all keys values in Redis


Cannot scroll vertical on chrome on android

My horizontal carousel doesn't allow me to scroll the page vertically when I try to scroll the page touching on the carousel.

removing touch-action: none did not work for me. I can't figure out why I cant scroll past the slider on mobile devices. My console doesnt throw any errors or anything. Really weird behavior.

adding on img {
z-index:-1;
position:relative;
}
worked for me.


feat: add budoux parser options

add option to set parser language

I think the format is not suitable for your preference, so please format it by your editor!

Thanks for the PR!
I'm planning to install biome later, so I'll keep it as is for now!


Alternatives

I've found your repo searching in google how yo hack waterrower S4. I've seen your code and it seems good and interesting.
But after searching how to update S4 firmware I've discovered this site:
http://waterrowerservice.com/software-drivers/#
There are some apps to achive the same goal. The only difference is that they do not use bluetooth.

Their software requires firmware 2.1 and to "upgrade" you need to pay $150 for a new S4. In my opinion a ripoff!


[iOS & Android][rendering][IsVisible toggling not rendering now visible element correctly ]

Platform
What platform is your issue or question related to? (Delete other platforms).

Android
iOS

Author
Sending to Teams iOS version 2.0.14 with and without developers preview. As well as Android, latest version.
Version of SDK:
JS - v1.6.0
Details
We use a collapsed view of our cards to show our users only the most relevant information then allow them to ToggleVisibility on additional information. This worked perfectly fine until recently. Now it appears the elements that were initially set to isVisible:false do not get rendered correctly when toggled to true.
We tested this on new cards as well as cards that were previously sent by the system to the mobile device. This is happening in both v1.0 and verison 1.2 of the schema
Steps to reproduce:

go to: https://amdesigner.azurewebsites.net/
select the Expense Approval sample
copy the JSON payload
use the App Studio within Teams to send yourself this card
observe expanding sections load correctly on the Desktop - ok
view same card on mobile app (just checked again on ios13.5.1) and notice that when expanding hidden panels they over lay with other ones, overall height is not adjusted - KO

@VikrantSingh01 / @anbah34 / @Aditya0603 have you guys seen reports of this issue wiht Teams mobile again?

@shalinijoshi19 Any update on this issue? thank you

@icatchx22i
This card won't load as it is for iOS and Android because parsing the payload will fail due to id collisions.
there are a few IDs that share the IDs.
c.c @jonmill

closing this issue as it is being tracked by #6015


Error when initializing NodeMediaServer Constructor

I get "Uncaught TypeError: Cannot read property 'prototype' of undefined"
when doing const NodeMediaServer = require("node-media-server");
Here is the traceback:
 at eval (webpack-internal:///./node_modules/express/lib/response.js:42)     at Object../node_modules/express/lib/response.js (app.js:5835)     at __webpack_require__ (app.js:727)     at fn (app.js:101)     at eval (webpack-internal:///./node_modules/express/lib/express.js:22)     at Object../node_modules/express/lib/express.js (app.js:5787)     at __webpack_require__ (app.js:727)     at fn (app.js:101)     at eval (webpack-internal:///./node_modules/express/index.js:11)     at Object../node_modules/express/index.js (app.js:5752)

at eval (webpack-internal:///./node_modules/express/lib/response.js:42): var res = Object.create(http.ServerResponse.prototype)
I've only installed the packages and tried to use it but I get this error.

You're having this error , just because node-media-server returns three different class, NodeMediaServer is one of those.
You can try this const { NodeMediaServer } = require("node-media-server");


Disable input of match.

Require #1137 to deduce properties by special visitor for match.
Close #1166
TODO now the properties deduce visitor is designed for ngql, so I need a properties deduce visitor for match which already in #1137

Move to https://github.com/vesoft-inc/nebula/pull/2537


Update sites.yml

Cleanup my site -  devprogrammer.com site - as it's no longer available.

This PR is:

[ ] Adding a new domain
[ ] Updating existing domain size
[ ] Changing domain name
[x] Removing existing domain from list
[ ] Website code changes (512kb.club site)
[ ] Other not listed

[ ] I used the uncompressed size of the site
[ ] I have included a link to the Cloudflare report
[ ] This site is not an ultra minimal site
[ ] The following information is filled identical to the data file

I confirm that I have read the FAQ section, particularly the two red items around minimal pages and inappropriate content and I attest that this site is neither of these things.

[x] Check to confirm

- domain:
  url:
  size:
  last_checked:

Cloudflare Report:

Please complete the template properly.


Need update to compatible  with prop-types

warning.js:33 Warning: Failed prop type: Calling PropTypes validators directly is not supported by the prop-types package. Use PropTypes.checkPropTypes() to call them. Read more at http://fb.me/use-check-prop-types

solved in version 1.0.12.
Thanks!


CI: don't use ubuntu-latest

relates to #155  but I need ASF Infra team to approve the sbt/setup-sbt so this should get the CI working in the mean time

#155 is viable now


lovr.system.isKeyDown() not working

I followed exactly what the document said but there's an error that says attempt to call field isKeyDown (is nil)
here's the code:
if lovr.system.isKeyDown("w") == true then     camy = camy - 0.01  end

This function was added to the nightly version and doesn't exist on stable
release v0.15.0.
If you want to stay on stable version, there is lovr-keyboard library which
has isDown() function.
On Tue, 6 Sept 2022, 14:34 Actravaz, @.***> wrote:

I followed exactly what the document said but there's an error that says
attempt to call field isKeyDown (is nil)
here's the code:
if lovr.system.isKeyDown("w") == true then camy = camy - 0.01 end
—
Reply to this email directly, view it on GitHub
https://github.com/bjornbytes/lovr/issues/580, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEHSSHXIHOPOCX4TPBFGHB3V4423DANCNFSM6AAAAAAQFZBLEI
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

The library worked, Thanks!


test, sriov: Configure ip addresses using cloud init network data

What this PR does / why we need it:
There is an issue with sriov test lane related to NetworkManager overrid ing the ip
configuration set by the tests using 'virtcl console'.
To fix that this PR will configure the ip address at cloud-init network data using
the "match" feature with the "macaddres attribute [1].
[1] https://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v2.html#examples
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #4432
Special notes for your reviewer:
Release note:

NONE

/test pull-kubevirt-e2e-kind-1.17-sriov

/test pull-kubevirt-e2e-kind-1.17-sriov

/test pull-kubevirt-e2e-kind-1.17-sriov

/retest

/hold sriov has to be ok
Also looks like the yaml parser is not correctly marshaling it, it's ignoring the golang tags with yaml field names.
STEP: Ethernets:
  id0:
    AcceptRA: null
    Addresses:
    - fd10:0:2::2/120
    - <IP_ADDRESS>/24
    DHCP4: null
    DHCP6: null
    DHCPIdentifier: ""
    Gateway4: ""
    Gateway6: fd10:0:2::1
    MACAddress: ""
    MTU: 0
    Match:
      MACAddress: 50:00:00:00:90:0d
    Nameservers:
      Addresses:
      - <IP_ADDRESS>
      Search:
      - default.svc.cluster.local
      - svc.cluster.local
      - cluster.local
    Routes: null
Version: 2

Can you please add that it fixes #4432 ?
also you might find this function interesting, to add to it wait for IPs exists
https://github.com/kubevirt/kubevirt/pull/4335/files#diff-ac233e621be1a1855fe741915d860d91484c7f061fc42beb213f4f9e0a756f86R1258
PR isnt merged yet

Added dependency to https://github.com/kubevirt/kubevirt/pull/4486

Split into separatre commits

Pushed new commit removing the skip from a pair of fixed PRs.

/retest

From what i understand you took here just 1 commit from #4486
As Edy suggested there, i would also suggest to either split #4486 to 2 PRs, one if them is just the 1 commit you need here,
or just cherry-pick the whole #4486 here ?

Rebased in top of https://github.com/kubevirt/kubevirt/pull/4492

Addressed @oshoval comments

Rebased on master

/retest

/test pull-kubevirt-e2e-kind-1.17-sriov

/hold cancel

/cc @alonSadan

/cc @EdDev

/cc @AlonaKaplan

/uncc kbidarkar

/uncc jerry7z

SRIOV lane is fine and tests are not skipped

force-push: Fixed linux bridge test failure

force-push: Addressed @EdDev comments

force-push: Fixing CI the set-name networkData field was missing.

force-push: Removing noise from my env.

force-push: Adressed  @EdDev comments

force-push: Fixed failing test using ests.WaitUntilVMIReady since we need virtctl console to be ready to run ping command between VMs.

force-push: Rebased and splitted in three commits, the refactoring the linux-bridge CNI test change and the SRIOV test change.

force-push: Fix error handling at cloud-init

force-push: Move networkData builder structs to the beginning of go file.

sriov lane is failing https://prow.apps.ovirt.org/view/gcs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/4448/pull-kubevirt-e2e-kind-1.17-sriov/1329721990516314114:
tests/vmi_multus_test.go:866
Unexpected error:
    <*errors.errorString | 0xc001622a50>: {
        s: "an empty namespace may not be set when a resource name is provided",
    }
    an empty namespace may not be set when a resource name is provided
occurred
tests/vmi_multus_test.go:859

and
Expected success, but got an error:
    <*fmt.wrapError | 0xc001654380>: {
        msg: "could not check interface: interface eth0 was not found in the VMI testvmishzjlz4v7xgdr2vnzb94v6vfprj77ngg6ztb9n89jxznh6z7: Can't connect to websocket (301): websocket: bad handshake\n",
        err: {
            err: "Can't connect to websocket (301): websocket: bad handshake\n",
            StatusCode: 301,
        },
    }
    could not check interface: interface eth0 was not found in the VMI testvmishzjlz4v7xgdr2vnzb94v6vfprj77ngg6ztb9n89jxznh6z7: Can't connect to websocket (301): websocket: bad handshake

force-push: Use default tests namespace at SRIOV tests

/retest

https://github.com/kubevirt/kubevirt/compare/a431109f2dd14c98d2e44910dfa0061e1cd053de..705307a308d3e91629c3dc3811c271b0257aae97: @EdDev comments
/hold
Let's wait for sriov lane.

force-push: Updating vmi at creation so vmi.Namespace can be used.

force-push:  Checking MAC address also at VMI as suggested offline by Alona.

force-push: Address @alonSadan  comments

force-push: Addressing @oshoval comments

force-push: Still using @alonSadan helpers but not retrieving interfaceName from status since we can force it to whatever we want at cloud init.

force-push: Introduce libvmi.GetInterfaceStatusByName to address last @EdDev comments, not sure we need to use Eventually there.

force-push: nits from @EdDev

/hold sriov has to be ok

SRIOV should be good now.
/hold cancel

force-push: Use HaveKey instead of Contains to fix the test.
force-push: rebase

force-push: Removed .log file

/approve
/lgtm
Really nice.

/hold
Again the issue with namespace at SRIOV lane https://prow.apps.ovirt.org/view/gcs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/4448/pull-kubevirt-e2e-kind-1.17-sriov/1331538968642064384

force-push:  vmi.Namespace is not filled in at .Create that was breaking SRIOV lane.

The change matches the approach taken in startVmi and is minimal. Raising lgtm again.
/lgtm

force-push: vmi.Namespace is not filled in at .Create that was breaking SRIOV lane.

It is filled with Create. The problem is due to startVMI that is not updating it (so the caller will get the update). :/
It is sad, but we will have to solve it in a following change.

force-push: More empty namespaces
https://prow.apps.ovirt.org/view/gcs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/4448/pull-kubevirt-e2e-kind-1.17-sriov/1331569922983071746

/hold cancel
sriov failures re not related
Can't connect to websocket (301): websocket: bad handshake\n

/lgtm

/retest

/hold
The webhook handshake issue is related to this PR.

force-push: fixed rebase after https://github.com/kubevirt/kubevirt/pull/4217 merged

force-push: Fix webhook handshake issue

/lgtm

/hold cancel
SRIOV lane is green https://prow.apps.ovirt.org/view/gcs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/4448/pull-kubevirt-e2e-kind-1.17-sriov/1331918476075339777


I get a error with the latest version

@2bc4 I get this error: [plugins.twitch][info] streamlink-ttvlol d314c5b8-master
[plugins.twitch][info] Please report issues to https://github.com/2bc4/streamlin
k-ttvlol/issues
[cli][info] Found matching plugin twitch for URL twitch.tv/mira
[plugins.twitch][info] Using playlist proxy 'https://eu.luminous.dev'
[plugins.twitch][error] Unable to open URL: https://eu.luminous.dev/live/mira?pl
ayer=twitchweb&type=any&allow_source=true&allow_audio_only=true&allow_spectre=fa
lse&fast_bread=true (request() got an unexpected keyword argument 'keywords')
error: No playable streams found on this URL: twitch.tv/mira
With the version Fix plugin args not being passed

The master branch requires git streamlink, currently with at least commit streamlink/streamlink@d314c5b8. Are you running that?

Maybe I should put that in the readme somewhere. It usually works without git streamlink but when upstream does a big change like recently it will break on normal releases.

Well that explains it.  I had a backup of the last version of this.  The new Streamlink release says it's gonna break a lot of things so I'll still with what works until either you add something I wanna try or Streamlink stabilizes

If I add something noteworthy before the next streamlink version I'll make a new release that's compatible with 5.5.1. So far it has just been refactors and log output stuff so I didn't see a reason to make a new release yet.
Anyways I'll close this now


Convert to monorepo

Convert to a monorepo with each component being published individually, for more reliable sharing and updating of components across projects using semver.

https://yarnpkg.com/lang/en/docs/workspaces/

https://gist.github.com/nolanlawson/457cdb309c9ec5b39f0d420266a9faa4

https://github.com/boennemann/alle

https://github.com/facebook/create-react-app/blob/next/packages/react-scripts/template/README.md#sharing-components-in-a-monorepo

https://github.com/facebook/create-react-app/pull/3741

https://github.com/bradfordlemley/cra-monorepo-examples

Blocked by #85

Likely going down the Jest model of using a combination of Lerna and Yarn workspaces


Typo per-create

Typo
Should be pre-create not per-create in this sentence.
"The following step enables monitoring of your AKS cluster using Azure CLI. In this example, you are not required to per-create or specify an existing workspace."

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 993779b5-188d-13ba-e4fc-9ce727c5f9f9
Version Independent ID: f81710f5-b542-fdff-522e-269d9406b4c0
Content: Monitor an Azure Kubernetes Service (AKS) cluster deployed - Azure Monitor
Content Source: articles/azure-monitor/insights/container-insights-enable-existing-clusters.md
Service: azure-monitor
GitHub Login: @bwren
Microsoft Alias: magoedte

@PCNZ Thanks for your comment! We will review and provide an update as appropriate.

@PCNZ Verified and updated the document as required. The change should go live soon (i.e., in less than 24 hours). Thanks.


Demo wont run locally

Hi.
I used the code from this official demo as the basis of some work in a project, but the drag did not work at all.
So I copied the demo cleanly into a new create-react-app project (which is what I'm using) and the drag still does not work. If I change the useGesture hook to explicitly use onDrag it sort of works. The picture scales down as I drag, but does not move until I get to a certain point and then it slides. I can see the numbers in transform() starting to go up, and as soon as I stop dragging they go down again.
I've been trying to work out what is going on for longer than I care to admit.
This is on Chrome Version 77.0.3865.90 (Official Build) (64-bit).
Here is my packages.json:
{
  "name": "swipe-test",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "lodash-es": "^4.17.15",
    "react": "^16.10.0",
    "react-dom": "^16.10.0",
    "react-spring": "^8.0.27",
    "react-use-gesture": "^6.0.1"
  },
  "devDependencies": {
    "react-scripts": "3.1.2"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": "react-app"
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  }
}

I also tried downgrading to React v 16.9.0

I've found the issue, as one often does after writing it down. ;-)
I did not check the version of react-use-gesture, and upon looking at the change log I spied breaking changes, and I have now got this fixed. The fix (for anyone else looking) is to simply to replace delta: [xDelta] with movement: [xDelta], and put the block into onDrag.

Indeed I need to work with @aleclarson and @drcmda to use the same sandboxed demos for react-spring and react-use-gesture (mine are up to date, just sayin’ 🙉

@dbismut No worries - I needed a 'bug' to help me get a better understanding of how to use the library anyway. 😺 This stuff is fantastic, team!


[Enhancement] Admin metrics thread filtering

Checklist

[X] I tested all changes and all related features with different users (student, tutor, editor, instructor, admin) on the test server https://artemistest.ase.in.tum.de.
[ ] *Server: I followed the coding and design guidelines.
[ ] *Server: I added multiple integration tests (Spring) related to the features (with a high test coverage)
[ ] *Server: I added @PreAuthorize and check the course groups for all new REST Calls (security)
[ ] *Server: I implemented the changes with a good performance and prevented too many database calls
[ ] *Server: I documented the Java code using JavaDoc style.
[X] Client: I followed the coding and design guidelines.
[X] Client: I added multiple integration tests (Jest) related to the features (with a high test coverage), while following the test guidelines.
[ ] *Client: I added authorities to all new routes and check the course groups for displaying navigation elements (links, buttons)
[X] Client: I documented the TypeScript code using JSDoc style.
[X] Client: I added multiple screenshots/screencasts of my UI changes
[X] Client: I translated all newly inserted strings into English and German.
[X] Language: I followed the guidelines for inclusive, diversity-sensitive, and appreciative language.

[ ]* not needed.
Motivation and Context

Enhance the filtering in the admin metrics page.
Description

Log in to Artemis as an admin
Go to System Administration and click on Metrics
Click on Expand button in the Threads column
The modal view should look like in the screenshot below
Play around with the buttons at the top and the filter text field to test if the filter functionality is working.

Review Progress

Code Review

[x] Review 1
[ ] Review 2

Manual Tests

[ ] Test 1
[ ] Test 2

Test Coverage

metrics-modal-threads.component.ts 89.36%
Screenshots

Thank you @imstreetspirit for the review 😊
I have incorporated all your suggestions.

Thank you for your suggestions @lschlesinger.
I applied both of them 😊

Nice idea @Malyuk-A
Now I made the filter to be case insensitive 👼