API Server hanging on raspberry pi

I have a 3 node microk8s cluster on raspberry PI running 1.21/edge - at least once daily one of the nodes will go into a Not Ready status and when I restart with microk8s stop ; microk8s start it will hang just after trying to start the API server.
inspection-report-20210521_093118.tar.gz
inspection-report-20210521_093759.tar.gz
The attached inspection reports are from one such event today which has impacted two of the three nodes.
Any ideas where to look?

Hi @CharlesGillanders. In the logs I see that the datastore (dqlite) cluster is not healthy. First, there is a period where the cluster does not reply to queries.
May 21 04:26:40 pi-node3 microk8s.daemon-kubelite[559691]: E0521 04:26:40.612941  559691 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"heimdall-5b46974d59-z2hbb.1680fa62c0e393c7", GenerateName:"", Namespace:"heimdall", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"heimdall", Name:"heimdall-5b46974d59-z2hbb", UID:"1accd027-d0f6-4b39-aa14-a1f70b2cfe5e", APIVersion:"v1", ResourceVersion:"2128206", FieldPath:""}, Reason:"NodeNotReady", Message:"Node is not ready", Source:v1.EventSource{Component:"node-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc021eb77f5d69dc7, ext:486531874504, loc:(*time.Location)(0x8aec560)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc021eb77f5d69dc7, ext:486531874504, loc:(*time.Location)(0x8aec560)}}, Count:1, Type:"Warning", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'rpc error: code = Unknown desc = exec (try: 500): database is locked' (will not retry!)
May 21 04:26:40 pi-node3 microk8s.daemon-kubelite[559691]: E0521 04:26:40.829712  559691 leaderelection.go:361] Failed to update lock: Put "https://<IP_ADDRESS>:16443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=30s": context deadline exceeded

This causes a restart and the dqlite cluster is not able to come up within 5 minutes so the restart process is triggered:
May 21 09:20:58 pi-node3 microk8s.daemon-kubelite[702459]: W0521 09:20:58.976958  702459 authentication.go:507] AnonymousAuth is not allowed with the AlwaysAllow authorizer. Resetting AnonymousAuth to false. You should use a different authorizer
May 21 09:25:58 pi-node3 systemd[1]: Stopping Service for snap application microk8s.daemon-kubelite...
May 21 09:26:07 pi-node3 microk8s.daemon-kubelite[702459]: Error: context deadline exceeded

Cat you share the output of the following on all three nodes:
cat  /var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml
ls -l  /var/snap/microk8s/current/var/kubernetes/backend/

FYI @MathieuBordere

I shutdown all three nodes and checked the cluster status according to this post https://microk8s.io/docs/ha-recovery
Only one node was in a good state for the dqlite replication sequence numbers and the two nodes that were completely off both had a zero length cluster.yaml file.
I was able to follow the instructions to restore the dqlite files but the cluster still didn't come up cleanly.
I took a chance on copying the cluster.yaml file from the one good node to the two others and that worked.
The cluster.yaml file currently looks like this on all three nodes

Address: <IP_ADDRESS>:19001
ID:<PHONE_NUMBER>608546238
Role: 0
Address: <IP_ADDRESS>:19001
ID:<PHONE_NUMBER>829213503
Role: 0
Address: <IP_ADDRESS>:19001
ID:<PHONE_NUMBER>264992893
Role: 0

Could you attach a microk8s.inspect tarball of the working cluster. I am interested to know the workload you have and try to reproduce the problem. Also, could you please share a ls -l  /var/snap/microk8s/current/var/kubernetes/backend/ as that would help us understand the size of the database.

inspection-report-20210521_111445.tar.gz
I am seeing something unexpected in the ls -l  (I don't know if it is important or not) two of the nodes have all of the files set to group microk8s but one node has some of them set to root.  I know when I tried to recover the cluster I did manually set the group for all files to root to match the spec in the FAQ but it looks like files created after that point in time one one node at least are being created with a different group.
Here's output from the unusual node
-rw------- 1 root microk8s  8378168 May 21 10:40<PHONE_NUMBER>514780-0000000006515091
-rw------- 1 root microk8s  8360600 May 21 10:41<PHONE_NUMBER>515092-0000000006515615
-rw------- 1 root microk8s  8368520 May 21 10:44<PHONE_NUMBER>515616-0000000006516021
-rw------- 1 root microk8s  8381912 May 21 10:46<PHONE_NUMBER>516022-0000000006516442
-rw------- 1 root microk8s  8378240 May 21 10:48<PHONE_NUMBER>516443-0000000006516869
-rw------- 1 root microk8s  8387288 May 21 10:51<PHONE_NUMBER>516870-0000000006517308
-rw------- 1 root microk8s  8372768 May 21 10:51<PHONE_NUMBER>517309-0000000006517887
-rw------- 1 root microk8s  8364488 May 21 10:54<PHONE_NUMBER>517888-0000000006518294
-rw------- 1 root microk8s  8371472 May 21 10:56<PHONE_NUMBER>518295-0000000006518684
-rw------- 1 root microk8s  8374280 May 21 11:00<PHONE_NUMBER>518685-0000000006519113
-rw------- 1 root microk8s  8370896 May 21 11:04<PHONE_NUMBER>519114-0000000006519438
-rw------- 1 root microk8s  8381552 May 21 11:07<PHONE_NUMBER>519439-0000000006519854
-rw------- 1 root microk8s  8379248 May 21 11:09<PHONE_NUMBER>519855-0000000006520295
-rw------- 1 root root      8386592 May 21 11:10<PHONE_NUMBER>520296-0000000006520838
-rw------- 1 root root      8358152 May 21 11:11<PHONE_NUMBER>520839-0000000006521214
-rw------- 1 root root      8353256 May 21 11:13<PHONE_NUMBER>521215-0000000006521579
-rw------- 1 root root      8383136 May 21 11:13<PHONE_NUMBER>521580-0000000006522017
-rw------- 1 root root       667592 May 21 11:15<PHONE_NUMBER>522018-0000000006522055
-rw------- 1 root root      8384256 May 21 11:17<PHONE_NUMBER>522056-0000000006522395
-rw------- 1 root root      8364560 May 21 11:18<PHONE_NUMBER>522396-0000000006522746
-rw------- 1 root root      8366576 May 21 11:19<PHONE_NUMBER>522747-0000000006523125
-rw-rw---- 1 root microk8s     1866 May  3 18:58 cluster.crt
-rw-rw---- 1 root microk8s     3268 May  3 18:58 cluster.key
-rw-rw---- 1 root microk8s      201 May 21 11:19 cluster.yaml
-rw-rw-r-- 1 root microk8s        2 May 21 11:15 failure-domain
-rw-rw---- 1 root microk8s       61 May  3 19:09 info.yaml
srw------- 1 root root            0 May 21 11:15 kine.sock
-rw-rw---- 1 root microk8s       67 May 21 11:15 localnode.yaml
-rw-rw---- 1 root microk8s       32 May 21 11:15 metadata1
-rw-rw---- 1 root microk8s       32 May 21 10:59 metadata2
-rw------- 1 root root      8388608 May 21 11:19 open-4
-rw------- 1 root root      8388608 May 21 11:18 open-5
-rw------- 1 root root      8388608 May 21 11:19 open-6
-rw------- 1 root root     77753400 May 21 11:13 snapshot-256-6522037-580700424
-rw------- 1 root root          128 May 21 11:13 snapshot-256-6522037-580700424.meta
-rw------- 1 root root     78305680 May 21 11:18 snapshot-256-6523061-580985737
-rw------- 1 root root          128 May 21 11:18 snapshot-256-6523061-580985737.meta
and the other
-rw-rw---- 1 root microk8s  8355056 May 21 10:39<PHONE_NUMBER>514588-0000000006514977
-rw-rw---- 1 root microk8s  8363192 May 21 10:41<PHONE_NUMBER>514978-0000000006515423
-rw-rw---- 1 root microk8s  8364848 May 21 10:43<PHONE_NUMBER>515424-0000000006515892
-rw-rw---- 1 root microk8s  8386160 May 21 10:46<PHONE_NUMBER>515893-0000000006516258
-rw-rw---- 1 root microk8s  8377880 May 21 10:47<PHONE_NUMBER>516259-0000000006516737
-rw-rw---- 1 root microk8s  8376512 May 21 10:50<PHONE_NUMBER>516738-0000000006517083
-rw-rw---- 1 root microk8s  8386520 May 21 10:51<PHONE_NUMBER>517084-0000000006517682
-rw-rw---- 1 root microk8s  8387168 May 21 10:53<PHONE_NUMBER>517683-0000000006518119
-rw-rw---- 1 root microk8s  8384072 May 21 10:55<PHONE_NUMBER>518120-0000000006518513
-rw-rw---- 1 root microk8s  8365856 May 21 10:57<PHONE_NUMBER>518514-0000000006518996
-rw-rw---- 1 root microk8s  8357432 May 21 11:02<PHONE_NUMBER>518997-0000000006519305
-rw-rw---- 1 root microk8s  8384072 May 21 11:06<PHONE_NUMBER>519306-0000000006519699
-rw-rw---- 1 root microk8s  8386448 May 21 11:08<PHONE_NUMBER>519700-0000000006520126
-rw-rw---- 1 root microk8s  8380616 May 21 11:10<PHONE_NUMBER>520127-0000000006520643
-rw-rw---- 1 root microk8s  8380616 May 21 11:11<PHONE_NUMBER>520644-0000000006521103
-rw-rw---- 1 root microk8s  8384216 May 21 11:12<PHONE_NUMBER>521104-0000000006521442
-rw-rw---- 1 root microk8s  8364488 May 21 11:13<PHONE_NUMBER>521443-0000000006521849
-rw-rw---- 1 root microk8s  8383640 May 21 11:16<PHONE_NUMBER>521850-0000000006522237
-rw-rw---- 1 root microk8s  8385296 May 21 11:17<PHONE_NUMBER>522238-0000000006522591
-rw-rw---- 1 root microk8s  8384648 May 21 11:18<PHONE_NUMBER>522592-0000000006522936
-rw-rw---- 1 root microk8s     1866 May  3 18:58 cluster.crt
-rw-rw---- 1 root microk8s     3268 May  3 18:58 cluster.key
-rw-rw---- 1 root microk8s      201 May 21 11:19 cluster.yaml
-rw-rw-r-- 1 root microk8s        2 May 21 10:08 failure-domain
-rw-rw---- 1 root microk8s       61 May  3 19:09 info.yaml
srw-rw---- 1 root microk8s        0 May 21 10:08 kine.sock
-rw-rw---- 1 root microk8s       67 May 21 10:08 localnode.yaml
-rw-rw---- 1 root microk8s       32 May 21 10:59 metadata1
-rw-rw---- 1 root microk8s       32 May 21 10:59 metadata2
-rw-rw---- 1 root microk8s  8388608 May 21 11:19 open-38
-rw-rw---- 1 root microk8s  8388608 May 21 11:17 open-39
-rw-rw---- 1 root microk8s  8388608 May 21 11:18 open-40
-rw-rw---- 1 root microk8s 77671000 May 21 11:13 snapshot-256-6522032-581581168
-rw-rw---- 1 root microk8s      128 May 21 11:13 snapshot-256-6522032-581581168.meta
-rw-rw---- 1 root microk8s 78202680 May 21 11:18 snapshot-256-6523056-581866958
-rw-rw---- 1 root microk8s      128 May 21 11:18 snapshot-256-6523056-581866958.meta

@CharlesGillanders When it happens again, can you again perform a ls -l /var/snap/microk8s/current/var/kubernetes/backend/ on all nodes while it's in the bad state?

inspection-report-20210521_111445.tar.gz
I am seeing something unexpected in the ls -l (I don't know if it is important or not) two of the nodes have all of the files set to group microk8s but one node has some of them set to root. I know when I tried to recover the cluster I did manually set the group for all files to root to match the spec in the FAQ but it looks like files created after that point in time one one node at least are being created with a different group.
Here's output from the unusual node
-rw------- 1 root microk8s 8378168 May 21 10:40<PHONE_NUMBER>514780-0000000006515091
-rw------- 1 root microk8s 8360600 May 21 10:41<PHONE_NUMBER>515092-0000000006515615
-rw------- 1 root microk8s 8368520 May 21 10:44<PHONE_NUMBER>515616-0000000006516021
-rw------- 1 root microk8s 8381912 May 21 10:46<PHONE_NUMBER>516022-0000000006516442
-rw------- 1 root microk8s 8378240 May 21 10:48<PHONE_NUMBER>516443-0000000006516869
-rw------- 1 root microk8s 8387288 May 21 10:51<PHONE_NUMBER>516870-0000000006517308
-rw------- 1 root microk8s 8372768 May 21 10:51<PHONE_NUMBER>517309-0000000006517887
-rw------- 1 root microk8s 8364488 May 21 10:54<PHONE_NUMBER>517888-0000000006518294
-rw------- 1 root microk8s 8371472 May 21 10:56<PHONE_NUMBER>518295-0000000006518684
-rw------- 1 root microk8s 8374280 May 21 11:00<PHONE_NUMBER>518685-0000000006519113
-rw------- 1 root microk8s 8370896 May 21 11:04<PHONE_NUMBER>519114-0000000006519438
-rw------- 1 root microk8s 8381552 May 21 11:07<PHONE_NUMBER>519439-0000000006519854
-rw------- 1 root microk8s 8379248 May 21 11:09<PHONE_NUMBER>519855-0000000006520295
-rw------- 1 root root 8386592 May 21 11:10<PHONE_NUMBER>520296-0000000006520838
-rw------- 1 root root 8358152 May 21 11:11<PHONE_NUMBER>520839-0000000006521214
-rw------- 1 root root 8353256 May 21 11:13<PHONE_NUMBER>521215-0000000006521579
-rw------- 1 root root 8383136 May 21 11:13<PHONE_NUMBER>521580-0000000006522017
-rw------- 1 root root 667592 May 21 11:15<PHONE_NUMBER>522018-0000000006522055
-rw------- 1 root root 8384256 May 21 11:17<PHONE_NUMBER>522056-0000000006522395
-rw------- 1 root root 8364560 May 21 11:18<PHONE_NUMBER>522396-0000000006522746
-rw------- 1 root root 8366576 May 21 11:19<PHONE_NUMBER>522747-0000000006523125
-rw-rw---- 1 root microk8s 1866 May 3 18:58 cluster.crt
-rw-rw---- 1 root microk8s 3268 May 3 18:58 cluster.key
-rw-rw---- 1 root microk8s 201 May 21 11:19 cluster.yaml
-rw-rw-r-- 1 root microk8s 2 May 21 11:15 failure-domain
-rw-rw---- 1 root microk8s 61 May 3 19:09 info.yaml
srw------- 1 root root 0 May 21 11:15 kine.sock
-rw-rw---- 1 root microk8s 67 May 21 11:15 localnode.yaml
-rw-rw---- 1 root microk8s 32 May 21 11:15 metadata1
-rw-rw---- 1 root microk8s 32 May 21 10:59 metadata2
-rw------- 1 root root 8388608 May 21 11:19 open-4
-rw------- 1 root root 8388608 May 21 11:18 open-5
-rw------- 1 root root 8388608 May 21 11:19 open-6
-rw------- 1 root root 77753400 May 21 11:13 snapshot-256-6522037-580700424
-rw------- 1 root root 128 May 21 11:13 snapshot-256-6522037-580700424.meta
-rw------- 1 root root 78305680 May 21 11:18 snapshot-256-6523061-580985737
-rw------- 1 root root 128 May 21 11:18 snapshot-256-6523061-580985737.meta
and the other
-rw-rw---- 1 root microk8s 8355056 May 21 10:39<PHONE_NUMBER>514588-0000000006514977
-rw-rw---- 1 root microk8s 8363192 May 21 10:41<PHONE_NUMBER>514978-0000000006515423
-rw-rw---- 1 root microk8s 8364848 May 21 10:43<PHONE_NUMBER>515424-0000000006515892
-rw-rw---- 1 root microk8s 8386160 May 21 10:46<PHONE_NUMBER>515893-0000000006516258
-rw-rw---- 1 root microk8s 8377880 May 21 10:47<PHONE_NUMBER>516259-0000000006516737
-rw-rw---- 1 root microk8s 8376512 May 21 10:50<PHONE_NUMBER>516738-0000000006517083
-rw-rw---- 1 root microk8s 8386520 May 21 10:51<PHONE_NUMBER>517084-0000000006517682
-rw-rw---- 1 root microk8s 8387168 May 21 10:53<PHONE_NUMBER>517683-0000000006518119
-rw-rw---- 1 root microk8s 8384072 May 21 10:55<PHONE_NUMBER>518120-0000000006518513
-rw-rw---- 1 root microk8s 8365856 May 21 10:57<PHONE_NUMBER>518514-0000000006518996
-rw-rw---- 1 root microk8s 8357432 May 21 11:02<PHONE_NUMBER>518997-0000000006519305
-rw-rw---- 1 root microk8s 8384072 May 21 11:06<PHONE_NUMBER>519306-0000000006519699
-rw-rw---- 1 root microk8s 8386448 May 21 11:08<PHONE_NUMBER>519700-0000000006520126
-rw-rw---- 1 root microk8s 8380616 May 21 11:10<PHONE_NUMBER>520127-0000000006520643
-rw-rw---- 1 root microk8s 8380616 May 21 11:11<PHONE_NUMBER>520644-0000000006521103
-rw-rw---- 1 root microk8s 8384216 May 21 11:12<PHONE_NUMBER>521104-0000000006521442
-rw-rw---- 1 root microk8s 8364488 May 21 11:13<PHONE_NUMBER>521443-0000000006521849
-rw-rw---- 1 root microk8s 8383640 May 21 11:16<PHONE_NUMBER>521850-0000000006522237
-rw-rw---- 1 root microk8s 8385296 May 21 11:17<PHONE_NUMBER>522238-0000000006522591
-rw-rw---- 1 root microk8s 8384648 May 21 11:18<PHONE_NUMBER>522592-0000000006522936
-rw-rw---- 1 root microk8s 1866 May 3 18:58 cluster.crt
-rw-rw---- 1 root microk8s 3268 May 3 18:58 cluster.key
-rw-rw---- 1 root microk8s 201 May 21 11:19 cluster.yaml
-rw-rw-r-- 1 root microk8s 2 May 21 10:08 failure-domain
-rw-rw---- 1 root microk8s 61 May 3 19:09 info.yaml
srw-rw---- 1 root microk8s 0 May 21 10:08 kine.sock
-rw-rw---- 1 root microk8s 67 May 21 10:08 localnode.yaml
-rw-rw---- 1 root microk8s 32 May 21 10:59 metadata1
-rw-rw---- 1 root microk8s 32 May 21 10:59 metadata2
-rw-rw---- 1 root microk8s 8388608 May 21 11:19 open-38
-rw-rw---- 1 root microk8s 8388608 May 21 11:17 open-39
-rw-rw---- 1 root microk8s 8388608 May 21 11:18 open-40
-rw-rw---- 1 root microk8s 77671000 May 21 11:13 snapshot-256-6522032-581581168
-rw-rw---- 1 root microk8s 128 May 21 11:13 snapshot-256-6522032-581581168.meta
-rw-rw---- 1 root microk8s 78202680 May 21 11:18 snapshot-256-6523056-581866958
-rw-rw---- 1 root microk8s 128 May 21 11:18 snapshot-256-6523056-581866958.meta

This output looks OK to me, but that's post-recovery I assume?
When it happens again, can you again perform a ls -l /var/snap/microk8s/current/var/kubernetes/backend/ on all nodes while it's in the bad state?

@CharlesGillanders On a sidenote, do you want to share one of those snapshot files with me, I want to find out if they compress well. If your system doesn't contain sensitive info that is ... :-)

I feel the cluster is in a dangerous state. If you look at the logs with journalctl -f -u snap.microk8s.daemon-kubelite you will see messages like May 21 11:11:43 pi-node1 microk8s.daemon-kubelite[3802920]: Trace[736557743]: ---"Object stored in database" 7755ms (11:11:00.704) indicating long delays in the datastore. When an entry is added to the data store all nodes need to communicate over the network and write on their disk. @CharlesGillanders what are the specs of the disk on each node and the network connecting the nodes? Is there any workload running that is particularly demanding in terms of network or disk requirements?

This is 3 raspberry PIs running from USB (SanDisk 3.2Gen1 USB Stick).  There is no particular load on this (test/home use only).  Networking is the 1GB onboard NIC for the PIs.
Am I reading that right that there is a nearly 8s delay in one datastore call?  That seems very excessive.  Thanks for the pointer, I will go do some performance testing and see if there's one or more of the PIs that may have a failing disk.

I checked all three PIs - all seem to have roughly equivalent disk read performance
Node1: hdparm -Tt /dev/sda
/dev/sda:
Timing cached reads:   1450 MB in  2.00 seconds = 724.44 MB/sec
Timing buffered disk reads: 314 MB in  3.01 seconds = 104.29 MB/sec
Node2: hdparm -Tt /dev/sda
/dev/sda:
Timing cached reads:   1424 MB in  2.00 seconds = 712.62 MB/sec
Timing buffered disk reads: 260 MB in  3.04 seconds =  85.50 MB/sec
Node3: hdparm -Tt /dev/sda
/dev/sda:
Timing cached reads:   1558 MB in  2.00 seconds = 780.15 MB/sec
Timing buffered disk reads: 286 MB in  3.00 seconds =  95.29 MB/sec
I'll need to test write performance but I do see that on all three nodes apiservice-kicker is using most of the CPU fairly constantly, I suspect that's not normal but it might be  an artifact of the issue I'm seeing rather than a  cause?
top - 12:52:42 up 6 days, 18:57,  2 users,  load average: 13.49, 12.57, 11.04
Tasks: 222 total,   5 running, 215 sleeping,   0 stopped,   2 zombie
%Cpu(s): 21.2 us, 74.5 sy,  0.0 ni,  2.7 id,  1.1 wa,  0.0 hi,  0.5 si,  0.0 st
MiB Mem :   7808.8 total,    248.1 free,   2532.0 used,   5028.7 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   4708.7 avail Mem
PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

3925859 root      20   0 3090028 884588  93588 S  93.8  11.1  16:00.99 kubelite
3803349 root      20   0    9760    704      0 R  92.5   0.0 134:12.32 apiservice-kick
1117080 root      20   0    9760    708      0 R  91.8   0.0   8653:58 apiservice-kick
3362076 root      20   0    9760    732      0 R  90.5   0.0 524:15.17 apiservice-kick
top - 12:53:24 up 6 days, 19:12,  2 users,  load average: 17.16, 16.09, 14.51
Tasks: 251 total,   5 running, 245 sleeping,   0 stopped,   1 zombie
%Cpu(s):  4.4 us, 79.5 sy,  0.0 ni,  0.5 id, 15.4 wa,  0.0 hi,  0.2 si,  0.0 st
MiB Mem :   7808.8 total,    225.3 free,   2869.3 used,   4714.3 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   4398.2 avail Mem
PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

1149649 root      20   0    9892    752      0 R  99.0   0.0   8856:55 apiservice-kick
4149852 root      20   0    9760    708      0 R  98.7   0.0 719:40.53 apiservice-kick
3451350 root      20   0    9892    744      0 R  97.7   0.0   1392:43 apiservice-kick
249774 root      20   0 3045796 804444  93160 S  20.7  10.1   2:37.57 kubelite
top - 12:53:49 up 6 days, 19:26,  2 users,  load average: 15.01, 13.48, 11.75
Tasks: 212 total,   6 running, 206 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.0 us, 99.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   7808.8 total,    356.5 free,   2367.5 used,   5084.9 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   4873.6 avail Mem
PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

2231802 root      20   0    9760    708      0 R  81.6   0.0   7754:43 apiservice-kick
691776 root      20   0    9760    708      0 R  80.6   0.0 252:51.77 apiservice-kick
2858799 root      20   0    9760    704      0 R  79.6   0.0   2279:30 apiservice-kick
1972701 root      20   0    9760    708      0 R  75.7   0.0   3319:28 apiservice-kick
347496 root      20   0    9892    752      0 R  73.7   0.0 567:24.52 apiservice-kick

@CharlesGillanders would you be able to run the following test:
sudo hdparm -Tt /dev/sda

Yeh I did that if you look in the previous comment.
Node1: hdparm -Tt /dev/sda
/dev/sda:
Timing cached reads: 1450 MB in 2.00 seconds = 724.44 MB/sec
Timing buffered disk reads: 314 MB in 3.01 seconds = 104.29 MB/sec
Node2: hdparm -Tt /dev/sda
/dev/sda:
Timing cached reads: 1424 MB in 2.00 seconds = 712.62 MB/sec
Timing buffered disk reads: 260 MB in 3.04 seconds = 85.50 MB/sec
Node3: hdparm -Tt /dev/sda
/dev/sda:
Timing cached reads: 1558 MB in 2.00 seconds = 780.15 MB/sec
Timing buffered disk reads: 286 MB in 3.00 seconds = 95.29 MB/sec

Write performance appears to be awful on all three nodes.
root@pi-node1:~# dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 635.829 s, 1.7 MB/s
root@pi-node2:/# dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 763.903 s, 1.4 MB/s
root@pi-node3:/# dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 435.923 s, 2.5 MB/s

@MathieuBordere I'm happy to send you one of the snapshots - can I do so without uploading it to github?

@MathieuBordere I'm happy to send you one of the snapshots - can I do so without uploading it to github?

Thanks! You can send me a link at<EMAIL_ADDRESS>

@CharlesGillanders It appears your issues are caused by the very limited write performance of your storage. Does your Pi have an internal flash? Might be worth experimenting with that and see if things improve, I would be interested to know if that solves it. We are planning to address similar issues like yours by compressing the snapshots and lowering the frequency with which they are taken, should take some pressure of the disk.

Digging around online I found this post https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=245931 about slow USB disk performance on Raspberry PI.  I followed those instructions to add a usb-storage.quirks entry to cmdline.txt and that has substantially improved write performance on these Raspberry PI nodes.
pi-node1:~# dd if=/dev/zero of=/tmp/test1.img bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.74865 s, 139 MB/s
pi-node2:~$ sudo dd if=/dev/zero of=/tmp/test1.img bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.50007 s, 143 MB/s
pi-node3:~$ sudo dd if=/dev/zero of=/tmp/test1.img bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.64548 s, 140 MB/s
I am going to restart the microk8s cluster and see if it is any more stable now.

Nice!

I'll need to test write performance but I do see that on all three nodes apiservice-kicker is using most of the CPU fairly constantly, I suspect that's not normal but it might be an artifact of the issue I'm seeing rather than a cause?
top - 12:52:42 up 6 days, 18:57, 2 users, load average: 13.49, 12.57, 11.04
Tasks: 222 total, 5 running, 215 sleeping, 0 stopped, 2 zombie
%Cpu(s): 21.2 us, 74.5 sy, 0.0 ni, 2.7 id, 1.1 wa, 0.0 hi, 0.5 si, 0.0 st
MiB Mem : 7808.8 total, 248.1 free, 2532.0 used, 5028.7 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 4708.7 avail Mem
PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

3925859 root 20 0 3090028<PHONE_NUMBER>8 S 93.8 11.1 16:00.99 kubelite
3803349 root 20 0 9760 704 0 R 92.5 0.0 134:12.32 apiservice-kick
1117080 root 20 0 9760 708 0 R 91.8 0.0 8653:58 apiservice-kick
3362076 root 20 0 9760 732 0 R 90.5 0.0 524:15.17 apiservice-kick

I see the same issue with apiservice-kicker running at 100% CPU load. Looking at apiservice-kicker (bash script) it should not cause load. The script should sleep most of the time and every 3 seconds look for some changes and if needed restart services... I'm new to microk8s and not sure how-to best debug apiservice-kicker.
After starting my system (currently a single node Raspberry PI 4 8GB - will add two more shortly) the system runs fine for a while, then the apiservice-kicker starts consuming too much CPU.
#top
top - 05:44:37 up 1 day, 20:40,  2 users,  load average: 2.34, 2.79, 2.92
Tasks: 207 total,   2 running, 205 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.7 us, 32.5 sy,  0.0 ni, 59.6 id,  0.1 wa,  0.0 hi,  0.2 si,  0.0 st
MiB Mem :   7808.9 total,   2506.1 free,   1309.9 used,   3992.8 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   6072.9 avail Mem
PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

2568444 root      20   0    9128    668      0 R  99.7   0.0 339:47.91 apiservice-kick
136424 root      20   0 2696572 666308  71952 S  16.5   8.3 600:26.70 kube-apiserver
136431 root      20   0    9128   3584   2956 S   0.0   0.0   5:47.40 apiservice-kick
2568440 root      20   0    9128   2120   1472 S   0.0   0.0   0:00.00 apiservice-kick
2568441 root      20   0    9128   2012   1348 S   0.0   0.0   0:00.00 apiservice-kick

#ps -eo pid,lstart,cmd | grep api
136424 Fri May 21 11:07:20 2021 /snap/microk8s/2217/kube-apiserver ....
136431 Fri May 21 11:07:20 2021 /bin/bash /snap/microk8s/2217/apiservice-kicker
2568440 Sun May 23 00:04:16 2021 /bin/bash /snap/microk8s/2217/apiservice-kicker
2568441 Sun May 23 00:04:16 2021 /bin/bash /snap/microk8s/2217/apiservice-kicker
2568444 Sun May 23 00:04:16 2021 /bin/bash /snap/microk8s/2217/apiservice-kicker
Any thought or pointers on how-to debug?
Best Regards
Bernhard

Any thought or pointers on how-to debug?
Best Regards
Bernhard

I found that the logging seems to be part of the problem (I saw some evidence of  similar issues and solutions with Rancher K3s)  - I adjusted journald to use volatile logging so that it isn't spending as much time writing log files to disk.
https://github.com/k3s-io/k3s/issues/1297
https://unix.stackexchange.com/questions/482920/store-more-logs-in-memory-with-systemd-journald
Since making this change I am no longer seeing the same constant load from apiserver-kicker.

@CharlesGillanders Has your cluster been more stable since performing the usb changes?

@CharlesGillanders Has your cluster been more stable since performing the usb changes?

Yes the cluster is more stable (nodes are no longer disconnecting or not being part of the cluster) but I am still seeing the apiservice-kicker script eventually hang and start to  consume large amounts of CPU.  When this happens the script cannot be killed and the only resolution seems to be a reboot of the node.
This morning I have taken the current snap I was running and on one node I have unsquashed the snap FS, edited the kicker script to add set -eux and I'm capturing the logged output from the script -hopefully that will allow me to  identify exactly what the script was doing when it hangs.

So I captured one of the hang  conditions.  The following is the last set of entries logged by the apiserver-kicker script.

2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + sleep 3
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + '[' -e /var/snap/microk8s/x1/var/lock/ha-cluster ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + getent group microk8s
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + chmod -R ug+rwX /var/snap/microk8s/x1/var/kubernetes/backend
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + chgrp microk8s -R /var/snap/microk8s/x1/var/kubernetes/backend
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + '[' -e /var/snap/microk8s/x1/var/lock/no-cert-reissue ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + grep -E '(--advertise-address|--bind-address)' /var/snap/microk8s/x1/args/kube-apiserver
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181730]: + ip route
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181731]: + grep default
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181732]: + snapctl services microk8s.daemon-kubelite
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181733]: + grep active
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ produce_certs
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ export OPENSSL_CONF=/snap/microk8s/current/etc/ssl/openssl.cnf
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ OPENSSL_CONF=/snap/microk8s/current/etc/ssl/openssl.cnf
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ for key in serviceaccount.key ca.key server.key front-proxy-ca.key front-proxy-client.key
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ '[' -f /var/snap/microk8s/x1/certs/serviceaccount.key ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ for key in serviceaccount.key ca.key server.key front-proxy-ca.key front-proxy-client.key
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ '[' -f /var/snap/microk8s/x1/certs/ca.key ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ for key in serviceaccount.key ca.key server.key front-proxy-ca.key front-proxy-client.key
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ '[' -f /var/snap/microk8s/x1/certs/server.key ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ for key in serviceaccount.key ca.key server.key front-proxy-ca.key front-proxy-client.key
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ '[' -f /var/snap/microk8s/x1/certs/front-proxy-ca.key ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ for key in serviceaccount.key ca.key server.key front-proxy-ca.key front-proxy-client.key
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ '[' -f /var/snap/microk8s/x1/certs/front-proxy-client.key ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ '[' -f /var/snap/microk8s/x1/certs/ca.crt ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ '[' -f /var/snap/microk8s/x1/certs/front-proxy-ca.crt ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ render_csr_conf
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181741]: +++ get_ips
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181742]: ++++ /snap/microk8s/x1/bin/hostname -I
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181741]: +++ local 'IP_ADDR=<IP_ADDRESS> <IP_ADDRESS> '
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181741]: +++ local CNI_INTERFACE=vxlan.calico
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181741]: +++ [[ -z <IP_ADDRESS> <IP_ADDRESS>  ]]
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181741]: +++ /snap/microk8s/x1/sbin/ifconfig vxlan.calico
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181745]: ++++ /snap/microk8s/x1/sbin/ip -o -4 addr list vxlan.calico
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181746]: ++++ /snap/microk8s/x1/usr/bin/gawk '{print $4}'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181747]: ++++ /snap/microk8s/x1/usr/bin/cut -d/ -f1
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181748]: ++++ head -1
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181741]: +++ CNI_IP=<IP_ADDRESS>
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181741]: +++ local ips=
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181741]: +++ for ip in $IP_ADDR
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181741]: +++ '[' <IP_ADDRESS> '!=' <IP_ADDRESS> ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181741]: +++ ips+=<IP_ADDRESS>
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181741]: +++ for ip in $IP_ADDR
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181741]: +++ '[' <IP_ADDRESS> '!=' <IP_ADDRESS> ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181741]: +++ IP_ADDR=<IP_ADDRESS>
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181741]: +++ echo <IP_ADDRESS>
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ local IP_ADDRESSES=<IP_ADDRESS>
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ cp /var/snap/microk8s/x1/certs/csr.conf.template /var/snap/microk8s/x1/certs/csr.conf.rendered
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ '[' <IP_ADDRESS> == <IP_ADDRESS> ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ '[' <IP_ADDRESS> == none ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ local ips= sep=
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ local -i i=3
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181750]: +++ echo <IP_ADDRESS>
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ for IP_ADDR in $(echo "$IP_ADDRESSES")
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ ips+='IP.3 = <IP_ADDRESS>'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ sep='\n'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ /snap/microk8s/x1/bin/sed -i 's/#MOREIPS/IP.3 = <IP_ADDRESS>/g' /var/snap/microk8s/x1/certs/csr.conf.rendered
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ '[' -f /var/snap/microk8s/x1/certs/csr.conf ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ local force
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ /snap/microk8s/x1/usr/bin/cmp -s /var/snap/microk8s/x1/certs/csr.conf.rendered /var/snap/microk8s/x1/certs/csr.conf
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ force=false
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ false
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ '[' '!' -f /var/snap/microk8s/x1/certs/front-proxy-client.crt ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181753]: +++ /snap/microk8s/x1/usr/bin/openssl x509 -noout -issuer
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ '[' 'issuer=CN = front-proxy-ca' == 'issuer=CN = <IP_ADDRESS>' ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181740]: ++ echo 0
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + csr_modified=0
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + [[ 0 -eq 1 ]]
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + restart_attempt=0
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + '[' -e /var/snap/microk8s/x1/args/cni-network/cni.yaml ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + '[' -e /var/snap/microk8s/x1/var/lock/ha-cluster ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + '[' -e /var/snap/microk8s/x1/var/lock/cni-loaded ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + '[' -e /var/snap/microk8s/x1/var/lock/cni-loaded ']'
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + is_nic_available
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + ip a show dev vxlan.calico
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + check_offloading
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181756]: + ethtool --show-offload vxlan.calico
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181758]: + grep -q ': on'

Here is the previous run of the same script a few seconds earlier

2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[1888]: + '[' -e /var/snap/microk8s/x1/var/lock/ha-cluster ']'
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[1888]: + getent group microk8s
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[1888]: + chmod -R ug+rwX /var/snap/microk8s/x1/var/kubernetes/backend
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[1888]: + chgrp microk8s -R /var/snap/microk8s/x1/var/kubernetes/backend
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[1888]: + '[' -e /var/snap/microk8s/x1/var/lock/no-cert-reissue ']'
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[1888]: + grep -E '(--advertise-address|--bind-address)' /var/snap/microk8s/x1/args/kube-apiserver
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181672]: + ip route
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181673]: + grep default
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181674]: + snapctl services microk8s.daemon-kubelite
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181675]: + grep active
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ produce_certs
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ export OPENSSL_CONF=/snap/microk8s/current/etc/ssl/openssl.cnf
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ OPENSSL_CONF=/snap/microk8s/current/etc/ssl/openssl.cnf
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ for key in serviceaccount.key ca.key server.key front-proxy-ca.key front-proxy-client.key
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ '[' -f /var/snap/microk8s/x1/certs/serviceaccount.key ']'
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ for key in serviceaccount.key ca.key server.key front-proxy-ca.key front-proxy-client.key
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ '[' -f /var/snap/microk8s/x1/certs/ca.key ']'
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ for key in serviceaccount.key ca.key server.key front-proxy-ca.key front-proxy-client.key
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ '[' -f /var/snap/microk8s/x1/certs/server.key ']'
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ for key in serviceaccount.key ca.key server.key front-proxy-ca.key front-proxy-client.key
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ '[' -f /var/snap/microk8s/x1/certs/front-proxy-ca.key ']'
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ for key in serviceaccount.key ca.key server.key front-proxy-ca.key front-proxy-client.key
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ '[' -f /var/snap/microk8s/x1/certs/front-proxy-client.key ']'
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ '[' -f /var/snap/microk8s/x1/certs/ca.crt ']'
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ '[' -f /var/snap/microk8s/x1/certs/front-proxy-ca.crt ']'
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ render_csr_conf
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181682]: +++ get_ips
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181683]: ++++ /snap/microk8s/x1/bin/hostname -I
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181682]: +++ local 'IP_ADDR=<IP_ADDRESS> <IP_ADDRESS> '
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181682]: +++ local CNI_INTERFACE=vxlan.calico
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181682]: +++ [[ -z <IP_ADDRESS> <IP_ADDRESS>  ]]
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181682]: +++ /snap/microk8s/x1/sbin/ifconfig vxlan.calico
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181686]: ++++ /snap/microk8s/x1/sbin/ip -o -4 addr list vxlan.calico
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181687]: ++++ /snap/microk8s/x1/usr/bin/gawk '{print $4}'
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181688]: ++++ /snap/microk8s/x1/usr/bin/cut -d/ -f1
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181689]: ++++ head -1
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181682]: +++ CNI_IP=<IP_ADDRESS>
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181682]: +++ local ips=
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181682]: +++ for ip in $IP_ADDR
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181682]: +++ '[' <IP_ADDRESS> '!=' <IP_ADDRESS> ']'
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181682]: +++ ips+=<IP_ADDRESS>
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181682]: +++ for ip in $IP_ADDR
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181682]: +++ '[' <IP_ADDRESS> '!=' <IP_ADDRESS> ']'
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181682]: +++ IP_ADDR=<IP_ADDRESS>
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181682]: +++ echo <IP_ADDRESS>
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ local IP_ADDRESSES=<IP_ADDRESS>
2021-05-26T15:56:02Z microk8s.daemon-apiserver-kicker[181681]: ++ cp /var/snap/microk8s/x1/certs/csr.conf.template /var/snap/microk8s/x1/certs/csr.conf.rendered
2021-05-26T15:56:03Z microk8s.daemon-apiserver-kicker[181681]: ++ '[' <IP_ADDRESS> == <IP_ADDRESS> ']'
2021-05-26T15:56:03Z microk8s.daemon-apiserver-kicker[181681]: ++ '[' <IP_ADDRESS> == none ']'
2021-05-26T15:56:03Z microk8s.daemon-apiserver-kicker[181681]: ++ local ips= sep=
2021-05-26T15:56:03Z microk8s.daemon-apiserver-kicker[181681]: ++ local -i i=3
2021-05-26T15:56:03Z microk8s.daemon-apiserver-kicker[181707]: +++ echo <IP_ADDRESS>
2021-05-26T15:56:03Z microk8s.daemon-apiserver-kicker[181681]: ++ for IP_ADDR in $(echo "$IP_ADDRESSES")
2021-05-26T15:56:03Z microk8s.daemon-apiserver-kicker[181681]: ++ ips+='IP.3 = <IP_ADDRESS>'
2021-05-26T15:56:03Z microk8s.daemon-apiserver-kicker[181681]: ++ sep='\n'
2021-05-26T15:56:03Z microk8s.daemon-apiserver-kicker[181681]: ++ /snap/microk8s/x1/bin/sed -i 's/#MOREIPS/IP.3 = <IP_ADDRESS>/g' /var/snap/microk8s/x1/certs/csr.conf.rendered
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181681]: ++ '[' -f /var/snap/microk8s/x1/certs/csr.conf ']'
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181681]: ++ local force
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181681]: ++ /snap/microk8s/x1/usr/bin/cmp -s /var/snap/microk8s/x1/certs/csr.conf.rendered /var/snap/microk8s/x1/certs/csr.conf
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181681]: ++ force=false
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181681]: ++ false
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181681]: ++ '[' '!' -f /var/snap/microk8s/x1/certs/front-proxy-client.crt ']'
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181710]: +++ /snap/microk8s/x1/usr/bin/openssl x509 -noout -issuer
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181681]: ++ '[' 'issuer=CN = front-proxy-ca' == 'issuer=CN = <IP_ADDRESS>' ']'
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181681]: ++ echo 0
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + csr_modified=0
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + [[ 0 -eq 1 ]]
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + restart_attempt=0
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + '[' -e /var/snap/microk8s/x1/args/cni-network/cni.yaml ']'
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + '[' -e /var/snap/microk8s/x1/var/lock/ha-cluster ']'
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + '[' -e /var/snap/microk8s/x1/var/lock/cni-loaded ']'
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + '[' -e /var/snap/microk8s/x1/var/lock/cni-loaded ']'
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + is_nic_available
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + ip a show dev vxlan.calico
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + check_offloading
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181713]: + ethtool --show-offload vxlan.calico
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181714]: + grep -E '^.x-checksumming:'
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181715]: + grep -q ': on'
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + return 0
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181716]: + snapctl services microk8s.daemon-apiserver
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181717]: + grep active
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + '[' 1 -eq 0 ']'
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + true
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + '[' 0 -ge 5 ']'

I think the obvious difference between the two is in the check_offloading section.

2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[1888]: + check_offloading
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181756]: + ethtool --show-offload vxlan.calico
2021-05-26T15:56:08Z microk8s.daemon-apiserver-kicker[181758]: + grep -q ': on'

2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + check_offloading
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181713]: + ethtool --show-offload vxlan.calico
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181714]: + grep -E '^.x-checksumming:'
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[181715]: + grep -q ': on'
2021-05-26T15:56:05Z microk8s.daemon-apiserver-kicker[1888]: + return 0

I am confused - how is the grep for the x-checksumming not happening in the hang situation?  The script isn't changing between runs.

I tried matching your output to my apiservice-kicker script and found you are on a different version than I am.
This entire section (which I assume is in your version) is not included in mine:
    if [ -e "${SNAP_DATA}/var/lock/cni-loaded" ] &&
      is_nic_available &&
      ! check_offloading
    then
      echo "Network interface $nic_name found! Disabling offloading on it..."
      deactivate_offloading
      sleep 2
      if check_offloading; then
        echo "Offloading successfully disabled on interface $nic_name"
      else
        echo "Offloading has not been disabled correctly on interface $nic_name. Please check what happened"
      fi
    fi

Still, we seem to be facing the same issue. looking at the difference between the "good" and the "bad" iteration of the logs you shared, I realize that the last strace line of each sub-process seems missing in the "bad" execution... Therefore I assume that the buffers did not fully flush and the strace output is incomplete. This would also explain the "changed grep"... it's the last output of one of the forks...
Therefore I assume the script hangs in the last part of the loop:
   if [ $installed_registry_help -eq 0 ] &&
       snapctl services microk8s.daemon-apiserver | grep active &> /dev/null &&
       "$SNAP/kubectl" "--kubeconfig=$SNAP_DATA/credentials/client.config" get namespace kube-public &> /dev/null
    then
      if ! "$SNAP/kubectl" "--kubeconfig=$SNAP_DATA/credentials/client.config" get configmap local-registry-hosting -n kube-public &> /dev/null
      then
        use_manifest registry-help apply
      fi

      installed_registry_help=1
    fi

I will wait for apiservice-kicker to hang again and then try the individual commands from this section...
Killing the "hanging" process does not work. I have to reboot and the "shutdown" takes very long!

some news...
I found some older threads talking about issues with apiserver restarting when metrics server is enabled. Therefore, I've setup a second Raspberry running microk8s, but did not add it to my cluster yet. I observed that this system did not show the hanging apiservice-kicker and the only difference was that I had not enabled any of the core dns, metrics server, ... services yet.
I decided to enable metrics server and disable metrics server on the other raspberry, and as of now (~ 12 hours of runtime) the new raspberry running media server had apiservice-kicker hang twice, on the first node (metrics server stopped) did not show the issue...
There seems a relation between the hang and metrics server being active!

Thanks for the tip @bkappler69 - I'm going to try that now and see what results I get over the weekend.

For the benefit of @MathieuBordere  and @ktsakalozos the advice from @bkappler69 has so far been perfect!
I have had no problems with the api-service-kicker in the last 72 hours, previously I could guarantee getting this issue on at least one node in any 24 hour period.
So it looks like the best practices for getting stable and reliable Microk8s on raspberry PI 4 has been a combination of

Configuring journald for volatile logging
Improving underlying disk storage performance (I ended up totally replacing my existing USB flash drives with USB attached 2.5" SSD with a USB to SATA convertor which properly supported UASP)
Disabling metrics server

new info on the hang!
I upgrade my cluster to 3 nodes and moved to snap install microk8s --classic --channel=1.21/stable
Still apiserver-kicker hangs every couple of days...
Analyzing the process states it turns out that the hang relates to the "check_offloading" - same as in @CharlesGillanders debugs above.
I created a simple script just containing the "check_offloading" part running in a loop with no sleep... With this script I was able to reproduce the issue - this is completely independent of API Server:
#!/bin/bash

set -eu

nic_name="vxlan.calico"

is_nic_available() {
  ip a show dev $nic_name > /dev/null 2>&1
}

check_offloading() {
  # Return an error if at least one offload is enabled (rx or tx)
  if ethtool --show-offload $nic_name 2>&1 | grep -E '^.x-checksumming:' | grep -q  ': on'; then
    return 1
  else
    return 0
  fi
}

while true; do
    echo -n "."
    if is_nic_available && ! check_offloading
    then
      echo "Network interface $nic_name found! Disabling offloading on it..."
      deactivate_offloading
      sleep 2
      if check_offloading; then
        echo "Offloading successfully disabled on interface $nic_name"
      else
        echo "Offloading has not been disabled correctly on interface $nic_name. Please check what happened"
      fi
    fi

done

The issue seems  to sit here:
if ethtool --show-offload $nic_name 2>&1 | grep -E '^.x-checksumming:' | grep -q  ': on'; then
In case of a hang, the ethtool comand executes and outputs into a pipe, but the grep statement is either not started or does not receive the data from the pipe.
A tail on the pipe from ethtool (tail /proc/PID/fd/4) returns the output once (meaning it was not read yet)...
My system was under load (~5-6) at the time of the failure....
Best Regards
Bernhard

@bkappler69 thank you for trying to dig into this bug. I am trying to reproduce it using your script with no luck so far.
You say the script hangs after a "while". How long do you have to wait? Maybe I am not giving it enough time.
Do you have MicroK8s with the metrics-server running in the background? I wonder if this has any effect as others reported in this issue.

@ktsakalozos it's hard to predict on how long it will take for the script to hang, but for me it's in the range of 1 to 6 hours relatively reliable.
I've reduced the script to an absolut minimum, which still shows the issue:
#!/bin/bash

while true; do

    echo -n "."

    if echo "hw-tc-offload: off [fixed]" 2>&1 | grep  'x-checksumming' ; then

         echo "should never get here...."

    fi

done

Overall the script does not really do anything ecept an "echo" which used to be the ethtool from the original and grep the output.
The problem must be either in bash, grep or the kernel directly.
The hanging process cannot be killed, it is in state running, buring 100% CPU in system space, no time in user space....
My environment is:

three node cluster of Raspberry PIs 4 with 8GB
booting of SSD directly
Ubuntu 64bit server (deployed via rpi-imager)
microk8s 1.21/stable installed
addons:
enabled:
dns                  # CoreDNS
ha-cluster           # Configure high availability on the current node
helm3                # Helm 3 - Kubernetes package manager
ingress              # Ingress controller for external access
metallb              # Loadbalancer for your Kubernetes cluster
storage              # Storage class; allocates storage from host directory
glusterfs 9.2

My cluster is not too busy yet (still in the setup phase).
I had my "test" script running on all 3 nodes over night, in the morning 2 got stuck. The hang seems to occur faster if there's load on the system (reboot one node, such that pods have to failover).

I installed microk8s on my Raspberry Pi3. microk8s inpsect will be stuck when it tries to inspect cluster. Its single node cluster. microk8s status will just hangs up for more than 20 mins!!
I modified cmdline.txt as suggested above and rebooted and there is no improvement.
I have not connected any USB device for disk. It's a plain rpi3 with SD card Ubuntu 20.04 LTS server image.
KUbectl, APi server and scheduler are taking up all CPU.
I cant give tarball as microk8s inspect just hangs on as shown below:
ubuntu@raspberrypi3:~$ microk8s inspect
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-flanneld is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver is running
Service snap.microk8s.daemon-apiserver-kicker is running
Service snap.microk8s.daemon-proxy is running
Service snap.microk8s.daemon-kubelet is running
Service snap.microk8s.daemon-scheduler is running
Service snap.microk8s.daemon-controller-manager is running
Service snap.microk8s.daemon-etcd is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
can anyone please suggest how to progress ahead with microk8s without adding any additional hardware.

I've been suffering from a lot of instability in trying to build up a cluser of RPi4's running microk8s and it would appear that this is the underlying cause. I'm seeing slow disk performance, high load (not from CPU but from disk wait time), and the indicative logs like this:
Sep 30 13:57:47 pi-06 microk8s.daemon-kubelite[1845]: Trace[1509214241]: ---"Object stored in database" 1627ms (13:57:47.649)
Sep 30 13:57:47 pi-06 microk8s.daemon-kubelite[1845]: Trace[922152618]: ---"Object stored in database" 272ms (13:57:47.671)
Sep 30 13:57:47 pi-06 microk8s.daemon-kubelite[1845]: Trace[1908037004]: ---"Object stored in database" 125ms (13:57:47.692)
Sep 30 13:58:52 pi-06 microk8s.daemon-kubelite[1845]: Trace[1136357025]: ---"Object stored in database" 1032ms (13:58:52.600)
Sep 30 13:58:53 pi-06 microk8s.daemon-kubelite[1845]: Trace[1526076370]: ---"Object stored in database" 1108ms (13:58:53.042)
Sep 30 13:58:53 pi-06 microk8s.daemon-kubelite[1845]: Trace[30915180]: ---"Object stored in database" 1242ms (13:58:53.252)
Sep 30 13:58:54 pi-06 microk8s.daemon-kubelite[1845]: Trace[263880530]: ---"Object stored in database" 520ms (13:58:54.638)
Sep 30 13:58:54 pi-06 microk8s.daemon-kubelite[1845]: Trace[2122212928]: ---"Object stored in database" 665ms (13:58:54.788)
Sep 30 13:58:55 pi-06 microk8s.daemon-kubelite[1845]: Trace[1513088012]: ---"Object stored in database" 1142ms (13:58:55.266)
Sep 30 13:59:32 pi-06 microk8s.daemon-kubelite[1845]: Trace[1857129140]: ---"Object stored in database" 665ms (13:59:32.053)
Sep 30 14:00:01 pi-06 microk8s.daemon-kubelite[1845]: Trace[1189483486]: ---"Object stored in database" 517ms (14:00:01.544)
Sep 30 14:00:01 pi-06 microk8s.daemon-kubelite[1845]: Trace[1679077799]: ---"Object stored in database" 546ms (14:00:01.848)
So firstly, can anyone define what time is considered too slow, and what time is "normal", or does this line not get logged at all if it was less than the defined threshold?
Secondly, I can see most of the disk usage is taken up by kubelite, is there some way to reduce the amount of disk writes it will use, or could this be redirected to a RAM disk or similar? In the event of a scheduled reboot, this could be written to disk, or in the event of a non-scheduled reboot, could I just add the node like it was a new node or similar?
Or, is it simply not possible to run microk8s on a system without better performing disk?
PS, currently I'm using the fastest SD cards I could find, SanDisk Extreme Pro 64GB UHS-I marked as U3, A2, XC and V30 which is reported as having performance of:
Up to 100MB/s read and up to 90MB/s write
The only option which I've now found with a higher performance level seems to be the SanDisk Extreme Pro 64GB UHS-II which has:
Up to 275MB/s read and up to 100MB/s write
Is there any other microSD card with better performance, or are there some other options for improving performance on RPi4 etc?
Appreciate any suggestions/ideas.

@adamgoryachev  I had a very similar experience with an SD performing as yours (Up to 100MB/s read and up to 90MB/s write).
I wasn't even able to start the cluster sometimes.
The next thing I want to try to rule out other factors, is with an external (USB) disk connected to the Raspberry Pi 4.
Also, how much RAM does yours have?

@adamgoryachev , @ferrarimarco
I had issues with my microk8s cluster - see above (3 Raspberry 4 systems with 8 GB and 1TB SSD attaches to each).
In https://github.com/ubuntu/microk8s/issues/2280#issuecomment-866912974 I gave an example of a simple script which when executed will hang after some time (sometimes after minutes, sometimes after hours, but it always got stuck).
I solved my issue by moving all my systems to Debian 64bit and running microk8s there - no issues since then. I suspect either the FW or the kernel version of Ubuntu being the root cause.
Hope this helps.

@adamgoryachev @ferrarimarco I now have a working and mostly reliable cluster which I got to by combination of a lot of things but I think most important were.

Booting each pi from a desktop 2.5" SSD which was attached through a USB to SATA convertor which properly supported UASP
Using Debian 64 as the OS

The need for UASP support is really important to call out - that made a huge difference in write IO latency.


Support migration from AWS IoT

Summary

Support migration from AWS IoT
Why do we need this?

To support users who want to migrate from AWS IoT.
What is already there? What do you see now?

We don't support this yet.
What is missing? What do you want to see?

We should support this LNS.
How do you propose to implement this?

Look into the AWS IoT API and check which ones need to be used: https://docs.aws.amazon.com/iot/latest/apireference/Welcome.html
How do you propose to test this?

With some test devices.
Can you do this yourself and submit a Pull Request?

No, I can help test.

This is moved to Dec 2024. We had issues with testing which we will resolve this month.


set SKIP_EMAIL_VALIDATION to true

email validation will be handled by the marketing app
Questions:

[ ] should I be adding this # Modified comment, like you do?
[ ] will this change apply to QA and production?

In the case where we're adding a new line that wasn't present I've been annotating it as # ADDED so that we can identify things that aren't part of the vanilla Ansible build. That will help us going forward as we figure out how to feed back upstream with a sane set of defaults for edx-platform.
And yes, this will apply to both production and QA


Site generator

We should agree on a static site generator. We could open the discussion for general web frameworks/CMS but atm I don't see a good reason to use them.
I would advocate going with Hakyll since it's slim and powerful and Igor knows a great deal about it.
Feel free to add your suggestions.

Using Jekyll now


get_appsearch_document_id() should return the id field in the serializer

The document ID is statically generated in get_appsearch_document_id(). If you set a custom "id" field in your serializer, delete_from_appsearch() fails to delete the object because the generated ID isn't the same as the actual document ID. This is potentially confusing behavior and should be documented.
Is there a way to allow the serializer be the source of truth on the ID?
Here's a possible implementation:
    def get_appsearch_document_id(self):
        """Get the unique document ID."""
        _serialiser = self.serialise_for_appsearch()
        if _serialiser.get('id'):
            return _serialiser['id']
        else:
            return '{}_{}'.format(type(self).__name__, self.pk)

If you need to set up a custom document id, you can simply override the get_appsearch_document_id method on your model class. This way, you don't have to change the id field on the serialser at all.

However, I do agree that it would be nice to have some documentation around this specific use case, so I'll leave this issue open until that's done.


Juno position

Hi,
Is it possible to calculate sign for the Juno asteroid?
Kind regards,
Nikola

At this time, no. However, this should be possible by updating a separate repo (which I'm also maintaining).
I haven't found the time yet to look up how to add more bodies like asteroids to the ephemeris dependency but I've added this issue to the list in the repo (under Issues)
https://github.com/0xStarcat/Moshier-Ephemeris-JS/
see this file:
https://github.com/0xStarcat/Moshier-Ephemeris-JS/blob/master/src/constants/celestialBodies.js


Docs: docker image

via https://hub.docker.com/r/fishtownanalytics/dbt
Let's:

update the dockerhub page
add info about the existence of these images, and how you can use them

Closing in favor of https://github.com/dbt-labs/docs.getdbt.com/issues/1028 :)


Gaudi fix issue with gaudi script

Issues Resolved by this Pull Request
Error during provisioning:
TASK [postscripts/common : Copy Intel Gaudi script to postscripts] *****************************************************************************************
Description of the Solution
Remove "#" form the script
Suggested Reviewers
@abhishek-s-a @priti-parate
TODO:

[ ] Finish internal testing

Can one of the admins verify this patch?


Change from unstable

#13  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @kirgirchrono and the rest of your teammates on  Graphite


Cant use in public socks ip address

i want to use your project for public socks ip
so i just copy and paste command it seems i have some problem
my public ip address is <IP_ADDRESS>:1080
my config:
tunnel:
  # Interface name
  name: tun0
  # Interface MTU
  mtu: 8500
  # Multi-queue
  multi-queue: false
  # IPv4 address
  ipv4: <IP_ADDRESS>
  # IPv6 address
  ipv6: 'fc00::1'

socks5:

  port: 1080
  address: <IP_ADDRESS>
  udp: 'tcp'

and last
sudo ip route add <IP_ADDRESS> dev eth0 metric 10
sudo ip route add default dev tun0 metric 20
but have problem and i cant do ping or curl or ...
what should i do?

socks5:
  udp: 'udp'

still not work
also i can try and get my server with
curl ip-api.com
but after route with these lines
sudo ip route add <IP_ADDRESS> dev eth0 metric 10
sudo ip route add default dev tun0 metric 20

got this error
curl: (56) Recv failure: Connection reset by peer
if can possible please provide an script for automatically do this thanks

If your server doesn't support UDP associations, DNS doesn't work, try TCP/IP without any domain resolve.

i changed my socks5 server for support udp and work fine but still problem for tunneling all server
before add two lines i used this command and work fine
curl --interface tun0 ip-api.com
but after add this two lines i have problem
sudo ip route add <IP_ADDRESS> dev eth0 metric 10 sudo ip route add default dev tun0 metric 20
and i cant use traffic of server i think i have problem for this line
sudo ip route add <IP_ADDRESS> dev eth0 metric 10
if socks5_server need input local ip?

Probably the gateway address is missing:
sudo ip route add <IP_ADDRESS> via [GATEWAY IP] dev eth0 metric 10

thanks work


Style load from JSON file

Support setting styles from JSON file.
Currently the chart's animate() method accepts style settings as objects:
let style = { ... };
chart.animate({ data, config, style });

If the parameter's style property is not an object, but a string, it should be considered as an URI for a style JSON file.
let style = '/mystyles.json';
chart.animate({ data, config, style });

Current implementation: https://github.com/vizzuhq/vizzu-lib/blob/main/src/apps/weblib/js-api/vizzu.js

CSS support added instead. Feature request cancelled.


1.3 Auth server deployment in Docker Swarm

Orchestrate authorization server in docker swarm
Add nginx proxy in front - centralised nginx model.
Add some test cases to see end-to-end working

@abhi4578 please check out the Auth server doc if any change is required, please mention them, as well as check the pull request also.

AAA server github repo :  https://github.com/datakaveri/iudx-aaa-server. Please go through the configs, docker-compose and orchestrate the AAA server accordingly in docker swarm @namanmanish


LXD 2.18 Broken on Ubuntu 16.04.3 LTS

Required information

Distribution: Ubuntu 16.04.3 LTS
The output of "lxc info" or if that fails:

Kernel version: GNU/Linux 4.4.0-92-generic x86_64
LXC version: 2.0.10
LXD version: 2.18-0ubuntu3~16.04.1
Storage backend in use: lvm

Issue description
Cannot install lxd on ubuntu 16.04.3 LTS with following error:
The following packages have unmet dependencies:
lxd : Depends: liblxc1 (>= 2.1.0~) but 2.0.8-0ubuntu1~16.04.2 is to be installed
Steps to reproduce

sudo apt-get install --yes --quiet --target-release=xenial-backports lxd

Oh crap, fixing.

Fix uploaded to the Ubuntu archive for 16.04 and 17.04, package is building now and should get published to the archive within the hour.

Sorry about that, my text system happened to have the more recent liblxc so I didn't catch the issue...

thx!


Exchanged 'secret police' and 'forced labour'

It appears these cards were switched in a later update.

Thanks.


fix: update strings and match designs

Fixes all text strings and aligns plugin with new designs (apart from follow up issues that I created)

Let's discuss this PR today

Updated PR with all of your review comments, except for the issues we are still discussing.
Here are the remaining issues I need your help with + anything you want to add / change.

Bold text in markdown using ** but doesn’t work
Empty State custom icon is not rendering in the empty state component for all managed services connected

@wtrocki

I think both issues are already mentioned upstream. Let's create PR upstream for this


[Bug]: Windows navigation

Similar Issues

[X] Before filing, I have searched for similar issues.

Neovim Version
NVIM v0.9.0-dev-1355+gb2d10aa01
Multiplexer Integration
Wezterm
Multiplexer Version
wezterm 20230421-075330-e0a92c73
Steps to Reproduce
Same as https://github.com/wez/wezterm/issues/3597
Just wanna ask have you tried this in Windows environment?, cos I wanna know if this is just really a windows-centric issue.
Expected Behavior
Should be able to navigate smoothly like tmux integration with vim using ctrl+ keymap
Actual Behavior
It is working but have a weird behavior as it using different keymap rather the specified keymap option
Minimal Configuration to Reproduce
local root = vim.fn.fnamemodify('./.repro', ':p')

-- set stdpaths to use .repro
for _, name in ipairs({ 'config', 'data', 'state', 'cache' }) do
  vim.env[('XDG_%s_HOME'):format(name:upper())] = root .. '/' .. name
end

-- bootstrap lazy
local lazypath = root .. '/plugins/lazy.nvim'
if not vim.loop.fs_stat(lazypath) then
  vim.fn.system({
    'git',
    'clone',
    '--filter=blob:none',
    '--single-branch',
    'https://github.com/folke/lazy.nvim.git',
    lazypath,
  })
end
vim.opt.runtimepath:prepend(lazypath)

-- install plugins
local plugins = {
  -- do not remove the colorscheme! it makes testing nicer
  'folke/tokyonight.nvim',
  'mrjones2014/smart-splits.nvim',
  -- add any other pugins here
}

require('lazy').setup(plugins, {
  root = root .. '/plugins',
})

require('smart-splits').setup({
  -- add any options here
})

-- recommended mappings
-- resizing splits
-- these keymaps will also accept a range,
-- for example `10<A-h>` will `resize_left` by `(10 * config.default_amount)`
vim.keymap.set('n', '<A-h>', require('smart-splits').resize_left)
vim.keymap.set('n', '<A-j>', require('smart-splits').resize_down)
vim.keymap.set('n', '<A-k>', require('smart-splits').resize_up)
vim.keymap.set('n', '<A-l>', require('smart-splits').resize_right)
-- moving between splits
vim.keymap.set('n', '<C-h>', require('smart-splits').move_cursor_left)
vim.keymap.set('n', '<C-j>', require('smart-splits').move_cursor_down)
vim.keymap.set('n', '<C-k>', require('smart-splits').move_cursor_up)
vim.keymap.set('n', '<C-l>', require('smart-splits').move_cursor_right)
-- swapping buffers between windows
vim.keymap.set('n', '<leader><leader>h', require('smart-splits').swap_buf_left)
vim.keymap.set('n', '<leader><leader>j', require('smart-splits').swap_buf_down)
vim.keymap.set('n', '<leader><leader>k', require('smart-splits').swap_buf_up)
vim.keymap.set('n', '<leader><leader>l', require('smart-splits').swap_buf_right)

-- add anything else here
vim.opt.termguicolors = true
-- do not remove the colorscheme! it makes testing nicer
vim.cmd([[colorscheme tokyonight]])

Additional Details and/or Screenshots
No response

Just wanna ask have you tried this in Windows environment?

I haven't, I don't use Windows and don't have any Windows computers.
I assume this applies specifically to Wezterm, is that correct? Do other multiplexers work as intended on Windows?
Can you try, from a different terminal, run Wezterm via the CLI by running the command wezterm, then open Neovim and run require('smart-splits).setup({ multiplexer_integration = 'wezterm' }) to ensure it's using Wezterm, then try again, does anything interesting show up in Wezterm's log output in the other terminal?

Also, are you able to move across Wezterm splits by themselves, just not when you add Neovim splits to the mix?

I updated my plugins today, and found ran into this issue. I'm running Arch Linux. When I create a wezterm split and run vim in it, I can move to the other window split for a second. After a second tho I can't move from vim to another wezterm window. I can however move from normal window to vim.

Can you try what I mentioned before:

Can you try, from a different terminal, run Wezterm via the CLI by running the command wezterm, then open Neovim (in the new Wezterm window you just opened) and run require('smart-splits).setup({ multiplexer_integration = 'wezterm' }) to ensure it's using Wezterm, then try again, does anything interesting show up in Wezterm's log output in the other terminal?

And add a print statement in the Wezterm mappings in the is_vim function, to check this comment from the wezterm issue? if wezterm is not reporting the foreground process name properly, then the keymaps won't work properly.

Ok, So I tried the first thing & that works. In my Config I had set multiplexer_integration to Wezterm.
As for the is_vim function. that was working fine. It returns nvim.

So it’s working for you now?

Yes

Sorry for the late reply but I also print the is_vim function and this is what I got

I also tried what other user say - https://github.com/wez/wezterm/issues/3597#issuecomment-1520220244, but it is still not working with this code
local function is_vim(pane)
  function test()
    return pane:get_title():find("~") ~= nil
  end
  print(pane:get_title())
  print(test())

  return pane:get_title():find("NVIM") ~= nil
end

I tried to navigate many times but its not letting me navigate to different wezterm panes as you can see in the image below.

Getting the same issue on windows, pane:get_foreground_process_name() simply is not getting nvim.exe as the process name, but one of its child process, in this case lua-language-server.exe as LSP was running, probably an upstream issue on Wezterm

Should be resolved on latest master with some small config changes. See README.md


Add function to support pre-hashed keys

Motivation
I'm working on a monad type with expensive method resolution (hash being one of them). To speed up the hashmap insertion I pre-hash my inner value when it is known, build my monad and insert into the HashMap using this new method with the pre-generated hash. It gave me a speed up of about 15%.
Implementation
The implementation is straightforward, using the same code path as insert but with a given hash.
The use of mem::swap is necessary to ensure that the old value (if any) is dropped and proved to be faster on my benchmarks than ptr::write.

Have you considered using the HashTable type instead? It is specifically designed for this kind of use case.
There is also the raw entry API which provides similar functionality, but that is deprecated in favor of HashTable.

Have you considered using the HashTable type instead? It is specifically designed for this kind of use case.
There is also the raw entry API which provides similar functionality, but that is deprecated in favor of HashTable.

I found the HashTable to low-level for what I'm doing, this Pull Request is just a suggestion, if considered not actually usefull I will just use my forked version keeping up-to-date with the library.

This method is unsuitable because we don't want to expose custom hash values in HashMap/HashSet. This could cause unsoundness because it would allow constructing a HashMap<usize, usize> which has inconsistent behavior. This is relevant because no types with custom Hash impl are used: you should be able to trust that maps of standard types behave predictably.
This is why all functionality related to custom hashing was moved to HashTable. Even though it has a low-level API, you can easily build a wrapper on top of it for your use case.

This method is unsuitable because we don't want to expose custom hash values in HashMap/HashSet. This could cause unsoundness because it would allow constructing a HashMap<usize, usize> which has inconsistent behavior. This is relevant because no types with custom Hash impl are used: you should be able to trust that maps of standard types behave predictably.
This is why all functionality related to custom hashing was moved to HashTable. Even though it has a low-level API, you can easily build a wrapper on top of it for your use case.

No problem, thanks for spending time looking into the PR 😄


[WIP] Custom reporter integration

Summary
This PR adds the ability to specify custom reporter to be used with Jest. Right now, there's no way for the user to specify the custom reporter they want to use, without tinkering the internals. This let's them specify the Custom reporter(s) if any users want to use with their tests.
This PR relies heavily on the work @dmitriiabramov did in #2115 for adding XUnitReporter. Right now, only the interface through which we can use different reporters has been developed. Reporters such as Nyan and XUnitReporter are work in progress.

This PR reuses
Test plan

Integration tests for custom reporters are work in progress.

The first custom reporter is going to be a Nyan Cat, as it's trivial to implement

@thymikee I am interested in knowing how we can add a better validation check for the Reporters being imported, in order to check if they have all the given functions we are going to dispatch implemented, maybe we can check if they all extend from the BaseReporter class.

@dmitriiabramov  In a Test Suite, Can we consider the the fullPath of the test as the value for the classname and package attribute within the test.?

Not adding it in this PR as @dmitriiabramov suggested, but here's the initial version of XUnitReporter written to be used a Custom Reporter. Any feedback is welcomed. https://gist.github.com/abdulhannanali/6560aa5f53e65ca3cf92fb9ad0836d02

One of the major concern I need to clear out is how the path resolution for the reporters will work.

@cpojer These are the rules we are using
{
    "rules": {
        "react/react-in-jsx-scope": 0,
        "react/jsx-no-undef": 0,
        "jest/no-focused-tests": 0,
        "no-undef": 0,
        "no-unused-vars": 0,
        "consistent-return": 0
    },
    "description": ".eslintrc-docs.json overrides the rules specified in .eslintrc, to make it more suitable for running on code examples in docs/ folder"
}

@dmitriiabramov The Reporters in the Jest extend from a BaseReporter, but in Mocha the biggest difference is that the BaseReporter is responsible for keeping track of the whole progress of the tests within it's stats object while the BaseReporter within the Jest is just exposing two more methods and removing the preRunMessage.
We can even remove the BaseReporter. However, it has a few advantages, as the code in the TestRunner would break if the reporters don't implement all of the methods on the base reporter. Also, we can additionally check in the code if the reports are inheriting from BaseReporter making things easier for us.

@abdulhannanali super excited you are working on this! Would you mind also including the Nyan reporter example in this diff so that we can see what it may look like? We should make sure that the API that Jest exposes to add reporters is actually useful and won't change too often.

Btw. there is also this proposal: https://github.com/js-reporters/js-reporters – I'm not sure if we should integrate with this.

Can we also add --reporters to the cli?  Right now this only allows it to be specified as a config option, but this would be a big win for jest-editor-support package if it could be passed in from the command line.  Should be simple enough to just add to https://github.com/facebook/jest/blob/master/packages/jest-cli/src/cli/args.js#L51 ?

@cpojer Thank you for sharing this js-reporters proposal seems quite nice of Qunit team. Something similar also came up in my mind after seeing the Mocha's Test Reporter interface, which is based on an EventEmitter, instead of calling methods on a class.
The EventEmitter interface really has it's advantages regarding the flexibility and making the TestDispatcher independent from how the reporters are implemented. Major parts in the TestRunner regarding the dispatching of methods and the existing runners will need to be updated for this to happen.  I still doubt if we can totally implement the proposal as we'll need to change concept around Suites too. However, it seems exciting, looking forward to get more feedback on this..
I am totally up for making implementing an EventEmitter interface though :)

@bookman25 IMO, It's too compicated to support this as a CLI option for now, how about first implementing it as a Config option and finalizing how we deal with it.

@cpojer js-reporters seems outdated, and popular frameworks such as Karma and Jasmine doesn't seem to be following this. It can be an overkill following this implementaiton. Jasmine's interface is actually quite similar to what we are already doing.

It's also very much focused on Testing frameworks that can be run client side. In order to reuse the reporters there.

@abdulhannanali sounds good. Let's build our own standard for reporters then ;)

@dmitriiabramov The only reason we seem to have a BaseReporter is to make sure all the methods exists so we can save us from method not found errors when we call the method on the given reporter, but  as we move towards custom reporters, a more flexible approach might be needed here.
The aggregatedResults provided by the TestRunner itself, is already doing the work of what Mocha does in it's Base.js class, similar to BaseReporter. So, for the standard reporters it might be okay to extend the BaseReporter class, but do we need to extend in the same way for other reporters.
We can have a _validateReporter method which checks if the given methods exist on the reporter, and if doesn't we just throw an error, or maybe we don't do this too, cos  the reporters should be flexible to implement only the methods they want which is the case when we are implementing simple reporters.
I am trying this _validateReporter approach and if it seems fine to you. Interested in hearing your thoughts about this.

@cpojer Can I take all these AggregatedResults related method and form them into one AggregatedResult class to be used by TestRunner.js?

@jest-bot will do, thanks for informing.

@dmitriiabramov @cpojer I have pushed the working version containing the tests. The custom reporters right now are reusing much of the Reporter functionality Jest already has for it's own reporters. The API to implement custom reporters is exactly the same. It's kind of really providing the hook with which you can add your own custom reporter.
In order for the reporters to confirm to the standard, I implemented a _validateReporters method which validates the reporter by checking if various methods are implemented for sure, but since this makes it very unflexible I removed it.
I would like to know any other suggestions, I have removed the [WIP] prefix from this PR too.
Thank you for your help everyone :heart:

Circle CI build is failed for some weird yarn reason, I have nothing to do with

Does this allow for reporters being passed in via the CLI?  It didn't look like it, but I might have missed it.  For jest-editor-support it would a big benefit to have CLI support for the reports arg.

@bookman25 As discussed before, this PR will only add the capability for the jest-config. A separate PR should be opened if we want to add the capability to CLI too. However, I haven't given a lot of thought to it, right now, even though Karma and Ava already does it.. Will think about it once this one is merged.

closing in favor of https://github.com/facebook/jest/pull/3349


Chromium 115 version

This just got released, which is Chromium 115 but it works on Vista, 7, and 8.x
https://github.com/win32ss/supermium
Maybe you could base this off of that instead and get a more up to date Chromium version.

@K4sum1 M115 is coming soon, and I definitely want to try to make a Thorium 115 version for Win7.

There's now a 117 version for supermium

well this would be awesome

@metapea @fenopa @K4sum1 I am probably going to target the M118 version for migrating to using Supermium as a base to build Thorium-Win7 on.

I am probably going to target the M118 version for migrating to using Supermium as a base to build Thorium-Win7 on.

Supermium 119 has been released. It seems that for ancient hardware (1GB RAM), an SSE2 build would be better on a base 109 or even older.
Practical usage shows that EDGE 115 and Supermium are 2-3 times slower than Thorium109. This is noticeable on heavy sites like Reddit, Discord.

hi @Alex313031 ,It would be helpful if you release thorium 119 based on supermium for windows 7.
Thank you

@Stepman123 @metapea @fenopa @K4sum1 Thorium M119 for Windows 7 BETA is out!
Final release should be coming any day now. I am still working out some kinks, but it is getting stabler and stabler.

@Stepman123 @metapea @fenopa @K4sum1 Thorium M119 for Windows 7 BETA is out!
Final release should be coming any day now. I am still working out some kinks, but it is getting stabler and stabler.

I wonder, could Supermium just become part of Thorium and make this Win7 version unnecessary?

So no, we wont be merging, but we hold each others hands lol.

I didn't mean that, I meant merging Thorium and Thorium-Win7. Probably should've just said that.

Please provide non avx sse4 builds

@K4sum1 Thats what I did. This repo is now a near identical mirror to the regular thorium repo. It just has a .patch file in the //patches dir to patch it for win7.
@libreom When the final M119 comes out, I will provide SSE3, SSE4, AVX, and AVX2 builds, as well as 32 bit builds. (However, the 32 bit builds are crashing, so they might have to come later)

@K4sum1 Thats what I did. This repo is now a near identical mirror to the regular thorium repo. It just has a .patch file in the //patches dir to patch it for win7.

Yeah, what I meant was just one repo. I don't think the Windows 7 fixes have any noticeable impact on performance or anything.

How come you "will never target Vista as my minimum baseline."? If Supermium supports it, why not?

@K4sum1 It requires linking in ntdll.dll, which slows the browser down, and requires progwrp.dll, which win32ss hasn't made open source yet. Also, most people aren't using Vista. The few Vista fans/users out there can use Supermium.
However, I DID specify the minimum NT version as 6.0 instead of 6.1, because you can still run it on Vista with Extended Kernel (also made by win32ss), and I didn't want to artificially limit it that way. It just can't run on stock Windows Vista.

@Alex313031 do you plan to update thorium-win7 as you have updated other versions to 120 and supermium also updated to 121 https://github.com/win32ss/supermium/releases/tag/v121 No hurries. 👍

@Alex313031 you should pin this if you want to keep this as a running dialogue about future updates for this flavor of Thorium.


New package: Polyester v0.3.0

Registering package: Polyester
Repository: https://github.com/JuliaSIMD/Polyester.jl
Created by: @chriselrod
Version: v0.3.0
Commit: a33a595579988fbcbe9f6ca49a7af1d911275613
Reviewed by: @chriselrod
Reference: https://github.com/JuliaSIMD/Polyester.jl/commit/a33a595579988fbcbe9f6ca49a7af1d911275613#commitcomment-51108609

[noblock]
This is renaming CheapThreads.jl -> Polyester.jl


Feedback on criterion in literature

A recent paper Robust benchmarking in noisy environments has feedback on criterion

While  it  is  impossible  to  eliminate  performance  variation entirely, benchmarking methodologies that attempt to account for both measurement error and external sources of variation do exist. For example, the Haskell microbenchmarking  package criterion attempts  to  thwart  error  due  to timer inaccuracy by growing the number of benchmark executions  per  timing  measurement  as  more  timing  measurements are  obtained.  After  all  measurements  are  taken,  a  summary estimate of the benchmark run time is obtained by examining the  derivative  of  the  ordinary  least  squares  regression  line at  the  point  of  a  single  evaluation  point.  There  are  three disadvantages  to  this  approach.  First,  the  least  squares  fit  is
sensitive to outliers (though criterion does warn the user if outliers are detected). Second, measurements made earlier in this experiment are highly vulnerable to timer error, since few benchmark repetitions are used. These early measurements can skew  the  regression,  and  hence  also  skew  the  final  run  time estimate.  Third,  measurements  made  later  in  the  experiment can  repeat  the  benchmark  more  times  than  are  necessary to  overcome  timer  error,  constituting  an  inefficient  use  of experiment time.

That's a cool paper. Thanks for the link!


The transection record, balance, balance in TokenScript action card are not synced

@James-Sangalli is this fixed with #1420 ?

I guess this requires the integration with services like AmberData?
Victor Zhang
CEO, AlphaWallet
Not only display information, @Alpha_Wallet let you preform the functions: DEPOSIT, WITHDRAW, PAYBACK, GENERATE, MO… https://t.co/6dJfdvnM3i

On 5 Sep 2019, at 17:10, Hwee-Boon Yar<EMAIL_ADDRESS>wrote:
@James-Sangalli is this fixed with #1420 ?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@zhangzhongnan928 ultimately yes, services like AmberData will help us to know that balances has changed and let us refresh it efficiently and quickly.
But even without AmberData, after the balance has changed for a while, and we visit this screen again, the app should be able to refresh it itself and get the correct balance. #1420 fixed an issue where we keep the old value of balances around and don't fetch the updated balances. So it might fix this issue to a certain extent (as long as we get the values all from the same node).

Not updated fast is ok, but need to be synced.
Cannot updated in one place, but not updated in others.
Victor Zhang
CEO, AlphaWallet
Not only display information, @Alpha_Wallet let you preform the functions: DEPOSIT, WITHDRAW, PAYBACK, GENERATE, MO… https://t.co/6dJfdvnM3i

On 5 Sep 2019, at 17:35, Hwee-Boon Yar<EMAIL_ADDRESS>wrote:
@zhangzhongnan928 ultimately yes, services like AmberData will help us to know that balances has changed and let us refresh it efficiently and quickly.
But even without AmberData, after the balance has changed for a while, and we visit this screen again, the app should be able to refresh it itself and get the correct balance. #1420 fixed an issue where we keep the old value of balances around and don't fetch the updated balances. So it might fix this issue to a certain extent (as long as we get the values all from the same node).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Not updated fast is ok, but need to be synced. Cannot updated in one place, but not updated in others

Right. We should at least fix this in this issue.

@zhangzhongnan928 use the latest build from testflight, it updates the cards on each click.

I believe this was fixed with the cache PR, not able to reproduce anymore as the values are given in real time from the TokenScript


[Bug] Pinterest Ads Validation Test Failing

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the issue
A validation test exists within this package to ensure that no combination of rows grouped by campaign ID, source relation, and _fivetran_synced timestamp should appear more than once. However, I'm encountering an issue in our raw data where multiple campaigns have campaign names but no campaign IDs. This test therefore fails and I'm unable to build a campaign-level model.
Relevant error log or model output
with validation_errors as (

    select
        source_relation, campaign_id, _fivetran_synced
    from ANALYTICS_DEV_AKSHAY_KAMAL.STAGING.stg_pinterest_ads__campaign_history
    group by source_relation, campaign_id, _fivetran_synced
    having count(*) > 1

)

select *
from validation_errors;

Expected behavior
I expected a unique ID to be assigned to my campaign data since it was originally synced via the Pinterest API. This may also be a source data quality issue, but in my investigation into the Pinterest Ads and Pinterest Ads Source packages, I encountered a number of macros from other packages that might be performing this work.
dbt Project configurations

models:
  +sql_header: "alter session set timezone = 'UTC';"
  +transient: "{{ false if target.name.upper() in ('PROD', 'DAGSTER_CLI_PROD', 'DAGSTER_SDA_PROD') else true }}"

  allbirds_dbt:
    marts:
      +materialized: table

      commercial:
        digital:
          +schema: commercial_digital
        retail:
          +schema: commercial_retail

      core:
        +materialized: table
        +schema: core

      finance_accounting:
        +schema: finance_accounting

      internal_metadata:
        snowflake:
          +schema: internal_metadata__snowflake

      observability:
        +schema: observability

      operations:
        +materialized: table
        +schema: operations

    scaffolds:
      +materialized: view
      +schema: staging

    staging:
      +materialized: view
      +schema: staging

  dbt_artifacts:
    +database: _dbt_artifacts
    +schema: internal_metadata__dbt_artifacts

  ad_reporting:
    +schema: staging

  facebook_ads:
    +schema: staging
  facebook_ads_source:
    +schema: staging

  google_ads:
    +schema: staging
  google_ads_source:
    +schema: staging

  microsoft_ads:
    +schema: staging
  microsoft_ads_source:
    +schema: staging

  pinterest:
    +schema: staging
  pinterest_source:
    +schema: staging

  tiktok_ads:
    +schema: staging
  tiktok_ads_source:
    +schema: staging

  twitter_ads:
    +schema: staging
  twitter_ads_source:
    +schema: staging

seeds:
  +schema: internal_metadata

vars:
  'dbt_date:time_zone': 'UTC'
  surrogate_key_treat_nulls_as_empty_strings: true

  allbirds_dbt:
    google_analytics_360__fivetran_cutover_date: '2022-07-01'
    unified_analytics__ga360_cutover_date: '2024-01-01 00:00:00.000 +0000'
    unified_analytics__rudderstack_start_date: '2022-12-01 00:00:00.000 +0000'
    unified_analytics__default_marketing_channel: 'Not_Set'
    unified_analytics__attribution_lookback_days: 14
    shopify__fivetran_instance_names: [
        'shopifyau',
        'shopifyca',
        'shopifyeu',
        'shopifyjp',
        'shopifynz',
        'shopifysk',
        'shopifyuk',
        'shopifyus'
    ]

    loop__fivetran_instance_names: [
        'fivetran_loop_returns_ca',
        'fivetran_loop_returns_eu',
        'fivetran_loop_returns_uk',
        'fivetran_loop_returns_us',
        'fivetran_loop_returns_ca_dev',
        'fivetran_loop_returns_eu_dev',
        'fivetran_loop_returns_uk_dev',
        'fivetran_loop_returns_us_dev'
    ]

  facebook_ads_schema: facebook_ads
  facebook_ads_database: fivetran

  google_ads_schema: google_ads
  google_ads_database: fivetran

  microsoft_ads_schema: bing_ads
  microsoft_ads_database: fivetran

  pinterest_schema: pinterest_ads_7_7_1
  pinterest_database: fivetran

  tiktok_ads_schema: tiktok_ads
  tiktok_ads_database: fivetran

  twitter_ads_schema: twitter_ads
  twitter_ads_database: fivetran

  ad_reporting__amazon_ads_enabled: False # by default this is assumed to be True
  ad_reporting__apple_search_ads_enabled: False # by default this is assumed to be True
  ad_reporting__linkedin_ads_enabled: False # by default this is assumed to be True
  ad_reporting__reddit_ads_enabled: False # by default this is assumed to be True
  ad_reporting__snapchat_ads_enabled: False # by default this is assumed to be True

  apple_search_ads__using_search_terms: False # by default this is assumed to be True
  pinterest__using_keywords: False # by default this is assumed to be True
  twitter_ads__using_keywords: False # by default this is assumed to be True

Package versions
packages:
  - package: dbt-labs/dbt_utils
    version: 1.1.1

  - package: calogica/dbt_expectations
    version: 0.10.1

  - git: https://github.com/brooklyn-data/dbt_artifacts.git
    revision: 2.6.2

  - package: fivetran/ad_reporting
    version: [">=1.7.0", "<1.8.0"]

What database are you using dbt with?
snowflake
dbt Version
dbt 1.7.5
Additional Context
No response
Are you willing to open a PR to help address this issue?

[ ] Yes.
[ ] Yes, but I will need assistance and will schedule time during our office hours for guidance
[ ] No.

After further investigation, this is likely an issue with either the Pinterest API or the Fivetran connector. Closing this issue.


Remove CryptoSwift

Fixes https://github.com/WalletConnect/WalletConnectSwiftV2/issues/27
Changes:

Add CommonCrypto based AES.CBC implementation
Implement hmac, sha256, sha512 using CryptoKit
Fix Autenticatin typo, I found more typos in JSONRPCSerialising, will open another pr if you need
Generate secure IV by SecRandomCopyBytes

Thank you @hewigovens and @vikmeup!


Add a mode that enables Save Actions for all projects

We love Save Actions, but find it a bit tedious to have to enable it on a per-project basis.
It would be great if we could check some box that makes it apply to every project.

i have same demand

The common approach on this request is to configure your default projects settings for new projects. Once you have have done this, new projects will have Save Actions enabled by default. To do so, go to File -> New project setup -> Preferences for new projects and setup your settings.

Wasn't aware of that option, and it works perfectly. Thank you!


Issue with latex files

I recently installed syntastic on my system. It works perfectly for my Python files, however it causes lot of issues with Latex files.
When trying to load a latex file with syntastic enabled, the following errors crop up:
Messages maintainer: Bram Moolenaar<EMAIL_ADDRESS>"Chapters/Chapter5.tex" 64L, 3396C
Error detected while processing function <SNR>28_BufWinEnterHook:
line    1:
E117: Unknown function: syntastic#log#debug
Error detected while processing function <SNR>28_BufWinEnterHook..58:
line    1:
E117: Unknown function: syntastic#log#debug
Error detected while processing function <SNR>28_BufWinEnterHook..58..75:
line    1:
E117: Unknown function: syntastic#log#debug
Error detected while processing function <SNR>28_BufWinEnterHook..58..28..29:
line    2:
E117: Unknown function: syntastic#log#debug
Error detected while processing function <SNR>28_BufWinEnterHook..58..28:
line    3:
E117: Unknown function: syntastic#log#debug
Error detected while processing function <SNR>28_BufWinEnterHook..58..3:
line    1:
E117: Unknown function: syntastic#log#debug
Error detected while processing function <SNR>28_BufWinEnterHook..58..3..4:
line    1:
E117: Unknown function: syntastic#log#debug
Error detected while processing function <SNR>28_BufEnterHook:
line    1:
E117: Unknown function: syntastic#log#debug

My syntastic options are:
let g:syntastic_mode_map = { 'mode': 'active',
\ 'active_filetypes': [],
\ 'passive_filetypes': []
}

let g:syntastic_tex_checkers = ['lacheck']
let g:syntastic_python_checkers = ['flake8']

I tried using chktex, but the same errors arise.

@darnir Hey!  Sorry to necro this old post, but do you remember what the interfering plugin was?

Sorry, it's been far too long. I can't remember what that plugin was.
Though, I think it was indentLine.

@darnir , No worries.  I don't remember what I did last month, let alone a year and a half ago.  Thanks for responding! :+1: