ConnectionReset Error when trying to read socket

Hello,
I am trying to use libertem_live to interface a MerlinEM.
I am first trying to stream data using the following example :
https://libertem.github.io/LiberTEM-live/merlin-lowlevel-example.html
The main modification I made is just changing the ip adress to connect to the PC that controls the MerlinEM hardware.
When executing :
data_source.socket.drain() or data_source.stream()
I get the following error :

Traceback (most recent call last):
File "C:\Users\supervisor\Git\medipix_server\NSLumiere\nionswift_plugin\IVG\QuantumPY\random_serv.py", line 118, in
client_handler(client_socket)
File "C:\Users\supervisor\Git\medipix_server\NSLumiere\nionswift_plugin\IVG\QuantumPY\random_serv.py", line 75, in client_handler
for frames in data_source.stream(num_frames = None, chunk_size = 16) :
File "C:\ProgramData\Anaconda3\envs\nionswift-medipix-server\lib\site-packages\libertem_live\detectors\merlin\data.py", line 616, in stream
yield from chunks
File "C:\ProgramData\Anaconda3\envs\nionswift-medipix-server\lib\site-packages\libertem_live\detectors\merlin\data.py", line 570, in _read_and_decode
acq_header = self.socket.read_acquisition_header()
File "C:\ProgramData\Anaconda3\envs\nionswift-medipix-server\lib\site-packages\libertem_live\detectors\merlin\data.py", line 382, in read_acquisition_header
length = self.read_mpx_length(cancel_timeout=cancel_timeout)
File "C:\ProgramData\Anaconda3\envs\nionswift-medipix-server\lib\site-packages\libertem_live\detectors\merlin\data.py", line 368, in read_mpx_length
hdr = self.read_unbuffered(15, cancel_timeout=cancel_timeout)
File "C:\ProgramData\Anaconda3\envs\nionswift-medipix-server\lib\site-packages\libertem_live\detectors\merlin\data.py", line 330, in read_unbuffered
bytes_read = self._socket.recv_into(
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

I am using libertem_live 0.2.1 and the MerlinEM readout software has the version : <IP_ADDRESS>
Could you help me ?

Hi, thanks for the report! For our setup, when we wanted to connect from the "outside" to the MerlinEM PC, we had to adjust the Windows Firewall settings to actually allow this. It should be possible to specifically allow connections on internal network adapters, for ports 6341 and 6342.
To diagnose if the firewall is the issue, I would suggest to (temporarily!) disable the firewall completely, and restart the script that calls data_source.stream(). If that works, it's clear that the firewall is the issue.

Hi,
Thanks for the very quick reply.
I disabled the firewall and tried again but it also failed with the same error. In the lab we are already using the MerlinEM with another interface through the same network connection and it works.
Is it worth trying myself to write a small python server and try to get the communication running ?

Great that the firewall can be excluded as a culprit that way.
Can you also confirm that there is no other data connection at the same time? IIRC there is a limit of one "user" data connection, the MerlinEM software itself will have another connection. This should be visible in the TCP server status window of the MerlinEM software (I can check tomorrow how it's called exactly)
If you have your other interface running at the same time, you'll get an error like the one you are reporting.

I restarted both computer and I also checked using netstat -a that no additional connection was active and I got the same behaviour anyway.
The with control, data_source : statement passes correctly.
The control.cmd('STARTACQUISITION) passes correctly too.
When the data_source socket is try to recieve some data I get the 10054 win error.

I'm a bit stumped - but it seems this specific error can also happen if the firewall on the PC running libertem-live blocks the outgoing connection. IIRC the Windows firewall shows a dialog the first time an application tries to create an outgoing connection, maybe that was denied at some point? Did you also try to deactivate the firewall temporarily on the PC running LiberTEM-live/nionswift? It's a bit weird that the control connection works, but the data connection doesn't...
In newer versions of the MerlinEM software there's also an error log - is anything logged there after trying to connect? Does the "TCP/IP Server (Data)" show a different status than Listening/Listen, or perhaps an error code that is not 0?

I am so sorry.... I made a mistake in the code and I was on the wrong port, it now works. I am closing this issue.

No worries, great to hear you got it solved!


Steve's vision for Hugo

This issue acts as a reminder for the presentation that @spf13 put together to share his thoughts about "The Future of Hugo".
At the time of writing certain things mentioned like i18n support and "make everything a node"-transition are already built-in to Hugo.

@digitalcraftsman when did @spf13 put together that presentation? Has his vision been updated since? Has anyone else presented their thoughts on a coherent vision (as opposed to grab-bag of pet features) since?

Steve created this presentation to communicate his vision on a private channel to the other maintainers. I don't know the exact date when he shared this document but it was months before I created this issue.

Has his vision been updated since?

No changes have been made since then. Some parts are can be ignored with the latest developments (i.e. the whole nodes vs. pages distinction). Others still have to finds it's way into Hugo.

Has anyone else presented their thoughts on a coherent vision (as opposed to grab-bag of pet features) since?

Not in this kind of presentation. @bep as lead developer will finally decide on which (new) features will be in the focus. He has a "mental roadmap" of improvements and major features that he would like to see implemented before making a v1.0 release.
AFAIK he hasn't written down what the missing pieces are to complete the puzzle. But the milestones and currently active issues should give you a good picture of where the development is heading towards.
Nonetheless, it would be still interesting to have some kind of outline of his vision.

I'm particularly interested in the concept of multi-part/multilingual metadata inheritance (Ref: Slide 16–17) as from what I can tell right now there's quite a bit of duplication in my bundles. Perhaps this is something which could be added with the introduction of multi-part pages, if that feature is still on the table, as I saw Steve mention meta inheritance as a feedback point in the multilingual implementation but met some resistance where I feel the original i18n implementor may not have considered the general use case.

Outdated, not actionable.


Design System > Components > Testimonials > Default > UI issues

Describe the bug
UI alignment issues.
Steps to reproduce bug

Go to 'https://dev-react.raaghu.ai/ & https://stage-react.raaghu.ai/'
Navigate to 'Design System > Components > Testimonials > Default'

Actual result-
UI alignment issues.
Screenshots

Expected result-

Expected result-
UI should be displays proper.
Desktop (please complete the following information):
-OS: [Windows 10 Enterprise]
-Browser [ edge]
-Version [ 125.0.2535.92]

Fixed on dev

Fixed on stage


[FEATURE] Support/verify installing Longhorn Helm chart using helm-controller which is built in k3s and rke2

Is your feature request related to a problem? Please describe (👍 if you like this request)
K3s/Rke2 include a Helm Controller that manages installing, upgrading/reconfiguring, and uninstalling Helm charts using a HelmChart Custom Resource Definition (CRD). Ref https://docs.k3s.io/helm
We want to verify and document the best practices for this new installation method. We can also add it to https://longhorn.io/docs/1.7.1/deploy/install/
One if the features of this installation method is that it will automatically uninstall the chart and reinstall the chart when an upgrade repeatedly failed. We all know that Longhorn chart is critical and should not be automatically installed like this otherwise there will be data lost. So for this installation method, we may want to set spec.failurePolicy  to abort  so that they can manually resolve the issue. Ref: https://docs.k3s.io/helm#helmchart-field-definitions

One additional item that we should investigate is what happens if we set spec.failurePolicy to abort and the Helm controller fails to upgrade.
My general feeling is that we should never rollback. We should somehow tell the Helm controller to retry. Rolling back is not supported in Longhorn
cc @james-munson

Also, I am assuming that (like for direct binary helm upgrade) we want to keep the pre-upgrade job active:
preUpgradeChecker:
  # -- Setting that allows Longhorn to perform pre-upgrade checks. Disable this setting when installing Longhorn using Argo CD or other GitOps solutions.
  jobEnabled: true
  # -- Setting that allows Longhorn to perform upgrade version checks after starting the Longhorn Manager DaemonSet Pods. Disabling this setting also disables `preUpgradeChecker.jobEnabled`. Longhorn recommends keeping this setting enabled.
  upgradeVersionCheck: true

To test, I built a HelmChart yaml file that looks like this:
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  annotations:
    helmcharts.cattle.io/managed-by: helm-controller
  creationTimestamp: "2024-09-18T18:21:13Z"
  finalizers:
  - wrangler.cattle.io/on-helm-chart-remove
  generation: 1
  name: longhorn
  namespace: longhorn-system
  resourceVersion: "187542"
  uid: 249c8726-f7ce-49d3-a516-ffe0cf272bf1
spec:
  chartContent: <base64 string of longhorn/charts tar>
  failurePolicy: abort
  set:
    defaultSettings.priorityClass: system-node-critical
    defaultSettings.replicaAutoBalance: least-effort
    defaultSettings.storageOverProvisioningPercentage: "200"
    image.csi.attacher.repository: longhornio/csi-attacher
    image.csi.nodeDriverRegistrar.repository: longhornio/csi-node-driver-registrar
    image.csi.provisioner.repository: longhornio/csi-provisioner
    image.csi.resizer.repository: longhornio/csi-resizer
    image.csi.snapshotter.repository: longhornio/csi-snapshotter
    longhornDriver.priorityClass: system-node-critical
    longhornManager.priorityClass: system-node-critical
    longhornUI.priorityClass: system-node-critical
    persistence.defaultClassReplicaCount: "2"
status:
  jobName: helm-install-longhorn

Then

Since the upgrade sequence includes the pre-upgrade check, which failed (although the reason for the failure is not exposed) no running longhorn components have been altered.  Intervention is simply a matter of deleting the helm-install-longhorn job (which we created in the longhorn-system namespace.)  Deleting the job will delete the failing helm-install-longhorn-7lkdt pod.

$ kl get jobs

NAME                    COMPLETIONS   DURATION   AGE
helm-install-longhorn   0/1           13m        13m

$ kl delete job helm-install-longhorn

job.batch "helm-install-longhorn" deleted

$ kl get pods

NAME                                                     READY   STATUS    RESTARTS      AGE
csi-attacher-7c8cdb8cc9-lmvpx                            1/1     Running   0             12d
csi-attacher-7c8cdb8cc9-pv4h6                            1/1     Running   2 (10d ago)   12d
csi-attacher-7c8cdb8cc9-sk9pp                            1/1     Running   0             12d
csi-provisioner-6c786d9b9-bvglt                          1/1     Running   2 (9d ago)    12d
csi-provisioner-6c786d9b9-qvqqk                          1/1     Running   0             12d
csi-provisioner-6c786d9b9-zs48c                          1/1     Running   1 (10d ago)   12d
csi-resizer-8f6bbc497-cr5zs                              1/1     Running   0             12d
csi-resizer-8f6bbc497-ml728                              1/1     Running   2 (10d ago)   12d
csi-resizer-8f6bbc497-nqc4z                              1/1     Running   2 (9d ago)    12d
csi-snapshotter-7c6c466646-dtjqz                         1/1     Running   0             12d
csi-snapshotter-7c6c466646-g4bkz                         1/1     Running   0             12d
csi-snapshotter-7c6c466646-jsbdf                         1/1     Running   1 (10d ago)   12d
engine-image-ei-b0369a5d-fhvs7                           1/1     Running   0             12d
engine-image-ei-b0369a5d-lvfbf                           1/1     Running   0             12d
engine-image-ei-b0369a5d-tnr4f                           1/1     Running   0             12d
instance-manager-59891674f992734191025d44edee0787        1/1     Running   0             12d
instance-manager-870c250b69a4fe01382ed46156d33f47        1/1     Running   0             12d
instance-manager-c367048286c0afd6e78dbc55cdf66e7e        1/1     Running   0             12d
instance-manager-da09e8123983053398c8d57d55d38b07        1/1     Running   0             12d
longhorn-csi-plugin-47c6f                                3/3     Running   0             12d
longhorn-csi-plugin-5k6wj                                3/3     Running   0             12d
longhorn-csi-plugin-tp78z                                3/3     Running   0             12d
longhorn-driver-deployer-68fcb95bd8-ppn9x                1/1     Running   0             12d
longhorn-manager-2w94t                                   1/1     Running   0             12d
longhorn-manager-8947w                                   1/1     Running   0             12d
longhorn-manager-jnsql                                   1/1     Running   0             12d
longhorn-ui-d6f459db6-cq5d6                              1/1     Running   0             12d
longhorn-ui-d6f459db6-lbgg8                              1/1     Running   0             12d
share-manager-pvc-2f0055f8-840b-4f50-a198-ee4b46d59d57   1/1     Running   0             12d

$ kl get helmcharts

NAME       JOB                     CHART   TARGETNAMESPACE   VERSION   REPO   HELMVERSION   BOOTSTRAP
longhorn   helm-install-longhorn

The running longhorn-managers are still untouched and 1.6.2:

$ kl get daemonset longhorn-manager -o yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "3"
    meta.helm.sh/release-name: longhorn
    meta.helm.sh/release-namespace: longhorn-system
  creationTimestamp: "2024-10-03T18:59:16Z"
  generation: 3
  labels:
    app: longhorn-manager
    app.kubernetes.io/instance: longhorn
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: longhorn
    app.kubernetes.io/version: v1.6.2
    helm.sh/chart: longhorn-1.6.2
  name: longhorn-manager
  namespace: longhorn-system
  resourceVersion: "122223909"
  uid: c3e2417d-d391-4828-a00b-2770a3c94a40
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: longhorn-manager
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: longhorn-manager
        app.kubernetes.io/instance: longhorn
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: longhorn
        app.kubernetes.io/version: v1.6.2
        helm.sh/chart: longhorn-1.6.2
    spec:
      containers:
      - command:
        - longhorn-manager
        - -d
        - daemon
        - --engine-image
        - longhornio/longhorn-engine:v1.6.2
        - --instance-manager-image
        - longhornio/longhorn-instance-manager:v1.6.2
        - --share-manager-image
        - longhornio/longhorn-share-manager:v1.6.2
        - --backing-image-manager-image
        - longhornio/backing-image-manager:v1.6.2
        - --support-bundle-manager-image
        - longhornio/support-bundle-kit:v0.0.37
        - --manager-image
        - longhornio/longhorn-manager:v1.6.2
        - --service-account
        - longhorn-service-account
        - --upgrade-version-check
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        image: longhornio/longhorn-manager:v1.6.2
        imagePullPolicy: IfNotPresent
        name: longhorn-manager
        ports:
        - containerPort: 9500
          name: manager
          protocol: TCP
        - containerPort: 9501
          name: conversion-wh
          protocol: TCP
        - containerPort: 9502
          name: admission-wh
          protocol: TCP
        - containerPort: 9503
          name: recov-backend
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /v1/healthz
            port: 9501
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /host/dev/
          name: dev
        - mountPath: /host/proc/
          name: proc
        - mountPath: /var/lib/longhorn/
          mountPropagation: Bidirectional
          name: longhorn
        - mountPath: /tls-files/
          name: longhorn-grpc-tls
      dnsPolicy: ClusterFirst
      priorityClassName: longhorn-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: longhorn-service-account
      serviceAccountName: longhorn-service-account
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /dev/
          type: ""
        name: dev
      - hostPath:
          path: /proc/
          type: ""
        name: proc
      - hostPath:
          path: /var/lib/longhorn/
          type: ""
        name: longhorn
      - name: longhorn-grpc-tls
        secret:
          defaultMode: 420
          optional: true
          secretName: longhorn-grpc-tls
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 100%
    type: RollingUpdate
status:
  currentNumberScheduled: 3
  desiredNumberScheduled: 3
  numberAvailable: 3
  numberMisscheduled: 0
  numberReady: 3
  observedGeneration: 3
  updatedNumberScheduled: 3

So no other rollback is needed.  The helmchart can stay in place - no action will be taken until it is modified.  However, the reason for the failure is not terribly clear.  (We know it is an attempt to jump from 1.6.2 to 1.8.0-dev, but that is not spelled out except in the logs of the failed pre-upgrade pod.)

@james-munson so, users don't need to set spec.failurePolicy to abort you mentioned after failing the pre-upgrade?

Note: the helmchart controller will create another upgrade job an hour later, it appears.  But don't attempt to prevent that by deleting the helmchart.  That implies an uninstall:

$ kl delete -f helmchart_master_head.yaml

helmchart.helm.cattle.io "longhorn" deleted

$ kl get jobs

NAME                   COMPLETIONS   DURATION   AGE
helm-delete-longhorn   1/1           24s        3m22s
longhorn-uninstall     0/1           3m18s      3m18s

$ kl get pods

NAME                                                     READY   STATUS      RESTARTS      AGE
csi-attacher-7c8cdb8cc9-lmvpx                            1/1     Running     0             12d
csi-attacher-7c8cdb8cc9-pv4h6                            1/1     Running     2 (11d ago)   12d
csi-attacher-7c8cdb8cc9-sk9pp                            1/1     Running     0             12d
csi-provisioner-6c786d9b9-bvglt                          1/1     Running     2 (10d ago)   12d
csi-provisioner-6c786d9b9-qvqqk                          1/1     Running     0             12d
csi-provisioner-6c786d9b9-zs48c                          1/1     Running     1 (11d ago)   12d
csi-resizer-8f6bbc497-cr5zs                              1/1     Running     0             12d
csi-resizer-8f6bbc497-ml728                              1/1     Running     2 (11d ago)   12d
csi-resizer-8f6bbc497-nqc4z                              1/1     Running     2 (10d ago)   12d
csi-snapshotter-7c6c466646-dtjqz                         1/1     Running     0             12d
csi-snapshotter-7c6c466646-g4bkz                         1/1     Running     0             12d
csi-snapshotter-7c6c466646-jsbdf                         1/1     Running     1 (11d ago)   12d
engine-image-ei-b0369a5d-fhvs7                           1/1     Running     0             12d
engine-image-ei-b0369a5d-lvfbf                           1/1     Running     0             12d
engine-image-ei-b0369a5d-tnr4f                           1/1     Running     0             12d
helm-delete-longhorn-lz7bp                               0/1     Completed   0             3m24s
instance-manager-59891674f992734191025d44edee0787        1/1     Running     0             12d
instance-manager-870c250b69a4fe01382ed46156d33f47        1/1     Running     0             12d
instance-manager-c367048286c0afd6e78dbc55cdf66e7e        1/1     Running     0             12d
instance-manager-da09e8123983053398c8d57d55d38b07        1/1     Running     0             12d
longhorn-csi-plugin-47c6f                                3/3     Running     0             12d
longhorn-csi-plugin-5k6wj                                3/3     Running     0             12d
longhorn-csi-plugin-tp78z                                3/3     Running     0             12d
longhorn-driver-deployer-68fcb95bd8-ppn9x                1/1     Running     0             12d
longhorn-manager-2w94t                                   1/1     Running     0             12d
longhorn-manager-8947w                                   1/1     Running     0             12d
longhorn-manager-jnsql                                   1/1     Running     0             12d
longhorn-ui-d6f459db6-cq5d6                              1/1     Running     0             12d
longhorn-ui-d6f459db6-lbgg8                              1/1     Running     0             12d
longhorn-uninstall-6krqd                                 0/1     Error       0             3m11s
longhorn-uninstall-txnm6                                 0/1     Error       0             3m20s
share-manager-pvc-2f0055f8-840b-4f50-a198-ee4b46d59d57   1/1     Running     0             12d

@james-munson so, users don't need to set spec.failurePolicy to abort you mentioned after failing the pre-upgrade?

@derekbit, they do need to set spec.failurePolicy to abort in the HelmChart yaml (as shown above https://github.com/longhorn/longhorn/issues/9506#issuecomment-2415277105).  If they do, the upgrade will just stop after pre-check failure, as it did here.  If not, it will go on to try an uninstall, which still should fail, but require a little more clean-up.  Unless they go on to set the deleting-confirmation-flag, which they probably don't really want.

@james-munson so, users don't need to set spec.failurePolicy to abort you mentioned after failing the pre-upgrade?

@derekbit, they do need to set spec.failurePolicy to abort in the HelmChart yaml (as shown above #9506 (comment)). If they do, the upgrade will just stop after pre-check failure, as it did here. If not, it will go on to try an uninstall, which still should fail, but require a little more clean-up. Unless they go on to set the deleting-confirmation-flag, which they probably don't really want.

Thanks @james-munson. Then, we need to create a page and a important node for the helm controller on the official doc.

Since the upgrade sequence includes the pre-upgrade check, which failed (although the reason for the failure is not exposed) no running longhorn components have been altered. Intervention is simply a matter of deleting the helm-install-longhorn job (which we created in the longhorn-system namespace.) Deleting the job will delete the failing helm-install-longhorn-7lkdt pod.

Sorry. Did we or Helm controller create the  helm-install-longhorn job ?

So no other rollback is needed. The helmchart can stay in place - no action will be taken until it is modified. However, the reason for the failure is not terribly clear. (We know it is an attempt to jump from 1.6.2 to 1.8.0-dev, but that is not spelled out except in the logs of the failed pre-upgrade pod.)

How does the user retry the upgrade?

Note: the helmchart controller will create another upgrade job an hour later, it appears. But don't attempt to prevent that by deleting the helmchart. That implies an uninstall:

This is perfhap due to the Helm controller has 1 hour resync so it will retry to create the helm installation job when it see that the job is missing the next time it resync.
I am wondering if the user set spec.failurePolicy to abort then:

When the upgrade fail, how does the user retry the upgrade? Do they need to delete the helm job and touch the Helm CR to trigger it. Is there a simpler way to retry?

Sorry. Did we or Helm controller create the helm-install-longhorn job ?
That's created by the Helm controller, although in whatever namespace is specified in the HelmChart yaml.

How does the user retry the upgrade?
The helm-install-longhorn-xxxx pod created by the job will retry a lot:

NAME                                  READY   STATUS             RESTARTS          AGE
helm-install-longhorn-install-cwtsd   0/1     CrashLoopBackOff   457 (4m55s ago)   38h

But if the reason for failure doesn't change beneath it, one can expect continued failures.  However, it does no harm to the running installation.

Maybe Helm controller resync the Helm CR somehow so it will retry to create the helm installation job when it sees that the job is missing
It appears to.  If the job is deleted, but the HelmChart is left intact, the job will be recreated, (I re-verified this over the weekend.)

I am wondering if the user set spec.failurePolicy to abort then:
When the upgrade fail, how does the user retry the upgrade? Do they need to delete the helm job and touch the Helm CR to trigger it. Is there a simpler way to retry?
That looks like the easiest way to get a retry to happen right away:  Fix the issue, if that can be determined (for instance, upgrade the engine in use), and then touch the CR to force a re-sync.

Whatever happens, we need to make it clear that the user should not delete the HelmChart CR itself, since that is regarded as asking for Longhorn to be un-installed.

It may also be necessary to delete a helm-controller secret. The controller appears to store the details of the most recent attempt here:
$ kubectl -n longhorn-system get apps.catalog.cattle.io
NAME               CHART      VERSION   RELEASE NAME       RELEASE VERSION   STATUS
longhorn-install   longhorn   1.7.2     longhorn-install   3                 failed

which, if dumped with -o yaml, is a very large record, with a binary encoded summary of the complete set of charts.  It is owned by a secret:
 kubectl -n longhorn-system get apps.catalog.cattle.io -o yaml | grep sh.helm
      objectset.rio.cattle.io/owner-name: sh.helm.release.v1.longhorn-install.v3
      name: sh.helm.release.v1.longhorn-install.v3

which has a record of the history of helm upgrades:
kubectl -n longhorn-systeml get secrets
NAME                                     TYPE                 DATA   AGE
longhorn-webhook-ca                      kubernetes.io/tls    2      3d22h
longhorn-webhook-tls                     kubernetes.io/tls    2      3d22h
sh.helm.release.v1.longhorn-install.v1   helm.sh/release.v1   1      3d22h
sh.helm.release.v1.longhorn-install.v2   helm.sh/release.v1   1      38m
sh.helm.release.v1.longhorn-install.v3   helm.sh/release.v1   1      36m

Deleting the most recent helm.sh secret and then touching the HelmChart will lead it to start over from the top, and re-run everything, including the longhorn pre-upgrade check.

Sounds good to me

Test Steps

Note Longhorn docs will include a section on using the Helm Controller to install, upgrade, or uninstall when the docs PR is commited.  Test sequences here should match that documentation.

Upgrade with repo - success
On a cluster with Longhorn 1.x installed, (and matching engine image running), create a HelmChart YAML like this:
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  annotations:
    helmcharts.cattle.io/managed-by: helm-controller
  finalizers:
  - wrangler.cattle.io/on-helm-chart-remove
  generation: 1
  name: longhorn
spec:
  version: <v1.x+1.y> # Replace with desired upgrade version
  chart: longhorn
  repo: https://charts.longhorn.io
  failurePolicy: abort
  set:
    defaultSettings.replicaAutoBalance: least-effort
    persistence.defaultClassReplicaCount: "2"
status:
  jobName: helm-install-longhorn

Then install it with kubectl apply -f <helm-chart.yaml>.  For instance. start with 1.6.3 and upgrade to 1.7.1.  Longhorn-manager and components should upgrade, and the engine upgrade must be done separately, just as for other upgrade method.  The HelmChart will be left in place, although the helm-install job can be deleted.
The spec.set settings can be any list of overridden defaults.

Important Do not delete the HelmChart CR.  Doing so is equivalent to requesting an uninstall of Longhorn.

Upgrade with chart contents - success
On a cluster with Longhorn 1.x installed, (and matching engine image running), create a HelmChart YAML like this:
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  annotations:
    helmcharts.cattle.io/managed-by: helm-controller
  finalizers:
  - wrangler.cattle.io/on-helm-chart-remove
  generation: 1
  name: longhorn
spec:
  chartContent: <tar cz of longhorn/chart for version 1.x+1.y | base64 -w 0>
  failurePolicy: abort
  set:
    defaultSettings.replicaAutoBalance: least-effort
    persistence.defaultClassReplicaCount: "2"
status:
  jobName: helm-install-longhorn

Make the chartContent entry by doing the tar and base64 encode of longhorn/chart and copying the large string into the field.  Then install it with kubectl apply -f <helm-chart.yaml>.  Longhhorn-manager and components should upgrade, and the engine upgrade must be done separately, just as for any other upgrade method.  The HelmChart will be left in place, although the helm-install job can be deleted.
Install - success
On a fresh cluster with no Longhorn, create a HelmChart YAML like this:
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  annotations:
    helmcharts.cattle.io/managed-by: helm-controller
  finalizers:
  - wrangler.cattle.io/on-helm-chart-remove
  generation: 1
  name: longhorn-install
  namespace: default
spec:
  version: v1.7.2 # Or whatever version is desired.
  chart: longhorn
  repo: https://charts.longhorn.io
  failurePolicy: abort
  targetNamespace: longhorn-system
  createNamespace: true
  set:
    defaultSettings.replicaAutoBalance: least-effort
    persistence.defaultClassReplicaCount: "2"
status:
  jobName: helm-install-longhorn

Install it with kubectl apply -f <heml-chart.yaml>.  As for other methods, it will take a little while for all the components to be created.  The main difference with this chart is that it specifies the creation of the longhorn-system namespace.
The HelmChart CR will be left in place.
Upgrade with chart contents - catch a failure
On a cluster with Longhorn 1.x installed, (and matching engine image running), create an upgrade HelmChart, but with a target release version of 1.<x+2>..  For instance, with 1.5.4 installed, edit the HelmChart to upgrade to 1.7.1 and apply with kubectl apply -f <helm-chart.yaml>.  Ensure that the YAML has the tag failurePolicy: abort.  Longhorn does not support install or upgrade with any other failurePolicy, including the default (reinstall).  The upgrade should fail at the pre-upgrade check` step, and the existing 1.5.4 components left in place.

Verification passed on master-head (longhorn 504ea01).
Install with chart content

Generate chart base64 string:

# cd longhorn
# tar cz chart | base64 -w 0

Copy the string to chartContent:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  annotations:
    helmcharts.cattle.io/managed-by: helm-controller
  finalizers:
  - wrangler.cattle.io/on-helm-chart-remove
  generation: 1
  name: longhorn-install
  namespace: default
spec:
  chartContent: <content of tar cz chart | base64 -w 0>
  failurePolicy: abort
  targetNamespace: longhorn-system
  createNamespace: true
  set:
    defaultSettings.replicaAutoBalance: least-effort
    persistence.defaultClassReplicaCount: "2"
status:
  jobName: helm-install-longhorn

Apply the yaml to create the helmchart custom resource:

$ kubectl create ns longhorn-system
$ kubectl apply -f install-chart-content.yaml

Verify Longhorn installed:

$ kubectl get pods -n longhorn-system -w
NAME                                                READY   STATUS    RESTARTS   AGE
longhorn-ui-c44bbbdd6-57zgk                         1/1     Running   0          7m19s
longhorn-ui-c44bbbdd6-7k5xp                         1/1     Running   0          7m19s
longhorn-manager-d8bdh                              2/2     Running   0          7m19s
longhorn-manager-fb6g5                              2/2     Running   0          7m19s
longhorn-manager-wxr5p                              2/2     Running   0          7m19s
engine-image-ei-b907910b-57lhg                      1/1     Running   0          7m3s
engine-image-ei-b907910b-v4mv7                      1/1     Running   0          7m3s
longhorn-driver-deployer-7b6b696f9f-86t4w           1/1     Running   0          7m19s
engine-image-ei-b907910b-45vzx                      1/1     Running   0          7m3s
instance-manager-df320dae8c962ccdd0c0e9775fbadce3   1/1     Running   0          6m33s
instance-manager-3e15168c64517b1cb98628231c6ea0de   1/1     Running   0          6m33s
instance-manager-a1ed841da163f6e819c9f2ef2e534906   1/1     Running   0          6m33s
csi-snapshotter-5c8cbbc487-sqnxg                    1/1     Running   0          6m
csi-attacher-6c657b85bb-4kt59                       1/1     Running   0          6m
csi-provisioner-7d98486f58-vrz7h                    1/1     Running   0          6m
csi-resizer-5cd684599d-j9qmw                        1/1     Running   0          6m
csi-attacher-6c657b85bb-tpfqx                       1/1     Running   0          6m
csi-resizer-5cd684599d-2jc4r                        1/1     Running   0          6m
longhorn-csi-plugin-4gbsj                           3/3     Running   0          6m
csi-snapshotter-5c8cbbc487-6dx47                    1/1     Running   0          6m
csi-provisioner-7d98486f58-dxnv9                    1/1     Running   0          6m
longhorn-csi-plugin-67xqp                           3/3     Running   0          6m
csi-provisioner-7d98486f58-k4rr7                    1/1     Running   0          6m
csi-attacher-6c657b85bb-wcg2v                       1/1     Running   0          6m
longhorn-csi-plugin-k4sjt                           3/3     Running   0          6m
csi-snapshotter-5c8cbbc487-pr5db                    1/1     Running   0          6m
csi-resizer-5cd684599d-wlczd                        1/1     Running   0          6m

Uninstall Longhorn

$ kubectl -n longhorn-system patch -p '{"value": "true"}' --type=merge lhs deleting-confirmation-flag
$ kubectl delete -f install-chart-content.yaml


USER'S DISPLAY fails to load in PKI mode for private-invite-only projects

looks like ../canAccess endpoint is using wrong id

PR: https://github.com/NationalSecurityAgency/skills-service/pull/2779

:+1:


$LATTICE_COORDINATOR_IP can be overridden for non-aws bootstrap

@davidwadden is correct that we would need to do a little more work to get this working.
In the Vagrantfile, you'd have to pass the LATTICE_COORDINATOR_IP into the install_from_tar script as an argument and propogate that value down to the upstart bootstrap script. With that change, the code you have provided should work. We'd be happy to look at an updated PR for this behavior.

I wasn't using the install_from_tar in the ansible playbook repo. I'll look at it for the PR
On Mon, Jan 5, 2015 at 8:26 AM, David Julia<EMAIL_ADDRESS>wrote:

@davidwadden is correct that we would need to do a little more work to get this working.
In the Vagrantfile, you'd have to pass the LATTICE_COORDINATOR_IP into the install_from_tar script as an argument and propogate that value down to the upstart bootstrap script. With that change, the code you have provided should work. We'd be happy to look at an updated PR for this behavior.
Reply to this email directly or view it on GitHub:
https://github.com/pivotal-cf-experimental/lattice/pull/7#issuecomment-68732428

We're closing this PR for now, as the files changed in the commit have been moved or no longer exist in our repository.  If you wish to have this feature implemented, I encourage you to open an issue so we can figure out how to best implement your request.


Rework ender gems with teleportation mechanic

The ender gem allows users to teleport a certain distance (According to the gem level) with a cooldown. In addition to this, it will damage the player according to the distance travelled.

Outdated new plan for gems


Add an easy way to find patterns

Sometimes it's not obvious where to look to find a particular pattern, that makes me sad. :cry:

This'll need some more chattin'.

Closing in favor of #170


Fix compilation

Fix compilation while compiling JS version with higher warnings.
strncasecmp() requires #include <strings.h>

This seems to be a POSIX header file, not a C standard header file?

Oh you are right. Should I add a C implementation for non-Posix?
Let me try to find a small implementation

It would certainly be better if there were.

Replaced by #412


Fatal error in , line 0 Check failed: has_pending_exception().

Current Behavior
my nodered runs on the office docker container and restarts because of an error:
#
# Fatal error in , line 0
# Check failed: has_pending_exception().
#
#
#
#FailureMessage Object: 0x7ffcd28a4440
 1: 0xb72001  [node-red]
 2: 0x1bfa314 V8_Fatal(char const*, ...) [node-red]
 3: 0xe22521 v8::internal::Isolate::UnwindAndFindHandler() [node-red]
 4: 0x11e513c v8::internal::Runtime_UnwindAndFindExceptionHandler(int, unsigned long*, v8::internal::Isolate*) [node-red]
 5: 0x15d9f95  [node-red]
./entrypoint.sh: line 14:     7 Trace/breakpoint trap   (core dumped) /usr/local/bin/node $NODE_OPTIONS node_modules/node-red/red.js --userDir /data $FLOWS "${@}"

this is since a few days without changing anything. before it was running without problems.
changed logging level to debug but didnt provide more infos. and the error message dont point me anywhere :/
Expected Behavior
No response
Steps To Reproduce
No response
Example flow
paste your flow here

Environment

Node-RED version: v3.1.0-beta.4 and latest release
Node.js version: v16.20.1
npm version: 8.19.4
Platform/OS: docker
Browser: Chrome

also i did a fresh install with the beta, imported my flow and installed the missing nodes

This type of issue is typically due to a native component of one of the extra modules you are using; the default palette of Node-RED doesn't include any native libraries.
What additional modules have you got installed?

Sry for the screenshots but don't have any other option ATM.

@wuast94 we aren't in a position to check all of those nodes for you. All I can do is offer some suggestions for you to identify which node it might be.
I would

check to see if any of those nodes have received an update recently
check for any issues reported against those nodes to see if someone else has hit a similar thing
look for any that include native components - that can be quite tricky to figure out without digging into the repo for each node as it is more likely they have a dependency on another npm module that has the native component.

Just for Documentation:

I searched for native modules with: find node_modules -type f -name "*.node" 2>/dev/null | grep -v "obj\.target"
I listed the dependency tree with: npm ls --all
Looked into the dependency tree that what modules are using the native components

I had 2 matches, openapi-red and node-red-contrib-matrix-chat
Booth arent updated recently, but after removing the matrix one my errors are gone. reinstall it node red works.
as soon as i connect to a matrix server nodered crashes again, and after removing the config from the node and it dont connect to server anymore i dont get crashes.
what makes me wonder is that this runs for over a month without problems and now breaks. sadly the node isnt maintained anymore and im not a programmer, so will have to find a way around this.
thanks for help 😊


New package: BPFnative v0.1.0

Registering package: BPFnative
Repository: https://github.com/jpsamaroo/BPFnative.jl
Created by: @jpsamaroo
Version: v0.1.0
Commit: 061752c37f6343fc6f2ae33796293528ba79540e
Reviewed by: @jpsamaroo
Reference: https://github.com/jpsamaroo/BPFnative.jl/commit/061752c37f6343fc6f2ae33796293528ba79540e#commitcomment-42919840

Please close this PR, I'll open a new one in a few weeks when the package can load properly.


Disable checksummed-hex encoding

This PR disables all instances of checksummed-hex encoding, leaving checksummed-hex decoding unmodified.
This will affect downstream consumers who expect checksummed-hex encoding to be applied to all hash or digest types output by the node or client.
Despite the output types no longer using mixed-case hex, mixed-case hex as input for all such types will continue to have checksum validation performed on them.
Closes #2514.

bors merge

@Fraser999 I went ahead and created a ticket to reenable this in 2.0


Windows binaries missing from the release page

Description
At the https://containerd.io/ homepage there's this paragraph:

containerd is available as a daemon for Linux and Windows.

But at the release page (e.g. https://github.com/containerd/containerd/releases/tag/v1.3.3) there is only the linux binaries (e.g. containerd-1.3.3.linux-amd64.tar.gz).
Describe the results you expected:
A windows binary in the release page.

When the website was designed we didn't have Windows support; I just moved this issue to the website repo and am re-opening it so we can fix this now that we have official Windows releases.


[SPARK-42896][SQL][PYSPARK] Make mapInPandas / mapInArrow support barrier mode execution

What changes were proposed in this pull request?

Make mapInPandas / mapInArrow support barrier mode execution
Why are the changes needed?

This is the preparation PR for supporting mapInPandas / mapInArrow barrier execution in spark connect mode. The feature is required by machine learning use cases.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

hmmm why do we need to care about the optimizer? The optimizer is not sensitive to the physical execution engine, e.g. Preso, Spark, Flink have many similar SQL optimizations.

hmmm why do we need to care about the optimizer? The optimizer is not sensitive to the physical execution engine, e.g. Preso, Spark, Flink have many similar SQL optimizations.

I am not familier with optimizer details but it is concern from @HyukjinKwon

Predicate pushdown is just an example. e.g., you might want to combine adjacent MapInPandass but it would need a special handling if is_barrier flag is added.

I am saying that real power of Catalyst optimizer is to optimize/reorder these logical plans, and I believe that's the reason why barrier execution wasn't introduced in SQL. But barrier has to be created exactly when the call is invoked, in which baiscally requires something like https://github.com/apache/spark/pull/19873 to have a sound implementation.
However, I am fine with having this as an exception if you guys are fine with this.

Barrier mode is only used in specific ML case, i.e. in model training routine, we will only use it in one pattern:
dataset.mapInPandas(..., is_barrier=True).collect()

To simply the implementation, we can implement a barrierMapInPandasAndCollect instead, and define a execution plan stage like BarrierMapInPandasAndCollectExec

If it is the only use case, i think it will be safe to add dedicated logical plan and physical plan for it.

I am saying that real power of Catalyst optimizer is to optimize/reorder these logical plans, and I believe that's the reason why barrier execution wasn't introduced in SQL. The barrier has to be created exactly when the call is invoked, in which basically it requires something like #19873 to have a sound implementation.
However, I am fine with having this as an exception if you guys are fine with this.

@cloud-fan What do you think of this ?

From a SQL engine's point of view, running all tasks at once or batch by batch doesn't matter. It doesn't change the semantics of the SQL operator, and the optimizer doesn't care about it. However, mapInPandas is a public API and you are free to define what's the expectation of the is_barrier parameter. To make our life easier, we can just define it as "the tasks of running the pandas function must all be launched at once", and it's not a barrier to the SQL operators. Then I think it's fine to just add a flag to the existing MapInPandas operator.

merged to master :)


Add metric producers to meter_provider configuration

Part of https://github.com/open-telemetry/opentelemetry-specification/issues/4015
Metric producers are registered to metric readers, so i've nested it under the periodic and manual readers.
@gouthamve @jack-berg

/easycla

/easycla


GH-3416 Initial LMDB-based SAIL.

based on NativeStore

currently uses same encodings for triples and values
may be switched to long IDs instead of int IDs

uses LWJGL interfaces to LMDB

fast implementation that is available for multiple platforms

uses one db for indexes and one db for values

GitHub issue resolved: #3416
Briefly describe the changes proposed in this PR:
This change adds a full LMDB-based SAIL that stores values and triples in two LMDB databases.

PR Author Checklist (see the contributor guidelines for more details):

[x] my pull request is self-contained
[x] I've added tests for the changes I made
[x] I've applied code formatting (you can use mvn process-resources to format from the command line)
[ ] I've squashed my commits where necessary
[ ] every commit message starts with the issue number (GH-xxxx) followed by a meaningful description of the change

#LMDB
Benchmark                                                       Mode  Cnt      Score      Error  Units
TransactionsPerSecondBenchmark.largerTransaction               thrpt    5      2.355 ±    0.128  ops/s
TransactionsPerSecondBenchmark.largerTransactionLevelNone      thrpt    5      2.849 ±    0.075  ops/s
TransactionsPerSecondBenchmark.mediumTransactionsLevelNone     thrpt    5   2502.306 ±  271.070  ops/s
TransactionsPerSecondBenchmark.transactions                    thrpt    5   9820.817 ±  819.341  ops/s
TransactionsPerSecondBenchmark.transactionsLevelNone           thrpt    5  12051.377 ± 1414.688  ops/s
TransactionsPerSecondBenchmark.veryLargerTransactionLevelNone  thrpt    5      0.023 ±    0.004  ops/s

Benchmark                                               Mode  Cnt     Score     Error  Units
QueryBenchmark.complexQuery                             avgt    5     0.283 ±   0.053  ms/op
QueryBenchmark.distinctPredicatesQuery                  avgt    5  3231.650 ± 157.282  ms/op
QueryBenchmark.groupByQuery                             avgt    5     0.121 ±   0.012  ms/op
QueryBenchmark.removeByQuery                            avgt    5     2.764 ±   0.061  ms/op
QueryBenchmark.removeByQueryReadCommitted               avgt    5     5.022 ±   0.347  ms/op
QueryBenchmark.simpleUpdateQueryIsolationNone           avgt    5     0.286 ±   0.014  ms/op
QueryBenchmark.simpleUpdateQueryIsolationReadCommitted  avgt    5     0.280 ±   0.017  ms/op

# Native Store
Benchmark                                                       Mode  Cnt    Score     Error  Units
TransactionsPerSecondBenchmark.largerTransaction               thrpt    5    4.757 ±   1.746  ops/s
TransactionsPerSecondBenchmark.largerTransactionLevelNone      thrpt    5    6.601 ±   1.447  ops/s
TransactionsPerSecondBenchmark.mediumTransactionsLevelNone     thrpt    5  602.569 ±  48.934  ops/s
TransactionsPerSecondBenchmark.transactions                    thrpt    5  625.174 ± 122.350  ops/s
TransactionsPerSecondBenchmark.transactionsLevelNone           thrpt    5  570.538 ± 190.161  ops/s
TransactionsPerSecondBenchmark.veryLargerTransactionLevelNone  thrpt    5    0.056 ±   0.025  ops/s

Benchmark                                               Mode  Cnt      Score      Error  Units
QueryBenchmark.complexQuery                             avgt    5     78.985 ±   67.389  ms/op
QueryBenchmark.distinctPredicatesQuery                  avgt    5   2353.538 ±  564.378  ms/op
QueryBenchmark.groupByQuery                             avgt    5     37.243 ±   21.944  ms/op
QueryBenchmark.removeByQuery                            avgt    5   2547.820 ± 1349.694  ms/op
QueryBenchmark.removeByQueryReadCommitted               avgt    5  10934.396 ± 1283.774  ms/op
QueryBenchmark.simpleUpdateQueryIsolationNone           avgt    5  10116.993 ± 4609.040  ms/op
QueryBenchmark.simpleUpdateQueryIsolationReadCommitted  avgt    5  11694.331 ± 2701.513  ms/op

That query performance is pretty impressive!

That query performance is pretty impressive!
Edit: Any chance it's too good to be true? It's miles ahead of the MemoryStore.

I'm also not sure if the numbers are correct. I've re-run the benchmarks several times with the same results.
To get more confidence I've also added corresponding tests in compliance/sparql. Those tests pass - at least on my computer.
BTW: I've used MDB_NOSYNC and MDB_NOMETASYNC for the benchmarks. Maybe most of the data is cached in (off-heap) memory.

I think this is fantastic. I would love to have it with longs as identifiers. Making it usable for large datasets like UniProt or Wikidata.
I put some small issues in already from a brief first look. But to me it looks like good quality code.

I think this is fantastic. I would love to have it with longs as identifiers. Making it usable for large datasets like UniProt or Wikidata.
I put some small issues in already from a brief first look. But to me it looks like good quality code.

I also think that longs are the way to go. To keep storage space efficiency also VarInts can be used.

@kenwenzel can you please create an issue in our issue tracker that describes the proposed improvement, and link this PR to it?

I'm on OS X with the new ARM based  M1 processor. Seems that support is coming in the next release (3.3.0).
https://github.com/LWJGL/lwjgl3/issues/601
I'm currently not able to run the tests or benchmarks. We could consider trying out the 3.3.0 snapshot version.

Short update from my side: I'm going to switch to varint encoded long ids and I'm planning to push this change within the next few days.
I'm also working on a zero-copy design for the record iterators on another branch.

@kenwenzel can you please create an issue in our issue tracker that describes the proposed improvement, and link this PR to it? And if you have them, I'd appreciate some pointers to the source of the LMDB libraries that you're using, and any licensing information for them - I'll need that info to log them for IP review.

You mean besides #3416? So I should create an LMDB-specific issue with some explanation?

Ah, no, you can use that issue, but it was not clear to me that was what you were using. Can you link them up by mentioning the issue number in your commit messages and the PR description please?

I'm on OS X with the new ARM based M1 processor. Seems that support is coming in the next release (3.3.0).
LWJGL/lwjgl3#601
I'm currently not able to run the tests or benchmarks. We could consider trying out the 3.3.0 snapshot version.

I've pushed a commit with an update to LWJGL 3.3.0.

A short update: I have some open issues with this.

drop custom comparator and use natural sort order for indexes (callbacks from native to Java are slow with LWJGL)
change group varint encoding to be compatible with natural sort order of the numbers (prerequisite for the above)
use varints everywhere in value store

I got it running on my laptop :) Thanks!
I took a look at the benchmarks and changed the complex query one to this:
	@Benchmark
	public long complexQuery() {

		try (SailRepositoryConnection connection = repository.getConnection()) {
			long count = connection
				.prepareTupleQuery(query4)
				.evaluate()
				.stream()
				.count();
			System.out.println(count);
			return count;
		}
	}

It print 0 which is wrong. You can change the one in the MemoryStore and see what it's supposed to be.

I got it running on my laptop :) Thanks!
I took a look at the benchmarks and changed the complex query one to this:
	@Benchmark
	public long complexQuery() {

		try (SailRepositoryConnection connection = repository.getConnection()) {
			long count = connection
				.prepareTupleQuery(query4)
				.evaluate()
				.stream()
				.count();
			System.out.println(count);
			return count;
		}
	}

It prints 0 which is wrong. You can change the one in the MemoryStore and see what it's supposed to be.

Thank you! I will dig into this.

Minor remark - you're using isse number 3415 in several commits, but the actual related issue is #3416.

Minor remark - you're using isse number 3415 in several commits, but the actual related issue is #3416.

Oops, I'll rewrite the commits when it is finally working.

That query performance is pretty impressive!
Edit: Any chance it's too good to be true? It's miles ahead of the MemoryStore.

@hmottestad I hope that I've fixed the problems now.
I have to investigate why complexQuery is much slower with LMDB than with NativeStore.

@kenwenzel this is up to you, but if you are comfortable with some of us pairing up to get this polished, can you give @eclipse/technology-rdf4j-committers access to your forked repo so we can push to your branch?

@kenwenzel this is up to you, but if you are comfortable with some of us pairing up to get this polished, can you give @eclipse/technology-rdf4j-committers access to your forked repo so we can push to your branch?

@jeenbroekstra  Sure, how can I do this? I can't find "eclipse/technology-rdf4j-committers" in the members list.

Besides polishing the following points are left:

use VarInts for namespace IDs in value store and reduce max key size before hashing is used (currently 512 bytes which is the max key size of LMDB)
invent a perfomant GC algorithm for values (Which was the initial motivation for this PR)

I've taken a look at the performance of ComplexQuery and found out that LMDB and the MemoryStore choose ever so slightly different query plans. This seems to make a massive difference for some reason, although it's not quite obvious why.
The main difference is that the MemoryStore does a join without any common variables as the initial (bottom) join, while LMDB has moved this join up a few nodes in the plan. I've added a visualisation of the two plans below.
In order to force the same plan to be used I disabled the query optimizer and ran both benchmarks again. The results are incredibly close 1030ms for LMDB and 930ms for the MemoryStore. This is very promising @kenwenzel. I would love to contribute some more benchmarks to your branch if you'd be ok with that.
MemoryStore

LMDB

If you want to produce these visualisations yourself you can take a look at the query explanation documentation here: https://rdf4j.org/documentation/programming/repository/#explaining-queries

I've taken a look at the performance of ComplexQuery and found out that LMDB and the MemoryStore choose ever so slightly different query plans. This seems to make a massive difference for some reason, although it's not quite obvious why.
The main difference is that the MemoryStore does a join without any common variables as the initial (bottom) join, while LMDB has moved this join up a few nodes in the plan. I've added a visualisation of the two plans below.
In order to force the same plan to be used I disabled the query optimizer and ran both benchmarks again. The results are incredibly close 1030ms for LMDB and 930ms for the MemoryStore. This is very promising @kenwenzel. I would love to contribute some more benchmarks to your branch if you'd be ok with that.

Thank you for the visualizations :-) What do you mean by ComplexQuery?

@hmottestad Thank you for the visualizations :-) What do you mean by ComplexQuery? I thought that I've fixed the issue.
BTW, cost estimates are currently done by iterating the first 1000 values, then moving to the last value and then extrapolating based on the difference in subject, predicate, object or context ids.

You've fixed the issue @kenwenzel. I was just running the benchmarks locally and your LMDB store is faster than the MemoryStore. After investigating I found out that the query plans are slightly different and I currently digging into it a bit trying to figure out what would make it faster for the MemoryStore. Even with caching I would still expect an in-memory database than a disk based one.

@kenwenzel this is up to you, but if you are comfortable with some of us pairing up to get this polished, can you give @eclipse/technology-rdf4j-committers access to your forked repo so we can push to your branch?

@jeenbroekstra Sure, how can I do this? I can't find "eclipse/technology-rdf4j-committers" in the members list.

Argh. I assumed too much on behalf of github, thinking that it would be possible to add the team as a whole as collaborators on your fork. But I've just given it a try and it looks like that's not possible and can only give access to individual collaborators.
Something simpler we can try first: there should be a checkbox "Allow edits by maintainers" visible for you on your pull request somewhere. If you enable that option it should be possible for me and other committers to add commits to your branch.

Never mind, I just gave it a quick try with some small changes to the pom.xml and it already worked! I've pushed an update to your branch, don't forget to pull before you continue working on it.

Never mind, I just gave it a quick try with some small changes to the pom.xml and it already worked! I've pushed an update to your branch, don't forget to pull before you continue working on it.

How should we move forward? Should I first rewrite the commit messages and potentially squash my commits before you and @hmottestad base your work on them?

Would be nice if you did that before I do any work. Could you also rebase onto develop so we get the latest changes from there too?
If you need to force push in the future then a heads up would be nice, but it's your branch after all!
FYI: I'm not about to do anything on your branch just yet.

I'll stay off your branch for now as well.

@jeenbroekstra The SPARQL compliance tests for LMDB fail now due to the optional dependencies to LWJGL native libraries.
Should I add those dependencies to the compliance tests or just remove the compliance tests for LMDB?

@kenwenzel Ah, yes, adding them as runtime dependence in the compliance test module should fix that

# LMDB
Benchmark                                                       Mode  Cnt      Score      Error  Units
TransactionsPerSecondBenchmark.largerTransaction               thrpt    5     31.845 ±    0.861  ops/s
TransactionsPerSecondBenchmark.largerTransactionLevelNone      thrpt    5     44.121 ±    2.161  ops/s
TransactionsPerSecondBenchmark.mediumTransactionsLevelNone     thrpt    5  19986.561 ±  774.509  ops/s
TransactionsPerSecondBenchmark.transactions                    thrpt    5  21565.831 ±  815.203  ops/s
TransactionsPerSecondBenchmark.transactionsLevelNone           thrpt    5  38754.785 ± 1597.893  ops/s
TransactionsPerSecondBenchmark.veryLargerTransactionLevelNone  thrpt    5      0.268 ±    0.016  ops/s

Benchmark                                               Mode  Cnt     Score     Error  Units
QueryBenchmark.complexQuery                             avgt    5    33.494 ±   2.694  ms/op
QueryBenchmark.distinctPredicatesQuery                  avgt    5  1231.341 ± 120.133  ms/op
QueryBenchmark.groupByQuery                             avgt    5    18.464 ±   1.735  ms/op
QueryBenchmark.removeByQuery                            avgt    5   231.470 ±  14.651  ms/op
QueryBenchmark.removeByQueryReadCommitted               avgt    5   683.210 ±  25.924  ms/op
QueryBenchmark.simpleUpdateQueryIsolationNone           avgt    5   514.449 ±  20.417  ms/op
QueryBenchmark.simpleUpdateQueryIsolationReadCommitted  avgt    5  1095.385 ± 216.371  ms/op

Findings:

insertion speed also improved for non-hashed values (benchmarks only generate very short literals)
degraded query perfomance because of 2 hops for lookups of hashed values (id -> hash and hash+id -> value)
TODO: rework storing of hashed values again

This is now feature complete in comparison to the native store. Two open points are:

add a perfomant GC algorithm for values (Which was the initial motivation for this PR)
maybe drop the complex logic for transaction isolation and solely rely on LMDB's transactions (problem: only one writer at a time is allowed)

Point 1 is really necessary while point 2 would be nice to have.
How should we proceed? Do you want to merge this PR first and then we add GC and maybe other features?

Previously we've marked features as experimental or for internal use only. That way we can merge early while still allowing for large changes later on.

As the work is stand alone as a new sail, I agree with @hmottestad to merge it with an experimental tag.

Great work @kenwenzel , thanks!


Smartcard redirection not working

Describe the bug
I am connecting to a Windows 10 desktop with smartcard passthrough, the desktop runs Activclient. Once connected Activclient freezes trying to read the card. There are times when I can remove and reinsert the card and the card will show up. Then if I try to rdp from the windows machine to another 2019/2012 windows server I get the same errors. From the freerdp logs I am seeing SCARD_E_CANCELLED errors. From the pcscd logs I am seeing rv=SCARD_E_CANCELLED and rv=SCARD_E_SHARING_VIOLATION. I have attached both logs. Please let me know if you need any further information. Thank you.
Expected behavior
The smartcard should be read and not freeze client applications.
Application details

FreeRDP version: FreeRDP version 2.8.1 (2.8.1

Command line used: xfreerdp /f /floatbar /microphone /smartcard:SCM /u: /d: /v: /sound -sec-rdp /log-level:trace

Output of xfreerdp /buildconfig

Build configuration: BUILD_TESTING=OFF BUILTIN_CHANNELS=ON HAVE_AIO_H=1 HAVE_EXECINFO_BACKTRACE=1 HAVE_EXECINFO_BACKTRACE_SYMBOLS=1 HAVE_EXECINFO_BACKTRACE_SYMBOLS_FD=1 HAVE_EXECINFO_H=ON HAVE_EXECINFO_HEADER=1 HAVE_FCNTL_H=1 HAVE_GETLOGIN_R=1 HAVE_GETPWUID_R=1 HAVE_INTTYPES_H=1 HAVE_JOURNALD_H=TRUE HAVE_MATH_C99_LONG_DOUBLE=1 HAVE_POLL_H=1 HAVE_PTHREAD_MUTEX_TIMEDLOCK=ON HAVE_PTHREAD_MUTEX_TIMEDLOCK_LIBS= HAVE_PTHREAD_MUTEX_TIMEDLOCK_SYMBOL=1 HAVE_SYSLOG_H=1 HAVE_SYS_EVENTFD_H=1 HAVE_SYS_FILIO_H= HAVE_SYS_MODEM_H= HAVE_SYS_SELECT_H=1 HAVE_SYS_SOCKIO_H= HAVE_SYS_STRTIO_H= HAVE_SYS_TIMERFD_H=1 HAVE_TM_GMTOFF=1 HAVE_UNISTD_H=1 HAVE_XI_TOUCH_CLASS=1 WITH_ALSA=ON WITH_CAIRO=ON WITH_CCACHE=ON WITH_CHANNELS=ON WITH_CLANG_FORMAT=ON WITH_CLIENT=ON WITH_CLIENT_AVAILABLE=1 WITH_CLIENT_CHANNELS=ON WITH_CLIENT_CHANNELS_AVAILABLE=1 WITH_CLIENT_COMMON=ON WITH_CLIENT_INTERFACE=OFF WITH_CUPS=ON WITH_DEBUG_ALL=OFF WITH_DEBUG_CAPABILITIES=OFF WITH_DEBUG_CERTIFICATE=OFF WITH_DEBUG_CHANNELS=OFF WITH_DEBUG_CLIPRDR=OFF WITH_DEBUG_DVC=OFF WITH_DEBUG_KBD=OFF WITH_DEBUG_LICENSE=OFF WITH_DEBUG_MUTEX=OFF WITH_DEBUG_NEGO=OFF WITH_DEBUG_NLA=OFF WITH_DEBUG_NTLM=OFF WITH_DEBUG_RAIL=OFF WITH_DEBUG_RDP=OFF WITH_DEBUG_RDPDR=OFF WITH_DEBUG_RDPEI=OFF WITH_DEBUG_RDPGFX=OFF WITH_DEBUG_REDIR=OFF WITH_DEBUG_RFX=OFF WITH_DEBUG_RINGBUFFER=OFF WITH_DEBUG_SCARD=OFF WITH_DEBUG_SND=OFF WITH_DEBUG_SVC=OFF WITH_DEBUG_SYMBOLS=OFF WITH_DEBUG_THREADS=OFF WITH_DEBUG_TIMEZONE=OFF WITH_DEBUG_TRANSPORT=OFF WITH_DEBUG_TSG=OFF WITH_DEBUG_TSMF=OFF WITH_DEBUG_TSMF=OFF WITH_DEBUG_TSMF_AVAILABLE=0 WITH_DEBUG_URBDRC=OFF WITH_DEBUG_WND=OFF WITH_DEBUG_X11=OFF WITH_DEBUG_X11_CLIPRDR=OFF WITH_DEBUG_X11_LOCAL_MOVESIZE=OFF WITH_DEBUG_XV=OFF WITH_DSP_EXPERIMENTAL=OFF WITH_EVENTFD_READ_WRITE=1 WITH_FAAC=OFF WITH_FAAD2=OFF WITH_FFMPEG=OFF WITH_GFX_H264=OFF WITH_GPROF=OFF WITH_GSM=OFF WITH_GSSAPI=OFF WITH_ICU=ON WITH_IPP=OFF WITH_JPEG=ON WITH_LAME=OFF WITH_LIBRARY_VERSIONING=ON WITH_LIBSYSTEMD=ON WITH_MACAUDIO=OFF WITH_MACAUDIO=OFF WITH_MACAUDIO_AVAILABLE=0 WITH_MANPAGES=ON WITH_MBEDTLS=OFF WITH_OPENCL=OFF WITH_OPENH264=OFF WITH_OPENSLES=OFF WITH_OPENSSL=ON WITH_OSS=ON WITH_PAM=ON WITH_PCSC=ON WITH_PROFILER=OFF WITH_PROXY=OFF WITH_PULSE=ON WITH_SAMPLE=OFF WITH_SANITIZE_ADDRESS=OFF WITH_SANITIZE_ADDRESS_AVAILABLE=1 WITH_SANITIZE_MEMORY=OFF WITH_SANITIZE_MEMORY_AVAILABLE=1 WITH_SANITIZE_THREAD=OFF WITH_SANITIZE_THREAD_AVAILABLE=1 WITH_SERVER=ON WITH_SERVER_CHANNELS=ON WITH_SERVER_INTERFACE=ON WITH_SHADOW=ON WITH_SMARTCARD_INSPECT=OFF WITH_SOXR=OFF WITH_SSE2=ON WITH_SWSCALE=OFF WITH_THIRD_PARTY=OFF WITH_VALGRIND_MEMCHECK=OFF WITH_VALGRIND_MEMCHECK_AVAILABLE=1 WITH_VERBOSE_WINPR_ASSERT=ON WITH_WAYLAND=ON WITH_WINPR_TOOLS=ON WITH_X11=ON WITH_XCURSOR=ON WITH_XDAMAGE=ON WITH_XEXT=ON WITH_XFIXES=ON WITH_XI=ON WITH_XINERAMA=ON WITH_XKBFILE=ON WITH_XRANDR=ON WITH_XRENDER=ON WITH_XSHM=ON WITH_XTEST=ON WITH_XV=ON WITH_ZLIB=ON
Build type:          RelWithDebInfo
CFLAGS:              -g -O2 -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Wall -Wno-unused-result -Wno-unused-but-set-variable -Wno-deprecated-declarations -fvisibility=hidden -Wimplicit-function-declaration -Wredundant-decls -g -fno-omit-frame-pointer -DWINPR_DLL
Compiler:            GNU, 12.2.0
Target architecture: x64

OS version connecting to (server side): Windows 10

I've attached freerdp and pcscd logs

Environment (please complete the following information):

OS: Ubuntu 22.10 - 6.1.11 x64
pcsc-lite version 1.9.9

Logs:
pcscd_trace.log
freerdp-trace.log

@wescode sorry for the delay, smartcard is kind of a hard topic with RDP redirection.
so. might I ask if this feature works if you connect with a mstsc (windows RDP client)?
need that as a baseline to know if we miss something  on our side (PCSC lite is not 100% compatible with the windows smarctard layer) or if it just does not work over RDP at all.

@wescode sorry for the delay, smartcard is kind of a hard topic with RDP redirection. so. might I ask if this feature works if you connect with a mstsc (windows RDP client)? need that as a baseline to know if we miss something on our side (PCSC lite is not 100% compatible with the windows smarctard layer) or if it just does not work over RDP at all.

No problem. Yes it works fine if I use windows mstsc. I am currently using a windows VM to do just that until this bug can hopefully be fixed. Thanks.

@wescode ok, good to know.
one simple (but kind of wtf) thing you can try is setting /client-build-number:<windows build number>.
the rdp server changes behaviour depending on version it receives, so maybe you´re better off emulating an older windows build. see MS-RDPECS for details

@akallabeth I forgot to include in my original post that I did indeed try several different build numbers and it didn't seem to make a difference.

If there is any other information I can provide in helping debug this issue let me know.

In case some further data for analysis is required, please find attached my trace files.
The client is running FreeBSD 13.1 and I'm connecting to Windows10. This works fine, however I cannot see any devices.
I'm trying to use the smartcard when connecting to a Citrix server - that's where things go wrong and the rdp logs show timeout messages.
When I use a windows-client to connect to the Windows10 machine, everything works fine.
Specifying the client-build-number also has no effect.
pcscd.log
freerdp.log

thank you for the logs.
smartcard is kind of hard to debug because:

the windows and linux smartcard drivers often differ in feature support for the same hardware (hardware dependent issues)
there are subtle differences in data exchanged (sometimes, horray for heisenbugs)
there are subtle differences in API
there are RDP protocol requirements pcsc-lite can not support fully
you have RDP protocol differences depending on /client-build-number
each software uses different means to access a smartcard

I don´t know if I can find something here, as I don´t exactly know how the calls should look like in your case, but maybe the listing above can help to identify additional data that can be gathered to debug this.
(ideal would be a call trace on the windows side with mstsc remoting the smartcard both, client OS and server OS side so we know how the calls should look like)

I can try to create some traces on windows machines. However I do not know how I would do this.
Do you have a hint how to create useful mstsc traces?

@mmsoft3 sorry, no I don´t know. but you should not trace mstsc but the PCSC API calls on windows.

Not sure how helpful this is, but I noticed in the buildconfig posted above (way above), it shows
WITH_SMARTCARD_INSPECT=OFF

@WingsLikeEagles no, that is a debugging option (log all smartcard calls)

I tried to investigate the issue a bit further.
Currently, I think at least my issue might be a problem with pcscd and my smart card.
Even when connecting the sc reader directly to a linux machine with citrix installed, the smart card does not work correctly.
My temporary solution to this issue is sharing the USB device over network with usbip.
In this scenario, I can import the virtual USB device to the Windows VM running Citrix.
This gives some flexibility, however I still cannot get rid of the Windows VM as a Citrix client.

@mmsoft3 base requirement is a working smartcard with pcscd. only alternative (if you have the key/cert) is a emulated smartcard (freerdp can be compiled with support for that) or the virtual smartcard for linux


rocfft_aot_helper rocfft_kernel_cache.db : Assertion `thread != NULL && "cannot lock() from (null)"' failed

What is the expected behavior

successful build

What actually happens
-  [61/389] cd /var/tmp/portage/sci-libs/rocFFT-5.4.0/work/rocFFT-rocm-5.4.0_build/library/src && /var/tmp/portage/sci-libs/rocFFT-5.4.0/work/rocFFT-rocm-5.4.0_build/library/src/rocfft_aot_helper /var/tmp/portage/sci-libs/rocFFT-5.4.0/work/rocFFT-rocm-5.4.0_build/library/src/rocfft_kernel_cache.db /var/tmp/portage/sci-libs/rocFFT-5.4.0/work/rocFFT-rocm-5.4.0_build/library/src/rocfft_rtc_helper gfx900
FAILED: library/src/rocfft_kernel_cache.db /var/tmp/portage/sci-libs/rocFFT-5.4.0/work/rocFFT-rocm-5.4.0_build/library/src/rocfft_kernel_cache.db
cd /var/tmp/portage/sci-libs/rocFFT-5.4.0/work/rocFFT-rocm-5.4.0_build/library/src && /var/tmp/portage/sci-libs/rocFFT-5.4.0/work/rocFFT-rocm-5.4.0_build/library/src/rocfft_aot_helper /var/tmp/portage/sci-libs/rocFFT-5.4.0/work/rocFFT-rocm-5.4.0_build/library/src/rocfft_kernel_cache.db /var/tmp/portage/sci-libs/rocFFT-5.4.0/work/rocFFT-rocm-5.4.0_build/library/src/rocfft_rtc_helper gfx900
rocfft_aot_helper: /var/tmp/portage/dev-util/hip-5.4.0/work/ROCclr-rocm-5.4.0/cmake/../thread/monitor.hpp:180: bool amd::Monitor::tryLock(): Assertion `thread != NULL && "cannot lock() from (null)"' failed.

How to reproduce

try build rocFFT 5.4.0 from source

Environment

Hardware
description

GPU
Radeon Vega Frontier Edition

CPU
AMD Ryzen 9 3900XT

Software
version

ROCK
v5.4.0

ROCR
v5.4.0

HCC
v5.4.0

Library
v5.4.0

PS. rocFFT 5.3.3 build succesfull, but 5.4.0 not.

With env var export ROCFFT_RTC_PROCESS=1 https://github.com/ROCmSoftwarePlatform/rocFFT/blob/rocm-5.4.0/library/src/rtc_cache.cpp#L377 case pass away.

You can to set AMD_LOG_LEVEL = LOG_DEBUG next time to debug this code.

Hi @perestoronin , thanks for the bug report.
This is ultimately a problem in hipRTC.  As you've discovered, the ROCFFT_RTC_PROCESS=1 workaround will help get rocFFT built.
Once built, rocFFT may run into the same problem at runtime, if it needs to compile any kernels.  The same workaround would be possible there too.
We'll aim to get the underlying hipRTC issue fixed in ROCm 5.4.1.


command line test cover

https://github.com/xudafeng/command-line-test

https://github.com/reliablejs/reliable-macaca-slave/blob/master/test/reliable-macaca-slave.test.js


Update token issue

Should be issued as json response
https://github.com/indieweb/indieauth-client-php#issuing-an-access-token

Fixed


softkinetic: 0.6.2-0 in 'indigo/distribution.yaml' [bloom]

Increasing version of package(s) in repository softkinetic to 0.6.2-0:

upstream repository: https://github.com/ipa320/softkinetic.git
release repository: https://github.com/ipa320/softkinetic-release.git
distro file: indigo/distribution.yaml
bloom version: 0.5.21
previous version for package: null

softkinetic
* Update CHANGELOG.rst
* Contributors: Nadia Hammoudeh García

softkinetic_camera

No changes

holding until indigo sync is finished


Implement MVP Invite Only auth

Authentication is one of those Platform level topics that can take a lot of effort and consideration. Do we use a dedicated Auth service and depend on their SDKs? Do we hand roll something more specialized?
I think we start with a simple "Invite Only" auth flow that skips the complications of signup components for now. We can utilize Firebase as our authentication platform, as they provide a decent amount of flexibility. I can use the Firebase SDK to simplify the implementation.

I'm kind of curious to implement my own JWT parsing and decoding/verification.


[Cosmos] Cosmos retry policy

Adding support for connection retry as a Pipeline policy rather than adding a urllib Retry object to the session.
Should be 99% backwards compatible.
Tests will require this fix for azure-core in order to pass: PR #7542

Can one of the admins verify this patch?

Yeesh.... rebase needed....


All messages pending until restart

Ever since I started using a Rabbitmq cluster, I see  that all the messages start showing as pending after a while. I will look at Rabbitmq admin though and I see that Postal IS connected. But, it doesn't send messages out until I restart Postal. Any idea which of all these logs I should be looking at to see why it won't send?
The Rabbitmq setup is 3 nodes load balanced behind HAPROXY. I also use the Consul plugin so it knows which nodes are available and HAPROXY will only serve up healthy nodes.
FYI.. I do not have to do anything to Rabbitmq to make things work. I restart postal, and then it works. So it definitely seems like a problem with Postal and its ability to get reconnected to RMQ.
But a valid first step here is to see what the logs are saying. Thx

UPDATE:
Okay, I did something that I should have known would cause things to break. On my HAPROXY nodes (inside the swarm), I specified some internal DNS servers that resolve names on my private network. This was so it would easier (so I thought) to point a service at a non-swarm private ip. Little did I know that the additional DNS servers specified conflicted with the swarm nodes names. This was causing all kinds of crazy problems with the swarm and all the services published.
I removed the private DNS servers from the HAPROXY nodes, and like magic, everything functioning again rock solid. Another lesson learned with Docker Swarm. DO NOT mess with DNS. Just create host entries in the stack or compose file if you really don't want to hard code the IP of a non swarm server.

Thanks for the information, your environment is quite specific but a reminder to be quite cautious of DNS is always good!


Verifiably unpredictable ids (revised)

To continue our conversation from the previous ticket,
I'd be interested in seeing what some other projects are doing.  In my project, I sign the values and include a hash of the signing key's fingerprint in the key.
However, without changes at the protocol level it's still possible for malicious entities to intercept traffic.  If an application signs and validates values then a hostile node can't forge data, but it can deny the existence of data which is enough to censor the network.
This patch changes node ID generation and adding to the routing table.
Each node creates a secret value (the seed).  From that, it creates a private signing key and a public verification key.  The node's ID is (hash of the verification key, the verification key).
When the local node encounters a new remote node, it sends a challenge message (random) to the new node.  The new node signs the challenge and sends the signature.  The local node validates the signature before adding the node to its routing table.
This serves two functions.  As with the previous patch, because the ID value used for routing is a hash, a malicious entity cannot choose it freely, say, to match the key of a known value.  Secondly, completing the challenge to new nodes requires knowledge of the secret key, so a malicious entity couldn't craft a node to impersonate another known node.
This is a significant departure from the original paper so I'd understand if it's not the sort of thing you want to include here.  I think censorship is a concern for many use cases though, so it may be of practical interest.
And, as with the previous patch, I'm not a security expert so please take this with a grain of salt.

Thanks @Rendaw!  I'll take a look when I get a chance.


Simple.OData.V4 in Test Project

Hello!
I am working on a project that realizes integration tests through the MSTest framework. Now the problem is, I installed the Simple.OData.V4 NuGet into the Test Project. However, when I run the tests, I get the error message:
Message: Initialization method KiamToKcrmCrudTestCases.KiamKcrmApiReadTestCase.Initialize threw exception. System.AggregateException: System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Unable to load OData adapter from assembly Simple.OData.Client.V4.Adapter ---> System.IO.FileNotFoundException: Could not load file or assembly 'Simple.OData.Client.V4.Adapter' or one of its dependencies. The system cannot find the file specified.WRN: Assembly binding logging is turned OFF. To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1. Note: There is some performance penalty associated with assembly bind failure logging. To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
I assume this happens because the assembly is present in the project, but it is not present in the context the test runner uses. Is there any workaround for this problem?

Hi,
You should check that Simple.OData.V4.Client is present in a test run directory. It's been long time since I used MSTest, so but as far as remember it runs tests in a different directory than the project bin folder.
You can check out Simple.OData.Client unit and integration tests. They all run fine, but they are using XUnit.

I found a solution. Apparently, dynamic assembly loading and any assembly loading of assemblies which MSTest does not recognize as "necessary" is quite ugly. I had to add the assembly as file to the project and add it as a deployment-item. But it works now. Thank you for your help!


Scrollbar not showing on load when dynamic width

The requirement is to have a dynamic width size for a scrollable area.
But scrollbar-container div only shows when scrollarea-content is dragged.
How can we enforce the scrollbar div to be shown on page load?

I implemented a workaround for a similar issue, perhaps it works in your case, too:
componentDidMount(){
    setTimeout(this.scrollArea.refresh, 100);
}

render(){
...
<ScrollArea ref={(component) => {this.scrollArea = component.scrollArea}} ... >
  ...
</ScrollArea>
...
}

@ChristianIO yes, this works. I thought of doing it like this but it seems to be a hack. Do you understand why the dynamic width prevents the scrollbar from showing? I can't understand the lifecycle.

@janejanejane In my case, pictures have not been loaded fast enough and the scrollbar could not render with the correct height. I rechecked my code due to your question and found a bug, that is now corrected in the example code above. Because component can be null (react calls the ref callback with null to release memory space), we have to check against it.
Actually I implemented another workaround today for my issue: Because I know the height of the pictures, I set this height of the img tag via css styles. Now the scrollbar is able to compute the correct overall height, even when the pictures have not been loaded yet.

I am unable to do the same as what you did. When I put a width limit in the css, say 915px, then not put a computation for the real width, the contents are not fully shown (of course). If I then put the computation with the css width, the horizontal scrollbar is still not displaying on first load. I know something is wrong but I can't figure it out for now.


ECONNREFUSED <IP_ADDRESS>:9222

Hi,
I deployed the package and for some reason chrome refuses to connect.
I have logged all the steps and this is the output(Added some custom log stepps). Link to pastebin for output.
Do you have any experience with this error? or know how to fix it.
With best regards,
Amarildo

Hi @AmarildoK,
Hm.. I'm not sure what to say. One thing I noticed in your logs is that the function was running for 62267.33 ms which is a long time for nothing to happen. It might help if you could share your code?

Hi @adieuadieu thanks for the response,
I have forked the repo and committed my changes.  Haven't changed much expect I get the url directly from event object. I plan on triggering the event trough our own back-end and won't need to extract the variables from queryStringParameters.
I also deploy the code with a windows machine & serverless as described in the readme file, idk if this matters when the code gets build.
[https://github.com/AmarildoK/serverless-chrome](Link to the fork)
With best regards,
Amarildo

Thanks for sharing the code, @AmarildoK. Indeed, not much changed.
Could you share here your config.js file here, as well as let me know how you invoke the lambda function (since you're not doing it via API Gateway?) What's the event payload that you're invoking the function with?
One thing you can try to help debug is un-comment lines 93 to 105 in chrome.js. This will cause stdout and stderr to get logged to CloudWatch. However, this will cause your function to timeout, because the event loop will never clear, since node will be continuously listening to those events. But, there might be some hints about what's happening with headless Chrome that gets logged.

Hej @adieuadieu,
config file is also bar minimum
export default {   logging: true, }
I invoke my functions trough the test button in lambda management and pass a simple object
{"url":"http://www.randomwebsite.com"}
I also un-commented the lines and for some unknown reason  stdout and stderr are null. In the logs you can see that
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: Cannot read property 'on' of null
The full log. I also logged the instance of the chrome variable.
`START RequestId: b9e93f5c-5203-11e7-87f1-377eb8c52f36 Version: $LATEST
2017-06-15T19:49:23.659Z	b9e93f5c-5203-11e7-87f1-377eb8c52f36	CHROME_PATH /var/task/headless-chrome/headless_shell
2017-06-15T19:49:23.684Z	b9e93f5c-5203-11e7-87f1-377eb8c52f36
$ ls /tmp
total 8.0K
drwxr-xr-x 21 root root 4.0K Jun 15 16:01 ..
drwx------ 2 sbx_user1070 476 4.0K Jun 15 19:49 .
2017-06-15T19:49:23.690Z	b9e93f5c-5203-11e7-87f1-377eb8c52f36
$ ps lx
F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
4 477 1 0 20 0<PHONE_NUMBER>6 ep_pol Ssl ? 0:00 /var/lang/bin/node --max-old-space-size=1228 --max-semi-space-size=77 --max-executable-size=154 --expose-gc /var/runtime/node_modules/awslambda/index.js
0 477 12 1 20 0<PHONE_NUMBER> - R ? 0:00 ps lx
2017-06-15T19:49:23.701Z	b9e93f5c-5203-11e7-87f1-377eb8c52f36	no running chrome yet?
2017-06-15T19:49:23.711Z	b9e93f5c-5203-11e7-87f1-377eb8c52f36	Is Chrome already running? false
2017-06-15T19:49:23.713Z	b9e93f5c-5203-11e7-87f1-377eb8c52f36	instance ChildProcess {
domain: null,
_events: {},
_eventsCount: 0,
_maxListeners: undefined,
_closesNeeded: 1,
_closesGot: 0,
connected: false,
signalCode: null,
exitCode: null,
killed: false,
spawnfile: '/bin/sh',
_handle: Process { owner: [Circular], onexit: [Function], pid: 13 },
spawnargs:
[ '/bin/sh',
'-c',
'/var/task/headless-chrome/headless_shell --headless --disable-gpu --window-size=1280x1696 --no-sandbox --user-data-dir=/tmp/user-data --hide-scrollbars --enable-logging --log-level=0 --v=99 --single-process --data-path=/tmp/data-path --ignore-certificate-errors --homedir=/tmp --disk-cache-dir=/tmp/cache-dir --remote-debugging-port=9222' ],
pid: 13,
stdin: null,
stdout: null,
stderr: null,
stdio: [ null, null, null ] }
2017-06-15T19:49:23.724Z	b9e93f5c-5203-11e7-87f1-377eb8c52f36	(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: Cannot read property 'on' of null
END RequestId: b9e93f5c-5203-11e7-87f1-377eb8c52f36
REPORT RequestId: b9e93f5c-5203-11e7-87f1-377eb8c52f36	Duration: 67.72 ms	Billed Duration: 100 ms Memory Size: 1536 MB	Max Memory Used: 28 MB`

@AmarildoK Hm.. definitely something isn't right.
Try the following config.js:
import Cdp from 'chrome-remote-interface'

export default {
  logging: true,
  async handler (event) {

    const versionInfo = await Cdp.Version()

    return {
      statusCode: 200,
      body: JSON.stringify({
        versionInfo,
      }),
      headers: {
        'Content-Type': 'application/json',
      },
    }
  },
}

This should print out the Chrome version info.

@adieuadieu I deployed the same code with the addition of your config.js change. From a mac and from a windows machine and for some reason when I deploy from a the mac the script gives me a return
{   "statusCode": 200,   "body": "{\"versionInfo\":{\"Browser\":\"HeadlessChrome/60.0.3095.0\",\"Protocol-Version\":\"1.2\",\"User-Agent\":\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/60.0.3095.0 Safari/537.36\",\"V8-Version\":\"6.0.184\",\"WebKit-Version\":\"537.36 (@947514553066c623a85712d05c3a01bd1bcbbffc)\"}}",   "headers": {     "Content-Type": "application/json"   } }
and from when I deploy it from a the windows machine it throws a timeout every time  I invoke the function.
{   "errorMessage": "2017-06-16T11:40:26.951Z 84567d9d-5288-11e7-ade6-d9a1173a80cc Task timed out after 30.00 seconds" }
Now deploying from a mac machine fixed it for me don't know why really. Both machines have node 7.4.0. Since I am on windows I generated the config file myself(haven't installed the ubuntu bash shell yet on windows). If I'm not mistaken you just generate a random uuid and make a simple object and write it to the config.js file. Are my assumptions wrong?
Thanks for the help.

@AmarildoK glad to hear you've had some success. To be honest, I've not yet had the chance to test on Windows—but I do want to support that platform.
The random uuid is actually not used anymore—so you don't need it. All you need is a config.js file which specifies your handler function code. (Also, note that: v1.0 of serverless-chrome will remove the config.js and postinstall script entirely.)
Perhaps you could share with me the artefact (zip package) which gets generated by Serverless on Windows: Install serverless globally with npm install serverless -g, then in the project directory run serverless package. This will generate the zip package without actually deploying. You'll find the zip in your-project-directory/.serverless/**something-something**.zip

@adieuadieu you can get the zip on this location. https://transferxl.com/08v1D5jth82yNt

Thanks @adieuadieu for serverless-chrome. I have the same problem as @AmarildoK - i.e. works when deployed from ubuntu, timesout when deployed from windows. I've done some poking about and not come up with a firm conclusion yet but wanted to share some insights.
TLDR chrome is not spawning correctly inside a lambda when deployed from windows - don't think cloudformation / webpack are the problem. My hunch is that either there's a jszip problem on windows or perhaps its the exec permission not being set correctly. I'm going to fiddle with the later now.
There are 4 processes that happen during deployment:
1. webpack
2. creation of cloudformation .ymls
3. creation of zip
4. anything else serverless might do whilst uploading .serverless/ to AWS
Webpack seems fine:

my .serverless/serverless-chrome.zip/handler.js is the same on windows and unbuntu - barring hashes and timestamps

Cloudformation .ymls seem fine:

again no real differences

Zip

This thread mentions that there may be issues with zipping on windows
Hmm.. I wonder whether it gets the rwx permissions correct inside the lambda after deployment? This has come up on other forums - can't remember where now... docker image building I think.

Serverless magic
TBH I'dunno what else goes on here but to check I also downloaded the files from the S3 ServerlessDeploymentBucket's (to which serverless uploads files). The only change appears to be whitespace stripping on the cloudformation-template-update-stack.json.
Finally - I tried to get the code working in a local amazonlinux docker image. I pulled amazonlinux, got headless_shell to run inside the shell and changed the src/ code from being "handler-based" to a CLI callable app (so I can run it in node directly). I've managed to get this to work however on my first attempt I forgot to include the spawn code and I got exactly the same ECONNREFUSED error mentioned in the title of this issue (which is how I found it). This squares with @AmarildoK's observations above of chrome not running in the windows deployed lambda.

Hi @ilanc thank you for such a thorough report! I'm away from my computer at the moment but will take a closer look at what you've found tomorrow. However, I quickly wanted to mention: v1.0 is in development in the develop branch and comes with a Serverless plugin: serverless-plugin-chrome And a standalone package @serverless-chrome/lambda. Do you still encounter the issue when using those? You can find more documentation in the develop branch.

On 2017/07/10, at 22:06, Ilan Copelyn<EMAIL_ADDRESS>wrote:

Thanks @adieuadieu for serverless-chrome. I have the same problem as @AmarildoK - i.e. works when deployed from ubuntu, timesout when deployed from windows. I've done some poking about and not come up with a firm conclusion yet but wanted to share some insights.
TLDR chrome is not spawning correctly inside a lambda when deployed from windows - don't think cloudformation / webpack are the problem. My hunch is that either there's a jszip problem on windows or perhaps its the exec permission not being set correctly. I'm going to fiddle with the later now.
There are 4 processes that happen during deployment:

webpack
creation of cloudformation .ymls
creation of zip
anything else serverless might do whilst uploading .serverless/ to AWS

Webpack seems fine:
my .serverless/serverless-chrome.zip/handler.js is the same on windows and unbuntu - barring hashes and timestamps
Cloudformation .ymls seem fine:
again no real differences
Zip
This thread mentions that there may be issues with zipping on windows
Hmm.. I wonder whether it gets the rwx permissions correct inside the lambda after deployment? This has come up on other forums - can't remember where now... docker image building I think.
Serverless magic
TBH I'dunno what else goes on here but to check I also downloaded the files from the S3 ServerlessDeploymentBucket's (to which serverless uploads files). The only change appears to be whitespace stripping on the cloudformation-template-update-stack.json.
Finally - I tried to get the code working in a local amazonlinux docker image. I pulled amazonlinux, got headless_shell to run inside the shell and changed the src/ code from being "handler-based" to a CLI callable app (so I can run it in node directly). I've managed to get this to work however on my first attempt I forgot to include the spawn code and I got exactly the same ECONNREFUSED error mentioned in the title of this issue (which is how I found it). This squares with @AmarildoK's observations above of chrome not running in the windows deployed lambda.
―
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Thanks @adieuadieu I'll have a look. I can confirm that the windows deployed binary doesn't have exec permission set. A quick hack would be to set the exec permission just prior to spawn. I'm sure this has happened elsewhere - there may be a better fix to get the rwx permissions correctly set inside the zip.

I'll give the develop stuff a whirl tomorrow. I just broke the unix version - by uncommenting the debugging code in chrome.js. - same error as above (i.e. TypeError: Cannot read property 'on' of null). This causes the same behaviour as in this issue - i.e. timeouts rather than CloudWatch log errors. This is a little confusing to the user. I suspect that the try/catch handlers are not functioning as intended in conjunction with various async functions. I've had issues with this in the past too. I'll try find some notes

Looks like the file permission thing on Windows is still an open issue in Serverless. I'll add some code to check for exec permission and chmod if missing.

Here's my total hack workaround (in chrome.js) - nb:

you have to copy not rename, and I added
I added an extra dependency = node-cp

var cp = require('cp')

const INSTALLED_CHROME_PATH = path.resolve('./headless-chrome/headless_shell')
const INSTALLED_CHROME_DEP = path.resolve('./headless-chrome/libosmesa.so')
const CHROME_PATH = '/tmp/headless_shell'; // process.env.CHROME_PATH && path.resolve(process.env.CHROME_PATH)
const CHROME_DEP = '/tmp/libosmesa.so'
const HEADLESS_URL = 'http://<IP_ADDRESS>:9222'
const PROCESS_STARTUP_TIMEOUT = 1000 * 5

var checkPermission = function (file, mask, cb) {
  fs.stat(file, function (error, stats) {
    if (error) {
      cb(error, false);
    } else {
      cb(null, !!(mask & parseInt((stats.mode & parseInt("777", 8)).toString(8)[0])));
    }
  });
};

function hackExecutableChrome() {
  console.log(INSTALLED_CHROME_PATH, 'exists', fs.existsSync(INSTALLED_CHROME_PATH));
  console.log(INSTALLED_CHROME_DEP, 'exists', fs.existsSync(INSTALLED_CHROME_DEP));
  if (fs.existsSync(CHROME_PATH))
    return;
  try {
    console.log('Move chrome to /tmp: ', INSTALLED_CHROME_PATH);
    cp.sync(INSTALLED_CHROME_PATH, CHROME_PATH);
    cp.sync(INSTALLED_CHROME_DEP, CHROME_DEP);
    // { "errno": -18, "code": "EXDEV", "syscall": "rename", "path": "/var/task/headless-chrome/headless_shell", "dest": "/tmp/headless_shell" }
    // /tmp must be on a different volume: see https://stackoverflow.com/questions/21071303/node-js-exdev-rename-error
    //fs.renameSync(INSTALLED_CHROME_PATH, CHROME_PATH);
    //fs.renameSync(INSTALLED_CHROME_DEP, CHROME_DEP);
    fs.chmodSync(CHROME_PATH, '555');
    fs.chmodSync(CHROME_DEP, '555');
    checkPermission(CHROME_PATH, 4, function(error, stats) { console.log(CHROME_PATH, 'read', stats); });
    checkPermission(CHROME_PATH, 2, function(error, stats) { console.log(CHROME_PATH, 'write', stats); });
    checkPermission(CHROME_PATH, 1, function(error, stats) { console.log(CHROME_PATH, 'exec', stats); });
    checkPermission(CHROME_DEP, 4, function(error, stats) { console.log(CHROME_DEP, 'read', stats); });
    checkPermission(CHROME_DEP, 2, function(error, stats) { console.log(CHROME_DEP, 'write', stats); });
    checkPermission(CHROME_DEP, 1, function(error, stats) { console.log(CHROME_DEP, 'exec', stats); });
  } catch(e) {
    console.log('Error moving chrome to /tmp');
    printJson(e);
  }
}

function printJson(data) {
	console.log(JSON.stringify(data, null, 4));
}

and call it from spawn():
export async function spawn () {
  hackExecutableChrome();

@ilanc thanks for sharing that. Ah you're right. Good point about copying instead of moving: the location of the lambda function code in /var/task is read-only to node.js. Which is super crappy because it means, as you noted, we have to make a copy of the chrome binary to /tmp before we can chmod it. Argh. That sucks.. It's kind of like a punishment of Windows users for deploying from Windows—which is lame.
I guess there are three things to do here:

Implement a exec-permission check and if not executable, copy to /tmp and make it executable, then using the new executable when spawning chrome.
Figure out how to write a useful unit test for the work-around.
Well-document this limitation for Windows users, as it may cost an extra 100ms (that's a guess) on cold-start.

An alternative to this for-Windows hackery may be to encourage Windows users to a Docker image we could provide. Perhaps based on docker-lambda, we create a Docker image that includes Serverless, and build/deploy the deployment package from a Docker container. Would requires that you have Docker installed—which adds some complexity. Does simplify life for people who wanted to include native node modules in their functions, though.

That's a useful link thanks - I was using https://hub.docker.com/_/amazonlinux/.
Short of adding features to serverless (e.g. in order to support explicit file includes and permissions) and JSZip (i.e. to support explicit permission setting - e.g. in here perhaps) I can't see any other alternative. On the plus such a solution would probably have wide application beyond just serverless-chrome.
I've been a bit sidetracked with a show stopper (for me) which I thought I'd bring to your attention - you can't download files in headless atm :(
https://bugs.chromium.org/p/chromium/issues/detail?id=696481
So for my purposes it looks like openwhisk and non-headless chrome + xvfb is going to be the only solution right now until 696481 is merged.

Serverless actually user archiver not JSZip - see zipService.js.
You can hack your local node_modules to work and just set all files to have 777 permissions:
# serverless-chrome\node_modules\serverless\lib\plugins\package\lib\zipService.js : 59
            mode: 511 // HACK: 511 === 0777 //stats.mode,

Then make sure you deploy using the hacked serverless:
node_modules/.bin/serverless deploy

@ilanc So it the main problem that fs.statSync doesn't correctly tell us that the file is executable on Windows? Seems like there may be a solution here somewhere that could be PRd on the Serverless repo?
I've engaged with the Serverless community on this issue.
About being able to download files: you could pass the URL you want to download back to Node and do the actual downloading in your Nodejs code. Might even offer you better control, depending on what you want to do with the downloaded file. That might be simpler than openwhisk + headed-chrome + xvfb—but I'm biased in favour of serverless-chrome :-P

BTW you don't need serverless-chrome.zip\headless-chrome\libosmesa.so with --disable-gpu


Wrong URL in description

example-dapps.lukso.tech/example-dapp-lsps
instead of
example-dapps.lukso.tech/lsp-dapp

Fixed thanks