sharegpt.com/explore not worked

just open sharegpt.com/explore, it seems the same as https://sharegpt.com/
which means I can't explore the gpt links shared by others

me too

metoo


make package/install fails on photoprism-g20240915

I am able to build with make however when I try to install it fails.
Any ideas how to overcome this ?
# uname -a
FreeBSD photoprism 13.3-RELEASE-p4 FreeBSD 13.3-RELEASE-p4 n257491-41f6a830f8e TRUENAS amd64

# make install
===>  Installing for photoprism-g20240915
===>  Checking if photoprism is already installed
===>   Registering installation for photoprism-g20240915
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/build/app.7c8896e06308b4527cda.js:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/build/app.7c8896e06308b4527cda.js.LICENSE.txt:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Bold/13056-1339.pbf:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Bold/45056-4539.pbf:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Italic/13056-1339.pbf:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Italic/45056-4539.pbf:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Regular/13056-1339.pbf:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Regular/45056-4539.pbf:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Semibold/13056-1339.pbf:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Semibold/45056-4539.pbf:No such file or directory
pkg-static: duplicate file listing: /usr/local/etc/rc.d/photoprism, ignoring
*** Error code 1

Stop.
make: stopped in /root/photoprism-freebsd-port

# make package
===>  Building packages for photoprism-g20240915
===>   Building photoprism-g20240915
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/build/app.7c8896e06308b4527cda.js:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/build/app.7c8896e06308b4527cda.js.LICENSE.txt:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Bold/13056-1339.pbf:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Bold/45056-4539.pbf:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Italic/13056-1339.pbf:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Italic/45056-4539.pbf:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Regular/13056-1339.pbf:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Regular/45056-4539.pbf:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Semibold/13056-1339.pbf:No such file or directory
pkg-static: Unable to access file /root/photoprism-freebsd-port/work/stage/var/db/photoprism/assets/static/font/Open Sans Semibold/45056-4539.pbf:No such file or directory
pkg-static: duplicate file listing: /usr/local/etc/rc.d/photoprism, ignoring
*** Error code 1

Stop.
make: stopped in /root/photoprism-freebsd-port

Thanks. Somehow after a number of make clean install the font file errors disappeared however the .js file errors remained.
I noticed the discrepancy between plist and asset directory contents


Staff notes

Add the ability for staff to leave notes on a user.
This serves as a way for staff to keep track of a user without having to issue an infraction, or message them directly.
Notes will be private and staff-only.
This is similar to how private notes work on user profiles, except this feature will be server-wide and shared between all staff.

Addendum:
Another possible idea would be to have Guru allowed to keep notes as well.
Guru could only view notes stored by other Guru, whereas Staff could view all notes.
Will need approval from Admin for this to be a thing


Context lenght exceded

I am using Mistral 7B and it was configured with context lenght 1500.  So when I asked a questiom it was not able to finish, after the first answer it errored out due to context lenght 1500.
So I tried to set it higher to 7500. But then the computer first fresed and  then the answer was empty.
I am using MacBook 'Pro with M1.

I have the same issue. Even if i just write "hi", i get an error
An error occurred

Input length 1581 exceeds context length 1500


⚠ ClamAV has degraded performance

In 777ce46, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 189 ms

Resolved: ClamAV performance has improved in e54b79c after 5 minutes.


Thanks!

Have been deployed to the online environment，very good，very simple！looking forward more similiar project for developer

是跑在 k8s 上的吗


Upgrading to new OCI 1.1 spec

A new release candidate of OCI 1.1 spec is released.

image-spec v1.1.0-rc.3
image-spec v1.1.0-rc4
distribution-spec v1.1.0-rc.2
distribution-spec v1.1.0-rc.3

We need to discuss how oras-go should support / implement the latest release candidate, or we should hold on for the stable release.

Summary of ORAS Related Changes
image-spec: v1.1.0-rc2 --> v1.1.0-rc4

BREAKING CHANGE Removed artifact manifest
BREAKING CHANGE Removed pre-defined annotation keys

org.opencontainers.artifact.created
org.opencontainers.artifact.description
org.opencontainers.referrers.filtersApplied

Minimum golang version changed to 1.18
Non-distributable layers a.k.a. foreign layers are deprecated
Introduced empty descriptor

Media type: application/vnd.oci.empty.v1+json

artifactType field is introduced to the image manifest. This field MUST be set if the config media type is set to empty
artifactType and subject fields are introduced to the image index
layers field of an image manifest SHOULD NOT be empty, which was a MUST. If the config media type is application/vnd.oci.image.config.v1+json, layers MUST NOT be empty.
Introduced Guidelines for Artifact Usage

Full change set: https://github.com/opencontainers/image-spec/compare/v1.1.0-rc2...v1.1.0-rc4
distribution-spec: v1.1.0-rc1 --> v1.1.0-rc.3

Removed docs to the OCI Artifacts repository
Updated conformance tests
BREAKING CHANGE Removed artifact manifest
Introduced the OCI-Chunk-Min-Length header
Introduced new endpoint end-13: GET /v2/<name>/blobs/uploads/<reference>.

Note This endpoint exists in the Docker Registry HTTP API V2

IMPORTANT The OCI-Subject header is returned on PUT manifest with subject if the remote registry supports Referrers API.
BREAKING CHANGE Applying filters on the Referrers API will not return a result JSON with the annotation org.opencontainers.referrers.filtersApplied. Instead, the registry server returns a header OCI-Filters-Applied.
Introduced Warning headers
Minimum golang version changed to 1.18
Repository name is relaxed. However, it is still a subset of the docker spec. Related oras-go code:
https://github.com/oras-project/oras-go/blob/0e20275bbbd71f4619d9c1b9da150c4f586dff3b/registry/reference.go#L30-L37

Full change set: https://github.com/opencontainers/distribution-spec/compare/v1.1.0-rc1...v1.1.0-rc.3

Detailed Changes by Release
image-spec: v1.1.0-rc2 --> v1.1.0-rc.3

BREAKING CHANGE Removed artifact manifest
BREAKING CHANGE Removed pre-defined annotation keys

org.opencontainers.artifact.created
org.opencontainers.artifact.description
org.opencontainers.referrers.filtersApplied

Minimum golang version changed to 1.18
Non-distributable layers a.k.a. foreign layers are deprecated
Introduced scratch config

Media type: application/vnd.oci.scratch.v1+json

artifactType field is introduced to the image manifest. This field MUST be set if the config media type is set to scratch
layers field of an image manifest SHOULD NOT be empty, which was a MUST. If the config media type is application/vnd.oci.image.config.v1+json, layers MUST NOT be empty.
Introduced Guidelines for Artifact Usage

Full change set: https://github.com/opencontainers/image-spec/compare/v1.1.0-rc2...v1.1.0-rc.3
image-spec: v1.1.0-rc.3 --> v1.1.0-rc4

artifactType and subject fields are introduced to the image index
scratch config is changed to empty descriptor

Media type: application/vnd.oci.empty.v1+json

Full change set: https://github.com/opencontainers/image-spec/compare/v1.1.0-rc.3...v1.1.0-rc4
distribution-spec: v1.1.0-rc1 --> v1.1.0-rc.2

Removed docs to the OCI Artifacts repository
Updated conformance tests
BREAKING CHANGE Removed artifact manifest
Introduced the OCI-Chunk-Min-Length header
Introduced new endpoint end-13: GET /v2/<name>/blobs/uploads/<reference>.

Note This endpoint exists in the Docker Registry HTTP API V2

IMPORTANT The OCI-Subject header is returned on PUT manifest with subject if the remote registry supports Referrers API.
BREAKING CHANGE Applying filters on the Referrers API will not return a result JSON with the annotation org.opencontainers.referrers.filtersApplied. Instead, the registry server returns a header OCI-Filters-Applied.
Introduced Warning headers
Minimum golang version changed to 1.18

Full change set: https://github.com/opencontainers/distribution-spec/compare/v1.1.0-rc1...v1.1.0-rc.2
distribution-spec: v1.1.0-rc.2 --> v1.1.0-rc.3

Repository name is relaxed. However, it is still a subset of the docker spec. Related oras-go code:
https://github.com/oras-project/oras-go/blob/0e20275bbbd71f4619d9c1b9da150c4f586dff3b/registry/reference.go#L30-L37

Full change set: https://github.com/opencontainers/distribution-spec/compare/v1.1.0-rc.2...v1.1.0-rc.3

Resolving this issue resolves

#491
#492

zot depends on this library to support ORAS artifacts. So a rc or a branch that removes/fixes at least the symbols much appreciated.
For example:
https://github.com/oras-project/oras-go/blob/main/internal/descriptor/descriptor.go#L73

@rchincha The PR #496 is going to fix that.

Hi,
We're testing the image spec master branch on zot, and we noticed this constant was removed from the spec:
there is a constant which got removed from the spec: https://github.com/opencontainers/image-spec/commit/7f43cb4fdf597b591197b46b4d67c2b6797de237
And is currently used in:
https://github.com/oras-project/oras-go/blob/2e7b65f1b60d11cd861ce6b68bc53e23a1fd9306/pack.go#L93

@andaaron Thanks for the reminder! We will fix that in the upcoming versions.

For the next minor version of oras-go, we will update it to comply with image-spec v1.1.0-rc4 and the upcoming distribution-spec v1.1.0-rc3. The updated version will remain compatible with image-spec v1.1.0-rc2 and distribution-spec v1.1.0-rc1, and will not introduce any breaking changes.
The required changes for oras-go include:

[ ] Upgrade to the latest image spec and fix missing types
[ ] Support Image Index with subject
[ ] Update oras.Pack to comply with the latest spec
[ ] Update the implementation of Referrers API (#443)
[ ] Update the implementation of pushing manifest
[ ] Update the implementation of deleting manifest
[ ] Support Warning headers (#469)

Closing this issue as oras-go has been update-to-date with image-spec v1.1.0-rc4 and distribution-spec v1.1.0-rc3.


Embedding PDFs

Description
The user should be able to view and download lesson plans and parenting guides as PDFs within the website. The admin should be able to upload lesson plans and parenting guides as PDFs within the website.
Create a component under components/PDF. For reference, the Figma design looks like this, but you shouldn't need it too much for this ticket. Feel free to use any library that may help make this easier (ex. react-pdf); it doesn't have to exactly match what the design shows.
Create a button (no frontend design necessary) that, upon clicking, allows you to upload a PDF.
Additionally, modify "lesson" and "parenting guide" in the Game schema and any necessary endpoints to account for the modified schema:

lesson: ObjectID of file saved in GridFS, optional
parenting guide: ObjectID of file saved in GridFS, optional

Acceptance Criteria

[ ] The embedded PDF is created at components/PDF
[ ] Create a /dummy-pdf/[id] page that contains a button (no frontend design necessary) that allows you to upload a PDF file for lesson plan and/or parenting guide and save it into the MongoDB database
[ ] Place a sample component on that same page that displays the lesson plan and/or parenting guide PDFs corresponding to the game id
[ ] The PDF should have the option to be downloaded by the user
[ ] The database schema and necessary endpoints are updated

Other Notes

For the button that uploads the PDF, do not worry about styling this button for now. Any button will work
Make sure to use Tailwind for styling! (There shouldn't be much styling for this ticket tho...)
Don't be afraid to reach out for help!

For reference, I've attached a few lesson plans the admin might upload. They should be pretty standard PDFs, but note they can be portrait or landscape!
colorful-lesson-plan.pdf
Critical Thinking with Mushwhom .pdf
print-friendly-lesson-plan.pdf


METRON-2308 Fix 'Degrees' Example Profile

One of the example profiles uses "in" as a variable name.  This is now a reserved keyword and cannot be used as a variable name.  The example should be updated to work.
Acceptance Testing
Try out the profile in your favorite Profiler flavor; REPL, Streaming, or Batch.
Pull Request Checklist

[x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
[x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:
[x] Have you ensured that any documentation diagrams have been updated, along with their source files, using draw.io? See Metron Development Guidelines for instructions.

Good find, thanks for the fix @nickwallen. +1 by inspection.


Readiness probe failing on upgrade to 2.6.4 on EKS

What you expected to happen?
weave starts okay when container is upgraded from 2.6.2 to 2.6.4
What happened?
Get the message:
Readiness probe failed: Get http://<IP_ADDRESS>:6784/status: dial tcp <IP_ADDRESS>:6784: connect: connection refused

How to reproduce it?
Running on EKS, originally I have ports 6783-6784 UDP and 6783 TCP allowed from EKS workers to cluster and runs okay on 2.6.2. After upgrading I get the readiness probe failure. If I open up 6784 I seem to be getting a different error:
DEBU: 2020/06/03 18:42:50.412103 [kube-peers] Checking peer "86:8d:08:e5:0d:a6" against list &{[{96:bb:40:05:dd:db ip-10-207-56-229.ec2.internal} {ca:b7:8d:a2:b9:49 ip-10-207-56-143.ec2.internal} {86:8d:08:e5:0d:a6 ip-10-207-56-225.ec2.internal} {62:67:8f:44:45:ec ip-10-207-56-183.ec2.internal}]}
FATA: 2020/06/03 18:42:50.654286 creating bridged fastdp veth pair: setting accept_ra to 0: open /proc/sys/net/ipv6/conf/vethwe-bridge/accept_ra: no such file or directory

Anything else we need to know?
EKS
Versions:

$ weave version
$ docker version
$ uname -a
$ kubectl version

Logs:
$ docker logs weave

or, if using Kubernetes:
$ kubectl logs -n kube-system <weave-net-pod> weave

Network:

$ ip route
$ ip -4 -o addr
$ sudo iptables-save

Presuming you mean 2.6.4, this is indeed a new fix that was put in for a CVE reported via Kubernetes.
Do your nodes have ipv6 disabled or something like that?  I can put in some protection against failure in this case.

Sorry about that.  Ironically the CVE is for nodes not using IPv6 but I didn't look at what happens when you have disabled it entirely.

If you're in a position to test an unreleased version, please try weaveworks/weave_kube:latest which should fix this problem.

I've tested that version, it fixed the problem


The project name should not allow any special characters

adding form validation on project name field so it does not allow any special characters on UI
adding form validation on other fields
Fixes #116

@yhfreeman What was the validation? I think we need (ATM) 40 characters max for the Name and 512 max for the description. @sbathgate we can either ask the Web or API to store the userId in lower case only. Although people should be able to enter them in mixed case in the Web form.

It might be worth converting the userId to lower case within the API in the event calls come from outside the Web interface?

Good point. Are you able to create a bug ticket for this. See the format we're using in other tickets. Hold off in implementing it; it might be something @j-pye should implement on his end as its only SSO that requires it to be in lower case.

@jleach To answer your question "what's the validation", this file defines the constraints https://github.com/bcgov/platform-services-registry/blob/63eaba1b0ff27ffddf9d2e69a9785608c343819d/web/src/utils/useValidator.tsx. I did look up the DB schema constraints, and then set up the front-end validations accordingly.


App not building and Firebase is not setup correctly

Im trying to build this project on latest android version and getting multiple errors.
Android studio version:
Android Studio Hedgehog | 2023.1.1 Patch 2
Build #AI-231.9392.1.2311.11330709, built on January 19, 2024
Runtime version: 17.0.7+0-b2043.56-10550314 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
Windows 10.0
GC: G1 Young Generation, G1 Old Generation
Memory: 2048M
Cores: 4
Registry:
external.system.auto.import.disabled=true
ide.text.editor.with.preview.show.floating.toolbar=false
ide.experimental.ui=true

were you able to build?


[FLINK-3057][py] Bidirectional Connection

This PR allows bidirectional data exchange during the plan phase.
To do this the following changes were made:

replaced OneWayMappedFileConnection with a simple TCP connection
added an iterator to the python ExEnv to deserialize data during the plan phase
added a new java Streamer class, providing the PlanBinder with a single access point to both send and receive data

A proof-of-concept JobExecutionResult was added as well, providing access to getNetRuntime()
Most of the changes in this PR ended up as a refactoring of related classes. The data exchange code was residing completely within the PythonStreamer, Sender and Receiver classes, regardless of the phase(plan or runtime)they are used. All these classes are now split into classes for the respective phase: PythonSender/-Receiver/-Streamer are used at runtime, PythonPlanSender/-Receiver/-Streamer during the plan phase. Additionally, these classes were renamed to be more consistent, and moved into separate packages.
These changes are mostly about consistency. Process spawning is always handled within a Streamer class, which is also always used as the access point for both sending and receiving data. There is also no longer a (non-obvious) overlap in methods used in both phases, which frequently caused issues when these were modified.
This PR also includes a fix for FLINK-3014, switching from InputStream.read() to DataInputStream.readFully().

I've quickly scrolled through the change. I'm wondering whether we should change the development process for the Python API a bit.
It seems that you are currently the only committer working on the Python API. Maybe it makes sense that you push changes directly to master if they don't touch the rest of the system. Do you think that would improve the ease of development of the python API?

@rmetzger Well it surely would speed things up, making it very tempting to go this route. QA goes out the window though, which leaves me quite torn on this one :/

Well, in practice, even with PR's, there is not much QA going on for Python.


Run as non-root, Kubernetes (Helm Chart)

Problem Statement:
The current Cribl Helm chart installs a container that is owned by the root user. Security best practices advise against this and many customers have internal security policies that do not allow. Cox Communications initially ran into permissions issues writing to /tmp/cribl/ volume. They were able to get around this specific issue by manually adding fsGroup: 1001 in the podSecurityContext section of the yaml config, but ran into other issues. Ultimately they were never able to successfully get the container to run as a non-root user.
Customer Request:
Cox Communications wants to be able to have their containers created/owned by UID 1001 (or higher). Ideally they would like the ability to specify which UID/GID to assign within the Helm chart.

@jballachino you'll need to have the customer look at using the securityContext values in the Helm charts. Setting securityContext or podSecurityContext should enable the charts to run as a non-root user.
Part of the issue is that the stock cribl/cribl container image is built as the root user (uid 0), so attempting to write anything to /opt/cribl inside the container fails when running as non-root.
The Leader has the ability to run as non-root and correctly configures the Git settings. The Worker Group chart needs a few extra values to enable running as non-root.
Here's a set of working values that makes Cribl run as the built-in nobody user:
Leader
securityContext:
  runAsUser: 65534
  runAsGroup: 65534
  runAsNonRoot: true

Worker Group
extraVolumeMounts:
  - name: config-volume
    mountPath: /opt/cribl/config-volume
env:
  CRIBL_VOLUME_DIR: /opt/cribl/config-volume
podSecurityContext:
  runAsUser: 65534
  runAsGroup: 65534
  runAsNonRoot: true

Thanks Brendan. I'll have the customer test and update back.
Jeff
On Fri, Oct 18, 2024 at 1:33 PM Brendan Dalpe @.***>
wrote:

@jballachino https://github.com/jballachino you'll need to have the
customer look at using the securityContext values in the Helm charts.
Setting securityContext or podSecurityContext should enable the charts to
run as a non-root user.
Part of the issue is that the stock cribl/cribl container image is built
as the root user (uid 0), so attempting to write anything to /opt/cribl
inside the container fails when running as non-root.
The Leader has the ability to run as non-root and correctly configures the
Git settings. The Worker Group chart needs a few extra values to enable
running as non-root.
Here's a set of working values that makes Cribl run as the built-in nobody
user:
Leader
securityContext:
runAsUser: 65534
runAsGroup: 65534
runAsNonRoot: true
Worker Group
extraVolumeMounts:

name: config-volume
mountPath: /opt/cribl/config-volume
env:
CRIBL_VOLUME_DIR: /opt/cribl/config-volume
podSecurityContext:
runAsUser: 65534
runAsGroup: 65534
runAsNonRoot: true

—
Reply to this email directly, view it on GitHub
https://github.com/criblio/helm-charts/issues/208#issuecomment-2422937330,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BMGORX67B7PCDL6XTP5JWBDZ4FA6HAVCNFSM6AAAAABQEC54FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRSHEZTOMZTGA
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi Brendan -
I shared your recommendations with the customer. They provided the
following response:
We have already tried the below recommendation, and the outcome was the
leader and worker processes were running but we were not able to configure
remote repository.
In our setup the git connectivity is via SSH-keys and when we run SSH
command in Leader pod shell it fails with error message “no user defined”.
We have also explained the same during our last connect with John Padilla,
and he acknowledged the issue is due to the Cribl base Image not configured
in a way to run as non-root user.
Please follow below cases that we worked on with support team.
00016512 – non-root user Issue
00016642 – Git Issue
On Fri, Oct 18, 2024 at 1:33 PM Brendan Dalpe @.***>
wrote:

@jballachino https://github.com/jballachino you'll need to have the
customer look at using the securityContext values in the Helm charts.
Setting securityContext or podSecurityContext should enable the charts to
run as a non-root user.
Part of the issue is that the stock cribl/cribl container image is built
as the root user (uid 0), so attempting to write anything to /opt/cribl
inside the container fails when running as non-root.
The Leader has the ability to run as non-root and correctly configures the
Git settings. The Worker Group chart needs a few extra values to enable
running as non-root.
Here's a set of working values that makes Cribl run as the built-in nobody
user:
Leader
securityContext:
runAsUser: 65534
runAsGroup: 65534
runAsNonRoot: true
Worker Group
extraVolumeMounts:

name: config-volume
mountPath: /opt/cribl/config-volume
env:
CRIBL_VOLUME_DIR: /opt/cribl/config-volume
podSecurityContext:
runAsUser: 65534
runAsGroup: 65534
runAsNonRoot: true

—
Reply to this email directly, view it on GitHub
https://github.com/criblio/helm-charts/issues/208#issuecomment-2422937330,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BMGORX67B7PCDL6XTP5JWBDZ4FA6HAVCNFSM6AAAAABQEC54FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRSHEZTOMZTGA
.
You are receiving this because you were mentioned.Message ID:
@.***>

@jballachino out of curiosity, what was the user id they configured for their security context?

1001
On Mon, Oct 21, 2024 at 1:08 PM Brendan Dalpe @.***>
wrote:

@jballachino https://github.com/jballachino out of curiosity, what was
the user id they configured for their security context?
—
Reply to this email directly, view it on GitHub
https://github.com/criblio/helm-charts/issues/208#issuecomment-2427278590,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BMGORX4E2G76NP47FQVPNMLZ4UYJBAVCNFSM6AAAAABQEC54FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRXGI3TQNJZGA
.
You are receiving this because you were mentioned.Message ID:
@.***>

@jballachino I am able to reproduce this, but it's not something specific to Cribl. The user MUST exist inside the container in order to use Git/SSH.
There's a very specific reason I chose 65534 with the cribl/cribl container image, as that user is nobody and already exists in the /etc/passwd file.
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
_apt:x:100:65534::/nonexistent:/usr/sbin/nologin

In order to allow SSH as non-root/nobody, you need to add a extraConfigmapMounts to the /nonexistent path, as that is the nobody's home folder.
extraConfigmapMounts:
  - name: nonexistent-user-gitconfig
    mountPath: /nonexistent/.gitconfig
    subPath: .gitconfig
    configMap: logstream-leader-gitconfig
securityContext:
  runAsUser: 65534
  runAsGroup: 65534
  runAsNonRoot: true

With these Helm values, I was able to commit and push configs to a remote git repo.
Alternatively, if the customer wants to use uid 1001, the will need to build a custom image and have it run the useradd so that it exists in /etc/passwd and has a home directory where a .gitconfig file can be mounted.

that user is nobody and already exists

Note that the Securing Debian Manual states that

Daemons that need not own any files run as user nobody and group nogroup. Thus, no files on a system should be owned by this user or group.

Thanks Brendan. I passed along the additional details. I'll let you know
how it goes.
Jeff
On Tue, Oct 22, 2024 at 9:17 AM Brendan Dalpe @.***>
wrote:

@jballachino https://github.com/jballachino I am able to reproduce
this, but it's not something specific to Cribl. The user MUST exist inside
the container in order to use Git/SSH.
There's a very specific reason I chose 65534 with the cribl/cribl
container image, as that user is nobody and already exists in the
/etc/passwd file.
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
_apt:x:100:65534::/nonexistent:/usr/sbin/nologin
In order to allow SSH as non-root/nobody, you need to add a
extraConfigmapMounts to the /nonexistent path, as that is the nobody's
home folder.
extraConfigmapMounts:

name: nonexistent-user-gitconfig
mountPath: /nonexistent/.gitconfig
subPath: .gitconfig
configMap: logstream-leader-gitconfig
securityContext:
runAsUser: 65534
runAsGroup: 65534
runAsNonRoot: true

With these Helm values, I was able to commit and push configs to a remote
git repo.
Alternatively, if the customer wants to use uid 1001, the will need to
build a custom image and have it run the useradd so that it exists in
/etc/passwd and has a home directory where a .gitconfig file can be
mounted.
—
Reply to this email directly, view it on GitHub
https://github.com/criblio/helm-charts/issues/208#issuecomment-2429262617,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BMGORX462AKXFDSL6MYUIM3Z4ZF5ZAVCNFSM6AAAAABQEC54FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRZGI3DENRRG4
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi Brendan,
Are you able to jump on a call with Cox Comms? They do not have configMap
setup similarly in their current environment and need some help
understanding the content of configMap: logstream-leader-gitconfig.
Please let me know. Thanks,
Jeff
On Thu, Oct 24, 2024 at 9:20 AM Jeff Ballachino @.***>
wrote:

Thanks Brendan. I passed along the additional details. I'll let you know
how it goes.
Jeff
On Tue, Oct 22, 2024 at 9:17 AM Brendan Dalpe @.***>
wrote:

@jballachino https://github.com/jballachino I am able to reproduce
this, but it's not something specific to Cribl. The user MUST exist inside
the container in order to use Git/SSH.
There's a very specific reason I chose 65534 with the cribl/cribl
container image, as that user is nobody and already exists in the
/etc/passwd file.
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
_apt:x:100:65534::/nonexistent:/usr/sbin/nologin
In order to allow SSH as non-root/nobody, you need to add a
extraConfigmapMounts to the /nonexistent path, as that is the nobody's
home folder.
extraConfigmapMounts:

name: nonexistent-user-gitconfig
mountPath: /nonexistent/.gitconfig
subPath: .gitconfig
configMap: logstream-leader-gitconfig
securityContext:
runAsUser: 65534
runAsGroup: 65534
runAsNonRoot: true

With these Helm values, I was able to commit and push configs to a remote
git repo.
Alternatively, if the customer wants to use uid 1001, the will need to
build a custom image and have it run the useradd so that it exists in
/etc/passwd and has a home directory where a .gitconfig file can be
mounted.
—
Reply to this email directly, view it on GitHub
https://github.com/criblio/helm-charts/issues/208#issuecomment-2429262617,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BMGORX462AKXFDSL6MYUIM3Z4ZF5ZAVCNFSM6AAAAABQEC54FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRZGI3DENRRG4
.
You are receiving this because you were mentioned.Message ID:
@.***>


Add generation of raw MFM images for tnt23 floppy emulator.

MFM file contains 160 tracks of raw MFM-encoded data.
Every track occupies 12800 bytes.
Total file length is 2048000 bytes.
For details about tnt23 floppy emulator see: http://www.sensi.org/~tnt23/megadrive/

See container/hfe.c and container/eadf.c for examples of how to open these files in libdisk by unmarshaling data into uniform_raw tracks, and how to implement the close() function to accept any format of track (not just raw_dd). Your case is simpler than HFE or EADF however.
Also please update disk-analyse.c:usage() to mention .mfm support.

Unacceptable in current form.

OK, I'll prepare a new pull request. Thanks for your comments.


Subtitle is not moved from output-extension folder

When I define a temporary save location in the output-extension option but when the media process ends, another save location moves. The subtitle not moved along with the file getting separated is there any solution for that?
Edit: I'm using manual.py to do the process

Could you post your current autoProcess configuration as well as a sample set of logs for a conversion that's exhibiting the behavior

Actually I was able to reproduce this without that info and should have it fixed in the latest update
ba94986dce19e4ba4fcff61631e942c92f724bf1
See if that works for you

Thanks. But I won't be able to see this until tomorrow.

Alright well seems to be fixed in my tests so I'll close this out but reopen if there's any issues.


Mercurial 4.9.1

Checklist

[x ] Used a fork of the feedstock to propose changes
[x ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x ] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe:

Whenever possible python packages should use pip. See https://conda-forge.org/docs/maintainer/adding_pkgs.html#use-pip

@conda-smithy please rerender

@conda-forge-admin please rerender

@ngoldbaum
I created this PR in order to be able to install a recent version of Mercurial (supporting hg absorb) with conda (in particular useful on Windows).
Do you think it can be merged so that the conda package would be built?

Please feel free to make a PR and add yourself as a maintainer.


Any eta for support for the newest version of freerdp?

Hi Cedric, you mentioned that you were working on implementing the newest version of freerdp. I was curious how that was going?

Hi Troy, yes I'm working on it the main problem being the low free time I have (like 1h/day).
I also had some technical challenges, with cmake, mixing up C/C++ code (myrtille uses C++ while freerdp is pure C), refactoring the myrtille code, playing with newest freerdp params and implementing the clipboard support.
The good news is it works, and well. The latest version fixes then NLA and mouse pointer issues. There is now clipboard and Hyper-V connection support. Performance may also be a bit improved as well, as freerdp now uses hardware gdi.
I'm currently testing and trying to fix some clipboard issues when multiple sessions are actives. Also, myrtille will support only unicode text clipboard format because OLE objects aren't supported by the browser.
ETA is 4 september, at best (I won't be at home this week-end so won't be able to progress).
I plan to do 2 more things in september: home page relooking (currently very basic) and IIS 7.5+ native websocket support (replacing Fleck).

There will be also an architectural change. I will make a freerdp fork as soon as I'm done with tests. This fork (another repository) will contain and tag the myrtille code (like it's done actually) and will be periodically synchronized with the freerdp main repository (stable branch).
The objective is to have myrtille always use the latest version of freerdp and thus benefit bug fixes and improvements.
But the main idea is to have myrtille loose coupled with freerdp, and the rdp protocol alltogether, in order to be able to be branched with other(s) project(s)/protocol(s) (ie: vnc). That is, myrtille will be a generic remote desktop gateway.
It will be transparent for the end user, I will have the installer manages that.

I committed the discussed changes. First, the rdp client (freerdp) is now decoupled from myrtille and is accessible there: https://github.com/cedrozor/FreeRDP. Second, I reworked the homepage to make it more sexy.
Just remains to update the installer, documentation and wiki.
I will release a new version this week (but the code is already functionnal is someone want to check it).

Cool. I can hardly wait to play with it :)

So I am a bit confused :) What is the relationship between cedrozer/myrtille/Myrtille.RDP... and cedrozer/FreeRDP?
The only thing I can see that you changed in Myrtille.RDP is wf_win.cpp which doesn't exist in FreeRDP?
Is any of the Myrtille.RDP still used?
I haven't built cedrozer/FreeRDP yet. I need to learn cmake I guess. I am really weak on c and c++

cedrozor/Myrtille (this repo) is the (http to rdp) gateway while cedrozor/FreeRDP is the rdp client (used by the gateway). It loose couples the 2, allowing tomorrow to have myrtille using telnet, SSH, VNC or whatever remote access protocol.
Before I have time to update the installer (and release a new version), I indeed left the old freerdp version (embedded with myrtille); it will be removed soon, when the decoupling is effective.
To build cedrozor/FreeRDP, you have to call cmake following this procedure: https://github.com/FreeRDP/FreeRDP/wiki/Build-on-Windows-Visual-C---2012-%2832-and-64-bit%29.
Once built, you have to copy the new freerdp executable and dlls into the myrtille bin folder, then comment and replace the old freerdp call by the new one (into Myrtille.Services/RemoteSessionProcess.cs.) But if you can wait a few days, I suggest you to wait the next release ;)
I will update the myrtille documentation and wiki according to it.

Myrtille 1.1.0 is now available. This is a pre-release (beta).


How to run Socket Mode alongside Flask SocketIO?

I am attempting to make a chatbot available both within my web app and via a Slack bot. I have tested both separately and they work well. However, when I attempt to run them in different threads of the same application, I get an SSL error from Bolt
ERROR:slack_bolt.App:on_error invoked (session id: b524fca5-51da-46cf-8817-6c71b133897a, error: SSLError, message: ('timed out',))
I am running the Flask-SocketIO app in a daemon thread and starting the SocketModeHandler in the main thread. For the Flask-SocketIO app, I am using eventlet and monkey patching it.
Reproducible in:
cryptography~=41.0
eventlet~=0.33
flask~=2.2
Flask-SocketIO~=5.3.0
#langchain~=0.0
langchain==0.0.266
looker-sdk~=23.10
openai~=0.27
opentelemetry-distro==0.38b0
pandas~=2.0.0
pylint==2.13.9
pyarrow~=12.0.0
scikit-learn~=1.3.0
sentry-sdk[flask]~=1.0
setuptools-lint==0.6.0
slack-bolt~=1.0
tiktoken~=0.4.0
The slack_bolt version
slack-bolt==1.18.0
Python runtime version
Python 3.10.13
OS info
Running Debian GNU/Linux 11 (bullseye) in a Docker container.
Steps to reproduce:
Unfortunately, I can't share source code. But, start a Flask-SocketIO app in a background thread and then call Slack SocketModeHandler#start() method. Observe SSL error.
Expected result:
There to be no SSL error, and the Bolt handler connects properly.
Actual result:
The SocketModeHandler#start() method fails. The above error message is printed several times over. The app does not respond to mentions in Slack.
Requirements
Please read the Contributing guidelines and Code of Conduct before creating this issue or pull request. By submitting, you are agreeing to those rules.

I've tried switching to gevent, but ran into the same issue. I'm not sure where else to look. Any guidance or pointers is appreciated! Thank you.

I got slightly more information now on the error
ERROR:slack_bolt.App:on_error invoked (session id: 1772e3df-7622-4151-96d9-f6fec3ff2759, error: SSLError, message: ('timed out',)) Traceback (most recent call last):   File "/usr/local/lib/python3.10/site-packages/slack_sdk/socket_mode/builtin/connection.py", line 305, in run_until_completion     received_messages: List[Tuple[Optional[FrameHeader], bytes]] = _receive_messages(   File "/usr/local/lib/python3.10/site-packages/slack_sdk/socket_mode/builtin/internals.py", line 223, in _receive_messages     return _fetch_messages(   File "/usr/local/lib/python3.10/site-packages/slack_sdk/socket_mode/builtin/internals.py", line 246, in _fetch_messages     remaining_bytes = receive()  # type: ignore   File "/usr/local/lib/python3.10/site-packages/slack_sdk/socket_mode/builtin/internals.py", line 221, in receive     raise e   File "/usr/local/lib/python3.10/site-packages/slack_sdk/socket_mode/builtin/internals.py", line 208, in receive     received_bytes = sock.recv(size)   File "/usr/local/lib/python3.10/site-packages/eventlet/green/ssl.py", line 243, in recv     return self._base_recv(buflen, flags, into=False)   File "/usr/local/lib/python3.10/site-packages/eventlet/green/ssl.py", line 270, in _base_recv     read = self.read(nbytes)   File "/usr/local/lib/python3.10/site-packages/eventlet/green/ssl.py", line 187, in read     return self._call_trampolining(   File "/usr/local/lib/python3.10/site-packages/eventlet/green/ssl.py", line 165, in _call_trampolining     trampoline(self,   File "/usr/local/lib/python3.10/site-packages/eventlet/hubs/__init__.py", line 159, in trampoline     return hub.switch()   File "/usr/local/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch     return self.greenlet.switch() ssl.SSLError: ('timed out',)

Hey @chadlieberman-flywire 👋
I tried to reproduce this but found that having the Flask app in the dameon thread caused this error from:
ValueError: signal only works in main thread of the main interpreter

Switching the apps on each thread, so that the Slack app runs on the dameon thread instead, seems to fix this and might also remove the SSLError too. With this example I didn't seem to encounter this, so I'm wondering if switching these threads might be an alright solution for you?
If not, I'd be curious to know more about the monkey patching happening with eventlet since the Slack app handler is sometimes working fine.
Also, formatting can get strange sometimes, so wanted to check that the slack-bolt version you're using is 1.18.0 and not 1.0?
Hopefully some of this can help, but let me know if this error continues to happen!


2.5.0 Gradle errors

When updating to version 2.5.0 it's throwing me this errors in Gradle app file:
Error:Failed to resolve: com.android.support:exifinterface:26.0.2
Error:Failed to resolve: com.android.support:appcompat-v7:26.0.2
Error:Failed to resolve: annotationProcessor
I tried to update the com.android.support.appcompat library to 26.0.2 and add the exifinterface library but with no success. Any ideas?

see the changes I had to do in this comment: https://github.com/ArthurHub/Android-Image-Cropper/commit/85b624ca736d343cb8790808af73613d9f11c48c

Same problem, i tried to add your library but giving errors:
Error:Failed to resolve: com.android.support:exifinterface:26.0.2
Error:Failed to resolve: com.android.support:appcompat-v7:26.0.2
Error:Failed to resolve: annotationProcessor
Any help would be really appreciated. Thanks a lot. btw I saw the changes in your comment, but i couldn't understand. I am new in Android Programming. Please Help

see #396


Support iterative fixes in the LSP "Fix All" command

The dart fix can apply multiple rounds of fixes, but the equiv LSP command (which is wired up to "Auto Fix" in VS Code, which some users like to run on-save) does not.
The dart fix command works by building overlays from the first set of changes and then running the fixes again. This will be more complicated inside the server run by the IDE as we can't modify the overlays without potentially introducing issues (even if they are applied and reverted, because the fixes are async, there's no guarantee the editor won't be sending requests that will have "misunderstood" offsets during that time).
It may be possible to lay another OverlayResourceProvider over the existing one to do this, but will need some testing - see discussion at https://github.com/dart-lang/sdk/issues/47886#issuecomment-991215561.

@bwilkerson this was the issue mentioned earlier. Based on the text above, it seems like the issue was that dart fix was modifying the overlays (which is fine for it because they're not also being used by an editor). Your linked comments suggest we may be able to wrap another temporary overlay provider over that one, though I haven't tested that yet. I'll make a note to do so soon, as this might actually be a fairly simple fix if that works (or can be made to).

@pq for visibility

@bwilkerson I started taking a look at this, but hit a snag. The way this currently works in dart fix is:

compute some edits
apply edits to overlay
reanalyze / compute more edits

If we collect all of the edits here, we don't end up with a single change that makes the necessary changes, but rather a set of changes that need applying sequentially to arrive at the expected output. For example in the case of prefer_final_locals and prefer_const_declarations and code var a = 'test'; we end up with:

A change that says "replace var with final"
A change that says "replace final with const"

These changes conflict, and can only be applied sequentially. In order to apply them in a single operation in VS Code (such that undo buffer works as expected), we need a single set of atomic edits (at least, that's how it seems.. though I'll see if there's a way around this.. I don't see why in theory VS Code couldn't apply some edits sequentially within a single undo record).
I think some of the issues (that is, edits from second round of edits offsets are wrong because they assumed the previous round of edits have already been made) might be fixable by using a shared ChangeBuilder, however I don't think conflicting edits (as occur in the example above) would work. In theory I think we should be able to merge the changes, but I'm not sure if that would be a simple task.

I think that merging the edits would be fairly difficult and hence error prone, but I haven't tried, so it might be more doable than I'm thinking it would be.
The only solution that comes to mind is to make all the edits as per dart fix, then return an edit to replace the entire content of the initial version with the entire content of the final version. We could potentially write a diff algorithm that would narrow it down, but I'm not convinced that it would be worth it.

I also have some concerns around performance. If we're iterating then we're re-analyzing the file on each iteration, which might take longer than we want to spend before returning an edit that might or might not be applied. (Unless I'm misremembering and 'Fix All' is actually a command rather than a fix.)

I think of those options, a full edit and diff is probably better Not doing the diff could be a problem though (which is why we do something similar for formatting - although it's not a real diff because we know only whitespace changes and can just walk two token streams together). A diff seems like the sort of thing there should be well-tested algorithms for we might be able to easily implement in Dart (if there isn't one already in the SDK somewhere 😄).

I also have some concerns around performance.

I was curious about perf too (I noticed the dart fix command on the CLI wasn't terribly fast when testing some of the fixes above on flutter_gallery), but didn't get as far as being able to measure it.
It is a command though, so we're not paying any cost until the user invokes it (we just return a static CodeAction referencing the command from the /codeActions request):
https://github.com/dart-lang/sdk/blob/15c8dad750f277753be7024cff536eb1a423a320/pkg/analysis_server/lib/src/lsp/handlers/code_actions/dart.dart#L302-L307
We can show progress notifications while it's running though (as we do for refactorings because the first invokcation of them can sometimes be a bit slow).

... if there isn't one already in the SDK somewhere ...

There might be one, but I wasn't able to find it when we were implementing dart fix. We considered having a flag to display a diff of the files rather than just a "we fixed 3 things in this file" message, but it looked like it would be too much work. If you find a diff algorithm, we might reconsider.

It is a command though ...

If it's a command, then I'm not concerned about the performance (the user will know what's happening).

I also couldn't find one in the SDK, although I did find this:
https://github.com/google/diff-match-patch/tree/master/dart
The repo hasn't been updated in a few years, but with some tweaks for null-safety the code (and tests) work in current Dart:
https://github.com/DanTup/diff-match-patch/tree/dart-null-safe
Although it doesn't seem like the project is maintained any more, so if we used this (it's Apache licence and Google-created, so I presume that would be possible) and there are any issues we'd have to fix them. Reducing it to only the useful diff-producing code gives ~1k lines:
https://gist.github.com/DanTup/a212ed0f02376322b1486af72f2fc52f
This is larger than I was expecting a diff to be, although it seems to describe some reasonable-sounding optimisations (I don't know if this is a common algorithm or something developed by that project though). Any thoughts?

I'm not going to address the question of whether we can use the code. My guess is yes, but I'd want someone else to make that call.
But if we can use some or all of that code, then it probably isn't going to be any bigger than the code we'd end up writing ourselves. Meyer's algorithm is definitely one of the best. The most recent committer is still working here, so we can also reach out if we have any additional questions.

@bwilkerson if we can get confirmation that it can be used, do you think that's the most sensible path forward? (further up you said you're not sure it's worth it, though I don't know if there are other options besides not supporting this - which might not be a problem, I guess it depends on the number of fixes may require multiple iterations.. I only know of the one given as an example above).
There may be some other benefits to having something like this - --dry-run could be updated to support a diff as you noted above, and we could also perhaps remove the custom handling for the formatter that does this by re-parsing both sets of code (although we might want to compare performance - I suspect since it's much simpler it may also be quicker).

Sorry, but I don't have a definitive answer for you at this point.

... I guess it depends on the number of fixes may require multiple iterations.

There are a couple of others (I don't remember which ones at the moment), but not many.
It would be consistent with our current direction for the fix for prefer_final_locals to check to see whether prefer_const_declarations is enabled and applicable, and if so to create a single edit that gets us to the final state in one operation. On the other hand, I think I might have expressed concerns about the combinatorial explosion of checking that could happen if we go too far down this path; I have mixed feelings about taking this kind of cross-checking too far.

disallow overlapping edits and make the user run fix multiple times, or

But if we combined this with fixing up the common fixes that we know overlap (as you suggested further up with final/const), we might be able to make this a much less common case?
Another option is that we just ditch the grouping and populate them all with a generic title - especially given VS Code doesn't show the title or grouping by default. The grouping would be nice though.

I was thinking more about this, and whether we could merge the edits. One thought I had to do this was to expand any overlapping edits to be the same range, then just replacing the first's replacement text with the replacement text from the second.
Expanding the edits such that these match up doesn't seem like it should be too complicated (we can expand an edit in either direction by just moving the start/end of the replacement range and adding the extra characters from the previous text into the replacement text). Inserts and deletes are just replacements with a zero-length range/text so I think they'd expand in the same way.
Once we have that new edit, we could also then minimize it by taking characters back off the start/end where they match the original source.
@bwilkerson WDYT? I couldn't find any existing algorithms that seem to do this (although it seems difficult to search for) but I think there only a limited number of combinations of edits we'd need to merge.

so I think we need to have some way of mapping ranges from one pass back to the ranges that existed prior to the original pass

For the non-overlapping edits, I think this is fairly straight-forward (compute a delta of the earlier edits replacementRange.length vs replacementText.length and offset the later edits by it). I'm fairly sure we already have to handle this case (because server produces sequential edits and LSP does not).
For the overlapped edits, I don't think it's possible to do this up-front. A second edit might be modifying code that didn't exist at the start, so there's no offset in the original document that represents it. But I think we can just handle this as part of normalising the edits to the same range.
I haven't tried coding it yet, but here's what I had in mind...
Given firstEdit and secondEdit there are only three possibilities for the start offset:

They are the same - nothing to do
firstEdit starts before secondEdit

Reduce offset of secondEdit to match firstEdit
For however many characters the offset was reduced by, read that many characters from firstEdit's replacementText and prepend it to secondEdit's replacementText. If firstEdit's replacementText didn't have enough characters, the additional characters would come from the original document starting at the end of firstEdit's replacementRange.

firstEdit starts after secondEdit

Reduce offset of firstEdit to match secondEdit
Prepend the substring from the new offset to the old offset from the original source into firstEdit's replacementText

Both edits should now have the same offset and applying them sequentially should have the same result. I think we can do something similar for the end. We can expand firstEdit by increasing the replacementRange and appending the additional characters from the source file, and we can expand secondEdit by doing the same (but taking from firstEdit's replacementText first).
It feels sound in my head, but I haven't tried coding it yet. It's possible there are cases I haven't considered or the logic doesn't work. I'll try coding it up soon and see where I get (though if you see any obvious flaws in the meantime, let me know 😄).

I think the logic works if there's exactly two edits to merge, but doesn't work when there are more edits. I'll give you an example below that should explain why I think it doesn't work, but it's possible that your algorithm works for reasons I'm not seeing.

Given firstEdit and secondEdit there are only three possibilities for the start offset:

They are the same - nothing to do

If the firstEdit was produced by a pass before the one that produced the secondEdit, then the offsets might not be comparable. For example, given in put of abcdefghijklmnopqrstuvwxyz, if the first edit inserted 012 at offset 3 and replace 4 characters starting at offset 7 with 34, the result would be abc012defg34lmnopqrstuvwxyz if the second pass inserted 5 at offset 7, then the result would be abc012d5efg34lmnopqrstuvwxyz. Note that the offset of 7 for the first edit is not related to the offset 7 from the second edit.

Thanks! I've looked at this a few times today, and it's definitely not as simple as I thought... For a single pass, server always produces edits that can be applied sequentially (they go from highest offset to lowest), but with multiple passes we can end up with the offsets having a mix of orders, so some assumptions that I (and the LSP code) made doesn't work.
I'm wondering if something like this might work:

Walk backwards through the edits
For each edit, walk backwards further looking for the nearest edit that intersects with it (computing an offset delta on the way)
Perform the steps above on those two edits, eliminating one of them (expanding edits will require having the computed source for related edit from start to end)
Repeat until there are no more intersecting edits

I'm not completely certain it'd work and it'd definitely be more complex. I wonder if it's worth trying though, as it feels like it still might be the closest we have to a solution that doesn't compromise too much on supporting things like the grouped refactor preview? 🤔

That might be worth trying. I don't have a case that proves that that approach wouldn't work, but I'm somewhat skeptical because the offsets of the edits are still relative to different versions of the files and my intuition is that that's the critical piece that needs to be solved. But maybe the offset delta you mentioned takes care of that in a way that I'm not currently understanding.
Point of clarification: I think the first step is to walk backwards through the current pass' edits, and the second step is to walk backward through the accumulated edits from previous passes. Otherwise I'd be concerned that it would be possible to incorrectly handle some of the edits in terms of which pass they're from.

I failed at trying to do the above. Tracking the offsets/deltas while working backwards got very complicated.
However, I think I may have a new idea 😄
First step is to go through the edits and re-order them where possible, so get them back into highest-offset-first (as each pass would have them). This is done like insertion sort (we work through them, then try to move each one as far as back as we can), but when we move items, we compensate for the delta of the previous item that's no longer being applied before it. If two edits touch or intersect (such that the swap would be invalid), we do not swap.
Now we have a list of edits in reverse order (as we want) except for those problematic overlapping/touching edits. But those edits are now adjacent, so we can use the original idea from above to merge them.
I have a basic version of this working for the cases I've tried, I'm just trying to get it into the server codebase so I can write tests and ensure all cases are covered.
I don't know how efficient it'll be (O(n^2)?), but I do expect our lists to be mostly sorted, and in the case where the second pass makes no changes we don't need to do any of this - and that's true for each individual file.. if only a single file has edits from two passes, only that files edits need merging. If I can confirm it all works and we can get a good set of tests, we can measure and if necessary, try to optimise.

Sounds promising. Eager to hear how it goes.

I've pushed what I've got so far here:
https://dart-review.googlesource.com/c/sdk/+/304180
I need to add more tests (deletions, multiple files) before it's ready but since I'm done for the day I thought it was worth pushing in case you want to have a scan of the general implementation (or think of any additional cases to test that I can work on tomorrow). Not urgent though :-)

Since the CPU profile wasn't particularly useful to me, I applied my rigged version of OperationPerformanceImpl that writes DevTools timeline events and it looks like this:

computeLibraryCycle in LibraryContext.load seems to take the bulk of the time that's specific to the first analysis in this context (I don't know if that's something that could be reused?).

I suspect, although the profile should show actual data, that the most time that it spent in computeLibraryCycle is because we build FileStates for transitive libraries. I think that we probably don't link libraries, because they should be already available in the shared ByteStore. We probably cannot reuse FileStates from a different AnalysisContextCollection, especially because it has different ResourceProvider.

although the profile should show actual data, that the most time that it spent in computeLibraryCycle is because we build FileStates for transitive libraries.

I looked through what computeLibraryCycle was calling, but couldn't see where the FileStates are created. If you can give me a pointer I can make it show up on the timeline or re-run the profiler to see if it shows up.

We probably cannot reuse FileStates from a different AnalysisContextCollection, especially because it has different ResourceProvider.

I'm not familiar with these parts so this might be a silly question, but since our new ResourceProvider is just overlays, would it be possible re-use the state for files that don't exist in it (and are being provided by the underlying resource provider, which is the same)?
Although, even if we completely eliminate that block of code, I still wonder whether the multiple analysis passes will still leave us slower than we'd like to be for something running on-save? 🤔

although the profile should show actual data, that the most time that it spent in computeLibraryCycle is because we build FileStates for transitive libraries.

I looked through what computeLibraryCycle was calling, but couldn't see where the FileStates are created. If you can give me a pointer I can make it show up on the timeline or re-run the profiler to see if it shows up.

Are we talking about this line? https://github.com/dart-lang/sdk/blob/305fa3d77c534eff0de9def487b184ebf8608b7b/pkg/analyzer/lib/src/dart/analysis/library_context.dart#L245
It invokes https://github.com/dart-lang/sdk/blob/305fa3d77c534eff0de9def487b184ebf8608b7b/pkg/analyzer/lib/src/dart/analysis/library_graph.dart#L16
And it will ask libraryImports here https://github.com/dart-lang/sdk/blob/305fa3d77c534eff0de9def487b184ebf8608b7b/pkg/analyzer/lib/src/dart/analysis/library_graph.dart#L142
This will cause building FileStates for imported libraries, which will become new nodes in the graph, for which we will ask their own dependencies, etc, recursively.

We probably cannot reuse FileStates from a different AnalysisContextCollection, especially because it has different ResourceProvider.

I'm not familiar with these parts so this might be a silly question, but since our new ResourceProvider is just overlays, would it be possible re-use the state for files that don't exist in it (and are being provided by the underlying resource provider, which is the same)?

No, I don't think this will work. AnalysisContextCollection uses a single ResourceProvider, in order to reuse anything, it has to be the same one.
So, maybe instead we should use the AnalysisContextCollection instance that already exists in DAS, and update the existing OverlayResourceProvider on which it is based. We will have to remember original content for files that we are going to update temporarily, and revert them back. We need to make sure that there is no race condition though, e.g. no new overlays asynchronously come from the client while we are running this command.

Although, even if we completely eliminate that block of code, I still wonder whether the multiple analysis passes will still leave us slower than we'd like to be for something running on-save? 🤔

Doing any significant work on save is a bad idea, IMHO.

And it will ask libraryImports here

Ah, that's what I was missing. I apparently clicked through to import.importedLibrary but ignored container.libraryImports. I'll check this out..

We will have to remember original content for files that we are going to update temporarily, and revert them back.

Hmm, this is an interesting idea. I was trying to avoid it because of async stuff, but perhaps we can use the existing mechanism for locking requests. I'll do some testing, thanks!

Doing any significant work on save is a bad idea, IMHO.

To clarify, this is if the user opts-in to wanting to automatically fix on-save (though we're discussing adding this to the "recommend settings" - but it definitely can't take this long!).

Hmm, this is an interesting idea. I was trying to avoid it because of async stuff, but perhaps we can use the existing mechanism for locking requests. I'll do some testing, thanks!

Actually, it's not as simple as that. We need to also ensure while we're modifying these overlays we're not firing off events such as new diagnostics, closing labels, etc.
So it would probably end up being something like:

Accept Fix All request
Lock the request queue so no more requests are processed
Ensure all in-flight requests are completed (since they might have been doing async work)
Ensure any in-progress analysis has completed
Prevent the server from sending any events that are usually triggered from analysis (diagnostics, closing labels, Flutter outline)
Perform the iterations of fix all, modifying overlays as they go
Reset all overlays
Wait for in-progress analysis to complete
Re-enable "events" from the server
Unlock the request queue

We'd also need to ensure however this is done, it's not brittle and can't easily accidentally be broken in the future (such as newly-added "events" being sent).
That said, I don't need to implement all of this to measure the change in performance - I can just ensure I don't violate any of those things manually and capture some numbers.

@scheglov you were right about the time being spent in libraryImports/libraryExports above.
So I changed the code to just manipulate the original overlays from the server (ignoring the other complexity for now). It's faster, but it's still quite slow.
One thing I noticed that might be duplicated work but I'm not sure, is that after we apply the edits to the overlays it triggers "computeAnalysisResults" (because the server thinks a file was just modified and needs to produce diagnostics). This holds up the next iteration a little, which is trying to get a resolved library. When computing analysis results completes, it continues, and then resolves the library.
I'm not too familiar with all the steps inside computeAnalysisResult and getResolvedLibrary but I assumed that computing the analysis result would have to resolve the library, so I'm wondering whether getResolvedLibrary should/could be faster here (because it could be reusing what was just computed)?

Hard to say for sure. If you mark a file as changed, the driver will clear cached results, and so all resolved units will be recomputed. It should not matter whether you ask for a resolved unit, or a library, some time ago I implemented caching the whole library.

@scheglov I am marking the file as changed, but only before both of the analysis that occurs on each pass.
I made a small test that I think reproduces the same which may make it clearer. It initialises the server and opens a file (which makes it priority). The file is modified (which triggers analysis) and then afterwards getResolvedLibrary is called. The call to getResolvedLibrary appears to result in another call through LibraryAnalyzer.analyze which I thought would be skipped (because some cache could be used).
I'm not sure exactly which caches exist (except priorityResults) though, so I'm not sure at which point we would expect it to reuse previous results (I couldn't see any obvious cache that was checked between getResolvedLibrary and LibraryAnalyzer.analyze).
https://github.com/DanTup/sdk/commit/dc512f7793ff0e978b747ea5c77669006173f4f7
  @soloTest
  Future<void> test_danny() async {
    instrumentationService.logInfo('Initializing...');
    await initialize();
    await openFile(mainFileUri, 'class A {}'); // Sets priority.
    await pumpEventQueue(times: 5000);

    // Replace file, which will trigger analysis
    instrumentationService.logInfo('Modifying priority file...');
    await replaceFile(1, mainFileUri, 'class B {}');
    await pumpEventQueue(times: 5000);

    // Ask for resolved library which has just been analyzed. Should it be
    // triggering `LibraryAnalyzer.analyze` again?
    instrumentationService.logInfo('Asking for resolved library...');
    await server.getResolvedLibrary(mainFilePath);

    // log output:
    // info: Initializing...
    // info: LibraryAnalyzer.analyze() for /home/my_project/lib/file.dart
    // info: LibraryAnalyzer.analyze() for /home/my_project/lib/main.dart
    // info: Modifying priority file...
    // info: LibraryAnalyzer.analyze() for /home/my_project/lib/main.dart
    // info: Asking for resolved library...
    // info: LibraryAnalyzer.analyze() for /home/my_project/lib/main.dart
    print(instrumentationService.log.join('\n'));
  }

Ah, right, getResolvedLibrary() results are not cached.
This is easier to implement than caching for individual files.
I will do it.

https://dart-review.googlesource.com/c/sdk/+/306302

@scheglov thanks! I'm still seeing duplicate calls with this change though. It looks like caching is done in _computeResolvedLibrary but I think my first analysis is happening inside _computeAnalysisResult (which also uses LibraryAnalyzer) but I think that it's caching any results.
It looks like _computeAnalysisResult doesn't create a ResolvedLibraryResultImpl so I'm not sure whether it has all the information to cache it?
I tested both my original case (the Fix All command I'm trying to improve) and the test in https://github.com/DanTup/sdk/commit/dc512f7793ff0e978b747ea5c77669006173f4f7, but both appear to be the same.
Is _computeAnalysisResult computing the full resolved library, and if so is it also able to add to the cache?

I have a possible fix for the above here: https://dart-review.googlesource.com/c/sdk/+/306620
With that, the duplicate analysis is gone and things are much faster at around half a second (for 3 passes). And when there are no fixes to apply, we're down to less than 1ms (because no new analysis takes place, the result from the analysis when the file was last modified is used from the cache).
I haven't implemented locking/reverting of the overlays yet so that might still change things, so I'll sort that and then do some more thorough profiling/timing and w can decide whether it's good enough.

Are these timings for a multiple fixes applied to a single file? How big is the file?
Or the whole Flutter repository?


Cannot launch IPython with matplotlib in devcontainer

@OCopping to fill out logs

(venv) root@diamtl04:/scratch/ryi58813/ophyd-async# ipython --matplotlib=qt5
Python 3.9.19 (main, Apr 10 2024, 15:19:04)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.18.1 -- An enhanced Interactive Python. Type '?' for help.
[TerminalIPythonApp] WARNING | Eventloop or matplotlib integration failed. Is matplotlib installed?
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
File /venv/lib/python3.9/site-packages/IPython/core/interactiveshell.py:3645, in InteractiveShell.enable_matplotlib(self, gui)
   3641         print('Warning: Cannot change to a different GUI toolkit: %s.'
   3642                 ' Using %s instead.' % (gui, self.pylab_gui_select))
   3643         gui, backend = pt.find_gui_and_backend(self.pylab_gui_select)
-> 3645 pt.activate_matplotlib(backend)
   3646 configure_inline_support(self, backend)
   3648 # Now we must activate the gui pylab wants to use, and fix %run to take
   3649 # plot updates into account

File /venv/lib/python3.9/site-packages/IPython/core/pylabtools.py:368, in activate_matplotlib(backend)
    363 # Due to circular imports, pyplot may be only partially initialised
    364 # when this function runs.
    365 # So avoid needing matplotlib attribute-lookup to access pyplot.
    366 from matplotlib import pyplot as plt
--> 368 plt.switch_backend(backend)
    370 plt.show._needmain = False
    371 # We need to detect at runtime whether show() is called by the user.
    372 # For this, we wrap it into a decorator which adds a 'called' flag.

File /venv/lib/python3.9/site-packages/matplotlib/pyplot.py:342, in switch_backend(newbackend)
    339 # have to escape the switch on access logic
    340 old_backend = dict.__getitem__(rcParams, 'backend')
--> 342 module = importlib.import_module(cbook._backend_module_name(newbackend))
    343 canvas_class = module.FigureCanvas
    345 required_framework = canvas_class.required_interactive_framework

File /usr/local/lib/python3.9/importlib/__init__.py:127, in import_module(name, package)
    125             break
    126         level += 1
--> 127 return _bootstrap._gcd_import(name[level:], package, level)

File <frozen importlib._bootstrap>:1030, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1007, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:986, in _find_and_load_unlocked(name, import_)

File <frozen importlib._bootstrap>:680, in _load_unlocked(spec)

File <frozen importlib._bootstrap_external>:850, in exec_module(self, module)

File <frozen importlib._bootstrap>:228, in _call_with_frames_removed(f, *args, **kwds)

File /venv/lib/python3.9/site-packages/matplotlib/backends/backend_qt5agg.py:7
      4 from .. import backends
      6 backends._QT_FORCE_QT5_BINDING = True
----> 7 from .backend_qtagg import (    # noqa: F401, E402 # pylint: disable=W0611
      8     _BackendQTAgg, FigureCanvasQTAgg, FigureManagerQT, NavigationToolbar2QT,
      9     FigureCanvasAgg, FigureCanvasQT)
     12 @_BackendQTAgg.export
     13 class _BackendQT5Agg(_BackendQTAgg):
     14     pass

File /venv/lib/python3.9/site-packages/matplotlib/backends/backend_qtagg.py:9
      5 import ctypes
      7 from matplotlib.transforms import Bbox
----> 9 from .qt_compat import QT_API, QtCore, QtGui
     10 from .backend_agg import FigureCanvasAgg
     11 from .backend_qt import _BackendQT, FigureCanvasQT

File /venv/lib/python3.9/site-packages/matplotlib/backends/qt_compat.py:112
    108         raise AssertionError(f"Unexpected QT_API: {QT_API}")
    111 if QT_API in [QT_API_PYQT6, QT_API_PYQT5, QT_API_PYSIDE6, QT_API_PYSIDE2]:
--> 112     _setup_pyqt5plus()
    113 elif QT_API is None:  # See above re: dict.__getitem__.
    114     if _QT_FORCE_QT5_BINDING:

File /venv/lib/python3.9/site-packages/matplotlib/backends/qt_compat.py:82, in _setup_pyqt5plus()
     80     _to_int = operator.attrgetter('value')
     81 elif QT_API == QT_API_PYSIDE6:
---> 82     from PySide6 import QtCore, QtGui, QtWidgets, __version__
     83     import shiboken6
     84     def _isdeleted(obj): return not shiboken6.isValid(obj)

ImportError: libGL.so.1: cannot open shared object file: No such file or directory

So, here is a repo with a working container that launches an IPython terminal with matplotlib: https://github.com/DiamondLightSource/htss-rig-bluesky
The main differences are in the Dockerfile and the settings to launch the container (which are in the README). You'll need to translate these settings from CLI arguments to the language of devcontainer.json

Turns out it just works using TK as a backend, However may be a related issue: https://github.com/bluesky/bluesky/issues/1693

You may run into other problems with tk.  This reads like there is something wrong with the pyside6 install.
https://github.com/bluesky/bluesky-pods/blob/main/compose/acq-pod/launch_bluesky.sh
has the magic needed to forward X11 out of a container to the desktop

X11 forwarding seems to work for us out-of-box (i.e. if I just do MPLBACKEND=tkagg python and plot something it works fine. This error only happened with qt

I misread as "containers locally" vs "containers on GH resources"


Missing information about parameter command_id

Type of issue
Missing information
Description
Description of parameter command_id is missing the parameter's accepted/known values.

Page URL
https://learn.microsoft.com/en-us/python/api/azure-mgmt-compute/azure.mgmt.compute.v2022_08_01.models.runcommandinput?view=azure-python
Content source URL
https://github.com/MicrosoftDocs/azure-docs-sdk-python/blob/main/docs-ref-autogen/azure-mgmt-compute/azure.mgmt.compute.v2022_08_01.models.RunCommandInput.yml
Document Version Independent Id
18f317c9-1d30-6651-da88-5c9077129041
Article author
@lmazuel
Metadata

ID: 6b52478a-aa4c-a2cb-c38c-ead396e49cdf
Service: virtual-machines

@juliabeltrao Thanks for the feedback! We are investigating and will get back to you when we have more information.

@juliabeltrao thank you for surfacing this issue, we are working to update the documentation for the commandid.


The directory modifiedIndex doesn't change.

Hi, all
Why I change a node's value in a directory, the directory's modifiedIndex doesn't change.
In my project, I have persist all the global settings in database and a etcd directory. And I persist the etcd directory modifiedIndex in the database.
Now, I need a sync from database to etcd. I don't want to compare every node instead of the modifiedIndex of the directory

@philips @xiang90

Why I change a node's value in a directory, the directory's modifiedIndex doesn't change.

This is the design of etcd store. If you want to know the difference between the database and etcd, you could watch on the directory and it will notify whenever there is a change: https://github.com/coreos/etcd/blob/master/Documentation/api.md#waiting-for-a-change

@fatduo The modified index of a directory only changes when you add/remove nodes into/from it. @yichengq has an answer for your use case. I am closing this issue. If you have any further questions, please feel free to follow up. Thanks!


PROV-N deserialization?

Hello, I would like to know whether a PROV-N deserializer is somewhere on the implementation roadmap? If not, I would like to contribute that; in case it is of interest of course.

Thanks, @MarcelPa. That'd be fab!
I've been thinking about doing this, but haven't found the time for it. @TomasKulhanek recently wrote an ANTLR grammar for PROV-N, which I believe can be used to build a PROV-N parser.
Are you interested in working on that? I'd be very happy to help with testing/integration when needed.

Great, I would like to work on that then. The ANTLR grammar will surely be really helpful for that, thanks @trungdong (and @TomasKulhanek of course).
I just forked the repo and will start to look into ANTLR; I hope to start pushing to the forked repo to the development branch starting next week.
Will keep you posted :-)

Excellent! Thanks a lot, @MarcelPa.

Quick question regarding testing: is there a pattern on how to create test files that can be found under tests/rdf for example? My approach would be to just copy the rdf documents and translate them into provn docs step by step.
FYI: ANTLR works fine so far, I got rid of raising NotImplementedError to successfully run all test cases in my virtual python environment.

Quite some type that I have pushed to my fork, therefore I am giving you an update via this issue:
Right now, the antlr-grammar seems to be erroneous in some cases, like langtags. Unfortunaly, I am not an expert in grammars, but I hope to get my head around them soon.

Thanks for the update, @MarcelPa. Unfortunately, I won't be of much help on ANTLR.
The PROV-N specs does use grammar rules, which you might find useful.

 I modified the grammar a little bit (basically just reordered some rules), now it seems to work properly :-)
I am down to 13 test cases which fail / error. Next step for me will be to rebase to the newest commit of the dev branch and keep on developing.
As for now, failed test cases seem to be incorrectly parsed float values from typed literals. What would you expect to insert into the attributes of an expression? Something like
Literal(somevalue, datatype="xsd:float", langtag=someLangtag)

or parsed native value, like
float(somevalue)

I think I used those a little bit inconsequently right now. I will need to refactor this one way or another ;-)

Thank you for the update and the work, @MarcelPa!
float(value) should work, I think. Do you have an example of a problematic case?
BTW, a Python float value is mapped to xsd:double by the package though.

I do: running test_entity_with_multiple_attribute fails. Both outputs are almost identical:
Parsed data from a debug print:
document
  prefix ex <http://example.org/>
  prefix ex_1 <http://example4.org/>

  entity(ex:emov, [ex:v_0="un lieu", ex:v_1="un lieu"@fr, ex:v_2="a place"@en, ex:v_3=1, ex:v_4=1, ex:v_5="1" %% xsd:short, ex:v_6="2" %% xsd:float, ex:v_7="1" %% xsd:float, ex:v_8="10" %% xsd:decimal, ex:v_9="1" %% xsd:boolean, ex:v_10="0" %% xsd:boolean, ex:v_11="10" %% xsd:byte, ex:v_12="10" %% xsd:unsignedInt, ex:v_13="10" %% xsd:unsignedLong, ex:v_14="10" %% xsd:integer, ex:v_15="10" %% xsd:unsignedShort, ex:v_16="10" %% xsd:nonNegativeInteger, ex:v_17="-10" %% xsd:nonPositiveInteger, ex:v_18="10" %% xsd:positiveInteger, ex:v_19="10" %% xsd:unsignedByte, ex:v_20="http://example.org" %% xsd:anyURI, ex:v_21="http://example.org" %% xsd:anyURI, ex:v_22='ex:abc', ex:v_23='ex:abcd', ex:v_24='ex_1:zabc', ex:v_25='ex_1:zabcd', ex:v_26="2019-03-27T12:52:02.266484" %% xsd:dateTime, ex:v_27="2019-03-27T12:52:02.266486" %% xsd:dateTime])
endDocument

versus the testcase data:
document
  prefix ex <http://example.org/>
  prefix ex_1 <http://example4.org/>

  entity(ex:emov, [ex:v_0="un lieu", ex:v_1="un lieu"@fr, ex:v_2="a place"@en, ex:v_3=1, ex:v_4=1, ex:v_5="1" %% xsd:short, ex:v_6="2" %% xsd:float, ex:v_7="1.0" %% xsd:float, ex:v_8="10" %% xsd:decimal, ex:v_9="1" %% xsd:boolean, ex:v_10="0" %% xsd:boolean, ex:v_11="10" %% xsd:byte, ex:v_12="10" %% xsd:unsignedInt, ex:v_13="10" %% xsd:unsignedLong, ex:v_14="10" %% xsd:integer, ex:v_15="10" %% xsd:unsignedShort, ex:v_16="10" %% xsd:nonNegativeInteger, ex:v_17="-10" %% xsd:nonPositiveInteger, ex:v_18="10" %% xsd:positiveInteger, ex:v_19="10" %% xsd:unsignedByte, ex:v_20="http://example.org" %% xsd:anyURI, ex:v_21="http://example.org" %% xsd:anyURI, ex:v_22='ex:abc', ex:v_23='ex:abcd', ex:v_24='ex_1:zabc', ex:v_25='ex_1:zabcd', ex:v_26="2019-03-27T12:52:02.266484" %% xsd:dateTime, ex:v_27="2019-03-27T12:52:02.266486" %% xsd:dateTime])
endDocument

The difference is noticable at ex:v_7="1.0" %% xsd:float, which will be parsed as a float but returned as ex:v_7="1" %% xsd:float.
So far, I did not notice any changes happening from float to double.

@MarcelPa Just a quick question what the status of the PROV-N deserialiser is. It's been a good year, and it looked like things weren't far off.

Oh my, I completely lost track of this issue, thanks for the reminder @pohutukawa !  Will keep you posted :-)

I am back at finding out how antlr4 works (any help is appreciated!). For reasons I do not yet understand, langtags and some int_literals will fail to parse, which gives me 57 fails of 185 unit tests. Once I figure out how to fix that, PROV-N deserialization should near its completion.

That looks promising!
Even if there are some "glitches" as in the comment above (where a float is parsed to ex:v_7="1" %% xsd:float), I'd be fully happy, as the value and its type is still preserved, and only the formatting (to 1.0) is lost.

Hi @MarcelPa, Wondering if you've had any progress on this deserializer? I'm wanting to work some more with Prov-n but seem quite limited without the ability to store and extract from Prov-n strings. I'm taking a bit of a look at the code and the tools to see if I could help but it's a little bit beyond me at this stage
Thanks!

I also found a mildly hacky way to convert in and out of Prov-n using the java ProvToolbox and provconvert,
Saving the file as a .provn then using provconvert to spit it out as .json, and then using the python deserialiser to get it back as a ProvDocument.

We've been trying to use @MarcelPa's feature branch that can parse PROV-N with decent results so far. Though, it's not based on the current 2.0 version, yet, so that's a bit of a pity.
If the ANTLR3 grammar by Luc is more complete, would that be an option to move forward on? (Even though it may be more "sexy" to use a current ANTLR4 grammar.) After all, there is a antlr3_python_runtime Python module as well.
I'm just searching for ways to not create any inconsistencies between individual approaches, for the case that the ANTLR4 grammar may differ from the ANTLR3 one ...


protoc-gen-go: new message struct structure prevents struct copying in Go

What version of protobuf and what language are you using?
protoc-gen-go v1.20.0,   protoc v3.11.4
What did you do?
Re-generating code with the newer protoc-gen-go breaks code that is copying generated message structs. Go vet is showing the following error:
assignment copies lock value to <var name>: <MsgStructType> contains google.golang.org/protobuf/internal/impl.MessageState contains sync.Mutex
Previous generated code for a simple struct:
type AgentRegisterMessage struct {
	ID                   string   `protobuf:"bytes,1,opt,name=ID,proto3" json:"ID,omitempty"`
	IP                   string   `protobuf:"bytes,2,opt,name=IP,proto3" json:"IP,omitempty"`
	Environment          string   `protobuf:"bytes,3,opt,name=Environment,proto3" json:"Environment,omitempty"`
	TestID               string   `protobuf:"bytes,4,opt,name=TestID,proto3" json:"TestID,omitempty"`
	Status               string   `protobuf:"bytes,5,opt,name=Status,proto3" json:"Status,omitempty"`
	XXX_NoUnkeyedLiteral struct{} `json:"-"`
	XXX_unrecognized     []byte   `json:"-"`
	XXX_sizecache        int32    `json:"-"`
}

Newly generated code, including the indirect reference to sync.Mutex:
type AgentRegisterMessage struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	ID          string `protobuf:"bytes,1,opt,name=ID,proto3" json:"ID,omitempty"`
	IP          string `protobuf:"bytes,2,opt,name=IP,proto3" json:"IP,omitempty"`
	Environment string `protobuf:"bytes,3,opt,name=Environment,proto3" json:"Environment,omitempty"`
	TestID      string `protobuf:"bytes,4,opt,name=TestID,proto3" json:"TestID,omitempty"`
	Status      string `protobuf:"bytes,5,opt,name=Status,proto3" json:"Status,omitempty"`
}

What did you expect to see?
The new message struct should be copy-able, or at least an explicit backwards compatibility change should be highlighted in the release notes.
What did you see instead?
assignment copies lock value to <var name>: <MsgStructType> contains google.golang.org/protobuf/internal/impl.MessageState contains sync.Mutex
Anything else we should know about your project / environment?

This is working as intended. Generated messages to be shallow-copyable since they posses fields that are accessed atomically. This was already the case prior to the new release and has been so for several years now.
If you look at the impl.MessageState type, you'll see that it embeds a pragma.DoNotCopy type, which is just a zero-sized array of sync.Mutex. The allows Go vet to flag what was already a pre-existing problem.
To copy a message, one should use proto.Clone.

This is somewhat surprising, since the old structs were often very simple Go structs, and I don't really understand what you mean by

Generated messages should not [be] shallow-copied since they posses fields that are accessed atomically.

I would be grateful if you could point me to some more information on this topic of atomic access vs struct copying.
Regardless, I have not been able to find any mention of this restriction in the docs, so if this was already assumed and it is now explicitly enforced, this is a clear positive change then. I will close the issue then.
Thank you for the quick reply!

posses fields that are accessed atomically.

The old XXX_sizecache, XXX_extensions, and others are examples of fields that are accessed atomically.

I would be grateful if you could point me to some more information on this topic of atomic access vs struct copying.

In general, its not safe to shallow copy any struct type where you don't control the implementation since you have no guarantee that you aren't accidentally copying atomic variables and locks (which is exactly the case here). This expectation is not specific to protobufs, but more of a general idiom.

Does that mean that I can't use range based for-loop on non-scalar repeated fields and MUST use index value to access them?


CC get_certs_from_web fails

Traceback below:
 File "/var/www/sec-certs/sec_certs_page/cc/tasks.py", line 58, in process
    dset.get_certs_from_web(update_json=False)
  File "/var/www/sec-certs/sec-certs/sec_certs/serialization/json.py", line 65, in inner_func
    result = func(*args, **kwargs)
  File "/var/www/sec-certs/sec-certs/sec_certs/dataset/common_criteria.py", line 268, in get_certs_from_web
    html_certs = self._get_all_certs_from_html(get_active, get_archived)
  File "/var/www/sec-certs/sec-certs/sec_certs/dataset/common_criteria.py", line 413, in _get_all_certs_from_html
    partial_certs = self._parse_single_html(self.web_dir / file)
  File "/var/www/sec-certs/sec-certs/sec_certs/dataset/common_criteria.py", line 488, in _parse_single_html
    certs.update(_parse_table(soup, cert_status, key, val))
  File "/var/www/sec-certs/sec-certs/sec_certs/dataset/common_criteria.py", line 453, in _parse_table
    x.dgst: x for x in [CommonCriteriaCert.from_html_row(row, cert_status, category_string) for row in body]
  File "/var/www/sec-certs/sec-certs/sec_certs/dataset/common_criteria.py", line 453, in <listcomp>
    x.dgst: x for x in [CommonCriteriaCert.from_html_row(row, cert_status, category_string) for row in body]
  File "/var/www/sec-certs/sec-certs/sec_certs/sample/common_criteria.py", line 509, in from_html_row
    raise

Happened together with log:
[2022-05-12 01:14:58,477: ERROR/ForkPoolWorker-2] Unexpected number of cells in CC html row.

This is likely a change on the CC side as the error happened on the nightly run on 11.05. -> 12.05. and there was no tool update at that point (the older version 0.0.3 was still used).

Btw, an empty raise in Python tries to reraise an existing exception, but that only works if there is one (i.e. it is being done in an except: handler, so using it without that is invalid.

@J08nY So, it appears that this was just temporary problem with CC page. I added some more details exceptions.
But detecting and reatemptting this robustly would constitute a non-trivial effort. I don't think we need to further address it at the moment.
But this reveals more general problem: Exception raising and handling. There's no coherent design w.r.t. this, it's all somewhat random and the CLI has no exception handling for instance.  But maybe this is an opportunity to finally learn how to do this well :)


easyengine installation on existing setup

Test - PHP upgrade
Ubuntu with php 5.3
apt-get install php
Apache 80
PHP 5.3

setup check
PHP 5.4
Apache disable (ask)

Update: We don't want to do this. It might get supported automatically as we are using dockers in v4, but we won't be putting additional efforts dedicated to this.


Fix typo in Builder.php

Codecov Report

Merging #45 into master will not change coverage.
The diff coverage is 0%.

@@            Coverage Diff             @@
##             master      #45    +/-   ##
==========================================
  Coverage     97.44%   97.44%
- Complexity        0      182   +182
==========================================
  Files            17       17
  Lines           430      430
==========================================
  Hits            419      419
  Misses           11       11

Impacted Files
Coverage Δ
Complexity Δ

src/Builder.php
94.79% <ø> (ø)
64 <ø> (+64)
:white_check_mark:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 90a5788...972585b. Read the comment docs.

Thank you very much!


Migrate tests to Node test runner

Keep it simple.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 92.617%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
222

Relevant Lines:
231

💛  - Coveralls


Return status code different from zero when there is an error

Picocli always returns status code 0, even when an error happens. For example, I forced an error by not passing an option value:
./myCLI myOption -s
Missing required parameter for option '--srcdir' (<path>)

And then, checking the status code, it's zero:
echo $?
0

I'm wondering if Pococli could return status 1 when an issue happens. That would be very useful for automation purposes. For instance, I have some automation and I am relying on the status code to know that an issue happened.

This should not happen.
Picocli’s execute method returns a non-zero value if required options are missing.
Your application should call System.exit with the return value of the execute method.
int exitCode = new CommandLine(new MyApp()).execute(args);
System.exit(exitCode);

Can you show what your code looks like?

I'm so sorry, I did a mistake. The tests did not include the System.exit(exitCode); as you mentioned. I fixed that and Picocli is working very well. sorry about the false alarm.

Great, glad to hear it works as expected. 😅
Enjoy picocli!


Run all tests in debug mode in CI

Running the WPT suite locally I noticed that some of the tests that pass in release mode hit asserts in the debug build. To avoid a proliferation of this, we should ensure that we run tests in debug mode, too.
To avoid making CI take way too long, we could mark more tests as SLOW, or perhaps even introduce a new DEBUG-SLOW annotation to skip them only in debug builds.
Additionally, we can look into running the suite with some parallelization. That wouldn't be too hard, with the main issue being that we'd want to pipe outputs correctly to not have completely garbled logs.

This was added in https://github.com/bytecodealliance/StarlingMonkey/pull/113.


Add support for Schema Contexts

The tool should support contexts for all capabilities.

Hi @abraham-leal, you may use contexts adding the SR URL with the context "embeded" (like https://<SR>.confluent.cloud/contexts/.mycontext), the problem is the GetSubjectsWithVersions will store the destination subjects with :.mycontext:subject and this will make the DeepEqual method inside the Sync to never find the subjects.
A quick fix is to remove the context when creating the map under GetSubjectsWithVersions.


Update zio, zio-test, zio-test-magnolia, ... to 2.0.18

About this PR
📦 Updates

dev.zio:zio
dev.zio:zio-test
dev.zio:zio-test-magnolia
dev.zio:zio-test-sbt

from 2.0.13 to 2.0.18
📜 GitHub Release Notes - Version Diff
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "dev.zio" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "dev.zio" }
}]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #25.