bbclient ignores --cache-folder on CLI

I was attempting to set a cache folder and I noticed that I had to set the environment variable to have bbclient use a non-default cache location. Using the argument on the CLI does not appear to work.

@ClaudeHu can you take a look

Fixed in 0.4.1


Jayanta | Update xp to reach level 2 per Patch 7.07

Can you replicate the change to build file in the build directory?

Yessir!


[release-4.6] Fix invalid symlink on second boot

This PR includes:

#8 - This replaces echo -n way to create files on host
Remove additional refs (FCOS version tag) which confuses bootstrap (it expects a single ref)
Avoid creating /etc/resolv.conf symlink to systemd-resolved managed file

Related to https://github.com/openshift/okd/issues/380

/test e2e-gcp

/test e2e-gcp


Context menu's showing up in light mode

I am not sure why, but out of nowhere the context menu's all are showing up in light mode now, even though my PC is in dark mode. All of my nss files are all the defaults and I have tried to uninstall and reinstall.

Shell: v1.9.15
Edition	Windows 11 Pro for Workstations
Version	23H2
Installed on	2023-07-08
OS build	22631.3235
Experience	Windows Feature Experience Pack 1000.22687.1000.0

https://github.com/moudey/Shell/issues/418#issuecomment-1949169104

الحل بسيط جدا
فقط في ملف theme.nss
قم بحذف كلمة dark.
لتبقى هكذا
dark = sys


Possible regression "weakly-referenced JS value"

Hello,
Lately I have this strange error popping up from my NAOT library which is built using the latest version 0.7.30 of node-api-dotnet:

Error: The weakly-referenced JS value not available.
at Microsoft.JavaScript.NodeApi.JSReference.GetValue() + 0xe6
at Microsoft.JavaScript.NodeApi.Interop.JSRuntimeContext.GetOrCreateObjectWrapper[T](T, Func'1) + 0x6a
at Microsoft.JavaScript.NodeApi.Interop.JSRuntimeContext.GetOrCreateObjectWrapper[T](T) + 0x8d
at Microsoft.JavaScript.NodeApi.Generated.Module.get_MicrosoftNodeapiNested_Parent_Child(JSCallbackArgs __args) + 0x90
at MicrosoftNodeapiNested!+0x128c39
at Microsoft.JavaScript.NodeApi.JSValue.InvokeCallback[TDescriptor](JSRuntime.napi_env, JSRuntime.napi_callback_info, JSValueScopeType, Func`2) + 0x1bc

I have made a small app and library to replicate the bug as it happens in my original NAOT library:
[JSExport]
public static class Program
{
    public static Parent GetParentInstance() {
        return new Parent();
    }
}`
[JSExport]
public class Parent {
    public string ParentName { get; set; } = "not set";
    public Child Child { get; set; } = new Child();
}
[JSExport]
public class Child {
    public string ChildName { get; set; } = "not set";
    public ChildChild ChildChild { get; set; } = new ChildChild();
}
[JSExport]
public class ChildChild {
    public string ChildChildName { get; set; } = "not set";
}

Then I've ran dotnet publish (with aot and node module settings) and then npm pack  ( type="module" in package.js) to obtain the local npm package and made this demo mjs app:
import {Program , Parent, Child, ChildChild} from "./node_modules/microsoftnodeapinested/MicrosoftNodeapiNested.mjs";
const p = Program.getParentInstance();
console.log("parent name:",p.parentName);
const ccname = p.child.childChild.childChildName;
console.log("immediate ccname:",ccname);
let iter = 0;
setInterval(() => {
    console.log("loop #",iter++, " ccname:",p.child.childChild.childChildName);
}, 3000);

After about 5 loops I get that error.
This could be similar to a previous resolved issue (#292 which I have opened a while ago) but no nullable properties are involved.
I am not sure since when this started happening but as far as I remember, version 0.7.0 - 0.7.1 did not had it, therefore this could be a regression bug (perhaps while tying to fix some other memory-leak related issues?)
Thanks in advance!

Thanks for the detailed report. There was a recent change related to references, so it's possible that caused this regression. I'll investigate.


Unsupported properties should be ignored

Right now if there is a property in the input CSS that is not supported by React Native, it will crash. It would be good to just ignore them.

Created a pull request at #23

PR accepted, @caiosba when you get a chance can you add tests for that PR?

Added tests and opened a new PR for that at #25

@caiosba awesome thank you.

Right now if there is a property in the input CSS that is not supported by React Native, it will crash.

Doesn't that mean you're using invalid CSS? If that's the case, it's probably better to be noticed early on or you'll end up with unknown bugs.

Not necessarily.  You might be reusing css from a web project; RN supports only a subset of  CSS.
Sent from my iPhone

On Jun 8, 2016, at 7:52 AM, Gabin Aureche<EMAIL_ADDRESS>wrote:
Right now if there is a property in the input CSS that is not supported by React Native, it will crash.
Doesn't that mean you're using invalid CSS? If that's the case, it's probably better to be noticed early on or you'll end up with unknown bugs.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.


Using docker executor

Is it possible to use the docker executor with this image? If so how do I supply the base image and link services?

+1

While I have never tried it, I don't see any reason why it will not work. To use the docker executor, you will need to volume mount your docker binary and unix socket into the runner. e.g.
docker run -it --rm \
  <other options>
  -v /var/run/docker.sock:/run/docker.sock \
  sameersbn/gitlab-ci-multi-runner

see https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/install/docker.md for more information

Hi,
It seems that the @sameersbn solution doesn't work.
It would be awesome if we could use the docker executor.
Otherwise, is it possible to do some 'nested docker'?

I have not tried this yet. I will give it a shot and let you know.
BTW, if you are eager to use docker as the executor, you can simply run the gitlab-ci-multi-runner binary on the host. Its a single binary. This image is more of a base image to build your self contained runner images in which case the shell executor makes more sense.
However, I will try the docker executor and let you know my findings.

problem was with access to the docker socket. Also during registration gitlab-ci-multi-runner asks some more questions. Dunno if they can be skipped.
First time run using:
docker run --name runner -it --rm \
  -v /run/docker.sock:/run/docker.sock \
  -v /srv/docker/gitlab-ci-multi-runner:/home/gitlab_ci_multi_runner/data \
  sameersbn/gitlab-ci-multi-runner

It will ask you to provide the relevant details.
I was able to trigger builds in docker containers. I have not tested an actual build, but the docker executor works.
FWIW i used the following build description
test:
  script:
  - pwd
  - ls -la

Thanks for this. I've pulled the latest image and I'm using the following command to start a runner:
docker run -d --name gitlab-ci-multi-runner --restart=always \
--volume /opt/gitlab-ci-multi-runner:/home/gitlab_ci_multi_runner/data \
--volume /var/run/docker.sock:/var/run/docker.sock \
--env='CI_SERVER_URL=http://<IP_ADDRESS>:10080/ci' \
--env='RUNNER_TOKEN=[redacted]' \
--env='RUNNER_DESCRIPTION=Docked runner' \
--env='RUNNER_EXECUTOR=docker' \
--env="DOCKER_IMAGE=node:0.12" sameersbn/gitlab-ci-multi-runner:0.5.5

But when I try to run a build I get:
gitlab-ci-multi-runner 0.5.5-1-g69bc934 (69bc934)
Using Docker executor with image node:0.12 ...
Pulling docker image node:0.12 ...

ERROR: Build failed with: dial unix /var/run/docker.sock: permission denied

I can get it working using the gitlab/gitlab-runner image but that requires executing gitlab-runner register separately and I'd like to have the whole thing automated from docker-compose.yml. Any ideas?

oh wait.. i was supposed to do another change. completely skipped my mind.

Have made a required change. Please check if it fixes your issue.
I was able to test using your docker run command.

Hmmm. Now I get groupadd: GID '100' already exists in the container log after startup. I'm using boot2docker 1.8.2 on OSX.

oh.. then I may have to add some more checks.

have introduce some more changes. Please pull in a couple of minutes (its building) and retry.

Awesome! That's working now. Thanks for all your help. 👍

@tamlyn
I hope you don't mind me jumping in here.  I subscribed to this thread earlier, since I was trying to get a better understanding of how a runner could be set up with docker-compose ... something that it seems you have been able to do.  Would you be able to offer a quick summary here of how you were able to get this working ... it seems you are using a base node image to create a runner dynamically ... or am I misunderstanding?  I have extended the docker-compose.yml in the sameersbn/docker-gitlab repo to include the runner? Thanks.

I haven't done a full end to end run through yet but I'll post back here when I do.

Much appreciated. Thanks.

one thing i did notice that the runner image needs to have git installed. But you must have figured that by now :)

@Analect try this http://tamlyn.org/2015/10/dockerised-gitlab-ci-runner/
I may well have missed some steps so let me know if you get stuck or something is wrong.

@tamlyn
Much appreciated.  Will work through this and revert back.

@tamlyn
Sorry for long delay in responding to this and thanks again for documenting this. I worked through your blog-post linked above, which was very thorough.  The only step that is explicity missing is the need to redeploy the gitlab container too .. since the GITLAB_HOST gets changed to reflect the container end-point on tutum.
The build is failing as per screenshot below.  Any idea what I might be missing.  Is the fact that the 'runner' container doesn't have docker installed?  ... or is it something else.

I notice in the docker-compose ... I presumably need to change the CI_SERVER_URL ... also to
gitlab-1.gitlab.xxxxx.cont.tutum.io/ci ... although, even if I do this, the build is still erroring-out as above.
The other thing I'm finding with tutum's script editor ... is that it randomly changes the docker-compose.yml to wrap certain elements in single quotes ... as strings ... but not others ... have you had any issues with that?
Thanks,
Colum
gitlab:
  image: 'sameersbn/gitlab:8.0.3'
  environment:
    - GITLAB_HOST=gitlab-1.gitlab.xxxxx.cont.tutum.io
    - GITLAB_PORT=10080
    - GITLAB_SECRETS_DB_KEY_BASE=long-and-random-alphanumeric-string
  links:
    - postgresql
    - 'redis:redisio'
  ports:
    - '10080:80'
  volumes:
    - '/srv/docker/gitlab/gitlab:/home/git/data'
node-runner:
  image: 'sameersbn/gitlab-ci-multi-runner:latest'
  environment:
    - 'CI_SERVER_URL=http://gitlab/ci'
    - 'DOCKER_IMAGE=node:0.12'
    - 'RUNNER_DESCRIPTION=Node runner'
    - RUNNER_EXECUTOR=docker
    - RUNNER_TOKEN=385072abd2d887484011
  volumes:
    - /var/run/docker.sock
    - '/opt/gitlab-ci-multi-runner:/home/gitlab_ci_multi_runner/data'
postgresql:
  image: 'sameersbn/postgresql:9.4-7'
  environment:
    - DB_NAME=gitlabhq_production
    - DB_PASS=password
    - DB_USER=gitlab
  volumes:
    - '/srv/docker/gitlab/postgresql:/var/lib/postgresql'
redis:
  image: 'sameersbn/redis:latest'
  volumes:
    - '/srv/docker/gitlab/redis:/var/lib/redis'


add asserts in bulk transfer for checks vector empty and one type symbol token

Issue moved to GolosChain/cyberway.contracts #161 via ZenHub


arm: sample apps and build support for PSA TF-M and Zephyr

This PR enables CI checks to pass for TF-M, and introduces build support to automatically pull in the external TF-M repository and all dependencies, and perform a build of TF-M as part of a standard Zephyr sample app build procedure. It is based on TF-M 1.0-rc2.

This should be a west project and a zephyr module. Please open an issue to create a TF-M repo under the zephyrproject-rtos and then be able to point west.yml to it. Please follow the instructions here:
https://docs.zephyrproject.org/latest/guides/modules.html

Hi,
The TF-M project and it's related repos are standalone projects. And the TF-M is not a module or a part of Zephyr. TF-M run in secure execution and Zephyr run in nonsecure execution, the compile seperately to different binaries. May be this requires more discussions or ideas on this point for further. Anyway, created an issue for new module request.
Thanks

@microbuilder I am working on Musca B1 to make it reproduce psa arch tests pass. The Zephyr is 2.0.0. Output like below:
`[Sec Thread] Secure image initializing!
***** Booting Zephyr OS build zephyr-v2.0.0-1393-gec3f603b8a7a *****
TF-M PSA Arch Tests with Zephyr on musca_b1_nonsecure
***** PSA Architecture Test Suite - Version 0.9 *****
Running.. Crypto Suite

....
************ Crypto Suite Report **********
TOTAL TESTS     : 44
TOTAL PASSED    : 43
TOTAL SIM ERROR : 0
TOTAL FAILED    : 0
TOTAL SKIPPED   : 1

Entering standby..
`

What is this commit for ?
"Merge github.com:zephyrproject-rtos/zephyr into psa_arch_tests_for_tf…"

This is a consequence of updating the PR against master, pulling recent changes into the PR ... I can remove this and most other superfluous commit messages before the final PR review to clean this up.

This should be a west project and a zephyr module. Please open an issue to create a TF-M repo under the zephyrproject-rtos and then be able to point west.yml to it. Please follow the instructions here:
https://docs.zephyrproject.org/latest/guides/modules.html

Hi Carles ... I understand the idea behind the module request, but I would argue in this case that this is specifically a sample with some BSP fixes thrown on top, and not a module as such since it doesn't add TFM in the PR itself. The cmake file does do some work to pull TF-M in as an external project as part of the normal Zephyr build process, build it with appropriate options, and link against that for the secure processing environment, but the core code here is exclusively a sample application for Zephyr, which doesn't seem inappropriate to me to include in the samples folder(?).

Why combine all commit history into one in the PR?

Why combine all commit history into one in the PR?

It makes the commit easier to refer to in the future once it gets merged in, versus 15 related but overlapping commits.

Why combine all commit history into one in the PR?

It makes the commit easier to refer to in the future once it gets merged in, versus 15 related but overlapping commits.

Hi Kevin,
I do not think so. The commits means history and the steps of how the feature enabled. All patches are trackable by git log. And also easier for reviewer. I do combine some patches because they just fix few words that in the same document. For other patches, I'd prefer keep them as original.

So I think we should see about having TF-M as a west module and setting up a repo under the zephyr project that tracks the TF-M repo (and maybe combines all the components that TF-M needs as its own west manifest).

Hi,
If put TFM and all dependencies as modules, is that means everyone will download them by west update, is it? And the modules seem using the zephyr CMake style to build each module, I think to do the same for TF-M project is a big effort. The TFM eventually not as a part of zephyr.bin, How do you think about the way we using now? To keep TF-M independent pull, and compile.

If put TFM and all dependencies as modules, is that means everyone will download them by west update, is it? And the modules seem using the zephyr CMake style to build each module, I think to do the same for TF-M project is a big effort. The TFM eventually not as a part of zephyr.bin, and all dependencies projects size is large, so implement all the work like it shows in this PR. To keep TF-M independent pull, and compile.

Yes, the modules would be installed via the usual west update command, and most likely this will require a fork of trusted-firmware-m.git into a new project like zephyrproject-rtos/tfm to host the TF-M code, and any support files similar to what happens with open-amp today: https://github.com/zephyrproject-rtos/open-amp

If put TFM and all dependencies as modules, is that means everyone will download them by west update, is it? And the modules seem using the zephyr CMake style to build each module, I think to do the same for TF-M project is a big effort. The TFM eventually not as a part of zephyr.bin, and all dependencies projects size is large, so implement all the work like it shows in this PR. To keep TF-M independent pull, and compile.

Yes, the modules would be installed via the usual west update command, and most likely this will require a fork of trusted-firmware-m.git into a new project like zephyrproject-rtos/tfm to host the TF-M code, and any support files similar to what happens with open-amp today: https://github.com/zephyrproject-rtos/open-amp
This would have the advantage of making it clearer which commit of TF-M we are currently using, and we can also add the TF-M dependencies into this new repo (mbed-crypto, CMSIS_5, etc.) so that we no longer rely on an awkward git clone solution to build.

I know that, and was wrok on the openamp module. Since that script can be updated. https://github.com/zephyrproject-rtos/zephyr/commit/469ad5218be949ee421dc6d6355d5553fbd7ee5b

So I think we should see about having TF-M as a west module and setting up a repo under the zephyr project that tracks the TF-M repo (and maybe combines all the components that TF-M needs as its own west manifest).

Hi,
If put TFM and all dependencies as modules, is that means everyone will download them by west update, is it? And the modules seem using the zephyr CMake style to build each module, I think to do the same for TF-M project is a big effort. The TFM eventually not as a part of zephyr.bin, and all dependencies projects size is large, so implement all the work like it shows in this PR. To keep TF-M independent pull, and compile.

If put it in module, west update will have 730 MB larger for every one to include TF-M and it's dependent projects.
All Cmake files in TFM does not using zephyr_xxxx() now, and I do not think it is vaulable to rework all Cmake files on a specific TF-M version, TF-M evolves all the time.
tfm_s.bin is not a part in zephyr.bin, not sure if it is properly as a module.
Just my oponion on it, thanks.

So I think we should see about having TF-M as a west module and setting up a repo under the zephyr project that tracks the TF-M repo (and maybe combines all the components that TF-M needs as its own west manifest).

Hi,
If put TFM and all dependencies as modules, is that means everyone will download them by west update, is it? And the modules seem using the zephyr CMake style to build each module, I think to do the same for TF-M project is a big effort. The TFM eventually not as a part of zephyr.bin, and all dependencies projects size is large, so implement all the work like it shows in this PR. To keep TF-M independent pull, and compile.

If put it in module, west update will have 730 MB larger for every one to include TF-M and it's dependent projects.

we should be able to construct the tf-m module repo in such a way that its not that big, we don't need all of CMSIS for example.

All Cmake files in TFM does not using zephyr_xxxx() now, and I do not think it is vaulable to rework all Cmake files on a specific TF-M version, TF-M evolves all the time.
tfm_s.bin is not a part in zephyr.bin, not sure if it is properly as a module.
Just my oponion on it, thanks.

So I think we should look at a middle ground here to start with.  We should use west as the means of getting the TFM source, how we build it after that we can figure out.  I agree it doesn't make sense to introduce/change the TFM CMake files.  However we should be able to do something similar to openamp.

@tejlmand could you please take a look at the integration between the 2 CMake build systems?

@tejlmand could you please take a look at the integration between the 2 CMake build systems?

I havent been able to get ninja to do the TF-M build first, the way that make does. I believe the dependencies are being setup correctly, with the add_dependencies function:
ExternalProject_Add(
  tfm
  SOURCE_DIR ${TFM_BASE_DIR}
  CMAKE_ARGS -DPROJ_CONFIG=${TFM_BASE_DIR}/configs/ConfigRegressionIPC.cmake
             -DTARGET_PLATFORM=${TFMBOARD}
	     -DBL2=True
	     -DCOMPILER=GNUARM
	     #-DGNUARM_PREFIX=arm-zephyr-eabi
	     #-DGNUARM_PATH=${ZEPHYR_SDK_INSTALL_DIR}/arm-zephyr-eabi
	     -DGNUARM_PREFIX=arm-none-eabi
	     -DGNUARM_PATH=/usr/local/prog/gcc-arm-none-eabi-7-2018-q2-update
  BUILD_BYPRODUCTS ${BINARY_DIR}/tfm/veneers/s_veneers.o
)
add_dependencies(app tfm)

With ninja:
$ cd samples/tfm_integration/tfm_ipc/
$ mkdir build && cd build
$ cmake -GNinja -DBOARD=mps2_an521_nonsecure ..
$ ninja
ninja: error: 'tfm-prefix/src/tfm-build/install/export/tfm/veneers/s_veneers.o', needed by 'zephyr/zephyr_prebuilt.elf', missing and no known rule to make it

With make:
$ cmake -DBOARD=mps2_an521_nonsecure ..
$ make run
...
[TF-M build output]
...
[Sec Thread] Secure image initializing!
TF-M isolation level is: 1
Booting TFM v1.0
*** Booting Zephyr OS build v1.12.0-rc1-19023-g10461e3708e7  ***
The version of the PSA Framework API is 256.
The minor version is 1.
Connect success!
TFM service support minor version is 1.

@microbuilder I changed a little in the CMakeLists.txt to get it running locally with ninja
You can try to fetch: https://github.com/karl-zh/zephyr/pull/1 and do west update.
This only corrects the dependency issues.
Note, there are still some build issues that @karl-zh  should look into.

Made a couple of small suggestions for #19985 just to get it to build.
Those are found here:
karl-zh#1
zephyrproject-rtos/trusted-firmware-m#3

Thanks, tested on the musca_b1 and an521 build targets, and resolved the linker issue (it was due to the output path changing for the TF-M build), and works as expected now, once zephyrproject-rtos/trusted-firmware-m#3 is merged in.

Very good to see this work started.
I have some thoughts about the IPC sample. The main file is largely copied from test/suites/ipc/non_secure/ipc_ns_interface_testsuite.c in TF-M. I would really prefer that this file is used as is, from TF-M. This involves adding some more files to the build to allow calling the PSA functions by their proper name (not calling the *_veneer() functions directly). Also, AFAIU "IPC" in TF-M means calling the Secure APIs through a mailbox, which will need additional files added (see app/CMakeLists.txt for more info).
The most important thing IMO is allowing the Zephyr user to call PSA APIs as they would if they developed on TF-M directly, even if we don't support both IPC and library calls.
See this if you haven't before: https://ci.trustedfirmware.org/job/tf-m-build-test-nightly/lastSuccessfulBuild/artifact/build-docs/tf-m_documents/install/doc/user_guide/html/docs/user_guides/tfm_build_instruction.html#configurations

I agree this could be done differently, and I have two other demos that I need to get in, one using library mode, and the other to run the PSA FF API, etc., tests as a sample app, but this might need to happen after the 2.3 merge window closes Friday.

@oyvindronningstad I agree this could be done differently, and I have two other demos that I need to get in, one using library mode, and the other to run the PSA FF API, etc., tests as a sample app, but this might need to happen after the 2.3 merge window closes Friday.

Sounds good. The one thing I want to get in before the merge window is allowing the PSA APIs to be used by their proper names, so that once the PSA APIs are added, it doesn't break things for the user.

@microbuilder Placed some additional code, just to provide a starting point, without changing current design.
https://github.com/karl-zh/zephyr/pull/2
https://github.com/zephyrproject-rtos/trusted-firmware-m/pull/6
Basically, it move the ExternalProject_Add() to the TFM Zephyr module, and then calls trusted_firmware_build instead.
Now, the question, which I think @oyvindronningstad is also addressing, is how do we want to integrate TF-M into Zephyr in general.
Cause I don't assume we want to add trusted_firmware_build throughout Zephyr samples.
Also, there could be downstream users with a different solution, or not wanting to clone TF-M, and such a function will break the build, if it is missing.
Do you have some thoughts regarding the future integration that  you can share ?

which actually makes me notice there is no documentation accompanying this PR.
It should be documented how users can enable and use TF-M.

which actually makes me notice there is no documentation accompanying this PR.
It should be documented how users can enable and use TF-M.

There is some documentation at the board level, but no high-level TF-M documentation, I agree. I'll add something to get started in with the next push, once I'm done some further cleanup with the build system.

@microbuilder this needs to be moved out of Draft ASAP if it is to be merged in time for 2.3.

@microbuilder this needs to be moved out of Draft ASAP if it is to be merged in time for 2.3.

@carlescufi Making some final changes to it tonight, and hope to flip the switch tomorrow morning for review. Sorry for the delay.

@oyvindronningstad I agree this could be done differently, and I have two other demos that I need to get in, one using library mode, and the other to run the PSA FF API, etc., tests as a sample app, but this might need to happen after the 2.3 merge window closes Friday.

Sounds good. The one thing I want to get in before the merge window is allowing the PSA APIs to be used by their proper names, so that once the PSA APIs are added, it doesn't break things for the user.

@oyvindronningstad This commit should resolve the issue you brought up, and PSA API functions can now be called directly from the sample app: https://github.com/zephyrproject-rtos/zephyr/pull/19985/commits/ce2726d1b603157b4d1c62c61ec019613ce11424

@oyvindronningstad I also added a more complex 'real world' example making use of the attestation, crypto and secure storage modules here, accessing the PSA APIs with the normal names in IPC mode of course: https://github.com/zephyrproject-rtos/zephyr/pull/19985/commits/4e2173e3269f70559ae6e9ea79bbd3ebe5aaa582

Just a gentle reminder @microbuilder (in case you 've forgotten) there are some commit messages to be filled in.

Just a gentle reminder @microbuilder (in case you 've forgotten) there are some commit messages to be filled in.

@ioannisg Was just fixing those, done now.

Tagging as DNM because of the missing SHA in west.yml. Otherwise this looks good to go to me.

@ioannisg take another look please, commits have been rebased into a logical set

@microbuilder thanks for taking over this PR and for bringing this to merge-able state
@tejlmand @oyvindronningstad thanks for the reviews!


update for 1.18.2

Should fix #1, tested with Sodium 1.18.2-0.4.1+build.15.
Also bumped version to 1.0.2, since the listed version on Modrinth is 1.0.1.
It's really just a one line change though :)

Thank you!


docs: channel object / $ref note.

title: ✏ Update docs: Channel Object / $ref note.
This is to prevent confusion around whether folks can/should make use of $refs under #/channels.
Hopefully it is

True
Does not discourage use of $ref in $.channels.*.$ref
Conforms to https://github.com/asyncapi/spec/issues/607#issuecomment-1016605546

cc @char0n @smoya

LGTM. @ponelat I changed the title to make it conformant to the rules.

Not sure you saw (I think its marked as resolved?) we were discussing about the wording here https://github.com/asyncapi/spec/pull/826#discussion_r954667501

No, didn't see. Thanks for pointing out.

Hi @derberg, would you mind looking in this PR? Thanks!

Hey, sorry but I'm not a native speaker, and Using the $ref field has been deprecated when accompanied by other fields is totally confusing for me. For me, deprecated means deprecated and when accompanied by other fields is confusing. What other fields? Users should not use $ref at all as it is deprecated and in 3.0 goes away -> https://github.com/asyncapi/spec/pull/777

Hey, sorry but I'm not a native speaker, and Using the $ref field has been deprecated when accompanied by other fields is totally confusing for me. For me, deprecated means deprecated and when accompanied by other fields is confusing. What other fields? Users should not use $ref at all as it is deprecated and in 3.0 goes away -> #777

@derberg

Deprecated: Usage of the $ref property has been deprecated.
vs
Deprecated: Using the $ref field has been deprecated when accompanied by other fields.

We're removing the $ref field in Channel Item Object and replacing it with union type of Channel Item Object | Reference Object. That means that all the shapes where union type Channel Item Object | Reference Object is expected and the shape will contain $ref field will always be semantically recognized as Reference Object and not as Channel Item Object. Reference Object ignores all additional properties, instead of merging them with referenced structure. Hence the more precise deprecation message.
Saying it also in different way:
All the Channel Item Object shapes with single $ref field will automatically become Reference Objects in future version of the spec and will have exactly the same behavior of resolution.
All the Channel Item Object shapes with $ref field and additional fields will automatically become Reference Objects in future version of the spec and and  all the additional fields will be ignored instead of merging them with referenced structure.
More info in: https://github.com/asyncapi/spec/issues/607 and https://github.com/OAI/OpenAPI-Specification/pull/2656#issuecomment-885539956 (just imagine that Channel Item Object === Path Item Object)

@derberg did my explanation helped a bit?

thanks a lot for the explanation 🙏🏼
How about:
`Channel Item Object` with $ref field and additional properties will automatically become Reference Objects in future versions of the spec, and all the additional properties will be ??? instead of merging with referenced structure.

how about this way?
The most important part is ??? as you wrote ignore. This is a problem. For example we use https://github.com/APIDevTools/json-schema-ref-parser in the parser and it basically overrides what comes from a referenced object with what is in the original object. In other words, what is in the original object is preserved, not overwritten or ignored.
JSON reference says:
A JSON Reference is a JSON object, which contains a member named
   "$ref", which has a JSON string value.  Example:

   { "$ref": "http://example.com/example.json#/foo/bar" }

   If a JSON value does not have these characteristics, then it SHOULD
   NOT be interpreted as a JSON Reference.

my question is:

shouldn't we make sure that our json schemas are written in a way that if you have properties other than $ref, then it is invalid ref object,
wouldn't be better to just say that $ref used with other properties will fail validation

??

Channel Item Object with $ref field and additional properties will automatically become Reference Objects in future versions of the spec, and all the additional properties will be ??? instead of merging with referenced structure.

As I pointed in https://github.com/asyncapi/spec/pull/826#discussion_r951337047 (and other issues and comments) - the wording of the suggested change is IMHO not self-contained. The wording inside one version of the spec should probably not refer to the future version of the spec (as this version of the spec can be the last one) - https://github.com/OAI/OpenAPI-Specification/pull/2656#issuecomment-885539956

For example we use https://github.com/APIDevTools/json-schema-ref-parser in the parser and it basically overrides what comes from a referenced object with what is in the original object. In other words, what is in the original object is preserved, not overwritten or ignored.

This looks like an issue of the tooling. The specification is crystal clear that Reference Object additional properties SHALL be ignored. In OpenAPI 3.1 for example the Reference Object allows two additional fields - summary and description (other fields SHALL be ignored as well).

If a JSON value does not have these characteristics, then it SHOULD NOT be interpreted as a JSON Reference.

IMHO this is why the Reference Object in AsyncAPI spec provides clarification that additional fields  SHALL be ignored and as soon as there is $ref field the characteristics of JSON Reference spec as satisfied.

shouldn't we make sure that our json schemas are written in a way that if you have properties other than $ref, then it is invalid ref object,

IMHO no. Additional fields allows to annotate Reference Object with additional non-semantic information, that tooling can use in auxiliary ways - as long as auxiliary ways are comfort-ant with the spec.

wouldn't be better to just say that $ref used with other properties will fail validation

As per my answer above.

Using the $ref property together with other properties is deprecated since 2.3 version of the specification.

IMHO together word here can be understand in ambiguous way. Mentioning the version number, I guess I'm fine with it but what purpose does it serve? Here are variations of your suggestion after analyzing the grammar with grammarly.com.

Using the $ref property along with other properties is deprecated since the 2.3.0 version of the specification.

Using the $ref property with other properties is deprecated since the 2.3.0 version of the specification.

Used grammarly.com to analyze the wording.
I guess I still prefer Using $ref property has been deprecated when accompanied by other properties. the most.

@derberg I'd like to move this forward a bit. Any preference on one of the following sentences? It's basically your suggestion run via grammarly.com.

Using the $ref property along with other properties is deprecated since the 2.3.0 version of the specification.

Using the $ref property with other properties is deprecated since the 2.3.0 version of the specification.

I'm ok with Using $ref property has been deprecated when accompanied by other properties..
I like Using the $ref property with other properties is deprecated since the 2.3.0 version of the specification. the most but will not argue to include info about 2.3.
It's just that the reality is that there are people that do not read spec on website, or on github specific tag. They read raw spec on master, so it would be useful to have text that specify since what version was the property deprecated. It is also common in other kind of docs to explicitly specify since what version something was deprecated, otherwise you need to play with blame.
Have a look at below, why I think people read from master 👇🏼
stats for last 2 weeks for spec repo, taken from repo Insights traffic info

@derberg I'm fine with either of your sentences. @ponelat what do you think?

They read raw spec on master

As I do ;]

It is also common in other kind of docs to explicitly specify since what version something was deprecated, otherwise you need to play with blame.

Right, I'm not arguing that. What I said was: The wording inside one version of the spec should probably not refer to the future version of the spec. So my argument is that we shouldn't refer to the future versions of the spec (to keep it self-contained) as this might be the last version of the spec released.

Thanks @char0n @derberg I'm happy with the word changes, and I see adding the version as a good thing .

@derberg seems you need to resolve some more conflicts 🙂

@jonaslagoni thanks
@char0n please have a final look and approve

@char0n hey man, would be good to merge it before 3.0

/ready-to-merge


Add primary key directives to types

It is difficult to determine the primary key columns when looking at the graphql schema. Adding directives would give people additional insight on the structure of the underlying tables.
Proposed directives:
Primary Key
Clustering Key
Static
MaterializedView
@MaterializedView(baseTable: "Cart")
@PrimaryKey(fields: ["cartId"])
@ClusteringKey(fields: [{field: userName, direction: DESC}]
type Cart_MV {
  cartId: TimeUuid
  quantity: Int
  userId: Uuid @static
  userName: String
}

There's no way to add directives to fields and types in the schema introspection:
https://spec.graphql.org/June2018/#sec-Schema-Introspection
The only directive supported is @deprecated via isDeprecated.

@mpenick is there any way that we can communicate to the user which fields are part of the primary key? This is critical for them to be able to know what they can use in filters

I'm giving it a try but I'm having a hard time defining the directive at object / input object level in the schema, it's not getting recognized by the builder... I might be doing something wrong.

@jorgebay You can added it the the builder, but it won't make it into the fields schema (per my previous comment).

I see two option:  1) We add to the comments for the fields, but that doesn't show up in things like graphiql or graphql playground, 2) have a naming convention where we add it to the field names e.g. pk_title, ck_author.

I'm not sure I understand what you are saying. The spec allows declaring directives at TypeSystemDirectiveLocation (i.e. OBJECT or INPUT_OBJECT) https://spec.graphql.org/June2018/#sec-Type-System.Directives
and there are examples object level directives in the wild: https://www.graphql-tools.com/docs/schema-directives/

https://www.apollographql.com/docs/apollo-server/schema/creating-directives/#declaring-schema-directives

@jorgebay It'll never make it into the schema introspection json: https://spec.graphql.org/June2018/#sec-Schema-Introspection. There's no directives on fields in the schema introspection json spec. So you can add directives, but there's no way to associate those directives with fields that's useful the the client.

Wow, nice findings!
True, it doesn't look it would surface via introspection.
Weird that Apollo Federation project is using it (see @key).
I only see it at __InputValue level...


Add pre-commit hooks for better code quality control

Feature details?
Git hooks using frameworks like husky + lint-staged or pre-commit will ensure bad coding standards & practices aren't introduced to the source code. They'll cleaned & taken care of right at the source.
Further quality control can be ensured by implementing an automated CI/CD pipeline but for starters, Git hooks is the way to go.
Code of Conduct

[X] I agree to follow this project's Code of Conduct

I have added the things you've mentioned. Can I push it ?


check for PMID:UNKNOWN

Do not allow Phenote Records to be saved if there are malformed PMIDs.

also add a check for the id having at least four digits

fixed


Published under the Apache 2.0 license
Source: server — frontend-ui
Built using Python 3, Polymer 2.6.0, Icons by Google and MaterialDesignIcons.com.
Frontend JavaScript version: latest
-->
Last working Home Assistant release (if known):
Operating environment (Hass.io/Docker/Windows/etc.):

Component/platform:

Description of problem:
config.yaml:
device_tracker:

platform: netgear
host: 192.168.1.xxx
username: admin
password: xxxxxxxxxx
interval_seconds: 10
consider_home: 180
track_new_devices: no

platform: google_maps
username<EMAIL_ADDRESS>password: xxxxxxxxxxxxx

Traceback (if applicable):
Caught exception
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/locationsharinglib/locationsharinglib.py", line 283, in _get_data
    output = json.loads(response.text.split("'", 1)[1])
IndexError: list index out of range

Additional information:

Similar situation, my error is the following:
2018-04-18 16:02:03 ERROR (SyncWorker_11) [locationsharinglib.Service] Caught exception Traceback (most recent call last):   File "/srv/homeassistant/lib/python3.6/site-packages/locationsharinglib/locationsharinglib.py", line 284, in _get_data     people = [Person(info) for info in output[0]] TypeError: 'NoneType' object is not iterable
This was the first set up of the component, followed the instructions but it never worked.

I'm also getting the same error as @andyfsimon, I've followed all the steps other people have suggested including stopping HA and deleting the cookie files. Nothing seems to work :/

Did you approve login from activity page?
https://myaccount.google.com/device-activity

I did - but in my stupidity I concentrated myself on the error and didn't look in the "known_devices.yaml" file! Now I have a "google" entity there,it's just that it does not show up as it's not tracked!!!
I expected it to show it in the main page automatically, but clearly it's not the case! :)
Now it works for me.

I approve mine on the activity page, however no device with google entity is shown. I have 2 android shared with the email. I do not know what it can be anymore, does anyone have an idea?

I don't know how this will help anyone else but I added the component yesterday, nothing worked. I authorized the login on device activity, deleted the cookies files, restarted, etc, etc. nothing. This morning I checked the known_devices.yaml, it wasn't there. So I stopped HA, deleted the cookies file, started HA (same way I did it yesterday) and finally it shows up! track in known_devices.yaml was already set to true! Maybe some of things authorizations just take a bit of time to propagate google's servers? No clue.

still not able to try by myself... can someone share the attributes this device tracker provides? lat/long GPS? Address name?

Here's the attributes I see: source type, latitude, longitude, gps accuracy, id, nickname, full name, last seen, address, friendly_name, entity_picture

Mine is working now. It might have been working from the beginning, but here are the sources of confusion for me:

The errors in the log file related to location sharing library.
The documentation states that "This platform will create a file named google_maps_location_sharing.conf where it caches your login session." It does not. At least not for me. It only creates the hidden cookie file:  ".google_maps_location_sharing.cookies "
I never had to go and authorize anything on my new tracking account. In fact it only shows the two desktop computers that I have used to access the account. It does not list my Rpi running Hass.io as a device that has accessed the account. But it is working!

That is why I assumed it wasn't working. I rebooted my home assistant server a few times as I was adding and removing the other router based tracking platform that I use, so I don't know exactly when it started to work. Maybe it was working all the time.

Ugh, so it had been working. I restarted HA and now I'm getting this:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/homeassistant/components/device_tracker/init.py",
line 184, in async_setup_platform
disc_info)
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3.6/site-packages/homeassistant/components/device_tracker/google_maps.py",
line 36, in setup_scanner
scanner = GoogleMapsScanner(hass, config, see)
File "/usr/lib/python3.6/site-packages/homeassistant/components/device_tracker/google_maps.py",
line 55, in init
self._update_info()
File "/usr/lib/python3.6/site-packages/homeassistant/components/device_tracker/google_maps.py",
line 67, in _update_info
for person in self.service.get_all_people():
File "/usr/lib/python3.6/site-packages/locationsharinglib/locationsharinglib.py",
line 292, in get_all_people
return self._get_data()
File "/usr/lib/python3.6/site-packages/cachetools/init.py", line
46, in wrapper
v = func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/locationsharinglib/locationsharinglib.py",
line 283, in _get_data
output = json.loads(response.text.split("'", 1)[1])
File "/usr/lib/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
On Thu, Apr 19, 2018, 8:40 PM mnikser<EMAIL_ADDRESS>wrote:

Mine is working now. It might have been working from the beginning, but
here are the sources of confusion for me:

The errors in the log file related to location sharing library.
The documentation states that "This platform will create a file
named google_maps_location_sharing.conf where it caches your login
session." It does not. At least not for me. It only creates the hidden
cookie file: ".google_maps_location_sharing.cookies "
I never had to go and authorize anything on my new tracking
account. In fact it only shows the two desktop computers that I have used
to access the account. It does not list my Rpi running Hass.io as a device
that has accessed the account. But it is working!

That is why I assumed it wasn't working. I rebooted my home assistant
server a few times as I was adding and removing the other router based
tracking platform that I use, so I don't know exactly when it started to
work. Maybe it was working all the time.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/home-assistant/home-assistant/issues/13978#issuecomment-382925756,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALB0PcfpQUr4ALR_1XbYbAt9Es32wFfgks5tqS6UgaJpZM4TZ76j
.

Working now! Restarted HA this morning and everything works again.... I changed nothing else.


Checksum :client instead :client_id

Hi, I've tried to create a new paypal transaction and bind it to a client.
It seems like the the attribute is called client instead of client_id
Reference: https://developers.paymill.com/guides/paypal/transactions.html#optional-transaction-details
With this code change paypal associates the payment to the given client instead of creating a new one every time.

Hi @nerdyglasses
Currently the API works with both attributes (client and client_id). I will leave this PR open until we got a decision in the API which name will be used.

API calls, at least as tested from the PHP client did not work with client_id. Once it was changed to client, everything worked fine.
Please see:  https://github.com/paymill/paymill-php/pull/114#issuecomment-167107694

okay cool thx for looking into it. Hope you had nice holidays.


Update examples to latest(2.x) sdk

This PR starts to update the examples in examples to use the latest released cloudevents-sdk(which is 2.x atm)
fixes #187

looks like i can't add reviewers,  so @lance @helio-frota @grant

Converted to a draft until 2.0.2 lands

@lholmquist you will need to modify your commits with --signoff before we can merge this (a CNCF requirement).

I've updated to the 2.0.2 version now


add draft of registering function for pandas

This PR adds a register function for pandas. It allows to directly push DataFrame objects to the hub and in return loading datasets on the hub from DataFrame. The motivation for this integration is to enable the vast number of pandas users to be able to easily push DataFrames to the hub.
Here is an example:
import pandas as pd
from datasets import register_pandas

register_pandas()

# push to hub
df = pd.DataFrame.from_dict({"test": [1,2,3]})
df.push_to_hub("my_test")

# load from hub
df_retrieved = pd.DataFrame.load_from_hub("lvwerra/my_test")

It follows a similar philosophy as the tqdm integration. Also see this issue on the pandas repository.
This is just a rough draft of what such integration could look like but I would like appreciate some feedback on this: is this something you would like to add the library and is this the way to go? cc @lhoestq @albertvillanova @julien-c

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

That's cool ! Though I would expect such an integration to only require huggingface_hub, not the full datasets library.
Indeed if users want to use the datasets lib they could just to Dataset.from_pandas(df).push_to_hub() already. Therefore I would explore something that doesn't not necessarily requires datasets.
For other could storage solutions (S3, GCS, etc.), pandas allows users to pass URIs like s3://bucket-name/path/data.csv to the read_xxx and to_xxx (for csv, parquet, json, etc). It also support passing the root directory like s3://bucket-name/dataset-dir instead of a single file name.
In the Hugging Face Hub case, we have one dataset = one repository. We can enter pandas' paradigm by saying one dataset = one repository = one root directory. Here is what we could have:
push to Hub:
"""
Demo script for writing a pandas data frame to a CSV file on HF using fsspec-supported pandas APIs
"""
import pandas as pd

HF_USER = os.getenv("HF_USER")
HF_TOKEN = os.getenv("HF_TOKEN")

books_df = pd.DataFrame(
    data={"Title": ["Book I", "Book II", "Book III"], "Price": [56.6, 59.87, 74.54]},
    columns=["Title", "Price"],
)

dataset_name = "books1"

books_df.to_csv(
    f"hf://{HF_USER}/{dataset_name}",
    index=False,
    storage_options={
        "repo_type": "dataset",
        "token": HF_TOKEN,
    },
)

load from Hub:
"""
Demo script for reading a CSV file from HF into a pandas data frame using fsspec-supported pandas
APIs
"""
import pandas as pd

HF_USER = os.getenv("HF_USER")
HF_TOKEN = os.getenv("HF_TOKEN")

dataset_name = "books1"

books_df = pd.read_csv(
    f"hf://{HF_USER}/{dataset_name}",
    storage_options={
        "repo_type": "dataset",
        "token": HF_TOKEN,
    },
)

print(books_df)

And you could do the same with Parquet data using read/to_parquet or other formats. Formats like CSV, Parquet or JSON Lines would work out of the box with datasets. This API would also allow anyone to use Dask with the Hugging Face Hub for example.
What do you think ?

I'm closing this issue hffs can now be used for reading/writing data frames from/to the Hub.


Support UPSs handled by APC

As someone whose computer is connected via USB to an external power supply, I would love to be able to monitor its status from within battop. However it doesn't appear in the sysfs and as such is invisible to this library. Being able to support more types of batteries and power supplies should be a worthy goal for this type of library, but I don't know how much effort it would take to implement.

Hey, @boomshroom!
I seriously doubt that this is the good idea, sorry. battery is nothing more than an abstraction over OS interfaces, and if target platform does not have the driver for your power supply, this crate is useless in that case.
Also, I do not want to commit myself to support all these various and rare devices for all supported platforms; that would be a hell of a burden.

I do have a driver in the form of apcupsd. I can access the statistics through apcaccess status. Unfortunately it doesn't seem to come with a library that can be linked to and needs the output to be parsed in order to be used, nor does it populate /sys/class/power_supply.

Oh, okay, I've probably misunderstood you.
As far as I see from manual, the only options to get data from the apcupsd without re-implementing it fully, is to run shell command apcaccess status or try to make the HTTP request to a daemon CGI interface (if it is even enabled).
I'm going to re-open that issue, but with a low priority, because ADP information is not supported at all right now and it should be done first.


Lookup for email in username field to match docs if email is undefined

Fixes #2502
When the email is not set on the user, the docs expressively says that the server should lookup in the username field.
This PR propose that change.
Note that the adapters should adapt to that change if implementing sendPasswordResetEmail, and lookup in the username field is email is not set.

@flovilmart updated the pull request - view changes

@flovilmart updated the pull request - view changes


Very is SITE in version 2.0, I need my SITE!!!

I am not sure who has decided to remove site concept and add the location type, location, parent idea. it is really complex and making any import of data from CSV is very complex and unusable!!!!!
just compare the CSV Header from version 1.x:
name,status,location_type,tenant,facility,asn,time_zone,description,physical_address
with version2.0:
name,display,id,object_type,natural_slug,tree_depth,time_zone,circuit_count,device_count,prefix_count,rack_count,virtual_machine_count,vlan_count,description,facility,asn,physical_address,shipping_address,latitude,longitude,contact_name,contact_phone,contact_email,comments,parent__name,parent__parent__name,parent__parent__parent__name,location_type__name,status__name,tenant__name,created,last_updated,tags
you can see Millions of parent!!!!
Please rollback the site concept

edit

Hi @saman-com, thanks for submitting the issue. We understand this is a drastic change that can take some time to adjust to. However, most of the csv column headers you pointed out are optional and you do not have to specify all the parents of the location. You just need to fill out the required columns.


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-target_a60c995a-9ff0-4715-af2a-8ef5e20ab905/pull/1 is merged into live branch.
Auto archive content to https://github.com/opstest2/docs-archive-test-target_a60c995a-9ff0-4715-af2a-8ef5e20ab905.git

Docs Build status updates of commit eaeaf99:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error-RuningBuildFailed] Some unexpected errors happened when running build, please open a ticket in https://aka.ms/SiteHelp and include the error report for our team to troubleshoot

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Generates a javascriptKey if not passed for CloudCode

Proposed fix for #651 #356

I'm not sure why this fixes it?  It looks like they are issues with not providing a serverURL...

Test that here: https://github.com/flovilmart/parse-server-testing

I can confirm that this fixes the issue.

Another related issue #654


Bazel: Support local_path type in package metadata

Please have a look at the individual commit messages for the details.

And please rebase onto latest main for the Python test fix.


4.0.0-alpha.0 crashes on node 0.10

blessc crashes on node 0.10 with error:
./.bin/blessc

fs-promise/node_modules/any-promise/any-promise.js:33
  throw new Error('Must install one of: '+libs.join());
        ^
Error: Must install one of: es6-promise,promise,native-promise-only,bluebird,rsvp,when,q

It will work if there is some of the listed libraries is installed in node scope.

PR #74 should fix this.

Looks like our test runner was actually failing those tests but the error wasn't getting surfaced correctly fooling the CI into thinking that it passed.
Note the 0 exit code from Travis though you can see red :cry:

I was hoping this would be a quick fix, but that does not appear to be the case. Thankfully the code coverage changes we made will surface this more quickly in the future.

Following the update of 4.0.0 to release, this is now occuring for us. The error occurs when running any bless function. E.g. blessc -v

When we roll back to 3.0.3 the error is no longer present.

I'm thinking Bless 4 will need to be on Node 4.0 and later. See #96.


Slight typo in code sample

Existing code sample
import requests
# Load image data
data = open('example.jpg', 'rb').read()
# Post raw data to scoring URI
res = request.post(url='<scoring-uri>', data=data, headers={'Content-Type': 'application/> octet-stream'})

Last line should be:
res = requests.post(url='<scoring-uri>', data=data, headers={'Content-Type': 'application/> octet-stream'})

Users copy/pasting will encounter :
NameError: name 'request' is not defined
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-11-103c3870e95b> in <module>
      3 data = open('someimage.jpg', 'rb').read()
      4 # Post raw data to scoring URI
----> 5 res = request.post(url='http://someuri', data=data, headers={'Content-Type': 'application/> octet-stream'})

NameError: name 'request' is not defined

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d9415d8d-a30c-42f9-989c-c7b3a79d7108
Version Independent ID: b7b8ae97-1801-42b1-431e-c96ca5e958f7
Content: Create client for model deployed as web service - Azure Machine Learning
Content Source: articles/machine-learning/how-to-consume-web-service.md
Service: machine-learning
Sub-service: core
GitHub Login: @aashishb
Microsoft Alias: aashishb

@krisbock Thanks for the question. We are investigating the issue and will update you shortly.

Thanks @krisbock . Raised PR #https://github.com/MicrosoftDocs/azure-docs-pr/pull/171468 to fix the typo.

@krisbock  We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.


soroban-cli/rpc: future releasing, semver, api stability

This issue is some thoughts I shared internally on slack yesterday, but should have posted in a public space for discussion:
v20 is both the protocol the CLI and RPC support, as well as its major version, so from here we should avoid breaking the interfaces (command line / API) to both the RPC and CLI.
Two things comes to mind:

There's a few PRs open that are usability improvements in the CLI, but they break the interfaces. Such as moving the config subcommands. We should move ahead with them, but make the changes backwards compatible so that someone who has a script running a specific subcommand will continue to work even if the subcommand has moved?
In 'clap' it's relatively easy to do this by keeping an old command that calls the new command, and then marking that old command as hidden or at least adding the word "deprecated" to its help text.
We can then plan to remove all deprecated commands in the v21 release.

We currently test each CLI release with only the latest RPC. That's not really enough. The CLI is a client of the RPC and I think a v20 CLI should work with any v20 RPC. It won't always be easy for folks to have multiple CLIs installed if they interact with multiple RPCs which could be different versions. It's also a poor user experience to force people to upgrade CLIs for every minor or patch release of the RPC. This problem isn't unique to the CLI/RPC, as there is also the js-stellar-sdk and other clients, and it won't be realistic to coordinate updates and upgrades to all synchronously.
We can maintain a compatibility guarantee for both tools by:

Don't add new required params to existing RPC endpoints that would cause existing CLI requests to fail, or requests from any client (js-stellar-sdk, or custom). (Until https://github.com/stellar/soroban-tools/issues/1145 is resolved don't add any new params.)

Keep the CLI tolerant of new RPC response fields being missing. If we add a new response field to the RPC that CLI will consume, the CLI should assume its optional. This principle can be applied to any client in development, so the js-stellar-sdk, or typescript bindings.

Test each CLI release with a range of RPC releases. Probably need to do same with other clients too.

Invest in some static API semver checks so that (iii) is less necessary.

Split up the CLI and RPC repos, so that we don't develop them in such a coupled manner, given it is just one of many clients of the RPC and the CLI needs to work with a range of RPC versions, not just the latest. Related discussion here: https://github.com/stellar/soroban-tools/issues/1115

Breaking changes in the CLI: imo it's the best practice for a tool to be backward compatible when possible for a least sometime to give developers the time to adopt a new command. I agree the idea of keeping the old commands which call the new commands. For breaking changes in tooling it could be good to also announce these changes on Community Dev Calls in addition to release notes.
An idea is the CLI could have a preferred version of RPC that's defined in the release. If the CLI is configured to a different RPC version that the dev needs to set a config param to --allow-mismatch so the dev is aware that she may run into issues. Allowing CLI to maintain compatibility guarantees to multiple versions of RPC also makes sense but unclear how much lift that adds to dev and testing. Devs ideally should be working with the latest stable release of RPC or an upcoming release if they want to test the newest features.

2. An idea is the CLI could have a preferred version of RPC that's defined in the release. If the CLI is configured to a different RPC version that the dev needs to set a config param to --allow-mismatch so the dev is aware that she may run into issues.

I think this shifts the responsibility too much onto the user. Other clients, such as the js-stellar-sdk, will have to work across multiple releases. It will be impractical to ask developers of applications to upgrade, or downgrade, to specific versions of the js-stellar-sdk before connecting to different RPC servers. If other clients need to be capable, I think the CLI can be too. The principles above are also not unique or extravagant. The development of any client/server software typically requires strategies for ensuring compatibility is maintained over time and there are standard patterns.

On other thing we need to ensure when adding optional params to the RPC, is that they always need to be appended to the list of params, not inserted earlier in the list. This assumes we continue to support array style param requests (https://github.com/stellar/soroban-tools/issues/1145).

Blocked due to recent release issues. Will work on this again by Platform

@tupui Thank you for sharing your learnings and insights. I agree, there needs to be an explicit guarantee / compatibility statement, and I like the idea of being explicit with deprecations about when they will eventuate into actual removal, and how to provide feedback on them as well. We do have some commands in the CLI marked as deprecated, but no other plans beyond that.
This is something I'm working on. ❤


cherry-pick(0.36.x): fix: ensure alias version isn't dropped round-trip (#6777)

Fixes #N/A
Description
Cherry picks #6777
How was this change tested?
Does this change impact docs?

[ ] Yes, PR includes docs updates
[ ] Yes, issue opened: #
[x] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>1
Details

15 of 15 (100.0%)  changed or added relevant lines in 1 file are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.08%) to 77.805%

Files with Coverage Reduction
New Missed Lines
%

pkg/providers/amifamily/ami.go
1
90.37%

Totals

Change from base Build<PHONE_NUMBER>1:
0.08%

Covered Lines:
5651

Relevant Lines:
7263

💛  - Coveralls


Attempt at impact intro

we should have a prose into to impact, so I had a go. Feel free to edit/suggest

I know this is not something you have changed but in "Summary" (for impact) we have:
This is the case in the PlasmaBio project, a Jupyter-based platform at Université de Paris for education in genomics.
I don't think we ever talk about it anywhere. Is it relevant?

Good point on the Summary - there shouldn't be anything only there, it should all be a summary of what's in the rest of section 2.


The move history bar is scrolling slowly instead of immediately

Why

Unfortunately only on iOS Safari, not on desktop

It got broken after the Refactor commit

Can also repro in Chrome on desktop

Turns out I was rerendering react all the time


Remove Caliptra profiles for 1.0

Fixes #66
Caliptra subsystem becomes a future effort.

@andreslagarcavilla, can we merge this and call the doc 1.0? Probably need one more PR to update version numbers?


uWSGI workers dying in 1.40.0

How do you use Sentry?
Sentry Saas (sentry.io)
Version
1.40.0
Steps to Reproduce
I don't have a reproduction yet due to the nature of the issue, for which I apologize, but with 1.40.0 my server processes are silently dying after about 90 minutes of run time. This does not happen with 1.39.2. I just want to get it on your radar that the latest update may be causing issues for some. I wish I had more information, but I don't at this time.
Expected Result
^
Actual Result
^

I also had a similar issue with 1.40, it was killing my uwsgi workers, had to downgrade to 1.39. Also wish I had more information to give but there wasn't much to go on. The issue was worse with profiling on (workers would be killed via SIGBUS)

@fiendish @seb-b Hey folks, thanks for bringing this to our attention. Even if there is no repro, more info could help point us in the right direction -- what kind of apps (Django, Flask, FastAPI, something else?) are you running? Which SDK integrations are active?
Looking at the changelog from 1.40.0 there's a couple things we could try:

turning off metrics collection:sentry_init(
    ... # your usual stuff
    _experiments={
        "enable_metrics": False,
    }
)

turning off DB query source capture:sentry_init(
    ... # your usual stuff
    enable_db_query_source=False,
)

Oops, yeah, sorry. In my haste I forgot to include. In my case it's Flask 2.1.3 + uWSGI 2.0.23, and
integrations = [
    FlaskIntegration(),
    LoggingIntegration(
        level=logging.INFO,
        event_level=logging.ERROR,
    )
]

I'll try to find time over the weekend or monday to try those things.

Same here with Django 3.2 LTS, uWSGI 2.0.23. This are my settings:
sentry_logging = LoggingIntegration(
    level=logging.INFO,  # Capture info and above as breadcrumbs
    event_level=logging.ERROR,  # Send errors as events
)
sentry_settings = {
    "dsn": SENTRY_LIVE_DSN,
    "integrations": [DjangoIntegration(), sentry_logging],
    "send_default_pii": True,
    "enable_tracing": True,
}

Yes, I'd like to try this. But I'll have to wait until it is a bit more calm here in the evening (in ~6hrs or so).

We hit the same issues to with  Python (version 3.11) Django (version 4.1.10) app running on uWSGI (version 2.0.23) and sentry-sdk 1.40.0
In our case, we have the following setting to recycle uwsgi workers every 1 hr
max-worker-lifetime= 3600 #seconds

After 1 hr of worker lifetime, we see that the uWSGI master complains that the listen queue is full. Log snippet attached below. The workers never get respawned and we eventually loose the kubernetes pod.
Fri Feb  2 17:53:20 2024 - worker 1 lifetime reached, it was running for 21 second(s)
Fri Feb  2 17:53:20 2024 - worker 2 lifetime reached, it was running for 21 second(s)
[WARNING] 032/175340 (10426) : Server uwsgi/uwsgi-7001 is DOWN, reason: Layer7 timeout, check duration: 6000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 032/175342 (10426) : Server uwsgi/uwsgi-7002 is DOWN, reason: Layer7 timeout, check duration: 6000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 032/175342 (10426) : Server uwsgi/uwsgi-7000 is DOWN, reason: Layer7 timeout, check duration: 6001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 032/175342 (10426) : backend 'uwsgi' has no server available!
 I could see the same behavior as logs above.
Additionally I noticed that the worker processes never went away when I did ps aux. Which makes me think that, supervisord which we use to restart the processes, thought the processes never died.
I also confirmed that, this issue does not occur when we kill by _raise_sigint() based on PSS memory
or when workers get reloaded using reload-on-rss parameter.
I have noticed this to happen only when  max-worker-lifetime value kicks-in.

Respawned uWSGI worker 1 (new pid: 150)

Disabling metrics fixed it.
Startup logs:
*** Starting uWSGI 2.0.23 (64bit) on [Sat Feb  3 19:00:44 2024] ***
compiled with version: 10.2.1 20210110 on 03 February 2024 18:58:51
os: Linux-4.18.0-477.27.1.el8_8.x86_64 #1 SMP Thu Aug 31 10:29:22 EDT 2023
nodename: [redacted]
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /app
detected binary path: /usr/local/bin/uwsgi
chdir() to /app/
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /var/opt/nomad/run/divvy-onboarding-ux-test-d31774b4-7f2e-62a7-356a-13ab41d8f3b1.sock fd 3
uwsgi socket 1 bound to TCP address <IP_ADDRESS>:20324 fd 4
Python version: 3.12.1 (main, Feb  1 2024, 04:52:14) [GCC 10.2.1 20210110]
Python main interpreter initialized at 0x7fb214c30028
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 385000 bytes (375 KB) for 4 cores
*** Operational MODE: preforking ***
 [sentry] DEBUG: Setting up integrations (with default = True)
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.fastapi.FastApiIntegration: Starlette is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.httpx.HttpxIntegration: httpx is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.pyramid.PyramidIntegration: Pyramid not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.starlette.StarletteIntegration: Starlette is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
 [sentry] DEBUG: Setting up previously not enabled integration flask
 [sentry] DEBUG: Setting up previously not enabled integration pure_eval
 [sentry] DEBUG: Setting up previously not enabled integration argv
 [sentry] DEBUG: Setting up previously not enabled integration atexit
 [sentry] DEBUG: Setting up previously not enabled integration dedupe
 [sentry] DEBUG: Setting up previously not enabled integration excepthook
 [sentry] DEBUG: Setting up previously not enabled integration logging
 [sentry] DEBUG: Setting up previously not enabled integration modules
 [sentry] DEBUG: Setting up previously not enabled integration stdlib
 [sentry] DEBUG: Setting up previously not enabled integration threading
 [sentry] DEBUG: Setting up previously not enabled integration redis
 [sentry] DEBUG: Did not enable default integration redis: Redis client not installed
 [sentry] DEBUG: Enabling integration flask
 [sentry] DEBUG: Enabling integration pure_eval
 [sentry] DEBUG: Enabling integration argv
 [sentry] DEBUG: Enabling integration atexit
 [sentry] DEBUG: Enabling integration dedupe
 [sentry] DEBUG: Enabling integration excepthook
 [sentry] DEBUG: Enabling integration logging
 [sentry] DEBUG: Enabling integration modules
 [sentry] DEBUG: Enabling integration stdlib
 [sentry] DEBUG: Enabling integration threading
 [sentry] DEBUG: Setting SDK name to 'sentry.python.flask'
 [sentry] DEBUG: [Profiling] Setting up profiler in thread mode
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x7fb214c30028 pid: 1 (default app)
spawned uWSGI master process (pid: 1)
spawned uWSGI worker 1 (pid: 6, cores: 1)
spawned uWSGI worker 2 (pid: 7, cores: 1)
spawned uWSGI worker 3 (pid: 8, cores: 1)
spawned uWSGI worker 4 (pid: 9, cores: 1)
 [sentry] DEBUG: [Tracing] Discarding <http.server> transaction <generic WSGI request> because traces_sampler returned 0 or False
 [sentry] DEBUG: [Profiling] Discarding profile because transaction is discarded.
 [sentry] DEBUG: Discarding transaction because sampled = False
[pid: 6|app: 0|req: 1/1] [redacted] () {28 vars in 388 bytes} [Sat Feb  3 19:00:47 2024] GET /ping => generated 21 bytes in 9 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)```

Repo is here: https://github.com/RoboJackets/divvy-onboarding-ux

The same problem with sentry-sdk 1.40 (Debian 11, python 3.9.2, django 4.2.9, uwsgi 2.0.23)
uWSGI workers hang after lifetime period.

The metrics code “just” spawns a thread. I don’t think we spawn this differently than our other background threads but that’s worth checking.
uwsgi is kinda crashy and I have seen bad failures in the past with threads. The metrics code other than the flusher thread should be dead unless someone calls into the metrics functions which is not happening by default.
One difference I think is that we do not sleep but wait on a condvar. Maybe we can repro with a dummy uwsgi and wait until it crashes, then inspect the core dump.

Folks, we're releasing SDK version 1.40.1 in a couple moments that includes a fix for the uWSGI workers hanging issue. Let us know if upgrading fixes the issue.

1.40.1 did not fix the issue on my end. I'm still struggling to find time to explore the code.

Thanks @fiendish, please keep us posted.
I see three possibly separate issues in this thread:

Workers hanging after their max lifetime. I was able to repro this and the fix we shipped in 1.40.1 should fix this. If this is not the case for someone with workers hanging after max lifetime, please let me know.
Workers dying after some time. Still investigating this, any extra info is welcome.
Workers segfaulting. Goes away when metrics are turned off. @kberzinch do you get the segfault consistently? Does it occur at startup or after some time?

The one good thing is that we can rule out this having anything to do with the other potential culprit (query source) since the app you linked @kberzinch is as far as I can tell not using a DB at all. So I'm pretty sure this is metrics, maybe in tandem with profiling since there was a report that profiling makes the issue worse.
Can any of you folks that are running into the workers dying share your uwsgi config? (I've already seen the one in the repo you linked @kberzinch, have been trying to use that to repro, no luck so far.) Maybe we can find some common denominator there that would help us reproduce this.

The worker segfaults immediately after requesting an endpoint that uses set_user (then nginx returns a 502 to the client for that request). That happens consistently: sending the same request again will kill the worker again. Other endpoints seem to work OK.
You are correct, that app does not use a DB, it is literally a glorified webform to send me an email 😄
I can try to fiddle around and get a minimum reproducible example, but probably not for a few days, sorry.

So I'm pretty sure this is metrics, maybe in tandem with profiling since there was a report that profiling makes the issue worse.

Indeed for us just toggling the metrics back off seems to fix the breakage.

Thanks for verifying and for the config @fiendish!
@kberzinch I'm obviously running a rather stripped down version of your app since I don't have all the necessary env vars but still can't repro. Will continue trying though.
We're continuing to look into this -- in the meantime I've also opened a uwsgi issue to possibly get some insight there.

@fiendish does adding enable-threads=true make a difference?

We just spent several engineering hours investigating why our service was crashing, and traced it to sentry-sdk 1.40.2.
Please yank the versions, this is a critical bug.

This issue just bit our services as well.  Python (version 3.11.7) Django (version 5.0.1) app running on uWSGI (version 2.0.23) using sentry-sdk (version 1.40.2). Spent a couple hours trying to figure out why our app was instantly crashing.

Unfortunately uwsgi has long standing threading bugs and crashes and sometimes in the metrics flusher is triggering one of those bugs. It’s almost equivalent to the flusher we use for years. The only difference I can see is the event. However it also relies on the GIL functioning heavily and I wonder if uwsgi does not hold the GIL properly.
Since uwsgi is no longer fixing such crashers I’m not sure what a good cause of action here is. For now we can just turn it off. Ultimately the issue lies with uwsgi :(

Since uwsgi is no longer fixing such crashers

Can you be more specific if you know something that is not commonly known?
https://uwsgi-docs.readthedocs.io/en/latest/ says "The project is in maintenance mode (only bugfixes [...])"

We reported quite a few segfaults over the years. Lack of test suite and invasiveness of some of these fixes impedes resolution.

https://github.com/unbit/uwsgi/issues/1950
https://github.com/unbit/uwsgi/issues/1651
https://github.com/unbit/uwsgi/issues?q=is%3Aissue+is%3Aopen+segfault+sort%3Acomments-desc

Folks, we're rolling out 1.40.3 which disables metrics for uWSGI for now.

Hi all,
I know I'm a bit late to the party, but it seems that --lazy is a "workaround" (on this particular Flask 3.0.2 app).
uwsgi  --processes 3 --master  -w index:app --need-app  --env mount=/api  --http-socket <IP_ADDRESS>:8080    --env SENTRY_DSN=https://xxx  --enable-threads  --die-on-term  --reload-mercy 1    --threads 1 --worker-reload-mercy 1

reliably produces the segfault on 2.0.23, 2.0.22, 2.0.21, and with py3.11.6 I cannot build older ones.
with a WSGI stub function I can't reproduce it (even if everything else - sentry, pymongo, and a bunch of other stuff - is loaded) .. but with a minimal Flask app it can be reproduced.
(a.py)
import sentry_sdk

sentry_sdk.init(
        dsn='insert valid sentry dsn',
        # Set traces_sample_rate to 1.0 to capture 100%
        # of transactions for performance monitoring.
        traces_sample_rate=1.0,
        # Set profiles_sample_rate to 1.0 to profile 100%
        # of sampled transactions.
        # We recommend adjusting this value in production.
        profiles_sample_rate=1.0,
        environment='local-dev',
        debug=True
    )

from flask import Flask

flask_app = Flask(__name__)

@flask_app.route('/')
def index():
    return 'k'

using
uwsgi  -w a:flask_app --need-app  --http-socket <IP_ADDRESS>:8080    -p 4

with -p 1 it does not occur, nor when the request gets assigned to the first worker. (and this kind of explains why lazy helps) probably the issue is that something gets initialized before forking (you don't say!? :))
*** Starting uWSGI 2.0.23 (64bit) on [Sat Feb 10 19:57:45 2024] ***
compiled with version: 13.2.0 on 06 February 2024 18:47:26
os: Linux-6.5.0-15-generic #15-Ubuntu SMP PREEMPT_DYNAMIC Tue Jan  9 22:39:36 UTC 2024
nodename: lima-default
machine: aarch64
clock source: unix
detected number of CPU cores: 4
current working directory: /opt/lima/xxx/xxx/wsgi-hack
detected binary path: /opt/lima/xxx/backend-venv/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 31425
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address <IP_ADDRESS>:8080 fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.11.6 (main, Oct  8 2023, 05:06:43) [GCC 13.2.0]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0xffff8d5207b8
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 291616 bytes (284 KB) for 4 cores
*** Operational MODE: preforking ***
/opt/lima/xxx/backend-venv/lib/python3.11/site-packages/sentry_sdk/_compat.py:161: Warning: We detected the use of uwsgi with disabled threads.  This will cause issues with the transport you are trying to use.  Please enable threading for uwsgi.  (Add the "enable-threads" flag).
  warn(
 [sentry] DEBUG: Setting up integrations (with default = True)
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.aiohttp.AioHttpIntegration: AIOHTTP not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.fastapi.FastApiIntegration: Starlette is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.httpx.HttpxIntegration: httpx is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.pyramid.PyramidIntegration: Pyramid not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.rq.RqIntegration: RQ not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sqlalchemy.SqlalchemyIntegration: SQLAlchemy not installed.
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.starlette.StarletteIntegration: Starlette is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
 [sentry] DEBUG: Setting up previously not enabled integration argv
 [sentry] DEBUG: Setting up previously not enabled integration atexit
 [sentry] DEBUG: Setting up previously not enabled integration dedupe
 [sentry] DEBUG: Setting up previously not enabled integration excepthook
 [sentry] DEBUG: Setting up previously not enabled integration logging
 [sentry] DEBUG: Setting up previously not enabled integration modules
 [sentry] DEBUG: Setting up previously not enabled integration stdlib
 [sentry] DEBUG: Setting up previously not enabled integration threading
 [sentry] DEBUG: Setting up previously not enabled integration flask
 [sentry] DEBUG: Setting up previously not enabled integration redis
 [sentry] DEBUG: Did not enable default integration redis: Redis client not installed
 [sentry] DEBUG: Enabling integration argv
 [sentry] DEBUG: Enabling integration atexit
 [sentry] DEBUG: Enabling integration dedupe
 [sentry] DEBUG: Enabling integration excepthook
 [sentry] DEBUG: Enabling integration logging
 [sentry] DEBUG: Enabling integration modules
 [sentry] DEBUG: Enabling integration stdlib
 [sentry] DEBUG: Enabling integration threading
 [sentry] DEBUG: Enabling integration flask
 [sentry] DEBUG: Setting SDK name to 'sentry.python.flask'
 [sentry] DEBUG: [Profiling] Setting up profiler in thread mode
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xffff8d5207b8 pid: 826678 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (pid: 826678, cores: 1)
spawned uWSGI worker 2 (pid: 826681, cores: 1)
spawned uWSGI worker 3 (pid: 826682, cores: 1)
spawned uWSGI worker 4 (pid: 826683, cores: 1)

 [sentry] DEBUG: [Tracing] Starting <http.server> transaction <generic WSGI request>
 [sentry] DEBUG: [Profiling] Initializing profile
 [sentry] DEBUG: [Profiling] Starting profile
 [sentry] DEBUG: [Profiling] Stopping profile
 [sentry] DEBUG: [Profiling] Discarding profile because insufficient samples.
!!! uWSGI process 826683 got Segmentation Fault !!!
*** backtrace of 826683 ***
uwsgi(uwsgi_backtrace+0x30) [0xaaaaab7d6a30]
uwsgi(uwsgi_segfault+0x34) [0xaaaaab7d6e14]
linux-vdso.so.1(__kernel_rt_sigreturn+0) [0xffff8d6e38f8]
/usr/lib/aarch64-linux-gnu/libpython3.11.so.1.0(_PyThread_CurrentFrames+0x94) [0xffff8d06bc54]
/usr/lib/aarch64-linux-gnu/libpython3.11.so.1.0(+0x1c4ac4) [0xffff8cf84ac4]
/usr/lib/aarch64-linux-gnu/libpython3.11.so.1.0(PyObject_Vectorcall+0x4c) [0xffff8cf35d4c]
/usr/lib/aarch64-linux-gnu/libpython3.11.so.1.0(_PyEval_EvalFrameDefault+0x38d4) [0xffff8cec9e14]
/usr/lib/aarch64-linux-gnu/libpython3.11.so.1.0(+0x262970) [0xffff8d022970]
/usr/lib/aarch64-linux-gnu/libpython3.11.so.1.0(+0x177ffc) [0xffff8cf37ffc]
/usr/lib/aarch64-linux-gnu/libpython3.11.so.1.0(_PyEval_EvalFrameDefault+0x79f4) [0xffff8cecdf34]
/usr/lib/aarch64-linux-gnu/libpython3.11.so.1.0(+0x262970) [0xffff8d022970]
/usr/lib/aarch64-linux-gnu/libpython3.11.so.1.0(_PyEval_EvalFrameDefault+0x79f4) [0xffff8cecdf34]
/usr/lib/aarch64-linux-gnu/libpython3.11.so.1.0(+0x262970) [0xffff8d022970]
/usr/lib/aarch64-linux-gnu/libpython3.11.so.1.0(+0x177ffc) [0xffff8cf37ffc]
/usr/lib/aarch64-linux-gnu/libpython3.11.so.1.0(+0x399194) [0xffff8d159194]
/usr/lib/aarch64-linux-gnu/libpython3.11.so.1.0(+0x2ba0e0) [0xffff8d07a0e0]
/usr/lib/aarch64-linux-gnu/libc.so.6(+0x837d0) [0xffff8cc837d0]
/usr/lib/aarch64-linux-gnu/libc.so.6(+0xef5cc) [0xffff8ccef5cc]
*** end of backtrace ***

(--enable-threads doesn't make any difference)

@PAStheLoD thanks for spending time on this. While you are at it, does lazy-apps suffice or do you really need lazy?

Thanks @PAStheLoD, this is very helpful. I'd also be interested in whether --lazy-apps has the same effect on your app or if you need --lazy.
My current thoughts on this are that there has to be a way to make this work since the transport worker and profiler work largely the same way and we're not seeing them explode people's apps. One difference (other than the use of threading.Event) that I see in the metrics code vs. other places where we have background threads is that in metrics we start the thread right off the bat, whereas elsewhere the thread is only started on demand. I can imagine this making a difference in prefork mode. I'll align this.
I'm also wondering if upgrading to the recent uWSGI 2.0.24 might make a difference?

We were also hit by uWSGI workers getting stuck, presumably due to changes in sentry-sdk - although we did try to revert upgrades without much luck. In the end we ended up migrating from uWSGI to Unit and it's been rock solid for us. Just in case anybody else also get frustrated with uWSGI...

Everyone, we've just released 1.40.4 with a tentative fix.
It'd greatly help us out if you could try it out if you're affected by this issue and report back. What you need to do is install sentry-sdk 1.40.4, add the following to your init, and run with your usual uWSGI config.
sentry_sdk.init(
    ... # your usual stuff
    # Metrics collection is still off by default in 1.40.4 if you're under uWSGI,
    # but this can be overridden by
    _experiments={
        "force_enable_metrics": True,
    }
)

Caveat: The fix in 1.40.4 should work as long as you're not manually emitting any metrics (with sentry_sdk.metrics.<metric_type>(...) on startup (on demand, e.g. on request, is fine). In that case the only known fix so far is to run uWSGI with --lazy-apps to disable preforking mode. We're looking into addressing this as well without requiring --lazy-app.
Also please note that for the SDK to not break in other unexpected ways uWSGI needs to be run with --enable-threads.

I can reproduce the issue.
uwsgi invocation with a.py from @PAStheLoD:
uwsgi -w a:flask_app --need-app --http-socket <IP_ADDRESS>:8080 -p 4 --enable-threads

The segfault is caused by uwsgi not handling threads correctly with default arguments.
When forking python with multiple threads running the fork hooks PyOS_BeforeFork, PyOS_AfterFork_Child and PyOS_AfterFork_Parent have to be run before/after fork. A correct fork sequence looks something like this. uwsgi runs none of those by default. It seems like this is by design, as uwsgi assumes that wsgi apps are single threaded (source).
To make the segfault disappear uwsgi has to be called with --enable-threads (initialize the GIL) and --py-call-uwsgi-fork-hooks (run PyOS_AfterFork_Child in the worker after fork).
uwsgi -w a:flask_app --need-app --http-socket <IP_ADDRESS>:8080 -p 4 --enable-threads --py-call-uwsgi-fork-hooks

Interestingly this behaviour is still not fully correct: Only PyOS_AfterForkChild is called. PyOS_BeforeFork and PyOS_AfterFork_Parent are not called. I'm not sure if that can lead to segfaults too, but in any case not all hooks registered by os.register_at_fork are called when the worker is forked. (The before and after_in_parent hooks are not called.)
Not sure if there is any viable workaround for the sentry SDK, as the corruption happens outside of its control.
IMHO this is not a sentry issue, but caused by unsafe defaults in uwsgi. I wish uwsgi would pick safer defaults.

Used software versions:
Linux 6.1.71-1-lts (arch linux kernel build)
Python 3.11.8 (arch linux PKGBUILD, but rebuilt with debug symbols)
uwsgi 2.0.23 and 2.0.24
sentry_sdk 1.40.2

Thanks a lot for looking into this @natano!
I think the best we can do on the SDK side moving forward is to issue a warning on startup if we detect we're in non-lazy mode and --py-call-uwsgi-fork-hooks is not on, just like we do currently if you're running uWSGI without --enable-threads.
The fix in 1.40.4 should work even without --py-call-uwsgi-fork-hooks in most cases because with that release we spawn all threads on demand BUT if you're unlucky and a thread is spawned at startup before forking, you might run into this again.
So to sum up for future reference, if you run into this issue:

run uWSGI with both --enable-threads and --py-call-uwsgi-fork-hooks
OR run uWSGI with --lazy-apps (or --lazy, but this is discouraged)

To add some extra seasoning to this issue.
We saw failures in our AWS Lambda executors after moving to 1.40.3:
Traceback (most recent call last):
  < ... omitted ... >
  File "/var/task/sentry_sdk/integrations/threading.py", line 56, in sentry_start
    return old_start(self, *a, **kw)
  File "/var/lang/lib/python3.9/threading.py", line 899, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

@thedanfields I'd wager a guess that's a AWS specific issue - reminds me of https://github.com/getsentry/sentry-python/issues/2632 without the upgrading to Python 3.12 part. I wonder if we've hit some kind of AWS Lambda limit on the number of threads since the SDK is now spawning one additional thread.
Could you open a new issue with this please?

@sentrivana , happy to oblige
I created a new issue as requested.


Removing extra github action. Not in use

Please always provide the GitHub issue(s) your PR is for, as well as test URLs where your change can be observed (before and after):
Fix #
Test URLs:

Before: https://main----.hlx.page/
After: https://----.hlx.page/

Update the description of the PR and add testURL for the PSI check


Add pwr.edu.pl

It's a domain belonging to the Wrocław University of Technology. Its other domain pwr.wroc.pl is already in the repository.
Should I add subdomains too or is just the topmost domain name sufficient?

@irth Pull request merged. No other actions needed, thank you!


Move watchdog kernel initialisation to server

Make the tornado server boot up the watchdog kernel.

This is as opposed to having the JavaScript call the watchdog service in start.

Don't do this. No need.


copy right footer is added in all pages

close issue -> #773

data is taken form your license a there mention 2022 so write as it is.

Good work!


Exception symbol not found in flat namespace '_SecItemExport' on Mac

I'm getting an exception trying to import pyroscope extension on mac
$ gem install pyroscope
$ irb
irb(main):001:0> require('pyroscope')
<internal:/usr/local/Cellar/ruby/3.1.1/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require': dlopen(/usr/local/lib/ruby/gems/3.1.0/gems/pyroscope-0.2.0/lib/pyroscope_c.bundle, 0x0009): symbol not found in flat namespace '_SecItemExport' - /usr/local/lib/ruby/gems/3.1.0/gems/pyroscope-0.2.0/lib/pyroscope_c.bundle (LoadError)
	from <internal:/usr/local/Cellar/ruby/3.1.1/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from /usr/local/lib/ruby/gems/3.1.0/gems/pyroscope-0.2.0/lib/pyroscope.rb:2:in `<top (required)>'
	from <internal:/usr/local/Cellar/ruby/3.1.1/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:160:in `require'
	from <internal:/usr/local/Cellar/ruby/3.1.1/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
	from <internal:/usr/local/Cellar/ruby/3.1.1/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
	from (irb):1:in `<main>'
	from /usr/local/Cellar/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/irb-1.4.1/exe/irb:11:in `<top (required)>'
	from /usr/local/opt/ruby/bin/irb:25:in `load'
	from /usr/local/opt/ruby/bin/irb:25:in `<main>'
<internal:/usr/local/Cellar/ruby/3.1.1/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- pyroscope (LoadError)
	from <internal:/usr/local/Cellar/ruby/3.1.1/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from (irb):1:in `<main>'
	from /usr/local/Cellar/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/irb-1.4.1/exe/irb:11:in `<top (required)>'
	from /usr/local/opt/ruby/bin/irb:25:in `load'
	from /usr/local/opt/ruby/bin/irb:25:in `<main>'

ruby -v
ruby 3.1.1p18 (2022-02-18 revision 53f5fc4236) [x86_64-darwin21]

System Software Overview:

  System Version:	macOS 12.2.1 (21D62)
  Kernel Version:	Darwin 21.3.0
  Boot Volume:	Macintosh HD
  Boot Mode:	Normal
  Computer Name:	MacBookPro
  Secure Virtual Memory:	Enabled
  System Integrity Protection:	Enabled
  Time since boot:	4 days 8:48

Okay, it's only happened on ruby version 3.1.1
It works okay on 3.0.4, 2.7.6 & 2.6.11


Fix storybook build

Storybook uses the 'analyze' webpack config, and some of the key-value data wasn't being pulled into the build. This submission fixes that issue so storybook will build correctly.

Thanks for the quick review!


memory-mapped DataFrame

There are several options here, from indexing CSV rows to allow random access to giant CSV files, to creating a new binary file format for storing DataVecs that supports mmap-style direct memory mapping.

I agree, we need a platform able to work transparently with data of any size, at least bigger than memory, and ideally distributed on several computers.
Solutions such as Spark are not complete, they only offer the basic functionalities to build something else.
We don't just need to be able to get some summaries, as we do with databases,  we need to be able to do all operations with big data, operations such as multiplying to big matrixes, fitting a mixed-effect models, MCMC, etc.
Bigmemory or ff allows you to do some simple things but they cannot be used by other packages like lme4.

I think this can be closed in light of the new Feather.jl package, which is essentially a a cross-language binary storage format for dataframes (R, pandas, etc.). It's fast and efficient because the files are mmapped directly and the each column is unsafe_wrap(Array) into NullableArrays=>DataFrame. Obviously, NullableArray support is not quite 100% with DataFrames, but should soon be.

Closing. Can be reopened in the context of Feather.jl.


When the stdlib is built with LTO and a custom toolchain, point -lto_library to the toolchain's lib directory when running lit tests

Let's see how much you hate this, @compnerd :)
The problem I'm trying to solve is this: I am building the 'freestanding' stdlib with a pre-built toolchain (that has ./usr/bin/clang, ./usr/bin/clang++, ./usr/bin/swiftc, ./usr/lib/libLTO.dylib) like this:
./swift/utils/build-script --preset stdlib_S_standalone_minimal_macho_x86_64,build,test toolchain_path=<PATH>

But since we're now enabling LTO on the 'freestanding' stdlib, linking must use the same toolchain. So I need a way of propagating -lto_library <PATH>/lib/libLTO.dylib into lit execution tests.

Build failed
Swift Test OS X Platform
Git Sha - 570f233d100d6f2ff85123d28d130ea8067c340f


Code Selection With Keyboard In win10 bug

How to reproduce: shift+ctrl+→ selects several words, and with shift+ctrl still holding, press ←.
Issue: the cursor goes back while the selection doesn't.

By the way, this software is what I want for months. What I want is a light-weight IDE to write shader and have some meshes to see the result immediately and this software meets my needs perfectly. Well done.

Just wanted to notify you that the v1.2.1 has fix for this + other annoying text editor bugs.