[Bug]:  Dependency aliases of transitive dependencies are not working

What happened?
Consider you are using a project that has a direct dependency on A. Where A then has a dependency on B, but the package is aliased to e.g. alias. e.g.
<EMAIL_ADDRESS>    name: '@isaacs/cliui'
    version: 8.0.2
    dependencies:
      string-width-cjs: registry.npmjs.org/string-width/4.2.3

rules_js will properly configure the store target for cliui to look for string-width-cjs as a store dependency. But the store target with the aliased name does actually not exist.
This causes errors like:
ERROR: /usr/local/google/home/pgschwendtner/projects/material.angular.io/BUILD.bazel:9:22: in deps attribute
of npm_package_store_internal rule
//:.aspect_rules_js/node_modules/@isaacs+cliui<EMAIL_ADDRESS>\
target<EMAIL_ADDRESS>does
not exist. Since this rule was created by the macro 'npm_link_all_packages', the error might have been caused by the macro implementation

Version
Linux amd64. Rules_JS rules_js-1.20.1. Bazel 5.1.0. A team member also tried with the latest version.
How to reproduce
No response
Any other information?
Looking into it without diving too much into implementation, the issue seems to be that pnpm_lock_file.packages() is used to create the store entries. But this never holds the aliased package names. i.e. rules_js iterates through the packages: entry in the lock file and then creates store entries for those.
There will never be an entry with the alias though because the alias name will only ever appear in the dependencies section of an entry. The transitive closure detection already has access to this information.
Likely we could use that to compile a more precise list of packages that will then be used
to create the links.

Have a repro here, https://github.com/aspect-build/rules_js/pull/1063. Working on a fix.

Hmm. It is a tricky one and more than an hour of work. I think this one will have to wait until next week when @jbedard or I have more time.

Notes from repro in #1063.
Some notes,... in /private/var/tmp/_bazel_greg/7aed427a991d86f99332ab79b9b11780/external/npm/repositories.bzl the generated npm_import for the package looks like this,
    npm_import(
        name = "npm__at_isaacs_cliui__registry.npmjs.org_at_isaacs_cliui_8.0.2",
        root_package = "",
        link_workspace = "",
        link_packages = {
            "npm/private/test": ["@isaacs/cliui"],
        },
        package = "@isaacs/cliui",
        version = "registry.npmjs.org/<EMAIL_ADDRESS>        url =<EMAIL_ADDRESS>        npm_translate_lock_repo = "npm",
        dev = True,
        generate_bzl_library_targets = True,
        integrity = "sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA==",
        deps = {
            "string-width"<EMAIL_ADDRESS>            "string-width-cjs"<EMAIL_ADDRESS>            "strip-ansi"<EMAIL_ADDRESS>            "strip-ansi-cjs"<EMAIL_ADDRESS>            "wrap-ansi"<EMAIL_ADDRESS>            "wrap-ansi-cjs"<EMAIL_ADDRESS>        },
        transitive_closure = {
            "@isaacs/cliui": ["registry.npmjs.org/<EMAIL_ADDRESS>            "string-width"<EMAIL_ADDRESS><EMAIL_ADDRESS>            "string-width-cjs"<EMAIL_ADDRESS>            "strip-ansi"<EMAIL_ADDRESS><EMAIL_ADDRESS>            "strip-ansi-cjs"<EMAIL_ADDRESS>            "wrap-ansi"<EMAIL_ADDRESS>            "wrap-ansi-cjs"<EMAIL_ADDRESS>            "ansi-styles"<EMAIL_ADDRESS><EMAIL_ADDRESS>            "ansi-regex"<EMAIL_ADDRESS><EMAIL_ADDRESS>            "emoji-regex"<EMAIL_ADDRESS><EMAIL_ADDRESS>            "is-fullwidth-code-point"<EMAIL_ADDRESS>            "color-convert"<EMAIL_ADDRESS>            "color-name"<EMAIL_ADDRESS>            "eastasianwidth"<EMAIL_ADDRESS>        },
    )

The transitive closure is calculated wrong from the looks of it, it should likely be:
        transitive_closure = {
            "@isaacs/cliui": ["registry.npmjs.org/<EMAIL_ADDRESS>            "string-width"<EMAIL_ADDRESS>            "string-width-cjs"<EMAIL_ADDRESS>            "strip-ansi"<EMAIL_ADDRESS>            "strip-ansi-cjs"<EMAIL_ADDRESS>            "wrap-ansi"<EMAIL_ADDRESS>            "wrap-ansi-cjs"<EMAIL_ADDRESS>            "ansi-styles"<EMAIL_ADDRESS><EMAIL_ADDRESS>            "ansi-regex"<EMAIL_ADDRESS><EMAIL_ADDRESS>            "emoji-regex"<EMAIL_ADDRESS><EMAIL_ADDRESS>            "is-fullwidth-code-point"<EMAIL_ADDRESS>            "color-convert"<EMAIL_ADDRESS>            "color-name"<EMAIL_ADDRESS>            "eastasianwidth"<EMAIL_ADDRESS>        },

Although even then I think there will also be fixed needed in npm_import itself that generates links the /private/var/tmp/_bazel_greg/7aed427a991d86f99332ab79b9b11780/external/npm__at_isaacs_cliui__registry.npmjs.org_at_isaacs_cliui_8.0.2__links/defs.bzl for the package. In particular, the deps, lc_deps & ref_deps there which currently look like
    deps = {
            ":.aspect_rules_js/{}/@isaacs+cliui@registry.npmjs.org+@isaacs+cliui@8.0.2/pkg".format(link_root_name): "@isaacs/cliui",
            ":.aspect_rules_js/{}/string-width@registry.npmjs.org+string-width@4.2.3/pkg".format(link_root_name): "string-width",
            ":.aspect_rules_js/{}/string-width@registry.npmjs.org+string-width@5.1.2/pkg".format(link_root_name): "string-width",
            ":.aspect_rules_js/{}/string-width-cjs@registry.npmjs.org+string-width@4.2.3/pkg".format(link_root_name): "string-width-cjs",
            ":.aspect_rules_js/{}/strip-ansi@registry.npmjs.org+strip-ansi@6.0.1/pkg".format(link_root_name): "strip-ansi",
            ":.aspect_rules_js/{}/strip-ansi@registry.npmjs.org+strip-ansi@7.0.1/pkg".format(link_root_name): "strip-ansi",
            ":.aspect_rules_js/{}/strip-ansi-cjs@registry.npmjs.org+strip-ansi@6.0.1/pkg".format(link_root_name): "strip-ansi-cjs",
            ":.aspect_rules_js/{}/wrap-ansi@registry.npmjs.org+wrap-ansi@8.1.0/pkg".format(link_root_name): "wrap-ansi",
            ":.aspect_rules_js/{}/wrap-ansi-cjs@registry.npmjs.org+wrap-ansi@7.0.0/pkg".format(link_root_name): "wrap-ansi-cjs",
            ":.aspect_rules_js/{}/ansi-styles@registry.npmjs.org+ansi-styles@6.2.1/pkg".format(link_root_name): "ansi-styles",
            ":.aspect_rules_js/{}/ansi-styles@registry.npmjs.org+ansi-styles@4.3.0/pkg".format(link_root_name): "ansi-styles",
            ":.aspect_rules_js/{}/ansi-regex@registry.npmjs.org+ansi-regex@6.0.1/pkg".format(link_root_name): "ansi-regex",
            ":.aspect_rules_js/{}/ansi-regex@registry.npmjs.org+ansi-regex@5.0.1/pkg".format(link_root_name): "ansi-regex",
            ":.aspect_rules_js/{}/emoji-regex@registry.npmjs.org+emoji-regex@9.2.2/pkg".format(link_root_name): "emoji-regex",
            ":.aspect_rules_js/{}/emoji-regex@registry.npmjs.org+emoji-regex@8.0.0/pkg".format(link_root_name): "emoji-regex",
            ":.aspect_rules_js/{}/is-fullwidth-code-point@registry.npmjs.org+is-fullwidth-code-point@3.0.0/pkg".format(link_root_name): "is-fullwidth-code-point",
            ":.aspect_rules_js/{}/color-convert@registry.npmjs.org+color-convert@2.0.1/pkg".format(link_root_name): "color-convert",
            ":.aspect_rules_js/{}/color-name@registry.npmjs.org+color-name@1.1.4/pkg".format(link_root_name): "color-name",
            ":.aspect_rules_js/{}/eastasianwidth@registry.npmjs.org+eastasianwidth@0.2.0/pkg".format(link_root_name): "eastasianwidth",
        }
    lc_deps = {
            ":.aspect_rules_js/{}/@isaacs+cliui@registry.npmjs.org+@isaacs+cliui@8.0.2/pkg_pre_lc_lite".format(link_root_name): "@isaacs/cliui",
            ":.aspect_rules_js/{}/string-width@registry.npmjs.org+string-width@4.2.3/pkg".format(link_root_name): "string-width",
            ":.aspect_rules_js/{}/string-width@registry.npmjs.org+string-width@5.1.2/pkg".format(link_root_name): "string-width",
            ":.aspect_rules_js/{}/string-width-cjs@registry.npmjs.org+string-width@4.2.3/pkg".format(link_root_name): "string-width-cjs",
            ":.aspect_rules_js/{}/strip-ansi@registry.npmjs.org+strip-ansi@6.0.1/pkg".format(link_root_name): "strip-ansi",
            ":.aspect_rules_js/{}/strip-ansi@registry.npmjs.org+strip-ansi@7.0.1/pkg".format(link_root_name): "strip-ansi",
            ":.aspect_rules_js/{}/strip-ansi-cjs@registry.npmjs.org+strip-ansi@6.0.1/pkg".format(link_root_name): "strip-ansi-cjs",
            ":.aspect_rules_js/{}/wrap-ansi@registry.npmjs.org+wrap-ansi@8.1.0/pkg".format(link_root_name): "wrap-ansi",
            ":.aspect_rules_js/{}/wrap-ansi-cjs@registry.npmjs.org+wrap-ansi@7.0.0/pkg".format(link_root_name): "wrap-ansi-cjs",
            ":.aspect_rules_js/{}/ansi-styles@registry.npmjs.org+ansi-styles@6.2.1/pkg".format(link_root_name): "ansi-styles",
            ":.aspect_rules_js/{}/ansi-styles@registry.npmjs.org+ansi-styles@4.3.0/pkg".format(link_root_name): "ansi-styles",
            ":.aspect_rules_js/{}/ansi-regex@registry.npmjs.org+ansi-regex@6.0.1/pkg".format(link_root_name): "ansi-regex",
            ":.aspect_rules_js/{}/ansi-regex@registry.npmjs.org+ansi-regex@5.0.1/pkg".format(link_root_name): "ansi-regex",
            ":.aspect_rules_js/{}/emoji-regex@registry.npmjs.org+emoji-regex@9.2.2/pkg".format(link_root_name): "emoji-regex",
            ":.aspect_rules_js/{}/emoji-regex@registry.npmjs.org+emoji-regex@8.0.0/pkg".format(link_root_name): "emoji-regex",
            ":.aspect_rules_js/{}/is-fullwidth-code-point@registry.npmjs.org+is-fullwidth-code-point@3.0.0/pkg".format(link_root_name): "is-fullwidth-code-point",
            ":.aspect_rules_js/{}/color-convert@registry.npmjs.org+color-convert@2.0.1/pkg".format(link_root_name): "color-convert",
            ":.aspect_rules_js/{}/color-name@registry.npmjs.org+color-name@1.1.4/pkg".format(link_root_name): "color-name",
            ":.aspect_rules_js/{}/eastasianwidth@registry.npmjs.org+eastasianwidth@0.2.0/pkg".format(link_root_name): "eastasianwidth",
        }
    ref_deps = {
            ":.aspect_rules_js/{}/string-width@registry.npmjs.org+string-width@5.1.2/ref".format(link_root_name): "string-width",
            ":.aspect_rules_js/{}/string-width-cjs@registry.npmjs.org+string-width@4.2.3/ref".format(link_root_name): "string-width-cjs",
            ":.aspect_rules_js/{}/strip-ansi@registry.npmjs.org+strip-ansi@7.0.1/ref".format(link_root_name): "strip-ansi",
            ":.aspect_rules_js/{}/strip-ansi-cjs@registry.npmjs.org+strip-ansi@6.0.1/ref".format(link_root_name): "strip-ansi-cjs",
            ":.aspect_rules_js/{}/wrap-ansi@registry.npmjs.org+wrap-ansi@8.1.0/ref".format(link_root_name): "wrap-ansi",
            ":.aspect_rules_js/{}/wrap-ansi-cjs@registry.npmjs.org+wrap-ansi@7.0.0/ref".format(link_root_name): "wrap-ansi-cjs",
        }

cc @jbedard

The other key observation is that this issue ONLY happens when the package name in the pnpm lock file starts with registry.npmjs.org/ such as
TODO

which only happens when the .npmrc registry setting is set for it in the repro:
When registry is not set for the package in npmrc then the pnpm-lock.yaml file package entry looks like this:
  /<EMAIL_ADDRESS>    resolution: {integrity: sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA==}
    engines: {node: '>=12'}
    dependencies:
      string-width: 5.1.2
      string-width-cjs<EMAIL_ADDRESS>      strip-ansi: 7.0.1
      strip-ansi-cjs<EMAIL_ADDRESS>      wrap-ansi: 8.1.0
      wrap-ansi-cjs<EMAIL_ADDRESS>    dev: true

with the aliases clearly distinguishable and rules_js correctly handles this default case.
The generated npm_import looks like this in this case,
    npm_import(
        name = "npm__at_isaacs_cliui__8.0.2",
        root_package = "",
        link_workspace = "",
        link_packages = {
            "npm/private/test": ["@isaacs/cliui"],
        },
        package = "@isaacs/cliui",
        version = "8.0.2",
        url =<EMAIL_ADDRESS>        npm_translate_lock_repo = "npm",
        dev = True,
        generate_bzl_library_targets = True,
        integrity = "sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA==",
        deps = {
            "string-width": "5.1.2",
            "string-width-cjs": "/string-width/4.2.3",
            "strip-ansi": "7.0.1",
            "strip-ansi-cjs": "/strip-ansi/6.0.1",
            "wrap-ansi": "8.1.0",
            "wrap-ansi-cjs": "/wrap-ansi/7.0.0",
        },
        transitive_closure = {
            "@isaacs/cliui": ["8.0.2"],
            "string-width": ["4.2.3", "5.1.2"],
            "strip-ansi": ["6.0.1", "7.0.1"],
            "wrap-ansi": ["7.0.0", "8.1.0"],
            "ansi-styles": ["6.2.1", "4.3.0"],
            "color-convert": ["2.0.1"],
            "color-name": ["1.1.4"],
            "ansi-regex": ["6.0.1", "5.0.1"],
            "emoji-regex": ["9.2.2", "8.0.0"],
            "is-fullwidth-code-point": ["3.0.0"],
            "eastasianwidth": ["0.2.0"],
        },
    )

and the deps, lc_deps and ref_deps look like this,
    deps = {
            ":.aspect_rules_js/{}/@isaacs+cliui@registry.npmjs.org+@isaacs+cliui@8.0.2/pkg".format(link_root_name): "@isaacs/cliui",
            ":.aspect_rules_js/{}/string-width@registry.npmjs.org+string-width@4.2.3/pkg".format(link_root_name): "string-width",
            ":.aspect_rules_js/{}/string-width@registry.npmjs.org+string-width@5.1.2/pkg".format(link_root_name): "string-width",
            ":.aspect_rules_js/{}/string-width-cjs@registry.npmjs.org+string-width@4.2.3/pkg".format(link_root_name): "string-width-cjs",
            ":.aspect_rules_js/{}/strip-ansi@registry.npmjs.org+strip-ansi@6.0.1/pkg".format(link_root_name): "strip-ansi",
            ":.aspect_rules_js/{}/strip-ansi@registry.npmjs.org+strip-ansi@7.0.1/pkg".format(link_root_name): "strip-ansi",
            ":.aspect_rules_js/{}/strip-ansi-cjs@registry.npmjs.org+strip-ansi@6.0.1/pkg".format(link_root_name): "strip-ansi-cjs",
            ":.aspect_rules_js/{}/wrap-ansi@registry.npmjs.org+wrap-ansi@8.1.0/pkg".format(link_root_name): "wrap-ansi",
            ":.aspect_rules_js/{}/wrap-ansi-cjs@registry.npmjs.org+wrap-ansi@7.0.0/pkg".format(link_root_name): "wrap-ansi-cjs",
            ":.aspect_rules_js/{}/ansi-styles@registry.npmjs.org+ansi-styles@6.2.1/pkg".format(link_root_name): "ansi-styles",
            ":.aspect_rules_js/{}/ansi-styles@registry.npmjs.org+ansi-styles@4.3.0/pkg".format(link_root_name): "ansi-styles",
            ":.aspect_rules_js/{}/ansi-regex@registry.npmjs.org+ansi-regex@6.0.1/pkg".format(link_root_name): "ansi-regex",
            ":.aspect_rules_js/{}/ansi-regex@registry.npmjs.org+ansi-regex@5.0.1/pkg".format(link_root_name): "ansi-regex",
            ":.aspect_rules_js/{}/emoji-regex@registry.npmjs.org+emoji-regex@9.2.2/pkg".format(link_root_name): "emoji-regex",
            ":.aspect_rules_js/{}/emoji-regex@registry.npmjs.org+emoji-regex@8.0.0/pkg".format(link_root_name): "emoji-regex",
            ":.aspect_rules_js/{}/is-fullwidth-code-point@registry.npmjs.org+is-fullwidth-code-point@3.0.0/pkg".format(link_root_name): "is-fullwidth-code-point",
            ":.aspect_rules_js/{}/color-convert@registry.npmjs.org+color-convert@2.0.1/pkg".format(link_root_name): "color-convert",
            ":.aspect_rules_js/{}/color-name@registry.npmjs.org+color-name@1.1.4/pkg".format(link_root_name): "color-name",
            ":.aspect_rules_js/{}/eastasianwidth@registry.npmjs.org+eastasianwidth@0.2.0/pkg".format(link_root_name): "eastasianwidth",
        }
    lc_deps = {
            ":.aspect_rules_js/{}/@isaacs+cliui@registry.npmjs.org+@isaacs+cliui@8.0.2/pkg_pre_lc_lite".format(link_root_name): "@isaacs/cliui",
            ":.aspect_rules_js/{}/string-width@registry.npmjs.org+string-width@4.2.3/pkg".format(link_root_name): "string-width",
            ":.aspect_rules_js/{}/string-width@registry.npmjs.org+string-width@5.1.2/pkg".format(link_root_name): "string-width",
            ":.aspect_rules_js/{}/string-width-cjs@registry.npmjs.org+string-width@4.2.3/pkg".format(link_root_name): "string-width-cjs",
            ":.aspect_rules_js/{}/strip-ansi@registry.npmjs.org+strip-ansi@6.0.1/pkg".format(link_root_name): "strip-ansi",
            ":.aspect_rules_js/{}/strip-ansi@registry.npmjs.org+strip-ansi@7.0.1/pkg".format(link_root_name): "strip-ansi",
            ":.aspect_rules_js/{}/strip-ansi-cjs@registry.npmjs.org+strip-ansi@6.0.1/pkg".format(link_root_name): "strip-ansi-cjs",
            ":.aspect_rules_js/{}/wrap-ansi@registry.npmjs.org+wrap-ansi@8.1.0/pkg".format(link_root_name): "wrap-ansi",
            ":.aspect_rules_js/{}/wrap-ansi-cjs@registry.npmjs.org+wrap-ansi@7.0.0/pkg".format(link_root_name): "wrap-ansi-cjs",
            ":.aspect_rules_js/{}/ansi-styles@registry.npmjs.org+ansi-styles@6.2.1/pkg".format(link_root_name): "ansi-styles",
            ":.aspect_rules_js/{}/ansi-styles@registry.npmjs.org+ansi-styles@4.3.0/pkg".format(link_root_name): "ansi-styles",
            ":.aspect_rules_js/{}/ansi-regex@registry.npmjs.org+ansi-regex@6.0.1/pkg".format(link_root_name): "ansi-regex",
            ":.aspect_rules_js/{}/ansi-regex@registry.npmjs.org+ansi-regex@5.0.1/pkg".format(link_root_name): "ansi-regex",
            ":.aspect_rules_js/{}/emoji-regex@registry.npmjs.org+emoji-regex@9.2.2/pkg".format(link_root_name): "emoji-regex",
            ":.aspect_rules_js/{}/emoji-regex@registry.npmjs.org+emoji-regex@8.0.0/pkg".format(link_root_name): "emoji-regex",
            ":.aspect_rules_js/{}/is-fullwidth-code-point@registry.npmjs.org+is-fullwidth-code-point@3.0.0/pkg".format(link_root_name): "is-fullwidth-code-point",
            ":.aspect_rules_js/{}/color-convert@registry.npmjs.org+color-convert@2.0.1/pkg".format(link_root_name): "color-convert",
            ":.aspect_rules_js/{}/color-name@registry.npmjs.org+color-name@1.1.4/pkg".format(link_root_name): "color-name",
            ":.aspect_rules_js/{}/eastasianwidth@registry.npmjs.org+eastasianwidth@0.2.0/pkg".format(link_root_name): "eastasianwidth",
        }
    ref_deps = {
            ":.aspect_rules_js/{}/string-width@registry.npmjs.org+string-width@5.1.2/ref".format(link_root_name): "string-width",
            ":.aspect_rules_js/{}/string-width-cjs@registry.npmjs.org+string-width@4.2.3/ref".format(link_root_name): "string-width-cjs",
            ":.aspect_rules_js/{}/strip-ansi@registry.npmjs.org+strip-ansi@7.0.1/ref".format(link_root_name): "strip-ansi",
            ":.aspect_rules_js/{}/strip-ansi-cjs@registry.npmjs.org+strip-ansi@6.0.1/ref".format(link_root_name): "strip-ansi-cjs",
            ":.aspect_rules_js/{}/wrap-ansi@registry.npmjs.org+wrap-ansi@8.1.0/ref".format(link_root_name): "wrap-ansi",
            ":.aspect_rules_js/{}/wrap-ansi-cjs@registry.npmjs.org+wrap-ansi@7.0.0/ref".format(link_root_name): "wrap-ansi-cjs",
        }

@devversion This is a potential work-around for the material repository: https://github.com/angular/material.angular.io/pull/1208/files

@gregmagolan Thanks for the investigation. So based on that, to me it sounds like the registry prefixes are currently just throwing off the logic, but in practice- with both variants (with registry prefix, or not), the aliases should be distinguishable, right? e.g.
with prefixes
  registry.npmjs.org/<EMAIL_ADDRESS>    resolution: {integrity: sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA==, registry: https://registry.yarnpkg.com/, tarball<EMAIL_ADDRESS>    name: '@isaacs/cliui'
    version: 8.0.2
    engines: {node: '>=12'}
    dependencies:
      string-width<EMAIL_ADDRESS>    <----- Same package name. No alias.
      string-width-cjs<EMAIL_ADDRESS><----- Different package name. Can be detected.
      strip-ansi<EMAIL_ADDRESS>      strip-ansi-cjs<EMAIL_ADDRESS>      wrap-ansi<EMAIL_ADDRESS>      wrap-ansi-cjs<EMAIL_ADDRESS>    dev: true

without prefix
  /<EMAIL_ADDRESS>    resolution: {integrity: sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA==}
    engines: {node: '>=12'}
    dependencies:
      string-width: 5.1.2                                             <----- No alias.
      string-width-cjs<EMAIL_ADDRESS>                         <----- Different package name. Can be detected.
      strip-ansi: 7.0.1
      strip-ansi-cjs<EMAIL_ADDRESS>      wrap-ansi: 8.1.0
      wrap-ansi-cjs<EMAIL_ADDRESS>    dev: true

Essentially, yes. The logic breaks for the aliases somewhere along the way with the registry.npmjs.org/ prefixed versions. I started down the path of finding fixing it but it was more work than I had time for and with a viable work-around I'm going to put fixing the registry.npmjs.org/ on the backlog since we have other higher priority work at the moment.

FYI that team members often run into issues, even with the workaround, because pnpm import seems to behave different when running under yarn bazel. The registry prefixes are always added then- reverting the workarounds basically

I think this may have been fixed with various lockfile cleanup/fixes. @devversion any idea if you're still running into this issue?


feature(test-runner): Restart Process After X Runs Decorator

Is your feature request related to a problem? Please describe.
When testing Stryker v4, I'm finding that there's still significant memory leak issues in our project with jest. Having the tests randomly fail due to a V8 memory allocation issue that can't be tracked down that well is a problem. On top of that, memory leaks might not always be in an end-users control. The leakage might just be a part of jest or a third party library that's leaking unexpectedly.
Describe the solution you'd like
I'd like to have a RestartDirtyProcessDecorator that takes in an option from the Stryker config to restart a process after x amount of mutant runs (a default of perhaps 20), with 0 disabling it altogether. I would also like an option to enable memory usage logging, that would log the memory usage for each process after each mutant, so that you can see the memory usage of each process and determine how/where you need to set this restart limit.
On top of that, the other thing I noticed was that one process for mutant runs started with a lot (600MB or Mb) of memory already allocated from the DryRun. I would like the process that executes the DryRun to restart after the DryRun so that all child processes are at the same level of "cleanliness" for mutant runs.
Describe alternatives you've considered
Other options

Determine Restart threshold by memory usage instead

Maybe collect memory usage from the first run and restart if total usage goes about 4x the initial usage?

Integrate some kind of memory snapshot tool to allow someone to view what the debug leak is

Hi @Garethp thanks for opening this issue.
We already have a decorator that helps to deal with memory leaks, the RetryDecorator. Isn't it working for you?
I don't want to restart after x times. This decorator would punish you if your tests and test runner don't a memory leak. Test runner processes can be really expensive. Think of karma with the karma-webpack plugin, you really don't want to ever restart that one.
I had this issue for Stryker itself as well, I think 2 years ago. The problem then was migrating sinon from v4 to v5. From version 5 onward, sinon.stub() registers the created stubs in the registry of its default sandbox. To clear this, you need to call sinon.restore() (preferably after each test). We didn't know this at the time. After hours of debugging, we finally found out that this was the root cause and we did a PR to improve sinon's documentation. What I'm trying to say with this example, is that it is far better to prevent memory leaks in the first place, instead of dealing with the consequences of them IMO.

I would like the process that executes the DryRun to restart after the DryRun so that all child processes are at the same level of "cleanliness" for mutant runs.

I was actually pretty happy with this side effect from moving to mutation switching. Take the karma-webpack example again. Reusing the process is a significant performance improvement already.

Integrate some kind of memory snapshot tool to allow someone to view what the debug leak is

I've actually been thinking about this as well when we had the Sinon issues. I've decided against it back then because it is very difficult to see if there is a memory leak from memory snapshots alone. JavaScript processes tend to grow in memory size until the garbage collector decides to run at arbitrary times. Thus a growing memory between runs isn't a smoking gun for a memory leak 🤷♂


blackconnect vs. black file watcher

What is the benefit of this over adding a file watcher that formats the file on save? It works pretty well even with large files. This is the settings I use

Same as with using blackd over invoking black separately - it saves a little bit of time over starting a new black process. Savings can be most noticeable when you work with a lot of files and change them frequently.
With File Watchers, on each file save PyCharm will spawn a new process and that takes time. With this plugin, you have to trigger reformatting manually (for the moment), but the webserver process is already started and only overhead of network connection is paid.
Also, there can be an issue with you updating VCS repo outside of PyCharm and triggering File Watcher for each file, thus spawning a lot of black processes.
All that said, at this time "File Watchers" is the more convenient solution since you do not need to start blackd by hand and manually trigger re-formatting. Hopefully, that will change soon :)


Removed the import of bits/stdc++

The header is compiler-bound, so clang or macos wouldn't be able to compile the project. The header itself, if you look at the GNU GCC source -> https://gcc.gnu.org/onlinedocs/gcc-4.8.0/libstdc++/api/a01541_source.html
It imports every header under the Sun, so even at the header level, makes unnecessary imports. Because it imports precompiled headers, it can theoretically speedup compilation (if used right, not intimate with the subtleties myself), but other compilers can't use it.
Removing it compiles the project on any platform now. Your set operations and other calls to C++ stdlib are already handled in other files.
Theoretically, you could maintain a local copy of bits/stdc++ to make it cross-platform -> https://gist.github.com/frankchen0130/9ac562b55fa7e03689bca30d0e52b0e5

Honestly can't remember why this was in there, must've been some artifact from something I was testing and forgot to remove


Interactive pane broken

Issue Type: Bug
With "jupyter.interactiveWindowMode": "single" multiple interactive panes insist on opening up. Also, when inputting code in the directly in the interactive pane, the window will jump to another separate interactive pane and the number of the original interactive pane that the code was executed from shown in the interactive pane tab name will auto increment by one.  Do the same thing again, and the same number in the tab name will decrease by one.  keep doing this over and over, and it will just cycle between the two number.
Something is severely broken.
Also, repeated pop ups saying Failed to request creation of interactive window from VS Code.  Not for all Interactive panes though, only approximately half of the Interactive panes.
None of the Interactive panes work.  I just get bounced around to different Interactive panes that pop up and change their number every time I execute code from a file or directly from an Interactive pane.
I have tried with localhost python instance, and also from remote Jupyter servers.  Python on the localhost computer and Jupyter labs or notebooks on the remote server run without issue.  It is only VS Code that is having the problems.
Extension version: 2021.10.1101450599
VS Code version: Code 1.62.3 (ccbaa2d27e38e5afa3e5c21c1c7bef4657064247, 2021-11-17T08:11:14.551Z)
OS version: Windows_NT x64 10.0.18363
Restricted Mode: No

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz (4 x 2496)

GPU Status
2d_canvas: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onrasterization: enabledskia_renderer: enabled_onvideo_decode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
15.88GB (5.07GB free)

Process Argv
--crash-reporter-id f652528b-5678-4b12-9ca0-3de6b1e1135f

Screen Reader
no

VM
0%

A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythontb:30283811
pythonvspyt551:30345470
pythonptprofiler:30281270
vsdfh931cf:30280410
vshan820:30294714
vstes263cf:30335440
vscoreces:30384385
pythondataviewer:30285071
vscod805:30301674
pythonvspyt200:30340761
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
pythonrunftest32:30373476
pythonf5test824:30373475
javagetstartedt:30391933
pythonvspyt187:30373474
vsqsis200cf:30402084
vsaa593:30376534
vssld246cf:30401891
vscexrecpromptc:30404946
vscop804cf:30404767
vscop940:30404999

Not sure what happened during the last VS Code update, but it broke a lot.
Uninstalled all MS Python and Jupyter extensions, then re-installed.  Seems to have fixed the issue ¯_(ツ)_/¯

Thanks for retrying. Maybe the update only partially updated. We have seen cases where it doesn't download fully for some reason.


RIFE denoise

Can this script be put in compiled form:
function RIFE_degrain1(clip src)
{
ConvertToRGB(src)
ConvertToPlanarRGB()
ConvertBits(32)
src32=last
even=SelectEvery(2,0)
odd=SelectEvery(2,1)
even_d=RIFE(even)
odd_d=RIFE(odd)
even_d=SelectEvery(even_d,2,1)
odd_d=SelectEvery(odd_d,2,1)
den_int=Interleave(even_d, odd_d).Trim(1,0)
src32_trim=Trim(src32,1,0)
den_int2=Average(den_int, 0.666, src32_trim, 0.3333)
return den_int2.ConvertToRGB24().ConvertToYV12()
}
?
May be as addional mode to this plugin or separate .dll.
And may be a bit increased options for 'larger temporal radius' like 2 or 3 or more. It may help to performance by decreasing AVS+ cashing inbetween script objects.
Also for AVS users the current documentation is a bit unclear - the 32bit RGB may be RGBA 8bit but RIFE only accepts RGBPS float32 planar format. May be add some example script or direct note about 32bit mean float32 each RGB channel samples encoding - not 8bit integer with 4 samples per RGBA pack.

As I see the original rife-ncnn-vulkan https://github.com/nihui/rife-ncnn-vulkan is simple (2 frames in -> 1 frame out) interpolation engine. So the script above is tr=1 temporal denoise using formula
out_frame = current_frame * (1/3) + rife_ncnn(+1, -1) * (2/3)
For better denosing with possible still not very great artifacts more tr-modes possible like tr=2 :
out_frame = current_frame * (1/5) + rife_ncnn(+1, -1) * (2/5) + rife_ncnn(+2, -2) * (2/5)
so for tr=N :
out_frame = sum _N (current_frame * (1/(N2+1)) + rife_ncnn(+1, -1) * (2/(N2+1)) +...+ rife_ncnn(+N, -N) * (2/ (N*2+1))
and so on. With AVS scripting it is more hard to simulate but with program compiled form is much easier.

At first look I would leave the colorspace conversions for the other tools.
I have to look at the rest but no eta.

Yes - it is enough if RIFE plugin will continue accept its 32bit float RGB.
Only need to add logic for fetching interpolated result from different
pairs of previous and next frames from RIFE and calculate weighted average
with the RIFE results and current frame. In float32 it is very easy and
precise.
On Thu, Oct 27, 2022 at 12:58 AM Asd-g @.***> wrote:

At first look I would leave the colorspace conversions for the other tools.
I have to look at the rest but no eta.
—
Reply to this email directly, view it on GitHub
https://github.com/Asd-g/AviSynthPlus-RIFE/issues/2#issuecomment-1292700859,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQMGLM4VWIOP7BF5V5YKDBDWFGSPRANCNFSM6AAAAAARIY3GA4
.
You are receiving this because you authored the thread.Message ID:
@.***>

Also for AVS users the current documentation is a bit unclear - the 32bit RGB may be RGBA 8bit but RIFE only accepts RGBPS float32 planar format. May be add some example script or direct note about 32bit mean float32 each RGB channel samples encoding - not 8bit integer with 4 samples per RGBA pack.

RGBA 8bit (RGB32) is not planar format.
From the  readme It must be in RGB 32-bit planar format.  - implies rgbps/rgbaps. Or I missing something?

For old AVS users 32bit format typically mean RGBA32. Yes - it is not planar. But 32bit RGB planar also may be unclear - like 32bit integer samples format. It is good to add note about float 32bit samples RGB planar (separated planes).

This is very interesting, is it going to happen?
Is it possible to use another method (maybe optional with a 'filterClip') such as temporalsoften or other temporal filters than average? In the same manner as mvtools mcompensated-Temporaldenoiser-selectevery().

@DTL2020, try the attached version. There is additional bool parameter denoise. Currently it does the exactly same operations as the script function from the first comment. It requires the plugin Average. Eventually that dependency could be removed.
https://github.com/Asd-g/AviSynthPlus-RIFE/tree/denoise
RIFE_denoise.zip

I do not have access to required GPU hardware to test so I ask users at
doom9 forum to make test.
I think with compiled implementation may be it is not required to duplicate
full script structure and only make call to RIFE (-1, +1) frame creation
for Average() processing ? With RIFE (-1, +1) I mean call to RIFE with a
pair of frames of -1 and +1 number from current requested frame number. So
RIFE will recreate interpolated frame (from previous and next) at timestamp
of 'current' frame for Average() to finish work of averaging (previous,
current, next) frames. If it possible with current RIFE implementation. May
be it will save lots of hardware resources if skip creation of non-needed
frames in 2 RIFE calls and SelectEvery() ? Or AVS engine do not really
request non-needed frame numbers and no resources wasted ?
As I remember from testing scripted denoise form it require lots of
hardware resources and can run at 6 GB GPU board with 1920x1080 frames with
only 3 threads (4 or more threads already can cause out of resources error).
On Thu, Feb 9, 2023 at 12:36 AM Asd-g @.***> wrote:

@DTL2020 https://github.com/DTL2020, try the attached version. There is
additional bool parameter denoise. Currently it does the exactly same
operations as the script function from the first comment. It requires the
plugin Average. Eventually that dependency could be removed.
https://github.com/Asd-g/AviSynthPlus-RIFE/tree/denoise
RIFE_denoise.zip
https://github.com/Asd-g/AviSynthPlus-RIFE/files/10690051/RIFE_denoise.zip
—
Reply to this email directly, view it on GitHub
https://github.com/Asd-g/AviSynthPlus-RIFE/issues/2#issuecomment-1423270259,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQMGLM26BP4HHZNBYM5MDTLWWQGULANCNFSM6AAAAAARIY3GA4
.
You are receiving this because you were mentioned.Message ID:
@.***>

RIFE (-1, +1) in one pass isn't possible. You can take a look at https://github.com/Asd-g/AviSynthPlus-RIFE/blob/main/src/rife.cpp#L352
The script implementation with prefetch(3) - GPU mem ~4500MB, RAM ~3300MB.
RIFE(denoise=true) with prefetch(3) - GPU mem ~3800MB, RAM ~2300MB.
Btw my GPU (RX480 8GB) is already saturated with prefetch(2).
The script implementation with prefetch(2) - GPU mem ~4000MB, RAM ~2500MB.
RIFE(denoise=true) with prefetch(2) - GPU mem ~2800MB, RAM ~1700MB.
Above tests are made with 1080p video and default RIFE model.

As I see in RIFE_process() it accept src0, src1 and dst pointers as input
params. So it was idea that RIFE takes only 2 input frames and timing
information in between (in the 0..1 float internal) for creation of output
frame (so it create motion estimation and compensation based only on 2
input frames only ?). Or it really takes and analyses (>2 frames) sequence
of many sequential frames to create the requested interpolated frame
between provided 0 and 1 src frames for RIFE_process()? I still not
understand completely how that RIFE_process() works.
If it takes only 2 frames - so it was idea in compiled form it is possible
to request 2 frames of -1 and +1 from previous filter and pass to
single RIFE processing environment. So it should save from creating 2
streams of half frame numbers and 2 RIFE processing streams to create both
even and odd interpolated frames and possibly save half of hardware and
compute resources, Only put workarounds to 0 and last output frame for
which -1 and +1 source frames not exist - simply replace -1 frame with
current for 0 output frame and +1 with current for last output frame.
In the AVS scripting form I think AVS caching may force 2 RIFE filters to
fill some cache buffers with every sequential frame and make 2x redundant
work because only half of frames from 2 RIFE streams actually requested for
Average().
On Thu, Feb 9, 2023 at 1:56 PM Asd-g @.***> wrote:

RIFE (-1, +1) in one pass isn't possible. You can take a look at
https://github.com/Asd-g/AviSynthPlus-RIFE/blob/main/src/rife.cpp#L352
The script implementation with prefetch(3) - GPU mem ~4500MB, RAM ~3300MB.
RIFE(denoise=true) with prefetch(3) - GPU mem ~3800MB, RAM ~2300MB.
Btw my GPU (RX480 8GB) is already saturated with prefetch(2).
The script implementation with prefetch(2) - GPU mem ~4000MB, RAM ~2500MB.
RIFE(denoise=true) with prefetch(2) - GPU mem ~2800MB, RAM ~1700MB.
Above tests are made with 1080p video and default RIFE model.
—
Reply to this email directly, view it on GitHub
https://github.com/Asd-g/AviSynthPlus-RIFE/issues/2#issuecomment-1423994783,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQMGLMYSUSUFFZ5VWVZVKZTWWTEMXANCNFSM6AAAAAARIY3GA4
.
You are receiving this because you were mentioned.Message ID:
@.***>

s(o it create motion estimation and compensation based only on 2
input frames only ?)

So it can run in 2-input frames mode only (not large sequence to analyse).

Yes, it's 2-input frames mode only.

So it can run in 2-input frames mode only (not large sequence to analyse).

You already have n and n+1 by default. So for denoise=true you want n-1 (auto src0{ avs_get_frame(fi->child, frameNum - 1) };)  (with adjusted timestep) instead n (auto src0{ avs_get_frame(fi->child, frameNum) };)?

Yes - for complete internal denoise with tr=1 we need to request 3 frames
from AVS environment:
n-1 is avs_get_frame(fi->child, frameNum - 1)
n is avs_get_frame(fi->child, frameNum)
n+1 is avs_get_frame(fi->child, frameNum + 1)
Frames n-1 and n+1 we send to RIFE_process and result of RIFE and n we send
to our simple linear blending engine Average().
It is simplest form but work only nice if RIFE provide ideal motion
compensation.
So some users may want to use more advanced non-linear blending engine in
AVS scripting so it may be good do add more control param like
'denoise_blending_mode' of 'average' or 'none'
In 'none' mode we simply output RIFE processed interpolated frame so user
can apply it into more advanced blending plugin/filter to attempt to save
from bad blends where RIFE fails significantly with motion compensation.
Also for experiments if may be added control param like 'denoise_tr' with
default of 1 and use it instead of +-1 in frames request for RIFE
interpolation.
So user may want to test more deep denoise in scripting with requesting
different interpolation 'temporal radius' of >1. Though the quality of
interpolation may quickly degrades with increasing tr but 2 may be still
working not very bad. So it will be possible to create more simple denoise
scripts like
int_tr1=RIFE(denoise=true, denoise_blending_mode=none, tr=1)
int_tr2=RIFE(denoise=true, denoise_blending_mode=none, tr=2)
Average(last, 1/5, int_tr1, 2/5, int_tr2, 2/5)
To simulate MDegrain2 activity. Also user can use other non-linear blending
instead of Average().
For the processing optimization it is better to perform Average() math
inside hardware accelerator and upload n-frame too for Average(). In
sequential AVS processing it is possible to upload 1 src frame and download
1 output frame for each output frame to have minimal possible bus traffic.
With uploaded resources referencing - for output frame n+1 our previously
uploaded resources is:
n-1 - skip
n become n-1 for RIFE input
n+1 become n for Average() input
uploaded next frame - become n+1 for RIFE input
and so on. So it need 3 frame resources in hardware accelerator buffer with
'shifting' for each next output denoised frame.
On Thu, Feb 9, 2023 at 4:24 PM Asd-g @.***> wrote:

s(o it create motion estimation and compensation based only on 2
input frames only ?)
So it can run in 2-input frames mode only (not large sequence to analyse).
Yes, it's 2-input frames mode only.
So it can run in 2-input frames mode only (not large sequence to analyse).
You already have n
https://github.com/Asd-g/AviSynthPlus-RIFE/blob/main/src/plugin.cpp#L146
and n+1
https://github.com/Asd-g/AviSynthPlus-RIFE/blob/main/src/plugin.cpp#L332
by default. So for denoise=true you want n-1 (auto src0{
avs_get_frame(fi->child, frameNum - 1) };) (with adjusted timestep)
instead n (auto src0{ avs_get_frame(fi->child, frameNum) };)?
—
Reply to this email directly, view it on GitHub
https://github.com/Asd-g/AviSynthPlus-RIFE/issues/2#issuecomment-1424188358,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQMGLM3JBQ2YR63CZZB5CIDWWTVZJANCNFSM6AAAAAARIY3GA4
.
You are receiving this because you were mentioned.Message ID:
@.***>

Try the attached version. It should be used only for denoising.
denoise parameter is removed.
New parameter denoise_tr. Default 1, it must be greater than 0.
New parameter denoise_blending_mode. 0 - average (still average plugin required); 1 - none. Default 0.
RIFE_denoise1.zip

The invernal blending looks like work good. At GTX1060 6 GB card its compute1 performance counter reach 100% at Prefetch(2) so any other increasing threads number in AVS+ not increase script fps and only increase GPU memory load and afterPrefetch(4) start to crash with vk_exceptions (like out of memory errors).
But when I try output frames for external blending something strange happens with AVS+ environment - it looks the 'last' variable get all frames processed by RIFE calls so Average() plugin not work because clips number of frames not equal:
Here is attempts:
ConvertToRGB()
ConvertToPlanarRGB()
ConvertBits(32)
src_den=last
#return last.ConvertToRGB24().ConvertToYV12()
tr1=RIFE(denoise_tr=1, denoise_blending_mode=1)
tr2=RIFE(denoise_tr=2, denoise_blending_mode=1)
return src_den.ConvertToRGB24().ConvertToYV12()
Average(src_den, 0.3333, tr1, 0.6666)
#Average(last, 0.2, tr1, 0.4, tr2, 0.4)
ConvertToRGB24()
ConvertToYV12()
Even simple return src_den.ConvertToRGB24().ConvertToYV12() after RIFE() calls return clip with frame number about several times more in compare with 'last' before RIFE calls. So all Average() calls for both tr1 and tr1+tr2 not work with either 'last' or even 'src_den' clip taken 'before' RIFE calls. Maybe RIFE with denoise_blending_mode=1 somehow corrupt AVS+ environment ? I use AVS+ version 3.7.3 (r3820, 3.7, x86_64)

As a temporal workaround I try Trim() 'last' clip close to its real number or frames before Average and it finally start to work:
src_den=last
tr1=RIFE(denoise_tr=1, denoise_blending_mode=1)
tr2=RIFE(denoise_tr=2, denoise_blending_mode=1)
tr3=RIFE(denoise_tr=2, denoise_blending_mode=1)
src_den=Trim(src_den, 0, 1700)
Average(src_den, 0.1428, tr1, 0.2857, tr2, 0.2857, tr3, 0.2857)
ConvertToRGB24()
ConvertToYV12()
AssumeFPS(50)
At least tr=3 is working OK. But the resulted 'last' clip got fps of about 400 as reported by VirtualDub - so I also need to assume it back to 50. So the RIFE calls somehow change fps of the AVS environment too.
At Win10 with 16 GB RAM and GTX1060 6 GB it work now up to tr6 looks like good, tr7 output no-colour and low Y values and crashes. At least working is
src_den=last
tr1=RIFE(denoise_tr=1, denoise_blending_mode=1)
tr2=RIFE(denoise_tr=2, denoise_blending_mode=1)
tr3=RIFE(denoise_tr=3, denoise_blending_mode=1)
tr4=RIFE(denoise_tr=4, denoise_blending_mode=1)
tr5=RIFE(denoise_tr=5, denoise_blending_mode=1)
tr6=RIFE(denoise_tr=6, denoise_blending_mode=1)
src_den=Trim(src_den, 0, 1700)
tr1=Trim(tr1, 0, 1700)
tr2=Trim(tr2, 0, 1700)
tr3=Trim(tr3, 0, 1700)
tr4=Trim(tr4, 0, 1700)
tr5=Trim(tr5, 0, 1700)
tr6=Trim(tr6, 0, 1700)
w6_1=1.0/13.0
w6_res=2.0/13.0
Average(src_den, w6_1, tr1, w6_res, tr2, w6_res, tr3, w6_res, tr4, w6_res, tr5, w6_res, tr6, w6_res)
With Prefetch(2)
When using Prefetch(1) it can go up to tr12 still working - good enough.

But when I try output frames for external blending something strange happens with AVS+ environment - it looks the 'last' variable get all frames processed by RIFE calls (or even sum of 'last' and RIFE) so Average() plugin not work because clips number of frames not equal:

It seems that fps/num_frames are from the rife processing and not from the postprocessing (selectodd is used). I'll take a look later.

The scene change detect engine looks like need significant enchancement - with default sc_threshold of 0.12 it looks still not detect at all and with 0.005 it start to skip RIFE process and things become soft. May be in current build scene detect engine not adapted for 'high-tr' ? When scene change detected in the +-tr interval (or between -tr and +tr frames pair ?) - do it return 'current/N' frame or what ? I think if scene change detected in the +-tr interval - for denoising it may be 'fail safe' to return current frame (in denoise_blending_mode=1). It will decrease denoising activity near scene change timestampts but will not cause possible bad blends or bad interpolation between scenes.

Try the attached version. sc and skip should be fine now for denoise_tr=1. Also the issue about fps/num_frames should be fixed. Added back parameter denoise (default false).
RIFE_denoise2.zip

Do I need to enable 'skip' param to get better avoiding interpolation on scene changes ? It require addtional VMAF2 plugin ? Or it is not used if 'denoise=true' ? My content typically do not have duplicated frames so either all valid frames need RIFE or if scene-change detected - simply return (or put to Average) current frame. Or VMAF2 plugin can help to scene detection if 'skip' enabled ?
As I see from last commit - https://github.com/Asd-g/AviSynthPlus-RIFE/blob/e714d8fa8ebc7a2d38bd729a642a86370dfb3261/src/plugin.cpp#L383 it is something about VMAF2 plugin and getting psnrY for scene detection ? Also do it mean new fixed scene detection only works with +-1 tr and not (yet) work for tr > 1 ?

After some testing: This version have enhancement in scene change detection before scene change but with tr=1 and internal blending still produce 1 error-blended frame right after scene change (also with lowered sc_threshold).
So string
RIFE(denoise=true, denoise_tr=1, denoise_blending_mode=0, skip=true, sc=true, sc_threshold=0.1)
looks like still not detect some scenechange example and produce 2 error-blended frames  - 1 before and 1 after scene change.
At the same scenechange string
RIFE(denoise=true, denoise_tr=1, denoise_blending_mode=0, skip=true, sc=true, sc_threshold=0.01)
looks like detect scenechange OK but produce 1 error-blended (interpolated from 2 scenes) frame after scenechange. May be still some issue remains in the handling scenechange algorithm ?

Do I need to enable 'skip' param to get better avoiding interpolation on scene changes ? It require addtional VMAF2 plugin ? Or it is not used if 'denoise=true' ? My content typically do not have duplicated frames so either all valid frames need RIFE or if scene-change detected - simply return (or put to Average) current frame. Or VMAF2 plugin can help to scene detection if 'skip' enabled ?

skip is opposite to sc. Frames with no motion, almost identical will be not be interpolated. VMAF2 is required for skip.

As I see from last commit - https://github.com/Asd-g/AviSynthPlus-RIFE/blob/e714d8fa8ebc7a2d38bd729a642a86370dfb3261/src/plugin.cpp#L383 it is something about VMAF2 plugin and getting psnrY for scene detection ? Also do it mean new fixed scene detection only works with +-1 tr and not (yet) work for tr > 1 ?

Yes, this line is only for VMAF2/psnrY.
Try the attached version. sc should be ok now and it works for any denoise_tr.
RIFE_denoise3.zip

It looks version 3 works as expected now. Tested with some scenechanges and now all frames good - before and after scenechange. I also try to ask RIFE developers about more dedicated mode of motion compensation for denoise.

I'm considering to remove parameter denoise_blending_mode. Output will be only interpolated frames when denoise=true and the user can use whatever he wants for the further filtering. Currently denoise_blending_mode=0 is exactly denoise_blending_mode=1 and calling filter Average. Yes, Average can be implemented in order to avoid the call of the external plugin but the impact (memory/speed) will be minimal.
Also implemented skip=true for any denoise_tr.

I agree - Average() is the simplest linear blending engine possible and return the lowest quality. So may be in most use cases the users will use much more advanced external blending engines so the 'denoise_blending_mode=0' with internal call to external Average() plugin may be removed.

Can not found direct way to contact via e-mail. It is a VS-to-AVSport request of the also hardware-accelerated plugin for motion compensation using NVIDIA Optical Flow (though looks like limited to NVIDIA hardware only). I found some in-development plugin for Vapoursynth only about NVIDIA optical flow - https://bitbucket.org/mystery_keeper/vapoursynth-nvof/src/master/
It is not very old and have some activity in 2022. Can you port it to Avisynth so we can test it in temporal denoising too ?
Though in readme.txt it is still marked as not very production-ready. May be quality is still not very good. But quality may be mostly depend on NVIDIA progress in Optical Flow implementation in hardware and drivers.

I don't have NVIDIA GPU. I prefer to not touch such plugins/code that I cannot test.


Making it work / getting started

Hello,
I've made a few prototypes to test out TriPoint circuits on TriBase and TriTag boards. The PCB's arrived mounted well and I'm able to program nRF51 and STM but afterwards it seems that there's an issue with getting the whole device to work... The only thing I can do is connect via BLE (so nRF51 works ok) but STM + DW1000 seem to not be doing anything...
Do you maybe have any tips on what to check? I'm attaching the activity on I2C and SPI lines from the device startup - as you see the I2C commands don't seem right but the FW tested is

Tried with tridev and check_version.py scripts but it also fails even though the device enumerates correctly in dmesg.
Traceback (most recent call last):
  File "check_version.py", line 8, in <module>
    tp = tripoint.TriPoint()
  File "/../lab11-polypoint/tripoint/tripoint.py", line 28, in __init__
    self.tripoint = mpsse.MPSSE(mpsse.I2C, mpsse.FOUR_HUNDRED_KHZ)
  File "/usr/local/lib/python2.7/dist-packages/mpsse.py", line 70, in __init__
    raise Exception, self.ErrorString()
Exception: device not found

Sorry for the slow response, I've been traveling a bunch lately and am way behind.
First thought is that the chip is really sensitive to the power supply, we never got things working with a switching power supply, so if whatever you're using to power the board is at all noisy, that could be causing problems.
As for first debugging steps, is the STM executing any code, i.e. if you probe the GPIO lines to the DW, do you see these toggles: https://github.com/lab11/polypoint/blob/master/software/firmware/main.c#L274 ?


Update README.md

Thank you for your contribution! 🙌
To get it merged faster, kindly review the checklist below:
Pull Request Checklist

[ ] Reviewed the Contributing Guidelines

Especially the How to Contribute section

[ ] No merge commits
[ ] Correct commit message style

Coverage remained the same at 94.759% when pulling 2bd548a4a28e1c10694874075d14f3faac216902 on btbernard:master into 6015d7c9be11c42ab6b605cbda4b2c1ac7afe919 on SAP:master.


Update tests for various commands

Codecov Report

Merging #77 (665b0b2) into master (57a3aa6) will increase coverage by 3.94%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master      #77      +/-   ##
============================================
+ Coverage     61.34%   65.28%   +3.94%
- Complexity      582      620      +38
============================================
  Files           117      117
  Lines          2103     2103
  Branches        222      222
============================================
+ Hits           1290     1373      +83
+ Misses          746      682      -64
+ Partials         67       48      -19

Impacted Files
Coverage Δ

...ain/java/seedu/address/model/application/Date.java
83.33% <0.00%> (+8.33%)
:arrow_up:

...in/java/seedu/address/model/application/Email.java
80.00% <0.00%> (+10.00%)
:arrow_up:

.../java/seedu/address/model/application/Company.java
80.00% <0.00%> (+10.00%)
:arrow_up:

.../java/seedu/address/model/application/Contact.java
80.00% <0.00%> (+10.00%)
:arrow_up:

...rc/main/java/seedu/address/model/ModelManager.java
100.00% <0.00%> (+10.63%)
:arrow_up:

...main/java/seedu/address/model/ApplicationBook.java
91.66% <0.00%> (+12.50%)
:arrow_up:

...a/seedu/address/model/application/Application.java
80.00% <0.00%> (+27.50%)
:arrow_up:

...java/seedu/address/logic/commands/EditCommand.java
97.01% <0.00%> (+47.76%)
:arrow_up:

...java/seedu/address/logic/commands/FindCommand.java
90.90% <0.00%> (+54.54%)
:arrow_up:

.../java/seedu/address/logic/commands/AddCommand.java
100.00% <0.00%> (+58.33%)
:arrow_up:

... and 3 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


[Bug][rubysrc2cpg] - support for ensure is missing

Describe the bug
support for ensure is missing
To Reproduce
Sample code snippet to reproduce the error
  def refund
    @charge.refund!
  rescue Pay::Error => e
    flash[:alert] = e.message
  ensure
    redirect_to paddle_charge_path(@charge)
  end

Refer the file where I got this error - https://github.com/pay-rails/pay/blob/master/test/dummy/app/controllers/paddle/charges_controller.rb

@xavierpinho tagging for parser error


Relocating to avoid conflict.

kryo-saster averted. :)

Also, for posterity, this is donated without the requirement of attribution in the final PR.


bug: Compatibility Issue with Cloudflare Workers Global Scope Initialization

Description:
While attempting to use hono-pino for logging in a Cloudflare Workers environment, I encountered the following error due to asynchronous operations being triggered within the global scope:
在 Cloudflare Workers 环境中使用 hono-pino 作为日志工具时，由于异步操作在全局作用域被触发，遇到了以下错误：
Uncaught Error: Disallowed operation called within global scope. Asynchronous I/O (ex: fetch() or connect()), setting a timeout, and generating random values are not allowed within global scope.

The error arises when trying to use pinoLogger directly in the global scope, as shown below:
这个错误是因为 pinoLogger 在全局作用域中调用时触发的。以下代码会导致此错误：
import { pinoLogger as logger } from 'hono-pino'

const app = new Hono<{ Bindings: Bindings }>()  // Bindings is Cloudflare Worker's Enviroment Variables

app.use(logger({
    pino: {
      level: 'debug',
    },
  }))

The issue is likely caused by asynchronous I/O operations or initialization within hono-pino or its dependencies. Cloudflare Workers restricts asynchronous operations within the global scope, and currently, hono-pino doesn’t handle this requirement.
Suggested Solution:
Consider refactoring hono-pino to delay or defer any asynchronous operations until the first request handler. Alternatively, providing an option to conditionally initialize the logger within a handler function could allow for compatibility with environments like Cloudflare Workers.
是否可以考虑优化 hono-pino 的异步操作，比如延迟异步操作的执行，或者提供一个选项将日志初始化放到第一个请求的 Handler 中？这样可以避免在全局作用域直接触发异步操作。
Environment:

hono-pino version: [0.5.0]
pino version: [9.5.0]
Hono version: [4.6.8]
Yarn version: [4.5.1]
Cloudflare Workers environment

Thank you for looking into this compatibility issue!
非常感谢提供这样的 package，我用了 Hono 官网的这个教程初始化的app，如果大佬需要复现可以直接初始化一个环境然后在这里面使用上面的代码进行测试。要提供更多的信息的话欢迎随时联系我！

這很奇怪, 這個package沒有使用異步初始化
我檢查了依賴包也沒有
從零建立新CF worker環境dev、deploy都沒問題

光從你的敘述看起來, 建議你檢查一下入口檔案是否正確export default app
或是可以提供最小實現嗎?

https://github.com/maou-shonen/hono-pino/releases/tag/v0.6.0 這個版本修復了一個會在cloudflare workers上發生錯誤的問題
你可查看這個修復是否適用於你嗎?

感謝，我晚一些試一下


bug/FPHS-290

Added auto capitalization for Other option feature that is defaulted to OFF

looks good


Iteration over files and Pandas dataframe

Hi,
I need to iterate over a number of files. In the examples, I do not understand if there's an easy way to convert the "chain" into a Pandas data frame. I may find a workaround but that may be a useful feature nonetheless.
Cheers,
Riccardo

That's a good idea. I put such a feature into release 2.8.4. The release notes have a few examples.
Instead of making a new DataFrame iteration method, I added pandas.DataFrame as a possible outputtype to all the existing methods. When building up an analysis, you can start with an iteration that yields dicts of Numpy arrays and then graduate to DataFrames, if need be.
The only problem with doing it this way is that methods to make lazy arrays also have an outputtype parameter (of the same meaning), and Pandas can't store lazy arrays. Dask can, and maybe that's the appropriate way to go with that.
It also leaves open the question of whether we need a tree.pandas.df() function, since the equivalent functionality is now provided by tree.arrays(outputtype=pandas.DataFrame). Currently the old form has an optimization that the new form doesn't: it fills data directly from ROOT into Pandas without intermediate Numpy arrays. Nah, it's not good to have different calling conventions yield different behaviors... I need to make them the same...

I did the things mentioned above and called it release 2.8.5.
No matter how you ask for Pandas DataFrames, all applicable optimizations will be applied, and tree.pandas.df(**args) is now just a thin wrapper around tree.arrays(outputtype=pandas.DataFrame, **args).


(Partially) Fixes: Erroneous listing of [BufExplorer] in buffer list

This pull request partially fixes issue #28. Instead of always calling BufExplorer() - to ensure the buffer list is active - before calling s:Close(), I put a check in the ToggleBufExplorer() function, based on the active window's current buffer name.
This should greatly reduce the frequency of the bad behavior described in the issue, but it can still be made to happen if you jump through the right hoops. One such scenario is:
:BufExplorer<CR>
:BufExplorer<CR>
q
:BufExplorer<CR>

It is the execute "drop" name statement in BufExplorer() that appears to be the issue, causing the [BufExplorer] buffer to be marked as listed.

After this pull request was merged I am unable to toggle the BufExplorer window anymore. After calling ToggleBufExplorer for the first time BufExplorer window is opened as expected, but calling it again won't close the BufExplorer window. BufExplorer window toggle worked properly before.
Please fix.

:ToggleBufExplorer was changed to take you back into a [BufExplorer] window if it is already open, much like the :BufExplorer command does. If you are already in the [BufExplorer] window, the :ToggleBufExplorer command will close the buffer.

No, now it is not closing the BufExplorer once it is opened.
Steps to reproduce:

Open .vimrc
Call ToggleBufExplorer. BufExplorer window is opened.
Call ToggleBufExplorer. Nothing happens.

Obviously, you've broke stuff.
In BufExplorer window :echo bufname(winbufnr(0)) returns [BufExplorer] and you compare it to s:BufExplorerName() that does escaping on non-Windows and returns \[BufExplorer\]. This is why ToggleBufExplorer is not working on non-Windows anymore. Also, you've changed comparison in s:ShouldIgnore in the same fashion, so I assume it is broken as well now.
Please fix.

OK. Thanks for the additional info. I will fire up a Linux VM and work out the kinks.


Add Buildpack source to cloud support + Patch commands

Enable the local source to cloud + patching in Container Apps. Building and Patching is done using buildpack. Building is integrated with the Up command, and patching introduces three new CLI commands (cf. below).

This checklist is used to make sure that common guidelines for a pull request are followed.
Related command
az containerapp up
az containerapp patch list
az containerapp patch apply
az containerapp patch interactive
General Guidelines

[x] Have you run azdev style <YOUR_EXT> locally? (pip install azdev required)
=> We have a couple of linting issues we're working on and will send an extra commit for soon, but given the short timeline we decided to send that as a second step.

About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify src/index.json.

Thank you for your contribution! We will review the pull request and get back to you soon.

@daniv-msft Please resolve the conflict

@daniv-msft Please resolve the conflict

@zhoxing-ms Could you please clarify which conflict you're referring to? @snehapar9 resolved a merge conflict with the Workload Profile PR just today. Was it the one you were thinking about?


run order

what is it the order execute of dom - cleanUp -and Mount in a render hook

what sequence of them ?

It’s not clear what you’re asking here, sorry! Do you still need assistance for this?


Thank you very much!

Thank you very much for writing this Firefox plugin! I totally use it and love it more and more. Just wanted to tell you somehow :)

@akavel thanks a lot for your warm comment. It motivates me a lot. Thanks again!


should not trigger pipeline on an template file

Pipeline generation tool found the ci.yml file under “sdk/template/.content/testResource/ci.yml” so it created this pipeline for it. Actually the “sdk/template/.content/testResource/ci.yml”  is just template file and it is not related to an resource, so it should not trigger a pipeline.

#29122

Thanks for fixing. I deleted the pipelines that were created for this.


Adding initial PLIO support to the platform

This is linked to the mlir-aie PR that adds PLIO support to the compiler and has an example that targets this platform.

Are you really still using 2022.1?  updating to 2023.2 would be good, which is where most of the other work is, although I expect we'll move to 2024.2 once it's released.

Are you really still using 2022.1? updating to 2023.2 would be good, which is where most of the other work is, although I expect we'll move to 2024.2 once it's released.

It is a good point, I made an issue to track this: https://github.com/Xilinx/ROCm-air-platforms/issues/14. Note that this version is solely for building the platform, so this does not preclude using 2023.2 or whatever newer version for mlir-aie (2023.2 is what I currently use), only that you need 2022.1 to build the platform. That being said, I get that now that we are opening up the platform for users to integrate their PL designs it makes sense to update the Vitis version for the platform as well.


Merging master into live

Docs Build status updates of commit 5f096ed:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/windows-vision-skills/index.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Official TypeScript support

Is there any interest in officially supporting TypeScript in create-react-app? I think having to fork the repo to add support is a bit unfortunate.

There is also https://github.com/timarney/react-app-rewired but i'm not sure how far that will get you compared to react-scripts-ts.

TypeScript support is only ~254 lines of code. Rebasing is painful.

https://github.com/wmonk/create-react-app-typescript/pull/183

Make a PR to  react-app-rewire-typescript to add ts-jest or write your own react-app-rewire plugin, its easy.
imo typescript doesnt belong in CRA.

Hi there! We're not quite ready to consider this yet but you can keep track of our Plugin System (https://github.com/facebookincubator/create-react-app/pull/2784) and TypeScript Plugin (https://github.com/facebookincubator/create-react-app/pull/2815).

Make a PR to react-app-rewire-typescript to add ts-jest or write your own react-app-rewire plugin, its easy.

I don't think react-app-rewire is the answer. There are known issues with that approach.
The Plugin System looks amazing. Hopefully that gets merged soon.

TypeScript is now officially supported as of Create React App 2.1. Read the release notes to get started!


Functional tests: disabled on flavor=usr-from-host & host systemd < v220

When the host does not have systemd >= v220, we cannot use stage1 built
with flavor=usr-from-host.

LGTM


sqlserver.database.state not sent for all databases

What does this PR do?
The metric sqlserver.database.state wasn't being sent for all the databases in the instance. The choice of databases to which the metric will be sent is pseudorandom - it depends on the customer's configuration. The technical reason is that the sqlserver.database.state is sent only for the database to which the agent session is currently connected to when the sysdatabases is read. As the metric should be sent for all the databases, I removed that condition.
In addition, I created a unit test that checks if the metric is emitted for msdb. msdb is used as a proxy for all other databases, as with the default configuration the session emits the metric only for master.
Last but not least, some of the tests were failing on EC2. This problem isn't related to the aforementioned bug. The reason for the failed tests is that newer versions of the ODBC/TDS drivers were generating slightly different error messages which weren't matched with the existing regular expressions in tests. I adapted the regular expressions so that they are working correctly also with the new versions of the drivers.
Motivation
The problem was noticed thanks to the following escalation: https://datadoghq.atlassian.net/browse/SDBM-302.
Additional Notes

Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] PR title must be written as a CHANGELOG entry (see why)
[ ] Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
[ ] PR must have changelog/ and integration/ labels attached
[ ] If the PR doesn't need to be tested during QA, please add a qa/skip-qa label.

Last but not least, some of the tests were failing on EC2. This problem isn't related to the aforementioned bug. The reason for the failed tests is that newer versions of the ODBC/TDS drivers were generating slightly different error messages which weren't matched with the existing regular expressions in tests. I adapted the regular expressions so that they are working correctly also with the new versions of the drivers.

ugh yeah, these tests are so brittle. I would like to implement them in a different way, but haven't had time to dig into it


[Feature] ability to edit 3D meshes

Ability to edit 3D meshes
We are researching 3D editors for WebGL that allow users to create and edit 3D meshes. E.g. we would like to be able to do things like move edges/verticies, divide faces, etc.
After adding a default cube, how do I edit its mesh?

Hi
Currently nunuStudio does not support proper 3D modeling.
Its possible to build models from primitives and CSG (adding, subtracting or intersecting models)
Thansk a lot!

Kool beans. It would be really nice to have an open source WebGL 3D editor. I also asked this question in the WebGLStudio.js repo, as that and nunuStudio seem to be the best open source WebGL 3D editing projects currently.

Neat, right now clara.io or the web version of sketchup (https://app.sketchup.com/app) are possible the best options for modeling right now.
There are a ton of CAD 3d editors also. nunuStudio is more focused on the programming part of the thing.
I would actually be fairly easy to implement some basic modeling since threejs already provides somewhat flexible structure and a ton of shape drawing / extrusion tools that could be used.
I might be a plan for future versions!
Thanks a lot!

For 3D modellnig, I'd rather use Blender for 3d
I believe, nunuStudio is best as it is, not as 3D modeller/animator.
I treat Nunustudio as my cooking pot. I prepare my cooking ingredients outside nunu, then once I chopped some polygons and marinated my texture maps, I bring everything into nunu and cook them together.

Neat, right now clara.io or the web version of sketchup (https://app.sketchup.com/app) are possible the best options for modeling right now.
There are a ton of CAD 3d editors also. nunuStudio is more focused on the programming part of the thing.

I agreed! After all if brylie wants free and open source modeling/animator/rendering/sculping/ unwarping and much more 3D editor there is always Blender. I belive that nunuStudio must stay focus to its initial  goals; a very good programming and software delivery 3D IDE.

To clarify, the request was for basic mesh editing. Like making simple building grimetries to be shown on an interactive 3D map, e.g. for city planning.
I agree Blender is excellent.

You can use boxes and make your city. nunuStudio has geometry shapes.

Hi
@sloothes Was @brylie requested was a more refined mesh editing solution, the ability do manioulate vertexes individually build triangles, have some more complex tools for surface creation.
The modeling abilities in nunuStudio are really basic and not enough for more than simple prototypes. Since its not the main focus of the tool i have not implemented it yet. But it would be a welcome feature.


KAFKA-6592: Follow-up

Do not require ConsoleConsumer to specify inner serde as s special property, but just a normal property of the message formatter.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

retest this please


I tried looking through the README.md on the GitHub page to see if there is a setting I needed to configure to see the parser suggestions, but I couldn't find anything I had to configure for it to show up. However, when I do use the command fuck, it does provide a suggestion then... It just appears to be different from the .gifs on the documentation.
  That's quite expected. Those GIFs were created a while ago, and since then not updated.

Sorry, I'm not sure I understand. Could you please try to describe your issue more clearly?

Sure! When a mistake is made while thefuck is running on my terminal, it is not pointed out with a suggested correction for the typo.

Ok. Could you please provide steps to reproduce your issue?   Wha else would you expect?

It's missing the "Did you mean..."

It never shows it. That's part of the error message.

I guess you were right about the gif then - about it not being up date.

Hey! I guess I got what you mean. That "Did you mean" thing comes from the command-not-found package on Ubuntu. You can activate that by just installing it with apt install -y command-not-found.

Ohhh!!!! Thank you so much for the info! This should totally be an optional setting for the code.


spark3.3.3 java exception thrown at native-engine/blaze/src/rt.rs:80

Describe the bug
A clear and concise description of what the bug is.

Caused by: java.lang.RuntimeException: called Result::unwrap() on an Err value: External("Java exception thrown at native-engine/blaze/src/rt.rs:80: org.apache.arrow.c.Data.importSchema(Lblaze/org/apache/arrow/memory/BufferAllocator;Lorg/apache/arrow/c/ArrowSchema;Lorg/apache/arrow/c/CDataDictionaryProvider;)Lblaze/org/apache/arrow/vector/types/pojo/Schema;")
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Isn't this compatible with centos7？

could you check the executor log for detailed error message? blaze has no special dependences to operating system. centos7 should be OK.

I have the same problem ,How is this problem solved ?  executor log has no other useful infomation ,Only   like this :
2024-09-23 18:50:23.483 INFO - 18:50:23.479 [task-result-getter-1] ERROR org.apache.spark.scheduler.TaskSetManager - Task 0 in stage 1.0 failed 4 times; aborting job
2024-09-23 18:50:23.773 INFO - org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 19) (117-116-13 executor 2): java.lang.RuntimeException: called Result::unwrap() on an Err value: External("Java exception thrown at native-engine/blaze/src/rt.rs:80: java.lang.NoSuchMethodError: org.apache.arrow.c.Data.importSchema(Lblaze/org/apache/arrow/memory/BufferAllocator;Lorg/apache/arrow/c/ArrowSchema;Lorg/apache/arrow/c/CDataDictionaryProvider;)Lblaze/org/apache/arrow/vector/types/pojo/Schema;")
2024-09-23 18:50:23.773 INFO - at org.apache.spark.sql.blaze.JniBridge.callNative(Native Method)
2024-09-23 18:50:23.773 INFO - at org.apache.spark.sql.blaze.BlazeCallNativeWrapper.(BlazeCallNativeWrapper.scala:66)
2024-09-23 18:50:23.773 INFO - at org.apache.spark.sql.blaze.NativeHelper$.executeNativePlan(NativeHelper.scala:89)
2024-09-23 18:50:23.773 INFO - at org.apache.spark.sql.execution.blaze.shuffle.BlazeShuffleWriterBase.nativeShuffleWrite(BlazeShuffleWriterBase.scala:80)
2024-09-23 18:50:23.773 INFO - at org.apache.spark.sql.execution.blaze.plan.NativeShuffleExchangeExec$$anon$1.write(NativeShuffleExchangeExec.scala:143)
2024-09-23 18:50:23.773 INFO - at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
2024-09-23 18:50:23.773 INFO - at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
2024-09-23 18:50:23.773 INFO - at org.apache.spark.scheduler.Task.run(Task.scala:136)
2024-09-23 18:50:23.774 INFO - at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
2024-09-23 18:50:23.774 INFO - at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
2024-09-23 18:50:23.774 INFO - at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
2024-09-23 18:50:23.774 INFO - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2024-09-23 18:50:23.774 INFO - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2024-09-23 18:50:23.774 INFO - at java.lang.Thread.run(Thread.java:748)
I compiled it in aarch64 machine which system page size is 64k.


gl crashes in repo with no commits

user@host /tmp/gl-core-test0l6rlbpr $ git init
Initialized empty Git repository in /tmp/gl-core-test0l6rlbpr/.git/

user@host /tmp/gl-core-test0l6rlbpr $ gl status
Traceback (most recent call last):
  File "/usr/local/bin/gl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/gitless/cli/gl.py", line 97, in main
    parser = build_parser(sub_cmds, repo)
  File "/usr/local/lib/python3.7/dist-packages/gitless/cli/gl.py", line 79, in build_parser
    sub_cmd.parser(subparsers, repo)
  File "/usr/local/lib/python3.7/dist-packages/gitless/cli/file_cmd.py", line 19, in f
    skip_dir_test=repo and repo.current_branch.path_is_ignored,
  File "/usr/local/lib/python3.7/dist-packages/gitless/core.py", line 222, in current_branch
    b = self.git_repo.head
_pygit2.GitError: reference 'refs/heads/master' not found

For someone who comes from git to gl it's easy to keep typing git init instead of gl init
There's a number of assumptions in the code that a branch is available and a head is present.


Cannot use functools.partial as title

I was surprised to see that you can pass normal functions and lambas to the title attribute,but partial functions (functools.partial) aren't accepted:
#!/usr/bin/env python
# coding: utf-8

import functools

def show_cmd(extra, task):
    return '{}: executing... {}'.format(extra, task.name)

def task_lambda_ok():
    return {
        'actions':['echo lambda'],
        'title': lambda task: show_cmd('LAMBDA', task)
    }

def task_partial_ko():
    title = functools.partial(show_cmd, extra='PARTIAL')
    return {
        'actions':['echo partial'],
        'title': title,
    }

gives
% doit list
ERROR: Task 'partial_ko' attribute 'title' must be {function, None}
got:functools.partial(<function show_cmd at 0x7f26847466a8>, extra='PARTIAL')
<class 'functools.partial'>

Is that on purpose or is it an oversight?
Seems prevented by https://github.com/pydoit/doit/blob/master/doit/task.py#L156, so it probably could be easily fixed by using collections.abc.Callable instead of types.FunctionType (but maybe I'm missing something).

Is that on purpose or is it an oversight?

You are right. Oversight again. Trivial fix as you pointed out..


Facing errors when trying to install (An error was encountered while executing action link)

Hello, so I am trying to configure dotbot wrote a yaml config faced a similar error as #287 I think its the indentation but I have tried to fix it but still doesn't work.

---
# Dotbot install config file for my dotfiles.

- defaults:
  link:
    create: true
    relink: true

- clean: ["~", "~/.ssh"]

- shell:
    - [
        test -f $HOME/.bashrc && mv $HOME/.bashrc $HOME/.bashrc_"$(date)".bak,
        Backing up default bashrc if it is present,
      ]
    - [
        if test -f $HOME/.bash_profile; then mv $HOME/.bash_profile $HOME/.bash_profile_"$(date)".bak; elif test -f $HOME/.profile; then mv $HOME/.profile $HOME/.profile_"$(date).bak"; fi,
        Backing up bash_profile & profile if either/both is present,
      ]

- create:
    - ~/.vim
    - ~/tools/{scripts, go}

- link:
  ~/.profile: dotfiles/bash/profile
  ~/.bashrc: dotfiles/bash/bashrc
  ~/.aliases: dotfiles/bash/aliases
  ~/.exports: dotfiles/bash/exports
  ~/.randline.py: dotfiles/eddie/randline.py
  ~/.greetings.txt: dotfiles/eddie/greetings.txt
  ~/.tmux.conf: dotfiles/tmux.conf
  ~/.vimrc: dotfiles/vim/vimrc
  ~/.vim/ycm_extra_conf.py: dotfiles/vim/ycm_extra_conf.py

Kindly assist.

Apologies, we need to figure out how to give better error messages for situations like this.
The link: needs to be underneath defaults:, and similarly, the items under the final link: need to be indented.
Try this (also attached as a file, if that's easier):
# Dotbot install config file for my dotfiles.

- defaults:
    link:
      create: true
      relink: true

- clean: ["~", "~/.ssh"]

- shell:
    - [
        test -f $HOME/.bashrc && mv $HOME/.bashrc $HOME/.bashrc_"$(date)".bak,
        Backing up default bashrc if it is present,
      ]
    - [
        if test -f $HOME/.bash_profile; then mv $HOME/.bash_profile $HOME/.bash_profile_"$(date)".bak; elif test -f $HOME/.profile; then mv $HOME/.profile $HOME/.profile_"$(date).bak"; fi,
        Backing up bash_profile & profile if either/both is present,
      ]

- create:
    - ~/.vim
    - ~/tools/{scripts, go}

- link:
    ~/.profile: dotfiles/bash/profile
    ~/.bashrc: dotfiles/bash/bashrc
    ~/.aliases: dotfiles/bash/aliases
    ~/.exports: dotfiles/bash/exports
    ~/.randline.py: dotfiles/eddie/randline.py
    ~/.greetings.txt: dotfiles/eddie/greetings.txt
    ~/.tmux.conf: dotfiles/tmux.conf
    ~/.vimrc: dotfiles/vim/vimrc
    ~/.vim/ycm_extra_conf.py: dotfiles/vim/ycm_extra_conf.py

install.conf.yaml.zip

Okay, now that fixed it... thanks a lot... one last question this line for create - ~/tools/{scripts, go} I want it to create two subfolders under tools, scripts and go folder but it doesn't work is there a way of achieving this using seq

No, they need to be listed separately:
- create:
    - ~/tools/scripts
    - ~/tools/go

Okay, thank you so much.


Add license

Description
Add license to the files.
Type of change

Documentation update

Testing steps
Checklist

[ ] make before_commit passes: no, due to code coverage.
[x] updated documentation wherever necessary
[x] added or modified tests if necessary
[x] updated schemas and validators in insights-data-schemas in case of input/output change

@tisnik PTAL now. I ignored the docs/ folder.


Sample project

Do you have a working sample project? Thanks

what kind of project do you mean?
do not hesitate to let we know,
join us on telegram
https://t.me/reactnative_id

as @infinitey say, It could be useful to read the code of a sample application.
I've got this error calling the function pay

TypeError: Cannot read property 'checkOut' of undefined

This is my code (just your code with some little modifications)
 import PaymentGateway from 'react-native-payment-gateway';

 export class InfoScreen extends React.Component {

 async pay(){

           const optionConect = {
               clientKey: "your client key",
               urlMerchant: "https://domain.net/" //-- will hit https://domain.net/charge
           }

           const transRequest = {
               transactionId: "0001",
               totalAmount: 4000
           }

           const itemDetails = [
               {id: "001", price: 1000, qty: 4, name: "peanuts"}
           ];

           const creditCardOptions = {
               saveCard: false,
               saveToken: false,
               paymentMode: "Normal",
               secure: false
           };

           const userDetail = {
               fullName: "jhon",
               email<EMAIL_ADDRESS>               phoneNumber: "0850000000",
               userId: "U01",
               address: "street coffee",
               city: "yogyakarta",
               country: "IDN", //-- must be standard country code
               zipCode: "59382"
           };

           const optionColorTheme = {
               primary: '#c51f1f',
               primaryDark: '#1a4794',
               secondary: '#1fce38'
           }

           const optionFont = {
               defaultText: "open_sans_regular.ttf",
               semiBoldText: "open_sans_semibold.ttf",
               boldText: "open_sans_bold.ttf"
           }

           const callback = (res) ={
               console.log(res)
           };

 console.log(PaymentGateway);

           PaymentGateway.checkOut(
               optionConect,
               transRequest,
               itemDetails,
               creditCardOptions,
               userDetail,
               optionColorTheme,
               optionFont,
               callback
           );
       }

 [...cut cut cut ...]

   render() {
       return (
         \<ScrollView key="ScrollViewPrincipale" \>
           \<Button
             key="KEYPAGO"
             icon={{name: 'camera'}}
             rounded={false}
             title="E io pago!"
             onPress={() ={
               this.pay()
               .then( (ok) =console.log(ok) )
               .catch( (e) =console.log(e) )
             } }
             backgroundColor="green"
             containerViewStyle={styles.marginBig}
           /\>
         \</ScrollView\>
     };
   }

as you can see, the console.log writes "undefined"
can you help?
thanks

I got the same problem.
Looks like the problem is in index.js of this library. MidtransModule in line 14 might detected as undefined.

solve on version 0.4.3


Better support for string-valued dates

When using JS/TS Date to represent a specific moment in time (in UTC), all is well. When using JS/TS Date to represent a year/month/day (e.g. a birth day), things get ambiguous. We should ensure that we can handle an ISO formatted date-only string and retain its stringiness.

We may already have this with IsDateString.


About coco related data

Hello,
I am a student on Master 1, doing a research internship on subject related to video captioning. I find your work on MEE very inspiring and is trying to test your model. But I encountered a problem when running train.py on MSRVTT:
these two files are missing:
coco_visual_path='data/X_train2014_resnet152.npy' ,coco_text_path='data/w2v_coco_train2014_1.npy'
can you add these two files to the public ftp server?
Thanks in advance

Hi, Thank you for reporting me these missing files !
I re-uploaded the data.zip file so it includes these files.
Sorry for the troubles !


added step-list core component

@GauriKudtarkar I think a merge may have gone wrong somewhere in this PR, because there are now thousands of changes all over the place. I think you may need to cancel this PR and open a new fresh branch from latest main, reapply your changes there, and open a new PR.


doctest_discover_tests - TEST_SUITE integration with CTest

Description
I'm grouping my tests using TEST_SUITEs and using doctest_discover_tests to register them with ctest.
I want to run only some group of tests using ctest, which can be done based on test name or labels.
Is there a way to automatically prepend the test suite to the test name when registering with ctest? Or adding it as a label?
I can obviously ignore the TEST_SUITE and manually prefix my TEST_CASEs with the suite name, but that seems easy to screw up.
I can also manually register each test suite using doctest_discover_tests with the PREFIX option (I think), but it would be nice to have some automatic once-off way. An additional flag to add suite names as labels/prefix maybe?
Is there an easy way to achieve this that I am missing?

From a brief glance I see that doctest_discover_tests uses --list-test-cases - doctest also has --list-test-suites. Or perhaps there needs to be a new option to extract the names of the test suites while listing the test case names... To be honest I have no clue how doctest_discover_tests works and I won't have the time to look into this anytime soon...
I guess it won't help but just as some extra information - you could set the test suite on a per-test-case-basis with a decorator.
Actually another option is to:

extend the --list-test-cases option at least for the xml reporter to also have them inside of test suites - the current xml output is like this but the test cases could be nested into their appropriate test suites
change the doctest_discover_tests cmake function to work with the xml output and parse that.

@RedDwarf69 introduced the ctest discovery so hopefully he could shed some light.

The xml reporter option looks like it would be the simplest.
The alternative is to implement a test-case + test-suite matcher from the outside. I'm not sure if it can be done without ambiguity?
Something like:

--list-test-suites
--list-test-cases for each test suite (enabling only the one test suite)
Disable all test suites to get the no-test-suite list

I don't know the workings of doctest well enough to know if this would have corner cases or not. Extending this to nested test suites would also be a pain.

What you described should work :)

I just extended the xml reporting when listing test cases - now each tag with the name of each test case as an attribute has 3 more things as attributes: filename, line and testsuite - that could be of relevance - see this issue: https://github.com/onqtam/doctest/issues/320


Modified MMM-Todoist.js for Todoist V8 API

Changed URL as follows: apiVersion: "v8", apiBase: "https://todoist.com/API"
Using .due rather than obsolete .due_date_utc.  Also using new Date format
Using .legacy_project_id for continuity

Is this PR working correctly or are the modifications of @simphide (https://github.com/cbrooker/MMM-Todoist/issues/50#issuecomment-528873586) needed?


Zib-EpisodeOfCare

Did a small fix, but cannot seem to find anything else. @ArdonToonstra maybe you can double check

EpisodeOfCare-Title (or has that now become EpisodeOfCareName?)
EpisodeOfCare-DateFirstEncounter in the nl-core profile
EpisodeOfCare-DateLastEncounter in the nl-core profile

Why? Where can we find these concepts in some dataset that will use zib2020-r4?
And even if we already know for sure that they will be applicable to zib2020-r4, we do not have any mapping guidelines for nl-core profiles.

Why? Where can we find these concepts in some dataset that will use zib2020-r4?
And even if we already know for sure that they will be applicable to zib2020-r4, we do not have any mapping guidelines for nl-core profiles.

Well, we have to put some effort into researching what those extensions are and what they are doing in the STU3 profiles. We can't just ditch everything not accounted for by the zib while we are processing the STU3 profiles. I am completely fine with not having them in the nl-core-profile but it should be a well-informed decision and stated in the release-notes.md.
I believe these extensions are added because of the GP use case, therefore, the most logical place to inform about these extensions would be Alexander. He can give us intel on whether these extensions will be needed or not.

I do not propose to ditch them, I suggest not adding them out of the blind as long as there isn't a 'zib2020-ready' dataset, which imo is a big difference between this use case and for example MP-920. If there will be a dataset at some point in the near future, we can add it then, including a clear mapping, because this is clearly lacking in STU3.  Process-wise I think this is better then copying everything that was present in STU3 without a proper mapping and I guess this is what we have been doing up until now (Patient.generalPractitioner for example)

But this probably all comes down to a lack of guidelines of what we actually should and want to do with nl-core, so maybe we should first focus on determining that

I agree that we should not blindly copying all that was present in STU3 and that we have somewhat lacking guidelines on what to do with the nl-core profiles. For these two extensions, I would still propose to discuss this with Alexander or GP product manager to notify them that we propose to not copy these extensions without a dataset and to state this in the release-notes.

@Ashwin-nictiz , I have created the ada2fhir mapping and noticed that guidance for the mandatory status element is missing. Could you work on that?

Handmatig gemerged.


Versioning and attribute value for :Conventions

Hi, simple question:
could we agree on some form of naming your simple geometry standard such that it can be referenced in the global :Conventions attribute of a NetCDF dataset? (at least for the time being, as long as this is not part of official CF).
For example, I'd like to refer to it as:
:Conventions = 'CF-1.6, UGRID-1.0, NCSG-0.1.0'
What are your thoughts on this?

I think that we are close enough on this over here: https://github.com/cf-convention/cf-conventions/pull/115 and CF-1.7 is so close to done, that this will be under CF-1.8 for sure. So I would say let's call this CF-1.8.


Client connections not properly closed?

We at robur (//cc @reynir) run a dream (alpha2) web application on the public Internet (behind a TLS reverse proxy) -- https://builds.robur.coop
Now, we observe after several days where lots of client have been served (and some have errored since the connection was preliminary closed/reset) there re lots of sockets owned by the dream process that are in the CLOSED state according to netstat.
May it be the case that there are code paths that do not call {Unix,Lwt_unix}.close on client sockets, and thus exhibit file descriptor leakages? May this be related to #118? While I suspect that cleanly terminating connection are doing the right thing [tm] (and call close()), may there be error cases where close is not called? I find the code rather hard to follow, but eventually you have an idea where the root cause is located?

I'll have to reproduce it. I'll check whether the Dream playground web server has the same issue.
As a note, Dream itself never closes sockets directly. This is done by the underlying web server, http/af. The issue could be either in http/af itself, or in how Dream interacts with it.

thanks for confirming. as a user of dream we also don't have to close anything specifically, or do we?
I suspect normal connections (curl etc.) are handled nicely, but maybe disappearing clients (or those not sending Connection: close but nevertheless closing the TCP connection) aren't gracefully closing the fd, eventually?

thanks for confirming. as a user of dream we also don't have to close anything specifically, or do we?

Ideally, Dream should take care of as many such things as it can automatically. I can't say much more without knowing which functions you're calling and how. But ideally, it shouldn't be easy to cause an fd leak.

I suspect normal connections (curl etc.) are handled nicely, but maybe disappearing clients (or those not sending Connection: close but nevertheless closing the TCP connection) aren't gracefully closing the fd, eventually?

I don't know at this point.
By the way, is this causing a problem besides the accumulation of fds? Is the process running out of them well before you might otherwise restart it, for example?

FWIW, a nginx server behind the same TLS reverse proxy does not have any sockets in state CLOSED.

I can't say much more without knowing which functions you're calling and how. But ideally, it shouldn't be easy to cause an fd leak.

Code is at https://github.com/roburio/builder-web :)
bin/builder_web_app.ml
    Dream.run ~port ~interface:host ~https:false
    @@ Dream.logger
    @@ Dream.sql_pool ("sqlite3:" ^ dbpath)
    @@ Http_status_metrics.handle
    @@ Builder_web.add_routes datadir
    @@ Dream.not_found

and inside the handlers (lib/builder_web.ml):
Dream.respond / Dream.redirect / Dream.empty

By the way, is this causing a problem besides the accumulation of fds? Is the process running out of them well before you might otherwise restart it, for example?

Yes. This is the issue.

Thanks. Do you know if the fds that remain open might be related to any events you observe in the logs?

I do not know that unfortunately. Maybe it would be a good idea to include the file descriptor number in the log output, so a correlation would be possible?


Relocation truncated to fit R_X86_64_32S failed

Despite invoking demo/hello is successful on sconified-worker, invoking other application (for example, demo/c_example) produces this error :
...
[12:15:00] [50] [T] Getting S3 key faasm/demo/c_example/function.aot as bytes
[12:15:00] [50] [T] Caching S3 key faasm/demo/c_example/function.aot at /usr/local/faasm/object/demo/c_example/function.aot
[12:15:00] [50] [T] WAMR loading 57636 wasm bytes

[12:15:00] [50] [E] Failed to load WAMR module:
AOT module load failed: relocation truncated to fit R_X86_64_32S failed. Try using wamrc with --size-level=1 option.
terminate called after throwing an instance of 'std::runtime_error'
  what():  Failed to load WAMR module
Caught stack backtrace:
[SCONE|DEBUG] src/enclave/enclave.c:1381:_enclave_terminate(): FSPF Synchronization Point: Persisting FSPF data before terminating enclave due to unhandled signal 6
...

The message comes from ./core/iwasm/aot/arch/aot_reloc_x86_64.c, line 198 : https://github.com/faasm/wasm-micro-runtime/blob/5e9dc3c7eb33167389d99b7e5851dc55b5911d33/core/iwasm/aot/arch/aot_reloc_x86_64.c#L198
Question : what is "relocation" mean in this case? Why we want to  relocate to 32-bit?
Pinging @csegarragonz since you might know something.

Commit https://github.com/ardhipoetra/faasm/commit/4d185f43407ad02f1dc41d3f4b031f715960dc07 closes this issue. On SGX, size-level needs to 1 (?).
Source : https://github.com/bytecodealliance/wasm-micro-runtime/issues/2274


Video file edit

Video file can be open by ark-retouch
After video is opened we can see only thumbail
We can draw on it and save
After save we have .mp4 file with size of some kB, not usible

Fixed in #22