Synthesis failed for nodejs-phishing-protection

Hello! Autosynth couldn't regenerate nodejs-phishing-protection. :broken_heart:
Here's the output from running synth.py:
f0c2ca25e6cd4d4bebdd198"
DEBUG: Call stack for the definition of repository 'com_googleapis_gapic_generator_go' which is a http_archive (rule definition at /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/bazel_tools/tools/build_defs/repo/http.bzl:296:16):
 - <builtin>
 - /home/kbuilder/.cache/synthtool/googleapis/WORKSPACE:261:1
DEBUG: Rule 'gapic_generator_typescript' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "ca322b5e7b0d03b3cc44a90444e3a7f944c9ba3345f0505ee48c8e715d19dd95"
DEBUG: Call stack for the definition of repository 'gapic_generator_typescript' which is a http_archive (rule definition at /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/bazel_tools/tools/build_defs/repo/http.bzl:296:16):
 - <builtin>
 - /home/kbuilder/.cache/synthtool/googleapis/WORKSPACE:280:1
DEBUG: Rule 'gapic_generator_csharp' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "40ddae63d2729ef5ccbd8b60123327ea200ce9400d0629238193ff530dcaea18"
DEBUG: Call stack for the definition of repository 'gapic_generator_csharp' which is a http_archive (rule definition at /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/bazel_tools/tools/build_defs/repo/http.bzl:296:16):
 - <builtin>
 - /home/kbuilder/.cache/synthtool/googleapis/WORKSPACE:318:1
DEBUG: Rule 'bazel_skylib' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "1dde365491125a3db70731e25658dfdd3bc5dbdfd11b840b3e987ecf043c7ca0"
DEBUG: Call stack for the definition of repository 'bazel_skylib' which is a http_archive (rule definition at /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/bazel_tools/tools/build_defs/repo/http.bzl:296:16):
 - <builtin>
 - /home/kbuilder/.cache/synthtool/googleapis/WORKSPACE:35:1
Analyzing: target //google/cloud/phishingprotection/v1beta1:phishingprotection-v1beta1-nodejs (1 packages loaded, 0 targets configured)
Analyzing: target //google/cloud/phishingprotection/v1beta1:phishingprotection-v1beta1-nodejs (1 packages loaded, 2 targets configured)
INFO: Call stack for the definition of repository 'npm' which is a yarn_install (rule definition at /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl:411:16):
 - <builtin>
 - /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/build_bazel_rules_nodejs/index.bzl:87:5
 - /home/kbuilder/.cache/synthtool/googleapis/WORKSPACE:293:1
Analyzing: target //google/cloud/phishingprotection/v1beta1:phishingprotection-v1beta1-nodejs (1 packages loaded, 2 targets configured)
ERROR: An error occurred during the fetch of repository 'npm':
   yarn_install failed: yarn install v1.19.1
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
info If you think this is a bug, please open a bug report with the information provided in "/home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/gapic_generator_typescript/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
 (error An unexpected error occurred<EMAIL_ADDRESS>Request failed \"404 Not Found\"".
Process stalled
Active handles:
  - Socket
  - Socket
  - Socket
  - TLSSocket
  - TLSSocket
  - TLSSocket
  - TLSSocket
  - TLSSocket
  - TLSSocket
)
ERROR: /home/kbuilder/.cache/synthtool/googleapis/google/cloud/phishingprotection/v1beta1/BUILD.bazel:218:1: //google/cloud/phishingprotection/v1beta1:phishingprotection_nodejs_gapic depends on @gapic_generator_typescript//:protoc_plugin in repository @gapic_generator_typescript which failed to fetch. no such package '@npm//@bazel/typescript': yarn_install failed: yarn install v1.19.1
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
info If you think this is a bug, please open a bug report with the information provided in "/home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/gapic_generator_typescript/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
 (error An unexpected error occurred<EMAIL_ADDRESS>Request failed \"404 Not Found\"".
Process stalled
Active handles:
  - Socket
  - Socket
  - Socket
  - TLSSocket
  - TLSSocket
  - TLSSocket
  - TLSSocket
  - TLSSocket
  - TLSSocket
)
ERROR: Analysis of target '//google/cloud/phishingprotection/v1beta1:phishingprotection-v1beta1-nodejs' failed; build aborted: no such package '@npm//@bazel/typescript': yarn_install failed: yarn install v1.19.1
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
info If you think this is a bug, please open a bug report with the information provided in "/home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/gapic_generator_typescript/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
 (error An unexpected error occurred<EMAIL_ADDRESS>Request failed \"404 Not Found\"".
Process stalled
Active handles:
  - Socket
  - Socket
  - Socket
  - TLSSocket
  - TLSSocket
  - TLSSocket
  - TLSSocket
  - TLSSocket
  - TLSSocket
)
INFO: Elapsed time: 5.971s
INFO: 0 processes.

DEBUG:synthtool:Wrote metadata to synth.metadata.
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/github/synthtool/synthtool/__main__.py", line 102, in <module>
    main()
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/tmpfs/src/github/synthtool/synthtool/__main__.py", line 94, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/kbuilder/.cache/synthtool/nodejs-phishing-protection/synth.py", line 29, in <module>
    library = gapic.node_library('phishingprotection', version)
  File "/tmpfs/src/github/synthtool/synthtool/gcp/gapic_bazel.py", line 52, in node_library
    return self._generate_code(service, version, "nodejs", **kwargs)
  File "/tmpfs/src/github/synthtool/synthtool/gcp/gapic_bazel.py", line 183, in _generate_code
    shell.run(bazel_run_args)
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 39, in run
    raise exc
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['bazel', '--max_idle_secs=240', 'build', '//google/cloud/phishingprotection/v1beta1:phishingprotection-v1beta1-nodejs']' returned non-zero exit status 1.
2020-08-06 04:21:05,376 autosynth [ERROR] > Synthesis failed
2020-08-06 04:21:05,377 autosynth [DEBUG] > Running: git reset --hard HEAD
HEAD is now at 902ccfb docs: add links to the CHANGELOG from the README.md for Java and Node (#175)
2020-08-06 04:21:05,382 autosynth [DEBUG] > Running: git checkout autosynth
Switched to branch 'autosynth'
2020-08-06 04:21:05,388 autosynth [DEBUG] > Running: git clean -fdx
Removing __pycache__/
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 690, in <module>
    main()
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 539, in main
    return _inner_main(temp_dir)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 670, in _inner_main
    commit_count = synthesize_loop(x, multiple_prs, change_pusher, synthesizer)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 375, in synthesize_loop
    has_changes = toolbox.synthesize_version_in_new_branch(synthesizer, youngest)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 273, in synthesize_version_in_new_branch
    synthesizer.synthesize(synth_log_path, self.environ)
  File "/tmpfs/src/github/synthtool/autosynth/synthesizer.py", line 120, in synthesize
    synth_proc.check_returncode()  # Raise an exception.
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 389, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['/tmpfs/src/github/synthtool/env/bin/python3', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']' returned non-zero exit status 1.

Google internal developers can see the full log here.

Autosynth passed, closing! :green_heart:


FE-Homework-15

Drift VIdeo Link: https://video.drift.com/v/abIqldyVE2TiDmphP2e8rt5UoJhFJvsvUeEfy4Ex8Ws0/
Screenshots:

Main page

Marking item as completed

Removing completed items by using clear completed button

Metric
Value

alert_status
:white_check_mark: OK

bugs
0

code_smells
1

reliability_rating
A (0 Bugs)Note: A being best, E being worst

security_rating
A (0 Vulnerabilities)Note: A being best, E being worst

vulnerabilities
0

Metric
Value

alert_status
:x: ERROR

bugs
0

code_smells
1

reliability_rating
A (0 Bugs)Note: A being best, E being worst

security_rating
A (0 Vulnerabilities)Note: A being best, E being worst

vulnerabilities
0

Proper commit message's format must be used
Edit feature is not implemented
Nice work on the UI (the clear completed button must come at the top of the list so that it'll be easy for the user to access)
Use utils, scss constant files


Using balanced_low_0 on single GPU crashes

System Info
- `Accelerate` version: 0.20.3
- Platform: Linux-6.3.7-arch1-1-x86_64-with-glibc2.37
- Python version: 3.10.8
- Numpy version: 1.23.5
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- PyTorch XPU available: False
- System RAM: 15.57 GB
- GPU type: NVIDIA GeForce GTX 970
- `Accelerate` default config:
	Not found

Information

[X] The official example scripts
[ ] My own modified scripts

Tasks

[ ] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
[X] My own task or dataset (give details below)

Reproduction
Single GPU machine
from transformers import AutoModel
model = AutoModel.from_pretrained("gpt2", device_map="balanced_low_mem_0")

Will crash:
│                                                                                │
│ /home/nicolas/.pyenv/versions/3.10.8/lib/python3.10/site-packages/accelerate/utils/modeling.py:5 │
│ 88 in get_balanced_memory                                                                        │
│                                                                                │
│    585 │   │   num_devices = len([d for d in max_memory if torch.device(d).type == "xpu" and ma  │
│    586 │                                                                                │
│    587 │   module_sizes = compute_module_sizes(model, dtype=dtype, special_dtypes=special_dtype  │
│ ❱  588 │   per_gpu = module_sizes[""] // (num_devices - 1 if low_zero else num_devices)          │
│    589 │                                                                                │
│    590 │   # We can't just set the memory to model_size // num_devices as it will end being too  │
│    591 │   # slightly less layers and some layers will end up offload at the end. So this funct  │

Expected behavior
Few different options:

Either using low_mem_0 is not wanted for single GPU setup, then just show a proper error beforehand
Or properly load the model sparing space on GPU0 (as much as possible).

The actual use case comes from quantization purposes :
We need to load the model efficiently (ideally not at all).
Then start to load the head of the model to run it, then layer per layer run things in order to create the quantization.
https://github.com/huggingface/text-generation-inference/pull/438/files#diff-0cf72039dd2a1e24a0663ccb387ccc49ad7a510ca4ed0ed354695c5dc6693294R659-R667
low_mem_0 seems like the simplest option. I also considered
model = AutoModel.from_pretrained("gpt2", device_map="auto", max_memory={0: "0GiB"})

which crashes with the same issue.
Bonus points if there's an easy way to recover layers that have actually been offloaded to disk (currently layer[i].to(dev)) will crash because they are on device meta, so we need to recover where the weights are.

balanced_low_0 is not meant to be used on one GPU only, but we can easily fix the bug by doing nothing.

Perfect. The fix seems reasonable to me.
That won't fix model = AutoModel.from_pretrained("gpt2", device_map="auto", max_memory={0: "0GiB"})
Nor
model = AutoModel.from_pretrained("gpt2", device_map="auto", max_memory={0: "1MiiB"})
(Different stacktrace:
 /home/nicolas/src/transformers/src/transformers/modeling_utils.py:3066 in <dictcomp>             │
│                                                                                │
│   3063 │   │   │   offload_index = {                                                             │
│   3064 │   │   │   │   p: {"safetensors_file": f, "weight_name": p, "dtype": str_dtype}          │
│   3065 │   │   │   │   for p, f in weight_map.items()                                            │
│ ❱ 3066 │   │   │   │   if param_device_map[p] == "disk"                                          │
│   3067 │   │   │   }                                                                             │
│   3068 │   │                                                                                │
│   3069 │   │   if state_dict is not None:                                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'h.0.attn.bias'

Also maybe what I'm trying to accomplish is too far off from the original intention and we'd be better doing the single loads manually (which would effectively be disk offloading), it's just slightly tedious because of the head. thing.

That's what I feared I figure some workaround ! Thanks.


fix ch2.md: fix ch2 ambiguous translation

core 核心,kernel 内核. I think core is more like a word in hardware level , often is called "核心" in Chinese,  while kernel is the english of "内核", is used in operating system, a word in software level. Here the author talks about cpu core, i think "核心" will be better

That make sense~
But you can't include your own comment in PR....

Sorry , i committed my own comments to the wrong branch. I will correct it tomorrow. Thank you for your patience.

@Vonng done, is this ok now?

Thanks for your contribution~~


downloading a space shared with secure viewer rule shouldn't be possible.

Describe the bug
User can download the space shared to them via secure viewer role
Steps to reproduce
1.Admin creates a space test.
2.Admin creates a new user Einstein
3.Admin share space test with Einstein via secure view role
Expected behavior
Einstein shouldn't be able to download the space
Actual behavior
Space could be download and the file reads as :
.space0000755000000000000000000000000014627513364010351 5ustar0000000000000000.space/image.png0000644000000000000000000013420114627513364012221 0ustar0000000000000000error: permission denied: storage_id:"cc1b6a48-4c00-4932-8f30-3b9a02edeec5" opaque_id:"4dd292ff-cb05-49aa-85ec-63d7a3478e87" space_id:"41b1b5b4-a3b0-485b-af80-c78e1d353ba6" %

Setup

OCIS_XXX=5.1.0-prealpha+55f8314b3b Community

Please, add some commands to reproduce this. would be easier for others. I don't see a download step there


Batch training tweaks

Improving and generalizing the training of GaussianProcessRegressionPipe in batches

@JamesCarzon I'm making some changes at the moment... this will change the way the training jobs are created, so the notebooks temporarily won't run... I'll update them when the new features are implemented

Okay, this is working... now implement selected features

Looks okay now. Time to adapt the job scripts

it looks like @db(unique) does not like collective fields...

I can't believe this actually provisioned fine

This is probably easier if implemented as Dynamic instead...

Time to close this long and winding PR


Make hashes self-contained

Hashes are used in several use-cases of the Identity Server. We need to make each hash self-contained by storing the hash metadata such as the hash function.
Effort BE::30

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


README contains broken link to architecture.md

https://github.com/src-d/lookout/blob/master/docs/architecture.md

I believe this link is prepared for the changes in https://github.com/src-d/lookout/pull/398. @dpordomingo please confirm.

yes :dancing_men:

yes :dancing_men: (I didn't expect that much time till merging)


fix(datepicker): increase month selector width

Proposed changes
Increase the month selector width because it is too small to accommodate "September", "November", and "December".
Screenshots (if appropriate) or codepen:

Types of changes

[x] Bug fix (non-breaking change which fixes an issue).
[ ] New feature (non-breaking change which adds functionality).
[ ] Breaking change (fix or feature that would cause existing functionality to change).

Checklist:

[x] I have read the CONTRIBUTING document.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

This is a quite easy non-breaking fix. Could you please give it a go?

Thanks!


Update sunraster to work with ndcube 2.0

Description

ndcube 2.0 is a major API-breaking update to ndcube.  The API will cause some code breakages in sunraster and so sunraster must be updated.
Make next version of sunraster only compatible with ndcube 2.x rather than having sunraster work with both ndcube 1.x and 2.x.

Hey Dan,
Can I start doing this in parts which require attention for ndcube? It would be a good chance for me to learn hands on for how sunraster really works with ndcube.


When will the "copy bots" feature be pushed out?

Do you know when the "copy bots" feature be pushed out?

Do you know when the "copy bots" feature be pushed out?

Im almost done with it. I will add it as soon as I hit 50 stars.


[CON-1760] Fix Flaky Oracle Server Test

This fixes the flaky oracle server test.

Instead of creating a listener to get a random available port, I made it so we could just use that listener directly when creating the server.
I added back the health check, the reason the heath check was causing issues before was that the listener was being closed as a defer call, so that make the port unavailable for the oracle server. However, using the listener directly solve this.
I used /version instead of /prices for health check since /prices ends up calling the mock, which we have not set mock functions for.
I changed the Timeout test to just check for an error instead of a specific error since the specific error it returns seems like an implementation detail of gRPC.

The test still seems a little flaky as I am getting this error after 20-30 runs
        server_test.go:215:
                Error Trace:    /Users/zrbecker/code/skip/connect/service/servers/oracle/server_test.go:215
                Error:          Received unexpected error:
                                rpc error: code = DeadlineExceeded desc = context deadline exceeded
                Test:           TestServerTestSuite/TestOracleMarketMap

But it is certainly less flaky than it was.

@zrbecker looks like theres some conflicts

@zrbecker looks like theres some conflicts

should be resolved now


Add debugSetting into deployment and deploymentOperation for arm-resources

Hi @wenwu449, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

It looks like you're working at Microsoft (wenwu). If you're full-time, we DON'T require a contribution license agreement.

If you are a vendor, DO please sign the electronic contribution license agreement. It will take 2 minutes and there's no faxing! https://cla.azure.com.

TTYL, AZPRBOT;

@wenwu449 please do not rename exising specs whenever the API version is changed. Create a new document instead to preserve the history.

@wenwu449 - Ping ???

@stankovski @amarzavery I have changed the PR to keep old API version spec and add new API version spec. Please review and merge. Thanks

@wenwu449
Line 24: description needs to be updated. We don't have GetLongRunninOperationStatus() anymore. This was the hydra (old codegen) way doing things. Some teams were planning to update the API reference doc and the msdn docs from the swagger specs. Hence, having crisp description everywhere is of prime importance.

Line 598: ResourceGroups_Delete update description.

Line 747: Resources_MoveResources update description

Line 1101: Why would you want to return a 204 for getting a resource under a resource group? Is that used to state that the resource does not exist in the resource group? If so then is that how it should be done. Just trying to understand, when would Resources_Get return a 204?

Line 2138: DebugSettings has only 1 property named "detailLevel". If you apply the extension "x-ms-client-flatten" : true, then detailLevel will be flattened while deserialization and unflattened while serialization. It gives users a better experience.
Since debugsettings is being added for the first time , we can add this extension right now.
This extension could also be applied on following models but could be breaking change for client side libraries:

PolicyAssignmentProperties
PolicyDefinitionProperties
DeploymentProperties
DeploymentPropertiesExtended


Bridge on mainnet down

A deposit was made with an empty memo which resulted in following error:
{"level":"info","time":"2022-10-03T07:27:18Z","message":"calling mint now"}
{"level":"error","time":"2022-10-03T07:27:18Z","message":"transaction with hash 1c840a95d126a98728cf09dacd6b28d850d20fbae220dee51d0772ff40e4d43c has empty memo, refunding now"}
{"level":"info","time":"2022-10-03T07:27:18Z","message":"call ready to be submitted"}
{"level":"error","time":"2022-10-03T07:27:18Z","message":"error occurred while submitting call Invalid Transaction\nfailed to submit extrinsic\ngithub.com/threefoldtech/substrate-client.(*Substrate).CallOnce\n\t/go/pkg/mod/github.com/threefoldtech/substrate-client@v0.0.0-20220613084053-f1f12e7dcf85/utils.go:178\ngithub.com/threefoldtech/substrate-client.(*Substrate).Call\n\t/go/pkg/mod/github.com/threefoldtech/substrate-client@v0.0.0-20220613084053-f1f12e7dcf85/utils.go:130\ngithub.com/threefoldtech/tfchain_bridge/pkg/bridge.(*Bridge).callExtrinsic\n\t/opt/tfchain/pkg/bridge/bridge.go:240\ngithub.com/threefoldtech/tfchain_bridge/pkg/bridge.(*Bridge).refund\n\t/opt/tfchain/pkg/bridge/bridge.go:349\ngithub.com/threefoldtech/tfchain_bridge/pkg/bridge.(*Bridge).mint\n\t/opt/tfchain/pkg/bridge/bridge.go:269\ngithub.com/threefoldtech/tfchain_bridge/pkg/stellar.(*StellarWallet).MonitorBridgeAccountAndMint.func1\n\t/opt/tfchain/pkg/stellar/stellar.go:362\ngithub.com/threefoldtech/tfchain_bridge/pkg/stellar.(*StellarWallet).StreamBridgeStellarTransactions\n\t/opt/tfchain/pkg/stellar/stellar.go:414\ngithub.com/threefoldtech/tfchain_bridge/pkg/stellar.(*StellarWallet).MonitorBridgeAccountAndMint\n\t/opt/tfchain/pkg/stellar/stellar.go:368\ngithub.com/threefoldtech/tfchain_bridge/pkg/bridge.(*Bridge).Start.func1\n\t/opt/tfchain/pkg/bridge/bridge.go:117\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"}
{"level":"error","time":"2022-10-03T07:27:18Z","message":"Error occured while minting: failed to submit extrinsic: Invalid Transaction"}
{"level":"info","time":"2022-10-03T07:27:19Z","message":"fetching events for blockheight 3996773"}
{"level":"info","time":"2022-10-03T07:27:24Z","message":"fetching events for blockheight 3996774"}
{"level":"info","ID":26,"time":"2022-10-03T07:27:24Z","message":"found burn transaction expired event"}
{"level":"info","time":"2022-10-03T07:27:25Z","message":"stellar account sequence number:<PHONE_NUMBER>36270588"}
{"level":"info","time":"2022-10-03T07:27:25Z","message":"call ready to be submitted"}
{"level":"error","time":"2022-10-03T07:27:25Z","message":"error occurred while submitting call Invalid Transaction\nfailed to submit extrinsic\ngithub.com/threefoldtech/substrate-client.(*Substrate).CallOnce\n\t/go/pkg/mod/github.com/threefoldtech/substrate-client@v0.0.0-20220613084053-f1f12e7dcf85/utils.go:178\ngithub.com/threefoldtech/substrate-client.(*Substrate).Call\n\t/go/pkg/mod/github.com/threefoldtech/substrate-client@v0.0.0-20220613084053-f1f12e7dcf85/utils.go:130\ngithub.com/threefoldtech/tfchain_bridge/pkg/bridge.(*Bridge).callExtrinsic\n\t/opt/tfchain/pkg/bridge/bridge.go:240\ngithub.com/threefoldtech/tfchain_bridge/pkg/bridge.(*Bridge).processEventRecords\n\t/opt/tfchain/pkg/bridge/bridge.go:206\ngithub.com/threefoldtech/tfchain_bridge/pkg/bridge.(*Bridge).processEventsForHeight\n\t/opt/tfchain/pkg/bridge/bridge.go:143\ngithub.com/threefoldtech/tfchain_bridge/pkg/bridge.(*Bridge).Start\n\t/opt/tfchain/pkg/bridge/bridge.go:125\nmain.main\n\t/opt/tfchain/main.go:52\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"}
{"level":"fatal","error":"failed to submit extrinsic: Invalid Transaction","time":"2022-10-03T07:27:25Z","message":"exited unexpectedly"}

Bridge account did not have funds anymore, refunded the wallet and restarted. Issue is resolved now :)


ci: install gke-auth-plugin in e2e image

This change installs the gke-gcloud-auth-plugin in the image which the e2e tests run in. This change will be needed to support authentication with GKE clusters starting with v1.26.

 > [kpt-config-sync-e2e 6/9] RUN wget https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.0.1/kustomize_v5.0.1_linux_amd64.tar.gz -O /tmp/kustomize_v5.0.1_linux_amd64.tar.gz &&   tar -zxvf /tmp/kustomize_v5.0.1_linux_amd64.tar.gz -C /tmp &&   mv /tmp/kustomize /usr/local/bin/kustomize &&   rm /tmp/kustomize_v5.0.1_linux_amd64.tar.gz:
#0 0.185 Connecting to github.com (<IP_ADDRESS>:443)
#0 0.381 Connecting to objects.githubusercontent.com (<IP_ADDRESS>:443)
#0 0.492 wget: server returned error: HTTP/1.1 503 Backend unavailable, connection timeout

Looks like a flake installing kustomize
/retest


[BUG] Layout issues when device font is set to larger than default

Describe the bug
When one sets the device font to be larger than default from the device settings, there are some layout issues that are produced in many parts of the app.
Expected behavior
The app should optimize its user interface according to the device font set. This can easily be fixed by setting the views' dimensions to be in sp (Scale-independent Pixels) instead of any other unit, for any view that can change size when text size is changed.
Screenshots
Some examples:

Version
Rocket.Chat.Android version: 3.3.0

Thank you for submitting the issue. I'll check it out.


chore: Only auto merge patch & minor version changes

Major changes will sometimes require some code changes. Most of these will be picked up by the build failing but that's not always the case like here: https://github.com/jorenn92/Maintainerr/pull/1340
There's also some merit in holding off using a new major release like https://github.com/jorenn92/Maintainerr/pull/1336 as from experience nextjs is initially pretty unstable.

:tada: This PR is included in version 2.2.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Update documentation of PINRemoteImageManager

To note that the manager holds a weak reference to its alternative representation provider.

          1 Warning

:warning:
Any source code changes should have an entry in CHANGELOG.md or have #trivial in their title.

  Generated by :no_entry_sign: Danger


⚠ Bofrost.es has degraded performance

In dee70b8, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1638 ms

Resolved: Bofrost.es performance has improved in fa7c2fe after 32 minutes.


Allow callbacks to access internal variables of training loop

🚀 Feature
Internal variables (batch, predictions, etc) of the training loop (training + validation step) should be made transparent to callbacks. Right now, there is no way to access these internal variables of the training loop through callbacks without making them as attributes of lightning module. This doesn't sound optimal as it pollutes the lightning module with non-essential code.
Motivation
Use case: Visualize images and predictions from a training batch. Right now, there are two ways:

Add a log method in pl module and call this from training_step method of pl module. By doing this, we are essentially polluting pl module with non-essential code.

Write a visualization callback. As of now, callback has access to pl module and trainer but still, it can’t access the variables (images, predictions, etc) in the training step. We can make these variables as attributes of pl module but then updating these attributes in every training step (so that callback can access it) would also count as “non-essential code” which would defeat the point of the callback. It also spoils the neat separation between callbacks and pl module as we'll be caching/updating attributes in pl module even if we switch off the callback.

each callback method has accessible trainer and model...
what PL version are using? because we made these changes quite recently... :]
cc: @ethanwharris

Hi @Borda, I am using pl version 0.7.1. In this version, callbacks do have access to trainer and pl module as you said. However, I wanted to point out that I can't access the interval variables of pl module training step method from a callback. Example:
Let's say we have the following module class from pl examples:
class LitModel(pl.LightningModule):

    def __init__(self):
        super().__init__()
        self.l1 = torch.nn.Linear(28 * 28, 10)

    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))

    def training_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self(x)
        return {'loss': F.cross_entropy(y_hat, y)}

I'd try to write the visualization callback as follows to visualize images from training batch:
class VizCallback(pl.Callback):

    def on_batch_end(self, trainer, pl_module):
        # here I wish to have access to images `x` from training_step
        grid_img = torchvision.utils.make_grid(x)
        self.logger.experiment.log_image(grid_img)

The issue is I can't access training samples x in the example above from a callback without linking it to pl module. As the code gets more complicated, I wish to have access to several variables of training_step which could be model predictions, attention maps and so on from a callback. In order to access these variables from a callback, I'd have to make them as attributes of pl module and update them in every training step which I think is non-ideal. See what I mean?

I have figured a way to retrieve these informations in a callback. Use hiddens, not sure if it's the right way to do it but at least it works with little modifications in your model, here is my training_step() function
def training_step(self, batch, batch_idx):
    inputs, targets = batch
    preds = self(inputs)
    loss = #whatever
    hiddens = {"inputs": inputs, "predictions": preds, "targets": targets}
    return {"loss": loss, "hiddens": hiddens}

with this you will get everything in your callback
class VizCallback(pl.Callback):
    def on_batch_end(self, trainer, pl_module):
        # here I wish to have access to images `x` from training_step
        if trainer.hiddens:
            inputs = trainer.hiddens["inputs"]
            preds = trainer.hiddens["predictions"]
            targets = trainer.hiddens["targets"]
            grid_img = torchvision.utils.make_grid(inputs)
            self.logger.experiment.log_image(grid_img)

But there is nothing for the validation loop as far as I see there is no on_batch_end for the validation am I wrong?

@czotti good hack haha but we need an actual way of doing this.
@PyTorchLightning/core-contributors ?

iirc, you can return any arbitrary key and it will be added to trainer.callback_metrics - you don't need to override the hiddens key

I think any key returning based idea (like overriding hiddens key/returning dictionary relevant for callback) would spoil the neat separation between callbacks and pl module because user would have to remember to cache/return variables in pl module which are relevant for callback and when user wishes to switch off the callback, they would have to delete the corresponding cache/return code in pl module. Thus, I suggested all the internal variables of training and validation to be exposed to callbacks by default so that they don't have to remember cache relevant stuff for callback in pl module. Please let me know if I wasn't clear. :)

iirc, you can return any arbitrary key and it will be added to trainer.callback_metrics - you don't need to override the hiddens key

Unfortunately, in the case of on_batch_end callback the Trainer's self.callback_metrics is not yet populated.

@williamFalcon I think the way to go here is to have more of the looping variables as trainer attributes. However, we need to be careful with this as we don't want our trainer to be bloated.

we can offer some read-only properties...

I haven't gone through this in detail but it seems relevant to the discussion here:
Repo: https://github.com/pberkes/persistent_locals
Blog post: http://code.activestate.com/recipes/577283-decorator-to-expose-local-variables-of-a-function-/
Quick summary from the README of above repo:
The proposed solution consists of a decorator that makes the local variables accessible from a read-only property of the function, 'locals'. For example:
@persistent_locals
def is_sum_lt_prod(a,b,c):
    sum = a+b+c
    prod = a*b*c
    return sum<prod

after calling the function, e.g. is_sum_lt_prod(2,1,2), we can analyze
the intermediate results as
is_sum_lt_prod.locals

which returns
{'a': 2, 'b': 1, 'c': 2, 'prod': 4, 'sum': 5}

Please ignore my above comment, I don't think the solution is neat and maintainable...

Yes, this would be useful. Currently there's no proper way to pass batch computation result to on_test_batch_end for a custom Callback (hiddens and callback_metrics won't work either). As a hacky workaround I'm just attaching the results of test_step to self lol.

I needed to do this in my code as well (log the results of inference as they occur), and I was thinking of setting model class attributes i.e. self.current_test_output = ... and then accessing it in the callback.
But then I thought maybe this would cause issues during multiprocessing, since I'm not exactly sure how the model is split/synced during DP/DDP.

Has this been resolved? I am trying to implement a callback for visualization and currently facing the same issue.

Are there any solutions \ workarounds for this?
having a similar issue of complex reporting of predictions that I want to separate from the main module

I'm using czotti's hack with self.hiddens (see above) but I agree that it poluted the pl module.

I don't know if this is relevant here but I was able to build a custom callback that was able to log images along with the predictions to the Weights and Biases dashboard.
class ImagePredictionLogger(Callback):
    def __init__(self, val_samples, num_samples=32):
        super().__init__()
        self.num_samples = num_samples
        self.val_imgs, self.val_labels = val_samples

    def on_validation_epoch_end(self, trainer, pl_module):
        val_imgs = self.val_imgs.to(device=pl_module.device)
        val_labels = self.val_labels.to(device=pl_module.device)

        logits = pl_module(val_imgs)
        preds = torch.argmax(logits, -1)

        trainer.logger.experiment.log({
            "examples":[wandb.Image(x, caption=f"Pred:{pred}, Label:{y}")
                           for x, pred, y in zip(val_imgs[:self.num_samples],
                                                 preds[:self.num_samples],
                                                 val_labels[:self.num_samples])]
            })

This is the associated colab notebook. Associated W&B's run page.

Your code indeed separates logic of lightning module from callback, but it does some additional work. I noticed that Callback now also has on_validation_batch_end so czozzi's method can be applied at validation time too (same for test). Also on_*_batch_end now provides an actual batch, so use case from https://github.com/PyTorchLightning/pytorch-lightning/issues/1243#issuecomment-605536194 is supported.

@festeh The problem of outputs remain though. For me I think the ideal solution would be inplace modification of a batch dictionary which is propagated by pytorch-lightning. Right now no inplace modifications are propagated, so the batch may only contain the original data which makes it rather useless. Until then I guess I will record the batch to the module.

I'm not sure how modifying a batch would help to solve original problem - coupling essential and non-essential code. For me it's either full decoupling and clean code (but you'll have to do additional computations in callback) or less nicer code that just returns everything in outputs.

For logging images or other media, I'm returning the values from the train/val/test step and using those outputs in my visualization callback. This does increase the memory overhead, but cleans up the code:
class MyModel(pl.LightningModule):
    def __init__(self):
        ...

    def training_step(self, batch, batch_idx):
        ...
        outputs = {
            "loss": total_loss,
            "rgb_imgs": batch["rgb"].detach(),
        }

Callback:
class LogMedia(Callback):
    def __init__(...):
        ...

    @rank_zero_only
    def on_train_batch_end(self, trainer, pl_module, outputs, batch, batch_idx):
        if self.every_n_steps_train > 0 and trainer.global_step % self.every_n_steps_train == 0:
            # 2D outputs - Shape: [b, ips, c, h, w], ips=imgs/scene
            rgb_imgs = outputs.get("rgb_imgs", None)
            if rgb_imgs is not None:
                viz_rgb = []
                rgb_imgs = rgb_imgs[:, : self.limit_samples_2d, ...]
                for idx, batch_rgb in enumerate(rgb_imgs):
                    rgb_imgs = torchvision.utils.make_grid(batch_rgb.detach(), nrow=4)  # create_grid
                    rgb_imgs = rgb_imgs.cpu().numpy().transpose((1, 2, 0))
                    rgb_imgs = (rgb_imgs * 255).astype(np.uint8)
                    rgb_imgs = Image.fromarray(rgb_imgs)
                    viz_rgb.append(wandb.Image(rgb_imgs))  # wandb uses PIL Image

                    if self.save_to_disk:
                        batch_num = 0 if self.overwrite_prev_files else batch_idx
                        rgb_imgs.save(self.exp_dir / f"batch-{batch_num}.{idx}.rgb.jpg")
                trainer.logger.experiment.log({f"{stage.value}/rgb": viz_rgb}, commit=False)

@Shreeyak Does your method not store every image of the epoch? IIRC the outputs of the training step are all stored for the at_epoch_end step in the model. For large datasets I think this would cause the memory to explode.

@Parskatt No, I actually don't. The above was a simplified snippet of code that uploads the images every batch. In my actual code, I log the latest N batches every M steps.
I keep a circular queue (collections.deque object) in the LogMedia callback that holds the latest N batches.
Here's a sample code:
from collections import deque
from pytorch_lightning.utilities import rank_zero_only

def flush_deque(deque_obj: deque) -> List:
    """Return all of the data in a que, in order oldest->newest in list"""
    data_r = []
    while len(deque_obj) > 0:
        data_r.append(deque_obj.popleft())
    return data_r

class LogMedia(Callback):
  def __init__(self, max_samples: int = 3, every_n_steps_train: int = 50, ...):
        self.deque_train = deque(maxlen=self.max_samples)
        self.every_n_steps_train = every_n_steps_train
        self.max_samples = max_samples

    @rank_zero_only
    def on_train_batch_end(self, trainer, pl_module, outputs, batch, batch_idx):
        self.deque_train.append(outputs)
        if self.every_n_steps_train > 0 and trainer.global_step % self.every_n_steps_train == 0:
            self._log_results(trainer, self.deque_train, Stage.TRAIN, batch_idx)

    def _log_results(self, trainer, deque_obj: deque, stage: Stage, batch_idx: Optional[int] = None):
        """Process and upload the media files to logger"""
        outputs_list = flush_deque(deque_obj)
        for outputs in outputs_list:
            self.visualize_scene_outputs(trainer, outputs, stage, batch_idx)

IIRC the outputs of the training step are all stored for the at_epoch_end step in the model.

I didn't realize that. I didn't see much memory increase in my project (5000 imgs only). In this case, the code can be modified so that the LightningModule holds the latest N batches, which it passes to the callback in the train_epoch_end() method.


Fix building problems caused by npm require of aws-sdk

Fixes issues raised on this thread: https://forums.meteor.com/t/error-in-test-mode-on-1-3-typeerror-cannot-read-property-stream-of-undefined/21696/7
Fix comes from https://github.com/tjconsult 's comment.
In short the problem is:
Due to the aws-sdk having it's own implementation of require, importing aws-sdk regularly can cause some build systems to break. For me, the breakage occurs when trying to run tests, specifically using dispatch:mocha-phantomjs or practicalmeteor:mocha
This is the responsible code: https://github.com/aws/aws-sdk-js/blob/master/dist/aws-sdk.js
See the thread mentioned above for more info

Sweet, thanks for the PR


[v0.2.0] ValueError: len() should return >= 0

Input Prompt Length: 75
Generation Parameters:

Width: 320
Height: 640
having 4 profiles generated by v0.1.0

Error Log
    Traceback (most recent call last):
      File "C:\stable-diffusion-webui\stable-diffusion-webui\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "C:\stable-diffusion-webui\stable-diffusion-webui\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\modules\txt2img.py", line 55, in txt2img
        processed = processing.process_images(p)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\modules\processing.py", line 734, in process_images
        res = process_images_inner(p)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\modules\processing.py", line 868, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\modules\processing.py", line 1142, in sample
        samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
      File "C:\stable-diffusion-webui\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 235, in sample
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "C:\stable-diffusion-webui\stable-diffusion-webui\modules\sd_samplers_common.py", line 261, in launch_sampling
        return func()
      File "C:\stable-diffusion-webui\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 235, in <lambda>
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "C:\stable-diffusion-webui\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 594, in sample_dpmpp_2m
        denoised = model(x, sigmas[i] * s_in, **extra_args)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\modules\sd_samplers_cfg_denoiser.py", line 188, in forward
        x_out[a:b] = self.inner_model(x_in[a:b], sigma_in[a:b], cond=make_condition_dict(c_crossattn, image_cond_in[a:b]))
      File "C:\stable-diffusion-webui\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
        eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
        return self.inner_model.apply_model(*args, **kwargs)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in <lambda>
        setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
      File "C:\stable-diffusion-webui\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
        return self.__orig_func(*args, **kwargs)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
        x_recon = self.model(x_noisy, t, **cond)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward
        out = self.diffusion_model(x, t, context=cc)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\modules\sd_unet.py", line 89, in UNetModel_forward
        return current_unet.forward(x, timesteps, context, *args, **kwargs)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\scripts\trt.py", line 70, in forward
        self.engine.allocate_buffers(feed_dict)
      File "C:\stable-diffusion-webui\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\utilities.py", line 304, in allocate_buffers
        tuple(shape), dtype=numpy_to_torch_dtype_dict[dtype]
    ValueError: __len__() should return >= 0

Image can be generated when SD Unet is disabled explicitly.

what i noticed with the new update is that your profile that loads has to cover the whole spectrum including your max resolution set by "hi-res fix". i don't know if you are using that, but force rebuild your profile might not hurt.

After removing Unet folder contents and restarted the instance, when I tried to regenerate the default engine, the Expected all tensors to be on the same device occurs.
@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=--xformers --opt-sdp-attention --opt-sub-quad-attention --opt-channelslast --precision autocast --no-half --device-id=0

set CUDA_MODULE_LOADING=LAZY
set 'PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512'
call webui.bat

ERROR:root:Exporting to ONYX failed. Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

v0.1.0 was able to generate the ONYX, so I think I will try to fallback the version first.

Also note that the checkpoint using is not SDXL, it should be based on SD1.5.

After removing Unet folder contents and restarted the instance, when I tried to regenerate the default engine, the Expected all tensors to be on the same device occurs.
@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=--xformers --opt-sdp-attention --opt-sub-quad-attention --opt-channelslast --precision autocast --no-half --device-id=0

set CUDA_MODULE_LOADING=LAZY
set 'PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512'
call webui.bat

ERROR:root:Exporting to ONYX failed. Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

v0.1.0 was able to generate the ONYX, so I think I will try to fallback the version first.

just aspeculation, but could it be that --device-id=0 is listing both the cpu and gpu? if so, then "Expected all tensors to be on the same device, but found at least two devices" would make sense.

Somehow force update xformers from 0.0.20 to 0.0.23 and torch from 2.0.1+cu118 to 2.1.3+cu118 has fixed the issue, ONNX can be exported and TensorRT engine can be built.

Nope, updating dependencies only resolved the rebuild issue, but same issue still occurs when

Hitting the same. Uncertain when it broke. Followed all the steps in this thread (inclusive of blowing away venv a couple times during testing).
ONNX/TRT that were generated prior to whatever broke continue to function as expected, inclusive of handling over 75 tokens (more on that below). I saved those off, then attempted to regenerate the ONNX/TRT models using the exact same input .safetensors and parameters, but continue to receive the same error if Max Tokens isn't 75.
If it's helpful: generating ONNX/TRT models without increasing the Max Tokens parameter appears to work (inclusive of Hires. fix, presuming final resolution is blow whatever maximum was set in the TRT options)  without throwing the initial __len__() should return >= 0 error. Unfortunately, it means text inputs greater than 75 tokens are discarded and new errors are logged in the console:
[E] 3: [executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2045] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2045, condition: profileMaxDims.d[i] >= dimensions.d[i]. Supplied binding dimension [2,154,768] for bindings[2] exceed min ~ max range at index 1, maximum dimension in profile is 77, minimum dimension in profile is 77, but supplied dimension is 154.

Addendum, in case it's helpful:
Using the same generation values (single batch size, 256-1280 L+W max/min, 75/75/300 min/opt/max prompt size) for exact same model, results in different sample filenames before and after breaking-update:
Working; generated before breaking change
[filename][hash]_sample=1x4x32x32+2x4x64x64+2x4x160x160-timesteps=1+2+2-encoder_hidden_states=1x77x768+2x77x768+2x308x768
Not working; generated after breaking change
[filename][hash]_sample=1x4x32x32+1x4x64x64+1x4x160x160-timesteps=1+1+1-encoder_hidden_states=1x77x768+1x77x768+1x308x768

Regenerated TRTs, no help.
Profile selected:

Loaded Profile: 1
        sample = [(2, 4, 80, 40), (2, 4, 120, 64), (2, 4, 160, 80)]
        timesteps = [(2,), (2,), (2,)]
        encoder_hidden_states = [(2, 77, 768), (2, 77, 768), (2, 77, 768)]
        latent = [(-1945833728), (-1945834240), (-1945833216)]

[filename]_[hash]_cc86_sample=2x4x80x40+2x4x120x64+2x4x160x80-timesteps=2+2+2-encoder_hidden_states=2x77x768+2x77x768+2x77x768
I also observed the following error log also before it yields __len__() should return >= 0:
[E] 3: [executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2046] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2046, condition: profileMinDims.d[i] <= dimensions.d[i]. Supplied binding dimension [1,4,80,40] for bindings[0] exceed min ~ max range at index 0, maximum dimension in profile is 2, minimum dimension in profile is 2, but supplied dimension is 1.

Another interesting observation is that, when I first launched WebUI and using default prompts without hires fix to generate, it works; when then I turned on hires fix, it fails; and then when I turned off hires fix and generate again, it fails again.

I usually use 20 steps and when it works, the error of the supplied dimension appears when it is >=15 steps.
 75%|█████████████████████████████████████████████████████████████▌                    | 15/20 [00:01<00:00, 14.94it/s][E] 3: [executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2046] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2046, condition: profileMinDims.d[i] <= dimensions.d[i]. Supplied binding dimension [1,4,80,40] for bindings[0] exceed min ~ max range at index 0, maximum dimension in profile is 2, minimum dimension in profile is 2, but supplied dimension is 1.
    )

BTW, when I recall the previous behavior that worked, I should have observed the generation process is actively switching between generated engines, when doing hires fix. This behavior was in v0.1.0 and I am not able to observe this in v0.2.0.

This has happened twice for me, once after doing a force-rebuild of a profile, which erroneously resulted in two identical profiles (according to the list of profiles in the TensorRT tab) instead of replacing the existing one, and the second time after creating a dynamic profile whose resolution range overlapped with another.  The first time, I resolved it by removing my Unet-trt and Unet-onnx directories.  This time I have resolved it by simply removing mention of one of the two profiles from Unet-trt/model.json.
In short, it seems that the trigger for this bug is the existence of profiles that overlap in some specific way that I haven't determined.  (Maybe if one is a perfect subset of the other?)
To get a little bit more specific about the cause of the false assertion:
            tensor = torch.empty(
                tuple(shape), dtype=numpy_to_torch_dtype_dict[dtype]
            ).to(device=device)

shape in this call is a Dims equal to (0), which apparently returns a negative number when len() is called on it (which happens as a result of the tuple() conversion).

I added a print statement to show the value of shape for each iteration of the loop in Engine.allocate_buffers (utilities.py).
When there are overlapping profiles and the problem occurs:
shape 0=torch.Size([1, 4, 64, 96])
shape 1=torch.Size([1])
shape 2=torch.Size([1, 77, 768])
[E] 3: [executionContext.cpp::resolveSlots::2868] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::resolveSlots::2868, condition: allInputDimensionsSpecified(routine)
    )
shape 3=(0)

When I pare down the model.json file to contain only one base profile and one dynamic profile for this checkpoint and the problem does not occur:
shape 0=torch.Size([1, 4, 64, 96])
shape 1=torch.Size([1])
shape 2=torch.Size([1, 77, 768])
shape 3=(1, 4, 64, 96)

When I pare down the model.json file to contain only one base profile and one dynamic profile for this checkpoint and the problem does not occur:

This did not resolve for me when doing my initial testing; I removed the ONNX and TRT folders entirely and relaunched Automatic1111 to rebuild the empty .json and folder structure, then recreated ONNX/TRT files. Zero models, zero configs.
Same errors occurred when I rebuilt models using known-good values that worked in previous versions of the extension.
Only fix in my case was to reduce min/opt/max tokens to 75. Highres. fix worked fine with batch 1/1/1:

@maxxrox, this definitely seems like a regression, so things that worked before do not necessarily work now.  I suspect you will have no problem as long as you don't have more than one profile that could apply to whatever resolution + batch size you're doing.  I can definitely create dynamic profiles that support multiple batch sizes.  Things only go sideways when I have two or more profiles for one checkpoint and, as far as I can tell, only when the resolution and batch size ranges overlap...perhaps in some specific way I don't know how to characterize, because sometimes even that works.

Not certain that model overlap is the issue in my case. This configuration (the one, sole, single, solitary config I have) works to generate native resolution and Highres. fix'd images:

I removed the Unet-TRT and Unet-ONNX folders entirely, then restarted Automatic1111. Here is confirmation that I have zero, zilch, nada, nil, (0) profiles available:

I then rebuild the exact same model as evidenced in screenshot 1 above, with the sole change being that the Text-Length parameter are set to 75/150/225 (instead of 75/75/75 in the above working model):

Then run it (without Highres. fix enabled, 512x512 default resolution):

Voila. Errors.
Crazy bit is that I can restore the ONNX/TRT built in the previous version of this extension and it works fine with >75 tokens. I simply cannot recreate that exact same model in v0.2.0.

It’d be interesting to recreate exactly the same profile that worked in the prior version and diff (1) the model.json files and (2) the lists of files in those two directories.On Jan 20, 2024, at 11:10, maxxrox @.***> wrote:
Not certain that model overlap is the issue in my case. This configuration (the one, sole, single, solitary config I have) works to generate native resolution and Highres. fix'd images:
image.png (view on web)
I removed the Unet-TRT and Unet-ONNX folders entirely, then restarted Automatic1111. Here is confirmation that I have zero, zilch, nada, nil, (0) profiles available:
image.png (view on web)
image.png (view on web)
image.png (view on web)
image.png (view on web)
I then rebuild the exact same model as evidenced in screenshot 1 above, with the sole change being that the Text-Length parameter are set to 75/150/225 (instead of 75/75/75 in the above working model):
image.png (view on web)
Then run it (without Highres. fix enabled, 512x512 default resolution):
image.png (view on web)
Voila. Errors.
Crazy bit is that I can restore the ONNX/TRT built in the previous version of this extension and it works fine with >75 tokens. I simply cannot recreate that exact same model in v0.2.0.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

Uncertain is has anything to do with the model.json file itself; the model-creation process itself outputs different filenames (see above). Regardless, here you go; this is the comparison between my the model.json files corresponding to the two versions tested in Automatic1111 detailed in my post above.:

Insofar as I can tell, the issue isn't in the models.json itself (it's simply recording what is), but rather the export process is specifying incorrect shapes at model creation. See my post above about filename differences/comparison of models exported in v0.2.0 and in the prior release.

I see what you mean, @maxxrox.
I tried to reproduce this by removing both of my Unet directories and creating a profile with the same settings as yours.  Oddly, when I do this, I encounter a different issue: It reports that there are no suitable engines found.

I have some that works with optimal/max tokens greater than 75, but they were all generated prior to the v0.2.0 release. I haven't been successful in generating them since.
Also interesting re: highres. fix not working for some folks using a batch size less than 2; the only bit I've noticed is that the post-upscaled image size must be lower than or equal to the max size supported by the ONNX/TRT. E.g., I could only use x2 upscaling on a base 1024x1024 image if my max supported resolution is 2040x2048. Using 2.5 upscaling on a base 1024x1024 image would result in an error (honestly, would be better if a console warning were throne and the UNET disabled for the generation, but that's a separate enhancement request rather than a bug report).

In the readme file said:
Resolution: When generating images, the resolution needs to be a multiple of 64. This applies to hires.fix as well, requiring the low and high-res to be divisible by 64.

I solved this problem by changing the resolution.

Thanks for weighing in, but this comment is not applicable. Please read the thread above. As you can see, this error occurs when using resolution 512x512:

For the previously mentioned error
[E] 3: [executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2046] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2046, condition: profileMinDims.d[i] <= dimensions.d[i]. Supplied binding dimension [1,4,40,80] for bindings[0] exceed min ~ max range at index 0, maximum dimension in profile is 4, minimum dimension in profile is 2, but supplied dimension is 1.)

When I injected the print log suggested by @maxxrox , I see the following:
 75%|█████████████████████████████████████████████████████████████▌                    | 15/20 [00:01<00:00, 17.34it/s]
shape 0=torch.Size([1, 4, 40, 80])
[E] 3: [executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2046] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2046, condition: profileMinDims.d[i] <= dimensions.d[i]. Supplied binding dimension [1,4,40,80] for bindings[0] exceed min ~ max range at index 0, maximum dimension in profile is 4, minimum dimension in profile is 2, but supplied dimension is 1.
shape 1=torch.Size([1])
[E] 3: [executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2046] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2046, condition: profileMinDims.d[i] <= dimensions.d[i]. Supplied binding dimension [1] for bindings[1] exceed min ~ max range at index 0, maximum dimension in profile is 4, minimum dimension in profile is 2, but supplied dimension is 1.
shape 2=torch.Size([1, 77, 768])
[E] 3: [executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2046] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2046, condition: profileMinDims.d[i] <= dimensions.d[i]. Supplied binding dimension [1,77,768] for bindings[2] exceed min ~ max range at index 0, maximum dimension in profile is 4, minimum dimension in profile is 2, but supplied dimension is 1.

It seems the batch_size in the shape suddenly changes from 2 to 1 at this step and I don't know why.

This is due to Negative Guidance minimum sigma set.
The value I set was 0.33, so the last 33% of steps (~6.6 = 6) will skip negative prompt, and therefore reduced a dimension.
Resetting value to zero eliminated the shape error.

tensor(... ...)
shape 0=torch.Size([2, 4, 80, 40])

tensor([999., 999.], device='cuda:0')
shape 1=torch.Size([2])

tensor([[[-0.3787,  0.0214, -0.0475,  ..., -0.4611, -0.3003,  0.0575],
         [-1.0053, -0.3166,  2.1954,  ..., -0.5814,  0.8775, -0.7874],
         [-1.9166, -0.3008,  1.5344,  ...,  0.2911,  0.5383, -0.2709],
         ...,
         [ 1.2781, -0.5123,  0.1328,  ...,  0.4920, -2.1129, -0.7022],
         [ 1.2668, -0.5150,  0.1203,  ...,  0.4973, -2.1449, -0.7328],
         [ 1.1520, -0.5386,  0.1315,  ...,  0.6207, -2.3053, -0.8234]],

        [[-0.3943,  0.0223, -0.0495,  ..., -0.4802, -0.3128,  0.0599],
         [ 0.5520, -0.7075,  0.2375,  ..., -0.3554,  1.5901, -0.9215],
         [ 0.8309,  0.6456,  0.5782,  ..., -0.8517, -0.0687, -1.1418],
         ...,
         [ 0.5583,  0.2160, -0.9537,  ..., -0.5698, -0.1652, -1.4681],
         [ 0.5301,  0.2512, -0.8903,  ..., -0.6384, -0.1873, -1.4783],
         [ 0.6110,  0.3418, -0.9354,  ..., -0.6222, -0.1316, -1.4487]]],
       device='cuda:0')
shape 2=torch.Size([2, 77, 768])

shape 3=(-1945994997)

ValueError: __len__() should return >= 0

Has anyone found a way to fix this? I'm also getting this error with multiple dynamic profiles.

From latest attempts, it becomes working when

using engine with batch size 2 or above; and
disable Negative Guidance minimum sigma


JSONDecodeError

I got this error:
Traceback (most recent call last):
  File "/usr/local/bin/ydl-podcast", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/ydl_podcast/__main__.py", line 56, in main
    download(ydl_mod, sub)
  File "/usr/local/lib/python3.11/dist-packages/ydl_podcast/__init__.py", line 246, in download
    entry = get_video_metadata(ydl_mod, md["url"], options, quiet=True)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/ydl_podcast/__init__.py", line 152, in get_video_metadata
    return json.loads(output.getvalue().split("\n")[0])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The config file is
subscriptions:
  - name: lurenqi
    url: https://space.bilibili.com/573606572/video
    retention_days: 14 # How old is the oldest video to download and keep locally
    initialize: False # Download the whole channel on first run
    download_last: 5 # Only download the last video at each run
    audio_only: True # Extract and keep audio only
    format: m4a      # Force end format
    filename_template: '%(id)s.%(ext)s' # Use this format for files saved from this subscription

output_dir: /root/ydl-podcast/
url_root: https://podcast.example.com
youtube-dl-module: yt-dlp

How to solve it, thanks.

I found the reason.
The config of retention_days is set to 14, and config of download_last is set to 5, while there are only two episodes in the last 14 days.  Therefore, the other three info of episodes is empty.


Documentation about managing connections in python.

I cannot find any docs/examples on how to in practice manage connections using Azure Functions python.
As I am running into SNAT port exhaustion and have been adviced to reuse connections (database clients and http).
Is there such documentation?

Hi @tiptop96, our current recommendation is to use global variables to manage connections. You can see an outline (though not specific to connections) of how to accomplish this here: https://docs.microsoft.com/en-us/azure/azure-functions/functions-reference-python#global-variables.

Closing.


Add unfurling capabilities to Webhook payload

Sending a chat message with ChatScheduleMessageRequest has both unfurlLinks and unfurlMedia, but com.slack.api.webhook.Payload does not. Testing the API manually, it looks that Slack does abide the unfurl_links and unfurl_media in the hooks URL
Misc
PR # 1000 🎉
Category

[ ] bolt (Bolt for Java)
[ ] bolt-{sub modules} (Bolt for Java - optional modules)
[ ] slack-api-client (Slack API Clients)
[x] slack-api-model (Slack API Data Models)
[ ] slack-api-*-kotlin-extension (Kotlin Extensions for Slack API Clients)
[ ] slack-app-backend (The primitive layer of Bolt for Java)

Requirements
Please read the Contributing guidelines and Code of Conduct before creating this issue or pull request. By submitting, you are agreeing to the those rules.

@jazcarate Congrats on the 1000th PR 🎉 Can you check my review comments above?

Sure!
I wanted to keep parity with ChatScheduleMessageRequests. But cappital B boolean is fine by me 😄
https://github.com/slackapi/java-slack-sdk/blob/main/slack-api-client/src/main/java/com/slack/api/methods/request/chat/ChatScheduleMessageRequest.java#L111-L119


Legend Align Labels like table format

Hello All,
I am trying align legend labels in table format like
Red and Green box to start at same line, Yellow and Blue boxes to start at same line,
Here's what it looks like when browser is resized,

Fiddle Link: https://jsfiddle.net/9u62nkb0/2/
I went through available legend styling options, but couldn't figure it out.
Is it possible to apply css styles directly on the label ? or is there any legend configuration to fix the same.
Any leads are appreciated.

Does the start alignment in v3 meet your needs? It's the red chart on https://www.chartjs.org/samples/master/legend/title.html

@etimberg,In my case aligning it to start only fixes the first column, is it possible to make the yellow and blue legend labels to start at the same line on smaller browser size.
with align: 'start'  as below,

Unfortunately there's no way to align the items in columns. The best solution I can offer is to create your own legend plugin that positions differently, or make a legend out of HTML


wsl leaves multiple background hosts alive

Your Windows build number:  Microsoft Windows [Version 10.0.17134.228]

What you're doing and what's happening:

i started a powershell window and typed wsl
in wsl i started an ssh-agent
i typed exit and closed the powershell window
repeat the previous steps a few time

we can now see multiple powershell hosts with their child processes running in the background and we can see all the ssh-agent processes in top in a new powershell window.

What's wrong / what should be happening instead:

when i manually end all three (or any number of) background powershell processes, all the ssh-agents are still alive, so my conclusion is only one background powershell (or console host or whatever) task is needed for keeping all wsl processes alive.
closing a powershell window should move all processes started under it to an already running host because this way I found myself having a few dozens of background powershells open while regularly using wsl.
this problem doesn't only apply to ssh-agent, the same thing happens with any process inside wsl that starts in the background.

my conclusion is only one background powershell (or console host or whatever) task is needed for keeping all wsl processes alive.

Zero (not one) powershell process(es) are needed to keep wsl processes "alive". The analogy is if you telneted into a Linux cloud server, started ssh-agent, exited the telnet session, and then exited PS. Another analogy would be if you started a win32 ssh-agent service, and exited PS.  Blog post is here.

Zero (not one) powershell process(es) are needed to keep wsl processes "alive"

I agree that this should be the expected behavior but when I end the last powershell (background) task, the wsl suddenly stops and all proccesses I started inside of wsl get terminated.
I read the blog post before opening the issue.
Are you sure this is by design, as this way users could potentially open hundreds of background powershell (or cmd) tasks after a few days/weeks

I can't reproduce this here. If I start one (and only one) powershell process, start wsl.exe, execute sudo service start cron, exit the WSL bash shell, and exit powershell, then cron remains running.

while trying to reproduce this, have you noticed that after exiting powershell (typing exit or clicking the X button on the window), it still stays as a background task (you can see it in task manager)?
if you then opened another powershell, run wsl and started another background service (such as sshd or nginx or ssh-agent), and then proceeded to exit that powershell window, it would also go to the background tasks section of your task manager leaving you with two powershell background tasks running.
now if you tried ending one powershell background task (through the task manager), both of your wsl background processes would continue working, no matter which powershell background task you ended. however if you end the second (last) powershell task, then the whole wsl stops and all your background services get terminated.
what i've been trying to say is, there's no need for keeping all the powershell tasks in background.

Thanks for your reply and patience. I'll try to find some time to setup a vm running the insider build you're using and will try to reproduce this behavior.

...I can't reproduce your accumulating PowerShell instances ... I have no clear explanation to give you on that.

Okay I have a working theory, though I'm a little sleep deprived and could be wrong. I think the change in the observed behavior might be because of the ConPTY changes, with a nod to #2370 (message) which shipped in 17618. I don't want to speculate too much, but at base, I couldn't see how your "background" PowerShells l could have to do with WSL proper. Those PowerShell instances aren't any of WSL's business (as alluded previously, you could have come in through say PuTTY and started nginx, and WSL sure as **** isn't going to hold PuTTY hostage in the background once you exit). I almost said out loud in the previous post "maybe something in PowerShell changed", but that didn't make much sense either. But something did change in the way PTYs are done.
Recommendation would be to try Insiders and see if your problem "goes away". If my theory is correct this was a long meandering non-obvious walk to dupe.

ConPTY is currently only used when hosting Windows processes from WSL, not the other way around.

Yeah I know. I didn't want to get in the weeds ("speculate too much"). Idea was that before the improvements, interop needed to hold a hidden console hostage, despite the TTY being detached by the daemon. LxManager.dll would then refcount them, and clean up the turds when necessary. I still could be very wrong. I wouldn't have said anything except late and weekend. Beats "closed - works for me".

Alright. I'm downloading a 17713 to look. In the mean time, what are your CLI steps just to be sure I do the exact same repro. Don't need screenshots. Just lines of text that can be cut and paste into two PS consoles before they're closed. If this reproduces in 17713 but not 18219 this is going to get a big grumble (which isn't on you).

alright, so start up a new powershell and type:
wsl
sudo service ssh start
exit
exit

for the second powershell you repeat the same steps but instead of ssh start any other background service e.g. cron, nginx, etc. or start an ssh-agent:
wsl
sudo service cron start
exit
exit

take a look at your task manager and you will see two background powershells running

If this reproduces in 17713 but not 18219 this is going to get a big grumble

why is that if i may ask?

why is that if i may ask?

Because it's a lot of effort just to find out it was fixed in skip-ahead. Makes for a questionable Sunday time-sink.

oh ok get it. im sorry if thats the case
speaking of, i tried upgrading to 18219 through regular windows update but its throwing me error 0xc0000005, tried multiple times after restarts and switching to fast in insider settings. i dont know how else to get the newest build since i cant find download iso anywhere

This is on a shiny new 17713 in a VM. Below is my background task list in Task Manager, with sshd running but no "Windows Powershell (n)" or "Windows Command Processor (n)" in the Double-ues. cron is in the scroll-up. Dunno what to tell you at this point.

how is that possible... im really confused, my installation is also fresh..
which distro did you install?

Ubuntu 18.04 from the MSFT Store. First thing I did after enabling WSL in 'features'. Thing I did after that was your repro steps verbatim. I didn't even stop for a coffee.

huh. i could try to test this on another computer, i have the 17134 build on my physical laptop and i emulated the 17713 build in virtualbox on the same laptop. both are x64. i doubt that could cause such a problem but i guess its worth testing.

but i guess its worth testing

Effort. Me I'd sit and wait for a "I'm seeing this too" from someone else, and maybe an explanation from the devs. I mean, okay, you try again on another box and find (a) same thing or (b) no PowerShells held hostage like me. Either way, then what. Suppose it depends a lot on how hard up you are for a time sink today.

hahah okay i see your point.
could you somehow ping the devs to read the issue and try to explain this behavior?

I can confirm this behavior. Running windows 10 x64 build 17134

you're right. this is very interesting. i didn't even think about checking process explorer or the details tab.
though, i think my original question still remains - but for wslhost.exe instead of powershell.
why do multiple wslhost.exe processes stay running in background? it's unnecessary and resource consuming (probably insignificant, but why keep e.g. 10 instances in background if a single one is enough)?

why do multiple wslhost.exe processes stay running in background?

Probable cause: If one background WSL-ed Linux service (sshd etc.) fails/corrupts it'll not affect other services.

that would make sense if a new wslhost.exe was forked for each new wsl service, but that's not the case. even if you start multiple background services or processes in a wsl session, only a single wslhost.exe will be spawned (along any old wslhost.exes)
also, out of curiosity, i tried killing the conhost.exe process that is left as a child process of each wslhost.exe and it doesn't effect wsl and its services.
so basically a single wslhost.exe is needed for the whole wsl to work in background.

i think my original question still remains - but for wslhost.exe instead of powershell

Nice to have an answer to that, given Dustin's excellent before and after pics. But at base the answer is "because the devs decided to do it like that" and we're basically back to by-design or question tag again.
[Better question would be why the Rube Goldberg host service on top of host service architecture when there was no backward compatibility constraints. A devpts.sys filesystem could be mounted in Windows, meaning porting PTY code would amount to changing the path string instead of having to shim a whole new obstinately different API. But I digress.]

i tried killing the conhost.exe process .., and it doesn't effect wsl

No.

I have an issue (I'm on the latest windows update) that's very similar. I've also followed this guide: https://evalcode.com/zsh-default-wsl-windows-10/ to setup ZSH on WSL (not sure if relevant)
Now I've started a shell, started a tmux server and attached to it. And now every time I close the terminal, reopen it and attach to the existing tmux server a new ZSH process keeps running in the background. That doesn't sound right and surely isn't how it works on linux (or I just didn't notice, correct me if I'm wrong).

@leongrdic - Wslhost is used so background processes (sshd in your example) are able to use NT interop (launching Windows binaries) with the right permission / environment as the process that launched WSL. The overhead of the process is minimal and the binary only does "work" when interop requests are received.


[Backport] Fix PyPI Packages and Python Docker Images nightly release (#18222)

remove OS from s3 library path

fix bash script run commands

Revert "remove OS from s3 library path"

This reverts commit 266511305ba1ac56582ee9b867d0a64ad3d60378.

hardcode s3 path for upload/download of binaries

Co-authored-by: Ubuntu<EMAIL_ADDRESS>Description
(Brief description on what this PR is about)
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

@leezu @ciyongch Please help to review. Thanks

Seems there's some issue with CI on this branch, can you help to take a check as well? Thanks!
[2020-05-29T08:56:46.595Z] + ln -s /usr/bin/ninja-build /usr/bin/ninja

[2020-05-29T08:56:46.595Z] ln: failed to create symbolic link '/usr/bin/ninja': File exists

[2020-05-29T08:56:47.515Z] The command '/bin/sh -c /work/centos7_core.sh' returned a non-zero code: 1

@mxnet-bot run ci [centos-cpu, centos-gpu, unix-gpu]


Add support for exclusion constraints

Closes #1098, replaces #1099

Thank you, I will review this later!

Merged, thank you so much! Are there any other constraint we should add? check constraints maybe?

Thanks for merging! I thought about CHECK constraints as well, but then I realized that they are actually better implemented using validations (which are already supported). I am not aware of cases where CHECK requires access to the db, but I might be wrong.


Handling of optional fields with non-None defaults

In TOML, unlike JSON and YAML, there is no null value. This means that an optional field (T | None annotation) will only be able to have a value of None when None is the default value.
For example, for the field foo: str | None = "bar":

when deserializing from TOML, foo cannot be assigned the value None
when serializing to TOML, foo being None would be a runtime error

I think it's best to reject unsupported constructs as early as possible, so when the Binder is created.
The only doubt I have is that if we ever add JSON support, forcing a None default on optional fields would be an unnecessary limitation. On the other hand, maybe it's better to focus first on making the best possible TOML binder and not worry about features that don't exist yet.

Labeling this as "bug" because the current behavior of serializing a None value to TOML is to silently omit it from the output, when means that it reverts to the default value. Therefore, if the default is not None, this alters the data on a round trip.


Upgrade to Jest 27.x

This makes it possible to run Jest with --detectOpenHandles, which will make it possible to ensure all async operations are completed before tests end.

Superseded by #3670.


Clear Written Documentation

More clear and explanatory documentation on

dependency injection
Route Context
Controller

Codecov Report

Merging #46 (3e0303e) into master (5cd509d) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 3e0303e differs from pull request most recent head 6819332. Consider uploading reports for the commit 6819332 to get more accurate results

@@           Coverage Diff           @@
##           master      #46   +/-   ##
=======================================
  Coverage   99.52%   99.52%
=======================================
  Files          44       44
  Lines        1695     1695
=======================================
  Hits         1687     1687
  Misses          8        8

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


I am unable to use the umd version

First of all, thank you so much @samjonesigd for creating and maintaining this. :heart:
I don't use a build step in our web app. So I am trying to use the UMD version.
I have the following in my HTML's head:
 <!-- Vue 3 JS -->
    <script<EMAIL_ADDRESS>
    <!-- Vuetify 3 JS -->
    <script<EMAIL_ADDRESS>
    <!-- Vuetify 3 CSS -->
    <link<EMAIL_ADDRESS>rel="stylesheet">

    <!-- Material Design Icons -->
    <link href="https://cdn.jsdelivr.net/npm/<EMAIL_ADDRESS>rel="stylesheet">

    <!-- Vue Image Zoomer -->
    <script<EMAIL_ADDRESS>    <link<EMAIL_ADDRESS>rel="stylesheet">

Then, in a script before </body>, I have:
const VueImageZoomer = window["vue-image-zoomer"];

In my Vue's createApp I have put the below:
components: {
                "vue-image-zoomer": VueImageZoomer
            },

I am using it as below in my HTML:
<vue-image-zoomer :regular="'data:image/png;base64,' + testCase.test_steps[stepIndexRangerPDFDialog].baselinePNGBase64">
</vue-image-zoomer>

It's showing an error on the developer console:
Uncaught TypeError: Cannot read properties of undefined (reading 'createElementVNode')
    at vue-image-zoomer.umd.js:1:6287
    at vue-image-zoomer.umd.js:1:214
    at vue-image-zoomer.umd.js:1:250

Please tell me what I am doing wrong? How I can successfully use vue-image-zoomer without a build step?
Thanks @samjonesigd!

@vkWeb
fixed in 2.4.1
use this const { VueImageZoomer } = window['VueImageZoomer'];

Thanks buddy @samjonesigd, appreciate your work!


fall back on url_name and func name if no route name is provided with Django

Related issues
closes #1092

:broken_heart: Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #1095 opened

Start Time: 2021-04-12T07:27:31.098+0000

Duration: 19 min 52 sec

Commit: c9317c84ee95f37ff2d5c790c295ed4854620102

Test stats :test_tube:

Test
Results

Failed
1

Passed
9685

Skipped
7883

Total
17569

Trends :test_tube:

Test errors
Expand to view the tests failures
Initializing / Test / Python-pypy-3-1 / test_ssl_verify_fails – tests.transports.test_urllib3

Expand to view the error details
 assert 'certificate verify failed' in '<ExceptionInfo TransportException("Unable to reach APM Server: HTTPSConnectionPool(host=\'<IP_ADDRESS>\', port=42855): M...or(SSLCertVerificationError(1, \'unable to get local issuer certificate\'))) (url: https://<IP_ADDRESS>:42855)") tblen=2>'
 +  where '<ExceptionInfo TransportException("Unable to reach APM Server: HTTPSConnectionPool(host=\'<IP_ADDRESS>\', port=42855): M...or(SSLCertVerificationError(1, \'unable to get local issuer certificate\'))) (url: https://<IP_ADDRESS>:42855)") tblen=2>' = str(<ExceptionInfo TransportException("Unable to reach APM Server: HTTPSConnectionPool(host='<IP_ADDRESS>', port=42855): Max ...Error(SSLCertVerificationError(1, 'unable to get local issuer certificate'))) (url: https://<IP_ADDRESS>:42855)") tblen=2>)

Expand to view the stacktrace
 waiting_httpsserver = <SecureContentServer(<class 'pytest_localserver.https.SecureContentServer'>, started<PHONE_NUMBER>72448)>
elasticapm_client = <tests.fixtures.TempStoreClient object at 0x0000000002ba3bb0>

    @pytest.mark.flaky(reruns=3)  # test is flaky on Windows
    def test_ssl_verify_fails(waiting_httpsserver, elasticapm_client):
        waiting_httpsserver.serve_content(code=202, content="", headers={"Location": "http://example.com/foo"})
        transport = Transport(waiting_httpsserver.url, client=elasticapm_client)
        transport.start_thread()
        try:
            with pytest.raises(TransportException) as exc_info:
                url = transport.send(compat.b("x"))
>           assert "certificate verify failed" in str(exc_info)
E           assert 'certificate verify failed' in '<ExceptionInfo TransportException("Unable to reach APM Server: HTTPSConnectionPool(host=\'<IP_ADDRESS>\', port=42855): M...or(SSLCertVerificationError(1, \'unable to get local issuer certificate\'))) (url: https://<IP_ADDRESS>:42855)") tblen=2>'
E            +  where '<ExceptionInfo TransportException("Unable to reach APM Server: HTTPSConnectionPool(host=\'<IP_ADDRESS>\', port=42855): M...or(SSLCertVerificationError(1, \'unable to get local issuer certificate\'))) (url: https://<IP_ADDRESS>:42855)") tblen=2>' = str(<ExceptionInfo TransportException("Unable to reach APM Server: HTTPSConnectionPool(host='<IP_ADDRESS>', port=42855): Max ...Error(SSLCertVerificationError(1, 'unable to get local issuer certificate'))) (url: https://<IP_ADDRESS>:42855)") tblen=2>)

tests/transports/test_urllib3.py:171: AssertionError

Steps errors
Expand to view the steps failures

Shell Script

Took 4 min 59 sec . View more details on here
Description: ./tests/scripts/docker/run_tests.sh pypy-3 none

Shell Script

Took 3 min 49 sec . View more details on here
Description: ./tests/scripts/docker/run_tests.sh pypy-3 none

Restore files previously stashed

Took 0 min 0 sec . View more details on here
Description: coverage-pypy-3-flask-0.12

Restore files previously stashed

Took 0 min 0 sec . View more details on here
Description: coverage-pypy-3-celery-3-django-2.0

Restore files previously stashed

Took 0 min 0 sec . View more details on here
Description: coverage-pypy-3-pymongo-newest

Restore files previously stashed

Took 0 min 0 sec . View more details on here
Description: coverage-pypy-3-elasticsearch-2

Restore files previously stashed

Took 0 min 0 sec . View more details on here
Description: coverage-pypy-3-gevent-newest

Restore files previously stashed

Took 0 min 0 sec . View more details on here
Description: coverage-pypy-3-graphene-2

Archive the artifacts

Took 0 min 0 sec . View more details on here
Description: [2021-04-12T07:46:20.647Z] Archiving artifacts
Python-pypy-3-none tests failed : hudson.AbortExcept

Error signal

Took 0 min 0 sec . View more details on here
Description: Python-pypy-3-none tests failed : hudson.AbortException: script returned exit code 2

Log output
Expand to view the last 100 lines of log output

[2021-04-12T07:45:35.111Z] ERROR: No such saved stash ‘coverage-pypy-3-flask-0.12’
[2021-04-12T07:45:35.867Z] ERROR: No such saved stash ‘coverage-pypy-3-celery-3-django-2.0’
[2021-04-12T07:45:36.616Z] ERROR: No such saved stash ‘coverage-pypy-3-pymongo-newest’
[2021-04-12T07:45:37.363Z] ERROR: No such saved stash ‘coverage-pypy-3-elasticsearch-2’
[2021-04-12T07:45:38.113Z] ERROR: No such saved stash ‘coverage-pypy-3-gevent-newest’
[2021-04-12T07:45:38.888Z] ERROR: No such saved stash ‘coverage-pypy-3-graphene-2’
[2021-04-12T07:45:40.823Z] + python3 -m coverage combine
[2021-04-12T07:46:19.577Z] + python3 -m coverage xml
[2021-04-12T07:46:19.615Z] [Cobertura] Skipping Cobertura coverage report as build was not SUCCESS or better ...
[2021-04-12T07:46:19.615Z]
[2021-04-12T07:46:20.066Z] + curl -sLO https://code.jquery.com/jquery-3.3.1.slim.min.js
[2021-04-12T07:46:20.378Z] + curl -sLO<EMAIL_ADDRESS>[2021-04-12T07:46:20.647Z] Archiving artifacts
[2021-04-12T07:46:20.835Z] Stage "Building packages" skipped due to earlier failure(s)
[2021-04-12T07:46:20.950Z] Stage "Integration Tests" skipped due to earlier failure(s)
[2021-04-12T07:46:21.065Z] Stage "Benchmarks" skipped due to earlier failure(s)
[2021-04-12T07:46:21.304Z] Stage "Prepare Release" skipped due to earlier failure(s)
[2021-04-12T07:46:21.358Z] Stage "Prepare Release" skipped due to earlier failure(s)
[2021-04-12T07:46:21.465Z] Stage "Prepare Release" skipped due to earlier failure(s)
[2021-04-12T07:46:21.574Z] Stage "Prepare Release" skipped due to earlier failure(s)
[2021-04-12T07:46:22.044Z] Running on Jenkins in /var/lib/jenkins/workspace/hon_apm-agent-python-mbp_PR-1095
[2021-04-12T07:46:22.205Z] [INFO] getVaultSecret: Getting secrets
[2021-04-12T07:46:22.464Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-12T07:46:23.468Z] + chmod 755 generate-build-data.sh
[2021-04-12T07:46:23.469Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-python/apm-agent-python-mbp/PR-1095/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-python/apm-agent-python-mbp/PR-1095/runs/1 FAILURE 1132092
[2021-04-12T07:46:23.719Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-python/apm-agent-python-mbp/PR-1095/runs/1/steps/?limit=10000 -o steps-info.json
[2021-04-12T07:46:27.842Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-python/apm-agent-python-mbp/PR-1095/runs/1/tests/?status=FAILED -o tests-errors.json

:bug: Flaky test report
:grey_exclamation: There are test failures but not known flaky tests.

Expand to view the summary

Test stats :test_tube:

Test
Results

Failed
1

Passed
9685

Skipped
7883

Total
17569

Genuine test errors
:broken_heart: There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Initializing / Test / Python-pypy-3-1 / test_ssl_verify_fails – tests.transports.test_urllib3

test failures are unrelated


add builtin RequestTemplate：COMMON to support CLF format

Codecov Report

Merging #268 into develop will decrease coverage by 0.09%.
The diff coverage is 9.67%.

@@            Coverage Diff            @@
##           develop    #268     +/-   ##
=========================================
- Coverage     52.6%   52.5%   -0.1%
=========================================
  Files          230     230
  Lines        22301   22332     +31
=========================================
- Hits         11732   11726      -6
- Misses        9239    9272     +33
- Partials      1330    1334      +4

Impacted Files
Coverage Δ

bfe_modules/mod_access/request_log.go
0% <0%> (ø)
:arrow_up:

bfe_modules/mod_access/conf_mod_access.go
28.28% <60%> (+1.68%)
:arrow_up:

bfe_util/fixed_pool.go
83.33% <0%> (-16.67%)
:arrow_down:

...e_modules/mod_trust_clientip/mod_trust_clientip.go
42.85% <0%> (-3.3%)
:arrow_down:

bfe_spdy/server_write_sched.go
78.46% <0%> (-1.54%)
:arrow_down:

bfe_http2/server.go
71.67% <0%> (-0.17%)
:arrow_down:

bfe_websocket/server_conn.go
76.51% <0%> (ø)
:arrow_up:

bfe_spdy/server_conn.go
78.53% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2336f73...a9c2df4. Read the comment docs.