Excessive memory use when uploading

Which version of the SDK was used?
v0.11.0
Which platform are you using? (ex: Windows, Linux, Debian)
Linux
What problem was encountered?
Excess memory usage compared to v0.8.0
How can we reproduce the problem in the simplest way?
Uploading lots of files with rclone works pretty well.
Have you found a mitigation/solution?
Setting GOGC=20 mitigates the problem.
Analysis
In c0650f8c2c04b124963ffb252cdd21b8f1dd4074 #167 by @element-of-surprise the chunked uploader was refactored and a sync.Pool was introduced.
This sync.Pool appears to be causing memory problems according to the memory profiles below.
Testing with the azureblob backend in rclone produces the memory profile attached
memprofile-1.53.3.log
This looks like

Compared to the profile from the previous version of rclone, which shows no use of sync.Pool.

This caused memory usage for rclone to go up from 400MB to 1.4 GB between the two versions. The old version of rclone was using v0.8.0
What appears to be happening is that memory is getting stuck in the sync.Pool.
From the sync.Pool docs:

On the other hand, a free list maintained as part of a short-lived object is
not a suitable use for a Pool, since the overhead does not amortize well in
that scenario. It is more efficient to have such objects implement their own
free list.

I think the use of sync.Pool is falling into that case. The Pool isn't shared between connections so it isn't really helping.
Because it is created and destroyed for each upload the old Pools will hang around in memory waiting to be garbage collected this is and the reason why GOGC=20 works around the problem I think.
Since you are recycling buffers of a fixed size maybe a chan []byte might work better instead of the sync.Pool - you could then flush it at the end of the transfer (unlike a sync.Pool which you can't flush).
Or alternatively implement a global sync.Pool. You'll need one for each size of chunk though which is a disadvantage.
This was originally discussed on the rclone forum.

Hi, as I said on the forum I'll be happy to test a release candidate for this fix, simply reach me here on this issue.

Forgive me, since its the holidays I'll be responding from a slight bit of ignorance, as I haven't gone back to look at the code in question.
My feeling is that this problem is unfortunately not about memory use or garbage made, I think it's more a problem where various use goals aren't aligning.
The end use of network copy package has the primary goal of moving bytes across a network as fast as possible.  As a secondary goal in some situations, you also want to constrict how much memory can be utilized at any moment (however, not always).
The achievement of those goals across multiple scenarios is where everything kinda goes bad.  Are we:

copying millions of small files,
1 large file
thousands of large files
is the network lossy
is this a short term process or always copying situations
...

What I remember of the existing structure (my memory may be faulty) was that there were three knobs:

Buffer size
Number of buffers
Concurrency restriction

My memory may be faulty on this, but I remember the design being so that the common case was to use a single client to transfer files to a single endpoint (reusing the sync.Pool).
The API provided restrictions through an options struct, if I remember.  I didn't want to make changes to the API without thinking about an overhaul of the package, so that lead to some restrictions on what I was going to change.
For use cases I tests (which revolved around kusto and azcopy), using these changes I could increase transfer speeds by up to 10x for many transfer types (though my main goal was to fix a couple of bugs).
So apparently I was sacrificing your use case to increase others.
I remember thinking when I was making the change that a good long term plan would be to have an interface that looked something like:
type TransferManager interface{
  Get() *bytes.Buffer
  Put(b *bytes.Buffer)
  Run(func() error)
}

This would get passed in whenever someone made a streaming call.
There would be several pre-defined types that could be used and users could define their own if none of those were satisfactory.  The code using TransferManager would never use more than the size of an allocated buffer (unless it was 0, which would error).  Run() would act as a thread-pool manager (which is more efficient than spinning goroutines off per copy).
So if static buffers were better for a use case, you could do that (the method before the change).  Or a sync.Pool for others.  Sometimes a combination of both is great (done this a few times).  For long term services doing copies in BE qos transfer class, you could write something that responds to network back pressure (dialing up or down your attempted copy speed).
And because you would pass your concrete type to the call, you could choose to share this between all calls or create a new one per call.
I'll dig into this deeper next week, going to try enjoying the holiday, but I am concerned specifically with the OOM mentioned.  I'm not familiar with rclone, so if you could link me the code where its using the streaming code, that would and whatever you were running to pull the memory profile, I'd appreciate it.
Thanks for bringing this up and tagging me.

The main upload code is here in rclone
https://github.com/rclone/rclone/blob/f7d9b157079bbf6c7145f94784a60f846f19c637/backend/azureblob/azureblob.go#L1441
Rclone calls azblob.UploadStreamToBlockBlob
https://github.com/Azure/azure-storage-blob-go/blob/456ab4777f89ceb54316ddf71d2acfd39bb86e1d/azblob/highlevel.go#L386-L388
which turn calls copyFromReader which is the code in question.
https://github.com/Azure/azure-storage-blob-go/blob/456ab4777f89ceb54316ddf71d2acfd39bb86e1d/azblob/chunkwriting.go#L31
I don't think there is any way of re-using the copier there which is what we'd need for the use of sync.Pool to be useful and not generate so much garbage.
Since we are making and throwing away a sync.Pool for each object uploaded, the sync.Pool retains its memory until some indeterminate time later, hence the memory leak.
It may well be that there is a better interface for rclone to be calling here - all it wants to do is upload a short blob of known size which doesn't need to be chunked. It has a separate multipart upload routine of its own and doesn't use the transfer manager (which I didn't know about until now!).
Here are docs on how to get memory profiles out of rclone: https://rclone.org/rc/#debugging-memory-use - you'll need the --rc flag and to uploads lots of smallish objects I think (@dcendents how would you describe the files you were uploading in terms of number and size distribution?)
Enjoy your holidays - no rush :-)

Thanks for getting back to me.  I'm making some changes now that hopefully will give a more robust and customizable solution.
Though, I feel that we should probably consider just re-writing the streaming interface into a client model, I'm going to keep this change API compatible.
Back to you next day or so.

Added PR: https://github.com/Azure/azure-storage-blob-go/pull/234

Any news on this? It came up again in the rclone forum.
@ajankovic are you working on a fix for this? Your idea of passing in a pool looks like a great fix to me.

Any news on this? It came up again in the rclone forum.
@ajankovic are you working on a fix for this? Your idea of passing in a pool looks like a great fix to me.

@ncw yes the PR is one way to solve it but it's not up to me to merge it.
Although I am not sure why the tests are failing because it's not related to my changes in any way, they are probably flaky.

@ncw yes the PR is one way to solve it but it's not up to me to merge it.
Although I am not sure why the tests are failing because it's not related to my changes in any way, they are probably flaky.

You need your environment setup to run some of the tests, I remember this from doing the original patch.
I left comments on this or you patch in your PR and linked in my PR.  I believe azblob has some type of release cadence so things get looked at as those approach.
  I detangled a bunch of stuff and had to write benchmarks that didn't go into the repo to make sure certain things were at least as fast as before for certain apps.  I lost the forrest because of the trees it appears.

You need your environment setup to run some of the tests, I remember this from doing the original patch.
I left comments on this or you patch in your PR and linked in my PR.  I believe azblob has some type of release cadence so things get looked at as those approach.
  I detangled a bunch of stuff and had to write benchmarks that didn't go into the repo to make sure certain things were at least as fast as before for certain apps.  I lost the forrest because of the trees it appears.

I've got a user testing your patch @ajankovic on the rclone forum. Hopefully it will fix their problem.
We need a fix for this ASAP though it is causing a lot of problems with rclone users in particular.

I've got a user testing your patch @ajankovic on the rclone forum. Hopefully it will fix their problem.
We need a fix for this ASAP though it is causing a lot of problems with rclone users in particular.

@ncw we have our own fork of rclone that uses my patch and it's definitely working in our tests. Having all of this upstream would be even better to cut on maintenance.

You need your environment setup to run some of the tests, I remember this from doing the original patch.

But this failed on CI where the environment is correctly setup. And my PR is not changing default behaviour nor integration tests, if pool is not provided it's not used. I'll check it out.

@ncw we have our own fork of rclone that uses my patch and it's definitely working in our tests. Having all of this upstream would be even better to cut on maintenance.

You need your environment setup to run some of the tests, I remember this from doing the original patch.

But this failed on CI where the environment is correctly setup. And my PR is not changing default behaviour nor integration tests, if pool is not provided it's not used. I'll check it out.

But this failed on CI where the environment is correctly setup. And my PR is not changing default behaviour nor integration tests, if pool is not provided it's not used. I'll check it out.

I am 99% sure that certain tests never works in the CI, that this only can be tested locally if you have the right setup.  I'm pretty sure only the devs have that setup.

But this failed on CI where the environment is correctly setup. And my PR is not changing default behaviour nor integration tests, if pool is not provided it's not used. I'll check it out.

I am 99% sure that certain tests never works in the CI, that this only can be tested locally if you have the right setup.  I'm pretty sure only the devs have that setup.

@element-of-surprise @ajankovic Thanks a lot for the contribution! We really appreciate it. I also apologize for the long delay in reviewing the PR.
We really like #242 but chose to merge #234 instead, as it is more future-proofing. Thanks for the detailed analysis! @ncw

@element-of-surprise @ajankovic Thanks a lot for the contribution! We really appreciate it. I also apologize for the long delay in reviewing the PR.
We really like #242 but chose to merge #234 instead, as it is more future-proofing. Thanks for the detailed analysis! @ncw

We'll make a release soon.

We'll make a release soon.

Thanks for fixing this :-)
If you could make the release in the next day or two I can fit it into rclone 1.54 which would be very convenient for me :-)

Thanks for fixing this :-)
If you could make the release in the next day or two I can fit it into rclone 1.54 which would be very convenient for me :-)

It is done! Sorry again for the long delay.

It is done! Sorry again for the long delay.

Thank you very much for making the release @zezha-msft
I've put this fix into rclone now and it will be released with rclone 1.54 and many users will rejoice!

Thank you very much for making the release @zezha-msft
I've put this fix into rclone now and it will be released with rclone 1.54 and many users will rejoice!


Color cues

When a user attempts to add a blog that is already in the list, a blue bar appears on the top of the screen alerting the user that the blog already exists. Similarly, when trying to download blogs, if the blog no longer exists, a blue bar appears at the top of the screen alerting the user. Finally, certain other error messages (timeouts, for example) also appear in a blue bar at the top of the screen.
Would it be possible to distinguish the type of message by color? For example, when a blog no longer exists, rather than use a blue bar, use a red bar. In the case of a timeout, you could use a green or purple bar. This would help a user quickly recognize errors by type, especially when sorting through several messages.

Hello Michael,
that sounds like a nice idea. We should look what different kind of errors could happen and which ones can be grouped together.
Maybe someone would like to take on that issue.

I'd be happy to take on that issue... but given that I haven't written a line of code (well, other than a macro here and there) since ... um ... well, I was writing in Pascal if that gives you any idea ... I don't think I'm the best guy for the task. Sigh.

Thanks for considering it. I also got to know Pascal... Maybe someone else reads it here and likes to work on it.

I'd be happy to take this up. So, since I have forked the repo already should I just create a new branch from that fork for this issue?

I've assigned it to you. Please have a look.
Yes, just create a new branch.

@arrowsmith001 Thanks for taking care of this issue.


skyscraper: init at 3.11.1

I don't like env.PREFIX = placeholder "out"; but I don't know how to set PREFIX properly, so if you know, please tell me/commit here.
Project repo.
To do
There's bash/zsh completion, I would try to understand how to make them work.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

also since this is an init PR, might as well squash all the commits for the package into one commit

also since this is an init PR, might as well squash all the commits for the package into one commit

I would like to, but I can't. My Internet is too slow too pull repo, so I do all the things through GitHub website. And I don't know of any way to squash commits here (except when merging).

This program require 7z (found in p7zip) during runtime (for some operations). How can I make it available for it? If p7zip is installed on my system, it finds it. But I would like not to install p7zip system-wide. I tried different buildInputs, path (I don't know what it actually do), no success so far.

This program require 7z (found in p7zip) during runtime (for some operations). How can I make it available for it? If p7zip is installed on my system, it finds it. But I would like not to install p7zip system-wide. I tried different buildInputs, path (I don't know what it actually do), no success so far.

You can add it using a wrapper. Search for makeBinPath


Width and height should be optional

I think width and height of widgets should be optional when margin is given，or  them should be optional even if margin is not given, and decided by font size，then widget would be more flexible，do hope awesome update. @AustL
Regards
larryw3i.

I have forked it and I will make a pull request later.

Buddy @AustL , may I think the width and the height are meaningless if they are < text's.
self.margin = kwargs.get('margin', 20)

# Width and height are meaningless if they are < font's.
_text_w,_text_h = self.text.get_size()
if self._width < _text_w:
    self._width = _text_w + 2*self.margin
if self._height < _text_h:
    self._height = _text_h + 2*self.margin


Short solution needed: "Check if key exists in Redis" (python-redis)

Please help us write most modern and shortest code solution for this issue:
Check if key exists in Redis (technology: python-redis)
Fast way
Just write the code solution in the comments.
Prefered way

Create pull request with a new code file inside inbox folder.
Don't forget to use comments to make solution explained.
Link to this issue in comments of pull request.

Check if key exists in Redis


Bug in extension tab

VSCode Version:1.26.1
Commit: 493869ee8e8a846b0855873886fc79d480d342de
Date: 2018-08-16T18:38:57.434Z
Electron: 2.0.5
Chrome: 61.0.3163.100
Node.js: 8.9.3
V8: 6.1.534.41
Architecture: x64
OS Version: Windows 10 1803

Steps to Reproduce:
Please see this gif picture:

Does this issue occur when all extensions are disabled?: No

@leslieeilsel I am not able to reproduce it. Are there any steps?

@sandy081

You can see vaguely that there are four numbers in the place where the problem occurs, 0010, which is the version number of the ‘Terminal’plugin. Maybe it has something to do with this.
As for the specific steps, I tested it normal this morning, which is very strange. I think it may be related to using a widescreen display while I am working. So, you can put this question on hold and reply to you after I verify it.
Sorry for my english.

@leslieeilsel Thanks for the response.
I would close it for now and wait for the steps to reproduce.


Add documentation about data breakpoint errors

This PR adds two md files about errors with managed data breakpoints
"unable to set data breakpoint" errors and hardware supported data
breakpoints.
error-unable-to-set-data-breakpoint.md:

Explains the errors messages you may get when using managed data
bps and how to either solve/workaround the issue

hard-ware-supported-data-breakpoints.md:

Displays a table on the number of supported hardware breakpoints and the limit of bytes it can watch

@mikejo5000

@WardenGnaw : Thanks for your contribution! The author(s) have been notified to review your proposed change. @waan

Microsoft authors of articles need to make updates through the private repository so the updates can be staged and validated by the current validation rules. After you review staging and fix all validation issues, you can add the sign-off comment to let the PR reviewers know the updates are ready to be merged.It's easy to access the private repo version of your articles in GitHub. Just add -pr to the URL, as follows:
Public repo URL: https://github.com/Microsoft/azure-docs/blob/master/articles/azure-portal-overview.md
Private repo URL: https://github.com/Microsoft/azure-docs-pr/blob/master/articles/azure-portal-overview.md
We have closed this pull request. Please resubmit the updates through the private repo. If you need perms to the private repo, see the contributor guide instructions


The SDK 'Microsoft.NET.Sdk' specified could not be found.

Hi! I'm trying to build the project using Visual Studio, but I'm getting these errors while trying to open it.
C:\Users\user\Desktop\Parse-SDK-dotNET-master\Parse\Parse.csproj : error  : The SDK 'Microsoft.NET.Sdk' specified could not be found.  C:\Users\user\Desktop\Parse-SDK-dotNET-master\Parse\Parse.csproj
C:\Users\user\Desktop\Parse-SDK-dotNET-master\Parse.Test\Parse.Test.csproj : error  : The SDK 'Microsoft.NET.Sdk' specified could not be found.  C:\Users\user\Desktop\Parse-SDK-dotNET-master\Parse.Test\Parse.Test.csproj
Originally posted by @WalGallen-Baktun in https://github.com/parse-community/Parse-SDK-dotNET/issues/294#issuecomment-441079930
EDIT: moved to a new issue as it is not directly related to the original topic

@WalGallen-Baktun I will be closing this issue now. If needed, you can request for it to be reopened for you.


KOGITO-4588: DataType concept in the editor should cover both "custom" and "out-of-the-box" data types

Data types should come from the DMN editor. So the React component will take a series of default + custom data types as input.

@ljmotta, this task moves data types retrieval from internal (enum) to external (component input).
AFAIU, it should not impact DMN Runner Table. But if it is causing any issues, please let me know.

jenkins retest this please

jenkins execute downstream

jenkins retest this please

jenkins retest this please

Manual checks todo list

[ ] open a diagram that was created a third party editor
[ ] change an existing type reference
[ ] create new data type definition, set node reference to the new data type created recently
[ ] use an imported data type
[ ] remove a data type that is used in graph
[ ] work in parallel with two nodes
[ ] keyboard to select a custom data type
[ ] sync context entries with custom data type

01 scroll to position of selected
with custom types support, the amount of items in dropdown can be much bigger than previously.
Please compare the user experience in old editor

with new editor

In the old editor, more items seems to be visible, including the currently selected one, can be this addressed somehow? If yes, as part of this PR or a separate ticket is preferred?

02 keyboard control
An issue when filtering types found
https://issues.redhat.com/browse/KOGITO-6451

02 keyboard control
An issue when filtering types found https://issues.redhat.com/browse/KOGITO-6451

Yes, I also noticed this issue. But, to be honest, I am not sure how and why this happens.
It was also present in the selector before introducing the custom data types.
The selector component itself provides keyboard navigability.
It does not reproduce the example provided for the Selector components (https://patternfly-react.surge.sh/components/select/#grouped-single-with-filtering)

01 scroll to position of selected
with custom types support, the amount of items in dropdown can be much bigger than previously.
Please compare the user experience in old editor
with new editor
In the old editor, more items seems to be visible, including the currently selected one, can be this addressed somehow? If yes, as part of this PR or a separate ticket is preferred?

It seems that the scroll functionality is not provided by the PF component (https://patternfly-react.surge.sh/components/select/)
I think that a different component must be used or a custom operation should be performed.
If this is a defect, it should be definitely traced separately.
Regarding the height of the selector menu, now it is 350 (https://github.com/vpellegrino/kogito-editors-java/blob/d707042c8988879a3867eebf9331584c239fa38f/kogito-editors-js/packages/boxed-expression-component/src/components/EditExpressionMenu/DataTypeSelector.tsx#L114) we can increase it if you prefer. Any suggestion?

03 BKM nodes
I am not able to save custom type for a BKM node. After save and reopen it is reset to <Undefined>.

It is not related to this PR.
There is defect traced for it: https://issues.redhat.com/browse/KOGITO-6449
It is a regression caused by Undo/redo feature.

04 sync context entry
Found an issue when trying to syn context entry https://issues.redhat.com/browse/KOGITO-6452

re 02
Ok, so can we increase so at least Custom group is visible partially. My intention is user is aware there are more entries. What do you think?

re 02
Ok, so can we increase so at least Custom group is visible partially. My intention is user is aware there are more entries. What do you think?

Ok, setting it to 500.

Additional test coverage created as https://github.com/kiegroup/kogito-tooling/pull/721/files

@ljmotta I pushed all commits implementing your suggestions.
@jomarko I solved all issues related to the keyboard navigability of the data types selector. Now we have a dedicated section where you can type and search. It looks very similar to the selector in the legacy implementation.
Please have a look at it.
I pushed one proposal to make the CUSTOM section visible. I have inverted the order: CUSTOM then DEFAULT. I'm expecting an experienced user who already knows CUSTOM data types to notice the existence of DEFAULT data types. WDYT? If you prefer, I can switch back to the classic order: DEFAULT then CUSTOM . Please let me know!

@vpellegrino I think we should keep DEFAULT first  and then CUSTOM because:

I tend to think that was proposed by UX in past, but I do not have a data to confirm this assumption
Imagine there is 30, 50 or more custom data types and you decide to add one more data type of single string. Then you need to scroll or to filter. The DEFAULT group size is static, that is why I think is better to have ot before CUSTOM.

@jomarko ok, I reverted my proposal. Thank you.

I just produced the .vsix with all commits belonging to this branch. Hope it helps.

jenkins retest this please

@vpellegrino thank you. As you can see I approved the PR already. in the VSIX you provided. once I expand types dropdown, CUSTOM is not visible for me. My screen resolution is: 1920 x 1080

The (build) kogito-tooling-editors-java check has failed. Please check the logs.

The (build) kogito-tooling-editors-java check has failed. Please check the logs.

Failure is due to:
[2021-12-23T16:33:06.316Z] [INFO] boxed-expression-component: $ yarn test && webpack --config webpack.production.js
[2021-12-23T16:33:06.316Z] [INFO] boxed-expression-component-showcase: $ yarn test:it
[2021-12-23T16:33:06.316Z] [INFO] boxed-expression-component: $ jest --collect-coverage
[2021-12-23T16:33:06.316Z] [INFO] boxed-expression-component-showcase: $ start-server-and-test start:it http://localhost:3000 cy:run
[2021-12-23T16:33:07.761Z] [INFO] boxed-expression-component-showcase: 1: starting server using command "npm run start:it"
[2021-12-23T16:33:07.761Z] [INFO] boxed-expression-component-showcase: and when url "[ 'http://localhost:3000' ]" is responding with HTTP status code 200
[2021-12-23T16:33:07.761Z] [INFO] boxed-expression-component-showcase: running tests using command "npm run cy:run"
[2021-12-23T16:33:09.980Z] [INFO] boxed-expression-component-showcase: ><EMAIL_ADDRESS>start:it
[2021-12-23T16:33:09.980Z] [INFO] boxed-expression-component-showcase: > BROWSER=none yarn start
[2021-12-23T16:33:09.980Z] [INFO] boxed-expression-component-showcase: $ react-app-rewired start
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: ℹ ｢wds｣: Project is running at http://<IP_ADDRESS>/
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: ℹ ｢wds｣: webpack output is served from
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: ℹ ｢wds｣: Content not from webpack is served from /home/jenkins/workspace/KIE/kogito/main/pullrequest/kogito-editors-java.tests.kogito-tooling-editors-java/kogito-editors-java/kogito-editors-js/packages/boxed-expression-component/showcase/public
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: ℹ ｢wds｣: 404s will fallback to /
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: Starting the development server...
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: node:events:342
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:       throw er; // Unhandled 'error' event
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:       ^
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: Error: ENOSPC: System limit for number of file watchers reached, watch '/home/jenkins/workspace/KIE/kogito/main/pullrequest/kogito-editors-java.tests.kogito-tooling-editors-java/kogito-editors-java/kogito-editors-js/packages/boxed-expression-component/showcase/public'
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at FSWatcher.<computed> (node:internal/fs/watchers:244:19)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at Object.watch (node:fs:2240:34)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at createFsWatchInstance (/home/jenkins/workspace/KIE/kogito/main/pullrequest/kogito-editors-java.tests.kogito-tooling-editors-java/kogito-editors-java/kogito-editors-js/node_modules/chokidar/lib/nodefs-handler.js:38:15)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at setFsWatchListener (/home/jenkins/workspace/KIE/kogito/main/pullrequest/kogito-editors-java.tests.kogito-tooling-editors-java/kogito-editors-java/kogito-editors-js/node_modules/chokidar/lib/nodefs-handler.js:81:15)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at FSWatcher.NodeFsHandler._watchWithNodeFs (/home/jenkins/workspace/KIE/kogito/main/pullrequest/kogito-editors-java.tests.kogito-tooling-editors-java/kogito-editors-java/kogito-editors-js/node_modules/chokidar/lib/nodefs-handler.js:233:14)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at FSWatcher.NodeFsHandler._handleDir (/home/jenkins/workspace/KIE/kogito/main/pullrequest/kogito-editors-java.tests.kogito-tooling-editors-java/kogito-editors-java/kogito-editors-js/node_modules/chokidar/lib/nodefs-handler.js:429:19)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at FSWatcher.<anonymous> (/home/jenkins/workspace/KIE/kogito/main/pullrequest/kogito-editors-java.tests.kogito-tooling-editors-java/kogito-editors-java/kogito-editors-js/node_modules/chokidar/lib/nodefs-handler.js:477:19)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at FSWatcher.<anonymous> (/home/jenkins/workspace/KIE/kogito/main/pullrequest/kogito-editors-java.tests.kogito-tooling-editors-java/kogito-editors-java/kogito-editors-js/node_modules/chokidar/lib/nodefs-handler.js:482:16)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at FSReqCallback.oncomplete (node:fs:195:5)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: Emitted 'error' event on FSWatcher instance at:
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at FSWatcher._handleError (/home/jenkins/workspace/KIE/kogito/main/pullrequest/kogito-editors-java.tests.kogito-tooling-editors-java/kogito-editors-java/kogito-editors-js/node_modules/chokidar/index.js:260:10)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at createFsWatchInstance (/home/jenkins/workspace/KIE/kogito/main/pullrequest/kogito-editors-java.tests.kogito-tooling-editors-java/kogito-editors-java/kogito-editors-js/node_modules/chokidar/lib/nodefs-handler.js:40:5)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at setFsWatchListener (/home/jenkins/workspace/KIE/kogito/main/pullrequest/kogito-editors-java.tests.kogito-tooling-editors-java/kogito-editors-java/kogito-editors-js/node_modules/chokidar/lib/nodefs-handler.js:81:15)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     [... lines matching original stack trace ...]
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at FSReqCallback.oncomplete (node:fs:195:5) {
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:   errno: -28,
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:   syscall: 'watch',
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:   code: 'ENOSPC',
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:   path: '/home/jenkins/workspace/KIE/kogito/main/pullrequest/kogito-editors-java.tests.kogito-tooling-editors-java/kogito-editors-java/kogito-editors-js/packages/boxed-expression-component/showcase/public',
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:   filename: '/home/jenkins/workspace/KIE/kogito/main/pullrequest/kogito-editors-java.tests.kogito-tooling-editors-java/kogito-editors-java/kogito-editors-js/packages/boxed-expression-component/showcase/public'
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: }
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: error Command failed with exit code 1.
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: Error: server closed unexpectedly
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at ChildProcess.onClose (/home/jenkins/workspace/KIE/kogito/main/pullrequest/kogito-editors-java.tests.kogito-tooling-editors-java/kogito-editors-java/kogito-editors-js/node_modules/start-server-and-test/src/index.js:75:14)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at ChildProcess.emit (node:events:365:28)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at maybeClose (node:internal/child_process:1067:16)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase:     at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: error Command failed with exit code 1.
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: error Command failed with exit code 1.boxed-expression-component-showcase: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
[2021-12-23T16:33:37.887Z] [INFO] boxed-expression-component-showcase: error Command failed with exit code 1.
[2021-12-23T16:33:37.887Z] [INFO] lerna ERR! yarn run build:prod exited 1 in 'boxed-expression-component-showcase'

https://eng-jenkins-csb-business-automation.apps.ocp-c1.prod.psi.redhat.com/blue/rest/organizations/jenkins/pipelines/KIE/pipelines/kogito/pipelines/main/pipelines/pullrequest/pipelines/kogito-editors-java.tests.kogito-tooling-editors-java/runs/426/nodes/95/steps/112/log/?start=0
It seems that issue is related to Node (Error: ENOSPC: System limit for number of file watchers reached). @tiagobento @jomarko what is your opinion there?

The (build) kogito-tooling-editors-java check has failed. Please check the logs.

@kiegroup/gatekeepers can you please merge? It seems that FDB is not available right now

@vpellegrino FDB is available. Github actions runners has 6 hours limitation, they are cancelled after that time (no matter what).
in this case the job is stuck at kogito-editors-java/kogito-editors-java/kogito-editors-java/kogito-editors-js tests
see
2021-12-23T21:49:14.3455103Z [INFO] dmn-loader: WARNING in webpack performance recommendations:
2021-12-23T21:49:14.3456339Z [INFO] dmn-loader: You can limit the size of your bundles by using import() or require.ensure to lazy load some parts of your application.
2021-12-23T21:49:14.3457634Z [INFO] dmn-loader: For more info visit https://webpack.js.org/guides/code-splitting/
2021-12-23T21:49:14.3458855Z [INFO] dmn-loader: webpack 5.41.1 compiled with 3 warnings in 90260 ms
2021-12-24T03:27:02.3619705Z ##[error]The operation was canceled.
2021-12-24T03:27:02.3970277Z Post job cleanup.
2021-12-24T03:27:02.5837029Z Post job cleanup.

dmn-loader: webpack 5.41.1 compiled with 3 warnings in 90260 ms message is at 2021-12-23T21:49:14 and the operation is cancelled at 2021-12-24T03:27:02
I will relaunch job to check whether it happens the same or not

@Ginxo thanks for the explanation.
Moreover, there are intermittent CI issues: https://github.com/kiegroup/kogito-editors-java/pull/196#issuecomment-1000424045


truvari collapse- inversions and duplications

Hello,
Thanks for this tool! I am using truvari to collapse SVs and I'm using --pctovl 0.5 to specify a 50% reciprocal overlap between deletions. Is there a reason that this isn't applied to inversions and duplications? From what I can tell, inversion and duplication matching is only restricted by the --refdist  and --pctsize (as with insertions).
Thanks,
Madeline

Hello,
Truvari was initially built for use with sequence-resolved breakpoint-exact DEL and INS, which meant the INS had no span over the reference. Only recently (f0a608877dbbf4c4dee13dbfad3fc52e3afb4b20) in v3.1 did we add a boundary inflation for INS. But I did accidentally leave the line which restricts matching to DEL.
I'll fix this and have it slated for the next release.

Please install from develop if you'd like to use the new fix (see wiki for details]. I've been planning to cut a v3.2.0 release on Apr 1st, so your changes will be available to install via pip at the end of the week.

The change is that --recovl is applied to all SV pairs during Matching. So yes, it will be applied to INV and DUPs as well. However, it will also be applied to INS. With the INS boundary changes mentioned above, it seems more faithful to the parameter description and general user expectations for the parameter to be applied to all SV pairs.
I will note that in our paper we found that reciprocal overlap isn't a very robust matching criteria for sequence-resolved, breakpoint exact SVs, which is why the default remains at 0.

Interestingly, 402 SV pairs have ≤5% reciprocal-overlap but ≥70% sequence and size similarity. These pairs may indicate alignment ambiguities across repetitive regions (e.g. left-shift vs. right-shift).

Got it, thank you! Looking forward to the next release. Point taken re: robustness of matching criteria, though- will take this into consideration.


VS Code - Diagram editor:  There is a white blink on each diagram update

On SWF VsCode plugin there is white blink on every canvas update. It also happens with the light theme but it is more evident on the dark theme.
Note: check the VsCodeKieEditorController.ts file, it seems there are fixed background colors that may be one of the causes of the white flash.
Steps to reproduce:

Open a valid swf diagram
Edit any property that does not break the diagram such as "description"
Note that on each change the canvas is reloaded, however, in between the reloads  and the rendered diagram it is possible to see a white background behind the canvas

@handreyrc Hey! I've seen that too but couldn't get a stable reproducer. @Josephblt and I noticed it on the BPMN Editor as well. Can you provide steps to reproduce please?

@tiagobento I'm not sure it is the same issue or at least the way to reproduce it is not applicable to BMPN. On BPMN editor, if my memory is not tricking me,  we to not edit the XML directly so we don't have full reloads of the diagram once it's open, an incremental refresh is performed following commands execution context instead.
Anyway, I updated the description of the issue above with the steps to reproduce the issue.
@Josephblt Maybe you can perform a quick test to confirm it.  If you change the white background color to the same background color of the BPMN canvas in the file I mentioned above and the issue is gone that's it.
Please, let me know.


In-memory graph population is very slow when running with Postgres backend

Background
I'm testing LND with Postgres locally and it seems to be running strangely slow. Database opens take 2-3 minutes before I can unlock. Running a getinfo command can take minutes to return. However, later on the getinfo calls take the normal 600ms. This isn't some slight performance hit, it's a substantial slow down so I'm wondering if I'm doing something wrong or there some other weird issue.
This is all running locally with Postgres in a Docker container. LND is using Neutrino as its chain backend. An additionally odd thing is I don't actually see LND doing operations against Postgres. It can definitely connect and it creates tables and value, but I don't see active connections hardly ever. I'm using this query to find this information:
SELECT * FROM pg_stat_activity WHERE datname = 'postgres' and state = 'active';

However, I do see some idle connections.
Command I'm running
lnd --bitcoin.active --bitcoin.node=neutrino --bitcoin.mainnet --lnddir=/Users/me/lnd --neutrino.feeurl=https://nodes.lightning.computer/fees/v1/btc-fee-estimates.json --db.backend=postgres --db.postgres.dsn=postgres://postgres:password@localhost:5432/postgres?sslmode=disable --db.postgres.maxconnections=1000

Logs
2021-12-21 06:46:52.978 [INF] CHDB: Checking for schema update: latest_version=24, db_version=24
2021-12-21 06:46:52.978 [INF] LTND: Database(s) now open (time_to_open=3m6.356551208s)!

GetInfo Command
$ time lncli getinfo
{
    "version": "0.14.1-beta commit=v0.14.1-beta",
    ...
}

real	1m40.783s
user	0m0.038s
sys	0m0.036s

Your environment

LND 0.14.1
MacOS
Neutrino
Postgres v14.1

Steps to reproduce
Run Postgres in a docker container locally and point LND at it and start it up.
Expected behaviour
I would expect LND to behavior similarly to bbolt in this scenario. Sure there's some differences in how it interacts with the database, but this is a large slow down.
Actual behaviour
Things get really slow.

Can you paste a few more logs lines, starting from Opening the main database, this might take a few minutes...  (this is where the timer starts)?
I tried to repro with lnd v0.14.1 and pg v14.1, but got Database(s) now open (time_to_open=25.087161ms)!. Didn't use neutrino and no macos either. Also using a fresh node on regtest.

Sure. Later today I can try to connect it to a bitcoin node instead of neutrino just to see what happens.
2021-12-21 06:56:44.776 [WRN] LTND: open /Users/gkrizek/lnd/lnd.conf: no such file or directory
2021-12-21 06:56:44.776 [INF] LTND: Version: 0.14.1-beta commit=v0.14.1-beta, build=production, logging=default, debuglevel=info
2021-12-21 06:56:44.776 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2021-12-21 06:56:44.776 [INF] RPCS: RPC server listening on <IP_ADDRESS>:10009
2021-12-21 06:56:44.777 [INF] RPCS: gRPC proxy started at <IP_ADDRESS>:8080
2021-12-21 06:56:44.777 [INF] LTND: Opening the main database, this might take a few minutes...
2021-12-21 06:56:44.814 [INF] LTND: Using remote postgres database! Creating graph and channel state DB instances
2021-12-21 06:59:52.093 [INF] CHDB: Checking for schema update: latest_version=24, db_version=24
2021-12-21 06:59:52.093 [INF] LTND: Database(s) now open (time_to_open=3m7.31988175s)!
2021-12-21 06:59:52.651 [INF] BTCN: Loaded 1626 addresses from file '/Users/gkrizek/lnd/data/chain/bitcoin/mainnet/peers.json'
2021-12-21 06:59:52.651 [INF] BTCN: Registering block subscription: id=1
2021-12-21 06:59:52.651 [INF] BTCN: Broadcaster now active
2021-12-21 06:59:52.682 [INF] LTND: We're not running within systemd
2021-12-21 06:59:52.682 [INF] LTND: Waiting for wallet encryption password. Use `lncli create` to create a wallet, `lncli unlock` to unlock an existing wallet, or `lncli changepassword` to change the password of an existing wallet and unlock it.
2021-12-21 06:59:52.773 [INF] BTCN: 4 addresses found from DNS seed x49.dnsseed.bluematt.me
2021-12-21 06:59:53.129 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 715091...
2021-12-21 06:59:53.129 [INF] BTCN: New valid peer <IP_ADDRESS>:8333 (outbound) (/Satoshi:22.0.0/)
2021-12-21 06:59:53.129 [INF] BTCN: Starting cfheaders sync from (block_height=715091, block_hash=000000000000000000063c0b5875484dab090035e176e56a9c294648ea3e39db) to (block_height=715091, block_hash=000000000000000000063c0b5875484dab090035e176e56a9c294648ea3e39db)
2021-12-21 06:59:53.129 [INF] BTCN: Starting cfheaders sync for filter_type=0
2021-12-21 06:59:53.132 [INF] BTCN: Syncing to block height 715091 from peer <IP_ADDRESS>:8333
2021-12-21 06:59:53.132 [INF] BTCN: Fetching set of headers from tip (height=715091) from peer <IP_ADDRESS>:8333
2021-12-21 06:59:53.219 [INF] BTCN: New valid peer <IP_ADDRESS>:8333 (outbound) (/Satoshi:22.0.0/)
2021-12-21 06:59:53.255 [INF] BTCN: New valid peer <IP_ADDRESS>:8333 (outbound) (/Satoshi:22.0.0/)
2021-12-21 06:59:53.341 [INF] BTCN: New valid peer <IP_ADDRESS>:8333 (outbound) (/Satoshi:22.0.0/)
2021-12-21 06:59:53.426 [WRN] BTCN: Banning peer <IP_ADDRESS>:8333: duration=48h0m0s, reason=peer was unable to serve compact filters
2021-12-21 06:59:53.461 [INF] BTCN: 31 addresses found from DNS seed x49.seed.bitcoin.jonasschnelli.ch
2021-12-21 06:59:53.465 [INF] BTCN: Lost peer <IP_ADDRESS>:8333 (outbound)
2021-12-21 06:59:53.481 [INF] BTCN: 36 addresses found from DNS seed dnsseed.bitcoin.dashjr.org
2021-12-21 06:59:53.509 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=715091, hash=e12d8dc1ba3c21455218f5c37260be60bc27a444114cc499070c58cac2a0f1a4
2021-12-21 06:59:53.509 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=715, checkpoints=715
2021-12-21 06:59:53.509 [INF] BTCN: Fully caught up with cfheaders at height 715091, waiting at tip for new blocks
2021-12-21 06:59:53.599 [INF] BTCN: New valid peer <IP_ADDRESS>:8333 (outbound) (/Satoshi:22.0.0/)
2021-12-21 06:59:53.644 [INF] BTCN: New valid peer <IP_ADDRESS>:8333 (outbound) (/Satoshi:22.0.0/)
2021-12-21 06:59:53.832 [INF] BTCN: 40 addresses found from DNS seed seed.bitnodes.io
2021-12-21 06:59:53.870 [INF] BTCN: New valid peer <IP_ADDRESS>:8333 (outbound) (/Satoshi:22.0.0/)
2021-12-21 06:59:54.083 [INF] BTCN: Lost peer <IP_ADDRESS>:8333 (outbound)
2021-12-21 06:59:54.083 [INF] BTCN: Syncing to block height 715091 from peer <IP_ADDRESS>:8333
2021-12-21 06:59:54.083 [INF] BTCN: Fetching set of headers from tip (height=715091) from peer <IP_ADDRESS>:8333

And is this with an empty database?

This node was brand new on Postgres. Those logs are from a start up where the tables existed and had some graph data in them and such. Basically this node as started up previously but no channels or funds.

And when you started it for the very first time without any graph data, was it also slow?

Also wondering why these two lines don't show up for you
2021-12-21 14:47:20.279 [DBG] CHDB: Populating in-memory channel graph, this might take a while...
2021-12-21 14:47:25.804 [DBG] CHDB: Finished populating in-memory channel graph (took 5.524251789s, num_node_features=1324, num_nodes=1316, num_channels=6030)

Right, you are probably on debug level info.
Switching to debug may be useful here. I suspect that populating the channel graph cache may be the culprit.

I am seeing the same thing with my "test" node which is on mainnet.
My environment:

lnd 0.14.1-beta
PostgreSQL 14.1 on a remote node
bitcoind 22.0 on a remote node

It takes a long time to start-up, but once up it works ok.  Creating graph and channel state DB instances
2021-12-21 22:10:25.820 [DBG] CHDB: Populating in-memory channel graph, this might take a while...
2021-12-21 22:19:55.322 [DBG] CHDB: Finished populating in-memory channel graph (took 9m29.499293195s, num_node_features=19307, num_nodes=19062, num_channels=164494)
2021-12-21 22:19:55.359 [INF] CHDB: Checking for schema update: latest_version=24, db_version=24
2021-12-21 22:19:55.359 [INF] LTND: Database(s) now open (time_to_open=9m29.729770273s)!
2021-12-21 22:19:55.363 [INF] LTND: Systemd was notified about our readiness
2021-12-21 22:19:55.363 [INF] LTND: Waiting for wallet encryption password. Use `lncli create` to create a wallet, `lncli unlock` to unlock an existing wallet, or `lncli changepassword` to change the password of an existing wallet and unlock it.
2021-12-21 22:20:26.060 [INF] LNWL: Opened wallet
2021-12-21 22:20:26.477 [INF] CHRE: Primary chain is set to: bitcoin
2021-12-21 22:20:27.314 [INF] LNWL: Started listening for bitcoind block notifications via ZMQ on [redacted]:28332
2021-12-21 22:20:27.316 [INF] LNWL: Started listening for bitcoind transaction notifications via ZMQ on [redacted]:28333
2021-12-21 22:20:27.356 [INF] CHRE: Initializing bitcoind backed fee estimator in ECONOMICAL mode
2021-12-21 22:20:29.155 [INF] LNWL: The wallet has been unlocked without a time limit
2021-12-21 22:20:29.181 [INF] CHRE: LightningWallet opened
2021-12-21 22:20:29.188 [INF] SRVR: Proxying all network traffic via Tor (stream_isolation=true)! NOTE: Ensure the backend node is proxying over Tor as well
2021-12-21 22:20:29.188 [INF] TORC: Starting tor controller
2021-12-21 22:20:29.194 [DBG] LNWL: Birthday block has already been verified: height=710459, hash=000000000000000000072a0fdd5acb02104b528060158240c1f0ace231014b72
2021-12-21 22:20:29.195 [DBG] LNWL: Waiting for chain backend to sync to tip
2021-12-21 22:20:29.516 [DBG] TORC: received protocol info: map[COOKIEFILE:"/run/tor/control.authcookie" METHODS:COOKIE,SAFECOOKIE Tor:"<IP_ADDRESS>"]
2021-12-21 22:20:29.603 [INF] HSWC: Cleaning circuits from disk for closed channels
2021-12-21 22:20:29.607 [DBG] HSWC: Found 0 closed channels
2021-12-21 22:20:29.607 [INF] HSWC: Finished cleaning: no closed channels found, no actions taken.
2021-12-21 22:20:29.607 [INF] HSWC: Restoring in-memory circuit state from disk
2021-12-21 22:20:29.612 [INF] HSWC: Payment circuits loaded: num_pending=0, num_open=0
2021-12-21 22:20:29.635 [INF] HSWC: Trimming open circuits for chan_id=713013:2009:0, start_htlc_id=2
2021-12-21 22:20:29.902 [DBG] CRTR: Instantiating mission control with config: Penalty Half Life: 1h0m0s, Apriori Hop Probablity: 0.6, Maximum History: 1000, Apriori Weight: 0.5, Minimum Failure Relax Interval: 1m0s
2021-12-21 22:20:29.908 [DBG] CRTR: Mission control state reconstruction started
2021-12-21 22:20:29.914 [DBG] CRTR: Reporting pair success to Mission Control: pair=03c4250004e0035c8ea9e78a8a44d92657982b0fec55d2d0c61d675e8b5679b4b0 -> 02c911ba09b21441200e0424279720ec13290b36cd51f4ab1a80cbb40b3baf0fe1, amt=10000000 mSAT
2021-12-21 22:20:29.914 [DBG] CRTR: Setting 03c4250004e0035c8ea9e78a8a44d92657982b0fec55d2d0c61d675e8b5679b4b0->02c911ba09b21441200e0424279720ec13290b36cd51f4ab1a80cbb40b3baf0fe1 range to [10000000 mSAT-0 mSAT]
2021-12-21 22:20:29.914 [DBG] CRTR: Reporting pair success to Mission Control: pair=03c4250004e0035c8ea9e78a8a44d92657982b0fec55d2d0c61d675e8b5679b4b0 -> 02c911ba09b21441200e0424279720ec13290b36cd51f4ab1a80cbb40b3baf0fe1, amt=9972000 mSAT
2021-12-21 22:20:29.914 [DBG] CRTR: Setting 03c4250004e0035c8ea9e78a8a44d92657982b0fec55d2d0c61d675e8b5679b4b0->02c911ba09b21441200e0424279720ec13290b36cd51f4ab1a80cbb40b3baf0fe1 range to [10000000 mSAT-0 mSAT]
2021-12-21 22:20:29.914 [DBG] CRTR: Mission control state reconstruction finished: n=2, time=5.881836ms
2021-12-21 22:20:29.914 [DBG] SRVR: Instantiating payment session source with config: AttemptCost=100 + 0.1%, MinRouteProbability=0.01
2021-12-21 22:20:29.950 [DBG] CHBU: Crafting backup for ChannelPoint([redacted]:0)
2021-12-21 22:20:29.957 [INF] LTND: Channel backup proxy channel notifier starting
2021-12-21 22:20:29.957 [INF] ATPL: Instantiating autopilot with active=false, max_channels=5, allocation=0.600000, min_chan_size=20000, max_chan_size=16777215, private=false, min_confs=1, conf_target=3
2021-12-21 22:20:29.960 [DBG] LTND: Populating dependencies for sub RPC server: SignRPC
2021-12-21 22:20:29.960 [DBG] LTND: Populating dependencies for sub RPC server: WalletKitRPC
2021-12-21 22:20:29.960 [DBG] LTND: Populating dependencies for sub RPC server: AutopilotRPC
2021-12-21 22:20:29.960 [DBG] LTND: Populating dependencies for sub RPC server: ChainRPC
2021-12-21 22:20:29.961 [DBG] LTND: Populating dependencies for sub RPC server: InvoicesRPC
2021-12-21 22:20:29.961 [DBG] LTND: Populating dependencies for sub RPC server: RouterRPC
2021-12-21 22:20:29.961 [DBG] LTND: Populating dependencies for sub RPC server: WatchtowerRPC
2021-12-21 22:20:29.961 [DBG] LTND: Populating dependencies for sub RPC server: WatchtowerClientRPC
2021-12-21 22:20:29.961 [DBG] RPCS: Starting sub RPC server: VersionRPC
2021-12-21 22:20:29.962 [DBG] RPCS: Starting sub RPC server: WatchtowerRPC
2021-12-21 22:20:29.962 [DBG] RPCS: Starting sub RPC server: SignRPC
2021-12-21 22:20:29.962 [DBG] RPCS: Starting sub RPC server: WalletKitRPC
2021-12-21 22:20:29.962 [DBG] RPCS: Starting sub RPC server: RouterRPC
2021-12-21 22:20:29.962 [DBG] RPCS: Starting sub RPC server: AutopilotRPC
2021-12-21 22:20:29.962 [DBG] RPCS: Starting sub RPC server: ChainRPC
2021-12-21 22:20:29.962 [DBG] RPCS: Starting sub RPC server: InvoicesRPC
2021-12-21 22:20:29.962 [DBG] RPCS: Starting sub RPC server: WatchtowerClientRPC
2021-12-21 22:20:29.962 [INF] LTND: Systemd was notified about our readiness
2021-12-21 22:20:29.966 [INF] LTND: Waiting for chain backend to finish sync, start_height=715100
2021-12-21 22:20:30.203 [DBG] LNWL: Chain backend synced to tip!
2021-12-21 22:20:30.307 [INF] LNWL: Started rescan from block<PHONE_NUMBER>0000000000f07e69240974837f29e0b9b9abdd6353f4f8da932127 (height 715099) for 0 addresses
2021-12-21 22:20:30.807 [INF] LNWL: Catching up block hashes to height 715100, this might take a while
2021-12-21 22:20:30.809 [INF] LNWL: Done catching up block hashes
2021-12-21 22:20:30.809 [INF] LNWL: Finished rescan for 0 addresses (synced to block<PHONE_NUMBER>000000000134a9e44e577fdd2cee468a30f8b775e8d435037def95, height 715100)
2021-12-21 22:20:30.983 [INF] LTND: Chain backend is fully synced (end_height=715100)!
2021-12-21 22:20:30.983 [WRN] HLCK: check: disk space configured with 0 attempts, skipping it
2021-12-21 22:20:30.983 [WRN] HLCK: check: tls configured with 0 attempts, skipping it
2021-12-21 22:20:30.983 [WRN] HLCK: check: tor connection configured with 0 attempts, skipping it
2021-12-21 22:20:30.983 [DBG] HLCK: Monitoring: chain backend
2021-12-21 22:20:31.004 [INF] PRNF: PeerNotifier starting
2021-12-21 22:20:31.009 [INF] NTFN: New block epoch subscription
2021-12-21 22:20:31.020 [INF] NTFN: New block epoch subscription
2021-12-21 22:20:31.026 [DBG] BRAR: Found 0 closing channels, 0 retribution records

Finished populating in-memory channel graph (took 9m29.499293195s, num_node_features=19307, num_nodes=19062, num_channels=164494)
That concludes it indeed.

yep, that's it.
2021-12-21 09:42:44.878 [INF] RPCS: gRPC proxy started at <IP_ADDRESS>:8080
2021-12-21 09:42:44.878 [INF] LTND: Opening the main database, this might take a few minutes...
2021-12-21 09:42:44.914 [INF] LTND: Using remote postgres database! Creating graph and channel state DB instances
2021-12-21 09:42:44.942 [DBG] CHDB: Populating in-memory channel graph, this might take a while...
2021-12-21 09:49:42.565 [DBG] CHDB: Finished populating in-memory channel graph (took 6m57.630952208s, num_node_features=12204, num_nodes=12108, num_channels=83810)
2021-12-21 09:49:42.619 [INF] CHDB: Checking for schema update: latest_version=24, db_version=24
2021-12-21 09:49:42.619 [INF] LTND: Database(s) now open (time_to_open=6m57.750526917s)!
2021-12-21 09:49:43.173 [INF] BTCN: Loaded 1701 addresses from file '/Users/gkrizek/lnd/data/chain/bitcoin/mainnet/peers.json'
2021-12-21 09:49:43.173 [DBG] BTCN: Starting block notifications subscription manager
2021-12-21 09:49:43.173 [DBG] BTCN: Waiting for peer connection...
2021-12-21 09:49:43.173 [INF] BTCN: Registering block subscription: id=1

Seems like this is a problem even with 1000 connections. Do we know the culprit?

Yes, nof connections is irrelevant, because the whole graph is loaded from the db in a single thread.
One thing worth trying that is relatively straight-forward is to parallelize the loading process. These are all reads and can be handled by lnd and postgres concurrently.
If that doesn't have a sufficiently large effect, we may need to look at query efficiency. We've been discussing this in various prs already.

Is there a reason you can't disable the in memory graph cache with Postgres?

See my answer above. Pathfinding would become unacceptably slow.

@joostjager how did your node start up in 5 seconds (from your log output?) Is this a matter of compute resources? I also confirmed that I started up a new node with --numgraphsyncpeers=0 and it starts up in 14ms 😁

That was not a new node anymore. It was already partially synced and the cache loading started to impact startup time.

I tried to parallelize the channel fetching for each node, but didn't make much of a difference. The loading of the graph data is just not very efficient, enormous numbers of queries.

Thanks for looking into it. So seems like this will be a reality in some capacity?

It is a reality, but I want to look into the options to make it better. I'd say it is borderline-acceptable.

Two options that come to my mind:

Finish https://github.com/lightningnetwork/lnd/pull/5595, although my earlier concerns about the risk level of the write part in particular are still there. Maybe a version stripped down to read only is an option? Also the cache is populated using ForEach, which still uses an inefficient cursor underneath, so we'd need to test if the gains are sufficient.

Add ReadBucket.GetAll and rewrite the graph cache population code to use that. It would translate to efficient select key,value from bucket=x-style queries for the node, edge and edgeIndex buckets.

@guggero @bhandras what are your ideas?

Two options that come to my mind:

Finish channeldb: write through cache for the graph and channel state #5595, although my earlier concerns about the risk level of the write part in particular are still there. Maybe a version stripped down to read only is an option? Also the cache is populated using ForEach, which still uses an inefficient cursor underneath, so we'd need to test if the gains are sufficient.

Not sure if the channel state cache would help with the initial in-memory graph population.

Add ReadBucket.GetAll and rewrite the graph cache population code to use that. It would translate to efficient select key,value from bucket=x-style queries for the node, edge and edgeIndex buckets.

Something like GetAll could be useful. Perhaps paginated too.

From the above flame graph it's clear that the number of roundtrips to the DB is indeed a big issue. The channel cache mentioned in the previous comment is meant to cache the channel state for the node's own channels. This is helpful when sending a lot of payments but not so helpful when reading the whole graph.
We can add GetAll to the etcd implementation too to help with startup.

I thought #5595 might help because at least it is doing a flat read of all the keys in a bucket, but maybe it doesn't matter because it uses the same inefficient cursor.
Initially I implemented ForEach using a range query, but the problem is that in many places, we fetch other data within the ForEach callback. That doesn't work within the same transaction.
I can give GetAll a try, see how that works.

WDYT about reconsidering the Prefetch model? It'd work when fetching multiple keys inside the closure and can be applied to hot paths selectively.

Although I still prefer to add GetAll as it may doors for overriding the STM completely for etcd for large "atomic" fetches on startup.

I think my initial objection to Prefetch still applies. It doesn't work in conjunction with server-side transactions if you also update keys. Maybe Prefetch on a read-only bucket could work, but to me GetAll looks more transparent and safer.

I think my initial objection to Prefetch still applies. It doesn't work in conjunction with server-side transactions if you also update keys. Maybe Prefetch on a read-only bucket could work, but to me GetAll looks more transparent and safer.

Quick note: Prefetch is optional. Hint for the underlying implementation about the coming fetches. Does not implies the need to prefetch at that point nor does it mean the need to prefetch everything. You don't need to implement it for write txns if you don't consider it applicable either. It's optional. Consider it a hint.

I see, yes that would make it simpler. But still Prefetch also prefetches keys in child buckets. I guess those could be ignored too? Still I am hesitant to write the code. Will need to make sure that all the read and cursor functionality still behaves exactly the same when it reads from memory instead of the database. I think there is a considerable risk associated with adding this kind of functionality to the database driver.
My preference would be to do GetAll. It also moves the application code a bit closer to the model of the data stores. There must be lots of other uses of cursor that don't need to be a cursor really.

I was thinking of only adding GetAll on the bucket level. It returns a list of all key/value pairs in the bucket? Seems we don't need anything else to fix the in-memory graph problem.

I was thinking of only adding GetAll on the bucket level. It returns a list of all key/value pairs in the bucket? Seems we don't need anything else to fix the in-memory graph problem.

Easy enough, sgtm!

Created PoC PR: https://github.com/lightningnetwork/lnd/pull/6111
On my machine with 13k channels in the database, it reduces cache population time from 26 seconds down to 160 ms.

Another option would be to make the process async: so start up, but then block any caller on graph access until it's fully populated. This would at least let lnd start up and start normal operations, but would block any path finding or gossip modifications until complete.

However, later on the getinfo calls take the normal 600ms.

Ok I think @joostjager's PR above address the slowness in start up. Do you have any profiles for the slow GetInfo call? Assuming you're running with the channel graph cache, then GetInfo should be pretty quick as it just is reading out local channel information.

I think that sounds like a good idea as well re: making cache population async. Would at least let some things continue.
I don't have a profile of the GetInfo but I can try to make one later. I'm sure it's related to the graph cache population. As soon as that finishes GetInfo returns like normal.

I think that sounds like a good idea as well re: making cache population async. Would at least let some things continue.

I spun this out (making the graph population async) into a new issue: #61111


Handle openfoodfacts.org links (Add deeplinking towards the product page)

Who for

Potential users
Current users

What

We have many product pages on the web
There are in-browser mechanisms to propose installing the native app for those who visit the page in a browser, or directly from Search Engines
Allow Smoothie to handle openfoodfacts.org links

Design doc with links to implementations in the classic apps
Why

Convenient for existing users
Chrome might propose app install as a result after a few visits.
Google Search might propose app install in some conditions.

Part of

#2515

Mockup

Code pointer

https://docs.flutter.dev/development/ui/navigation/deep-linking

Status
At the moment we handle these urls:

[x] Android

[x] /product
[x] /additive
[x] /allergen
[x] /category
[x] /cgi/search.pl
[x] /contributor
[x] /country
[x] /label
[x] /manufacturing-place
[x] /origin
[x] /packaging
[x] /state
[x] /store

[ ] iOS:

[ ] ...

Note that we should support those URLs in plural (list of all existing additives), and in singular (list of products for a given additive)
Urls we'd like to support
Depends on

[x] https://github.com/openfoodfacts/smooth-app/issues/260

Note: Its not straight forward, we have to switch to either named routes or router 2.0

@teolemon Could you make the document publicly readable, please?

Fixed @g123k

On Android, here are all the supported links:

/product
/additive
/allergen
/category
/cgi/search.pl
/contributor
/country
/label
/manufacturing-place
/origin
/packaging
/state
/store

We run Google AdGrants campaigns (free Adwords for NGOs), and the console says deeplinking will improve conversion rates:

I would like this PR to be merged: https://github.com/flutter/packages/pull/2416
It will ease the migration.
Can we wait for it? Or should I start with a fork containing it?

Before any code is started, I would like to understand the goal, the strategy, and the impact of waiting/not waiting for https://github.com/flutter/packages/pull/2416 to be merged. I'm old-fashioned, sorry about that ;)
As far as I understand, the goal is to allow deep-linking, and the most relevant way to do that is "named routes" or "router 2.0". Btw are both solution equivalent? Is that just the same concept with a different name?
Does that mean that

all current Smoothie pages are supposed to be accessed via a named route, e.g. /history? Of course there are parameters too I guess, like /product/1234
we have to connect "manually" Smoothie named routes and the website URLs via some regexp?
same implementation on Android and iOS?
is there a way we can code that gradually (in successive PRs), like convert to named routes first, then to router 2.0 (if needed/relevant), then deep linking on Android, then deep-linking on iOS?

So let me explain: to achieve the goal of having deep-links, the best solution is to migrate to Navigator 2.0.
As we all know, Navigator 2.0 requires hundreds of lines of code and the use of external libraries is encouraged by Google itself. The most popular solution is from an ex-Google employee: go_router.
I had started a few weeks ago an implementation (never pushed, my fault), but I was soon stuck with codes like this:
var res = Navigator.of(context).push()
// Do something with res

Basically go_router doesn't allow receiving a result from a closing route. The only is by tricking and making the code totally awful and unreadable. And the PR I mentioned finally fixes that issue.

@g123k Thank you for your answer!
That's why I wanted to discuss the matter before any code writing: it looks like what causes problems is:

either the choice of Navigator 2.0
or the fact that, if we choose Navigator 2.0, we cannot pop/return a value (before the merge you indicated anyway)

My comments (a bit late in the evening to my standards) are:

perhaps there is one less problematic solution than Navigator 2.0 (I'm just speculating)
or perhaps we should rethink our current use of pop in order to return a value - either by not returning values and just notifying listeners, or by using a slightly different solution for specific pages that really need to return a value (that could be converted to dialogs?), a bit like what happened today with @simonbengtsson's WillPopScope where 2 specific solutions were implemented for the problematic 2 specific cases.

The first step would be to list the pop that return a value, and see what we could do with them.
Assuming that they can be converted to dialogs, we won't have to wait for the flutter merge.
And we can start migrating to Navigator 2.0, then (or simultaneously) deep-linking.
How does that sound?

Working on step one: transforming all await Navigator.push<MyObject?>( calls into await Navigator.push<void>( when the pop/return value is not used.
Current status:
% grep -r "await Navigator.push" *
cards/data_cards/image_upload_card.dart:          final bool? refreshed = await Navigator.push<bool>(
cards/product_cards/smooth_product_card_not_found.dart:                    final String? result = await Navigator.push<String>(
cards/product_cards/product_title_card.dart:                await Navigator.push<Product?>(
cards/product_cards/smooth_product_card_found.dart:            await Navigator.push<Widget>(
pages/scan/scan_header.dart:                      await Navigator.push<Widget>(
pages/scan/scan_product_card.dart:    await Navigator.push<Widget>(
pages/product/edit_product_page.dart:                      await Navigator.push<Product?>(
pages/product/edit_product_page.dart:                      final bool? refreshed = await Navigator.push<bool>(
pages/product/edit_product_page.dart:                      await Navigator.push<Product?>(
pages/product/edit_product_page.dart:                      await Navigator.push<Product?>(
pages/product/edit_product_page.dart:                      await Navigator.push<Product?>(
pages/product/edit_product_page.dart:        await Navigator.push<Product>(
pages/product/edit_product_page.dart:        await Navigator.push<Product>(
pages/product/product_image_gallery_view.dart:        final File? photoUploaded = await Navigator.push<File?>(
pages/product/add_ingredients_button.dart:          await Navigator.push<bool>(
pages/product/add_nutrition_button.dart:          await Navigator.push<Product>(
pages/product/common/product_list_page.dart:                    await Navigator.push<Widget>(
pages/product/summary_card.dart:              await Navigator.push<Product?>(
pages/product/summary_card.dart:                await Navigator.push<Widget>(
pages/product/new_product_page.dart:              await Navigator.push<void>(
pages/product/add_category_button.dart:          await Navigator.push<Product>(
pages/product/add_new_product_page.dart:          final File? finalPhoto = await Navigator.push<File?>(
pages/product/add_new_product_page.dart:          final Product? result = await Navigator.push<Product?>(
pages/product/add_new_product_page.dart:          final Product? result = await Navigator.push<Product?>(
pages/all_user_product_list_page.dart:                    await Navigator.push<void>(
pages/question_page.dart:                            await Navigator.push<Widget>(
pages/user_management/login_page.dart:                            final bool? registered = await Navigator.push<bool>(

Just FYK the mentioned PR is closed (not merged)

We should not merge this until we figure out this how to support imperative API moving forward
https://github.com/flutter/flutter/issues/99112. Closing for now

Thank you @M123-dev!
Looks like we need to clean our code first and figure out how to distinguish "edit dialogs with returned values" from "real pages".
First step is reviewing #2799 ;)

Now I need your opinion guys.
We still have 8 cases where "pages" are pushed and a returned value is expected:
cards/data_cards/image_upload_card.dart:          final bool? refreshed = await Navigator.push<bool>(
cards/product_cards/smooth_product_card_not_found.dart:                    final String? result = await Navigator.push<String>(
pages/product/edit_product_page.dart:                      final bool? refreshed = await Navigator.push<bool>(
pages/product/product_image_gallery_view.dart:        final File? photoUploaded = await Navigator.push<File?>(
pages/product/add_new_product_page.dart:          final File? finalPhoto = await Navigator.push<File?>(
pages/product/add_new_product_page.dart:          final Product? result = await Navigator.push<Product?>(
pages/product/add_new_product_page.dart:          final Product? result = await Navigator.push<Product?>(
pages/user_management/login_page.dart:                            final bool? registered = await Navigator.push<bool>(

We need to code those 8 cases differently - as they are now we won't be able to switch to named routes (where no value is returned).
And if you have a look at those 8 cases, it's about a place where the user inputs data (e.g. editing a product or logging in). Which has nothing to do with deep-linking - unless we want to be able to connect directly to the "edit product picture 15" page, but from what I saw in the Android URL list this is not the case.
Therefore I suggest to switch those 8 cases from a page to a dialog or - perhaps better - to a bottom sheet.

MaterialPageRoute / fullscreenDialog: true (current mode)
showModalBottomSheet / isScrollControlled: true (suggested mode)

The UI difference is minimal, and it takes 1 minute to switch from one mode to the other.
Are you guys OK with this idea of converting pages that return values, to bottom sheets?
After that we would be able to migrate to named routes, or Navigator 2.0.

AFAIK named routes and Router 2.0 implementations like go_router are completely different and for deeplinking we likely need router 2.0 so we can skip named routes. I just checked and it looks like we can (with what seems like replacing routes) indeed somewhat pop a value
https://stackoverflow.com/questions/71359432/flutter-go-router-how-to-return-result-to-previous-page
https://gorouter.dev/user-input

@M123-dev If you ask me, from what I read I prefer the dynamic named routes solution: it's easier to implement, and it allows deep-linking all the same.
That said, I barged into this issue 4 days ago because I assumed that there was something I could be helpful with, considering that pages returning values were a problem (cf. @g123k and https://github.com/flutter/packages/pull/2416)
If actually that's not a problem that's good news, I'm done here.
Now you guys are all set for the implementation!

https://android-developers.googleblog.com/2022/08/monitor-your-deep-links-in-one-place.html?m=1&s=09

FYI: https://github.com/flutter/packages/pull/2416

Heyy just one little detail I've seen in the Medium article for flutter 3.3 go_router now supports async push.

The latest version (4.3) enables apps to redirect using asynchronous code, and includes other breaking changes described in the migration guide.

Can't say if that allows to pop values but still interesting to know

"It reduces the amount of code needed", dixit Edouard

@monsieurtanuki does the move to Flutter 3.7 change things on this one ?

@teolemon I think we were alright even before, as since months ago we no longer expect pop'ed values from our navigator calls. Except for minor cases that are now handled as "full screen dialog"s rather than as pages.

What kind of solution do you plan to use @g123k a self build one or a package like go_router

What kind of solution do you plan to use @g123k a self build one or a package like go_router

My idea is to use go_router, as it's the most commonly used package.

Sounds good 👍🏼

The config file (assets links) is now deployed on the website
https://github.com/openfoodfacts/openfoodfacts-server/pull/8306#event-8963728420

Hi everyone,
After multiple implementations, supporting only '/product' links is OK, but I have to give a name to all pages. Would it somehow be possible to have a list of page names / screens to show?


Adds weekly and monthly stars to repo object

This change checks whether the selected period is "daily", "weekly" or "monthly" and includes the number of stars in the repository object.
Previously only "daily" was being handled.

I like it, but it would be better to always use the same property for this value.
The API would be more consistent that way.
Could you rename starsToday to starsInPeriod and use this property for all periods.

You're right, makes sense.
I left null as the default value for starsInPeriod then.
Let me know if that's the solution you were looking for.


Migrated OpenCV to 3.1 (default in Kinetic)

I made the changes required to compile and run mapviz on kinetic.  This is one of them.

Relevant:

Kinetic Migration Guide: OpenCV
ROS Wiki: OpenCV3

Is there anything else that need to be done before this can be merged?

This looks good to me, although I'd like to get somebody else to test it out first (I don't have a Kinetic environment right at the moment), and it couldn't hurt to get @malban's approval.

Looks good to me.  We haven't tested it since we don't use kinetic yet, but I'll go ahead and merge it anyway.


[do not merge] Custom unittest runner + explicit thread shutdown

I want to see if the combo of not running ocean tests + having explicit shutdown will get rid of these random failures we have in travis on a mac.

Travis seems to pass on this one (docker fails because I pointed the localrest submodule to my fork).
So I think the combo will work well. I'll retrigger the build a few more times.

The build passed 3 times in a row, so I think this confirms it, the shutdown functionality is required.


refactor/feat: improve organization of server directory, serve frontscope

This PR improves the organization of the server subdirectory, adds some documentation, adds a script for symlinking server config files to their appropriate locations, and serves frontscope.

I read all the changes and they look good. This is hard for me to test locally. Does this match the code currently running on numberscope.colorado.edu? If so, please just go ahead and merge this PR, I have reviewed it by reading code to the extent I can. If there are any differences between this PR and what is running on the numberscope machine, please let me know what they are and I will review.
NOTE that going forward (immediately after this merge) we need to stop making code changes on the server first and then merging them. We need to adopt the workflow of PR, local/GitHub CI testing, merge, then install on the server. We should only ever be installing main on the server. (Except possibly for a rare emergency hotfix.) Thanks!

Does this match the code currently running on numberscope.colorado.edu?

Yes, this is the code running on the server.

we need to stop making code changes on the server first and then merging them. We need to adopt the workflow of PR, local/GitHub CI testing, merge, then install on the server.

I would love this.
One of the problems I foresee is adding tests to backscope is going to be hard without doing some refactoring first. (I've been doing a lot of reading about how to test Flask/SQLAlchemy/Postgres.)
What are your thoughts, @gwhitney?

I suppose we could test refactors locally/manually, then add tests.

Also, I don't have a great way to test server config changes (like the changes in this PR) locally.
Maybe I could try to get the server config running on my Linux laptop.

I'm going to merge now since @gwhitney said it was okay to merge.

Yes, that is correct, going forward at least one reviewer needs to bite the bullet and get the server config running on their local machine for testing purposes, or we need to get a CI server able to run the server config. Let us cross that bridge the next time we need to modify backscope (which will be fairly soon to deal with those sequences that can't load because of crawling too fast).


Hard-coded Ewald tolerance

Description
It looks like the Ewald tolerance is hardcoded to a 'magic' number. I'm guessing this is legacy from the toolkit, but it would be good if this number was made available somewhere (either in the FF or via a global constant)
Reproduction
Please include a minimally reproducing example of this bug.
Output
Please include the output, including full tracebacks of any errors, resulting from the reproduction.
Software versions

Which operating system and version are you using?
How did you install Interchange?
What is the output of running conda list?

I'm guessing this is legacy from the toolkit

Right you are! I wasn't a fan of it being set away from OpenMM's default, but I think @j-wags would have been even less of a fan of that (subtle) behavior change while switching the backend over.

it would be good if this number was made available somewhere (either in the FF

Agreed, and I'm not alone https://github.com/openforcefield/standards/issues/50

or via a global constant

I hadn't really considered this, and I see the utility is not having it be a number tucked away 8 files deep that the user can't change. I'm not sure exactly where it would go ... maybe there's nothing better than shoving it under the rug using keyword arguments. But that's an implementation detail.
This gets into murky territory of what is and is not a part of a force field - my first impression is that exposing it but maintaining the legacy default is still "within" the spec, in some sense. 99.9% of users would be unaffected, and if somebody wanted to go by the strictest letter of the law, I don't know how one could say that 1e-4 is more or less valid than any other value fitting the current description which says nothing more than

Ewald3D-ConductingBoundary (default) denotes that a method like particle mesh Ewald should be used.

Does this sounds reasonable, are there other details I'm unaware of here?

Should be snuck into the next release (probably by early next week?) which I think makes 0.3.15


[DCC_Tabbed_Sheet] Release v1.6.2 - Bugfix and small feature

Submission Checklist
Required

[x] The pull request title clearly contains the name of the sheet I am editing.
[x] The pull request title clearly states the type of change I am submitting (New Sheet/New Feature/Bugfix/etc.).
[x] The pull request makes changes to files in only one sub-folder.
[x] The pull request does not contain changes to any json files in the translations folder (translation.json is permitted)

Changes / Description

Bug fix: Fixed display of roll templates in dark mode (Thanks @DBendit !)
Feature: Automatically fill backstab damage value if it's not already set

Character Sheet Info Roll20 Internal Use only.


feat: Add SNAT property to firewall resources

Overview/Summary
Add SNAT property to firewall resources
Related Issues/Work Items
Closes #739
This PR fixes/adds/changes/removes

Add SNAT property to firewall resources

Breaking Changes
None
Testing Evidence
Replace this with any testing evidence to show that your Pull Request works/fixes as described and planned (include screenshots, if appropriate).
As part of this Pull Request I have

[x] Read the Contribution Guide and ensured this PR is compliant with the guide
[x] Ensured the resource API versions in .bicep file/s I am adding/editing are using the latest API version possible
[x] Checked for duplicate Pull Requests
[ ] Associated it with relevant GitHub Issues
[ ] (ALZ Bicep Core Team Only) Associated it with relevant ADO Items
[ ] Ensured my code/branch is up-to-date with the latest changes in the main branch
[ ] Performed testing and provided evidence.
[ ] Updated one or more of the following tests (if required)

Unit
Linting
E2E (End-To-End)
ValidateAzCloud (Base validation in Azure Cloud)
ValidateMcCloud (Base validation in Azure China Cloud)

[ ] Updated relevant and associated documentation (e.g. Contribution Guide, Module READMEs, Wiki Docs etc.)
[ ] If relevant, created or updated Code Tours here

\azp validate azcloud

/azp validate azcloud

/azp run validateazcloud

/azp run validateazcloud


We are working on similar thinks

Hi,
me and thebradleysanders (author of IOTSmartHome) joined the forces and start development of new platform (Simple-Home) since we both are sharing similar ideas, i From nowhere saw your project homify, whitch look awesome :) but same like IOTSmartHome is no longer Actively developed. So i started wondering is you will be interested like thebradleysanders and joined you skill , patience, knowledge, etc.  with ours on similar project :) which is actively worked on :) your contribution would be appreciated :) in any quantity :)
Thank you, for your time!
Best Regards
JonatanRek

Hi Jonatan,
thanks for reaching out to me! And it's pretty cool that you are developing a new SmartHome-Platform in PHP :)
Homify is already 2 years old, but in my opinion, the concept and its modular approach is a good foundation. I've invested more time in the entire backend logic instead of creating a powerful frontend. Currently, I do not have the time to contribute to other projects, since I have many other development & work topics at SAP.
Since Homify is open-source, you can of course contribute to it / use code out of it. It's licensed under the MIT License, so feel free to use it. (But also credit me, if so).
If you have any questions about my project, feel free to contact me, either via git or also per Mail / LinkedIn
Best regards and good luck with your dev project!
Markus


[Enhancement]: Enlist all the contributors

Is your enhancement request related to a problem? Please describe.➕
Game enhancement

Enlist all the contributors on the main website
Showcase them into a hexagonal grid structure of all the contributors

Describe the solution you'd like
Solution steps

Using Javascript, fetching all the data of the contributors
using css, maintain their styling and giving them a hexagonal grid structure of all the profiles

Select program in which you are contributing
GSSoC23
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINE of this project.

Hey @Avdhesh-Varshney !
Thank you for raising an issue 💗
You can self assign the issue by commenting /assign in comment 😀
Make sure you follow CODE OF CONDUCT and CONTRIBUTING GUIDELINES 🚀
Don’t Forget to ⭐ our GameZone🎮
Make sure you join our Discord🕹

/assign

Hey @Avdhesh-Varshney ! Thank you so much for your raising the issue💗
It’s all yours, you can come anytime again and make some contributions! 🚀
Alone, we can do little, but together we can do so much! 😇


Guest Login feature for user's

Guest Login feature for user's

Guest changes looks fine. Just check the commented code once.
And also need to see where to keep in home page but for now I am approving this.


split snakemake file into multiple.

Addresses https://github.com/rotary-genomics/rotary/issues/90
Split rotary into multiple snakemake files.

@jmtsuji Split the snakemake file into multiple. Can you double-check that all the rules are in the right places?


chore: created .sh version of sass compilation script

:tada: This PR is included in version 2.0.0-next.50 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 3.2.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Move Oembed endpoint validation onto the serializer

Fixes

Alternative to #3068 by @dhruvkb
Fixes #3057
Description

This PR checks that the param passed to the endpoint is an actual URL and that the image exists. In the future, we should make it more strict and check it's actually an Openverse URL (as specified in oembed.com) and review the JSON returned, there are fields missing.
Testing Instructions

See instructions in the linked PR.
Checklist

[x] My pull request has a descriptive title (not a vague title likeUpdate index.md).
[x] My pull request targets the default branch of the repository (main) or a parent feature branch.
[x] My commit messages follow best practices.
[x] My code follows the established code style of the repository.
[x] I added or updated tests for the changes I made (if applicable).
[ ] I added or updated documentation (if applicable).
[x] I tried running the project locally and verified that there are no visible errors.
[x] I ran the DAG documentation generator (if applicable).

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

This actually introduces some breaking behavior, previously the URL http://localhost:50280/v1/images/oembed/?url=2982c0bf-9c26-4543-a712-4d50a117ded8 would correctly return the oembed information

IMO it was an error to allow only the image identifier here. It should receive a valid Openverse URL, not just the image identifier. But if that is considered important, feel free to discard this and my review on the linked PR to merge the first proposed solution instead. I also think we are less prone to errors reducing the possible accepted params here.

This actually introduces some breaking behavior

Ah, I commented the same thing on the other PR 😅 For the record here, I generally agree that it makes more sense for the url param to be a validated url, and this solution gets rid of some confusing serializer structure. I think you could reasonably call that fixing a bug in the API's intended behavior rather than changing/breaking the API, based on the documentation of the param. But I would feel more comfortable discussing/fixing that separately from the critical bug fix.

I definitely don't disagree! It does seem like it's in error, but given that it's current behavior of the API I feel like we should continue to honor that contract rather than introduce a breaking change here.

Then, I'll reduce the priority of this PR while that discussion happens.

Yes, we kept the url param very relaxed so making it stricter now will break any apps that rely on the old behaviour. So far we have been accepting any URL (or rather any string in fact) as long as it contains a valid identifier as the end segment of the path.

Yes, we kept the url param very relaxed so making it stricter now will break any apps that rely on the old behaviour. So far we have been accepting any URL (or rather any string in fact) as long as it contains a valid identifier as the end segment of the path.

@krysal Can you query the Django request logs and see how many requests use the unintended format, if any? If the number is very low, e.g., below 100 requests per week, I think it would be fine to change this, especially considering the error message is unambiguous about the expected format and what it takes to fix the request.

Facts about oembed feature (derived from Cloudflare traffic analysis):

43 total requests in the previous 30 days.
21 used the correct format (url=https://...).
8 used the incorrect format (url={uuid}).
The rest (14) had no url query param.
29 of those requests resulted in a 504, 400, or 404, meaning they failed anyway (regardless of parameter format).
Only 5 of requests resulted in a 200 response, of which only 3 weren't cached.
Excluding the 504s, which the API failed to process and which were not cached, the API processed only 5 requests: the 3 uncached 200s and 2 404s.

Two things to draw from that:

The successful usage is statistically 0 when compared to our overall request count. The vast majority of requests resulted in a failure, either 504 or 400.
In other words, the endpoint is essentially unused and most of the time it doesn't work, either because the application crashes (504) or because the requests are incorrectly formatted (usually missing query params altogether) or the referenced resource does not exist.

From my perspective, these are the valid options, given the usage:

Change the query parameter format to be strictly in a full URL. There are so few uses of this endpoint and none appear regular, we won't be breaking any ongoing use cases by making this change.
Do the above change, but then use a Cloudflare transform rule to rewrite query params that are just the UUID into the full URL. The benefit is that our application can be "correct", while our actual public interface still supports the backwards compatible behaviour. The drawback is that this behaviour isn't in our codebase and is therefore magical and easily misunderstood or forgotten.
Change the serializer, but update the request handler to return a redirect in line with the transform behaviour. The benefit here is that we don't rely on easily-lost Cloudflare configuration knowledge to understand why things work and it's less "magical" from the application perspective. The drawback is that it doesn't necessarily simplify our code, overall.

I don't think it's worth doing nothing here. The improvements to the serializer are an obvious benefit.
We should also investigate the 504s on this endpoint. We only log requests we actually responded to, so we don't know what the 504s were, but I would guess it has something to do with the image size calculation.

@sarayourfriend are the 504 Gateway Timeouts continuing after the PR #3068 was deployed? That PR was supposed to fix the app crashing when supplied with an invalid UUID.
Personally I don't have any strong leaning here and would be fine with any approach to proceed.

The Cloudflare rewrite can be documented in a Sphinx page and linked to from the serializer, making it more known and less magic.
We could also break backwards compatibility and just make the usage stricter, making the API more compliant with the oEmbed spec.
We could also do nothing (for now) and consider this as yet another breaking change slated for an eventual v2 of the API.

@sarayourfriend are the 504 Gateway Timeouts continuing after the PR https://github.com/WordPress/openverse/pull/3068 was deployed? That PR was supposed to fix the app crashing when supplied with an invalid UUID.

I didn't pull the exact date but I think it was about in the middle of time period, so probably before 28 sept

This PR is moving out of sync with main over time. @krysal would you like to update it, resolve conflicts and eventually merge it? Or, given that the endpoint is working normally now, should we close it and leave it as a potential future update to the endpoint?

Sorry it took me so long to get to this, folks! I'd like us to move to the stricter serializer @dhruvkb.
Luckily the conflicts were easy to solve, and this is ready for re-reviewing.


See Reactions

Describe the solution you'd like
Allow Discord users to react and Messenger will use the available emojis if the user used one that transfers over. Better yet, show reacts from Messenger to Discord
Describe alternatives you've considered
Showing a "Bob reacted with :) to >Message "
^ this may become necessary for Discord reacts to Messenger
Issues
Allow users to set how long before a messages reaction is no longer tracked on the message to prevent tracking too many messages.

This has the potential for so much spam and abuse, I think it's a bad idea...

@fortepc what spam and abuse do you expect could happen with this?
I think this would be a helpful feature. You can't implement sending reactions from Discord to Messenger in a straightforward manner. However, I can think of a couple options:

Only send the first reaction from Discord that is one of the 7 reactions that exist on Messenger (this works especially well if the bot is nicknamed Someone on Discord)
When a reaction is made on Discord, send it in a message on Messenger that is a reply to the message that was reacted to on Discord

Reactions from Messenger to Discord would be more straightforward, I think.
But, I wonder how tricky it would be to keep track of individual messages? Do they all have a unique ID that would make this easy? Or would you have to search strings? Maybe if you have to search strings, you only search the last X messages that were sent, and if you don't find it (either because the message is very old, or it was edited on Discord), skip sending the reaction?

@abcasada I see spam and abuse, on the group I'm in people often spam reactions, because it's sending notifications if the person is not using Messenger and it's annoying even in that form, it could be much worse if for every reaction there was a message on Discord
Also I thought about locating the message, because... Facebook does not send content with any events regarding messages, only IDs. I would have to store these IDs with corresponding messages on Discord, for things like removing a message, reactions or replies.

But why can't just use native reactions on either sides? This app is not event based?

Miscord cannot use native reactions for a few, quite simple reasons:

Multiple people add a reaction on Discord. How would it look like on Messenger?
Someone reacts with :tada: on Discord. Again, how would it look like on Messenger?
etc.

Multiple people add a reaction on Discord. How would it look like on Messenger?

on the Messenger multiple people are allowed to react as well. not sure I understand the problem.
oh wait.. the same messenger user represents all discord users.

Someone reacts with 🎉 on Discord. Again, how would it look like on Messenger?

either ignore unknown ones, or use fallback 👍 (make configurable), or even allow to supply translation between two systems.
... I guess the sync needs to be one way only then Messenger -> Discord.
but, ... as discord also has one user (bot), there's no way to distinguish which messenger user used react..., unless you create custom reactions for each user on the discord side, which seems overkill and probably hits custom reaction limit very fast.

There's some update on the Messenger side, they also have now custom reactions:

https://www.cnet.com/news/facebook-messenger-gets-a-new-logo-custom-reactions/

These look like a pre-picked list for one conversation; doesn't solve the issue with multiple people on Discord nor custom emoji on Discord

I guess it should be an opt-in feature that should be off by default in order to prevent spamming. I would be interested in such feature for instance :p
For the reactions from Facebook to Discord, a very rough method could be to edit the text from the message, like so :

Miscord is great isn't it ?
3 👍 2 ❤

And for the tracking, I would suggest using the time AND the amount of messages.
After 15 messages, the tracking could stop (on my screen 15 messages fill up the screen)
For the reactions from Discord to Facebook, another very rough method could be to send a Facebook reaction corresponding to the most popular reaction after x people react (x to be define in the settings, default value could be 1 so that discussions 1-to-1 work normally)
And for the custom Discord emojis, I would simply not process them ¯_(ツ)_/¯


fix: disallow emb cols as metadata

https://app.shortcut.com/galileo/story/9579/dq-adding-x-y-data-x-or-data-y-as-metadata-succeeds-but-should-not
Customer ran into issues when logging x as metadata

@setu4993 yes we were getting quite an odd error on the runners side!