dpkg collector should attempt parsing /var/lib/dpkg/info/*.list files

What would you like to be added:
Currently syft parses only .md5sums and .conffiles. There are also .list files which appear to track more files.
Here is an example:
$ docker run -ti ubuntu:impish cat /var/lib/dpkg/info/base-files.md5sums
18f746c5c90ff506445f615ae44f5a7b  lib/systemd/system/motd-news.service
79b82d315319fdb55ccf0f2b4c83fe62  lib/systemd/system/motd-news.timer
34a1699ffecaec1086587f266075f95b  usr/bin/locale-check
75176163fdc8090948d909cc7e4ab4bf  usr/lib/os-release
cf277664b1771217d7006acdea006db1  usr/share/base-files/dot.bashrc
d68ce7c7d7d2bb7d48aeb2f137b828e4  usr/share/base-files/dot.profile
6db82730e03aaeeecb8fee76b73d96d4  usr/share/base-files/dot.profile.md5sums
e8217ac03d4101a8afb94cb0d82c4da4  usr/share/base-files/info.dir
9830e3dbb6a828f2cc824db8db0ceaf7  usr/share/base-files/motd
7028a4bd02c1d276aa45f0973b85fe1b  usr/share/base-files/networks
9926b56bc6e576d4ad206dd82d38deff  usr/share/base-files/profile
7146d00681e32d626ac96f70bd38e108  usr/share/base-files/profile.md5sums
f3b332b9a376a0567236f54d7d87f85e  usr/share/base-files/staff-group-for-usr-local
3b83ef96387f14655fc854ddc3c6bd57  usr/share/common-licenses/Apache-2.0
f921793d03cc6d63ec4b15e9be8fd3f8  usr/share/common-licenses/Artistic
3775480a712fc46a69647678acb234cb  usr/share/common-licenses/BSD
65d3616852dbf7b1a6d4b53b00626032  usr/share/common-licenses/CC0-1.0
24ea4c7092233849b4394699333b5c56  usr/share/common-licenses/GFDL-1.2
10b9de612d532fdeeb7fe8fcd1435cc6  usr/share/common-licenses/GFDL-1.3
5b122a36d0f6dc55279a0ebc69f3c60b  usr/share/common-licenses/GPL-1
b234ee4d69f5fce4486a80fdaf4a4263  usr/share/common-licenses/GPL-2
d32239bcb673463ab874e80d47fae504  usr/share/common-licenses/GPL-3
5f30f0716dfdd0d91eb439ebec522ec2  usr/share/common-licenses/LGPL-2
4fbd65380cdd255951079008b364516c  usr/share/common-licenses/LGPL-2.1
e6a600fd5e1d9cbde2d983680233ad02  usr/share/common-licenses/LGPL-3
0c5913925d40b124fb52ce84c5deb3f3  usr/share/common-licenses/MPL-1.1
815ca599c9df247a0c7f619bab123dad  usr/share/common-licenses/MPL-2.0
57e7e94034b0d1220f7b7fd4682c8a94  usr/share/doc/base-files/README
fbd937e067f0a83fb9422713a6b84a8a  usr/share/doc/base-files/README.FHS
f343e1edb724b5a85de2e03974b1fa6f  usr/share/doc/base-files/changelog.gz
c686090b1ff44554e2d6c541280a55b1  usr/share/doc/base-files/copyright
07223424da25c119e376b74f04a1cb2b  usr/share/lintian/overrides/base-files
$ docker run -ti ubuntu:impish cat /var/lib/dpkg/info/base-files.list
/.
/bin
/boot
/dev
/etc
/etc/debian_version
/etc/default
/etc/dpkg
/etc/dpkg/origins
/etc/dpkg/origins/debian
/etc/dpkg/origins/ubuntu
/etc/host.conf
/etc/issue
/etc/issue.net
/etc/legal
/etc/lsb-release
/etc/profile.d
/etc/profile.d/01-locale-fix.sh
/etc/skel
/etc/update-motd.d
/etc/update-motd.d/00-header
/etc/update-motd.d/10-help-text
/etc/update-motd.d/50-motd-news
/home
/lib
/lib/systemd
/lib/systemd/system
/lib/systemd/system/motd-news.service
/lib/systemd/system/motd-news.timer
/proc
/root
/run
/sbin
/sys
/tmp
/usr
/usr/bin
/usr/bin/locale-check
/usr/games
/usr/include
/usr/lib
/usr/lib/os-release
/usr/sbin
/usr/share
/usr/share/base-files
/usr/share/base-files/dot.bashrc
/usr/share/base-files/dot.profile
/usr/share/base-files/dot.profile.md5sums
/usr/share/base-files/info.dir
/usr/share/base-files/motd
/usr/share/base-files/networks
/usr/share/base-files/profile
/usr/share/base-files/profile.md5sums
/usr/share/base-files/staff-group-for-usr-local
/usr/share/common-licenses
/usr/share/common-licenses/Apache-2.0
/usr/share/common-licenses/Artistic
/usr/share/common-licenses/BSD
/usr/share/common-licenses/CC0-1.0
/usr/share/common-licenses/GFDL-1.2
/usr/share/common-licenses/GFDL-1.3
/usr/share/common-licenses/GPL-1
/usr/share/common-licenses/GPL-2
/usr/share/common-licenses/GPL-3
/usr/share/common-licenses/LGPL-2
/usr/share/common-licenses/LGPL-2.1
/usr/share/common-licenses/LGPL-3
/usr/share/common-licenses/MPL-1.1
/usr/share/common-licenses/MPL-2.0
/usr/share/dict
/usr/share/doc
/usr/share/doc/base-files
/usr/share/doc/base-files/README
/usr/share/doc/base-files/README.FHS
/usr/share/doc/base-files/changelog.gz
/usr/share/doc/base-files/copyright
/usr/share/info
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/base-files
/usr/share/man
/usr/share/misc
/usr/src
/var
/var/backups
/var/cache
/var/lib
/var/lib/dpkg
/var/lib/misc
/var/local
/var/lock
/var/log
/var/run
/var/spool
/var/tmp
/etc/os-release
/usr/share/common-licenses/GFDL
/usr/share/common-licenses/GPL
/usr/share/common-licenses/LGPL
/usr/share/doc/base-files/FAQ
$

Why is this needed:
This would help for more accurate file<->package tracking.
It would also enable someone to use data from syft to build equivalent of dpkg -S.
Consider the following:
# dpkg -S /usr/share/common-licenses/GPL
base-files: /usr/share/common-licenses/GPL

It's not possible to directly answer this question with data from syft right now.

This is an interesting enhancement since it would help a lot with #931. If we can link the package information back up to the correct package manager then when we go to do the vulnerability scanning we won't mislabel said package as installed on its own vs coming from a strict Linux distributions source.
We'll start investigating @errordeveloper so we can get more fidelity around these file relationships!

After investigating this one we're going to close this as not planned.
*.list files do contain useful information regarding what files/directories are needed for a package to exist, however it's not authoritative on ownership in the same way that syft represents ownership of directory/file.
Example:
cat /var/lib/dpkg/info/tar.list
/.
/bin
/bin/tar
/etc
/usr
/usr/lib
/usr/lib/mime
/usr/lib/mime/packages
/usr/lib/mime/packages/tar
/usr/sbin
/usr/sbin/rmt-tar
/usr/sbin/tarcat
/usr/share
/usr/share/doc
/usr/share/doc/tar
/usr/share/doc/tar/AUTHORS
/usr/share/doc/tar/NEWS.gz
/usr/share/doc/tar/README.Debian
/usr/share/doc/tar/THANKS.gz
/usr/share/doc/tar/changelog.Debian.gz
/usr/share/doc/tar/copyright
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/tar.1.gz
/usr/share/man/man1/tarcat.1.gz
/usr/share/man/man8
/usr/share/man/man8/rmt-tar.8.gz
/etc/rmt

In this case we don't want to associate all of the shared directory paths as owned by the tar package.
In dpkg - multiple packages can own the same directories. This is not a paradigm we want to introduce as it will cause too much noise when we try to examine ownership-by-file-overlap relationships.
If there is more information we're missing happy to discuss and reopen but for now this is marked as not planned.

@spiffcs thanks for looking into this! What if syft could just provide an optional collector for these files and store results in some additional field that a downstream tool can take into consideration only when needed? See my example with that licence file, I couldn't work out where it comes from using syft's output at the time.

@errordeveloper could you expand a bit on what you are looking to accomplish? Is it just trying to determine which package "owns" a file?

Is it just trying to determine which package "owns" a file?

@kzantow yes, in that particular case I wished I had some kind of a hint, instead I ended up hard-coding some lists of files.

...we found is that the same files are referenced in multiple .list files, so there isn't a clear package owner of a file.

Are you referring to directories, as @spiffcs pointed out, or files also?

Is there any more information you could point us to that might help identify the right package for a particular file?

No, I am afraid I don't. I don't believe any package managers do it perfectly well. I guess the only thing I can keep referring to is the dpkg output I shown above, which is an important factor.


reduce_ex function implementation

Deduping this with #1182 because dataframe implementation should be nearly the same.

Duplicate of #1182


Rows are narrower than the jstree (in Firefox)

The rows of the grid are displayed too narrow. So, the complete grid gets out of sync with the tree.

I don't understand. Is this an issue, or is it not?

Sorry, a colleague of mine told me that in Firefox it does not work for him as I usually develop in Chrome.
However, somebody else does not had the same issue in Firefox so I suspect this is a misconfiguration on my colleague's Firefox.

No worries. Unfortunately, because of how it has to leverage the existing tree, it always will be either sensitive due to css to slight changes either in columns (the old way) or rows (since ~3.0.0).
I have wanted to do an overhaul of jstree-grid for a long time, perhaps make it an Aurelia/Polymer/Angular/Angular2 component, remove some dependencies. But that is at least several intensive days of work, so until someone wants it enough to hire me to consult for it, it probably won't happen.
May I ask, how are you using it?


Sentry Flutter SDK is ~3 MB

Platform
Flutter Mobile
Obfuscation
Enabled
Debug Info
Enabled
Doctor
Hi all,
We're big fans of Sentry but are unfortunately might need to move away from it: we're building an iOS App Clip (which constrains us to a ~15 MB uncompressed+thinned IPA). Flutter is ~11 MB and Sentry is ~3 MB (which is much larger than we thought it would be).
Here's a section of the output of zipinfo -m RunnerAppClip.ipa:
drwxr-xr-x  2.1 unx        0 bx  0% stor 24-Jul-19 16:19 Payload/RunnerAppClip.app/Frameworks/Sentry.framework/
drwxr-xr-x  2.1 unx        0 bx  0% stor 24-Jul-19 16:19 Payload/RunnerAppClip.app/Frameworks/Sentry.framework/_CodeSignature/
-rw-r--r--  2.1 unx     2296 bX 74% defN 24-Jul-19 16:19 Payload/RunnerAppClip.app/Frameworks/Sentry.framework/_CodeSignature/CodeResources
drwxr-xr-x  2.1 unx        0 bx  0% stor 24-Jul-19 16:19 Payload/RunnerAppClip.app/Frameworks/Sentry.framework/Sentry.bundle/
-rw-r--r--  2.1 unx     2108 bX 81% defN 24-Jul-19 16:19 Payload/RunnerAppClip.app/Frameworks/Sentry.framework/Sentry.bundle/PrivacyInfo.xcprivacy
-rw-r--r--  2.1 unx      757 bX 27% defN 24-Jul-19 16:19 Payload/RunnerAppClip.app/Frameworks/Sentry.framework/Sentry.bundle/Info.plist
-rwxr-xr-x  2.1 unx  2817968 bX 72% defN 24-Jul-19 16:19 Payload/RunnerAppClip.app/Frameworks/Sentry.framework/Sentry
-rw-r--r--  2.1 unx      782 bX 28% defN 24-Jul-19 16:19 Payload/RunnerAppClip.app/Frameworks/Sentry.framework/Info.plist
drwxr-xr-x  2.1 unx        0 bx  0% stor 24-Jul-19 16:19 Payload/RunnerAppClip.app/Frameworks/sentry_flutter.framework/
drwxr-xr-x  2.1 unx        0 bx  0% stor 24-Jul-19 16:19 Payload/RunnerAppClip.app/Frameworks/sentry_flutter.framework/_CodeSignature/
-rw-r--r--  2.1 unx     1798 bX 77% defN 24-Jul-19 16:19 Payload/RunnerAppClip.app/Frameworks/sentry_flutter.framework/_CodeSignature/CodeResources
-rwxr-xr-x  2.1 unx   219344 bX 71% defN 24-Jul-19 16:19 Payload/RunnerAppClip.app/Frameworks/sentry_flutter.framework/sentry_flutter
-rw-r--r--  2.1 unx      797 bX 28% defN 24-Jul-19 16:19 Payload/RunnerAppClip.app/Frameworks/sentry_flutter.framework/Info.plist

(the fourth column is number of bytes)
As you can see, Sentry is ~3 MB. Are there optimizations that can be made?
Version
8.4.0
Steps to Reproduce

flutter create my_app
add an App Clip target: https://docs.flutter.dev/platform-integration/ios/ios-app-clip
flutter pub add sentry_flutter and call SentryFlutter.init in main
flutter build ipa --release --export-method development --analyze-size
open build/ios/archive/*.xcarchive
"Distribute App" in Xcode
"Custom" method of distribution -> "Release Testing" method for distribution
select the App Clip target for distribution
"All compatible device variants" for App Thinning distribution option

open the resulting directory and open App Thinning Size Report.txt and see the uncompressed size
We're happy to help on reducing the SDK size, but may need some guidance on how to approach this.
Expected Result
N/A
Actual Result
N/A
Are you willing to submit a PR?
Yes

Maybe the SDK can support better tree-shaking? For us, we primarily only care about SentryHttpClient. Being able to use SentryHttpClient in our App Clip without eating all 3 MB would be amazing

@sumanthratna hey thanks for raising this issue.
theoretically you could do Sentry.init instead of SentryFlutter.init but you would probably miss a lot of device context, stacktrace symbolication etc...
We'll discuss this and get back to you

do you know if there are any signifcant changes in earlier versions?

-rwxr-xr-x  2.1 unx  2817968 bX 72% defN 24-Jul-19 16:19 Payload/RunnerAppClip.app/Frameworks/Sentry.framework/Sentry

this is the sentry-cocoa sdk taking up space, so I'm not sure if there is much we can do from the flutter side to optimize this.
this might be an issue that could be brought up in the sentry-cocoa repo

@buenaflor Thanks so much for your replies. I tried using Sentry.init instead of SentryFlutter.init and that removed the sentry-cocoa SDK from the build, which has helped us significantly -- Sentry Dart SDK is only ~0.3 MB.
Obviously, this solution is not ideal since now we don't get the features from sentry_flutter, but it's good enough for our App Clip needs. I think the ideal solution would be for Dart+Flutter to support tree-shaking native libraries, so then the Cocoa SDK could be stripped down to what we need but that's probably out of scope for you all. As a workaround, maybe splitting the Cocoa SDK into smaller SDKs with specific functionalities might allow Dart tree shaking to strip un-needed native code from the build.
Feel free to close since I don't think there are any easy solutions for this on your end.
Thanks for the help!

@sumanthratna modularizing the sentry-cocoa SDK is a long-term goal we have, but as you point out, not in the immediate scope for us. We definitely appreciate your input in any case. I'll close this issue for now, please leave a comment if there's a reason to reopen.

@sumanthratna we have released sentry-dart/flutter8.9.0 which allows you to have stacktrace symbolication with only using Sentry.init on iOS, macOS and Android. Please try it out.

@sumanthratna we have released sentry-dart/flutter8.9.0 which allows you to have stacktrace symbolication with only using Sentry.init on iOS, macOS and Android. Please try it out.

Tried Sentry.init instead of SentryFlutter.init in our App Clip but I cannot anymore get anythin visible in the sentry website. Not even in releases. With SentryFlutter.init it works.
await Sentry.init(       (SentryOptions options) {         options.dsn = ConfigDefs.FM_SENTRY_DSN;         options.environment = ConfigDefs.getEnvironmentNameForSentry();         options.release = getReleaseNameForSentry();       },       appRunner: () => runApp(         const MyAppClip(),       ),     );

Do you mean you can't get issues to show up?
There are some limitations though since Sentry.init doesn't support release health compared to SentryFlutter.init but that shouldn't affect your issue

Do you mean you can't get issues to show up?
There are some limitations though since Sentry.init doesn't support release health compared to SentryFlutter.init but that shouldn't affect your issue

Yes - I cannot seem to see issues. Assuming the exceptions and error logs should appear normally from Flutter? And yes in "Releases" I cannot see either

Okay got it, that does not seem right, errors should show up in your sentry issues list.
could you enable debug and paste some of the logs?

@johannesvuorinen which sentry dart version are you using?

I am using the latest one and finally found the reason. This was needed:
options.httpClient = CupertinoClient.fromSessionConfiguration(URLSessionConfiguration.defaultSessionConfiguration());
Apple disallows different things in App Clips like sockets so devs need to use Apple's own network framework which the cupertino_http package enables.


more use of writeTable

we have a wrapper for writing tables. there are a few places like save_fun in the fitting class where writeTable should be utilized. it formats much more nicely

done


Remove synopsis validation in episode model again

Deja vu? #97

Dafuq how did that get back in there

Maybe from a merge conflict :thinking:

Git blame says:
44ce234ac (Matthew Dias 2017-04-16 02:07:41 -0500 39)   validates :synopsis, length: { in: 50..600 }, allow_blank: true

👌

😅


Dg fixes -model

Codecov Report

Merging #216 (d7354ff) into master (5dfe781) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #216   +/-   ##
=========================================
  Coverage     44.72%   44.72%
  Complexity      683      683
=========================================
  Files           175      175
  Lines          3557     3557
  Branches        409      409
=========================================
  Hits           1591     1591
  Misses         1863     1863
  Partials        103      103

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5dfe781...d7354ff. Read the comment docs.


[Protection] Reality Pump

possibly related to #195
https://store.steampowered.com/app/254060/KnightShift/
Incorporates 3rd-party DRM: Reality Pump
no machine activation limit

I'm frankly morbidly curious, since this is, again, just the name of the developer. And it even says there's no activation limit, so is there even any DRM? Even "DRM-Free" had an activation limit, this is just seemingly telling you that the DRM is that there's actually no DRM. Will need further investigation.


feat(frontend): support \d, \dt and \dm

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.
What's changed and what's your intention?

Temporarily implement pg_table_is_visible by simply return true, since pg_class only returns all relations under connected database.
fix relkind format in pg_class and make it compatible with PG, note that, (materialized) source are not supported yet in \d:
r = ordinary table, i = index, S = sequence, t = TOAST table, v = view, m = materialized view, c = composite type, f = foreign table, p = partitioned table, I = partitioned index
add binary operation support for PGRegexMatch and PGRegexNotMatch.

Besides, for different version of psql, the sql generated by \d might be a little bit different.
Checklist

[x] I have written necessary rustdoc comments
[x] I have added necessary unit tests and integration tests
[x] All checks passed in ./risedev check (or alias, ./risedev c)

Documentation
If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.
Types of user-facing changes
Please keep the types that apply to your changes, and remove those that do not apply.

Installation and deployment
Connector (sources & sinks)
SQL commands, functions, and operators
RisingWave cluster configuration changes
Other (please specify in the release note below)

Release note
Please create a release note for your changes. In the release note, focus on the impact on users, and mention the environment or conditions where the impact may occur.
Refer to a related PR or issue link (optional)
Resolve #5128
Resolve partially #2954

Please fill the "Documentation" section in the PR description ❤

BTW, \d <table_name> is not supported.

BTW, \d <table_name> is not supported.

NTFS, we still need to support OPERATOR like postgreSQL for that. The actual query is as follows:
SELECT c.oid,
  n.nspname,
  c.relname
FROM pg_catalog.pg_class c
     LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE c.relname OPERATOR(pg_catalog.~) '^(t)$'
  AND pg_catalog.pg_table_is_visible(c.oid)
ORDER BY 2, 3;


Can't change programming language for Zero Sum

Player reports:

I've been unable to change my language from python when playing ZeroSum. I can change languages in the campaign portion, but not in ZeroSum. I've tried different browsers and icognito sessions but nothing works. Regardless of which language I choose before the match starts, python is the language I have to work in. Please advise.

Maybe related to #2420.


Using nonce for dst in call to Seal

Hi,
On https://github.com/GGP1/kure/blob/ad75a75514bc32646617e335844a4b7758d38deb/crypt/crypt.go#L53 it seems odd to use nonce as dst in AEAD.Seal(). It appears that in Seal sliceForAppend is called which increases the size of dst if it is not large enough for the result. If it increases the size, when it gets to the deriveCounter call, it will take the wrong branch and not use the intended counter calculation, see https://golang.org/src/crypto/cipher/gcm.go?s=3384:3423#L397
I'd suggest creating a new slice for dst in the call to Seal.

Hi, thanks for mentioning it, I've created a new slice for dst. It will be fixed in the next release.

Hi, I've created a new slice for dst. Thanks for your help.


[Feature Request]IGW (In-Game Wiki) support

Wiki in game you can press I and view description of block you are looking at also good for explaining on how to use machines or tools such as book press
https://github.com/MineMaarten/IGW-mod

If you add support for it, please do not add (or supress it if you can) that annoying spam message every time you login if IGW isn't installed.

@Sandriell this message is disabled in IGWMod.cfg. Config is created without a mod.

@Tsyklop Thanks! Never thought to look for a config for a mod I didn't have. hehe

I don't think I'll be adding IGW support anytime soon.  If I ever add in-game documentation, it will probably be with some kind of book that I design.


Threads do not die

The latency with Minerva seems to grow and grow becoming increasingly problematic.
Option 1, switch off reasoning
Option 2 from @kltm

The other option is just to have something like a nightly cycle for the
time being.
What I think is going on, so far:
swdev@toaster:~$ ps -AlF | grep minerva-cli.jar | wc
3      97    1352
Looking, this is the grep command, the shell script, and the actual java
call. Now then, look at this:
swdev@toaster:~$ ps -AlFT | grep minerva-cli.jar | wc
765   29818  479144
That's 765...what is this? They are all child...threads? I'm not
entirely sure what is going on here, but the number looks /something/
like every command to the server is generating a new one of whatever
these are. Any thoughts? Reasoner issues? Something odd with how the
calls are being taken by the web server?
We can start experimenting about when exactly these are generated, but
it would seem that there is some kind of leak at the thread level.

One possibility is that HermiT is not tidying up after itself. We could experiment with some different slme wrapped reasoners here

Almost unusable now, need to restart

TL;DR: Likely in the reasoner/model neck of the woods, not with the webapp.
Starting:
~/local/src/git/noctua$:) ps -eLF | grep minerva-cli.jar | wc -l
24

Refresh to view model (get, model):
~/local/src/git/noctua$:) ps -eLF | grep minerva-cli.jar | wc -l
29

Repeats (get, model) do not increase the count.
Go back to landing page (get, meta); an increase of 1, up to 30, but after a breif pause, back down to 29; repeats do nothing.
Back to the original model (Scratch, btw), using ubernoodle added an individual. No noticeable change.
Added an edge between the new individual and something else: drops to 25, then back up to 29.
Go to a different model, this takes us up to 33...so 4 threads per model? At (/ (- 765 25) 4) => 185, being that we don't have 185 models (last time I checked), that's not it.
Opening a new browser, as a new user, to the same model, does nothing.
Saving causes the drop and return (29 to 33).
Added another edge and individual, no change.
Killed minerva, down to 1--the grep command again.
So there must be /some/ command to minerva that is getting us up to that high number. I'm assuming that the high number is causing the slowdown (which would be nice to double check).

I suggest to run VisualVM on minerva to have more insights of what's going on. Try to sample the method calls, on the memory tab if I recall correctly.

Also, if this is not something related to running in jetty, then it should be easy to replicate this with a junit stress/integration test that performs multiple operations. Based on @kltm's analysis above it might be useful to test cycling through models.
I'm still not clear about the role of the reasoner in this. Yesterday I thought we restarted minerva with reasoning off, yet I see inferences there...
Is there a minerva health check and/or config query call?

May not be directly related to performance: production is up to 12,600 threads.

No jetty should be involved here--isn't jersey-server just it's own thing?
We did not turn of the reasoner; I think we just turned off one of the settings for it.

Found the reasoner flags in minerva-server/src/main/java/org/geneontology/minerva/server/StartUpTool.java, restarting with reasoner off; will report back later.

Well, after a few hours, we're at 43 still. Will watch some more, but this seems like a huge improvement so far.

And by the way, when minerva was killed at ~13k threads, it has also managed to finally eat through all of its allocated 32GB memory.

Over 24hrs use, at 116, and this is also after a very large number of manual repl operations across a whole bunch of models.

I've been looking into this and have a reproducible way to poke minerva-server and get it to generate 8 new Elk reasoner threads, without throwing away the first 8. So you get 8 new threads every time. Either a Factory isn't being reused, or it's not doing it's job. I'm investigating.
Here's the URL, fyi. I haven't decomposed it to make it more readable:
curl -v 'http://localhost:6800/m3BatchPrivileged?intention=action&requests=%5B%7B%22entity%22%3A%22model%22%2C%22operation%22%3A%22add%22%2C%22arguments%22%3A%7B%7D%7D%5D&uid=http%3A%2F%2Forcid.org%2F0000'

I also discovered that you can send a signal to a Java process to get it to log its threads, which is sometimes handier than using jvisualvm:
> ps |grep minerva-server.jar
 4803 ttys000    6:27.67 /usr/bin/java -Xmx2G -jar ./minerva-server/bin/minerva-server.jar -g ./cache/merged.owl -f ../noctua-models/models --port 6800 -c ./cache/catalog-v001.xml --use-golr-url-logging --slme-elk --use-golr-url-logging --use-golr-url-logging exit
> kill -3 4803

(copied from gitter...)
I think I found at least one bug that is responsible for the thread explosion. Basically, a missing reasoner.dispose() in JsonOrJsonpBatchHandler.java. I added a dispose at the end of the workhorse  m3batch method like this:
		if (useReasoner) {
			if (reasoner != null) {
				reasoner.dispose();
			}
		}

		return response;
	}

	public static void initResponseData(JsonModel jsonModel, ResponseData data) {

and the Elk-Reasoner threads were disposed of by the end of the run.
Looking at JsonOrJsonpSeedHandler.java, I see that the dispose() is already present, AND it is in a finally clause. I’m building a PR with my provisional fix. Going to see about making a unit test to detect thread explosion, but  I think that is really a different type of unit test.

I have a Pull Request here:
https://github.com/geneontology/minerva/pull/30
that adds the dispose() call.

The proposed change is a stop-gap until we can fix the underlying issues. Even with disposing of the reasoners, we will run into situations with too many threads (i.e. more than model creates a reasoners) and thrashing.

Stepping through the ELK code, I can confirm that each ELK reasoner instance creates it's own thread pool. There is no way to override that behavior, see org.semanticweb.elk.reasoner.ReasonerFactory#createReasoner

@kltm @cmungall would you say this is no longer a problem? I'm not sure anything changed with ELK, but we aren't actively using it now.

Those were exciting times, but alas over.


Control of prior distribution

If you are doing Bayesian optimization, it is probably a good idea to give the control of prior probability distribution to the user. Any plans to add that to your API?

Hi, thank you for your interest.
For the Bayesian Optimization part, we rely on the scikit-optimize library. The ways to control the prior over BO are the following:

use different surrogate models, i.e. Gaussian Process and Random Forest, and this can be specified both in the python library and from the dashboard

choose a specific kernel for the Gaussian Process (this feature is available only in the python library at the moment). The default kernel is 1.0 * Matern(length_scale=1.0, length_scale_bounds=(1e-1, 10.0), nu=1.5), but you can pass any other kernel from sklearn (See "sklearn.gaussian_process.kernels" here: https://scikit-learn.org/stable/modules/classes.html).

Thanks for the prompt reply!
I now see that control on surrogate model is possible. What I also meant is for example in hyperparameter optimization, one usually want to be able to control the distribution (normal, log-normal, uniform, log-uniform etc.) to sample from for each hyperparam separately. skopt.space classes (e.g. Real, Integer) has some control on this if I remember correctly.

Hi,
yes, skopt.space allows you to specify a prior over the numeric hyperparameters. The possible values are "uniform" and "log-uniform".
Thanks for your questions, we are going to make this clearer in the OCTIS documentation.
Cheers,
Bruno


docs: update Rust and CUDA version in docs (PROOF-902)

Rationale for this change
The docs should match recent version bumps. Rust 1.80 was updated in the blitzar-rs repo in https://github.com/spaceandtimelabs/blitzar-rs/pull/36 and CUDA 12.6 was updated in Blitzar in https://github.com/spaceandtimelabs/blitzar/pull/172.
What changes are included in this PR?

Bump Rust to version 1.80 in the docs
Bump CUDA to version 12.6 in the docs

Are these changes tested?
Yes

:tada: This PR is included in version 3.2.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


how to unique face find ?

I found face but i need unique face only. one persons may time acquire but i need unique people count only need which algorithm best share the code and refer link

You could try some unsupervised clustering without the need of cluster number (K), such like DBSCAN, Chinese Whisper Algorithm, if you need example code, please check
http://dlib.net/face_clustering.py.html
Thanks.

I tried same person different  cluster find problem. any other algorithm refer me


nixos/wakapi: StateDirectory is not set

Describe the bug
I'm not 100% sure but shouldn't StateDirectory be set to wakapi so that /var/lib/wakapi is created and accessible from the service?
Steps To Reproduce
Steps to reproduce the behavior:

...
...
...

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Notify maintainers

@isabelroses
Metadata

system: "x86_64-linux"
host os: Linux 6.11.1, NixOS, 24.11 (Vicuna), 24.11.20241004.bc947f5
multi-user?: yes
sandbox: yes
version: nix-env (Lix, like Nix) 2.90.0-lixpre20240527-0b91a4b
nixpkgs: /run/current-system/nixpkgs

Add a :+1: reaction to issues you find important.

Yes, I would assume so. The module sets StateDirectoryMode so I assume the service uses it's StateDirectory.

I set StateDirectory on my side and it seems to work. I can make a PR.


[Hostmetrics Receiver]: e2e testing- CW validator, Linux expectation template, Testcase, and ECS script

Description:
Added Hostmetrics Receiver e2e testing

CW validator
Linux expectation template
testcase
and ECS script

Link to tracking Issue:
https://github.com/aws-observability/aws-otel-ops/issues/137
PR is long but can be spit into sections. Kept it long so that whole PR can be tested by running as one testcase with below command.
Testing:
Manually testing on local box by running:
cd terraform/ecs
terraform init
terraform apply -auto-approve \
    -var="aoc_image_repo=public.ecr.aws/q1w6y6d0/hostmetricsamd" \
    -var="aoc_version=latest" \
    -var="testcase=../testcases/hostmetrics_receiver" \
    -var-file="../testcases/hostmetrics_receiver/parameters.tfvars"

Note - Requires Collector with hostmetrics_reciever component
Documentation:

[ ] Tracking errors that came up during execution of these tests: https://quip-amazon.com/nXJSAzstMBsW/Issues-discovered-during-hostmetrics-e2e-testing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

I noticed commit 83c7fc1 was pushed that added a lot of changes related to EKS and EC2 test scripts. Can these be extracted into a separate PR that builds on top of this one? Adding this amount of changes on top of an existing PR isn't very helpful for reviewers. Creating PRs that build on top of each other can be good because we can review PRs one at a time.
If this PR just focuses on adding validation + ecs tests that would make review easier.


minor inaccuracy in winlogon graph

[Enter feedback here]
hi, just  small technical edit
@https://learn.microsoft.com/en-us/windows-server/security/windows-authentication/credentials-processes-in-windows-authentication
@Logon tile enumeration the graph incorrectly says winlogon uses the secure32.dll which should be changed to secur32.dll (minus the extra e)

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 1ebfccb0-3ec2-67f2-5566-68815e97b884
Version Independent ID: 96e7a317-573e-3d79-f08d-10cf76dcb3de
Content: Credentials Processes in Windows Authentication
Content Source: WindowsServerDocs/security/windows-authentication/credentials-processes-in-windows-authentication.md
Product: windows-server
Technology: security
GitHub Login: @JasonGerend
Microsoft Alias: jgerend

Hey hamzawinix,
Thanks again for contacting Microsoft. We value your feedback.
A documentation change has been made where instances in the graphs of "Secure32.dll" being corrected to Secur32.dll and will be live after the next business day. We will now be closing this thread.
Thank you and have a most pleasant day.
#please-close


Update akka-actor, akka-stream to 2.6.15

Updates

com.typesafe.akka:akka-actor
com.typesafe.akka:akka-stream

from 2.6.9 to 2.6.15.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.typesafe.akka" } ]

labels: test-library-update, semver-patch

Superseded by #87.


api->ShowOpenDialog of file_selector freeze UI main thread on windows after start printer

1.file_selector:0.9.2+4 + file_selector_windows-0.9.1+6
2.the ui freezed after start printer;
3.It work fine for run the function in a work thread!
code changed: messages.g.cpp line 180  FileSelectorApi::SetUp
std::thread([=]{                   flutter::EncodableMap wrapped;                   const auto& options_arg = std::any_cast<const SelectionOptions&>(                       std::get<flutter::CustomEncodableValue>(                           encodable_options_arg));                   const auto& encodable_initial_directory_arg = args.at(1);                   const auto* initial_directory_arg =                       std::get_if<std::string>(&encodable_initial_directory_arg);                   const auto& encodable_confirm_button_text_arg = args.at(2);                   const auto* confirm_button_text_arg =                       std::get_if<std::string>(&encodable_confirm_button_text_arg);                   ErrorOr<flutter::EncodableList> output = api->ShowOpenDialog(                       options_arg, initial_directory_arg, confirm_button_text_arg);                   if (output.has_error()) {                       wrapped.emplace(flutter::EncodableValue("error"),                           WrapError(output.error()));                   }                   else {                       wrapped.emplace(                           flutter::EncodableValue("result"),                           flutter::EncodableValue(std::move(output).TakeValue()));                   }                   reply(wrapped);                 }).detach();

https://user-images.githubusercontent.com/62126718/226576233-1b6b7b26-5767-4671-a5c7-1613c63a1f28.mp4

Hi @liqping
For the issue to be workable, it needs to be reproducible with a completed and minimal reproducible code sample that doesn’t include 3rd party plugins or complex production code. Please provide such a sample.
You may also try running plugin sample code to check if the issue can reproduce or not.
Also, please provide the output of flutter doctor -v as well. Thanks!

flutter doctor -v
`[√] Flutter (Channel stable, 3.3.6, on Microsoft Windows [版本 10.0.19045.2728], locale zh-CN)
• Flutter version 3.3.6 on channel stable at D:\fluter
• Upstream repository https://github.com/flutter/flutter.git
• FLUTTER_GIT_URL = https://github.com/flutter/flutter.git
• Framework revision 6928314d50 (5 months ago), 2022-10-25 16:34:41 -0400
• Engine revision 3ad69d7be3
• Dart version 2.18.2
• DevTools version 2.15.0
• Pub download mirror https://pub.flutter-io.cn
• Flutter download mirror https://storage.flutter-io.cn
[√] Android toolchain - develop for Android devices (Android SDK version 32.0.0)
• Android SDK at C:\Users\liping\AppData\Local\Android\sdk
• Platform android-33, build-tools 32.0.0
• Java binary at: D:\Android\Android Studio\jre\bin\java
• Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)
https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)
[√] Connected device (3 available)
• Windows (desktop) • windows • windows-x64    • Microsoft Windows [版本 10.0.19045.2728]
• Chrome (web)      • chrome  • web-javascript • Google Chrome 111.0.5563.65
• Edge (web)        • edge    • web-javascript • Microsoft Edge 111.0.1661.44
[√] HTTP Host Availability
• All required HTTP hosts are available
• No issues found!
`

The code sample: https://github.com/liqping/file_select_test.git

The simple_printer_plug is a test printer plug which only pop a windows native dialog to select printer.
PrintDlg see https://learn.microsoft.com/en-us/windows/win32/printdocs/retrieving-a-printer-device-context "Using the PrintDlg Function"

https://user-images.githubusercontent.com/62126718/226855974-aeeb262a-b4d1-4da6-8758-0f2539159c96.mp4

Thanks for your response with more information. I check this on my Windows but it works normally.
https://user-images.githubusercontent.com/104349824/227094581-10c4bf04-074b-4425-83ec-4fab06c392ff.mp4
I'm verifying this on the stable 3.7.7. Can you check this on the latest stable/master channels?

flutter doctor -v (stable & master)
[√] Flutter (Channel stable, 3.7.7, on Microsoft Windows [Version 10.0.19045.2604], locale en-US)
    • Flutter version 3.7.7 on channel stable at C:\WIP\flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 2ad6cd72c0 (33 hours ago), 2023-03-08 09:41:59 -0800
    • Engine revision 1837b5be5f
    • Dart version 2.19.4
    • DevTools version 2.20.1

[√] Windows Version (Installed version of Windows is version 10 or higher)

[√] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
    • Android SDK at C:\AndroidSDK
    • Platform android-33, build-tools 33.0.0
    • ANDROID_HOME = C:\AndroidSDK
    • Java binary at: C:\Program Files\Android\Android Studio Electric Eel 2022.1.1\jbr\bin\java
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)
    • All Android licenses accepted.

[√] Chrome - develop for the web
    • Chrome at C:\Program Files (x86)\Google\Chrome\Application\chrome.exe

[√] Visual Studio - develop for Windows (Visual Studio Community 2022 17.4.4)
    • Visual Studio at D:\DOWNLOADWORK\VS2022
    • Visual Studio Community 2022 version 17.4.33213.308
    • Windows 10 SDK version 10.0.19041.0

[√] Android Studio (version 2021.2)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+7-b1504.28-7817840)

[√] Android Studio (version 2022.1)
    • Android Studio at C:\Program Files\Android\Android Studio Electric Eel 2022.1.1
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[√] VS Code (version 1.76.0)
    • VS Code at C:\Users\ADMIN\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.60.0

[√] Connected device (3 available)
    • Windows (desktop) • windows • windows-x64    • Microsoft Windows [Version 10.0.19045.2604]
    • Chrome (web)      • chrome  • web-javascript • Google Chrome 110.0.5481.180
    • Edge (web)        • edge    • web-javascript • Microsoft Edge 110.0.1587.41

[√] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

[!] Flutter (Channel master, 3.9.0-7.0.pre.33, on Microsoft Windows [Version 10.0.19045.2604], locale en-US)
    • Flutter version 3.9.0-7.0.pre.33 on channel master at C:\WIP\flutter_master
    ! Warning: `flutter` on your path resolves to C:\WIP\flutter\bin\flutter, which is not inside your current
      Flutter SDK checkout at C:\WIP\flutter_master. Consider adding C:\WIP\flutter_master\bin to the front of your
      path.
    ! Warning: `dart` on your path resolves to C:\WIP\flutter\bin\dart, which is not inside your current Flutter SDK
      checkout at C:\WIP\flutter_master. Consider adding C:\WIP\flutter_master\bin to the front of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision fb7e82826d (2 hours ago), 2023-03-15 04:34:21 -0400
    • Engine revision b7297f0259
    • Dart version 3.0.0 (build 3.0.0-322.0.dev)
    • DevTools version 2.22.2
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git"
      directly to perform update checks and upgrades.

[√] Windows Version (Installed version of Windows is version 10 or higher)

[√] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
    • Android SDK at C:\AndroidSDK
    • Platform android-33, build-tools 33.0.0
    • ANDROID_HOME = C:\AndroidSDK
    • Java binary at: C:\Program Files\Android\Android Studio Electric Eel 2022.1.1\jbr\bin\java
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)
    • All Android licenses accepted.

[√] Chrome - develop for the web
    • Chrome at C:\Program Files (x86)\Google\Chrome\Application\chrome.exe

[√] Visual Studio - develop for Windows (Visual Studio Community 2022 17.4.4)
    • Visual Studio at D:\DOWNLOADWORK\VS2022
    • Visual Studio Community 2022 version 17.4.33213.308
    • Windows 10 SDK version 10.0.19041.0

[√] Android Studio (version 2021.2)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+7-b1504.28-7817840)

[√] Android Studio (version 2022.1)
    • Android Studio at C:\Program Files\Android\Android Studio Electric Eel 2022.1.1
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[√] VS Code (version 1.76.1)
    • VS Code at C:\Users\ADMIN\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.60.0

[√] Connected device (3 available)
    • Windows (desktop) • windows • windows-x64    • Microsoft Windows [Version 10.0.19045.2604]
    • Chrome (web)      • chrome  • web-javascript • Google Chrome 111.0.5563.65
    • Edge (web)        • edge    • web-javascript • Microsoft Edge 110.0.1587.41

[√] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.


Added Search Insert Position in Task 1

I've added a new problem and it's solution in the Task 1 folder, please review! Thanks :)

Hey, could you please review and merge this? Thanks!


[Feature][BitBucket] Support BitBucket server version

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Use case
I'm using a BitBucket self-hosted version and want to integrate BitBucket data with DevLake.
Description
TODO:

[ ] Research Auth differences between BitBucket server and cloud
[ ] Research API differences between BitBucket server and cloud
[ ] Determine the version to be supported
[ ] Implementation

Related issues
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I also would like to use bitbucket server. Do you have any hints on how to get started besides the plugin implementation in the developer manuals?

I also would like to use bitbucket server. Do you have any hints on how to get started besides the plugin implementation in the developer manuals?

I made an exporter using python bitbucket so you don't dont have to use plugin and you use the api instead

Do you have a repo link to your implementation? I couldn't find it on your profile.

Sorry @Startrekzky , this is low prio for me at the moment. I might pick it up at a later date if devlake is deemed useful by the devs in my team

Sorry @Startrekzky , this is low prio for me at the moment. I might pick it up at a later date if devlake is deemed useful by the devs in my team

No problem.

still an open issue

still important to me

Would love to see a plugin which supports BB server. At this point this is a blocker for us from being able to fully utilize DevLake as our company uses BB server.
Is there any work-around for not having a plugin based  solution?
For instance, I was wondering if one could write some custom code to populate some db tables with data from BB server and then point DevLake at the data directly.
Someone above mentioned something about an "exporter".
What exactly would that entail?

I also would like to use bitbucket server. Do you have any hints on how to get started besides the plugin implementation in the developer manuals?

I made an exporter using python bitbucket so you don't dont have to use plugin and you use the api instead

@seumassoltysik this might be easier than creating a plugin for bb server

Hi @seumassoltysik @tim-goto You can look at how bitbucket plugin is implemented.
One of the major tasks for supporting a new version of BitBucket server is to check if the existing APIs used in the BitBucket (cloud) collectors also support the corresponding BitBucket Server version you use. May I ask what BitBucket Server version are you on?

Hi @tim-goto , any updates about this issue? Let me know if you have any plan of attack.

Hey, sorry this has lost priority for my org (the managers have decided..). We are on v7.21.14.

@tim-goto Got it. No problem👌

I also would like to use bitbucket server. Do you have any hints on how to get started besides the plugin implementation in the developer manuals?

I made an exporter using python bitbucket so you don't dont have to use plugin and you use the api instead

@seumassoltysik this might be easier than creating a plugin for bb server

Hi @seumassoltysik @tim-goto You can look at how bitbucket plugin is implemented.
One of the major tasks for supporting a new version of BitBucket server is to check if the existing APIs used in the BitBucket (cloud) collectors also support the corresponding BitBucket Server version you use. May I ask what BitBucket Server version are you on?

We are using BB server version: 7.10.1

I also would like to use bitbucket server. Do you have any hints on how to get started besides the plugin implementation in the developer manuals?

I made an exporter using python bitbucket so you don't dont have to use plugin and you use the api instead

@seumassoltysik this might be easier than creating a plugin for bb server

@tim-goto
At a high level what would this exporter look like.
Would it involve an out of band process for populating DevLake db/tables with BB data?
Relatively new to DevLake so have not had opportunity to fully dig into source code.

I am not quite sure about that, there are some python files in the devlake repo that do just that but for other services, those should get you started

23.9.21 Updates: @tim-goto is integrating the BitBucket Server v7.21.14 and we're collaborating in Slack

Using BitBucket Server version v7.21.4.  Hope the work underway to integrate with BitBucket Server can support this version.  Will be happy to try out Beta version as well.

Using BitBucket Server version  v5.14.1. Hope the work underway to integrate with BitBucket Server can support this version as well.  Will be happy to try out Beta version as well. Thanks

I'm also interested in this feature.

Hello @Startrekzky, we are willing and ready to take over the BitBucket server integration, please contact me.

Hello @Startrekzky, we are willing and ready to take over the BitBucket server integration, please contact me.

Hi @ilian-cleverpine , thanks for volunteering. We have a community member working on this issue now. Can you DM me in Slack so I can add you to the channel for BitBucket-server development?

Nov 6th, 2023 Updates:

Marais is working on this issue.
Since we have a variety of BitBucket Server versions, Marais's plan is to support the latest BitBucket v8.9 first, and then keep supporting the previous v8+ and v7+ versions (May be implemented by other contributors).
Please reach out to me in DevLake's Slack community, I can add you to the BitBucket dev channel if you want to get first-hand updates about this issue.


creates a SHIT ton but none are working??

not sure why i cant use them to login ?

Not a actual generator. its fake and all it does is checks if random emails are registered on Spotify, and if they aren't it just spits out a random password that isn't real and sent actually create an account, either way script is outdated and doesn't work on current version of Spotify.


User space wireguard not working?

First of all, thanks a lot for this awesome software! We are in the process of migrating everything to netbird and so far the experience is great.
I already succeed in running netbird on Teltonika routers, and now I'm trying to run netbird 0.28.4 on AXIS cameras. The goal is to be able to connect to the cameras (HTTP/RTSP) from other peers.
The problem is that the user running the application (ACAP) is very limited, but someone managed to run tailscale on them (https://github.com/Mo3he/Axis_Cam_Tailscale), so I'm pretty sure it'd be feasible to run netbird too.
Apparently they do it with --tun=userspace-networking when using tailscale (https://github.com/Mo3he/Axis_Cam_Tailscale/blob/main/aarch64/app/Tailscale_VPN).
So, I made a script like this:
#!/bin/sh

PACKAGE_DIR='/usr/local/packages/netbird'
TMP_DIR="$PACKAGE_DIR/tmp"

ARGS="--config $TMP_DIR/config.json --log-file $TMP_DIR/client.log --daemon-addr unix://$TMP_DIR/netbird.sock"
NETBIRD="$PACKAGE_DIR/lib/netbird $ARGS"

export NB_WG_KERNEL_DISABLED=true
# export NB_USE_NETSTACK_MODE=true
# export NB_SOCKS5_LISTENER_PORT=30000

echo '----------------------------------------'
echo 'Starting netbird'
mkdir -p $TMP_DIR
$NETBIRD service run &

echo 'Waiting 5 seconds'
sleep 5

echo 'Joining network'
$NETBIRD up --management-url https://netbird.foo.com --setup-key 123123123

echo 'Logs'
tail -f $TMP_DIR/client.log

This script runs fine as root (I can access my cameras), but when I run it as the application user (which will is the non-negociable default in latest firmares) then it outputs this:
2024-10-28T15:59:04.396+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14426]: Starting netbird
2024-10-28T15:59:04.398+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14426]: Waiting 5 seconds
2024-10-28T15:59:09.397+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14426]: Joining network
2024-10-28T15:59:12.398+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14493]: Connected
2024-10-28T15:59:12.406+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14426]: Waiting 5 seconds
2024-10-28T15:59:17.476+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14863]: Error: status failed: create wg interface: operation not permitted

And while the peer shows up in the netbird ui, of course I cannot connect to it. So next if I uncomment the line with NB_USE_NETSTACK_MODE, then it connects fine and I can even ping the camera, but I cannot remotely access it.
With NB_USE_NETSTACK_MODE here is what it status prints:
stvs@axis-b8a44fb94617:/tmp/netbird/lib$ ./netbird $ARGS status
OS: linux/arm64
Daemon version: 0.28.4
CLI version: 0.28.4
Management: Connected
Signal: Connected
Relays: 2/2 Available
Nameservers: 0/0 Available
FQDN: (retracted)
NetBird IP: <IP_ADDRESS>/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 5/9 Connected

Here's what the client.log show:
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/cmd/service_controller.go:64: started daemon server: /usr/local/packages/netbird/tmp/netbird.sock
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:257: generated new SSH key
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:273: using default Wireguard port 51820
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:284: using default Wireguard interface wt0
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:337: filling in interface blacklist with defaults: [ wt0 wt utun tun0 zt ZeroTier wg ts Tailscale tailscale docker veth br- lo ]
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:383: using default DNS route interval 1m0s
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:02+01:00 INFO client/internal/config.go:216: new Management URL provided, updated to "https://netbord.foo.com" (old value "https://api.netbird.io:443")
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:03+01:00 WARN client/server/server.go:259: failed login: rpc error: code = InvalidArgument desc = invalid setup-key or no sso information provided, err: invalid UUID length: 0
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:05+01:00 INFO client/internal/connect.go:119: starting NetBird client version 0.28.4 on linux/arm
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:05+01:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 ERRO iface/uapi.go:15: failed to open uapi socket: remove /var/run/wireguard/wt0.sock: no such file or directory
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 ERRO iface/wg_configurer_usp.go:191: failed to open uapi listener: remove /var/run/wireguard/wt0.sock: no such file or directory
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/firewall/create_linux.go:58: no firewall manager found, trying to use userspace packet filtering firewall
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/internal/connect.go:267: Netbird engine started, the IP is: <IP_ADDRESS>/16
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/internal/dns/host_unix.go:68: System DNS manager discovered: file
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/internal/engine.go:1479: Network monitor is disabled, not starting
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO iface/tun_netstack.go:46: create netstack tun interface
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO signal/client/grpc.go:158: connected to the Signal Service stream
2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:07+01:00 ERRO client/internal/dns/server.go:322: unable to configure DNS for this peer using file manager without a nameserver group with all domains configured
2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:07+01:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 208.831µs, total rules count: 2
2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:07+01:00 INFO management/client/grpc.go:164: connected to the Management Service stream
2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:07+01:00 WARN client/internal/engine.go:587: running SSH server is not permitted

Can you clarify:

Wether NB_USE_NETSTACK_MODE is actually a dead end and won't help me connect from outside to my camera.
What rights "userspace wireguard" requires so I can ask AXIS about what I should do.

We currently haven't implemented any forwarding from the netstack NIC to elsewhere (host or routes), so unfortunately, this access won't work in netstack mode.

Thanks for the answer. Can you maybe clarify what netstack mode is for? I think it basically means "create a socks server that allows other applications to reach other peers through it", but that's "outgoing" only correct?

The intended use is in serverless environments: https://docs.netbird.io/how-to/netbird-on-faas

I think it basically means "create a socks server that allows other applications to reach other peers through it", but that's "outgoing" only correct?

correct

Thanks. Is it planned to allow for some --tun=userspace-networking like tailscale does?
I asked AXIS about what I can do with this limited user, we'll see what they say. Maybe I could create the wg interface as root and then netbird could work under the ACAP user.

@lixmal I'm a developer. How easy would it be to provide --tun=userspace-networking like tailscale does? I can maybe make a PR.
It's a shame because I find netbird much better than tailscale in pretty much every aspects, except I need a way to access remote cameras 😢

Thanks. Is it planned to allow for some --tun=userspace-networking like tailscale does?

We currently don't have any plans for that, there's been no request so far.

@lixmal I'm a developer. How easy would it be to provide --tun=userspace-networking like tailscale does? I can maybe make a PR.
It's not trivial. You'd have to implement a forwarder for tcp and udp (and possibly others like icmp, sctp, ...) packets that arrive on the wireguard netstack interface and decide where to forward them (host vs elsewhere).

Starting point would be somewhere here https://github.com/netbirdio/netbird/blob/10480eb52f305cbe235ba7c63349f7e89db42bbc/client/iface/device/device_netstack.go#L48. That's when we get ahold of the netstack interface.

Thanks!
I noticed the AXIS camera sort the logs so they appear out of order. Here are the correct logs:
2024-10-29T16:34:48+01:00 INFO client/cmd/service_controller.go:24: starting Netbird service
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:147: generating new config /usr/local/packages/netbird/tmp/config.json
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:209: using default Management URL https://api.netbird.io:443
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:233: using default Admin URL https://api.netbird.io:443
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:251: generated new Wireguard key
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:257: generated new SSH key
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:273: using default Wireguard port 51820
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:284: using default Wireguard interface wt0
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:337: filling in interface blacklist with defaults: [ wt0 wt utun tun0 zt ZeroTier wg ts Tailscale tailscale docker veth br- lo ]
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:383: using default DNS route interval 1m0s
2024-10-29T16:34:48+01:00 INFO client/cmd/service_controller.go:64: started daemon server: /usr/local/packages/netbird/tmp/netbird.sock
2024-10-29T16:34:53+01:00 INFO client/internal/config.go:216: new Management URL provided, updated to "https://netbird.foo.com" (old value "https://api.netbird.io:443")
2024-10-29T16:34:54+01:00 WARN client/server/server.go:259: failed login: rpc error: code = InvalidArgument desc = invalid setup-key or no sso information provided, err: invalid UUID length: 0
2024-10-29T16:34:55+01:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service
2024-10-29T16:34:55+01:00 INFO client/internal/connect.go:119: starting NetBird client version 0.28.4 on linux/arm64
2024-10-29T16:34:56+01:00 INFO iface/tun_usp_unix.go:33: using userspace bind mode
2024-10-29T16:34:56+01:00 WARN client/internal/routemanager/manager.go:122: Failed cleaning up routing: 4 errors occurred:
        * rule with suppress prefixlen v4: remove routing rule: operation not permitted
        * rule with suppress prefixlen v6: remove routing rule: operation not permitted
        * rule v4 netbird: remove routing rule: operation not permitted
        * rule v6 netbird: remove routing rule: operation not permitted
2024-10-29T16:34:56+01:00 ERRO client/internal/routemanager/systemops/systemops_linux.go:100: Error setting up sysctl: 1 errors occurred:
        * write sysctl net.ipv4.conf.all.src_valid_mark: open /proc/sys/net/ipv4/conf/all/src_valid_mark: permission denied
2024-10-29T16:34:56+01:00 ERRO client/internal/routemanager/systemops/systemops_linux.go:108: Error cleaning up routing: 4 errors occurred:
        * rule with suppress prefixlen v4: remove routing rule: operation not permitted
        * rule with suppress prefixlen v6: remove routing rule: operation not permitted
        * rule v4 netbird: remove routing rule: operation not permitted
        * rule v6 netbird: remove routing rule: operation not permitted
2024-10-29T16:34:56+01:00 ERRO client/internal/engine.go:322: Failed to initialize route manager: setup routing: rule with suppress prefixlen v4: add routing rule: operation not permitted
2024-10-29T16:34:56+01:00 INFO iface/tun_usp_unix.go:48: create tun interface
2024-10-29T16:34:56+01:00 ERRO client/internal/engine.go:332: failed creating tunnel interface wt0: [operation not permitted]
2024-10-29T16:34:56+01:00 ERRO client/internal/routemanager/manager.go:166: Error cleaning up routing: 4 errors occurred:
        * rule with suppress prefixlen v4: remove routing rule: operation not permitted
        * rule with suppress prefixlen v6: remove routing rule: operation not permitted
        * rule v4 netbird: remove routing rule: operation not permitted
        * rule v6 netbird: remove routing rule: operation not permitted
2024-10-29T16:34:56+01:00 ERRO client/internal/connect.go:263: error while starting Netbird Connection Engine: create wg interface: operation not permitted

It looks like the Management Service connects just fine, but then the tun interface creation fails. I forwarded this to AXIS in the hope they can losen up restrictions using some ACAP flags.

@lixmal just found out that tailscale works because it uses https://github.com/WireGuard/wireguard-go
How feasible would it be for netbird to fallback to wireguard-go as a backup strategy? I assume it'd be a lot of dev.

Interface type: Userspace

Always uses wireguard-go

Then I'm confused about why tailscale works and netbird does not. Will investigate more and report.

Ok, so the crux is not wireguard-go, tailscale uses a Userspace Network Stack which allows them bypass the need of CAP_NET_ADMIN.
Implementing the same in netbird is likely to be a lot of work.


fix: resolve tx calls based on current runtime

Description
This PR refactors existing call selection logic into the resolveCall method on the AssetTransferAPI which determines the correct call type and call based on information obtained from the current runtime.
Changes

Refactored call selection logic into the resolveCall method
Added resolveCall unit tests for ParaTo,* SystemTo* and RelayTo* XCM directions
Fixed parachain primary asset tx construction to allow for xTokens transfers of parachain native assets (e.g. SDN -> Moonriver, MOVR -> Bifrost, etc.) which were previously constructed using only polkadotXcm.
Added RuntimeCallNotFound error variant
Updated parachain integration tests

closes: #368

There are some errors when I run yarn lint but they were not introduced from this PR. Just in case you would like to fix them.

It's probably something specific to your local machine since the CI is passing.


More efficient hooks

Whilst looking at #8540, it was noticed that our pre-receive and post-receive hooks are highly inefficient during migration. Each updated/created reference causes a separate request on the private hooks URL.
This patch batches up the updated/created references in batches up to 30 and tries to reduce the amount of duplicated requests to the db.
Fixes #8540 - although the underlying issue with db connections potentially causing a TIME_WAIT connection explosion requires #8528.
EDIT: dropped the batch size to 30.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@85e4190). Click here to learn what that means.
The diff coverage is 44.91%.

@@            Coverage Diff            @@
##             master    #8602   +/-   ##
=========================================
  Coverage          ?   41.68%
=========================================
  Files             ?      528
  Lines             ?    68722
  Branches          ?        0
=========================================
  Hits              ?    28646
  Misses            ?    36417
  Partials          ?     3659

Impacted Files
Coverage Δ

models/update.go
19.04% <ø> (ø)

modules/private/hook.go
0% <0%> (ø)

cmd/hook.go
0% <0%> (ø)

routers/private/internal.go
23.63% <100%> (ø)

models/repo_watch.go
51.36% <39.39%> (ø)

modules/repofiles/update.go
39.85% <45.33%> (ø)

routers/private/hook.go
50.86% <52.52%> (ø)

modules/repofiles/action.go
65.36% <68.83%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 85e4190...10c2f12. Read the comment docs.

Why not send only one request to internal web router when one post_receive triggered?

I'm not sure I understand the question.
There's 3 options:

Send a post-receive request for each line in the stdin. Simple, but inefficient: reasonable if you never expect more than say 10 references updated in a push.
Send a single post-receive request for all lines in the stdin. Not quite as simple, the problem is that once you accept pushes can update large numbers of references - the post becomes unbounded. It could get arbitrarily large.
Send them in batches, basically free once you've done the second step. A little more inefficient but theres a sort of hard bound on how big each post is and how long each one might take.

I think sending them in batches with a large enough batch size means that in most circumstances it's essentially send them all in one go. I'm not certain what the correct size for that batch is, it could be reasonable to make it 500 but setting a bound means that we know how big it could be.

From #8540 it looks like >6,000 tags and >14,000 commits are quite a real scenario. Perhaps some feedback from @mansshardt could help us decide a good batch size. One thing to take into account is that long transactions tend to:

Lock parts of the database.
Produce dead-locks or similar side effects (e.g. snapshot too old) for the same reason.

So, too large a batch size is not a good thing. Also, I think there will be a curve of diminishing returns, meaning that a batch size of 900 will not be any better than 500 (it will probably be worse), and 100 could still be overkill. I believe that a relative small number between 10 and 30 will work wonders.

It would also be helpful if this actually worked though first :stuck_out_tongue_closed_eyes:

OK, so with some more testing the issue is that 200 takes too long for Gitea to process in one fell swoop. If you change the batching to 50 it works - although it appears it takes 15s to process 50 tags  - I've dropped it down to 30.
That seems rather slow to me but probably is consistent with previous code and not likely to be due to these changes.

From #8540 it looks like >6,000 tags and >14,000 commits are quite a real scenario. Perhaps some feedback from @mansshardt could help us decide a good batch size.

@zeripath @guillep2k  I just checked out your recent changes and built again. With this build all releases got created and it's noticeably faster. So 30 seems to be a good batch number.

Excellent. I'm glad that this approach does actually reduce the time. I've never actually tried to send 400 references before so was kind of horrified at how long it took.
Some thoughts:

I think it should be possible to prevent a timeout, better get it to predict a reasonable timeout given the amount of data to be sent, on the internal POST too. We could do that as a further PR though.
Clearly the underlying action and notifywatchers functions need more optimization too - I'm a little horrified that 50 tags took 15s to process - surely some of what they're doing can be sped up or moved out of the main goroutine. (Again probably not for this PR unless there's something obvious.)

The git client failing is interesting and I would be interested to know what caused it.

The git client failing is interesting and I would be interested to know what caused it.

I'd suspect that the git client doesn't get a proper signal that everything has been transmitted as Gitea is still working on tags/releases. I think the client only gets that signal after Gitea "think's" it's finished. Which is why I got no error yesterday but Gitea stopped after creating only 190 releases. Does that make any sense to you @zeripath
What should happen here is that Gitea sends the git client some kind of an "OK" after everything got transmitted and should proceed in the background creating releases from tags.
It should also be possible to restart the release creating process in case it fails. But thats a feature request.

Hmm I had put some logging with fmt.Fprintf(os.Stderr,...) into the code whilst testing but I couldn't get git to emit it until the end of the push so removed it as it wasn't very helpful.
That being said the original code never logged so I don't think that could need the logging - especially if it took even longer previously. Is there anything in the Gitea logs? I think something must be erroring somewhere near the end and that's cause.
I suspect I need to stack up the errors rather than just blindly fail when one happens.

Agreed about backgrounding release creation. Most of this stuff should just be entering a queue and not happening on the main goroutine. Errors from creating a release can go in to the release.

In my experience, log.Printf is buffered, but there's a log.Flush function as well that helps with that.
I think the client simply timed out waiting for a message. If you add some feedback messages and flush them from time to time, I think the ultimate client timeout could probably be avoided.
For tests, you should be able to easily simulate the delay by aritificially adding sleep calls between operations.

So in the case of the hooks within cmd/hook.go you can't just use log annoyingly - it's not running in the server process. You can log out to stderr and git will display it in the end though but it appears it caches that. Perhaps it's enough though to keep git happy?

I haven't try it, but testing that should be easy enough by adding artificial delays in the code. Some experimenting (or some Google) will be required to determine what that delay should be to make the client fail.

@zeripath This should give some hint: https://stackoverflow.com/questions/6458790/is-there-a-way-to-make-git-over-http-timeout
There might be some ssh timeout setting as well.

Is there anything in the Gitea logs?

@zeripath I cant't see anything in the logs indicating the error on the git client side.

I think the client simply timed out waiting for a message.

@guillep2k That's what I think as well.

This page suggests that git will effectively timeout in ssh operations as well and proposes a fix by changing the timeout configuration.
So, pushing a message from time to time should be an effective solution for long operations like this. Git does that with "counting objects ..." etc., and I suspect it's not just to give the user some feedback but to keep the client alive.
I don't know a thing about the HTTP GIT protocol. How are partial messages sent to the client, with '206 Partial Content' blocks?

@zeripath Just tested with your latest commit:
Counting objects: 409817, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (108243/108243), done.
Writing objects: 100% (409817/409817), 1.14 GiB | 28.09 MiB/s, done.
Total 409817 (delta 201401), reused 409812 (delta 201396)
error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.
fatal: The remote end hung up unexpectedly
error: error in sideband demultiplexer
Everything up-to-date

Damn. Is there any way of getting git to print out the information as it is getting info? I mean the hook is now printing to stderr and we don't see it. There's ways of getting SSH to be more verbose and to get git to be more verbose.
I really don't understand what's happening differently here - it's definitely quicker than the original so why is ssh timing out? (Assuming that's what's happening)
My local tests with 403 tags don't suffer this (at least over http with sqlite). The adjusted hook cmd shouldn't be erroring as far as I can see and we only fail if there's a dB error.

fatal: The remote end hung up unexpectedly
That's no timeout.

Damn. Is there any way of getting git to print out the information as it is getting info? I mean the hook is now printing to stderr and we don't see it. There's ways of getting SSH to be more verbose and to get git to be more verbose.
I really don't understand what's happening differently here - it's definitely quicker than the original so why is ssh timing out? (Assuming that's what's happening)
My local tests with 403 tags don't suffer this (at least over http with sqlite). The adjusted hook cmd shouldn't be erroring as far as I can see and we only fail if there's a dB error.

@zeripath Just to clarify thins. We're using git over http.

@mansshardt Do you know the approximate time before it failed? (seconds? minutes? hours?) BTW, an easy way to time an operation in Linux is using time before our command:
$ time sleep 3

real    0m3.022s
user    0m0.000s
sys     0m0.003s

$ time fgrep -R no-find-string vendor

real    0m2.189s
user    0m0.065s
sys     0m0.162s

@guillep2k Here you go:
user@linux:~/repo.git$ time git push --mirror https://gitea.example.org/org/repo.git
Counting objects: 409817, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (108243/108243), done.
Writing objects: 100% (409817/409817), 1.14 GiB | 29.07 MiB/s, done.
Total 409817 (delta 201401), reused 409812 (delta 201396)
error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.
fatal: The remote end hung up unexpectedly
error: error in sideband demultiplexer
Everything up-to-date

real	16m42.025s
user	0m38.338s
sys	0m7.839s

As mentioned, Gitea continues to create releases.

Would it be possible to add one or two integration tests that go full-circuit with multiple commits? Preferably generating notifications.

With Version 1.11.0+dev-169-g577241bc5 I get the following output from git when I migrate a repository with 548 tags:
time git push --mirror https://gitea.example.org/org/repo.git
Counting objects: 25199, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (6916/6916), done.
Writing objects: 100% (25199/25199), 94.41 MiB | 31.19 MiB/s, done.
Total 25199 (delta 12485), reused 25199 (delta 12485)
remote: **............................
remote: ..............................
remote: ..............................
remote: ..............................
remote: ..............................
remote: ..............................
remote: ..............................
remote: ..............................
remote: ..............................
remote: ..............................
remote: ..............................
remote: ..............................
remote: ..............................
remote: ..............................
remote: ..............................
remote: ..............................
remote: ..............................
remote: ..............................
remote: ........ Checking 2 branches
remote: Checked 548 references in total
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: .............................. Processing 30 references
remote: ........ Processing 8 references
remote: Processed 548 references in total
remote:
remote: Create a new pull request for 'master':
remote:   https://gitea.example.org/org/repo/compare/xxx...master
remote:
To https://gitea.example.org/org/repo.git
 * [new branch]        xxx -> xxx
 * [new branch]        master -> master
 * [new branch]        origin/xxx -> origin/xxx
 * [new tag]           0.0 -> 0.0
 * [new tag]           0.1 -> 0.1
 * [new tag]           0.10 -> 0.10
[...]
 * [new tag]           99 -> 99

real	3m21.681s
user	0m1.249s
sys	0m0.840s

So everything looks good now. Thanks @zeripath and @guillep2k for fixing this!

Oh interesting - I was considering abandoning/rewriting this to instead make PushUpdate run over a graceful shuttting down queue.
If the hacky stdout thing works - then maybe that would be enough.

Oh interesting - I was considering abandoning/rewriting this to instead make PushUpdate run over a graceful shuttting down queue.
If the hacky stdout thing works - then maybe that would be enough.

@zeripath I think it's not that hacky. If you add some tests we can test it in Windows too to cover all the bases.

So it will be difficult to actually create a reasonable test case for this.
The problem with timeouts is dependent on how long PushUpdate takes to run.
In particular, I suspect that the git count-objects inefficiency is largely responsible for a lot of the timeouts - so although a noddy example with 500 tags and a single commit will show the batching - it's not really going to exactly replicate the problem that @oaxiento (I guess you've changed your username) had with the original large batch size.
I do still think we should be batching our calls to pre- and post-receive, but if we can speed up PushUpdate(s) and other parts so that they actually do their work in a queue then the logging will likely become unnecessary.
I guess we could cross that bridge when we come to it.
I'll look at resolving the conflicts.

So it will be difficult to actually create a reasonable test case for this.
The problem with timeouts is dependent on how long PushUpdate takes to run.

I just mean a test for correctness (e.g. will it break on Windows or under some unforseen condition?), and to avoid breaking functionality in the future for some regression problem. I agree that a test for performance is at the very least unreliable.

@oaxiento (I guess you've changed your username)

That's right :)

Please resolve the conflicts.

Rebased on the current master to fix conflicts.
It's probable that I need to fix the notifier to work on a number of refs rather than just send one. I will have a think.

Please resolve conflicts.

The default branch is the first branch but not master when push to a new empty repository. Maybe we could change it when we found master and if this is the first push of this repository.

I find the PR title a litle bit misleading

@lunny Done

Ping lgtm


Support Laravel Nova 4.0

I was finally able to get back to this.
The other day I only made it so it will stop crashing in Nova 4.  But the field was not working.
Now this has everything working in Nova 4.

@metrixinfo thanks. I'll review it.


adds a configuration setting to disable app metrics for specific apps

Closes DNE
What's Changed
Metrics can be disabled on a per-app basis. This can be used to disable slow app metrics.

Would it make sense to put this in PLUGINS_CONFIG instead of adding another global setting?
PLUGINS_CONFIG = {
    "nautobot_plugin_xyz": {
        "metrics_enabled": False,
    }
}

Would it make sense to put this in PLUGINS_CONFIG instead of adding another global setting?
PLUGINS_CONFIG = {
    "nautobot_plugin_xyz": {
        "metrics_enabled": False,
    }
}

Not sure we can unilaterally add things to the PLUGINS_CONFIG dict, might be impactful to Cloud and its use of JSON schemas for app configurations. Other than that concern, I like the idea!

Would it make sense to put this in PLUGINS_CONFIG instead of adding another global setting?
PLUGINS_CONFIG = {
    "nautobot_plugin_xyz": {
        "metrics_enabled": False,
    }
}

Not sure we can unilaterally add things to the PLUGINS_CONFIG dict, might be impactful to Cloud and its use of JSON schemas for app configurations. Other than that concern, I like the idea!

Yes, this was the exact concern I shared. This would be a new pattern to include in plugin config and it would make this feature far more complicated. If you say I should explore this I can do it, but it would be a far deeper change.

Settings documentation needs to be ported over to settings.yaml - we can probably handle this part.

Test is failing because as currently implemented the environment variable is space-separated, but the expectation (for every env variable except ALLOWED_HOSTS) is for it to be comma-separated instead. We probably should change to match the expectation rather than changing the test.

Alright, should be good now. Thanks for taking care of porting over to settings.yaml.

Hm I don't quite understand the CI failure, why should my constant be part of the config?

Because it's being imported into nautobot.core.settings, the test sees it and things it's a configurable setting. Probably need to update the test to ignore it.

I have instead opted for an import ... as, I like that a little better than implementing an explicit exception


fix: Avoid icons imports

Summary:
Related to https://github.com/snapshot-labs/sx-monorepo/pull/577#discussion_r1703907938

Add a new rule to avoid icon imports

Closes https://github.com/snapshot-labs/config/issues/24
How to test:

Go to any project, for example, in monorepo, edit package.json

Under eslintConfig add config:

"no-restricted-imports": [
        "error",
        {
          "patterns": [
            {
              "group": ["~icons/*"],
              "message": "No need to import icons directly, they are auto-imported"
            }
          ]
        }
      ]
    }

Try to  import icon
see error

TODO:

[ ] Run yarn release to update versions, before merging this PR

ahh now i see what you mean


Update CodeQL to 2.12.2

Adapt to https://github.com/jenkins-infra/jenkins-codeql/pull/25

I'm thinking if we should also specify version for codeql/java-queries:AlertSuppression.ql codeql/java-queries:AlertSuppressionAnnotations.ql in https://github.com/jenkins-infra/jenkins-security-scan/blob/main/.github/workflows/jenkins-security-scan.yaml#L66
I don't remember the syntax exactly, maybe codeql/java-queries@0.5.2:AlertSuppression.ql?
I'm unsure which version it would use if not specified explicitly.

I'm unsure which version it would use if not specified explicitly.

Hopefully the one downloaded a few lines up :)

I'm unsure which version it would use if not specified explicitly.

Hopefully the one downloaded a few lines up :)

Oh, that would actually make sense! 🙈


[QUERY] Correlation information in DiagnosticSource event ServiceBusProcessor.ProcessMessage.Start

Query/Question
When implementing OpenTelemetry instrumentation for the Azure.Messaging.ServiceBus SDK I have run into a a problem regarding correlation-id handling.
When consuming the DiagnosticListener with name Azure.Messaging.ServiceBus and the event ServiceBusProcessor.ProcessMessage.Start the correlation information has not yet been set on the DiagnosticActivity. In the code the DiagnosticActivity is first started, triggering the Start-event, and then after the correlation data is set with DiagnosticScope.SetMessageData in the form of a linked Activity.
When handling the event it is then not possible to set the parent trace id in the Activity to sent to OpenTelemetry as the linked activity does not exist yet.
One way to solve it is to send the OpenTelemetry tracing from the ServiceBusProcessor.ProcessMessage.Stop event as the Activity.Current then has linked activity containing the correlation-id/trace-id. But unsure if that is the way to go.
Is there any reason that the correlation information is not set before the DiagnosticScope.Start is invoked? That is to reverse the order of Start and SetMessageData.
https://github.com/Azure/azure-sdk-for-net/blob/850df4dc21246f6d216e8ed472313b741a8362b7/sdk/servicebus/Azure.Messaging.ServiceBus/src/Processor/ReceiverManager.cs#L118-L120
Environment:
Azure.Messaging.ServiceBus 7.1.1

//cc: @pakrym

Thank you for your feedback.  Tagging and routing to the team members best able to assist.

I think it is a reasonable expectation that the message data would be included before the start of the scope. @jsquire do you know how this is handled in Event Hubs?

I think it is a reasonable expectation that the message data would be included before the start of the scope. @jsquire do you know how this is handled in Event Hubs?

Event Hubs reads the diagnostic identifier from the event and creates the link before starting the scope.
https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/eventhub/Azure.Messaging.EventHubs/src/Primitives/EventProcessor.cs#L541-L562

@mangeg thanks for filing the issue, we will get this fixed and we also accept contributions 😄


Issue in installing signalRclient using vcpkg

On installing vcpkg using command "vcpkg install signalrclient" getting error :
The following packages will be built and installed:
signalrclient[core]:x86-windows
Starting package 1/1: signalrclient:x86-windows
Building package signalrclient[core]:x86-windows...
-- Using cached C:/Users/dataman/vcpkg/downloads/SignalR-SignalR-Client-Cpp-1.0.0-beta1.tar.gz
-- Using source at C:/Users/dataman/vcpkg/buildtrees/signalrclient/src/.0.0-beta1-a6e6078b32
-- Configuring x86-windows
-- Building x86-windows-dbg
CMake Error at C:/Users/dataman/vcpkg/scripts/cmake/vcpkg_build_cmake.cmake:170 (message):
Command failed: C:/Program Files/CMake/bin/cmake.exe;--build;.;--config;Debug;--target;install;--;/p:VCPkgLocalAppDataDisabled=true;/p:UseIntelMKL=No
Working Directory: C:/Users/dataman/vcpkg/buildtrees/signalrclient/x86-windows-dbg
See logs for more information:
C:\Users\dataman\vcpkg\buildtrees\signalrclient\install-x86-windows-dbg-out.log
Call Stack (most recent call first):
C:/Users/dataman/vcpkg/scripts/cmake/vcpkg_install_cmake.cmake:24 (vcpkg_build_cmake)
C:/Users/dataman/vcpkg/ports/signalrclient/portfile.cmake:24 (vcpkg_install_cmake)
C:/Users/dataman/vcpkg/scripts/ports.cmake:71 (include)
Error: Building package signalrclient:x86-windows failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with .\vcpkg update, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
Package: signalrclient:x86-windows
Vcpkg version: 0.0.113-nohash
for Reference i have attached install-x86-windows-dbg-out.log
install-x86-windows-dbg-out.log
install-x86-windows-dbg-out.log

LINK : fatal error LNK1104: cannot open file 'C:\Users\dataman\vcpkg\installed\x86-windows\debug\lib\cpprest_2_10.lib'

Hi,
I go around this problem simply copying libcpprest_2_10d.lib into libcpprest_2_10.lib in the same folder.
But... there is another problem.... The package is missing openssl lib . We get this error now:
unresolved external symbol _CRYPTO_cleanup_all_ex_data referenced public: __thiscall boost::asio::ssl::detail::openssl_init_base::do_init::~do_init(void) and so on...
Is there any way to change locally the package to build successfully this package?
TIA

@alangab  I didn't find any problems on the latest version. Maybe you should update and rebuild this port.

Hi @ShailjaDataman , thanks for reporting this issue!
This issue has been fixed. Plz update vcpkg and rebuild signalrclient.
Thanks.

Hi    JackBoosY
Thanks for the information , i tryed again and it works now.
I cannot answer immediately,
Bye
Il giorno gio 18 apr 2019 alle ore 06:06 JackBoosY<EMAIL_ADDRESS>ha scritto:

Closed #4075 https://github.com/Microsoft/vcpkg/issues/4075.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vcpkg/issues/4075#event-2284556797, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ALBLUR5MMW73ST4KODLUZV3PQ7XVNANCNFSM4FPIGKKA
.


Schema view doesn't render after a number with default value is defined

Using a number with a default value causes the schema view to not render the next time it's loaded.
max : {
    type  : "number",
    attrs : {
        min   : 1,
        max   : 80,
        value : 80
    }
}

The error is: input.js:26 Uncaught TypeError: Cannot read property 'child' of undefined
I changed the check on line 25 of input.js to val && options.root to work around this, no idea if it's the right solution.

Yeah that's the correct fix, send a PR?

Fixed by 7584c3a7c0e3ac720c133a1f3c4022fff206e89d


Compute Shaders + ExecutionMode's

Adds a wgpu compute shader example, "wgpu-example-compute-runner" and "wgpu-example-compute-shader". At minimum to compile and execute a compute shader, ie GLCompute, the execution mode LocalSize must be set. So this PR additionally adds a mechanism to do that:
#[spirv(gl_compute(local_size_x=64, local_size_y=16, local_size_z=4))]
pub fn main_cs() {}

Emulating GLSL, dimensions not specified default to 1 (gl_compute only).
The execution modes specified within a execution model, are collected into a Vec along with their arguments, and then this is sent as a SpirvAttribute::Entry, which is now special Entry struct. In entry.rs, the stub functions submit these execution modes.
I also added the fn compute_shader_entry_stub as a duplicate of shader_entry_stub, which in the future may handle function parameters differently.
I added the complete list of execution modes. There may be some additional logic for default values that I'm not familiar with, notably you can now specify OriginLowerLeft to vertex shaders, (OriginUpperLeft is the default). There isn't any error handling about which modes are valid per execution model, or specifying multiple conflicting modes. In general they are collected and submitted left to right.
All the tests passed.

wow thanks for the contribution!
linking to #180

I'm not sure why CI isn't running on this - perhaps the merge conflicts need to get resolved first?

Yes, merge conflicts are fixed.


Support for killing previous session after connecting to a new session

What would you like sesh to do?
Sometimes I use sesh like cd/zoxide but automatically creates a new tmux session at the same time.
So I would usually go like
# this is just in simple command form but i usually interact with this in fzf

# go to my project directory
sesh connect ~/Code

# create a new project like
mkdir new-project

# and then connect to it
sesh connect ~/Code/new-project

but then I would be left with a session that will just clutter my list of sessions and I have to remove later.
I also like to use it after I know I'm done with a session as a bitty bit faster way of killing one and also I get to select the next session I want to view.
So I have made a workaround via a script.
#!/bin/sh
# t is just this script without the kill-session
[ ! -n "$TMUX" ] && {
	t
	return
}
prev_session=$(tmux display-message -p '#S')
out=$(
	sesh list -i -H | fzf-tmux -e --ansi -p 80%,80% \
		--no-sort --border-label ' sesh relinquish ' --prompt '⚡ ' \
		--header '  ^a all ^t sessions ^f find ^k kill selected session' \
		--bind 'tab:down,btab:up' \
		--bind 'ctrl-a:change-prompt(⚡ )+reload(sesh list -iH)' \
		--bind 'ctrl-t:change-prompt(  )+reload(sesh list -it)' \
		--bind 'ctrl-f:change-prompt(  )+reload(fd -td --absolute-path | sed -E "s|$HOME|~|g; s|.*|\x1b[36m \x1b[0m&\x1b[0m|")' \
		--bind 'ctrl-k:execute(tmux kill-session -t $(echo {} | cut -c 5-))+change-prompt(⚡ )+reload(sesh list -i -H)'
)

sesh connect "$out" || return
if [ "$out" != "" ]; then
	tmux kill-session -t "$prev_session"
fi

So I would use Ctrl-F to show all the relative files and connect to them

this type of workflow also influenced this #133 issue because without using --absolute-path It will just break

But I would very much appreciate an option in sesh connect and maybe in sesh clone too so I wouldn't have to use this weird script
OPTIONS:
   --abandon -a       Only works while within a session, after connecting to a different session, kill previous session.

Ooo, I really like this idea. I think I can play around with it more next month.
Thanks for the suggestion!


IdProvinder should support not only from the property

In Neo4j bold library, the returned Node, org.neo4j.driver.internal.InternalNode has it's id which wasn't in the properties, should have a provider which has a interface like,
T processIdentifier(Vertex node);
I just gremlin traversal the neo4j database.

Yes, it is possible to have a provider that uses the native id supplied by the database. I will add it as a feature request for next versions

here, https://github.com/caivega/neo4j-gremlin-bolt/commit/8cc41bda2d90cd7ff991a488479d36dce0d809ae, for reference, i just meet my requirement, for traversal, check and see.

This enhancement has been completed, use version 0.2.10.


0.7.7 -> 1.0 Migration Issues w/ Draggable and/or Event Bubbling

Filed a bug with leaflet.DistortableImage for compatibility with 1.0: https://github.com/publiclab/Leaflet.DistortableImage/issues/64
For some reason, dragging a DistortableImage around the map fails to work at all.
Implementation is here.
Would changes to L.Draggable or to event bubbling account for this issue?

And just a note -- L.DistortableImageOverlay extends  L.ImageOverlay -- and dragging is handled here: https://github.com/publiclab/Leaflet.DistortableImage/blob/master/src/edit/DistortableImage.Edit.js#L127-L157