Some songs cant be played

Hello!
I'm using raspotify at my old cubieboard 2, its work well but some songs dont play there, anyone here know something about it? I've seacher but i might be using wrong words and I didnt find nothing, my english is basic as it is not my primary language.
log:
Aug 18 05:12:06 cubieboard2 librespot[1921]: [2020-08-18T05:12:06Z INFO  librespot_playback::player] Loading <Running Outta Love> with Spotify URI <spotify:track:1aSTn7F9RUijD31mAzjlQx> Aug 18 05:12:06 cubieboard2 librespot[1921]: [2020-08-18T05:12:06Z ERROR librespot_core::channel] channel error: 2 0 Aug 18 05:12:06 cubieboard2 librespot[1921]: [2020-08-18T05:12:06Z ERROR librespot_playback::player] Unable to load encrypted file. Aug 18 05:12:06 cubieboard2 librespot[1921]: [2020-08-18T05:12:06Z WARN  librespot_connect::spirc] The player has stopped unexpectedly.

Thanks and sorry for my english

I have the same problem, Spotify Play only some songs and it stops unexpectedly. I can do a sudo systemctl restart raspotify but the album or playlist starts from the start and it stops again, usually on the same song. I'm using the latest version of raspbian (or Raspberry OS) with the last version of Raspotify from the raspotify repository under a Raspberry Pi 3 model B.
Here is my log:
ago 18 19:47:54 rpi-mk librespot[582]: [2020-08-18T23:47:54Z WARN  librespot_audio::fetch] Error from channel for data receiver for range 5045536 (+16384).
ago 18 19:49:02 rpi-mk librespot[582]: [2020-08-18T23:49:02Z INFO  librespot_playback::player] Loading <Hazard Profile, Pt. 1 - Live at the Baked Potato> with Spotify URI spotify:track:5rGM9EVIwg57HV2wL13ZnL
ago 18 19:49:03 rpi-mk librespot[582]: [2020-08-18T23:49:03Z INFO  librespot_playback::player] <Hazard Profile, Pt. 1 - Live at the Baked Potato> (389306 ms) loaded
ago 18 19:55:32 rpi-mk librespot[582]: [2020-08-18T23:55:32Z INFO  librespot_playback::player] Loading <Kings and Queens - Live at the Baked Potato> with Spotify URI spotify:track:4iNAHHWfRL1XVmesLu6Kqw
ago 18 19:55:32 rpi-mk librespot[582]: [2020-08-18T23:55:32Z ERROR librespot_core::channel] channel error: 2 0
ago 18 19:55:32 rpi-mk librespot[582]: [2020-08-18T23:55:32Z ERROR librespot_playback::player] Unable to load encrypted file.
ago 18 19:56:02 rpi-mk librespot[582]: [2020-08-18T23:56:02Z INFO  librespot_playback::player] Loading <Kings and Queens - Live at the Baked Potato> with Spotify URI spotify:track:4iNAHHWfRL1XVmesLu6Kqw
ago 18 19:56:03 rpi-mk librespot[582]: [2020-08-18T23:56:03Z ERROR librespot_core::channel] channel error: 2 0
ago 18 19:56:03 rpi-mk librespot[582]: [2020-08-18T23:56:03Z ERROR librespot_playback::player] Unable to load encrypted file.
ago 18 19:56:03 rpi-mk librespot[582]: [2020-08-18T23:56:03Z WARN  librespot_connect::spirc] The player has stopped unexpectedly.

I have the same problem with certain tracks.  It will fail consistently on some tracks and log messages similar to this.  Raspberry Pi 4
18.08.20 20:59:31 (-0500)  spotify  [2020-08-19T01:59:31Z INFO  librespot_playback::player] Loading <Gendèr> with Spotify URI <spotify:track:73J0XZfBhvGgCloBnkXVCP> 18.08.20 20:59:34 (-0500)  spotify  [2020-08-19T01:59:34Z ERROR librespot_core::channel] channel error: 2 0 18.08.20 20:59:34 (-0500)  spotify  [2020-08-19T01:59:34Z ERROR librespot_playback::player] Unable to load encrypted file. 18.08.20 20:59:34 (-0500)  spotify  [2020-08-19T01:59:34Z WARN  librespot_connect::spirc] The player has stopped unexpectedly.

I believe it has something to do with the librespot that is included with raspotify.  I went to that project's site and followed their install/compile directions, then symlinked it where raspotify installed it's build of it.  It's been over a day since I've seen the issue now.  Previously, I was seeing it every time I listened.

Hmmmmm, I don't know if I have enough experience to do it, but I'll try...

@mk-360 what are you running this on?  I'm running on a raspberry pi.  If you are running on a raspberry pi, I can write what I did to install/compile it.  It might be a day or so before I get the chance.

@JRansomed That would be great, I'm using a headless raspberry pi 3b+ with raspbian with all updates.
Thanks!

@mk-360 That matches my setup.  I'll write something up and hopefully hit all the points I found.  If something is missed, we can work through and improve the documentation.  It really wasn't all that bad.  Just takes the pi some time to compile.

@natn
If I restart the raspotify it will work again too but other files become unavailable.
@JRansomed Hey, thanks for the tips, i'll try compile it, i'm not a expertise on linux but i hope i can make it work as well. If you have any tips wich or where get the files to compile it will help a lot. I know i need the source code but i'm not sure if it only this.
If help you too, I'm using Cubieboard 2 with armbian

@BrazilianArmy I just put a fresh microsd card in my pi.  Going through the motions from scratch and documenting it.  I'll cover creating the symlinks.  The reason I'm using symlinks is to make it easy to change to the binary and config we are building.  Likewise, easy to change back to the one dtcooper created.  I imagine these steps should work for the cubieboard 2.
Waiting on the longest part, the software compilation now.  Should be able to finish my notes soon and organize them into something readable, then post them.  : )

Ok, this is going to be a lot to post.  It's not as bad as it looks.  : )

Run these commands as your normal, logged in user.  Use sudo when a privileged
command is needed.
Install some prerequisites:
sudo apt-get install build-essential libasound2-dev
I needed a newer version of rust than my pi installed from apt. Per the
instructions at https://www.rust-lang.org/tools/install:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
When prompted, proceed with the default by selecting 1
When the install finishes, we need to update our paths for our currently logged in
session:
source ~/.cargo/env
Compile and install librespot (This step will take quite a while, 41 minutes for me)
cargo install librespot
If it completes successfully, you should see something similar to:
    Finished release [optimized] target(s) in 41m 46s
  Installing /home/pi/.cargo/bin/librespot
   Installed package librespot v0.1.3 (executable librespot)
If we ask the OS where librespot is installed, we should see two installs:
whereis librespot
Example output:
librespot: /usr/bin/librespot /home/pi/.cargo/bin/librespot
The new copy is in your home directory.  Before we make changes, lets shutdown
Raspotify:
sudo systemctl stop raspotify
Move old librespot and symlink to the new one.  I'm getting /usr/bin from my
whereis output.  My install had dtcooper's install in /usr/bin:
cd /usr/bin/
sudo mv librespot librespot.old
sudo ln -sf ~/.cargo/bin/librespot /usr/bin/librespot
The new librespot has changed some of it's option flags.  We will need to edit
our /etc/default/raspotify.  But first we'll copy and symlink.
cd /etc/default
sudo cp raspotify raspotify.old
sudo mv raspotify raspotify.new
Edit raspotify.new with your preferred editor.  You will need to sudo to edit
this file.  Raspotify by default uses --linear-volume.  This option changed in
the new version of librespot.  Uncomment the following line by removing the #
at the start of the line:
#VOLUME_ARGS="--enable-volume-normalisation --linear-volume --initial-volume=100"
Edit the line so that --linear-volume is now --mixer-linear-volume:
VOLUME_ARGS="--enable-volume-normalisation --mixer-linear-volume --initial-volume=100"
The default backend is incorrect with the new version.  At the bottom of the
file should be the following line.  Remove the # to uncomment it:
#BACKEND_ARGS="--backend alsa"
Make it look like the following (alsa no longer worked for me, at librespot's
github site, https://github.com/librespot-org/librespot, they indicate that
rodio is now the default.  This worked for my install):
BACKEND_ARGS="--backend rodio"
Finally symlink to our new config:
sudo ln -sf raspotify.new raspotify
Now start raspotify and test it.
sudo systemctl start raspotify
To check the logs:
journalctl -u raspotify

To change everything back to the dtcooper's config and binary:
sudo systemctl stop raspotify
sudo ln -sf /usr/bin/librespot.old /usr/bin/librespot
sudo ln -sf /etc/default/raspotify.old /etc/default/raspotify
sudo systemctl start raspotify

@JRansomed Excellent! I've been playing a playlist for a couple of hours so far without any problem.
In the first attempt the instructions didn't work, cause I depend on ALSA for other things (a hifiberry like sound card, mopidy) but surfing a bit through the librespot documentation I found a solution. If someone else needs ALSA can use the install command described in the instructions in this manner:
cargo install librespot --no-default-features --features "alsa-backend"
All the rest worked perfectly :)
Thanks for your help!

@JRansomed Hey thanks you!
When i get home i'll reinstall my system and try it, yesterday i was trying to compile it by myself but got stuck at some erros, hope today with you explanation i can pass it.
Thanks!

@mk-360 Awesome!  I'll edit the instructions to show the alsa-backend option.  Thanks for sharing!
@BrazilianArmy Hopefully these instructions will get you going.  If not, post back here.  Maybe we can figure it out.
I'm still hopeful that Raspotify's repo will get updated with the new version of librespot.  We should see it when it does.  It should overwrite the symlink.  ; )  Happy to have helped!

Hi @JRansomed after a while trying to figure out others issue while compiling i finally made it work, everything is edited but when i do the command:
journalctl -u raspotify
i see a error there, i'm pretty sure its is easy to find out for those who already had a good experience with linux, but im stuck there since yesterday trying to figure out what 'im doing wrong.
By now, seems something about permission but i dont find a way by my self to correct this, could you help me?
Aug 23 13:26:59 cubieboard2 systemd[1290]: raspotify.service: Failed to execute command: Permission denied
Aug 23 13:26:59 cubieboard2 systemd[1290]: raspotify.service: Failed at step EXEC spawning /usr/bin/librespot: Permission denied
raspotify.service - Raspotify
Loaded: loaded (/lib/systemd/system/raspotify.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Sun 2020-08-23 13:47:27 -03; 3s ago
Process: 1807 ExecStart=/usr/bin/librespot --name ${DEVICE_NAME} $BACKEND_ARGS --bitrate ${BITRATE} $CACHE_ARGS $VOLUME_ARGS $OPTIONS (code=exited, status=203/EXEC)
Process: 1806 ExecStartPre=/bin/chown raspotify:raspotify /var/cache/raspotify (code=exited, status=0/SUCCESS)
Process: 1805 ExecStartPre=/bin/mkdir -m 0755 -p /var/cache/raspotify (code=exited, status=0/SUCCESS)
Main PID: 1807 (code=exited, status=203/EXEC)
Thanks!

Hello @BrazilianArmy ,
I had the same annoying error (203/EXEC).
First I tried from the CLI with my set-up:
/usr/bin/librespot --name Player --backend alsa --bitrate 320 --disable-audio-cache --enable-volume-normalisation --initial-volume=5 --mixer-linear-volume --onevent /player/spotifyevent.sh
That worked! The things that might stick out here is that I want a low start-up volume and I catch the start and stop in a script. Besides that I think it is a rather normal option set-up.
Quick workaround
The problem is as you suggest a permission problem, it can be found in the Raspotify systemd service definition, i.e. in the file /lib/systemd/system/raspotify.service. I just changed the user from raspotify to root in order to get any permission hinders out of the way.
User=root
Restarted the Raspotify service by:
 systemctl daemon-reload
 systemctl restart raspotify.service
Now librespot works with the Raspotify wrap.
However, it is generally believed that you should not run programs as root. I understand the reasoning for that... well, in my case, I own and control the computer in question, it only streams Spotify, it is connected to a private network,... That said, yes there has to be a better solution than editing the raspotify.service file and using root.
Finally I want to thank @JRansomed and @mk-360  for the instructions above. It was really helpful. So far it has felt a little bit overwhelming to compile librespot myself, but now I am more confident about that. I actually think that I in the future will skip Raspotify and just compile from the source.


PROD: Generate keys for Sudo and collators

Please generate keys for:

10 collator accounts
1 sudo account
genesis account holders zero gamedao game3

Blocker for #139

What's the place for storing keys for PROD? 1Password?

@2075 assigning this to you. You can create a subtask for collator keys generation and assign it for one of us, if needed.

@2075 as we decided to use multisig for the sudo, we need to decide about members of this multisig and collect their keys.
Can you please provide your pub key and suggest other members to be part of signatories?


WIP: basic p2p

took a quick peek, wondering if you're able to get pretty far just using openbazaar's message sending... think i saw some of that in there so maybe you're taking their lead already, but i'm not sure what the goal is here either so maybe doesn't make sense after all, there's is for text-like messages

yeh that is kinda what we need (arbitrary msgs between peers), so need to dig deeper into their stuff. for sure. right now, just testing mostly


Proxy File Example

Will be awesome provide proxy example file....
Please someone help me create proxy file..... some free to test before to buy a payded proxies.....

It's already there... if you pick rotating it goes to about 4 links in github and pulls some free proxies, you the docs first.


Fix a left group chat still shows in the search page

Details

When we leave a group, the report is removed from the storage, but the cache doesn't remove it. This PR handle that.
Fixed Issues

$ https://github.com/Expensify/App/issues/40493
PROPOSAL: https://github.com/Expensify/App/issues/40493#issuecomment-2081892408
Tests

Same as QA Steps

[x] Verify that no errors appear in the JS console

Offline tests

Same as QA Steps
QA Steps

Create a new group chat
Open the search page and verify the group chat is in the search list
Open the group chat and press the header
Press Leave to leave the group
Open the search page again
Verify the group chat isn't in the search list anymore

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native

https://github.com/Expensify/App/assets/50919443/c2b6c6f4-8baa-476d-aa81-610c9b68524c

Android: mWeb Chrome

https://github.com/Expensify/App/assets/50919443/7ab373d6-9869-4fe1-8438-c1c188d7249c

iOS: Native

https://github.com/Expensify/App/assets/50919443/5713ef01-7396-4fea-86bf-f98cecea5af6

iOS: mWeb Safari

https://github.com/Expensify/App/assets/50919443/69cef5d2-31b5-4387-8890-57c22f831f57

MacOS: Chrome / Safari

https://github.com/Expensify/App/assets/50919443/ad1db445-486b-4a48-9140-fd74a161fd07

MacOS: Desktop

https://github.com/Expensify/App/assets/50919443/184882dd-a6ba-4560-95ae-002a0400dbca

🚀 Deployed to production by https://github.com/marcaaron in version: 1.4.71-6 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


[feature] Add local-joint negative sampler for link prediction

Issue #, if available:
Description of changes:
Add LocalJoint negative sampler. This can help reduce the possibility that a negative sample is remote, which helps reduce the cost of doing graph sampling remotely and pull node features from remote. Here is a profiling result:
Dataset: MAG-LSC with 38.6M training edges.
Environment: 4 * g5.48xlarge
GNN model setting: 2 layer RGCN with a fanout of 10,10; Training batch size: 1024
Training time:
| epoch time
joint         |  17999.55s
localjoint |  7266.78s
The training speed is improved by 2.5X
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Can you please add model accuracy too in the description?


Import OpenCV

This seems to be the only correct way to make openCV usable with the IDE.
https://opencv.org/android/
Best source for this.

https://proandroiddev.com/android-studio-step-by-step-guide-to-download-and-install-opencv-for-android-9ddcb78a8bc3
Android Studio + OpenCV
Doing this.


Update branch dependencies

UPDATE_BRANCH=release-1.7 ./bin/update_deps.sh; make gen
Pull Request Attributes
Please check any characteristics that apply to this pull request.
[X] Does not have any changes that may affect Istio users.

@jacob-delgado: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-telemetry-k8s-tests_istio_release-1.7
658a87e54dd05d81e6190c7632c64859e5d7100b
link
/test integ-telemetry-k8s-tests_istio_release-1.7

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

hmm, this same test failed for the proxy update: https://github.com/istio/istio/pull/28591
let's give it another try

/retest

@jacob-delgado: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-telemetry-k8s-tests_istio_release-1.7
658a87e54dd05d81e6190c7632c64859e5d7100b
link
/test integ-telemetry-k8s-tests_istio_release-1.7

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

cc @gargnupur This is related to https://github.com/istio/istio/pull/28500 (and same for 1.6 and I'm imaging 1.8)
I will retest this once changes have pushed into the releases.

@jacob-delgado Do you want to update this PR? Proxy SHA was updated in https://github.com/istio/istio/pull/28767


mono-support.h:36:10: error: 'xamarin/xamarin.h' file not found

Path of "include" and "lib" directory for MonoTouch.*.sdk has been changed.
Example:
Old Path: /Library/Frameworks/Xamarin.iOS.framework/Versions/<IP_ADDRESS>/SDKs/MonoTouch.iphonesimulator.sdk/usr/include
New Path: /Library/Frameworks/Xamarin.iOS.framework/Versions/<IP_ADDRESS>/SDKs/MonoTouch.iphonesimulator.sdk/include
Path has been updated to resolve this issue.

@dasmihir20
Would you please also fix the macos case at line 481?
This problems occurs in macos too.


Memory Game

🎮 Game Request
Hey @kunjgit  I would like to add Memory Game in GameZone.
I have created a similar game and able to contribute to this repository as well please assign me this so that I can build Memory game for GameZone.
Point down the features
These a simple mind game were user's memory will be tested.
Select program in which you are contributing
No response
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINE of this project.

Hey @TusharUtane07 !
Thank you for raising an issue 💗
You can self assign the issue by commenting /assign in comment 😀
Make sure you follow CODE OF CONDUCT and CONTRIBUTING GUIDELINES 🚀
Don’t Forget to ⭐ our GameZone🎮
Make sure you join our Discord🕹

/assign

Hey @TusharUtane07 👋!
@kunjgit here,
We have this annoucement to make 📢!!
We have updated our README 🚀
Make sure you follow proper guideline to raise your next issues !!
Make sure you join our Discord🕹 if you haven't 😇
Thank you 💗!

Hello @TusharUtane07, Time's Uppp!⏰
Sorry for closing your issue!
But it's more than a week since we haven't received anything from your side 😢 .
Come up with new ideas, create a new issue and make sure you finish it within a week! 🔥
All the best! 🚀
Happy Hacking! 💗

Hey @TusharUtane07 ! Thank you so much for your raising the issue💗
It’s all yours, you can come anytime again and make some contributions! 🚀
Alone, we can do little, but together we can do so much! 😇


[Django 1.10] CharFields must define a 'max_length' attribute

When I try to run the server, I get the following SystemCheckErrors:

myapp.HistoricalPackaging.origin: (fields.E120) CharFields must define a 'max_length' attribute.
myapp.Packaging.origin: (fields.E120) CharFields must define a 'max_length' attribute.

My bad.


TLS Passthrough with Caddy autogenerating Certificate is broken

Describe the bug
We use Caddy so when users hit the site via a domain, if the certificate doesn't exists on first connection to port 443 Caddy will fetch the cert, set it up on disk and then allow the connection to use the cert that was dynamically generated (via letsencrypt).
Problem is that first time Caddy is hit the connection is closed.  Therefore connection errors, the certificate is never downloaded, so the page never starts working.  If I use a Service LB, then the TCP connection works great and fetching the Cert and showing me a valid SSL cert but I started getting errors when trying to login to the site, etc.  Seems like headers were getting dropped.
Everything works great if I setup a Service resource of type LB, but just can't seem to get it to work.  I'll post some more details
Expected behavior
The SSL certificate gets dynamically created without reseting the connection, so on the first connection to port 443 the cert is made available and SSL/TLS just work.
Steps to reproduce the bug
Setup Caddy running in a pod, I'm fairly new to Caddy but can def share some configs if needed
Version
1.1.0
Installation
helm
Environment
GKE
Cluster state
N/A

Anyone have any ideas?  I don't even really know where to start

@bradenwright Can you please share your helm chart and any values that you might be using?
I shall attempt to reproduce the issue you are seeing.

Working on getting that put together and will post soon

Bug scrubbing
Closing since this can be fixed by changing configuration.


subscribe() returns function to unsubscribe

This makes it even more compatible to the actual
api of subscribe(...) as documented here:
http://redux.js.org/docs/api/Store.html#subscribe

Could you tell me what has to be done to get this change merge ? Maybe it's not an option, and I should close this PR instead ?
Thanks

@Byron-TW I think you should ask author with @ directly

@arnaudbenard Do you think you need anything else to merge this one ? Thank you
@dmitry-zaets Thanks for the hint :) !

@Byron-TW I added a comment on your code. I'm not really sure why you would need that in your testing suite, care to elaborate on the goal of this functionality?

@arnaudbenard Good catch - the check has been removed.Nonetheless I consider the current implementation buggy, as it will actually remove all but the last subscriber in that case (e.g. [1,2,3].splice(-1, 1)yields [3]).
We need the unsubscribe callback to be a function as our code relies on it. Personally I don't mind to make this functionality actually work, instead of returning a noop() for example.

@arnaudbenard Oh well, admittedly, I didn't really look  at the code and blindly trusted the unit-tests. In the latest commit, this functionality should be correctly implemented.

@Byron-TW What is the status of this PR?

@arnaudbenard I am waiting for feedback or the merge.

@Byron-TW Once the syntax stuff are fixed we can merge. Sorry about the time it took to merge, been pretty busy lately. Thank you for the contribution 👍
Maybe we should have eslint or standard, it would help the contributors :)

@arnaudbenard All but one syntax issues have been addressed. A linter as part of the tests would definitely be helpful. Besides that, given the triviality of the changes and the fact that the comments already contained the fix, I would argue that it would have been easier for all of us if you would have fixed them yourself. Please also note that in the meanwhile, we have made a hard fork of the mock-store, but would be happy to go back once a new release is available.

@Byron-TW You are welcome to make a PR with your hard fork. I think that having a linter will avoid that problem in the future. 👍


fine tune bubble/spike scaling

follow up from #416 to find a better scaling logic than:
const SHRINK_FACTOR = 0.75;
const zoom = Math.pow(2, (current - base) * SHRINK_FACTOR);

in addition, the zoom factor is limited such that the bubbles have at most a radius of 50px and the spike height is at most 100px.

The parameters in ( https://deploy-preview-446--cmu-delphi-covidcast.netlify.app/ ) seem like a big improvement over what we currently have, so I'd love to see this make the next release (demo on Tuesday) and we can perhaps fine tune them further moving forward.

Originally posted by @adamperer in https://github.com/cmu-delphi/www-covidcast/pull/416#issuecomment-694421283

@sgratzl do you have a proposal for better scaling logic?

no, I don't, my latest proposal is the one currently implemented. It is also related to issues like https://github.com/cmu-delphi/www-covidcast/issues/421 and https://github.com/cmu-delphi/www-covidcast/issues/435

Closing as obsolete.  My current opinion is that bubble/spikes have limited utility and we may leave both out of the redesign.


Add Fabric API as a dependency on Modrinth

Since better-fabric-console depends on Fabric API, it would be nice if this was shown on Modrinth so that not only people could see that they needed to download the mod, but automatic mod downloading tools like ferium could automatically download Fabric API along with better-fabric-console.

Done with 1.1.6


Can't combine domains and subdomain hostnames

To replicate
apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: '*.solace-system.net.,www.purplebooth.co.uk.,purplebooth.co.uk.'

Expected
All hosts to correctly resolve to the same IP.
Actual
Only the wildcard domain is configured
time="2018-09-17T22:16:44Z" level=info msg="Change zone: solace-system-net"
time="2018-09-17T22:16:44Z" level=info msg="Del records: a.solace-system.net. A [<IP_ADDRESS>] 300"
time="2018-09-17T22:16:44Z" level=info msg="Del records: a.solace-system.net. TXT [\"heritage=external-dns,external-dns/owner=siQueeSeid2fi5eo2iem,external-dns/resource=service/istio-system/knative-ingressgateway\"] 300"
time="2018-09-17T22:16:44Z" level=info msg="Add records: *.solace-system.net. A [<IP_ADDRESS>] 300"
time="2018-09-17T22:16:44Z" level=info msg="Add records: *.solace-system.net. TXT [\"heritage=external-dns,external-dns/owner=siQueeSeid2fi5eo2iem,external-dns/resource=service/istio-system/knative-ingressgateway\"] 300"
time="2018-09-17T22:16:44Z" level=info msg="Change zone: purplebooth-co-uk"
time="2018-09-17T22:16:44Z" level=info msg="Del records: b.purplebooth.co.uk. A [<IP_ADDRESS>] 300"
time="2018-09-17T22:16:44Z" level=info msg="Del records: c.purplebooth.co.uk. A [<IP_ADDRESS>] 300"
time="2018-09-17T22:16:44Z" level=info msg="Del records: b.purplebooth.co.uk. TXT [\"heritage=external-dns,external-dns/owner=siQueeSeid2fi5eo2iem,external-dns/resource=service/istio-system/knative-ingressgateway\"] 300"
time="2018-09-17T22:16:44Z" level=info msg="Del records: c.purplebooth.co.uk. TXT [\"heritage=external-dns,external-dns/owner=siQueeSeid2fi5eo2iem,external-dns/resource=service/istio-system/knative-ingressgateway\"] 300"
time="2018-09-17T22:17:44Z" level=info msg="All records are already up to date"

Note
Non-wildcard domains work fine, for example
apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: 'a.solace-system.net.,b.purplebooth.co.uk.,c.purplebooth.co.uk.'

is configured as expected
Provider: google cloud dns

I'm going to close this issue
The problem was that I have txt records on the "A" for "purplebooth.co.uk"


[Space webdav]Can't delete Shares named folder in Personal space

Describe the bug
Can't delete Shares folder on the Personal spaces
Steps to reproduce
Steps to reproduce the behavior:

einstein creates a folder FOLDER
einstein shares the folder with the role editor to admin
admin accepts the shares
admin  create Shares folder on the Personal space
admin tries to delete the Shares folder on the Personal space

curl https://host.docker.internal:9200/remote.php/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157%24some-admin-user-id-0000-000000000000/Shares -X DELETE -u admin:admin -vk

Expected behavior
User should able to delete Shares  folder on the personal space
Actual behavior
Response:
> DELETE /remote.php/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157%24some-admin-user-id-0000-000000000000/Shares HTTP/1.1
> Host: host.docker.internal:9200
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.68.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Access-Control-Allow-Origin: *
< Content-Length: 220
< Content-Security-Policy: default-src 'none';
< Content-Type: text/xml; charset=utf-8
< Date: Tue, 01 Nov 2022 04:56:07 GMT
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< X-Robots-Tag: none
< X-Xss-Protection: 1; mode=block
<
<?xml version="1.0" encoding="UTF-8"?>
* Connection #0 to host host.docker.internal left intact
<d:error xmlns:d="DAV" xmlns:s="http://sabredav.org/ns"><s:exception>Sabre\DAV\Exception\Forbidden</s:exception><s:message>Permission denied to delete ./Shares</s:message></d:error>%

@dragonchaser @amrita-shrestha Why should we delete the Shares folder? It only appears on the legacy webDAV endpoints. From my POV the behavior is fine.

Shared jail can not be deleted, works as intended (and functionality might differ from oc10 behaviour => may need adaption in the testsuite) /cc @individual-it

Misunderstanding on my behalf, this seems to be an issue with the hard-coded string for the shares jail, will investigate.

https://github.com/cs3org/reva/pull/3432

Will be fixed with next reva version bump in ocis.

Currently this is not in the 2.0.0-rc.1


ci-docs workflow broken

https://github.com/askap-vast/vast-tools/actions/runs/6256626592
Possible solution is updating this to v8?

Closed by #496.


Terminus on Windows tries to unistall itself every time I close it

Hello. I'm using Terminus on Windows 10. Every time I close it, I'm asked by Windows' user account control if I want to allow the app to make changes to my system. If I press 'Yes', Terminus is removed. It happens all the times. How can I avoid this?

I think I had the same issue. Yesterday (or more: since around a month) it worked the whole time. Today, terminus is just gone. Empty link on my desktop...

Same behavior here


move the summarization code from dart-pad

Move the summarization code from dart-pad to here; have a summarization field come back on every call to analyze.

I don't think we want to put that field on the response of every call to analyse. That call already does too much.
I'd rather we added a specific summarise end point.

Let's talk then. It's in the interest of optimization. Perhaps as a separate service call, and have an optional param on analyze to return the summary?

Sure - lets chat. We're only showing the summary on share right? Or are we planning it put it elsewhere as well?

Just share currently, and don't envision it elsewhere right now.


Spock tests work only for "--lang=groovy"

Fix for https://github.com/micronaut-guides/creating-your-first-micronaut-app-groovy/issues/4
The issue is in state "closed", but the problem can still be reproduced if following the guide instead of just cloning the git repo.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

This was already fixed a few months ago: https://github.com/micronaut-projects/micronaut-guides/commit/0f905cc4e4f4edaf31165e6085eef76933cdaf2b and https://github.com/micronaut-projects/micronaut-guides/commit/96f26996fb51d5c3e1b64e2d8951c1b3adbc8c5b


Add Status Filters to Compliance Node List View

User Story
As a Automate User, I would like to filter out the failed/passed/skipped nodes on the compliance node tab so that I may find my failed nodes easier.
Definition of Done
Page: /compliance/reports/nodes
The nodes table will have our standard filter buttons above the nodes table.
Filters:
Total Nodes | Failed Nodes | Passed Nodes | Skipped Nodes
Demo Script / Repro Steps

A user can navigate to goes to Compliance > Nodes Tab
User clicks on a status filter
System then filters the data table.

related: https://github.com/chef/automate/issues/328


Amend care card pattern plus links to it

Description
Make changes to care card pattern.
Update links from care card component to care card pattern.
Keep a care card component page with a link to new pattern
Add a new review date component page with a link to new pattern.
Related issue

Checklist

[ ] Tested against the NHS.UK frontend testing policy (Resolution, Browser & Accessibility)
[ ] Code follows the NHS.UK frontend coding standards
[ ] Version number has been increased in package.json (using SEMVER)
[ ] CHANGELOG entry - not yet
[ ] Whats new page entry - not yet
[x] Page updated date

@henocookie to make the changes to add commas following a review, along with creating updated examples.


Added check for cls in classmethod

I see now that there is already a PR and issue for this:
#2
#3
Oops 😅


Bisect improvements

Change described https://github.com/microsoft/TypeScript/issues/50670#issuecomment-1244232538:

The issue with the bisect was that the culprit commit was cherry-picked from main to release-4.5, and I only look at the timeline of commits going into main. I’m far from a git expert, but (warning: likely butchering terminology ahead) git bisect gets mad if you give it a pair like v4.4.4 and v4.5.5 because those tags exist on separate branches off of main that never reunite with main. To account for that, I actually run the bisect between the merge-base of the respective inputs and main, so I’m actually bisecting between the point where release-4.4 and release-4.5 branched off of main. But in hindsight I think I only need to take the merge-base of the old ref. There still might be a problem if the behavior is different between the old ref and its merge base with main, though. I think that should be less common.

This also changes the GH Action API a bit; TypeScript PR incoming.

I do bisects between tags on different branches all time time when bisecting by hand; it just starts by testing the merge base first automatically and bails out if that commit doesn't have the "old"/"good" behavior. (Maybe I'm confused and that's what you're reimplementing in TS?)

Hm, I had some kind of trouble with it when I was first setting this up. Don’t recall exactly how.


Integration Tests

Integration Tests in progress but mostly complete, besides deserialization/ serialization.
Updated Action Listener to include server timeout  parameter for testing considerations
modified main script

Let's merge this after https://github.com/owaiskazi19/IndependentPlugin/pull/8.

Looks like gradle check is failing.
@owaiskazi19 how much of a work is it to get gradle check running?
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find org.opensearch:opensearch:2.0.0-SNAPSHOT.
     Searched in the following locations:
       - file:/home/runner/.m2/repository/org/opensearch/opensearch/2.0.0-SNAPSHOT/maven-metadata.xml
       - file:/home/runner/.m2/repository/org/opensearch/opensearch/2.0.0-SNAPSHOT/opensearch-2.0.0-SNAPSHOT.pom
       - https://repo.maven.apache.org/maven2/org/opensearch/opensearch/2.0.0-SNAPSHOT/maven-metadata.xml
       - https://repo.maven.apache.org/maven2/org/opensearch/opensearch/2.0.0-SNAPSHOT/opensearch-2.0.0-SNAPSHOT.pom
     Required by:
         project :

Looks like gradle check is failing. @owaiskazi19 how much of a work is it to get gradle check running?
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find org.opensearch:opensearch:2.0.0-SNAPSHOT.
     Searched in the following locations:
       - file:/home/runner/.m2/repository/org/opensearch/opensearch/2.0.0-SNAPSHOT/maven-metadata.xml
       - file:/home/runner/.m2/repository/org/opensearch/opensearch/2.0.0-SNAPSHOT/opensearch-2.0.0-SNAPSHOT.pom
       - https://repo.maven.apache.org/maven2/org/opensearch/opensearch/2.0.0-SNAPSHOT/maven-metadata.xml
       - https://repo.maven.apache.org/maven2/org/opensearch/opensearch/2.0.0-SNAPSHOT/opensearch-2.0.0-SNAPSHOT.pom
     Required by:
         project :

Hey @saratvemulapalli! Gradle check will fail because maven dependencies are getting fetched from local and not from maven central. We need to publish the code from the feature branch to maven central and it will resolve the gradle check failire.

Looks like gradle check is failing. @owaiskazi19 how much of a work is it to get gradle check running?
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find org.opensearch:opensearch:2.0.0-SNAPSHOT.
     Searched in the following locations:
       - file:/home/runner/.m2/repository/org/opensearch/opensearch/2.0.0-SNAPSHOT/maven-metadata.xml
       - file:/home/runner/.m2/repository/org/opensearch/opensearch/2.0.0-SNAPSHOT/opensearch-2.0.0-SNAPSHOT.pom
       - https://repo.maven.apache.org/maven2/org/opensearch/opensearch/2.0.0-SNAPSHOT/maven-metadata.xml
       - https://repo.maven.apache.org/maven2/org/opensearch/opensearch/2.0.0-SNAPSHOT/opensearch-2.0.0-SNAPSHOT.pom
     Required by:
         project :

Hey @saratvemulapalli! Gradle check will fail because maven dependencies are getting fetched from local and not from maven central. We need to publish the code from the feature branch to maven central and it will resolve the gradle check failure.

Could you open an issue? one of us can pick it up.
We should be able to publish OpenSearch to maven local in the github CI.

Looks like gradle check is failing. @owaiskazi19 how much of a work is it to get gradle check running?
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find org.opensearch:opensearch:2.0.0-SNAPSHOT.
     Searched in the following locations:
       - file:/home/runner/.m2/repository/org/opensearch/opensearch/2.0.0-SNAPSHOT/maven-metadata.xml
       - file:/home/runner/.m2/repository/org/opensearch/opensearch/2.0.0-SNAPSHOT/opensearch-2.0.0-SNAPSHOT.pom
       - https://repo.maven.apache.org/maven2/org/opensearch/opensearch/2.0.0-SNAPSHOT/maven-metadata.xml
       - https://repo.maven.apache.org/maven2/org/opensearch/opensearch/2.0.0-SNAPSHOT/opensearch-2.0.0-SNAPSHOT.pom
     Required by:
         project :

Hey @saratvemulapalli! Gradle check will fail because maven dependencies are getting fetched from local and not from maven central. We need to publish the code from the feature branch to maven central and it will resolve the gradle check failure.

Could you open an issue? one of us can pick it up. We should be able to publish OpenSearch to maven local in the github CI.

https://github.com/opensearch-project/OpenSearch/issues/2758. This is a great idea to publish the dependencies to maven local in GHA.

Rebase the branch with main to run the latest CI for gradle check

@joshpalis gradle check failure
TransportCommunicationIT > testInvalidMessageFormat() STANDARD_ERROR
    Exception in thread "Thread-5"
    java.io.UncheckedIOException: java.net.SocketTimeoutException: Accept timed out

    	at transportservice.ActionListener$1.run(ActionListener.java:36)

    Caused by: java.net.SocketTimeoutException: Accept timed out

    	at java.base/java.net.PlainSocketImpl.socketAccept(Native Method)

    	at java.base/java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:458)

    	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:565)

    	at java.base/java.net.ServerSocket.accept(ServerSocket.java:[53](https://github.com/owaiskazi19/IndependentPlugin/runs/5826049630?check_suite_focus=true#step:6:53)3)

    	at transportservice.ActionListener$1.run(ActionListener.java:31)

TransportCommunicationIT > testInvalidMessageFormat() PASSED

TransportCommunicationIT > testMismatchingPort() STANDARD_OUT
    [2022-04-05T00:42:37,391][INFO ][o.o.t.TransportService   ] [[Test worker]] Registered handler with actioninternal:transport/handshake
    [2022-04-05T00:42:37,397][INFO ][o.o.t.TransportService   ] [[Test worker]] publish_address {<IP_ADDRESS>:9301}, bound_addresses {<IP_ADDRESS>:9301}
    [2022-04-05T00:42:37,398][INFO ][o.o.t.TransportService   ] [[Test worker]] Registered handler with actioninternal:discovery/extensions

TransportCommunicationIT > testMismatchingPort() STANDARD_ERROR
    Exception in thread "Thread-7"
    java.io.UncheckedIOException: java.net.SocketTimeoutException: Accept timed out

    	at transportservice.ActionListener$1.run(ActionListener.java:36)

    Caused by: java.net.SocketTimeoutException: Accept timed out

    	at java.base/java.net.PlainSocketImpl.socketAccept(Native Method)

    	at java.base/java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:458)

    	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:[56](https://github.com/owaiskazi19/IndependentPlugin/runs/5826049630?check_suite_focus=true#step:6:56)5)

    	at java.base/java.net.ServerSocket.accept(ServerSocket.java:533)

    	at transportservice.ActionListener$1.run(ActionListener.java:31)

TransportCommunicationIT > testMismatchingPort() FAILED
    org.junit.ComparisonFailure at TransportCommunicationIT.java:130


coco annotation

https://github.com/Daniil-Osokin/lightweight-human-pose-estimation.pytorch/blob/1590929b601535def07ead5522f05e5096c1b6ac/scripts/prepare_train_labels.py#L111
I try to print "annotation['segmentation']"
{'counts': [43835, 3, 339, 5, 337, 8, 6, 2, 6, 1, 319, 14, 1, 9, 319, 24, 319, 24, 319, 24, 319, 8, 2, 14, 319, 6, 4, 14, 319, 4, 4, 2, 334, 2, 357, 3, 6153, 4, 338, 8, 335, 12, 331, 12, 331, 12, 331, 12, 331, 12, 331, 12, 332, 5, 339, 4, 331, 6, 4, 1, 331, 8, 334, 8, 334, 11, 332, 12, 331, 12, 331, 12, 331, 12, 331, 11, 332, 9, 335, 9, 335, 8, 337, 7, 337, 8, 334, 10, 332, 12, 331, 12, 331, 12, 331, 12, 331, 12, 331, 11, 332, 10, 334, 3, 2, 3, 336, 1, 41156, 6, 336, 8, 334, 10, 332, 12, 331, 12, 331, 12, 331, 12, 331, 12, 331, 12, 331, 12, 332, 10, 334, 8, 337, 4, 87108, 6, 336, 8, 334, 10, 332, 12, 331, 12, 331, 12, 331, 12, 331, 12, 331, 12, 331, 12, 332, 10, 334, 8, 337, 4, 17921], 'size': [343, 640]}

But the coco format  "segmentation"
{
	"segmentation": [[510.66,423.01,511.72,420.03,510.45......]],
	"area":<PHONE_NUMBER>999998,
	"iscrowd": 0,
	"image_id": 289343,
	"bbox": [473.07,395.93,38.65,28.67],
	"category_id": 18,
	"id": 1768
},

Where are the 'counts' and  'size'

Hi! Do you have the error running this script with the COCO dataset?

I dont meet any warning.
The original json (coco) will generate the 'counts' and 'size' after doing prepare_train_labels.py

If you mean difference in the formats, segmented polygons can be in 2 different formats: list of vertices coordinates and in RLE. counts and size refer to RLE. You may check here for the details.


upload_binary_to_app_store

We currently have pilot/upload_to_testflight and deliver/upload_to_app_store to upload binaries to App Store Connect / Testflight (among other things).
The code doing the actual binary uploading is pretty similar:
https://github.com/fastlane/fastlane/blob/69a77679a58868d202e36fa14317e841cda4f5ef/pilot/lib/pilot/build_manager.rb#L31-L38
https://github.com/fastlane/fastlane/blob/69a77679a58868d202e36fa14317e841cda4f5ef/deliver/lib/deliver/runner.rb#L140-L162
Both create the "package" via FastlaneCore::IpaUploadPackageBuilder.new.generate, then use FastlaneCore::ItunesTransporter.new and .upload to actually upload this to Apple.
I suggest we extract that logic into a new upload_binary_to_app_store action that can be used independently from the currently existing actions as well (and is used here of course).
Why?
3 reasons:

The other App Store Connect stuff doesn't work with application specific passwords. ItunesTransporter does, so this would make it easier to show people a way to not have to worry about 2FA if they just want to upload a binary to Apple.
Having this as its own action would enable users to just upload a binary build of their app, without anything else. No metadata, no changelog, no icons etc.
There is an alternative upload mechanism available via altool that we ignored in fastlane for now, but that would be nice to add anyway as it has some advantages over ItunesTransporter.

How?
Find other uses of FastlaneCore::ItunesTransporter.new and .upload to make sure we have the full picture, then extract a generalized version into its own action, then replace the current code in pilot and deliver with a call to that action. Done.

@janpio I do like the idea of combining these into one shared thing but don't think we can make this an "action" if we want it to also be used in pilot and deliver. Those are things that I don't think we want being dependent on an action. If anything, this new shared code should probably live in fastlane or fastlane_core and pilot and deliver should access it there.
I do like the idea of having an action that does not touch any of the legacy or new App Store Connect APIs (if all people want to do is upload binary) but I'm pretty sure deliver can do that already if you set all meta stuff to false. I would prefer to not introduce another action for uploading App Store Connect binaries if we don't need to 😇 But it would be cool if we could make it easier for people to upload just the binary from deliver if that is what they want 🙃

Sure, it makes more sense for the code to live somewhere in fastlane or fastlane_core - and if we decide to introduce the new action it can also just use it.
And yes, deliver probably can achieve a similar result with some specific collection of parameters. But looking at the list of available parameters, I have no idea which ones exactly have to be set to what values to achieve that. And to be honest I was not aware that deliver could also be used to upload builds for Testflight.
Because of these reasons I think the added overhead of a new action is justified. (We would probably add some documentation to all 3 actions here that crosslink between each other ("if you want to also update metadata, look at ..." or "if you only want to upload a binary and use an app specific password, look at ...").

This still sounds like a great idea to me.


Precision of Calculations

I have been trying to fix the failing test cases and came across a problem.  In SolarHeatingWaterOnly\Python\Testing\energyTest.py, lines 64 and 65 keep failing because, as far as I can tell, the actual value is not within the expected error range.  In this case, I was using the equation
params.C_w * params.Mw * (temp - params.Tinit)
to calculate the new expected energy values without PCM and this worked fine for every other test case except the aforementioned last two (values obtained from SolarHeatingWaterOnly\Python\Testing\loadParamsTest.py).  However, if I change the precision of the error (1e-9 to 1e-8) the code runs fine.  To this end, is the factor of 10 change acceptable or is there some other approach I should take in fixing this error?

This is very much a @smiths kind of issue.

@AKM11, yes a relative error of 1e-8 is fine.  This is a very tiny error and nothing to worry about.  The choice of 1e-9 was arbitrary.  It was probably the smallest value we could get away with.   Now 1e-8 is the smallest value.  The problem you are solving is slightly different, so the change is no thing to worry about.

Alright, thank you.


eth/vm/logic/arithmetic: handle exp(X,0) == 1.

What was wrong?
0^0 == 1, but this wasn't tested for.
Caught by exp8.json test (not yet tracked in tests fixture on master).
Semi-accidentally caught in PR #1181, which updated the JSON test fixtures to a newer version, which does contain the exp8.json test. The failure got lost among others.
Can be seen in py3{5,6}-vm CI tests: for the EVM bytecode 0x600060000a600055, which is:
000000: PUSH1 0x00
000002: PUSH1 0x00
000004: EXP
000005: PUSH1 0x00
000007: SSTORE

... there was a 15000 gas use delta: as if the storage was reset to 0, not set to 1. Mentioned CI runs:

py35-vm
py36-vm

How was it fixed?
An additional short-circuit check for exponent == 0 was added.
Cute Animal Picture

Source: not sure; via @ano at peepeth

I have an unrelated question to this PR:
I was trying to understand this line:
https://github.com/ethereum/py-evm/blob/08674f611f4f692e76bbcd7ea1cefcdfd15c723a/eth/vm/logic/arithmetic.py#L155
In my understanding mod=constants.UINT_256_CEILING is used to prevent an overflow.
I was trying to see what would happen when base would be negative.
In [0]: pow(-1, 3)
Out[0]: -1

In [1]: pow(-1, 3, 2**256)
Out[1]:<PHONE_NUMBER>37316195423570985008687907853269984665640564039457584007913129639935

is this intended behaviour?

@voith I think so?..
The yellowpaper is a bit terse on this, but it does specify some signed arithmetic operations (e.g. SDIV in addition to DIV (division, /), or SGT in addition to GT (greater than, >); but not SEXP in addition to EXP).
It could be assumed that the stack values are to be interpreted as unsigned integers, but I haven't found this stated explicitly anywhere.
There is a section on precompile 5 ("arbitrary-precision exponentiation under modulo"), for which one could assume the logic follows that of EXP. For this precompile, inputs are non-negative integers.

@voith I am pretty sure @veox is correct that arguments are treated as unsigned integers.

@pipermerriam Should be able to...

For this precompile, inputs are non-negative integers.

Yeah, just tested some solidity code. solidity doesn't allow exponentiation when the base is negative!

@pipermerriam See commit e7b0af9 - 0^0==1, 0^1==0. Is this enough? Too much?..


Type inference error after upgrading to version 1.4.x

Description
After upgrading vue-tsc to version 1.4.1, type checking failed and this issue still exists even in version 1.4.4.
I defined a ComponentProps<C> type in global.d.ts as follows:
import type { VNodeProps } from 'vue';

declare global {
  type ComponentProps<C> = Omit<InstanceType<C>['$props'], keyof VNodeProps>;
}

This is to make it easier for me to obtain the type of props that needs to be used to other components. I use it like this:
<script setup lang="ts">
import { computed } from 'vue';
import AppKanban from<EMAIL_ADDRESS>
const columns = computed<ComponentProps<typeof AppKanban>['columns']>(() => [
  // columns setting
])
</script>

<template>
  <AppKanban :columns="columns" />
</template>

Currently, the latest version of Volar does not show any errors and correctly infers types, but when running type checking, it fails with the following error message:
src/pages/grade/system/index.vue:63:7  - error TS2322: Type 'unknown' is not assignable to type 'AppKanbanColumnItem[]'.

This issue did not exist in version 1.2.x.

Is the TS version in your project consistent with VSCode? If yes, please provide a repro case so we can investigate it.

Thank you for your response. The following link contains a reproduction of the issue:
https://stackblitz.com/edit/vitejs-vite-f2muxx?file=src%2FApp.vue,src%2Fcomponents%2FAppKanban.vue
I have attempted various methods to replicate the issue and have found that it only occurs when using <slot>. Once I removed the <slot> from the reproduction, the issue was resolved.

Fixed in v1.5.3.


Upgrade to Helm 3.7.0

As noted in https://github.com/helm/helm/releases/tag/v3.7.0 a few major changes were made to the experimental OCI feature:
helm chart commands were removed and integrated into helm cli. Backward compatibility is not guaranteed. For more details: https://github.com/helm/community/blob/main/hips/hip-0006.md.
To upgrade to helm 3.7.0 the following changes need to be done:

update the commands in hack/make-rules/helm.mk
update the commands in pkg/helm/
remove the  fybrik/helm dependency as the new chartLoad function in fybrik/helm loads the chart from the cache and in the new helm oci feature the cache is removed.
update all the modules that use helm chart subcommands under fybrik to use the new helm commands:
https://github.com/fybrik/arrow-flight-module
https://github.com/fybrik/hello-world-module
https://github.com/fybrik/hello-world-read-module

Please also check if the requirement to use semver when pushing/pulling is already in place. We're currently using latest which according to the original helm plans is invalid.

@roee88 Yes, it's in place - I will add that to the list of changes. Thanks. I wonder if it makes sense to use semver pre-release 0.0.0-master to denote master and latest charts...

For the master use case we should move towards snapshot versioning. For the latest use case, is it really required?

I never used that... I do not think so.


Implemented mock data node

Description

Resolves #451

Verification

Mock data appearing on console when running package and previous warning messages regarding not all subscribers are active in the physics engine node have diminished.

Resources

my big ol'brain

Resources:

Resources:

HAHAHAHAAHAHA

new CLI arguement for boat_simulator:
enable-mock-data:=true

Nice work so far! It would be great if you could also configure the mock data node using the globals.yml file. Ideally, we should be able to specify which topics to publish mock data to, the exact data to publish, and the frequency at which it’s published. For an example of how this configuration was added for the data collection node, please refer to #222. Within the node, you’ll need to use ROS parameters to access the values defined in globals.yml. You can see how these parameters are used in the data collection node in this section:
https://github.com/UBCSailbot/sailbot_workspace/blob/6c5d7f3f2f51d0d3ac93baf2f5cb80aa5af30302/src/boat_simulator/boat_simulator/nodes/data_collection/data_collection_node.py#L80

I have added the following parameters to globals.yaml:
mock_data_node:
   ros__parameters:
      qos_depth: 10
      mock_desired_heading: True
      mock_sail_trim_tab: False
      pub_period_sec: 1.0

and added logic so the node only publishes mock data to the topics that have a true value in globals.yaml


Make shutdown on standstill optional.

Add a shutdown_on_standstill option: If this is disabled, nodes no longer shut down if no progress is made in consensus.
Closes #1680.

LGTM

bors merge

bors merge

Just FYI, I've restarted the test. Error was irrelevant to this PR – address already in use.


Export raw text transcriptions

Ability to export raw text to Google Drive (that's the one I want - could be others)

done


.withRequirement being called once you tab complete for arguments

Description
I'd like CommandAPI to call the withRequirement lambda once you try to tab complete the first position ("test", "toggle", etc.)
I found out instead it calls the lambda several times once you join.

Expected code
  private final CommandTree command = new CommandTree("request")
    .then(new LiteralArgument("test")
      .executesPlayer((sender, args) -> {
        sender.sendMessage(Component.text(QualityEconomyAPI.hasRequest(sender.getUniqueId())));
      }))
    .then(new LiteralArgument("toggle")
      .executesPlayer(this::toggleRequests))
    .then(new MultiLiteralArgument("answer", "accept", "deny")
      .withRequirement(sender -> {
        if (sender instanceof Player) sender.sendMessage(Component.text("GOOD"));
        else sender.sendMessage(Component.text("BAD"));
        return (sender instanceof Player player) && QualityEconomyAPI.hasRequest(player.getUniqueId());
      })
      .then(new PlayerArgument("target")
        .replaceSuggestions(ArgumentSuggestions.strings(info -> {
          if (info.sender() instanceof Player player && requests.containsKey(player.getUniqueId()))
            return requests.get(player.getUniqueId()).keySet().stream().map(Bukkit::getOfflinePlayer).map(OfflinePlayer::getName).toArray(String[]::new);
          return new String[0];
        }))
        .executesPlayer(this::answerRequest)))
    .then(new LiteralArgument("send")
      .then(new PlayerArgument("target")
        .replaceSuggestions(CommandUtils.getOnlinePlayerSuggestion())
        .then(new DoubleArgument("amount", Number.getMinimumValue())
          .executesPlayer(this::request))));

Extra details
No response

This sounds like it is working as intended. In vanilla, the client never asks the server which commands it can access. Instead, when the player joins, the server will figure out which commands they can access based on the requirements. This means that the server only needs to tell the client about the commands that they can access, so less information needs to be sent over the network.
If the requirements for your command change, you can trigger the server to resend commands to the player by calling Player#updateCommands or CommandAPI#updateRequirements. This is described in the documentation here: https://commandapi.jorel.dev/9.3.0/requirements.html#updating-requirements.

So how can I update the list of acceptable arguments when they tab complete the suggestions?

You probably want to call Player#updateCommands or CommandAPI#updateRequirements whenever the information being accessed by QualityEconomyAPI.hasRequest(player.getUniqueId()) (as seen in your code's call to withRequirement) changes.
I don't know how your system is structured, but from a quick look at your repo (which might be out of date compared to your current code), I think you would need to expand on these lines:
https://github.com/ImNotStable/QualityEconomy/blob/089471e192cb2cfa9beb81d8e33722dea598d325/src/main/java/com/imnotstable/qualityeconomy/commands/RequestCommand.java#L80-L81
requests.computeIfAbsent(requesteeUUID, uuid -> new HashMap<>()).put(requesterUUID, amount);
Bukkit.getScheduler().runTaskLater(QualityEconomy.getInstance(), () -> requests.get(requesteeUUID).remove(requesterUUID, amount), 1200);

To properly update the requestee client on whether or not they are able to access the answer, accept, and deny multiliteral, you can do something like this:
requests.computeIfAbsent(requesteeUUID, uuid -> new HashMap<>()).put(requesterUUID, amount);
CommandAPI.updateRequirements(requestee);

Bukkit.getScheduler().runTaskLater(QualityEconomy.getInstance(), () -> {
    requests.get(requesteeUUID).remove(requesterUUID, amount);
    CommandAPI.updateRequirements(requestee);
}, 1200);

Whenever a player gains or loses a request, the result returned by QualityEconomyAPI#hasRequest in the call to withRequirement might change, so you should call CommandAPI#updateRequriements or Player#updateCommands to reevaluate the requirements.

Thank you for the help!


Performance regression in allreduce_grad in Chainer 5.2 -> 6.0 w/fp16

We've observed performance regression between 5.2 and 6.0 in our internal cluster at allreduce_grad with fp32 model.  Benchmark code: https://github.com/chainer/comm_bench
6.0

5.2

cc: @shu65

The problem may be solved if we do the following.
https://github.com/chainer/chainer/blob/master/chainer/optimizer.py#L201

Also, performance improved drastically with batched_copy=True .

We have confirmed the issue, and figured out that it is caused by massive implicit cupy.copy invoked by astype at https://github.com/chainer/chainer/blob/3a6e9ad0baf58175ba3e71459007fd13c134715c/chainermn/communicators/_memory_utility.py#L121
As we have batched_copy, a cuda kernel, to do the type casting in a much faster way, we have decided to make batched_copy default and available in all the communicators.

TODO:

[ ] Apply batched_copy to all the communicators (#7761 )
[ ] Switch to use batched_copy as default

solved by #7885


Logo proposal!

Hello! I made that logo for you.I hope you will like it.Regards!

Below you can find the Gdrive with all the files that you need:
https://drive.google.com/open?id=1_9Z4GnC_PqqGAE3eVdkbougTlIIRc1wi

amazing, i love it,
I like the design, I will implement for this project,
thanks

logo has been activated via commit https://github.com/yussan/seal-middleware/commit/7596940258644aa4467f427f282295cf3c470a74


Do not bold annotated transcript until the annotation is published

As of now the annotation is not there but the preceding text is bolded

This may not be worthy of the effort since there are plenty of edge cases like two annotations over the same transcript text, or annotations being switch back to draft and then republish.
Either we leave it as it is signaling users that a new annotation is coming over that text which might seem appropriate or we need to make this an editorial task each time a draft annotation will be published, bold the associated transcript text, but if we go down that road we will need to somehow link an annotation with the associated transcript text to make editors life easier...


feat(fsc): Integrate feature file support for testing POC - WIP

Summary

Added support for executing BDD tests directly through feature files using godog package.
Current implementations include:

Support for is_feature_enabled and all get_feature_variable API's.
Support for dispatched_events and listeners and their Subset comparison.
Support added for Template parsing for the feature-files.
Currently, only verified execution of the given feature files:
features/get_feature_variable/feature_rollouts/audience_targeting.feature
features/is_feature_enabled/feature_rollouts/audience_targeting.feature

TODO

Verify execution of all Feature-files and add missing implementations.
Add support for rest of the API's.
Add support for DFM, event-batching etc.

@mikeng13 am working on filtering out output to just show only failed scenarios.


Removes proto and golang tooling

manifest.proto moves to the canonical repo https://github.com/tetratelabs/getenvoy
Fixes #102

I'll repurpose this PR to delete the proto stuff instead of adding more tooling

the change to getenvoy is in, but I've not reworked this yet to delete the proto. I will

ok this is updated

https://github.com/tetratelabs/getenvoy-ci/pull/233 for cleanup


Light mode

This is probably very inefficient, and the Light Mode menu might be moved, but it's functional. I was going to change the renderer's background but I noticed ssbh_wgpu has no way to update the background color after initialization so I left that be.

This is breaking the font colors in dark mode. It should be relatively easy to fix. The widget colors for dark mode are modified in these lines of code. The modified widget visuals should stay the same for dark mode for improved text contrast. I'll probably end up tweaking the light mode as well, but that's not important for this PR. You just need to use the same values for the egui::style::Visual object created here for dark mode.
https://github.com/ScanMountGoat/ssbh_editor/blob/ed79dbc5779c5a2cb7345cf3f071bb971653b7e9/src/main.rs#L200-L207
Unless it's a small change like a typo, I would open an issue first to get feedback before starting an implementation. This helps avoid having to make major changes later or have the PR closed. I may set up more information on guidelines for contributing in the future. I'd be happy to list a high level overview for implementing an issue if it looks doable and someone is interested in tackling it.
In terms of efficiency, I would do some informal benchmarking before trying to optimize it. You can easily print elapsed time in Rust to get a feel for how long something takes. The times really only matter when using cargo build --release since Rust in debug mode can be quite slow.
let start = std::time::Instant::now();
// your code
println!("{:?}", start.elapsed());

The UI for this looks ok. I haven't added a preferences menu yet, so there really isn't a better place for it at the moment.

Is the location for the light_mode boolean okay? I wasn't sure whether to put it in SSBHUIState or SSBHApp. I actually originally made it as dark_mode, but then I realized I would have to initialize the value as true.
Also what about the renderer's background?

This is now being tracked in #36. Putting the field in UiState like you did is fine. I'll make a commit to ssbh_wgpu for updating the clear color after initialization. Just making a copy of widgets_dark that contains the light mode colors is fine. I'll tweak the colors later. You can find the values used here:
https://docs.rs/egui/latest/src/egui/style.rs.html#769-807

I did an efficiency test in a release build and it seemed to be alright after this edit. (It might be clunky but this was the best thing I could think of.)


[QST]GPU and CPU memory allocation

What is your question?
I want to ask whether the raw data read is stored in CPU memory or GPU video memory? And where are the intermediate results of GPU calculation stored? Exchange via shared memory? If all operators are executed on the GPU, assuming that video memory can store all data, do you still need to use CPU memory?

Okay, thanks for your reply. A data structure of the Table type is constructed when the host (CPU) memory is transferred to the device (GPU) video memory, as shown in the following figure.
Table{
columns=[ColumnVector{
rows=1,
type=STRING,
nullCount=Optional.empty,
offHeap=(ID: 1017fc576bc87d0)
},
ColumnVector{
rows=1,
type=STRING,
nullCount=Optional.empty,
offHeap=(ID: 1027fc575543a00)
}],
cudfTable=140486077350896,
rows=1
}
The batch operation of the GPU operator is the columns above. The offHeap indicates the out-of-heap memory. Does the CUDF copy the offHeap data to the device (GPU) video memory? But I didn't see it.
In addition, if the data calculated by the GPU operator is still calculated on the GPU, will the data in the video memory be directly used instead of being transmitted to the CPU? The data is copied to the CPU only when a rollback operation is performed?
During data transmission, is it considered to copy only the data of the calculation part? If the data is not involved in the calculation, do not copy the data by adding indexes?
For example:select a, b from table where a != 0
If the filter needs to be rolled back, can I copy only the data in column a to the CPU? After the CPU calculates the data, the CPU returns the data and reassembles the data?

Sorry this gets a little complicated. Most of the complexity is due to memory management and trying to make something like java which relies on garbage collection play nicely with C++/CUDA that have explicit memory management.
There are a number of object that hold some memory offHeap. This could be a memory buffer as is the case in HostMemoryBuffer or it could be a pointer to a C++ class instance as is the case for something like Scalar, or it could be a combination of them which is what happens with ColumnVector and Table. The C++ class instances may point to other resources too like GPU memory. But it is up to the C++ class to handle cleaning that all up properly.  This is why we call it offHeap, even if what it is pointing to is mostly related to GPU memory.
Internally in java for most of these java objects we are doing reference counting. So when you add a ColumnVector to a Table the Table will increment the reference count on that ColumnVector so it is not released until the Table is also closed. This lets us create a ColumnVector and treat it a lot more like a regular java object.  We can pass it around. We can add it to other objects. We just have to be sure that there are clear rules about when to decrement the reference count and when to increment it.  It is not perfect but generally it works okay. When it does not work well, then we want ways to debug this and also let the garbage collector clean up after us. To do this we separate out the parts that need to be tracked for debugging/cleanup into a separate hidden class from the public facing class.  We register it with a MemoryCleaner that will let us query what is outstanding. Keep track of when and where a reference count was changed, and also lets something similar to a finalize run that will clean up the off heap parts if the garbage collector got to it before we could free the resources through the proper means, and we can output error messages when this happens.
Generally speaking a HostColumnVector will store its data in CPU memory off heap. a regular ColumnVector will store its data in GPU memory, but will have some small amount of CPU memory to store metadata about it. A Table cannot point ot a HostColumnVector only to data on the GPU.
Now for an example like
select a, b from table where a != 0

If we assume that table is a set of parquet files stored in HDFS.  Spark on the CPU will get the list of files to scan. It will pass that information on to the rapids plugin along with the list of columns to read a and b and the predicate a != 0.  The rapids plugin will then translate that information into a set of tasks that will run on Spark, but use the GPU and CPU to do the processing. First the tasks will get a set of files and splits to read the data from. They will read the parquet footer for the files that they care about and determine the set of row groups within those files that should be read.  This would be based off of how the files are split up and if the predicate a != 0 would make it so no data in that row group would be read.  After that the data associated with the row groups in the file we care about and the column are read into CPU memory.  That memory is passed on to CUDF as a HsotMemoryBuffer.  CUDF will then parse the parquet data out of it and turn it into a Table that is returned.  The data will be on the GPU.  After that the Table, or a Spark equivalent of that called a ColumnarBatch would be passed onto a GpuFilterExec operation that would run the a != 0 expressions on the GPU to get a boolean mask, and then a new Table would be created with the filtered results. The boolean predicate column and the original input table would then be closed and the GPU memory that they point to would be released.
After that it really depends on what the next operation in the query is. It could be written out to a file in another format. It could be collected back to the CPU for further processing/etc.
I hope that this helps.

Well, thank you very much for your reply, which has helped me a lot!
Another question is, when going back from the GPU to the CPU for execution, do you need to copy all the data on the GPU to the CPU?


WIP 4.x

Implements #491
A new start from scratch for a 4.x version. Continues #732.

[x] asynchronous section rendering
[x] Better custom section integration
[x] Migrate documentation (#670)
[x] Use (Ferris the) crab for Rust symbol (#516)
[x] Make sure all options follow chosen naming conventions (#594)
[x] Get user environment with spaceship::env (#579)
[x] Drop pyenv section in favour of python section (#431)
[x] CLI interface for getting version, configuring and sending bug-reports
[x] Drops ember section. It's now an external section.
[x] Require minimal Zsh version. Closes #578

@Kadrian hi, this is basically ready. The only thing is left is to rebase that on top of the master branch, write release announcement, and it's ready to go.


devtunnel login fails with 403 on the localhost callback

The console then keeps waiting.
Tunnel CLI version: 1.0.791+d10ee4be43

Tunnel service URI        : https://global.rel.tunnels.api.visualstudio.com/
Tunnel service version    : 1.0.944.532 (14026cffe3; 2023-10-04 01:45:30Z)
Tunnel service cluster    : usw2

Thanks for reporting the issue @gakama!C an try updating to the latest Tunnel CLI and let us know if you can still repro the issue?
If so, can you try logging in as follows:
devtunnel login -d
This should allow you to login with a GitHub account with device code login, when the local interactive browser login isn't possible.

Closing this as stale and we've made improvements in this area. If you feel it deserves to stay open, please clarify your use case and provide additional information. Thanks.


⚠ Guya.moe Proxied has degraded performance

In b932eca, Guya.moe Proxied (https://ice.guya.moe/) experienced degraded performance:

HTTP code: 200
Response time: 13209 ms

Resolved: Guya.moe Proxied performance has improved in d19c5f9 after 1 hour, 51 minutes.


Date formating in "ru" locale.

Greetings!
I have:
Sugar.Date.format(new Date(), "{month}", "ru") > декабря
How can i get "декабрь" instead of "декабря", "сентябрь" instead of "сентября" and etc?

Hi,
variants can be specified with numbers in the formatting tokens. In this case, {month4} will return what you want. Since all languages are different, there's not an overall applicable rule to the different types of tokens... it's easiest just to try them out for yourself.

Thanks, Andrew, your project is awesome.

Hello... I'm considering a better system for Sugar's formatting tokens and I need your help! Could you please fill out this sheet for Russian? If you have any questions let me know! Thank you in advance!

Greetings!
Done.

Awesome! Thank you!


Assignment 1 - 6ITA - Dabhi Rajdeepsinh

#38
Number of Problems Solved: 50

@Rajdeepsinh1410
Please be mindful of the instructions for the PR

Issue number should have been linked in the Pull Request description
The number of problems solved should be mentioned in the Pull request description


feat(codegen): reduce needed imports

Opening this PR to see if you are open to directly importing from @ethersproject packages rather than ethers umbrella.
This may be a breaking change?
Here is a trivial example before we get too deep:
These two imports can be taken directly from the @ethersproject/contracts package. no need for umbrella import, should be a drop in.
next
import type { BaseContract, Signer, utils } from "ethers";

Note. this a naive commit , do not merge

I think that the current way of importing stuff assumes less about the project that the user has set up. (what if the user uses only particular packages instead of the full ethers.js package?). For that reason, I think the current state of things is better. Thanks for the PR though!


missing inheritance

Coverage remained the same at 92.403% when pulling 1cc9d34c0f9648df59252e5d3e88c3d98ceb1e2e on dev into 98cf86b74d747f2673e5175be9c4e34f12ec5dcf on master.


Change ARM client style to Operation suffixes

Fixes: https://github.com/Azure/autorest.csharp/issues/811

Merging this as I'd like to get the change into main repo sooner. Feel free to leave comments I'll address them in another pass.


Memory leak when using Metric with list state

🐛 Bug
I tried implementing a custom Metric to use as a loss when training. It seems to compute the desired values fine, however like the title states the metric quickly consumes all the memory on my single GPU. Models that previously required less than half of my GPU memory now run into OOMs after less than one epoch.
I tried replicating the issue in this Colab notebook, however the dataset and training procedure are too lightweight to replicated the resource consumption issue.
Can anyone confirm whether it's an issue with how I handle the list state myself or a bug with Metric itself? The only metric I found that uses a list state that I could draw inspiration from is the explained_variance. However, I thought I should be able to do things differently and wouldn't need to store the targets and preds in the state, only the computed results.
To Reproduce
Colab notebook that tries to reproduce the issue (with no success) on a toy example.
If it can be of further help, my real use-case is the following, where I implement the dice loss as a metric with which to train my model:
class DifferentiableDiceCoefficient(Metric):
    """Computes a differentiable version of the dice coefficient."""

    def __init__(
        self,
        include_background: bool = False,
        nan_score: float = 0.0,
        no_fg_score: float = 0.0,
        reduction: str = "elementwise_mean",
        compute_on_step: bool = True,
        dist_sync_on_step: bool = False,
        process_group: Optional[Any] = None,
    ):
        super().__init__(
            compute_on_step=compute_on_step, dist_sync_on_step=dist_sync_on_step, process_group=process_group
        )
        self.include_background = include_background
        self.nan_score = nan_score
        self.no_fg_score = no_fg_score
        assert reduction in ("elementwise_mean", "none")
        self.reduction = reduction

        self.add_state("dice_by_steps", [])

    def update(self, input: torch.Tensor, target: torch.Tensor) -> None:
        self.dice_by_steps += [
            differentiable_dice_score(
                input=input,
                target=target,
                bg=self.include_background,
                nan_score=self.nan_score,
                no_fg_score=self.no_fg_score,
                reduction=self.reduction,
            )
        ]

    def compute(self) -> torch.Tensor:
        return torch.mean(torch.stack(self.dice_by_steps), 0)

Expected behavior
I would expect to be able to use metrics/losses implemented using the Metric API exactly like I would if they inherited from nn.Module.

@SkafteNicki Thanks for the explanations. I think I might not be getting what Metric is designed to be used for. I can't use detach() because I indeed want to backprop through the metric, i.e. I implement my loss using the Metric API. Is this not recommended?
And I understand that the memory footprint might be deceptively high, since whole graphs need to be kept in memory to be able to backproped through. However, I would have assumed that since I call it using compute_on_step=True, backprop should be performed for each step, followed by a reset of the state that clears the graph in memory. Or is there no guarantee about when memory will be freed once I use a Metric?
Thanks again for the clarifications on the recommended use of Metrics!

@ananyahjha93

@nathanpainchaud all loss functions are metrics, but not all metrics are loss functions. Two requirements to loss functions that does not necessarily apply to all metrics:

Be differentiable. Accuracy is a good example of a metric that is non-differentiable and therefore is not a loss function
Can be decomposed into individual terms per sample (at least necessary if we want to do minibatching). ExplainedVariance is a good example of a metric, where the value is not a simple sum/mean over each individual sample

When you set compute_on_step=True, it is correct that the value returned from forward is the metric calculated on the current input, however it is also stored in state. Please see this part of the code:
https://github.com/PyTorchLightning/pytorch-lightning/blob/0474464c454a000e4cacfe188c86a0b8317288d5/pytorch_lightning/metrics/metric.py#L148-L165
As you can see the current state gets cached, get used to calculate for the current input, and then gets restore afterwards.
The hole idea of implementing using the Metrics class is that you want to compute a value over multiple batches, not just a single one. I would have to see more of your code to judge if using the Metric class is correct for your usecase (what does differentiable_dice_score look like and how to you call and use the metric).

@SkafteNicki I do understand the difference between metrics and loss functions :wink:
What I think I don't understand is to take into account self._cache. I now understand the caching mechanism (thanks for the explanation!) but I'm not sure how to work with it to implement a loss with a reasonable memory footprint. I don't see where self._cache gets flushed, so should I assume that the cache only grows during training, and even across epochs? Would that mean the whole computational graph MUST be kept in memory in the case of a loss that is not detached from the graph?
As for my own use-case, differentiable_dice_score closely resembles the functional dice_score (it has the same API) provided in the Metrics package, only it is the loss version of the dice coefficient that is differentiable and can be backproped through to train models. And is use it has a regular metric, i.e. create the object in the model's init, and call it in the training loop, kind of like below:
def __init__(self, ...):
    ...
    self._dice = DifferentiableDiceCoefficient(include_background=False)

def training_step(self, batch: Tuple[Tensor, Tensor], batch_idx: int) -> Tensor:
    x, y = batch
    y_hat = self.module(x)
    dice = self._dice(y_hat, y)
    return loss

@nathanpainchaud just wanted to make sure that we were on the same page 😉
If the metric update just add to a list, then yes, the memory footprint will just grow and grow. This is the very reason why most of our metric (all except for ExplainedVarariance) have a two tensors which they just add. It would seem to me that in your usecase you could do something similar:
class DifferentiableDiceCoefficient(Metric):
    """Computes a differentiable version of the dice coefficient."""

    def __init__(
        self,
        include_background: bool = False,
        nan_score: float = 0.0,
        no_fg_score: float = 0.0,
        reduction: str = "elementwise_mean",
        compute_on_step: bool = True,
        dist_sync_on_step: bool = False,
        process_group: Optional[Any] = None,
    ):
        super().__init__(
            compute_on_step=compute_on_step, dist_sync_on_step=dist_sync_on_step, process_group=process_group
        )
        self.include_background = include_background
        self.nan_score = nan_score
        self.no_fg_score = no_fg_score
        assert reduction in ("elementwise_mean", "none")
        self.reduction = reduction

        self.add_state("sum_dice_score", default=torch.tensor(0), dist_reduce_fx="sum")
        self.add_state("counter", default=torch.tensor(0), dist_reduce_fx="sum")

    def update(self, input: torch.Tensor, target: torch.Tensor) -> None:
        self.sum_dice_score += differentiable_dice_score(
                input=input,
                target=target,
                bg=self.include_background,
                nan_score=self.nan_score,
                no_fg_score=self.no_fg_score,
                reduction=self.reduction,
            )
        self.counter+=1

    def compute(self) -> torch.Tensor:
        return self.sum_dice_score / self.counter

@SkafteNicki Thanks a lot for the recommended implementation. I've just now gotten around to implementing it. However, I still get the CUDA OOM after the same number of steps.
Like I mentioned in one my earlier message, I suspect that behind the scenes PyTorch keeps the computational graph of the previous steps because the results (requiring grad) are saved as part of the state of the metric. Reading through the Metric class implementation, I didn't find anything in the built-in metrics that seemed to ensure this doesn't happen. Would you have any recommendation on what inner Metric state to monitor to understand why my metric seems to behave differently from the built-in ones?
Thanks in advance, as well as for all your help until now!

@nathanpainchaud if all you want to do is to backpropergate the metric, why don't you just do that?
def training_step(self, batch: Tuple[Tensor, Tensor], batch_idx: int) -> Tensor:
    x, y = batch
    y_hat = self.module(x)
    dice = self.differentiable_dice_score(y_hat, y)
    return dice

@SkafteNicki That is what I'm doing right now. But I wanted to see if my loss could be implemented as a Metric instead, to benefit from all the built-in functionalities the Metric API offers. But it seems that it might be more trouble than it's worth at this point...

@nathanpainchaud we are reevaluating if our metric API should support backprop, as it seems to give troubles.
You are completely right that there is nothing in the build in Metric class that prevents the computational graph from being stored.
Maybe the compromise would be to internally call detach before values are stored in the cache. With this:
val = metric(pred, target) # this can be backprop
val = metric.compute() # this cannot be backprop

@SkafteNicki how about we call detach before caching for the epoch but let it be a part of the graph for the step? That way you can backprop for the step but detach from the graph before adding it to compute over epoch and GPUs?

@ananyahjha93 that was also what I was thinking. I will send a PR soon with the update.


Update README.md

Improve data directory notes and development process description.

@dagurval Feel free to use the edit button and commit to this branch.  We can use "Squash and merge" when ready.

@xt-bot retest this


what type is a surrogate model (does it need to be its own class?)

@donghwan-shin trainLocal and trainGlobal return surrogate models what type is a surrogate model (does it need to be its own class?) i guess it could maybe have a parent class surrogate model and then have multiple types of surrogate model children (allows user to code their own child of this class).
https://github.com/SashaSaw/SAMOTA-Lib/blob/c5bc9174ad2f125b570aec91665ccee8017e93c9/LocalSearch.py#L23
https://github.com/SashaSaw/SAMOTA-Lib/blob/c5bc9174ad2f125b570aec91665ccee8017e93c9/GlobalSearch.py#L16

Yes, that sounds sensible. The common functions for the SurrogateModel instances should be: train() and predict().


What's up?
Hi, author
In this article "new directions in automated traffic analysis", I saw a heatmap of the feature importance, such as Figure 4 in the paper.
I'd like to ask how this figure is obtained and whether to used rf.feature_ importances_ in sklearn this function to draw
Due diligence

[X] I have read the docs

Hello.
Here is the old code used to generate the heatmaps. Where the feature importance for each feature comes from ultimately doesn't matter, though in the paper we used the feature importance from the random forest model trained by autogluon.
Note that the above code is over a year old and need minor tweaks to work with any newer autogluon changes.
Cheers


Question to understand resulting placement

Hello, thank you for providing a great paper code reproduction! I would like to ask, after I run the script.py, the generated placement.txt  shows that the vnf in the sfc is only placed on node 6. I don't know what the problem is. Can you help me answer it?

Hi, thanks for the kind words.
Without yet looking at the setup or script, is the outcome surprising in terms of placement? Could it be possible that placement at node 6 is actually intended and not a problem?
@stwerner97 @NilsLuca Maybe you know more?

Hi @yingchaot, sorry for the late reply!
What configuration did you use for the run? Did the training of the agent converge or increase at all? As far as I recall, we were unable to reproduce the results published in the paper. Although the agent improved over time, it could not match the results of some heuristics we tried. Still, I don't recall issues where the agent used only a single location for the placement.
Unfortunately, I am quite busy at the moment, but I will try to look into the issue at the end of next week.

Hello @stefanbschneider @stwerner97 ! Thank you for your reply!
Sorry for not accurately describing the problem before. I used abilene.gpickle and requests.json under the data folder, using PPO as the agent. The following figures are my final training results:
!

If possible, I would like to ask two more questions: 1) When I run the script.py, I change an agent, such as DQN, the request acceptance rate, average bandwidth consumption and other indicators are always 0, and no training is conducted. The training situation is shown in the following figure:

2) You set the bandwidth unit coust in the environment/netwrok.py file to 0.006, while in the original paper it was 0.0006. Did you specifically set this?
You can reply me at a convenient time, thank you again!

Hi @yingchaot, thanks for the reply! Did you also take a look at the episode returns of the agent?  I think stable_baselines3 logs these as eval/mean_reward or similar. In response to your other questions

We mainly tested the implementation with PPO. That the script does not run for other agents is a bug that I'll try to correct.
This is just an oversight; thanks for spotting the typo! If I recall correctly, we had some general issues reproducing their simulation results since the paper does not state the parameter values for node capacities nor for request demands in detail.

I think it's best to treat our implementation as an attempt to reproduce their general approach to service coordination, not their simulation results.

Hi @stwerner97, Thank you for your patient answer! However, the logs file does not display the forward parameter. I only find two related parameters, as shown in the following figure:

Have a nice weekend!

Hi @yingchao, the linked PR should fix the issues you have mentioned with regard to the evaluation. Moreover, it changes the default request scenario (i.e., increases the load).
I'll close the issue for now. If you have any additional questions, feel free to reopen it!

Thanks for all the effort @stwerner97 !

Hello sir, .I can understand the rough process, but I do not know how to use the experiment results and use them to make the figures like that. Could you give me an instruction about the experimental indicators ?(logs, placements in script.py and ray results in tuning.py). I am looking forward to your reply very much.
image
image
image


Don't crash in edgeCell when posAtCoords returns null

The current code assumes it will never do that, but that doesn't appear to be a valid assumption—a customer ran into a crash when the returned object is dereferenced.

I'm just going to go ahead and merge this, then.


TE-18.3 gRIBI and Scheduler mpls-in-udp scale feature

README is at https://github.com/openconfig/featureprofiles/pull/3375


Update request_list.html

added data table

@kira0204  someone added Django-filter to it  , and we added client-side pagination , please disable one

@JijinJayakumar @kira0204  The changes should consider different client types like low end mobile phones, bandwidth etc. means do not try to include any heavy load JS libs, any client side rendering techniques

This PR is fragmented with #8 #9 #11 and #12 please reconsider merging changes in this pattern

as we added server-side I think we can  close this :)


BF: closes #1484 for unordered map

Closes issue #1484

Otherwise looks good to me now.

Thanks!

thank you for the help


[18.06] luci-app-https_dns_proxy: bugfix: subnet address datatype, add DNS.SB provider

This addresses the issue mentioned in #3148.
Signed-off-by: Stan Grishin<EMAIL_ADDRESS>

@hnyman @jow- Should I also merge this in 18.06?
Is there a release coming?

I have seen that this is a new package.
As I know 18.06 is feature complete and new package should only go to master.
I will close this pullrequest if there is not a good reason to merge this.

Florian, the bugfix is a part of this PR.


Invalid content type error fix

If the user submits an invalid content type, the swagger validator doesn't set the failedValidation field in the error object.

I'm not sure I agree with the approach taken.  Here are my concerns:

There are a lot of changes for what could had been just a few lines of change.  I don't understand the need for so much work just to set the failedValidation property on the error
This PR breaks consistency with request validation error messages

If I were fixing this, I would use the same approach here that I do here.  That would be a few lines change maximum.

The amount of lines I added are around the same, the rest is just change of the indenting.
True looks like a lot in the diff view. I changed it that way to be consistent with the rest of the error handling in that file. didn't see any reason to handle that differently.
The only thing that changes to the outside is an anyway dynamic error message as far as I know.  (the required and the offending content types are inserted) True, thats the breaking change there. But it should be more consistent in handling (just checking for the failedValidation flag, same error handling in code, same prefix as for the other messages)
catching the error and just adding the flag seems a bit hacky for me as most errors already have that flag ;-)
thought it would be cleaner that way.

I went ahead and fixed this myself: https://github.com/apigee-127/swagger-tools/commit/beded2129620aa92df0d971508999bd8aa94d42b  I really appreciate your help but I think the simplest approach was to just stick to the existing standard and not to deviate from existing error message patterns.  The end result is the same, the implementation is just a little different.


Language docs

Applied review comments, ready for review.

@Schaeff can we merge this for now and do another iteration after Powdr2 is merged?