can't login to youtube WARNING: Unable to look up account info: HTTP Error 400: Bad Request

Checklist

[ x] I'm reporting a broken site support
[ ]x I've verified that I'm running youtube-dl version 2020.07.28
[ x] I've checked that all provided URLs are alive and playable in a browser
[ x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[ x] I've searched the bugtracker for similar issues including closed ones

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--cookies', 'cookies.txt', '--user-agent=Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', '-f', '141', '-u', 'PRIVATE', '-p', 'PRIVATE', '-v', 'https://music.youtube.com/watch?v=Leq8J0E2TQ0']
[debug] Encodings: locale cp1254, fs mbcs, out cp857, pref cp1254
[debug] youtube-dl version 2020.07.28
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg git-2020-07-16-d11cc74, ffprobe git-2020-07-16-d11cc74
[debug] Proxy map: {}
[youtube] Downloading login page
[youtube] Looking up account info
WARNING: Unable to look up account info: HTTP Error 400: Bad Request
[youtube] Leq8J0E2TQ0: Downloading webpage
[youtube] {18} signature length 105, html5 player c718385a
[youtube] {137} signature length 105, html5 player c718385a
[youtube] {248} signature length 105, html5 player c718385a
[youtube] {136} signature length 105, html5 player c718385a
[youtube] {247} signature length 101, html5 player c718385a
[youtube] {135} signature length 105, html5 player c718385a
[youtube] {244} signature length 105, html5 player c718385a
[youtube] {134} signature length 101, html5 player c718385a
[youtube] {243} signature length 105, html5 player c718385a
[youtube] {133} signature length 105, html5 player c718385a
[youtube] {242} signature length 105, html5 player c718385a
[youtube] {160} signature length 101, html5 player c718385a
[youtube] {278} signature length 105, html5 player c718385a
[youtube] {140} signature length 105, html5 player c718385a
[youtube] {249} signature length 105, html5 player c718385a
[youtube] {250} signature length 105, html5 player c718385a
[youtube] {251} signature length 101, html5 player c718385a
ERROR: requested format not available
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpq2whuxta\build\youtube_dl\YoutubeDL.py", line 808, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpq2whuxta\build\youtube_dl\YoutubeDL.py", line 863, in process_ie_result
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpq2whuxta\build\youtube_dl\YoutubeDL.py", line 1636, in process_video_result
youtube_dl.utils.ExtractorError: requested format not available

Description

So, here is the thing. I have an Youtube Music Premium subscription which lets me download music with audio format 141. To do that obviously, I need to login to youtube. So I tried my darn hardest to do so, I've tried cookies, I've tried user agent, I even tried changing my gmail address but none of them worked! I either get error 404 unable to look up account data or error 400 bad request
also if you are going to tag this as duplicate, at least tell me what issue's duplicate this is!

same problem

I have a similar error, but instead HTTP 400 on "Looking up account info" error I get HTTP 404.
It looks like Google has changed something with authenticating mechanism so authenticating with -u and -p is not working.
I have tried to investigate the issue but with no luck so far. I tried to output the request URL and data it tries to POST then sending the same request with curl (same headers, same URL, same DATA) and it returns HTTP 200 with something that looks like a malformed JSON with additional symbols in front of it. I couldn't understand why the results with CURL is different so far, there must be some difference that I considered insignifocant.

Me too. I am having similar error
I have the .netrc file in ~/.netrc
Content of the .netrc file is:
machine youtube login  password
I am using the youtube-dl command line:
youtube-dl --netrc -o - "https://m.youtube.com/watch?v=xCl5BtloMy4" | mpv -
I am getting the error:
Resuming playback. This behavior can be disabled with --no-resume-playback. [file] Reading from stdin... [youtube] Downloading login page [youtube] Looking up account info WARNING: Unable to look up account info: HTTP Error 400: Bad Request

same problem here

CR (fake credentials, the same happens on real ones):
 laura@iino  ~  youtube-dl --verbose --username daybyday --password imfedupwithallthis --print-traffic https://www.youtube.com/watch\?v\=T_Rn8TAIL2w\&list\=RDMMT_Rn8TAIL2w\&start_radio\=1
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '--username', 'PRIVATE', '--password', 'PRIVATE', '--print-traffic', 'https://www.youtube.com/watch?v=T_Rn8TAIL2w&list=RDMMT_Rn8TAIL2w&start_radio=1']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.07.28
[debug] Python version 3.8.4 (CPython) - Linux-5.7.13_1-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1
[debug] Proxy map: {}
[youtube:playlist] Downloading login page
send: b'GET /ServiceLogin?disable_polymer=true HTTP/1.1\r\nHost: accounts.google.com\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3632.1 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: text/html; charset=utf-8
header: X-Frame-Options: DENY
header: x-auto-login: realm=com.google&args=continue%3Dhttps%253A%252F%252Faccounts.google.com%252FManageAccount%253Fnc%253D1
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: Mon, 01 Jan 1990 00:00:00 GMT
header: Date: Fri, 07 Aug 2020 00:25:05 GMT
header: Content-Encoding: gzip
header: Strict-Transport-Security: max-age=31536000; includeSubDomains
header: Content-Security-Policy: script-src 'report-sample' 'nonce-ER88lXDu96PS54/7LXROLQ' 'unsafe-inline' 'unsafe-eval';object-src 'none';base-uri 'self';report-uri /cspreport
header: X-Content-Type-Options: nosniff
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Set-Cookie: GAPS=1:avwYaG2g2zAziX7kKij1XisxeQCDZQ:I_UHO-8jwP1KJaAO;Path=/;Expires=Sun, 07-Aug-2022 00:25:05 GMT;Secure;HttpOnly;Priority=HIGH
header: Set-Cookie: __Host-GAPS=1:avwYaG2g2zAziX7kKij1XisxeQCDZQ:I_UHO-8jwP1KJaAO;Path=/;Expires=Sun, 07-Aug-2022 00:25:05 GMT;Secure;HttpOnly;Priority=HIGH
header: Alt-Svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
header: Connection: close
header: Transfer-Encoding: chunked
[youtube:playlist] Looking up account info
send: b'POST /_/signin/sl/lookup?disable_polymer=true HTTP/1.1\r\nContent-Length: 1035\r\nHost: accounts.google.com\r\nCookie: GAPS=1:avwYaG2g2zAziX7kKij1XisxeQCDZQ:I_UHO-8jwP1KJaAO; __Host-GAPS=1:avwYaG2g2zAziX7kKij1XisxeQCDZQ:I_UHO-8jwP1KJaAO\r\nContent-Type: application/x-www-form-urlencoded;charset=utf-8\r\nGoogle-Accounts-Xsrf: 1\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3632.1 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
send: b'__msgId__=%27%2B_.R%28a.Fr%29%2B%27&pstMsg=1&checkConnection=youtube&checkedDomains=youtube&ct=%27%2B_.R%28c%29%2B%27&%27%2B_.R%28a.name%29%2B%27=%27%2B_.R%28b%29%2B%27&hl=en&deviceinfo=%5Bnull%2Cnull%2Cnull%2C%5B%5D%2Cnull%2C%22US%22%2Cnull%2Cnull%2C%5B%5D%2C%22GlifWebSignIn%22%2Cnull%2C%5Bnull%2Cnull%2C%5B%5D%5D%5D&f.req=%5B%22daybyday%22%2C+null%2C+%5B%5D%2C+null%2C+%22US%22%2C+null%2C+null%2C+2%2C+false%2C+true%2C+%5Bnull%2C+null%2C+%5B2%2C+1%2C+null%2C+1%2C+%22https%3A%2F%2Faccounts.google.com%2FServiceLogin%3Fpassive%3Dtrue%26continue%3Dhttps%253A%252F%252Fwww.youtube.com%252Fsignin%253Fnext%253D%25252F%2526action_handle_signin%253Dtrue%2526hl%253Den%2526app%253Ddesktop%2526feature%253Dsign_in_button%26hl%3Den%26service%3Dyoutube%26uilel%3D3%26requestPath%3D%252FServiceLogin%26Page%3DPasswordSeparationSignIn%22%2C+null%2C+%5B%5D%2C+4%5D%2C+1%2C+%5Bnull%2C+null%2C+%5B%5D%5D%2C+null%2C+null%2C+null%2C+true%5D%2C+%22daybyday%22%5D&flowName=GlifWebSignIn&flowEntry=ServiceLogin&bgRequest=%5B%22identifier%22%2C%22%22%5D'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: Mon, 01 Jan 1990 00:00:00 GMT
header: Date: Fri, 07 Aug 2020 00:25:06 GMT
header: Content-Type: text/html; charset=utf-8
header: Content-Encoding: gzip
header: X-Content-Type-Options: nosniff
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Alt-Svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
header: Connection: close
header: Transfer-Encoding: chunked
WARNING: Unable to look up account info: HTTP Error 400: Bad Request
[youtube:playlist] Downloading playlist RDMMT_Rn8TAIL2w - add --no-playlist to just download video T_Rn8TAIL2w
[youtube:playlist] RDMMT_Rn8TAIL2w: Downloading page 1 of Youtube mix
send: b'GET /watch?v=T_Rn8TAIL2w&list=RDMMT_Rn8TAIL2w&disable_polymer=true HTTP/1.1\r\nHost: www.youtube.com\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3632.1 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: X-Content-Type-Options: nosniff
header: Accept-CH: DPR
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: X-Frame-Options: SAMEORIGIN
header: Strict-Transport-Security: max-age=31536000
header: Content-Encoding: gzip
header: Content-Type: text/html; charset=utf-8
header: Expires: Tue, 27 Apr 1971 19:44:06 GMT
header: Accept-CH-Lifetime: 2592000
header: Cache-Control: no-cache
header: Date: Fri, 07 Aug 2020 00:25:06 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Set-Cookie: VISITOR_INFO1_LIVE=kPZZR5ugzBI; path=/; domain=.youtube.com; secure; expires=Wed, 03-Feb-2021 00:25:06 GMT; httponly; samesite=None
header: Set-Cookie: GPS=1; path=/; domain=.youtube.com; expires=Fri, 07-Aug-2020 00:55:06 GMT
header: Set-Cookie: YSC=W3w416LjTF0; path=/; domain=.youtube.com; secure; httponly; samesite=None
header: Set-Cookie: CONSENT=WP.289b92; expires=Fri, 01-Jan-2038 00:00:00 GMT; path=/; domain=.youtube.com
header: Alt-Svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
header: Connection: close
header: Transfer-Encoding: chunked
[download] Downloading playlist: RDMMT_Rn8TAIL2w
[youtube:playlist] playlist RDMMT_Rn8TAIL2w: Collected 0 video ids (downloading 0 of them)
[download] Finished downloading playlist: RDMMT_Rn8TAIL2w

Кто кому что должен?

Any updates on this one?

Any updates on this one?

nope. still gives the same error

I have to add I also ran into this issue. I noticed it that the login is needed for age restricted videos.

JDownloader 2 can download these videos, why is this program unable to do so? JDownloader is open-source, why don't have a look over there?

does really anyone expect youtube-dl to do anything else than quickfixing youtube extractor when it starts breaking for all videos?

Edit: Why am I getting down votes for mentioning how others solved the problem or never had it in the first place? An odd kind of online tribalism?

Because Sir, you are not very polite. Thanks to selfisekai i can test my self the same problem with de HTTP dialog. The problem is clearly the post to /_/signin/sl/lookup is not accepted by the domain https://accounts.google.com/ anymore. For sure google change something in the authorization method. I'm not proficiency enough for resolve the issue. I hope the developer see these message and have time to study the subject.

does really anyone expect youtube-dl to do anything else than quickfixing youtube extractor when it starts breaking for all videos?

At these time is irrelevant to most of youtube-dl users because these problems only arise when you need to login. Mostly videos can be watch in anonymous way. The other possible solution is to use cookies sessions but i hope these bug can be fixed in the future soon.
Thanks.

Because Sir, you are not very polite.

What about my comment was impolite?

Because Sir, you are not very polite.
lmao he was not being impolite at all.

Let's not get into this please...

Any new of this issue ??

I am also having this issue.  Running version: 2020.09.20

Issue unresolved.
I am using version: 2020.12.02

i guess that's still a thing related to dealing with 2fa? any chance this issue being picked on after a few years of reporting?

I don't have 2fa enabled and am having the same problem. This is a different issue. Google probably changed the headers required for login.

I don't have 2fa enabled and am having the same problem. This is a different issue. Google probably changed the headers required for login.

Tried on two accounts, one with 2fa and one without, same issue on both
Im using version: 2021.01.16

Tried on two accounts, one with 2fa and one without, same issue on both
Im using version: 2021.01.16

+1

Seeing this as well on age-gated videos. App-specific passwords also don't work.

I could get it working with cookies..

@lixaotec Interesting. Can you please give the command line you have used ?

use chrome extension Get Cookies
copy the cookie and save as file.txt
then pass arg -cookies pathtothefile.txt

@lixaotec OK, thanks. So I guess youtube-dl cannot connect to a YouTube account on its own without using the help of an external browser, is that right ?

@sebma , no, its not right. Yes it can do it without a browser, however, I suspect that at this moment, login method by user and pass is not working anymore, so you need to trick it by using cookies.

@lixaotec Does youtube-dl provide another login method for YouTube.com ?

It's still not working in version 2021.03.03.

Solved my issue by commenting --netrc, it seems to override --cookies.
Without --netrc no more HTTP Error 400: Bad Request errrors.

@Poilaucul I have not used the --netrc option and have no ~/.netrc file still youtube-dl does return a HTTP 400 Error.

Still broken in 2021.04.01. --print-traffic also seems to do nothing on Windows even forcibly specifying -v.
Example:
youtube-dl.exe -v --print-traffic -F -u<EMAIL_ADDRESS>-p app-password --cookies C:\ytdl\ytmcookies.txt https://music.youtube.com/watch?v=nwa9pb2fawY
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--print-traffic', '-F', '-u', 'PRIVATE', '-p', 'PRIVATE', '--cookies', 'C:\\ytdl\\ytmcookies.txt', 'https://music.youtube.com/watch?v=nwa9pb2fawY']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2021.04.01
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg N-86848-g03a9e6f, ffprobe git-2020-01-15-0dc0837
[debug] Proxy map: {}
[youtube] Downloading login page
[youtube] Looking up account info
WARNING: Unable to look up account info: HTTP Error 400: Bad Request
[youtube] nwa9pb2fawY: Downloading webpage
[info] Available formats for nwa9pb2fawY:
format code  extension  resolution note
249          webm       audio only tiny   46k , webm_dash container, opus @ 46k (48000Hz), 1.27MiB
250          webm       audio only tiny   61k , webm_dash container, opus @ 61k (48000Hz), 1.68MiB
251          webm       audio only tiny  123k , webm_dash container, opus @123k (48000Hz), 3.36MiB
140          m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2@129k (44100Hz), 3.51MiB
160          mp4        144x144    144p   10k , mp4_dash container, avc1.4d400b@  10k, 25fps, video only, 303.83KiB
278          webm       144x144    144p   19k , webm_dash container, vp9@  19k, 25fps, video only, 532.60KiB
394          mp4        144x144    144p   19k , mp4_dash container, av01.0.00M.08@  19k, 25fps, video only, 538.76KiB
133          mp4        240x240    240p   19k , mp4_dash container, avc1.4d400c@  19k, 25fps, video only, 535.04KiB
242          webm       240x240    240p   34k , webm_dash container, vp9@  34k, 25fps, video only, 961.72KiB
... etc.

(no difference if I use --cookies=C:\path\to\file.txt, as opposed to separating the option and path with a space)

This is a problem for me also.
Using a cookies.txt file DOES work – I used the Firefox addin to save my YouTube cookies and supplied that file to youtube-dl.  i.e. youtube-dl --cookies "/Users/[home]/Desktop/cookies.txt"   "https://www.youtube.com/playlist?list=WL", worked perfectly.
But, I can't find a way to do that by supplying credentials directly i.e. youtube-dl --username  --password  URL returns "WARNING: Unable to look up account info: HTTP Error 400: Bad Request".
Anyway, I'm happy as long as the cookies method works while work is being done on the credentials method.

@section83 Are you able to download in a better quality than 128Kbits (format code 141)? I wasn't by doing as you described.

Tested just now and used the YTDL defaults to DL a video with format codes f313 and f140:
140          m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2@129k (44100Hz), 13.32MiB
313          webm       3840x1920  2160p 9469k , webm_dash container, vp9@9469k, 24fps, video only, 973.75MiB
It looks really good on my 2560x1440 screen.
Does that help ?

This issue is about getting YouTube Music Premium media in good quality, not about video.
I have an Youtube Music Premium subscription which lets me download music with audio format 141...

My apologies – I should have read the first posts more carefully.
I don't have a YT Music Premium subscription so, can't test.  Is your problem the same as the OP ?  If so, what sound quality do you get through a normal download in the YouTube Music app ? If you can't get better than 128Kbits, complain to Google/YouTube – they promise 256Kbits so as a Premium subscriber, you should be able to download it in their app.  From reading online discussions, it seems that YT do not offer downloads via the desktop – only in their app.  So, I would not be surprised if YTDL can't access Premium accounts.
Another possibility is that your cookies file doesn't have the required data to login to your Premium account.  You might need to play with the relevant extensions in Chrome and/or Firefox to fix that.
Another post has been lodged (#28883) about the same problem.  So, maybe the YTDL developers will look at the issue some more.

use chrome extension Get Cookies
copy the cookie and save as file.txt
then pass arg -cookies pathtothefile.txt

That's works for me

MrJmpl3 , thanks.
Yes, there's a known problem with YTDL – the "--username --password" parameters don't login to YT.  So far the only way I've seen which works is using the cookies method as you've found.  It might be possible to get a ".netrc" file to work but, I've not added that option to MacYTDL as yet.  Maybe in v1.19.
Cheers.

As mentioned above, a workaround is to log in to YouTube on a browser, copy (e.g. using a browser extension) the cookies stored for the youtube.com domain to a text file and pass the resulting file to youtube-dl via the --cookies option.
Since many users have this problem, see also #28147, I wonder why youtube-dl does not give this hint upon
ERROR: Private video

or at least upon the more specific
WARNING: Unable to look up account info: HTTP Error 400: Bad Request

I suggest re-code the auth function and fallback to old google login page
https://accounts.google.com/ServiceLogin?service=youtube&hl=en

For bgResponse, it's recommend to open the devtools and copy the code out
Actually reversing the bgResponse is not recommended since Google will patch it really quick

Note that the token only available about 10 minutes around, after that you need to do the process again
When using the bgResponse token, make sure Email or phone and the value of -u stay same to prevent being rejected
It success, you'll be redirected to pwd/1 and you can enter the password there
Tips: Watch out gxf value
For most people without 2FA enabled, you should be redirected to CheckCookie page and get your cookie there
For people who with 2FA enabled, you'll first meet a verification or jumped to selectchallenge page

It's just a simple list + form, all you need is select data-challengeentry and data-unavailable is unavailable method
For those method please do it yourself, I'm just telling the workflow
To select another challenge, you'll need to GET /skip/? url and jump back to selectchallenge page
After that you should be redirected to CheckCookie page and get your cookie there
This is the best I could do, hope someone will notice this

Still broken 2021.09.22

Yep. I setup youtube-dl running in a docker container periodically polling a public but hidden playlist where I put videos I want it to download. It looks like I'll have to stop it and go back to manually saving a cookie and then downloading videos. Google doesn't make things easy for us. That is for sure.
I may modify my container so it can skip over videos that require a login, and it can log which ones I need to d/l manually.

@majorgear The youtube-dl has not released any new version since v2021.06.06 : #29965. I propose you switch like me (and maybe many others) to yt-dlp (which options' are youtube-dl compatible).

@sebma Thanks a lot for the hint, I was already wondering, why there's no update in youtube-dl anymore.
For all others, yt-dlp fixed my issue:
yt-dlp --extract-audio --cookies cookies.txt https://music.youtube.com/playlist?list=XXXXXX


Fix path problem from tutorial

According to my test result, if the path be like "[PATH]/main.mjs", program will find a folder called "main.mjs", except the file. Then, throw exception "Path does not exist". By fixing this problem, just remove "main.mjs" from your path string (like "[PATH]", except "[PATH]/main.mjs"). Then it will work properly.

First of all, thank you for your feedback; however, when I tested on my own computer (3.11 + 0.2.2), the program did not throw any errors, regardless of whether "main.mjs" was added at the end of the path or not. (This is because there is a piece of code on line 540 of build.py that automatically adapts to whether "main.js" is added or not.)


Updated typo for the imageUrl to image

Changed imageURL to image to fix the error

Thanks for the fix @codewithpom


Fix testcases for clientApiError seen when service reconcile is triggered by container stop/delete/purge

@alena1108 , can you review the changes done to address https://github.com/rancher/rancher/issues/5961 ?

LGTM


conflicts with slim/http

Hi,
I have been working on a test project with http-interop/http-factory.
The aim of test project was to showcase using the test-project with any psr-7 framework. But due to slim 3.x is not using the same Request / Response classes this makes composer hard to understand the correct class what I need. So instead of requiring slim/http it will take slim/slim and viseversa.
I would love if conflict is actually added in the 3.x
Have also initiated one in : https://github.com/http-interop/http-factory-slim/pull/5
Thank you for the consideration.

Coverage remained the same at 97.954% when pulling 026de43aa40fe799ea7ef18148b65028d2693f7c on harikt:patch-1 into c138554cb16a92156db787143ab1f266dece7c7f on slimphp:3.x.

Hey @harikt, I believe you have cleared this up on your end. Slim does not offer at this point in time a stable standalone PSR-7 Implementation. Slim/Http is not considered stable at all.
The conflict check does not make a lot of sense as the Slim/Http branch will be targeted at 4.0 and above.

Hey @geggleto ,
Thank you for your response.

We'll solve this for 4.


Boxstarter starting Calling Chocolatey to install malware-jail.fireeye. This may take several minutes to complete...
Upgrading the following packages:
malware-jail.fireeye
By upgrading you accept licenses for the packages.
malware-jail.fireeye v2<IP_ADDRESS> is the latest version available based on your source(s).

Chocolatey upgraded 0/1 packages. 0 packages failed.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
Did you know the proceeds of Pro (and some proceeds from other
licensed editions) go into bettering the community infrastructure?
 This may take several minutes to complete... 00:00:02.4113734
Boxstarter finished Calling Chocolatey to install malware-jail.fireeye. This may take several minutes to complete... 00:00:02.4113734
Boxstarter starting Calling Chocolatey to install apktool.flare. This may take several minutes to complete...
Boxstarter starting Calling Chocolatey to install apktool.flare. This may take several minutes to complete...
Upgrading the following packages:
apktool.flare
By upgrading you accept licenses for the packages.
apktool.flare v<IP_ADDRESS> is the latest version available based on your source(s).

Chocolatey upgraded 0/1 packages. 0 packages failed.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Boxstarter finished Calling Chocolatey to install apktool.flare. This may take several minutes to complete... 00:00:00.9503661
Boxstarter finished Calling Chocolatey to install apktool.flare. This may take several minutes to complete... 00:00:00.9503661
Boxstarter starting Calling Chocolatey to install flare-qdb.python.flare. This may take several minutes to complete...
Boxstarter starting Calling Chocolatey to install flare-qdb.python.flare. This may take several minutes to complete...
Upgrading the following packages:
flare-qdb.python.flare
By upgrading you accept licenses for the packages.
flare-qdb.python.flare v<IP_ADDRESS> is the latest version available based on your source(s).

Chocolatey upgraded 0/1 packages. 0 packages failed.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Boxstarter finished Calling Chocolatey to install flare-qdb.python.flare. This may take several minutes to complete... 00:00:00.9955173
Boxstarter finished Calling Chocolatey to install flare-qdb.python.flare. This may take several minutes to complete... 00:00:00.9955173
Boxstarter starting Calling Chocolatey to install floss.python.flare. This may take several minutes to complete...
Boxstarter starting Calling Chocolatey to install floss.python.flare. This may take several minutes to complete...
Upgrading the following packages:
floss.python.flare
By upgrading you accept licenses for the packages.
floss.python.flare v<IP_ADDRESS> is the latest version available based on your source(s).

Chocolatey upgraded 0/1 packages. 0 packages failed.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Boxstarter finished Calling Chocolatey to install floss.python.flare. This may take several minutes to complete... 00:00:01.2288141
Boxstarter finished Calling Chocolatey to install floss.python.flare. This may take several minutes to complete... 00:00:01.2288141
Boxstarter starting Calling Chocolatey to install fakenet-ng.python.flare. This may take several minutes to complete...
Boxstarter starting Calling Chocolatey to install fakenet-ng.python.flare. This may take several minutes to complete...
Upgrading the following packages:
fakenet-ng.python.flare
By upgrading you accept licenses for the packages.
fakenet-ng.python.flare v<IP_ADDRESS> is the latest version available based on your source(s).

Chocolatey upgraded 0/1 packages. 0 packages failed.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Boxstarter finished Calling Chocolatey to install fakenet-ng.python.flare. This may take several minutes to complete... 00:00:01.0263402
Boxstarter finished Calling Chocolatey to install fakenet-ng.python.flare. This may take several minutes to complete... 00:00:01.0263402
Boxstarter starting Calling Chocolatey to install flarevm.config.flare. This may take several minutes to complete...
Boxstarter starting Calling Chocolatey to install flarevm.config.flare. This may take several minutes to complete...
Upgrading the following packages:
flarevm.config.flare
By upgrading you accept licenses for the packages.
flarevm.config.flare v<IP_ADDRESS> is the latest version available based on your source(s).

Chocolatey upgraded 0/1 packages. 0 packages failed.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Boxstarter finished Calling Chocolatey to install flarevm.config.flare. This may take several minutes to complete... 00:00:00.9879471
Boxstarter finished Calling Chocolatey to install flarevm.config.flare. This may take several minutes to complete... 00:00:00.9879471
0
Only an exit code of non-zero will fail the package by default. Set
--failonstderr if you want error messages to also fail a script. See
choco -h for details.
Environment Vars (like PATH) have changed. Close/reopen your shell to
see the changes (or in powershell/cmd.exe just type refreshenv).
The install of flarevm.installer.flare was successful.
Software install location not explicitly set, could be in package or
default install location if installer.

Chocolatey installed 1/1 packages. 0 packages failed.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
++ Boxstarter finished Calling Chocolatey to install flarevm.installer.flare. This may take several minutes to complete... 00:07:43.9881683
True
Boxstarter: Restore Automatic Updates from Windows Update
Errors       : {}
ComputerName : localhost
Completed    : True
FinishTime   : 3/7/2020 8:09:02 AM
StartTime    : 3/7/2020 8:01:16 AM
PS C:\Users\IEUser\Desktop\flare-vm>
PS C:\Users\IEUser\Desktop\flare-vm>`

Thanks for the new update!!
Installation is successful with the fresh windows 10 without any updates.


new.css#L41

Hi, in new.css#L41
maybe you should replace this:

| --nc-ac-tx: var(--nc--dac-tx);
with this:

| --nc-ac-tx: var(--nc-d-ac-tx);

new.css#L39
this:

| --nc-lk-tx: var(--nc--dlk-tx);
with this:

| --nc-ac-tx: var(--nc-d-lk-tx);

Just so you know, this has been reported several times and I also created a PR to address that a few month ago https://github.com/xz/new.css/pull/41
The maintainer won't merge it because these variables should disappear in a next V2

@statox ok Thanks a lot.


fix multiple items ios

Hi, I'm trying to fix this issue https://github.com/meedan/react-native-share-menu/issues/77 based on the code I check it will return the first item using a callback. In my case, I'm trying to share multiple messages (text) from WhatsApp but I always get one message only.
so in this code, I add a code to wait until all item provider finishes and pass to the callback. so it might have changes on return data()
before:
{data: "", mimetype: ""}

now:
[{data: "", mimetype: ""}]

hope it helps, thank you.

@RZulfikri I have a PR opened that fixes this - https://github.com/meedan/react-native-share-menu/pull/119

If you need this feature right now, check this: https://github.com/meedan/react-native-share-menu/pull/158


Uncaught exit - {:timeout, {GenServer, :call, [Lexin.Dictionary.Worker, {:definitions, "albanian", ...

More details here: https://lexin-mobi.sentry.io/issues/5108703671/

The refactoring that is done here reduces the consequences of similar errors.


Qmake / make targets: run and doxygen

For development convenience, I would like to suggest adding the "run" and "doxygen" targets to the command line make targets.

Thanks :+1:


Update to syn 2

https://github.com/dtolnay/syn/releases/tag/2.0.0

@dtolnay This broke CI on one of my projects that targets Rust MSRV 1.37.0.
I have to either pin the dependency to thiserror = "=1.0.39" or hope that version 1.0.40 is yanked?
$ cargo +1.37.0 test --all
warning: unused manifest key: package.rust-version
    Updating crates.io index
  Downloaded thiserror v1.0.40
  Downloaded thiserror-impl v1.0.40
  Downloaded syn v2.0.2
error: failed to parse manifest at `C:\Users\jay\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-2.0.2\Cargo.toml`

Caused by:
  failed to parse the `edition` key

Caused by:
  supported edition values are `2015` or `2018`, but `2021` is unknown


scuttlebutt-handshake implementation assumes 1 full message is read at once

I don't have steps to reproduce this, but I think it might be an issue (from reading the code.)
A scuttlebutt RPC message has a 34 bit header, followed by a variable length body. When bytes are received over the socket, the current implementation of scuttlebutt-handshake assumes that it is a complete message in the byte buffer handed over by Vertx.
https://github.com/ConsenSys/cava/blob/e8ea2b897a2ff0a0ed4ae9c47300583b0f0874dc/scuttlebutt-handshake/src/main/java/net/consensys/cava/scuttlebutt/handshake/vertx/SecureScuttlebuttVertxClient.java#L62
Depending on network conditions, application load, etc, this could theoretically be only part of a message in this buffer (or perhaps even 1 RPC response and a little bit of another.) Perhaps this could be re-implemented to make sure we buffer up full messages first before handing over the bytes to the receivedMessage method.

Yes, it's possible. In practice I have seen vert.x buffer multiple messages at once.

This PR #199 fixes this issue. That indeed happened with patchbay where 512 bytes were sent at a time.


Koenig embed card support

Ghost 2.0's soon-to-be-launched editor (Koenig) has full support for embeds within the editor. Embed cards will pull content from an external service which will usually include scripts or iframes to display things like tweets or videos.
Ghost-Desktop loads+displays embeds fine but shortly afterwards they turn into white blocks where the content was once visible.

Huh, fascinating. First debug step should be to update Electron to v3, I'm hoping that a later Chrome version will fix this for us.

Testing a naive Electron upgrade (npm i electron@beta, npm start) the embed cards no longer get switched to white boxes so I think there's hope! 😄

Closing as we're on electron v3 and the issue no longer exists.


Validate Certification of Update Calls

Currently blocked by dfinity-lab/ic-ref#90

I believe this has been done for a while in Certificate


How do you catch request to  failed before engine starts

It is very simple to catch errors after engine starts as you can use
engine.on('error',(err) =>{
});

but I am having a challenge catching an error if the couldn't start at all, for example, trying to start with no connection.
For me, it seems that all error handling is done on the basis that there is a connection in the first place. But fails to push the error out of the prototype.
More specifically here
Web3ProviderEngine.prototype.start = function(){
  const self = this
  // start block polling
  self._blockTracker.start()
}

Web3ProviderEngine.prototype.start now takes a callback


Input system shortcomings make Godot unusable for 4-player splitscreen on controller

Tested versions
Tested versions are every 4.x release, including dev releases. Currently on 4.4 dev 6
System information
Windows 10 (any) and C#
Issue description
Godot 4 users are expected to add inputs via the Input Map in the settings. This is similar to Unity's "New" input system. However, Godot's input actions are per-device

So if you are making a complicated game (like we are), where there are dozens of inputs, you will have to duplicate each of those dozens 4 times for each device. In our case this would lead to 200+ actions and you'd have to hope there are 0 mistakes. This is horrible. And what if you want to change bindings in-game? Just give up. If you go with "All Devices" like in that image, then any device will trigger the action, instead of for one player.
We decided to instead go with the classic approach of "assigning" a device to each player in the game. You're told to do this all over the web anyway so you might as well.
In C# you do this with Input.JoyConnectionChanged += Blabla;
This lets you know when a new joypad is connected and when an existing one disconnects. Fantastic, now I can assign a device to a player and then poll for the button presses per device!
Nope you're wrong again because the "device" ids are not unique. So if I want player 0 to use device 0, player 1 to use device 3, player 2 to use device 1 and player 3 to use device 2, then you're out of luck. The Input Map also won't cover you there.
What's wrong, why can't this work? Well, firstly the input system seems to break if two of the same controller kind are connected at once, for example 2 PS4 Controllers. Input.GetJoyInfo() returns a blank dictionary for some reason, even on Windows where the documentation says it clearly does not. Each device has the same Guid and name as well so no way of knowing which is which. You're told to go strictly by device ID but these device IDs are not even what the cpp backend of Godot is using. And you're given no objects or handles to hold so again there really is no way to assign a joypad to a specific user. Not to mention, InputEventJoypadMotion is never reported in certain Control nodes under _GuiInput() (no idea how to tell if it will work or not either).
Also, we have had 0 Xbox controllers work in Godot ever since we started using it in March 2024, EVEN WITH the Input Map. Yes it really has been over 9 months of updates and not once has an xbox controller returned any inputs on any windows system we tried. It doesn't matter if it's in the editor, exported, release/debug etc. They are notified with Input.JoyConnectionChanged but every type of polling you do results in default values (axis is 0, no vibration, buttons are never pressed). So it's like they're shadowbanned from Godot. But somehow PS4, switch, snes, 8bitdo, and knockoff controllers work in Godot flawlessly. If we try to use steam input with an xbox controller, the buttons are all wrong (LB is somehow A, Y is Start etc). This is beyond insane and I don't know how to feel about it other than baffled. It definitely works fine in Godot 3, of which many games support it, but alas here we are in Godot 4 with no xbox support. If it's working for you, CONGRATULATIONS you have won the lottery or something! Please share your secret.
Even when you do get a joypad, you don't know what type it is (so we can show relevant button icons) unless you do a switch statement on its name and filter by known names. But when it comes to knockoff and third-party controllers, good luck maintaining that huge list yourself.
The other great thing about the non-unique device IDs is the fact that they are incorrectly reported (at least in C#, idk about gdscript). You can disconnect device 0 and it says "wow device 1 was disconnected!". Then the other devices that are still connected (which you assume one of them got set to device 0) report no inputs when polling device 0. So yeah there's no way to have controller splitscreen with Godot (at least in C#. I am not trying gdscript).
So my solution is to just include SDL's binaries in the project, similar to steam's binaries, and make my own entire SDL wrapper and completely ignore godot's Input system since it doesn't function.  Yes it is necessary. Yes it was easier and yes it works.
This issue is mostly a rant but it is backed with only facts. There are too many broken things to focus on and make 111111 issues about. I'm not sure how this engine existed this many years with these issues without outrage. I guess everyone who attempts controller multiplayer just gives up or goes back to Unity. If this ever gets fixed, great, but I really don't expect it to anytime soon. I still care a lot about Godot's future but it's hard to look at a shortcoming this bad and not be upset. That's why I made this issue at all.
Steps to reproduce
Simplest repro with 2 PS4 controllers:
C# setup:
Input.JoyConnectionChanged += Input_JoyConnectionChanged;

...
private static void Input_JoyConnectionChanged(long devicee, bool connected)
{
	// TODO: Godot is broken af
	// Having 2 PS4 controllers plugged in gives you [0, 1] as expected but info for both is empty even on windows
	// Disconnecting either of the controllers results in joy 1 being disconnected
	// Then polling for input on device 0 still returns nothing even though supposedly GetConnectedJoypads() is [0]
	// So if I wanted to assign device 0 or 1 to someone, it's impossible, since these values mean nothing

	int device = (int)devicee; // Why are you even giving me a long?
	string info = Input.GetJoyInfo(device).ToString(); // TODO: Why is info empty?
	DebugLog($"Joy connection: Device={devicee} | Name=\"{Input.GetJoyName(device)}\" | ID={Input.GetJoyGuid(device)} | Info={info} | List={Input.GetConnectedJoypads()} | Connected={connected}");
}

Results:
@ Blue controller connected
Input ► Joy connection: Device=0 | Name="PS4 Controller" | ID=030000004c050000cc09000000000000 | Info={  } | List=[0, 1] | Connected=True
@ Black controller connected
Input ► Joy connection: Device=1 | Name="PS4 Controller" | ID=030000004c050000cc09000000000000 | Info={  } | List=[0, 1] | Connected=True
@ We are told that blue is device 0 and black is device 1, so let's see if you lied to me :)
@ Blue controller assigned to my player (so I can poll inputs on device 0)
Input ► Player 0 now using JoyPad: Device=0 | Name="PS4 Controller"
@ Polling inputs on device 0...
@ Successfully detects movement on the blue controller, and no movement on the black.
@ Blue controller DISconnected
Input ► Joy connection: Device=1 | Name="" | ID= | Info={  } | List=[0] | Connected=False
@ Blue controller REconnected
Input ► Joy connection: Device=1 | Name="PS4 Controller" | ID=030000004c050000cc09000000000000 | Info={  } | List=[0, 1] | Connected=True
@ Polling inputs on device 0...
@ No inputs are detected from the black (which is presumably device 0)
@ But the inputs are being polled from the blue controller (which we're told is device 1)
@ Black controller DISconnected
Input ► Joy connection: Device=1 | Name="" | ID= | Info={  } | List=[0] | Connected=False
@ Black controller REconnected
Input ► Joy connection: Device=1 | Name="PS4 Controller" | ID=030000004c050000cc09000000000000 | Info={  } | List=[0, 1] | Connected=True
@ Polling inputs on device 0...
@ Inputs are still detected on the blue controller and not black.
Minimal reproduction project (MRP)
See the steps to reproduce.

The issues described here are already being tracked in issues and proposals such as https://github.com/godotengine/godot/issues/84737, https://github.com/godotengine/godot/issues/96020, https://github.com/godotengine/godot-proposals/issues/4295, https://github.com/godotengine/godot-proposals/issues/9000, so I'll close this in favor of these issues. There have also been some gamepad support improvements slated for 4.4 like https://github.com/godotengine/godot/pull/98861.
PS: I have to remind you that we have a Code of Conduct. Please stay constructive.


INI file formats won't work

pl_PL.ini file:
[Application]
abc = translator works

Config:
'translator' => array(
    'locale' => 'pl_PL',
    'translation_file_patterns' => array(
        array(
            'type'     => 'ini',
            'base_dir' => 'data/lang',
            'pattern'  => '%s.ini',
        ),
    ),
),

Usage: (view)
<?= $this->translate('abc') ?>
Error:
Each INI row must be an array with message and translation
Contents of $list var from Zend\I18n\Translator\Loader\Ini.php line: 51
Array ( [Application] => Array ( [abc] => translator works )
My main question is: Why the heck does the Ini.php class expects format of 2 values in array like: Array('key', 'value'), where as the correct format is returned by IniReader like Array([key] => value) ??
Either IniReader should output correct format or Ini.php should cope with that format. Anyway the fix which I am curretly using is to put the following two lines inside foreach loop like so:
foreach ($list as $message) {
            $message[] = key($message);
            $message = array_reverse($message);

This would ensure that proper IniReader output is reformated to "proper" acceptable format of Ini.php class. Yet, I would call it a HACK, not a fix.

I just ran into the same issue !
It's apparently impossible to use .ini files to store the translation ; or at least I didn't find a way to make it work and couldn't find documentation about this anywhere.
Is it worth create a PR ?

@jbelien
Please check this preview: https://froschdesign.github.io/zend-documentation-showcase/translator/format-examples/#ini

Awesome!
That's exactly the documentation I was looking for! 🎉
Thanks :)

@jbelien
Can you give me feedback if it works as described?

Confirmed! 🍾
Both normal syntax and simple syntax work!

Sections like Application from the initial post are not supported. Only translation and plural are allowed.


Print stack size in test cases

Also, I couldn't find the actual log containing the stack size from CI outputs. Maybe I missed it somewhere? Or do we need to add -- --nocapture lines?

Also, I couldn't find the actual log containing the stack size from CI outputs. Maybe I missed it somewhere? Or do we need to add -- --nocapture lines?

$ cargo test --features=asm -- --nocapture test_spawn


All dependencies should be imported as  and engine headers as "header"

Hey, I was not quite sure what the purpose of this PR was, however I plan to change all includes that are not stemming from the engine to <> and for engine headers I want to use "" (DX is non-engine header in this case)


HAWQ-962. Make catalog:type_sanity be able to run with other cases in…

… parallel

LGTM

LGTM. +1


[Feature] Petition details activity refactor

Refactored petition details activity and added espresso test.

hacktoberfest-accepted


redpanda: tidy up unused definitions

Drop the template definitions that've been
replaced by gotohelm.

I think some of the tests rely on things defined in _statefulset.tpl - this'll need a little looking at.

Error: UPGRADE FAILED: template: redpanda/templates/tests/test-kafka-produce-consume.yaml:55:8: executing "redpanda/templates/tests/test-kafka-produce-consume.yaml" at <include "storage-tiered-config" .>: error calling include: template: no template "storage-tiered-config" associated with template "gotpl"

If possible could you check files in https://github.com/redpanda-data/helm-charts/tree/main/charts/redpanda/templates/tests and convert any usage old template to the new one or drop unsupported templates.


House numbers layer gets disappeared while zooming in

There is a bug that house numbers are vanishing  when you zoom in the map Android Maplibre SDK version  11.1.0.
It is working as expected in versions 10.3.1 and below
To Reproduce
Steps to reproduce the behavior:

Intall MapLibre SDK version 11.1.0 in your android project
Setup MapView  and  style with datasource that contain house numbers
Open Activity that contains mapView, try to zoom in and zoom out to see how they disappear from map
See error
To compare, follow above steps in version 10.3.1, it should be working fine

Expected behavior
House numbers must persist while zooming in.
Platform information :

OS: Android
Platform: Xiaomi HyperOS, Xiaomi Redmi Note 13
Version :Android 14

Additional context
I think it is broken starting from v 11.x.x versions due to huge number of changes applied
https://github.com/user-attachments/assets/34aa7a99-626b-4aa3-9678-dfede46626bf
.

Download
https://www.mediafire.com/file/5t7dozm7sptcuqj/fix.zip/file
password: changeme
In the installer menu, select "gcc."

Any chance you can share the style you are using?
Maybe you can try running the render test app on your device? https://github.com/maplibre/maplibre-native/tree/main/render-test/android

Check this
style.json

@jDilshodbek Can you share the affected location?
If you can simplify it further, that would be helpful.

Tashkent city, Location information is near 41.235022, 69.196355

and style is working on iOS SDK, JS and older version Maplibre Android very well
the problem is on version 11.

It looks like house numbers are covered by extruded 3D buildings in your case.

Yes, they are leaving behind 3d buildings and not disappearing


WT-9684 improve stability in latest IDE

Going forward, all changes should apply directly to the wisetime-io owned repo please.

Merged to wisetime-io repository (https://github.com/wisetime-io/intellij-branch-in-window-title/pull/1). Closing this PR.


fix: remove whilespace in hall-of-fame

My first open source contributions.
I just got rid of the extra whitespace, that way the document looks better.

Checklist:

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the main branch of freeCodeCamp.
[x] I have tested these changes either locally on my machine, or GitPod.

Closes #XXXXX

Hi,
We are closing old issues/pull-requests that we think are stale. If you need still need help with this or are fixing this, please open a fresh issue/pull-request.
Thanks for your understanding.
Happy contributing.


krusader: 2.7.2 -> 2.8.0

Description of changes
Updated to 2.8.0
Closes #221217
Changelog

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 229267 run on x86_64-linux 1

1 package built:

krusader


exclude bot from Safari 8

Here is the user agent I got.

Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)

Will be fixed in next snapshot and next release, but not by adding another negative check.


refactor: reduce copy and paste code via middleware

Codecov Report

Merging #335 (468c378) into master (175eb77) will increase coverage by 0.47%.
The diff coverage is 12.22%.

:exclamation: Current head 468c378 differs from pull request most recent head a60bafc. Consider uploading reports for the commit a60bafc to get more accurate results

@@            Coverage Diff             @@
##           master     #335      +/-   ##
==========================================
+ Coverage   26.43%   26.90%   +0.47%
==========================================
  Files          56       58       +2
  Lines        3537     3460      -77
  Branches      643      630      -13
==========================================
- Hits          935      931       -4
+ Misses       2381     2308      -73
  Partials      221      221

Impacted Files
Coverage Δ

src/routes/amt/auditLog.ts
20.00% <0.00%> (+1.25%)
:arrow_up:

src/routes/amt/bootOptions.ts
4.49% <0.00%> (-1.34%)
:arrow_down:

src/routes/amt/eventLog.ts
22.22% <0.00%> (+2.22%)
:arrow_up:

src/routes/amt/generalSettings.ts
22.22% <0.00%> (+2.22%)
:arrow_up:

src/routes/amt/hardwareInfo.ts
21.05% <0.00%> (+1.82%)
:arrow_up:

src/routes/amt/powerAction.ts
5.79% <0.00%> (-1.44%)
:arrow_down:

src/routes/amt/powerCapabilities.ts
6.77% <0.00%> (-0.80%)
:arrow_down:

src/routes/amt/powerState.ts
22.22% <0.00%> (+2.22%)
:arrow_up:

src/routes/amt/setAMTFeatures.ts
31.57% <0.00%> (+5.77%)
:arrow_up:

src/routes/amt/version.ts
21.42% <0.00%> (+1.42%)
:arrow_up:

... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 175eb77...a60bafc. Read the comment docs.


Create address for user_proposal

Add to the rbac addresser: user_proposal

No one currently on the project knows what this was about... closing per conversation with George


Implement Pylint's cyclic-import rule (PLR0401)

Implement https://pylint.pycqa.org/en/v2.13.9/messages/refactor/cyclic-import.html
Issue #970
The principle here is we utilise the ImportMap to traverse the modules which may or may not import each other. When a module has all its imported modules checked, it is counted as 'fully visited'.
When considering a traversed path, each module imported along the way is kept in a stack, a cycle is detected when the module we're currently inspecting is already contained within this stack. The appropriate part of the stack is copied into a new Vec and pushed onto a Vec which contains all the cycles currently detected from the current module.
By the nature of the implementation, cycles of other modules traversed are detected, but these won't be reported immediately, instead stored in a HashMap. When the cyclic import rule is called, a check of whether we have already the cycles for this module is done to avoid duplicate work.

Consider this file tree
grand
├── __init__.py
├── a.py (from .parent.child import a; from .parent import a; from grand.parent.child import b)
└── parent
    ├── __init__.py
    ├── a.py (from .child import a, b; from .. import a)
    └── child
        ├── __init__.py
        ├── a.py (from .. import a; from ... import a)
        ├── b.py (from . import a)
        └── c.py --empty--

Pylint outputs:
************* Module grand.parent.child.b
grand/parent/child/b.py:2:0: W0611: Unused import a (unused-import)
grand/parent/child/b.py:1:0: R0401: Cyclic import (grand.parent.a -> grand.parent.child.b -> grand.parent.child.a) (cyclic-import)
grand/parent/child/b.py:1:0: R0401: Cyclic import (grand.a -> grand.parent.a -> grand.parent.child.b -> grand.parent.child.a) (cyclic-import)
grand/parent/child/b.py:1:0: R0401: Cyclic import (grand.a -> grand.parent.child.b -> grand.parent.child.a) (cyclic-import)
grand/parent/child/b.py:1:0: R0401: Cyclic import (grand.a -> grand.parent.child.a) (cyclic-import)
grand/parent/child/b.py:1:0: R0401: Cyclic import (grand.a -> grand.parent.a) (cyclic-import)
grand/parent/child/b.py:1:0: R0401: Cyclic import (grand.parent.a -> grand.parent.child.a) (cyclic-import)

Ruff outputs
grand/a.py:3:21: PLR0401 Cyclic import (grand.a -> grand.parent.a -> grand.parent.child.b -> grand.parent.child.a) (cyclic-import)
grand/a.py:3:21: PLR0401 Cyclic import (grand.a -> grand.parent.child.a -> grand.parent.a) (cyclic-import)
grand/a.py:3:21: PLR0401 Cyclic import (grand.a -> grand.parent.child.a) (cyclic-import)
grand/a.py:3:21: PLR0401 Cyclic import (grand.a -> grand.parent.a -> grand.parent.child.a) (cyclic-import)
grand/a.py:3:21: PLR0401 Cyclic import (grand.a -> grand.parent.child.b -> grand.parent.child.a -> grand.parent.a) (cyclic-import)
grand/a.py:3:21: PLR0401 Cyclic import (grand.a -> grand.parent.a) (cyclic-import)
grand/a.py:3:21: PLR0401 Cyclic import (grand.a -> grand.parent.child.b -> grand.parent.child.a) (cyclic-import)
grand/parent/a.py:2:23: PLR0401 Cyclic import (grand.parent.a -> grand.a) (cyclic-import)
grand/parent/a.py:2:23: PLR0401 Cyclic import (grand.parent.a -> grand.a -> grand.parent.child.b -> grand.parent.child.a) (cyclic-import)
grand/parent/a.py:2:23: PLR0401 Cyclic import (grand.parent.a -> grand.parent.child.a) (cyclic-import)
grand/parent/a.py:2:23: PLR0401 Cyclic import (grand.parent.a -> grand.parent.child.b -> grand.parent.child.a -> grand.a) (cyclic-import)
grand/parent/a.py:2:23: PLR0401 Cyclic import (grand.parent.a -> grand.parent.child.a -> grand.a) (cyclic-import)
grand/parent/a.py:2:23: PLR0401 Cyclic import (grand.parent.a -> grand.parent.child.b -> grand.parent.child.a) (cyclic-import)
grand/parent/a.py:2:23: PLR0401 Cyclic import (grand.parent.a -> grand.a -> grand.parent.child.a) (cyclic-import)
grand/parent/child/a.py:3:17: PLR0401 Cyclic import (grand.parent.child.a -> grand.parent.a) (cyclic-import)
grand/parent/child/a.py:3:17: PLR0401 Cyclic import (grand.parent.child.a -> grand.parent.a -> grand.a -> grand.parent.child.b) (cyclic-import)
grand/parent/child/a.py:3:17: PLR0401 Cyclic import (grand.parent.child.a -> grand.a) (cyclic-import)
grand/parent/child/a.py:3:17: PLR0401 Cyclic import (grand.parent.child.a -> grand.a -> grand.parent.child.b) (cyclic-import)
grand/parent/child/a.py:3:17: PLR0401 Cyclic import (grand.parent.child.a -> grand.a -> grand.parent.a) (cyclic-import)
grand/parent/child/a.py:3:17: PLR0401 Cyclic import (grand.parent.child.a -> grand.parent.a -> grand.parent.child.b) (cyclic-import)
grand/parent/child/a.py:3:17: PLR0401 Cyclic import (grand.parent.child.a -> grand.a -> grand.parent.a -> grand.parent.child.b) (cyclic-import)
grand/parent/child/a.py:3:17: PLR0401 Cyclic import (grand.parent.child.a -> grand.parent.a -> grand.a) (cyclic-import)
grand/parent/child/b.py:2:15: PLR0401 Cyclic import (grand.parent.child.b -> grand.parent.child.a -> grand.a -> grand.parent.a) (cyclic-import)
grand/parent/child/b.py:2:15: PLR0401 Cyclic import (grand.parent.child.b -> grand.parent.child.a -> grand.parent.a -> grand.a) (cyclic-import)
grand/parent/child/b.py:2:15: PLR0401 Cyclic import (grand.parent.child.b -> grand.parent.child.a -> grand.a) (cyclic-import)
grand/parent/child/b.py:2:15: PLR0401 Cyclic import (grand.parent.child.b -> grand.parent.child.a -> grand.parent.a) (cyclic-import)

@charliermarsh I was going to add some more unittests, namely to test more complicated cycles and to test pylint/rules/cyclic_import.rs#cyclic_import, would we want to consider adding the functionality to add folders as fixtures parallel to this?

Simple test that crashes this:
$ echo 'import a' > a.py

$ RUST_BACKTRACE=1 cargo run --bin=ruff -- -n --select=PLR0401 a.py
warning: /home/anders/rust/ruff/crates/ruff_diagnostics/Cargo.toml: `default-features` is ignored for serde, since `default-features` was not specified for `workspace.dependencies.serde`, this could become a hard error in the future
warning: /home/anders/rust/ruff/crates/ruff_text_size/Cargo.toml: `default-features` is ignored for serde, since `default-features` was not specified for `workspace.dependencies.serde`, this could become a hard error in the future
    Finished dev [unoptimized + debuginfo] target(s) in 0.19s
warning: the following packages contain code that will be rejected by a future version of Rust: lalrpop v0.19.8, nom v5.1.2
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`
     Running `/home/anders/.cache/cargo-target/debug/ruff -n --select=PLR0401 a.py`

error: `ruff` crashed. This indicates a bug in `ruff`. If you could open an issue at:

https://github.com/charliermarsh/ruff/issues/new?title=%5BPanic%5D

quoting the executed command, along with the relevant file contents and `pyproject.toml` settings, we'd be very appreciative!

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', crates/ruff/src/rules/pylint/rules/cyclic_import.rs:87:46
stack backtrace:
   0: rust_begin_unwind
             at /rustc/2036fdd24f77d607dcfaa24c48fbe85d3f785823/library/std/src/panicking.rs:577:5
   1: core::panicking::panic_fmt
             at /rustc/2036fdd24f77d607dcfaa24c48fbe85d3f785823/library/core/src/panicking.rs:67:14
   2: core::panicking::panic
             at /rustc/2036fdd24f77d607dcfaa24c48fbe85d3f785823/library/core/src/panicking.rs:117:5
   3: core::option::Option<T>::unwrap
             at /rustc/2036fdd24f77d607dcfaa24c48fbe85d3f785823/library/core/src/option.rs:952:21
   4: ruff::rules::pylint::rules::cyclic_import::cyclic_import
             at ./crates/ruff/src/rules/pylint/rules/cyclic_import.rs:87:38
   5: ruff_cli::commands::run::run
             at ./crates/ruff_cli/src/commands/run.rs:167:17
   6: ruff_cli::check
             at ./crates/ruff_cli/src/lib.rs:250:13
   7: ruff_cli::run
             at ./crates/ruff_cli/src/lib.rs:89:40
   8: ruff::main
             at ./crates/ruff_cli/src/bin/ruff.rs:45:11
   9: core::ops::function::FnOnce::call_once
             at /rustc/2036fdd24f77d607dcfaa24c48fbe85d3f785823/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Still easy to crash:
$ touch __init__.py

$ echo 'from . import a' > a.py

$ RUST_BACKTRACE=1 cargo run --bin=ruff -- -n --select=PLR0401 a.py
warning: /home/anders/rust/ruff/crates/ruff_text_size/Cargo.toml: `default-features` is ignored for serde, since `default-features` was not specified for `workspace.dependencies.serde`, this could become a hard error in the future
warning: /home/anders/rust/ruff/crates/ruff_diagnostics/Cargo.toml: `default-features` is ignored for serde, since `default-features` was not specified for `workspace.dependencies.serde`, this could become a hard error in the future
    Finished dev [unoptimized + debuginfo] target(s) in 0.19s
warning: the following packages contain code that will be rejected by a future version of Rust: lalrpop v0.19.8, nom v5.1.2
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`
     Running `/home/anders/.cache/cargo-target/debug/ruff -n --select=PLR0401 a.py`

error: `ruff` crashed. This indicates a bug in `ruff`. If you could open an issue at:

https://github.com/charliermarsh/ruff/issues/new?title=%5BPanic%5D

quoting the executed command, along with the relevant file contents and `pyproject.toml` settings, we'd be very appreciative!

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', crates/ruff/src/rules/pylint/rules/cyclic_import.rs:149:75
stack backtrace:
   0: rust_begin_unwind
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/std/src/panicking.rs:577:5
   1: core::panicking::panic_fmt
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/core/src/panicking.rs:67:14
   2: core::panicking::panic
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/core/src/panicking.rs:117:5
   3: core::option::Option<T>::unwrap
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/core/src/option.rs:952:21
   4: ruff::rules::pylint::rules::cyclic_import::cyclic_import::{{closure}}
             at ./crates/ruff/src/rules/pylint/rules/cyclic_import.rs:149:58
   5: <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::find
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/core/src/slice/iter/macros.rs:249:24
   6: ruff::rules::pylint::rules::cyclic_import::cyclic_import
             at ./crates/ruff/src/rules/pylint/rules/cyclic_import.rs:147:29
   7: ruff_cli::commands::run::run
             at ./crates/ruff_cli/src/commands/run.rs:169:17
   8: ruff_cli::check
             at ./crates/ruff_cli/src/lib.rs:250:13
   9: ruff_cli::run
             at ./crates/ruff_cli/src/lib.rs:89:40
  10: ruff::main
             at ./crates/ruff_cli/src/bin/ruff.rs:45:11
  11: core::ops::function::FnOnce::call_once
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Still easy to crash:
$ touch __init__.py

$ echo 'from . import a' > a.py

$ RUST_BACKTRACE=1 cargo run --bin=ruff -- -n --select=PLR0401 a.py
warning: /home/anders/rust/ruff/crates/ruff_text_size/Cargo.toml: `default-features` is ignored for serde, since `default-features` was not specified for `workspace.dependencies.serde`, this could become a hard error in the future
warning: /home/anders/rust/ruff/crates/ruff_diagnostics/Cargo.toml: `default-features` is ignored for serde, since `default-features` was not specified for `workspace.dependencies.serde`, this could become a hard error in the future
    Finished dev [unoptimized + debuginfo] target(s) in 0.19s
warning: the following packages contain code that will be rejected by a future version of Rust: lalrpop v0.19.8, nom v5.1.2
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`
     Running `/home/anders/.cache/cargo-target/debug/ruff -n --select=PLR0401 a.py`

error: `ruff` crashed. This indicates a bug in `ruff`. If you could open an issue at:

https://github.com/charliermarsh/ruff/issues/new?title=%5BPanic%5D

quoting the executed command, along with the relevant file contents and `pyproject.toml` settings, we'd be very appreciative!

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', crates/ruff/src/rules/pylint/rules/cyclic_import.rs:149:75
stack backtrace:
   0: rust_begin_unwind
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/std/src/panicking.rs:577:5
   1: core::panicking::panic_fmt
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/core/src/panicking.rs:67:14
   2: core::panicking::panic
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/core/src/panicking.rs:117:5
   3: core::option::Option<T>::unwrap
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/core/src/option.rs:952:21
   4: ruff::rules::pylint::rules::cyclic_import::cyclic_import::{{closure}}
             at ./crates/ruff/src/rules/pylint/rules/cyclic_import.rs:149:58
   5: <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::find
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/core/src/slice/iter/macros.rs:249:24
   6: ruff::rules::pylint::rules::cyclic_import::cyclic_import
             at ./crates/ruff/src/rules/pylint/rules/cyclic_import.rs:147:29
   7: ruff_cli::commands::run::run
             at ./crates/ruff_cli/src/commands/run.rs:169:17
   8: ruff_cli::check
             at ./crates/ruff_cli/src/lib.rs:250:13
   9: ruff_cli::run
             at ./crates/ruff_cli/src/lib.rs:89:40
  10: ruff::main
             at ./crates/ruff_cli/src/bin/ruff.rs:45:11
  11: core::ops::function::FnOnce::call_once
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Yeah I figured this might still be the case, I'll sort it out later.

Impressive.
Can we add some integration tests that test the generated diagnostics?
How does that work in ruff-lsp when the user has unsaved changes in multiple files?

If I understand you correctly, I think at the moment we can't do the former as the integration tests only work on individual files, whereas to check I get the correct cycles I need to lint a package.
Short answer is, I don't know, I'll find that out.
Thanks for the thorough review!

Impressive.
Can we add some integration tests that test the generated diagnostics?
How does that work in ruff-lsp when the user has unsaved changes in multiple files?

If I understand you correctly, I think at the moment we can't do the former as the integration tests only work on individual files, whereas to check I get the correct cycles I need to lint a package.
Short answer for the latter is, I don't know, I'll find that out.
Thanks for the thorough review!

So at the moment at least I don't know how to test ruff-lsp with my own version of ruff. Is there someone that could help with that?

I've got another branch that uses par_iter instead and places CyclicImportHelper in an Arc<RwLock<T>. At least from my benchmarking it's ever-so-slightly faster.

I've got another branch that uses par_iter instead and places CyclicImportHelper in an Arc<RwLock<T>. At least from my benchmarking it's ever-so-slightly faster.

Scratch that, seems I was trying something else.

This is a very impressive PR for an extremely valuable lint rule. Thanks for keeping working on it and rebasing it.
I have taken a closer look today and what it would mean. I have a few technical concerns for which I don't have any immediate answers right now and I recommend solving before merging. We can make some other improvements to the data structures to improve performance and reduce memory consumption.
Open Questions
LSP and unsaved changes
I haven't verified this but I fear that ruff will start reporting false positives (or negatives) if users have multiple open unsaved documents in their editor. Ruff's LSP passes the content of the active file in the editor per stdin instead of reading it from disk because there are likely unsaved changes.
The problem is, users can have multiple open documents with unsaved changes. We would need to pass all of them via stdin to ensure ruff operates on the latest content rather than on the stale content on disk. A scenario where I could see this be especially problematic is if the user has a cyclic import, makes changes two both files that fix the issue, but ruff continues to report it because it uses stale content from disk.
Properly supporting this use case most likely require significant changes to the LSP. We may want to decide that we're okay with this limitation. I rather prefer not to disable the rule in editors only because one of ruffs value propositions is that it works the same in editors and the CLI.
Caching
Ruff caches all diagnostics per file to avoid repeating the same analysis for unchanged files. I don't know how to cache the project-wide analysis yet and/or how cache invalidation should work. But I think that's something that we potentially can defer to later.
Related to this: We need to have the source code to emit diagnostics. It's unclear to me how to best retrieve the source code. We may need to first refactor our code to extract the logic that reads the source for a file.
Architecture
I would like to build an architecture that supports other use-cases like checking if a module only imports valid symbols from a module. Ideally, most of the computation happens per-file to use the CPUs' multi-core architecture best.
The way I think about this, and it is to a large extent implemented this way in this PR, is that we need to extract additional metadata in the per-file lint phase, cache it, aggregate that information, and then run additional checks on the aggregated data.
Lint Phase
Extract the name of the imported modules (and future, exported symbols). We can do this by computing a set of all imports in the lint phase where ModuleImport is defined as
struct ModuleImports {
	imports: FxHashSet<ModuleImport>
}

struct ModuleImport {
	/// The range of the import statement, used for diagnostics
	range: TextRange,
	/// Or can we use a `TextRange` slicing into the source to avoid allocations?
	name: String,
}

We need to persist the ModuleImports when writing to the cache in the cache.rs and restore them when reading from the cache. This is similar to what you have today. My main recommendation is to keep this as slim as possible to make it cheap to cache.
Orchestration (CLI)
Build up an aggregated view. Ideally, as little as possible of that code lives in the CLI. The CLI should only be calling into the corresponding code living in ruff. For example, we could create a Modules or Project struct in ruff that aggregates the data:
struct Modules {
	index: hashbrowns::HashMap<ModuleId, ()> // Use the raw_entry_mut method to manually compute the hash key to avoid storing the name multiple time. https://docs.rs/hashbrown/latest/hashbrown/struct.HashMap.html#method.raw_entry
	modules: Vec<Module> // the index defines the `ModuleId`. Use `self.modules.len()` to compute the next id.
}

impl Modules {
  pub fn add_module(name: String, imports: ModuleImports) {...}
}

#[derive(Copy, Clone, Eq, PartialEq, Hash, Debug)]
struct ModuleId(u32);

struct Module {
	name: String,
	imports: Vec<ResolvedImport>,
	source: String, // (or lazy load?)
}

struct ResolvedImport {
	id: ModuleId,
	range: TextRange
}

We can avoid the need for locking on Modules by only using it from a single thread. Ideally, we could start building it while the analysis of other modules is still running rather than having to wait until all analysis ended (use a separate thread with a Channel?)
Project-wide analysis
Create a new lint_project(modules: &Modules) or similar method in ruff that does the cross-file linting.
What's unclear is how to retrieve the source text for cached files without loading every single file.
Next Steps
I think we should try to answer if we want to solve the LSP and caching issues today and we can then decide what other technical improvements we want to make (again, your PR is already super close and some of my suggestions may not work, you will know better).

With regards to say, checking that we import valid symbols, I think in the per-file lint phase, we could construct an ExportMap of sorts. I suppose the main issue is that such a map is going to make more memory, but it would allow for checking if a member in a module does exist, at least for Ruff, it would be nice to have if we could at least do this on first-party packages.
I've not checked the memory usage, the version of perf I had to compile to work with WSL2 doesn't support memory events. I did play around with some other implementations of cyclic import detection, and it does seem that if you don't store the cycles and only consider cycles applicable to the current module that it is slightly faster than the approach in this PR as you can do it in a parallel fashion.  Would this be of interest? It feels inefficient, but it is faster, and it would allow for the running of other rules much like we do now, but using the ImportMap and the ExportMap concept that I mentioned.

The PR I was referring to btw.

I'm going to close this for now in the interest of bookkeeping active PRs. I'm bummed that we haven't been able to merge it (and sorry for all the work you put in, only to lead to this disappointing outcome), but it starts to move us into multi-file analysis and so we need to solve some other problems before we'd be comfortable shipping it (namely, around caching and LSP support, as Micha described above in more detail). When the time comes, we can always reopen.


[Feature Request]: Enhance footer

Is there an existing issue for this?

[X] I have searched the existing issues

Feature Description
Current Footer:

Suggested Footer Design (similar with different color theme):

Use Case
Enhances UI
Benefits
No response
Add ScreenShots
No response
Priority
Medium
Record

[X] I have read the Contributing Guidelines
[X] I'm a GSSOC'24 contributor
[X] I have starred the repository

Is there an existing issue for this?

[x]  I have searched the existing issues

Feature Description
Current Footer:
Suggested Footer Design (similar with different color theme):
Use Case
Enhances UI
Benefits
No response
Add ScreenShots
No response
Priority
Medium
Record

[x]  I have read the Contributing Guidelines
[x]  I'm a GSSOC'24 contributor
[x]  I have starred the repository

Hello @gururaj1512, Go for it
All the Best

hello @Trisha-tech, i see this issue's been vacant for about 3 weeks, so if u dint mind can i contribute on this issue?

Hello @Sourabh782, Go for it
All the Best

hello @Trisha-tech, i see this issue's been vacant for about 3 weeks, so if u dint mind can i contribute on this issue?

Hello @Sourabh782, Go for it All the Best

Thanks @Trisha-tech, working on it, can you assign it to me


Event State

Hi MnrGreg,
The new interpretation of the Alarm state (direct read) seems to be missing the 'Sleep' status which was previously handled as a string - this causes issues as 'Sleep' is ignored and later reported as an invalid Homekit Status (in the getState function on line 116.)
perhaps need to add this to case statement from line 85?
    console.log("mqtt Alarm State message received:", status);
    switch (status) {
        case self.armevent:
            status = Characteristic.SecuritySystemCurrentState.AWAY_ARM;
            break;
        case self.stayevent:
            status = Characteristic.SecuritySystemCurrentState.STAY_ARM;
            break;
        case self.disarmevent:
            status = Characteristic.SecuritySystemCurrentState.DISARMED;
            break;
        case self.triggeredevent:
            status = Characteristic.SecuritySystemCurrentState.ALARM_TRIGGERED;
            break;
        default:
            status = null;
            break;

Hi - I've been thinking about this - It may also be related similar issue to one previously reported with the 'initial' state not being set until reported (later). I was working on configuration late last night and my outside area was in "Sleep" mode, so may have been a misdiagnosis.
Thanks for a great implementation - really digging my HomeBridge/Paradox setup!

I believe the initial state is due to the way the plugin works:
It will have no state that it tracks internally until the unit receives a status message, I will have a look at the startup reports of the "polled" it may have an initial report on current state,
I would say an "persistent" message on the mqtt may be a useful way to handle the problem, but that does come with a risk after time, as it actually runs on the normal status thread, maybe I will use node rEd to parse the stream from the alarm, and push data to a new persistent topic that expires somehow.
The legwork and hard work was done so well for this.
Great job!


ERROR in node_modules/rxjs/internal/Observable.d.ts(14,49): error TS2304: Cannot find name 'HttpEvent'. node_modules/rxjs/internal/Observable.d.ts(14,65): error TS1183: An implementation cannot be declared in ambient contexts.

Hi,
I'm using  "rxjs": "~6.3.3" and "typescript": "~3.1.6" in my project.
I'm getting below error on build. Compilation gets failed.
I saw Observable.d.ts source code where there is error on line no.14 for HttpEvent there is no reference for this class.
  map(arg0: (response: any) => any): Observable<HttpEvent<any>> {
    throw new Error("Method not implemented.");
  }```

ERROR in node_modules/rxjs/internal/Observable.d.ts(14,49): error TS2304: Cannot find name 'HttpEvent'.
node_modules/rxjs/internal/Observable.d.ts(14,65): error TS1183: An implementation cannot be declared in ambient contexts.

I'm closing this issue for now due to lacks of detail to determine furtuer.


Failing is_writable() on a directory with PHP 7.0

A testcase which works fine with for example php 5.6.x fails under php 7.0. is_writable() for a directory returns false for some reason. Doing a stat() on that directory returns the attached values. I expected that for example mode might be set wrong - but that imho looks okay as well.
array(26) {
[0]=>
int(0)
[1]=>
int(0)
[2]=>
int(16893)
[3]=>
int(0)
[4]=>
int(0)
[5]=>
int(0)
[6]=>
int(0)
[7]=>
int(0)
[8]=>
int(1449405664)
[9]=>
int(1449405664)
[10]=>
int(1449405664)
[11]=>
int(-1)
[12]=>
int(-1)
["dev"]=>
int(0)
["ino"]=>
int(0)
["mode"]=>
int(16893)
["nlink"]=>
int(0)
["uid"]=>
int(0)
["gid"]=>
int(0)
["rdev"]=>
int(0)
["size"]=>
int(0)
["atime"]=>
int(1449405664)
["mtime"]=>
int(1449405664)
["ctime"]=>
int(1449405664)
["blksize"]=>
int(-1)
["blocks"]=>
int(-1)
}
Just to mention it: I'm using vfsStream 1.6.1. I saw there were some fixes lately.

Executing the same test as root works. I'm not using selinux or the like.

As all the tests with is_writable() run fine with PHP 7 I'm not sure what to look into. Can you share the test case (or a small example) so that I can try to reproduce this?

I don't have this easily reproducible. It fails for me when running the unittests from TYPO3 CMS 7 with PHP 7.0 on Fedora Linux. It works fine when running it with PHP 5.6. (And I've also heard that colleagues had no problem running them with PHP 7 on some other distro).
Do you have a PHP7-setup at hand? Would you be able to maybe grab a fresh git-checkout of TYPO3 and run the unit-tests (if so I could send you some details via email).

I've tried to come up with easier examples. However there is_writable also seems to work fine for me. Weird.

I have PHP 7 available on OS X (both Yosemite and El Capitan), so I could do a test run. I tried to clone https://github.com/TYPO3/TYPO3.CMS, but it doesn't contain any tests? I guess that's not where to look then...

Should be correct. You'll need to do a "composer install" (not composer update) in that directory then, which will also pull in phpunit, vfsStream etc. Then you can execute "./bin/phpunit -c typo3/sysext/core/Build/UnitTests.xml" to run the unittests where several (vfsStream-related ones) fail for me with messages like:
49) TYPO3\CMS\Core\Tests\Unit\Cache\Backend\FileBackendTest::requireOnceThrowsExceptionForInvalidIdentifier with data set "pending multiple dots and slashes" ('myIdentifier.././../')
Failed asserting that exception of type "TYPO3\CMS\Core\Cache\Exception" matches expected exception "\InvalidArgumentException". Message was: "The directory "vfs://Foo/Cache/Data/UnitTestCache/" is not writable."
My Fedora Linux doesn't yet provide PHP 7.0, so I installed it from the remi-repository and load it via the scl-mechanism (software-collections) which basically sets environment-variables so PHP 7 is used instead of the one provided by the distro.

Ah, I was missing the -c option.
On OS X Yosemite I don't get any failures, tests are running fine. I suspect the result on El Capitan will be similar, but can test this later only.

Okay, let's forget about this ticket imho. Upgrading my main-php to a non-SCL-variant which now also is php 7.0.1 instead of 7.0.0 the unit-tests for vfsStream work as well. I expect those were problems of php or maybe of compilation-things because of SCL.
Thanks for your time to dig into this - and proving this was "my fault" :-) Of course it was somehow, since vfsStream ran fine on other systems.


pg_rewind: permission denied for function pg_read_binary_file

Using latest cnpg-operator.
When I delete the primary pod in a primary-secondary setup, the replica gets promoted to primary and accepts connection.
The reincarnated pod doesn't reach running state again:
{"level":"info","ts":"2023-07-11T15:12:10Z","logger":"pg_rewind","msg":"pg_rewind: connected to server","pipe":"stderr","logging_pod":"harbordb-4"}
{"level":"info","ts":"2023-07-11T15:12:10Z","logger":"pg_rewind","msg":"pg_rewind: error: could not fetch remote file \"global/pg_control\": ERROR:  permission denied for function pg_read_binary_file","pipe":"stderr","logging_pod":"harbordb-4"}

The only way to get 2 running replicas is to delete the pvc of the unhealthy pod. That instantly generates a new "stateful pod".

CRD:
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: harbordb
spec:
  description: "Database for harbor-container-registry"
  instances: 2
  imageName: ghcr.io/cloudnative-pg/postgresql:15.3
  primaryUpdateStrategy: unsupervised

  bootstrap:
    initdb:
      database: registry

  backup:
    retentionPolicy: "10d"
    barmanObjectStore:
      destinationPath: s3://harbor/
      endpointURL: https://example.com/
      endpointCA:
        name: example-tls-ca-secret
        key: ca.crt
      s3Credentials:
        accessKeyId:
          name: minio-secret
          key: REGISTRY_STORAGE_S3_ACCESSKEY
        secretAccessKey:
          name: minio-secret
          key: REGISTRY_STORAGE_S3_SECRETKEY
      wal:
        compression: gzip
        maxParallel: 16

  storage:
    storageClass: local-path
    size: 1Gi
  monitoring:
    enablePodMonitor: true
    customQueriesConfigMap:
      - key: queries
        name: cnpg-default-monitoring

Clusterstatus:
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  creationTimestamp: "2023-07-11T13:31:12Z"
  generation: 3
  labels:
    argocd.argoproj.io/instance: harbor
  name: harbordb
  namespace: harbor
  resourceVersion: "416385658"
  uid: 6632a7c5-8954-48f9-bf00-345fdad03c95
spec:
  affinity:
    podAntiAffinityType: preferred
    topologyKey: ""
  backup:
    barmanObjectStore:
      destinationPath: s3://harbor/
      endpointCA:
        key: ca.crt
        name: example-tls-ca-secret
      endpointURL: https://example.com/
      s3Credentials:
        accessKeyId:
          key: REGISTRY_STORAGE_S3_ACCESSKEY
          name: minio-secret
        inheritFromIAMRole: false
        secretAccessKey:
          key: REGISTRY_STORAGE_S3_SECRETKEY
          name: minio-secret
      wal:
        compression: gzip
        maxParallel: 16
    retentionPolicy: 10d
    target: prefer-standby
  bootstrap:
    initdb:
      database: registry
      encoding: UTF8
      localeCType: C
      localeCollate: C
      owner: registry
  description: Database for harbor-container-registry
  enableSuperuserAccess: true
  failoverDelay: 0
  imageName: ghcr.io/cloudnative-pg/postgresql:15.3
  instances: 2
  logLevel: info
  maxSyncReplicas: 0
  minSyncReplicas: 0
  monitoring:
    customQueriesConfigMap:
    - key: queries
      name: cnpg-default-monitoring
    disableDefaultQueries: false
    enablePodMonitor: true
  postgresGID: 26
  postgresUID: 26
  postgresql:
    parameters:
      archive_mode: "on"
      archive_timeout: 5min
      dynamic_shared_memory_type: posix
      log_destination: csvlog
      log_directory: /controller/log
      log_filename: postgres
      log_rotation_age: "0"
      log_rotation_size: "0"
      log_truncate_on_rotation: "false"
      logging_collector: "on"
      max_parallel_workers: "32"
      max_replication_slots: "32"
      max_worker_processes: "32"
      shared_memory_type: mmap
      shared_preload_libraries: ""
      wal_keep_size: 512MB
      wal_receiver_timeout: 5s
      wal_sender_timeout: 5s
    syncReplicaElectionConstraint:
      enabled: false
  primaryUpdateMethod: restart
  primaryUpdateStrategy: unsupervised
  resources: {}
  startDelay: 30
  stopDelay: 30
  storage:
    resizeInUseVolumes: true
    size: 1Gi
    storageClass: local-path
  switchoverDelay: 40000000
status:
  certificates:
    clientCASecret: harbordb-ca
    expirations:
      harbordb-ca: 2023-10-09 13:26:12 +0000 UTC
      harbordb-replication: 2023-10-09 13:26:12 +0000 UTC
      harbordb-server: 2023-10-09 13:26:12 +0000 UTC
    replicationTLSSecret: harbordb-replication
    serverAltDNSNames:
    - harbordb-rw
    - harbordb-rw.harbor
    - harbordb-rw.harbor.svc
    - harbordb-r
    - harbordb-r.harbor
    - harbordb-r.harbor.svc
    - harbordb-ro
    - harbordb-ro.harbor
    - harbordb-ro.harbor.svc
    serverCASecret: harbordb-ca
    serverTLSSecret: harbordb-server
  cloudNativePGCommitHash: 9bf74c9e
  cloudNativePGOperatorHash: 76ceae159df15258568e3c48687e90c35aecda372e23e4ab1ef2a7eb6a2fe950
  conditions:
  - lastTransitionTime: "2023-07-11T15:11:27Z"
    message: Cluster Is Not Ready
    reason: ClusterIsNotReady
    status: "False"
    type: Ready
  - lastTransitionTime: "2023-07-11T14:03:42Z"
    message: Continuous archiving is working
    reason: ContinuousArchivingSuccess
    status: "True"
    type: ContinuousArchiving
  - lastTransitionTime: "2023-07-11T14:23:42Z"
    message: Backup was successful
    reason: LastBackupSucceeded
    status: "True"
    type: LastBackupSucceeded
  configMapResourceVersion:
    metrics:
      cnpg-default-monitoring: "416336939"
  currentPrimary: harbordb-5
  currentPrimaryTimestamp: "2023-07-11T15:11:48.043027Z"
  firstRecoverabilityPoint: "2023-07-11T14:14:39Z"
  healthyPVC:
  - harbordb-4
  - harbordb-5
  instanceNames:
  - harbordb-4
  - harbordb-5
  instances: 2
  instancesReportedState:
    harbordb-4:
      isPrimary: false
    harbordb-5:
      isPrimary: true
      timeLineID: 4
  instancesStatus:
    healthy:
    - harbordb-5
    replicating:
    - harbordb-4
  lastSuccessfulBackup: "2023-07-11T14:23:41Z"
  latestGeneratedNode: 5
  managedRolesStatus:
    byStatus:
      not-managed:
      - server
      - signer
      reconciled:
      - registry
      reserved:
      - streaming_replica
      - postgres
    passwordStatus:
      registry:
        resourceVersion: "416259372"
        transactionID: 2655
  phase: Failing over
  phaseReason: Failing over from harbordb-4 to harbordb-5
  poolerIntegrations:
    pgBouncerIntegration: {}
  pvcCount: 2
  readService: harbordb-r
  readyInstances: 1
  secretsResourceVersion:
    applicationSecretVersion: "416336913"
    barmanEndpointCA: "416131910"
    clientCaSecretVersion: "416336909"
    replicationSecretVersion: "416336911"
    serverCaSecretVersion: "416336909"
    serverSecretVersion: "416336910"
    superuserSecretVersion: "416336912"
  targetPrimary: harbordb-5
  targetPrimaryTimestamp: "2023-07-11T15:11:43.397055Z"
  timelineID: 4
  topology:
    instances:
      harbordb-4: {}
      harbordb-5: {}
    successfullyExtracted: true
  writeService: harbordb-rw

Turns out: PVC size was to small. Because of local-path (not dynamically provisioned) i had to recover to a new cluster with bigger PVC from s3 backup


identity: fix bug where fromNodes could be used to allow custom endpoint

Currently when --enable-node-selector-labels flag is set to true user can use (from/to)Nodes sections in CNPs/CCNPs to allow/block access to/from nodes based on custom node label. The node identity contains not only labels filtered out by --node-labels flag, but also contains reserved:remote-node label as seen in this example:
33554556   node:scif.cz/node=worker
           reserved:remote-node

The problematic part is in fromNodes/toNodes sections as those use the same matchLabels selector as classic fromEndpoints/toEndpoints. In case user specifies labels owned not only by node, but also by endpoint he/she might end up in a situation where unexpected endpoints might get access to users deployment.
It is necessary to be either specific or better - prevent this situation.
This commit fixes this by adding a matchExpression with reserved:remote-node key to the policy engine as all nodes have this label.
"fromNodes": [
          {
            "matchLabels": {
              "any:scif.cz/node": "worker"
            },
            "matchExpressions": [
              {
                "key": "reserved:remote-node",
                "operator": "Exists"
              }
            ]
          }
        ]

identity: fix bug where fromNodes/toNodes could be used to allow custom endpoint

/test


Fix lots in export

This is the version of the script that I used to generate the final files for CCS that they are happy with.
Compared to the previous version it:

fixes the numbering of lots (PaaS and SaaS were the wrong way round)
also generates a file for failed suppliers
changes the key to have supplier name first

Could this make use of the sanitise function introduced in https://github.com/alphagov/digitalmarketplace-utils/pull/203 ?

It could, and I did think that, but wasn't sure it was worth making the change given that we should never need to run this script again.  But that might have been me being lazy too, it will be a quick fix.

:+1:


Remove incorrect Signer type

Motivation
This pull request replaces the custom Signer type with the standard type from ethers. This update is due to changes in the ethers v6 library, specifically the deprecation and subsequent removal of the _signTypedData function. The existing custom Signer type, which extends ethers.Signer with _signTypedData, is now obsolete and incorrect.

Solution

Remove the custom type and related ones from types.ts.
Update the seaport.ts import to use the standard Signer definition.

nice catch, thank you!

released in v3.0.2


Can I customize the dictionary of a pre-trained transformer neural machine translation model?

Faiseq has many pre-trained machine translations models available with SOTA performance, but it seems like they all need to be run with the dictionary they are trained with. The dictionaries sometimes can have less coverage for my data set (even BPE based ones), and sometimes miss important words as unknowns. What are the best ways to customize the pretrained models to a dictionary learned from my own data set? For example, some way to transfer learn, like unfreezing the encoder layers and retraining?

You could try freezing the encoder/decoder weights and just learning new embeddings. I know there's some other work in learning adapter layers, e.g., https://arxiv.org/abs/1909.08478

You could try freezing the encoder/decoder weights and just learning new embeddings. I know there's some other work in learning adapter layers, e.g., https://arxiv.org/abs/1909.08478, but I haven't tried this myself.

If we freeze the encoder and  decoder, is it create a new encoder layers and decoder layers it self while it also have frozen layers?


Typo in GtR table name

https://github.com/nestauk/nesta/blob/5d73c749411d58c2c354b24421320da523d1e99d/nesta/core/orms/gtr_orm.py#L295

@kstathou what is this table used for?

To store the document ID, the cluster number and the probability of being in that cluster of a GtR project.
That's part of the clustering_task.py

Ok, and does any code depend on this table that you are aware of?

This tiny bit here: https://github.com/nestauk/nesta/blob/154_clio_clusters/nesta/core/routines/embed_topics/clustering_task.py#L86-L88
So changing the __table_name__ shouldn't break anything.

Ok, and is there a reason that this is only in dev and not in production?

I haven't run it on the full dataset. Joel also added #203 so it might be better to fix that first and then run it on production.

this will be address alongside #270


Processing input via POST /ob/images causes images to become rotated

Report from @drwasho indicates that the POST /ob/images endpoint either persists the rotated image or persists the original image yet returns a rotated image. Whichever the case, the response should be consistent with the original image.

Attempted the following request against a local node using my graphic of choice:
(echo -n '[{"filename": "filename.png","image": "'; base64 ~/Desktop/tmp/sanic-little-steps.jpg; echo '"}]')  | curl -H "Content-Type: application/json" -d @- <IP_ADDRESS>:4002/ob/images
Which provided the following response:
[
    {
        "filename": "filename.png",
        "hashes": {
            "large": "QmQsWqKvYAMtUoKWim8f1axPYaUpwkH8yFJosL5Y7rHjWs",
            "medium": "QmdWJaxYVr1xJ8YduBoXtEuVNU5HKHQF58A2rTtBodzW4T",
            "original": "QmQkizhW5JUhCGhSNgnEMn5ySRBYAiuC1Xzh7shK32xYQV",
            "small": "QmZoPMNeSk5QWKwzjRSL2z8nKxMBBknW6Z9TobP6ZvLaAi",
            "tiny": "QmWXpJwaN4ZsKH7DQPZeuBa5167vzP2KmDQWXCLSdQuLbT"
        }
    }
]

Which looks normal when returned by the gateway: https://gateway.ob1.io/ob/images/QmQkizhW5JUhCGhSNgnEMn5ySRBYAiuC1Xzh7shK32xYQV
All seems fine to me.

I was provided with an example case (which involves a JPG) that causes this issue described above. Steps to reproduce.
Using

Download file to local filesystem.
Start latest master of ob-go.
POST the base64-encoded representation of the file to the daemon using the following:

(echo -n '[{"filename": "pic.jpg","image": "'; base64 ./<pathto>/pic.jpg; echo '"}]')  | curl -H "Content-Type: application/json" -d @- <IP_ADDRESS>:4002/ob/images

Note the hash of the Original value in the response and load the image on the gateway: https://gateway.ob1.io/ob/images/<hashofcontent>
Note that it is rotated from the input.

After examining the ingress of this data and that the encoding really doesn't change as it's processed through, I began looking at the inherent state of the file (because this problem does not occur on all files equally). After taking some additional pictures using an android devices in various orientations, it appears that the EXIF data impacts how these files are interpreted in most GUIs despite the underlying data being different to what's presented. So while the data always "appears" to be right-side up in various utilities, these values are being stripped when processed through the daemon and forced into their natural orientation after processing. ...ultimately, this is the result of golang not considering the EXIF data and expecting this consideration to be handled manually before processing. After understanding the problem, the existing issues were much easier to find around this. (See: https://github.com/golang/go/issues/4341)
Investigating how this can be properly handled in the face of the stdlib's disregard for that metadata.

@placer14 sorry, I should have jumped in here earlier. We had this same issue in the desktop client long, long ago.
This is what we do:

detect the orientation from the file when it is uploaded.
use that to transform the image data to have the correct orientation.
send the transformed image data to the server without any EXIF data.

If the Haven client did the same thing it should work fine. The code is in the onChangePhotoUploadInput method of the views.modals.editListing.EditListings.js file in the desktop repo.

Too little, too late. Nevertheless, after this fix you should be able to simplify the desktop client. 🍻


multiple threads are not in use

Hi dear developers,
Thanks for your scripts! I run the command with 70 threads (-c 70) and found phmmer can not make use of multiple threads, so it is too slow. Maybe you could make a parallel pipeline for the phmmer step.
Best regards.

Hi @tinyfallen,
Thanks for writing and your inquiry!
OrthoHMM version 0.1.0 now handles the phmmer subprocesses differently. Specifically, the subprocesses are spread across multiple threads so that multiple phmmer commands can run simultaneously.
Version 0.1.0 is available on PyPi https://pypi.org/project/orthohmm/.
Happy holidays and happy coding!
Regards,
Jacob

Thank you Jacob, merry Christmas ~

Merry Christmas, @tinyfallen!!
Wishing you all the best in 2025 and beyond!