Compilation issue with libadwaita main branch due to AboutWindow breaking change

Hey there!
So in the stable version of the book, we recommend that users build libadwaita from the main branch. However, we ran into a bit of a hiccup due to a recent breaking change in AboutWindow, which prevent us from being able to compile Libadwaita on Windows as it introduced appstream as a hard-dependency.
To get around this, we should suggest that users clone libadwaita-1-3 for now, although I should note that it's only a matter of time before we'll need to transition to version 1.4 where the breaking change lies.
For more information, you may want to check out the relevant discussions over at:
https://gitlab.gnome.org/GNOME/libadwaita/-/merge_requests/833 and https://github.com/ximion/appstream/issues/231
cc @Exalm

The only issue to be solved here is to make the book use the latest stable release. Anything else should be resolved upstream.

I mean I'll just reiterate what I said in the libadwaita MR. AppStream issue has had the help wanted tag for more than 3 years now and ximion mentioned they would be fine with a port, just worried there would be no users. Libadwaita would be that user.
If nobody wants to do that - I mean then 1.4 is gonna be released as is. Because just as in the appstream issue, libadwaita does not explicitly target Windows - if it works, cool, but it's not a requirement and nobody working on libadwaita is testing it there.


Two factor as second login step?

In most cases where I've used Two-Factor Auth the auth code is a second login step.
Are they any good examples anywhere on how to make this a two step process?

Howdy folks, late to the discussion, but wondering if this might work: (modified from @jonathansimmons)

User logs in
A before filter fires on Session create

Finds the user based on e-mail provided
Validates the email and password
renders the Two-Factor step two if two factor enabled

Two-Factor step two is a second form requesting ota_attempt

hidden fields store the plaintext email/password/remember_me
User submits ota_attempt and we validate to stored user

Initially this seemed insecure, but how insecure is it really? The only vulnerability I can think of is if the user walks away from the computer while their browser is on the two-factor page, and a potential attacker comes and inspects the DOM for their plaintext password.... and this isn't really a vulnerability, right?
What am I missing? ;)

Although it seems innocent, putting a plaintext password on a page is a really really bad idea.  It's the same reason it's important to disable autocomplete for password fields - people at an internet cafe or library will be vulnerable to this.
Pages get cached. A page that gets cached in a browser on a public computer that includes your plaintext password is bad news.
Pages get saved by accident. All it takes is one accidental hit of Ctrl-S and there's a page on the desktop for anyone to open up later and get the plaintext password.
...

You kind of get where this is going - including the plaintext password on a page is a really bad idea due to the myriad number of ways in which an attacker can later access it; even if they don't have local access to the computer, but they have a RAT set up, they can later look at the cache and pull down their password.
The best way to do this, I think, is to pass some sort of secure signature along that you can verify, server-side, as having been a user authenticated via password. That way you don't have to pass along a uid or anything like that either. Ideally, you'd want to time-bound and expire the signature somehow as well.
Additionally, rails sessions are conveniently signed already, so you could simply add a parameter to the session as well, but you may need to clean it up after they're logged in or if they choose not to complete the login process.
Does that make sense?

Yup, thanks for the sanity check. Pushing a fix to our integration right now.

I think this requires at least small section in README. Since every service implements 2FA in 2 steps - its really unclear how to implement it with current library.

Hey all I found https://github.com/AsteriskLabs/devise_google_authenticator which appears to solve with workflow out of the box.
It's a has bad name which is why I missed it in my initial report. You can vote for a rename of the gem [here[(https://github.com/AsteriskLabs/devise_google_authenticator/issues/41) and please do!

Updating my comment. I've ending up using https://github.com/wmlele/devise-otp and have spent a good amount of time activley contributing to it.
The gem resolved this flow and has a lot of flexibility. Again just an FYI for those looking for this feature inparticular.

Hey @jonathansimmons, I just stumbled upon this thread as I'm looking at creating a two-stage login using this gem and your fork is dead. Did you manage to get this working nicely with tinfoil/devise-two-factor before I jump in and try it myself? It would be good to hear what troubles you encountered before I tackle it myself if you've got a moment to comment.
Cheers,
Rikki

@rikkipitt The last time I used my fork it was working fine. Devise has been updated a few times since then.
I did make some significant customizations in my exact implementation. I needed it for a project a littler over a year ago. I tried to make some progress with the maintainer but he stopped communicating.
That being said it's lightweight and should work. Email me if you have questions and I'll see if I can help.

Cheers @jonathansimmons. What I've got so far is a normal login with a before_filter to check if the user has otp_required_for_login set to true - redirects to a verification form view. I'm storing the email/password in session (encrypted).
The problem I'm now having is merging the email/password params and the verification code before triggering the default sessions_controller create method. How did you do the auth step in the end?
Cheers

@rikkipitt email me at jon{at}jsdev.co so we can move it off this thread as it's not relevant to this repo.

Hi @jonathansimmons, I am wondering if the gem that you suggested: https://github.com/wmlele/devise-otp is building on top of tinfoil/devise-two-factor, or is it a solo gem by itself?
I have implemented all the backup codes and the verification code with tinfoil/devise-two-factor and I am wondering if I can combine the implementation with the gem that you suggested. I am currently stuck at the two-step log-in feature.

Hey, I am wondering if anyone managed to implement the second login step using tinfoil/devise-two-factor gem? I have implemented a bulk of the features using this gem, and the only thing that is lacking is the second login step.

GitLab uses devise-two-factor with a two-step login. I believe the relevant controller is their SessionsController and this. I believe they store the user ID in the signed/encrypted Rails session.
I'd definitely welcome any clarifications to the README about this from someone who has implemented it. I believe GitLab's method addresses @borski's security concerns above.

@bsedat Thanks for letting me know that big project like Gitlab is using tinfoil/devise-two-factor as well. I will try to tinker with it and see if it works for my project.

We run a bug bounty program so our implementation should be pretty secure :)

@jeffreyyong no it does not build upon this gem. I believe it was based off a google authenticator gem.
It's been a while since I've used either of these gems personally but I know each of them can do what you're looking for with a little elbow grease.

@connorshea I managed to implement the two-step login in my system after looking at the GitLab example! Thank you very much.

@jeffreyyong Would you be kind enough to demonstrate your implementation please? When you get the time of course!

@jeffreyyong that's really great to hear! Open source at work <3

@rikkipitt  Sorry for the late reply. Unfortunately, I can't post my implementation to you because it's my company code that is not open source. However, I referred to the Gitlab open source code: Gitlab The two main files that I looked at were app/controllers/sessions_controller.rb and app/controllers/concerns/authenticates_with_two_factor.rb

@connorshea Yes! Open source rocks! :)

Got mine working too with help of GitLab's implementation.

Isn't GitLab's two step login method is leaking info (whether OTP enabled/disabled) about users? If you know the username of course, without knowing the password.
Isn't better to use :database_authenticatable before checking if OTP enabled/disabled for the user?

A few of you have suggested using the session cookiestore to store the state of the user
I would recommend storing it in a custom cookie (specifically an encrypted one) so that you can set a separate expiry time (5 minutes) instead of using a rails session cookie (where setting a short expiry can be very inconvenient)

I've consolidates some of the pieces of advice in the various comments into the following step by step guide with a sample project:
https://www.jamesridgway.co.uk/implementing-a-two-step-otp-u2f-login-workflow-with-rails-and-devise/

A security improvement idea. Instead of storing user's id in the session (to load the use on 2FA screen)
session[:otp_user_id] = user.id

I suggest storing a random token
session[:otp_user_id] = SecureRandom.alphanumeric(64)

In addition, store token's hash and its expiration date in the users table. On 2FA screen, load the user by token's hash and check the token's expiration date. This gives additional security if you store your session in an encrypted cookie. It remains secure even if  cookie encryption key leaks out. It also allows to invalidate the token on the server side.


could not find installed module

I have Pandoc installed globally (since I use it for general file format conversion, not as part of a project), and I'm trying to write a filter for it. This requires import Text.Pandoc. In ghci, I can import it just fine, but hdevtools gives the following error:
Could not find moudle 'Text.Pandoc.JSON'. Use -v to see a list of the files searched for.

Does hdevtools have any way of picking up global packages?

Hey this sounds a lot like an existing issue. How did you install pandoc, i.e. where is it located?
Sent from TypeApp
On Nov 8, 2016, 11:04 PM, at 11:04 PM, Alok Singh<EMAIL_ADDRESS>wrote:

I have Pandoc installed globally (since I use it for general file
format conversion, not as part of a project), and I'm trying to write a
filter for it. This requires import Text.Pandoc. In ghci, I can
import it just fine, but hdevtools gives the following error:
Could not find moudle 'Text.Pandoc.JSON'. Use -v to see a list of the
files searched for.

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/hdevtools/hdevtools/issues/45

I installed pandoc with stack into ~/.local/bin (the standard global bin for stack packages).


Sort filters values

Hello!
This issue has been discovered some time ago and fixed on our production version. I would like to take some to discuss about a final patch for this issue.
Filters values sorting
Here is an example, we have 100g, 150g, 200g and 1kg. But we now have a few new products (yay!): 50g and 2kg.

So, here we go, we add everything, everything is great and we take a look into the groceries category. Issue: the filters values are unordered. Which means, the customer will have a few logical issues when filtering.
Proposition
I'm thinking, maybe we could reorder the values when we are adding them (by adding a position column for example and cache the sorted data).

Edit: while I'm looking at my Saleor checklist, I also note that we shouldn't be showing filters containing no values. This case was happening when using attributes containing no pk values, only text values in products.

ref #915

I think we had a discussion about it in one of the PRs recently. We agreed that there should be the ability to reorder attribute values.


u-boot: ROCK64 RAM init improvements

Description of changes
This switches the ROCK64 over to the open-source RAM init as it now works flawlessly. It also removes the HDCP flag from the ATF for the RK3328 as it cannot use it, it is only used in the RK3399. This makes the ROCK64 u-boot now fully open.
There is also an issue with the ROCK64 v2 revision where the DRAM routing is marginal, making some of them unstable. So also package a variant which uses a lower-speed DDR3 timing configuration which is stable on these boards.
Things done

Built on platform(s)

[x] x86_64-linux (cross-built to aarch64)
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

I tested ubootRock64v2 on my ROCK64 v2 and it boots, although not completely reliably. I don't have physical or UART access to the board right now (just a remote power switch), so I don't have any idea where it is getting stuck (it might not be U-Boot).

I've actually never had it fail at the u-boot stage already (with the full 1600MT/s), always after a few minutes to hours of operation. The special v2 version has been running on my V2s for ~3 months without any incident. Would be interesting to know if yours actually fails to start u-boot.

See also #145506 and #146725

@lopsided98 Please retest this, I figured out that the patch hook for the reduced-clock version applied too late. Before I PRed this, I used a patch which applied earlier and it worked.

That explains why my Rock64 locked up after a few days as well. I just tested the new version and it seems to boot reliably.

Pinging @lopsided98 @dezgeg @samueldr

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/1392

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/1945

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/installing-nixos-on-rock64-2024-experience/40954/1


create redundant table for aggregated bill reports

Description
this pr creates a redundant table for aggregated bill reports, and lets the contracts sql query make use of it.
Related Issues

#231
#308

it is worth noting that currently some contracts on devnet have +30K bill reports, so even with the redundant table, it takes time just to fetch them.
another solution is to separate bill reports information from contracts.


Restart not working if application was started with shell script

Hi @Dansoftowner
My application can be installed and started in two ways - via an installer or via downloading a ZIP file.
(See https://document-archiver.github.io/index.html#download)
When running the application from an installation, the restart works fine.
When running the application from the ZIP file (via shell script), the restart fails:
Exception in thread "Thread-3" com.restart4j.RestartException: Couldn't execute the starter command with the OS: [/usr/bin/java --module-path lib --module DocumentArchiver/com.sophisticatedapps.archiving.documentarchiver.Launcher]
	at Restart4j@1.4.1/com.restart4j.ApplicationRestart.execArray(ApplicationRestart.java:94)
	at Restart4j@1.4.1/com.restart4j.ApplicationRestart.exec(ApplicationRestart.java:82)
	at Restart4j@1.4.1/com.restart4j.ApplicationRestart.lambda$buildNewProcessCreation$4(ApplicationRestart.java:74)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Cannot run program "/usr/bin/java --module-path lib --module DocumentArchiver/com.sophisticatedapps.archiving.documentarchiver.Launcher": error=2, No such file or directory
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
	at java.base/java.lang.Runtime.exec(Runtime.java:592)
	at java.base/java.lang.Runtime.exec(Runtime.java:451)
	at Restart4j@1.4.1/com.restart4j.ApplicationRestart.execArray(ApplicationRestart.java:92)
	... 3 more
Caused by: java.io.IOException: error=2, No such file or directory
	at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
	at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
	at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
	... 7 more

This is the shell script:
myMac2014:document-archiver-2.1.0-mac stephan$ cat DocumentArchiver.sh
java --module-path lib --module DocumentArchiver/com.sophisticatedapps.archiving.documentarchiver.Launcher

Hi @stephansann ,
Thanks for reporting this issue!
I will look into it!
Best regards
Daniel

Hi again, @Dansoftowner
Just checked on Linux and there it works in both cases.
Seems to be a problem on macOS only.
Let me know if I can help you with input.
Best regards
Stephan

Hi @Dansoftowner
Any progress on this? Anything I could do from my side?
Just asking because I want to do a new release of my app soon, and it would be great to have a fixed restart in it 🤓
Thanks and best regards
Stephan

Hi @stephansann ,
Try this code:
ApplicationRestart.builder().modifyCmd(cmd -> cmd.replaceAll("\\s+", "\00")).build().restartApp();

And let me know if it works or not!
Sorry for the late reply!
Best regards
Dsaniel

Hi @stephansann ,
Try this code:
ApplicationRestart.builder().modifyCmd(cmd -> cmd.replaceAll("\\s+", "\00")).build().restartApp();

And let me know if it works or not!
Sorry for the late reply!
Best regards
Dsaniel

Hi @Dansoftowner
Nice! With this code it now works on macOS, started from ZIP-distribution.
The other five ways (three platforms, two distributions - the one above) I didn't test.
Let me know if you need that - I would have to build an installation package and fire up my VirtualBox.
Best regards
Stephan

Hi @stephansann ,
Well, on Windows it probably does not work, so I will make a new release soon :)

Great - looking forward to it 😊

Check out version 1.4.4!
If the problem is still present on MacOS, please let me know!
Also, If you don't mind, please test it on Windows too (I already tested it on Linux).

Use the normal ApplicationRestart.builder().build().restartApp().

Hi @Dansoftowner
Thanks for the new version.
Now the ZIP distribution version restarts correctly, but the installed version has a problem.
It's a problem with folders which have a blank in the name. I think we had this before, right!?
The installation folder is:
"/Applications/Document Archiver"
Exception in thread "Thread-10" com.restart4j.RestartException: Couldn't execute the starter command with the OS: [/Applications/Document, Archiver/DocumentArchiver.app/Contents/MacOS/JavaApplicationStub]
	at Restart4j@1.4.4/com.restart4j.CommandLineExecutor$MacOSExecutor.exec(CommandLineExecutor.java:70)
	at Restart4j@1.4.4/com.restart4j.ApplicationRestart.exec(ApplicationRestart.java:80)
	at Restart4j@1.4.4/com.restart4j.ApplicationRestart.lambda$buildNewProcessCreation$4(ApplicationRestart.java:72)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Cannot run program "/Applications/Document": error=2, No such file or directory
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
	at java.base/java.lang.Runtime.exec(Runtime.java:592)
	at java.base/java.lang.Runtime.exec(Runtime.java:451)
	at Restart4j@1.4.4/com.restart4j.CommandLineExecutor$MacOSExecutor.exec(CommandLineExecutor.java:68)
	... 3 more
Caused by: java.io.IOException: error=2, No such file or directory
	at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
	at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
	at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
	... 7 more

I removed the previous command parsing strategy and updated OSHI in version 1.4.6.
If the problem is still present, let me know!
Thanks for your patience!

With this version we are back to field 0 😕
When started from ZIP distribution, I once again get this:
Exception in thread "Thread-3" com.restart4j.RestartException: Couldn't execute the starter command with the OS: [/usr/bin/java --module-path lib --module DocumentArchiver/com.sophisticatedapps.archiving.documentarchiver.Launcher]
	at Restart4j@1.4.6/com.restart4j.CommandLineExecutor$MacOSExecutor.exec(CommandLineExecutor.java:69)
	at Restart4j@1.4.6/com.restart4j.ApplicationRestart.exec(ApplicationRestart.java:80)
	at Restart4j@1.4.6/com.restart4j.ApplicationRestart.lambda$buildNewProcessCreation$4(ApplicationRestart.java:72)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Cannot run program "/usr/bin/java --module-path lib --module DocumentArchiver/com.sophisticatedapps.archiving.documentarchiver.Launcher": error=2, No such file or directory
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
	at java.base/java.lang.Runtime.exec(Runtime.java:592)
	at java.base/java.lang.Runtime.exec(Runtime.java:451)
	at Restart4j@1.4.6/com.restart4j.CommandLineExecutor$MacOSExecutor.exec(CommandLineExecutor.java:67)
	... 3 more
Caused by: java.io.IOException: error=2, No such file or directory
	at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
	at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
	at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
	... 7 more

Okay, so let me explain the problem.
In order to restart the application, Restart4j has to retrieve the command-line that started the app.
Normally,on MacOS and Linux, this command line is delimited by null-chars (\00) that allows you to separate the arguments.
The bug is (probably in OSHI there is a problem) that if the app was started with shell script, this command is not delimited by null-chars, only by spaces. So if I split the command based on the spaces, I might get a wrong array (for example if the application folder has spaces in it's name like "Document Archiver").
So, until this bug is not solved, my solution for you is this:
In your MacOS shell script, pass an application argument for Document Archiver like --mac-script that indicates that the app was started with shell script:
java --module-path lib --module DocumentArchiver/com.sophisticatedapps.archiving.documentarchiver.Launcher --mac-script

Save the main-class app arguments:
public class Launcher {

     private static List<String> arguments;

     public static void main(String[] args) {
           arguments = List.of(args);
           ....
     }

     public static List<String> getArguments() {
           return arguments;
     }

Then:
ApplicationRestart.builder()
              .modifyCmd(cmd -> {
                       if (Launcher.getArguments().contains("--mac-script") {
                             return cmd.replaceAll("\\s+", "\00");
                        }
                        return cmd;
                }).build().restartApp();

It's just an example but I hope you got the point.
If you have any questions, tell me please!
I will keep in mind this issue and notify you when the issue is solved.
Best regards
Daniel

Hi @Dansoftowner
Okay - I see the problem. Thanks for the suggestion. I will implement something in this direction.
You say there may be a problem in OSHI. Are you going to open an issue for this or should I do that? (Even though you know the specifics better than I do.)
Thanks again and best regards
Stephan

Hi @stephansann ,
Don't worry, I will report the issue to the OSHI guys soon :)
Best regards
Daniel

Great.
For the time being I will have a look at the environment variables, which will reveal which way the app was started.
But I will be happy to remove this hack again as soon as possible.
Thanks for the always pleasant interaction 🙏🏻
If we ever meet on some kind of Java Conference (do you visit such?), we should have a beer together 😉
Best regards
Stephan

OSHI maintainer here.  Found the bug. It's intermittent/unpredictable whether you'll get nulls or spaces, actually, so a better test of how to parse is whether there are any embedded nulls at all .... if not, assume it's spaces.
I'll have a fixed version of OSHI out this weekend.

Hi @dbwiddis ,
Thanks for your cooperation!
Yes, my first idea was your temporary solution ("check whether there are any embedded nulls at all .... if not, assume it's spaces"), but it's not reliable: if an executable's file path has spaces in it, I will get a wrong cmd-array.
For example:
The command line: "/Applications/Document Archiver/DocumentArchiver.app/Contents/MacOS/JavaApplicationStub"
Check it has nulls: no
So if I split it based on spaces, I get this:
["/Applications/Document", "Archiver/DocumentArchiver.app/Contents/MacOS/JavaApplicationStub"]
And it obviously won't work:
...
Caused by: java.io.IOException: Cannot run program "/Applications/Document": error=2, No such file or directory
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
	at java.base/java.lang.Runtime.exec(Runtime.java:592)
	at java.base/java.lang.Runtime.exec(Runtime.java:451)
	at Restart4j@1.4.4/com.restart4j.CommandLineExecutor$MacOSExecutor.exec(CommandLineExecutor.java:68)
	... 3 more
Caused by: java.io.IOException: error=2, No such file or directory
	at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
	at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
	at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
	... 7 more

Yep. I agree parsing spaces is hard (if not impossible).  My suggestion was that if this is always used in a consistent manner (e.g., your own process) you might have more information about your own command line to help the parsing.   Alternately You could try to catch errors like the above and retry different parsing if it looks like a file path, etc...
All a pain... or wait for me to get out of a crazy busy work week and fix it properly in a few days. :)

Alternately if you are on JDK 9+, you can use the ProcessHandle API to get the value correctly.

Thank you for the advices!
We will wait for your bug fix, no problem;).
Thanks for your work.

FYI, I've just released OSHI 5.7.4 with this fix.

@dbwiddis Great - thanks a lot for that 🙏🏻
@Dansoftowner let me know if/when I can help you testing on Mac 🤓
A nice evening to both of you 🍀

@stephansann you should be able to place the oshi-core dependency in your own dependency manager config (build.gradle or pom.xml) and it should override the transient dependency in Restart4J.  So you could test now without waiting for Restart4j to update.

@dbwiddis Thanks for the suggestion. But AFAIR Daniel already tried to include workarounds in his code for the problem. So just exchanging the the OSHI version may not do the trick and may even end up in unexpected results. But as I know Daniel he is very quick with new versions, so I'll wait for that.

@stephansann The workarounds were for having spaces instead of nulls.  This issue mentions being started by a script, but in actuality the underlying bug was random (improperly byte sized values on the native stack), so any patterns you detected were not consistent.
oshi-core version 5.7.4 will only have nulls so there are no workarounds needed.  (Actually the bug was introduced in 5.3.7 so versions 5.3.6 and earlier should also be safe.). Unless one of the workarounds assumes spaces when there are actually nulls, it shouldn't work.   In that case, reverting to the version of Restart4J just before the workarounds, plus the newest OSHI, should work.

@stephansann Check out version 1.4.7! :)

Hi @Dansoftowner
Thanks a lot for the quick implementation.
I now included it in Document Archiver and it works for both "startup ways" 🎉
BTW had to exclude the old OSHI version in jSystemThemeDetector to make it work. So maybe you could update it there, too!?
@dbwiddis - sorry I wasn't able to answer your post yesterday evening, had to go. Anyhow this is fixed now (at least for Big Sur (I hope we don't have a problem with older macOS versions now 😬)).
Both of you have a great start into the new week 🍀
Best regards
Stephan


Remove edit/delete permission for index pages

See https://praekeltorg.atlassian.net/browse/MOLO-99

👍 but I don't like the template overrides - this will make upgrading to future versions of wagtail hella  difficult. But I'm not seeing an easier way at the moment so I'm happy with this.
Also a note for @codiebeulaine with the Merged CMS - We'll need to allow Main to be deletable so we'll have to remove the mixin from Main.


Default download location

Still refining it.
#266

Sorry it's taken so long - I'll merge this and give it a cleanup on master. Thanks for the contribution, and for your patience!


openAI Node default settings are very confusing

This is based on a partner question related to the openAI Extension.
Based on my testings a few things should be changed:

the default value for prompt should be {{input.text}} , the current value is wrong and utterly confusing, the model already has the information
based on the openAI documentation and my testing the following default settings should be changed to the following values:

temperature: 1
max_tokens: 100
top_p: 0.1

In terms of postprocessing the following code provides good results in extracting the first proper paragraph:
input.openai.match(/\w.*[!"#$%&'()*+,\-.\/:;<=>?@[\]^_{|}~]/m)`
this should probably be part of the readme

Thank you very much @cabullmann . I will implement and test the suggested changes.
Just for my understanding: Do you want to have {{input.text}} as default value for the text fields of the Flow Node?

@alexteusz yes, exactly


support task nesting for exported / crosslinked tasks

Problem Description:
when one or more blocks are crosslinked (with the crosslink open tasks or the create tasks from selection feature) they end up as "parent tasks" in Todoist.
The indentation of the blocks is not considered
Preferred Solution:
export tasks and blocks like they are nested in Craft - so create subtasks if the upper level block is also a task

+1 for this

@jshum please note that I'm not actively adding features to the extension anymore.
The Craft API didn't receive any updates in more than a year and I personally moved away from Craft.
If something is NOT working anymore in the current state I'm happy to help.

Ah Sorry to hear that, but understandable. Can I ask what note taking and todo solution you are using now?

It's been hard to find a note taking solution that works somewhat well with todoist

I moved my notes to obsidian but that's also on the edge right now since I don't need that much from my notes app. Currently I'm just using Drafts (getdrafts.com) and I'm happy with it. I didn't move my notes from obsidian over to Drafts yet, but i think I will do that soon. I'm a long term Drafts user 😃
Tasks are still in Todoist 👍


Supabase Functions to supports GET (and other) requests

Feature request
Is your feature request related to a problem? Please describe.
I may not be seeing everything so I apologize for whatever I may be overlooking.
There are problems that present themselves when you have only POST requests to work with.  If you consider that command-query separation is a well-appreciated approach for writing software, you are reminded that nearly all software relies on commands and queries.  The commands effect change in the system and the queries provide views of data to inform decisions and, in turn, are used to issue commands.  Serverless functions should, thus, afford both.
POST has only the connotation of issuing a command.  So what you have when POST is all you can do is a system which issues commands but cannot be queried.  It's true you can query the database directly via a client library, but this loses one of the native benefits of HTTP: caching.
When I put my database calls behind an HTTP GET endpoint I gain this for free.  It reduces network chatter and improves app response time, all without any special coding.  Whenever the app asks for something it previously requested (e.g. same endpoint, same query string), I get it for free because my GETs are idempotent, and cached in my browser!
I'm offering an argument for having the complete set of HTTP methods on the roadmap.  Otherwise, once a developer needs a query function he'll be forced, as I was, to look elsewhere.  It wouldn't make sense for a developer to use Supabase's Edge Functions along side Cloudflare Workers.  Rather, for consistency's sake, he'll opt into the more robust offering.
Obviously, my use case is specific.  I benefit from cacheable GET requests.  It would seem, however, that providing serverless functions that, in certain situations, require developers to look elsewhere make them too fragile to build upon.  That is, why would one build on POST-only functions knowing the minute he needs a cacheable GET request he'll require another vendor?
Describe the solution you'd like
Functions which comprehensively support other methods, esp. GET, fill the gaps needed by many apps.  With GET implemented, at the very least, many network responses can be cached and thus ignored altogether.  It also permits JAMStack-friendly service providers like Cloudflare to cache on the edge and not just in the browser.
Describe alternatives you've considered
I considered using the Supabase client to hit the database directly in lieu of HTTP GETs.  It gets me the data but not the cache.  I could implement caching logic in the client but that's work I don't have to do with an HTTP GET.
I'm considering using Cloudflare Workers which handle HTTP GET/POST.

@mlanza Thanks for sharing this feedback. Yes, support for other HTTP verbs other than POST is something we're working on (incl. GET). Will update this issue when we release it.

Super.  I figured it was probable but not seeing mention in the docs/blog esp. pertaining to Edge Functions, I just wanted to be sure it was on the roadmap.
Thank you.

Closing this issue. We'll update when we release this feature.

I have implemented my app's functionality with Workers so this issue is not critical to my work.  I still think it's a critical offering for Edge Functions for reasons already mentioned.


Dimension problem

i am trying to video capture. There is a black area padding right of video. I tried resize_to but it doesnt work. can you help me?

i solved this using chrome driver.


Hey,I made a new distro,please add the ascil logo to neofetch！

##logo

##Distro name
Trix OS
thanks～

Can you please provide an url where Trix OS is published?

There is nothing at the moment https://github.com/pyqi-Bilibili/trix_os


Use case study: Intheon

We're very happy to see progress in getting the array API standardized -- obviously a monumental undertaking when all is considered.
We've been using backend-agnostic numpy-style code for over a year now in research and now production, and are gradually rolling it out across a multi-megabyte codebase (supporting mainly numpy, cupy, pytorch, tensorflow, jax (also dask, but not using that at the moment), so I thought I'd share our user story in case it's helpful. It's a similar use case than what the array API addresses, but it was built with the current numpy workalike APIs in mind.
Our codebase has an entrypoint equivalent to get get_namespace, although we call them backends, and a typical use case looks like:
def sqrt_sum(arr1, arr2):
    be = backend_for(arr1, arr2)
    return be.sqrt(be.asarray(arr1) + be.asarray(arr2))

For each of the backends we have a (usually thin) compatibility layer that adds any missing functions or fixes up issues with the function signature. In our case, backend_for looks at __array_priority__ to return the namespace for the highest-priority array, although we rarely use it with more than one array (but it results in the above function accepting multiple array types and promoting according to np < dask < {jax, tf, cupy} < torch). Getting a namespace this way is about as fast as doing .T on a 1x1 numpy array (thanks to some caching) so we use it on even the smallest subroutine.
We use a lot of the API surface of these backends, and typically the most compute-intensive subroutines have an option to choose a preferred backend (via a function backend_get(shorthand)) out of the subset that supports the necessary ops, or to keep the same backend. We've been extremely impressed with the compatibility of cupy, the performance of torch (notably also its cpu arrays), and we have a few places where we prefer tf or jax because they might have a faster implementation of a critical op or parallelize better (e.g. jax). We find that even over a half-year time span, things evolved rapidly enough that the ideal backend pick for a function may change from one to another (one of the reasons we aim for that much flexibility).
We traced our API usage and found that perhaps 90% of our backend-agnostic call sites would be covered by the array API as it exists now. There are a few places that use different aliases for the same functionality (due to some forms being more popular with current backends than others) the most frequent issues being absolute(x) and concatenate(x), other examples being arccos(x) (all our backends) vs acos(x) (no backend). We also frequently use convenience shorthands like hstack(x), vstack(x), ravel(x) or x.flatten(), but those could be substituted easily (or provided in a wrapper).
We found a few omissions that would require a bit more code rewriting, among others the functions minimum(a,b), maximum(a,b), clip(x, lower, upper) (presumably that would turn into where(x>upper,upper,where(x<lower,lower,x)). Also we frequently use moveaxis(x,a,b) and swapaxes(x,a,b), e.g., in linear algebra on stacks of matrices (>30 call sites for us). All of these are supported by the above 6 backends already and they're pretty trivial, fortunately. Our code uses real(x) in some places since some of the implementations might return spurious complex numbers; that may be reason enough to at least partially specify that function already now. Also reciprocal(x) occurs frequently in our code, I guess that's from a suspicion that writing 1/x may fail to use the reciprocal instruction if it's available.
A few things that we use have no substitute at this point, unfortunately, namely einsum(), lstsq(x,y), eig(x), and sqrtm() (though the latter could be implemented via eigh). We hope that these eventually find their way (back) into the API. I realize that lstsq was removed as per a previous discussion (and it's understandable given that the API is a bit crufty), but then our code base has 26 unique call sites of that alone, since we're dealing mostly with engineering and stats. One might reasonably assume that backends that already have optimized implementations of that (all 6 do, and torch/tf support batched matrices) will provide it anyway in their array API namespace. However, we do worry that, given that it has been deliberately removed, we can't be sure that some of the existing backends won't be encouraged to minimize their maintenance surface and drop functions like that from their formerly numpy-compatible (soon array API compatible) namespace, forcing users like us to deal with their raw EagerTensor, xla_extension.DeviceArray, or whatever it may be called, and go find it in whichever ancestral tensorflow namespace the functionality may have been buried in before. We're wondering if a tradeoff could be made where e.g., some of the rarely-used outputs could be marked as as "reserved" placeholder and are allowed to hold unspecified values (e.g., None) until perhaps at some future date the API specifies them. There's also the option to go the same route as with svd, where some arguments were removed in the interest of simplicity. On the plus side, it's good to see diag retired in favor of diagonal (especially so in the age of batched matrices).
Other than that, for multi-GPU we use a backend-provided context manager where available (torch, tf, cupy) a custom context manager where it's not (jax), and a no-op context manager for numpy & dask (usage looks like with be.select_device(id):). That's because passing device= through all compute functions down into the individual array creation calls (from arange to eye) just isn't all that practical with a large and deeply nested codebase, and it's easy to overlook call sites, causing hidden performance bugs that only turn up on multi-accelerator runs -- however, since the user can write their own context manager (with a stack in thread-local storage and wrappers around the array creation functions), that can be worked around with some effort.
Lastly, our indexed array accesses in our main codebase (the parts that we hope to eventually port) look like the following:
x[:, :, slice,:]        # similar expressions occur hundreds of times with variable counts of :'s
x[:,:,slice,:,slice,:]  # hundreds of times, but can fall back to chained x[:,:,:,:,slice,:][:,:,slice,:,:,:]
x[:,:,indices,:]        # similar uses occur dozens of times
X[:, an_int, :, :]      # probably a few dozen times
X[:, bools, :, :]       # a few to a dozen times
x[:, reverse_slice,:]   # occasional (not supported by current pytorch)
x[indices, :, indices]  # never (we don't use numpy advanced indexing with multiple arrays)
x[:, None, slice]       # used as equivalent to np.newaxis

We use a high-level array wrapper (similar in spirit to xarray) that supports arbitrarily strided views and allows writes into those views, which results in low-level calls (in the guts of the array class) equivalent to the form:
be.reshape(be.transpose(x, order), shape)[:,slice,:,:,slice,:] = y  # invoked similarly from hundreds of places (indirectly)

... that's because we spend much of our time dealing with multi-way tensors (e.g., neural data) that have axes such as space, time, frequency, instance, statistic, or feature (often 3-5 at a time), and most subroutines are agnostic to the presence or order of most axes except for one or two, so they create views that move those few to specific places and then read/write through the transposed view. Our way of dealing with backends that don't support that is not enabling them for those functions (and having feature flags on the backends for reverse indexing, slice assignment, and mesh indexing support to catch cases where we do).
I wasn't sure if this is the right place to report relevant API usage "in the field", hopefully it is.

Hi @chkothe, thanks so much for the very detailed write-up of your use case! This is a good place to share.
My overall takeaway from the statistics and details you provide is that we've gotten things mostly right and our minimalist API surface covers a pleasantly large amount of what you need. However you also point out some gaps that would be helpful to address, so let me try and respond to each and see which ones we need to follow up on:

We use a lot of the API surface of these backends, and typically the most compute-intensive subroutines have an option to choose a preferred backend [...]

This part surprised me. I'd expect that converting between array types per function call will be quite expensive, introducing both overhead and possible bugs. Do you have some examples for functions where you do or don't allow this?

the most frequent issues being absolute(x) and concatenate(x),

concatenate vs. concat I understand (we went with conciseness + Python's naming), but in NumPy abs and absolute are aliases, and np.abs is the much more commonly used one. So for absolute just making changes in your code base seems okay?

other examples being arccos(x) (all our backends) vs acos(x) (no backend).

TensorFlow and PyTorch (and C99) use acos. NumPy also tends to follow C in such naming, but the arc* functions ended up differently for historical reasons. I think the choice is reasonable.

We also frequently use convenience shorthands like hstack(x), vstack(x), ravel(x) or x.flatten(), but those could be substituted easily (or provided in a wrapper).

There is something here that we still need to follow up on (not enough hours in a day ...): can we popularize/adopt einops as a better API for these sort of manipulations? https://github.com/data-apis/consortium-feedback/issues/3

We found a few omissions that would require a bit more code rewriting, among others the functions minimum(a,b), maximum(a,b), clip(x, lower, upper) (presumably that would turn into where(x>upper,upper,where(x<lower,lower,x)).

I'm fairly sure minimum and maximum came up somewhere else recently and they have good support across array libraries. I am just failing to find the relevant discussion right now.
clip is fairly heavily used and the use of where to emulate it is a bit cumbersome - this may be worth splitting off as a separate issue to discuss standardizing it (or have a good reason for why not).

Also we frequently use moveaxis(x,a,b) and swapaxes(x,a,b),

moveaxis: discussed in https://github.com/data-apis/array-api/pull/42#issuecomment-695986375, was left out initially because PyTorch and TensorFlow don't have it.
swapaxes: pointed out in that same PR as missing, but not discussed there further. TensorFlow also doesn't have it, but PyTorch does have it.

Both of these are good candidates to add I'd say. Let's open a separate issue for them.

Also reciprocal(x) occurs frequently in our code, I guess that's from a suspicion that writing 1/x may fail to use the reciprocal instruction if it's available.

I'm not sure if this is actually guaranteed - for example the numpy docs just say "calculates 1/x". reciprocal is implemented by all libraries though - it was just not considered for inclusion yet because it's fairly niche (I just checked SciPy for example: just 2 instances of np.reciprocal).

A few things that we use have no substitute at this point, unfortunately, namely einsum(), lstsq(x,y), eig(x), and sqrtm() (though the latter could be implemented via eigh). We hope that these eventually find their way (back) into the API.

einsum is planned, and there's a PR up for eig (gh-113).
sqrtm (and related functions like expm, logm, etc.) are very hard to implement, so I'm fairly certain we don't want that one in the standard.
lstsq: as you comment on, this one is a little complicated. gh-240 points out that it can be considered for re-adding in the future. It's a bit of a mess though, so TBD what it would look like.

However, we do worry that, given that it has been deliberately removed, we can't be sure that some of the existing backends won't be encouraged to minimize their maintenance surface and drop functions like that from their formerly numpy-compatible

I very much doubt this will happen. I have so far not seen any suggestions like this, nor do I think it's a valid reason to remove functionality. Each library is going to provide a superset of what's in the standard. Deprecation and removal of existing functionality requires a good reason, like "confuses users", "there are now superior alternatives", "it's broken", etc.

We're wondering if a tradeoff could be made where e.g., some of the rarely-used outputs could be marked as as "reserved" placeholder and are allowed to hold unspecified values (e.g., None) until perhaps at some future date the API specifies them. There's also the option to go the same route as with svd, where some arguments were removed in the interest of simplicity.

The svd route seems preferred here. Or just transitioning to a new name with saner behavior, just like we left out unique and replaced it with (in that case multiple) new functions.

Other than that, for multi-GPU we use a backend-provided context manager where available (torch, tf, cupy) a custom context manager where it's not (jax), and a no-op context manager for numpy & dask (usage looks like with be.select_device(id):). That's because passing device= through all compute functions down into the individual array creation calls (from arange to eye) just isn't all that practical

We went back and forth on that quite a bit. I think we need to see this work in practice for a bit during adoption of the standard in array-consuming libraries. There were certainly arguments for adopting a context manager. One pain point I remember is that semantics were hard to agree on - for example PyTorch has a rule of never allowing implicit device transfers, while TensorFlow does allow that.

Lastly, our indexed array accesses in our main codebase (the parts that we hope to eventually port) look like the following:

That's super useful, thanks. I think there's work in progress to propose adding single-integer-array indexing to the specification, which I think address the most common gap you're seeing.

Thanks for the detailed feedback! I agree with the overall assessment.

This part surprised me. I'd expect that converting between array types per function call will be quite expensive, introducing both overhead and possible bugs. Do you have some examples for functions where you do or don't allow this?

Yes, and to be more clear, for the typical few-liner subroutine it would be too much overhead -- the places where we do that mainly involve iterative solvers that can take 10s of seconds (or more) to run on large data, where the conversion overhead is small. Some examples of that are robust estimators, large-scale least-squares interpolation, machine learning code, or costly sliding-window ops on time series (so the analogy would be the typical high-level scikit-learn or statsmodels method). Another case is with workflows where only the few most expensive steps run on the GPU, meaning that a conversion happens anyway, and then one may as well pick the fastest backend.

So for absolute just making changes in your code base seems okay?

Yes for sure -- those are quick search-and-replace changes. We're also fine with putting light wrappers around namespaces to fix up small gaps like those where needed, and have been doing that all along (that may be necessary if one wishes to have side-by-side support for backends that have and have not already transitioned to the array API). Overall I agree that concise and uniform naming seems preferable.

Re moveaxis/swapaxes: I checked, and those are by now available in TF (move, swap) and PyTorch (move, swap). I fear I won't have much time to spin out separate issues and follow up on them at this point (my apologies!), but just having those on the collective radar is hopefully a good start.

Re minimum / maximum / clip: right, and another reason would be that common GPUs have dedicated/efficient support for some typical element-wise clipping (saturating) ops (e.g., nVidia, AMD; it's my understanding that they can even be zero-cost), and it'd be nice if the API didn't leave that silicon unused. Of course it wouldn't be guaranteed that those strength were leveraged by any actual implementations (though one may hope that XLA et al. would have an easier time doing a good job when presented with an easy-to-transform expression compared to nested where clauses).

can we popularize/adopt einops as a better API for these sort of manipulations? data-apis/consortium-feedback#3

My two cents are that having those added to the API seems appealing for newly written code, at least in places where it makes numpy easier to use or follow (requires some case-by-case judgment). When porting existing numpy code, I'd ideally be able to search and replace np. by xp. for the most part without much rethinking/reengineering. Likewise when reviewing a PR that does such a port, it's a much lower burden if it's a light syntactic change (especially when unit tests are incomplete).
Thanks for addressing the lstsq concern. As long as there's awareness that there's users like us out there who more or less depend on it, we should be covered for now (and presumably more people would come out of the woodwork if that ever broke in existing implementations).

We went back and forth on that quite a bit. I think we need to see this work in practice for a bit during adoption of the standard in array-consuming libraries. There were certainly arguments for adopting a context manager. One pain point I remember is that semantics were hard to agree on - for example PyTorch has a rule of never allowing implicit device transfers, while TensorFlow does allow that.

Re context managers -- yes, those look like a candidate for a later extension (seems like a potential tar pit that could slow down standardization). After all, there's sort of an escape hatch that allows users to retrofit one that controls the device= argument of the creation functions if they need it (using function wrappers), and that may even be the most sensible scope for that. I could imagine that someone might develop a utility library around the array API that fills in some bits like that, and otherwise broadens the numpy compatibility by implementing more common aliases for ops etc. (easier to do things like that in a take-it-or-leave-it library than in a standard).

That's super useful, thanks. I think there's work in progress to propose adding single-integer-array indexing to the specification, which I think address the most common gap you're seeing.

Yeah, and I'm not even sure all our single integer array indexing use cases are all that relevant (oftentimes we could as well use a 1-element slice). What's more important to us is actually that implementations continue to preserve their numpy-style view semantics and write-through capabilities that they already have (and one may hope that more implementations achieve it in case it's low-hanging fruit for them).

Hi @chkothe, thanks for your very detailed report! It's valuable and as a CuPy contributor I'm happy to hear both CuPy and the array API help your work.
Sorry to digress here, though, as I was hoping to discuss this offline but couldn't find your contact. Could you kindly share why you need eig() instead of eigh()? What's your use cases? On the CUDA math library side (which I am part of), this need (ex: https://github.com/cupy/cupy/issues/3255) was raised multiple times internally, but the cuSOLVER team considered the routine that can back eig() both mathematically ill-defined and much less used in the wild. If you can help us justify/motivate, it would be very nice (so we can unblock you and many other libraries like CuPy) 🙂

Thanks for following up on that! Emailed you.

sqrtm (and related functions like expm, logm, etc.) are very hard to implement, so I'm fairly certain we don't want that one in the standard.

These are indeed tricky to implement, but probably not significantly harder than matrix factorization? Yes, they are in SciPy rather than NumPy, but that's a somewhat artifcial distinction. Many (not all) of these can be found in PyTorch, JAX and TensorFlow.
My two cents is that it would be valuable if the standard specified the interface for these functions, even if not every library is going to implement them. It's really not a big deal to need to look up a compatibility table for advanced linear algebra functionality.
The other option is to encourage people to write their own shims for specific backends, which is not terrible but goes a little against the spirit of the array standard.

linalg is already an optional extension in the standard. Maybe it would also be helpful to state that individual functions in an extension might not be implemented?

linalg is already an optional extension in the standard. Maybe it would also be helpful to state that individual functions in an extension might not be implemented?

Yes, definitely!

Maybe it would also be helpful to state that individual functions in an extension might not be implemented?

Hmm, I'm not so sure I agree with that. Unfortunately https://data-apis.org/array-api/latest/extensions/index.html does not explain this yet - we should fix that. I had a look through older discussions, and it wasn't conclusive resolved if this is okay or not. What was resolved is that an extension is optional. My working assumption was that if the extension is implemented, it is complete. If every single function in an extension is optional, then this will be very tricky to use from a user perspective. Checking if __array_namespace__.linalg exists is a lot easier (also to document by a downstream library as required yes/no).

My two cents is that it would be valuable if the standard specified the interface for these functions, even if not every library is going to implement them. It's really not a big deal to need to look up a compatibility table for advanced linear algebra functionality.

I had a look at expm, which is probably the most widely used one of this set of functions:

https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.expm.html
https://jax.readthedocs.io/en/latest/_autosummary/jax.scipy.linalg.expm.html
https://pytorch.org/docs/stable/generated/torch.linalg.matrix_exp.html
https://www.tensorflow.org/api_docs/python/tf/linalg/expm
CuPy, Dask and MXNet don't have an implementation

For the four libraries that do have an implementation, the signatures don't match. So it's also a matter of effort whether we'd like to do this. Maybe rather than adding it to the linalg extension, there should be a separate list of functions for which folks want to align on an interface, but it's not part of the standard? Not sure exactly how that should look like (separate html docs section, some qualifier/status for an API, or ...), but I'd be keen on not adding so much that it becomes ever harder to be compliant.

clip is fairly heavily used and the use of where to emulate it is a bit cumbersome - this may be worth splitting off as a separate issue to discuss standardizing it (or have a good reason for why not).

Raised in issue ( https://github.com/data-apis/array-api/issues/482 )


Fix dependencies

Move dependencies that are also peer dependencies to dev dependencies

Fixes #140
Fixes #155

https://github.com/GR34SE/react-switch-selector/commit/7a595868200290767e8f8cbd72c77405ece6ea9e


Client.Timeout or context cancellation while reading body

https://github.com/rakyll/openai-go/blob/3c4e9fb5a90611ce64db2d75da85345abc560960/openai.go#L40
Can we increase the timeout period or set the environmental variable? Sometimes there will be a timeout when encountering extensive data.
Failed to complete: context deadline exceeded (Client.Timeout or context cancellation while reading body)

You can provide a custom timeout by using a custom HTTPClient.
sess := openai.NewSession(....)
sess.HTTPClient = &http.Client{
         Timeout: 120 * time.Second,
}


Normalize test directories

The test package directories didn't match the package names which goes
against the best practice so this has been fixed.
No functional changes.

🤖 AutoRest automatic publish job 🤖
success (version: 2.1.122)


Distribute xcframework in separate repo

As done in automerge-swift, the binary can be packaged and deployed into a separate GitHub repo, which helps to reduce the size of this repository

Resolved as of the 0.1.1 release - there's a release process detailing the setup, but the XCFramework is now hosted from GitHub releases, although not from a separate repository.


Public certs are only available at the app level and not webspace

This page contains the following note which is not entirely accurate:
Note
After you upload a certificate to an app, the certificate is stored in a deployment unit that's bound to the App Service plan's resource group, region, and operating system combination, internally called a webspace. That way, the certificate is accessible to other apps in the same resource group and region combination. Certificates uploaded or imported to App Service are shared with App Services in the same deployment unit.
This only applies to private (.pfx) certificates.  Public (.cer) certificates must be uploaded to each individual web app that needs access - so you may have many multiples of the same cert uploaded on the webspace.  Can we get this clarified on the page as it has led to some customer confusion.
Thanks.
Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: cc2ee874-df45-1de2-1b30-1fd75c7fd709
Version Independent ID: ee181722-8386-9842-407f-d0549012d2e9
Content: Add and manage TLS/SSL certificates - Azure App Service
Content Source: articles/app-service/configure-ssl-certificate.md
Service: app-service
GitHub Login: @msangapu-msft
Microsoft Alias: msangapu

#please-close


Looker API session times out after an hour

Hi everyone
I've got ldeploy content export working locally, but I'm trying to export content from a large instance, which takes a long time.
By default, the Looker API sessions on the SDK only last one hour, after which point I hit an authentication error.
Is there any way to configure the set up so that it automatically reauthenticates?

Hi everyone
I've got ldeploy content export working locally, but I'm trying to export content from a large instance, which takes a long time.
By default, the Looker API sessions on the SDK only last one hour, after which point I hit an authentication error.
Is there any way to configure the set up so that it automatically reauthenticates?

I am facing the similar issue.

Needs a PR to introduce logic for obtaining new access tokens when expiration is breached before attempting further requests.


Introduce ActionMap to Terminal Settings Model

Summary of the Pull Request
This entirely removes KeyMapping from the settings model, and builds on the work done in #9543 to consolidate all actions (key bindings and commands) into a unified data structure (ActionMap).
References
#9428 - Spec
#6900 - Actions page
Closes #7441
Detailed Description of the Pull Request / Additional comments
The important thing here is to remember that we're shifting our philosophy of how to interact/represent actions. Prior to this, the actions arrays in the JSON would be deserialized twice: once for key bindings, and again for commands. By thinking of every entry in the relevant JSON as a Command, we can remove a lot of the context switching between working with a key binding vs a command palette item.
#9543 allows us to make that shift. Given the work in that PR, we can now deserialize all of the relevant information from each JSON action item. This allows us to simplify ActionMap::FromJson to simply iterate over each JSON action item, deserialize it, and add it to our ActionMap.
Internally, our ActionMap operates as discussed in #9428 by maintaining a _KeyMap that points to an action ID, and using that action ID to retrieve the Command from the _ActionMap. Adding actions to the ActionMap automatically accounts for name/key-chord collisions. A NameMap can be constructed when requested; this is for the Command Palette.
Querying the ActionMap is fairly straightforward. Helper functions were needed to be able to distinguish an explicit unbinding vs the command not being found in the current layer. Internally, we store explicitly unbound names/key-chords as ShortcutAction::Invalid commands. However, we return nullptr when a query points to an unbound command. This is done to hide this complexity away from any caller.
The command palette still needs special handling for nested and iterable commands. Thankfully, the expansion of iterable commands is performed on an IMapView, so we can just expose NameMap as a consolidation of ActionMap's NameMap with its parents. The same can be said for exposing key chords in nested commands.
Validation Steps Performed
All local tests pass.

So close! I just have ONE failing test: DeserializationTests::TestRebindNestedCommand.
Summary
Test

✅ bind nested commands to "parent"
❌ rebind "parent" to a plain "newTab" command

Discussion
This should work. ActionMap is doing the following:

bind nested commands to "parent"

deserialize the command
it's a nested command!
add it to the action map

check if it's a nested command (it is!)
add it to _NestedCommands

when name map is requested...

go through non-nested commands (none)
go through nested commands...

"parent" -->  is added

rebind "parent" to a plain "newTab" command

deserialize the command
it's a "newTab" command!
add it to the action map

check if it's a nested command (it isn't!)
add it to the _ActionMap

when name map is requested...

go through non-nested commands (none)
go through parents (I SEE THAT THIS IS THE PROBLEM NOW)
add nested commands

Ok, I see what the problem is now. Just had to think out loud for a bit. Looks like the order is wrong in _PopulateNameMap. I'll marinate on a fix and submit that tomorrow morning. But other than that (and a code format), we're good to go!

So close! I just have ONE failing test: DeserializationTests::TestRebindNestedCommand.

Fixed! I'm a bit annoyed by the fix but it works and it's not bad, tbh. Basically, I split up _PopulateNameMap into two functions:

_PopulateNameMapWithNestedCommands:

top-down approach. We're just adding the nested commands starting from the root and moving down until we get to our current layer.

_PopulateNameMapWithStandardCommands:

bottom-up approach. Start from the current layer, update a set of visited action IDs, then visit our parents and add any actions that we're missing.
This is so that we don't have to do a dance of figuring out the old name of a command when we have to update it (because we'd have to remove the old name, and add the new name).

An important assumption is that in the current layer, there is no overlap in names between the names of _NestedCommands and standard commands. We enforce this assumption via ActionMap::AddAction.
By taking a two-step approach to populating the name map, we are able to handle collisions across multiple layers.

New problem: new tab dropdown doesn't display key chords.
Seems to be a problem where GetKeyBindingForAction isn't able to find the matching action. We're just using HashProperty here, so I'm gonna have to write up some tests for the hashing to make sure we're handling all of that correctly.

New problem: new tab dropdown doesn't display key chords.

Fixed! Had to move HashProperty into its own file HashUtils.h and make them all inline (Leonard suggested constexpr as a way to make them inline explicitly). This gave me more space to create specializations for IReference, Color, and other types. Doing so allows me to make sure that we always get a consistent hash from two structs composed of the same data.
While fixing this, I ran into a problem with a deserialization test. This PR introduces new behavior for actions, but I think this is the correct behavior. It is as follows:
{ "name": "foo", "command": "copy" },
{ "name": "bar", "command": "copy" }

Before PR
After PR

NameMap keys
foo, bar
bar

I believe this is the desired behavior because every command should only have one name.
Using this behavior, we can fix #7441. In fact, #7441 needs this. Consider the following:
{ "name": "foo", "command": "copy" },
{ "name": "bar", "command": "copy" },
{ "name": null, "command": "copy" },

Before PR
After PR

NameMap keys
foo, bar
<none>

The value of a null name comes from being able to find the names that map to the command, and remove them from the name map. Thus, we can leverage internal action IDs (aka hashing) to accomplish this.

Bug: Actions page is missing default actions

I set ctrl+shift+w to unbound but it still shows up as the official closePane shortcut in the command palette.

ctrl+shift+w bug is now fixed.
Merged in and fixed the changes requiring global hotkeys.
Dismissing @zadjii-msft's review since there's been a good amount of changes (mostly due to consolidated actions).

This PR will not merge until #9428 merges


build on both 7.10 and 7.8 (WIP)

groovy

:watch:

Superseded by https://github.com/ambiata/icicle/pull/320


feat: rename 'Builder' section to 'Build tool'

So that it better reflects the intention behind the section.
Created as from comment to #130

Thx :)


parsedown markdown glitch: blockquotes

see this SO thread
I'm pretty sure that:

quote; line1
quote; line2

should be rendered in a <blockquote> as 2 distinct lines
(as in github and muut)

This discussion: https://github.com/erusev/parsedown/issues/226
suggested that setting breaksEnabled should do the trick.
This is done by Parsedown->setBreaksEnabled()
And indeed ParsedownGravTrait.php calls this function passing the auto_line_breaks parameter.
but it passes $defaults['auto_line_breaks']
I not sure, but dumping the value it seems that my

markdown:
auto_line_breaks: True
in the header ofd my page was not taken into account (could be another bug).
(still I have to see if changing the auto_line_breaks value in the settings makes Parsedown processes quotes correctly... without side-effects)

3rd bug, I added:
   markdown:
        auto_line_breaks: true

To the pages YAML section of the default user/config/system.yaml
I'd expect to override the "false" value from system/config/system.yaml
It does but.... other values of the same section, like auto_url_links become "undefined":

Whoops\Exception\ErrorException, …/system/src/Grav/Common/Markdown/ParsedownGravTrait.php41

ok, I can confirm that setting auto_line_breaks does the trick (hopefully without side-effect).
The issue is now "how to set auto_line_breaks?"

Overriding 1 value of the markdown section in user/config/system.yaml undefines all others
Overriding from the page's header does not seem to work
I hope a maintainer could close this one about Parsedown while tracking the 2 others about overriding markdown values.

It should work from the page header by using the syntax:
https://github.com/getgrav/grav/issues
===
title: Your Page
markdown: true
  extra: false
  auto_line_breaks: false
  auto_url_links: false
  escape_markup: false
  special_chars:
    '>': 'gt'
    '<': 'lt'
===

Your content

Can you confirm that is not working for you?

it's not working (ie: the values aren't taken into account)
Note: header is delimited by --- rather than ===

I'll take a look

fix confirmed
thx!


removed tex and man output

Fix https://github.com/votca/tools/issues/326
blocked by https://github.com/votca/xtp/pull/655
@junghans did I miss anything?

We will need to merge https://github.com/votca/xtp/pull/655 first as xtp is actually still using this functionality.

We will need to merge https://github.com/votca/xtp/pull/655 first as xtp is actually still using this functionality.

Wait for https://github.com/votca/votca/pull/593

Wait for https://github.com/votca/votca/pull/593

@votca-bot changelog: remove tex and man output from app class

@votca-bot changelog: remove tex and man output from app class

@votca-bot format

@votca-bot format


bean 'oauth2ClientContext' duplicated

i am a newbie in oauth2 , i find there is a issue when i want to implement credential grant type client.
log as below:
The bean 'oauth2ClientContext', defined in class path resource [org/springframework/boot/autoconfigure/security/oauth2/client/OAuth2RestOperationsConfiguration$SingletonScopedConfiguration.class], could not be registered. A bean with that name has already been defined in BeanDefinition defined in class path resource [org/springframework/security/oauth2/config/annotation/web/configuration/OAuth2ClientConfiguration$OAuth2ClientContextConfiguration.class] and overriding is disabled.
Action:
Consider renaming one of the beans or enabling overriding by setting spring.main.allow-bean-definition-overriding=true
TestForOauth2.zip
The sample code is very simple, i think this issue can be reproduced very easy.
when you set grant type =  client_credentials, it will trying to registry another oauth2ClientContext bean which is already there.
I know i can set overriding=true to avoid this exception. But i want to avoid to do that if possible, I am afraid bean overriding will produce other issues in future.
I think it is not so hard to just add a @ConditionalOnMissingBean there.
And finally i have to say i am not very confident that it is a issue, because i am new to use oauth2, maybe there is some inappropriate usages in my code. Please feel free to commit on this issue at any time.
thank you.

@loki8217 sorry to hear you are having trouble.
Have you already checked https://github.com/spring-projects/spring-security-oauth2-boot/issues/85#issuecomment-464084127 and https://github.com/spring-projects/spring-security-oauth2-boot/issues/51#issuecomment-428252825 to see if they address your issue?

@loki8217 I haven't heard back from you on my question.
If I still haven't heard from you by the end of the week, I'll close this ticket due to inactivity.

@loki8217 I haven't heard back from you on my question.
If I still haven't heard from you by the end of the week, I'll close this ticket due to inactivity.

I am sorry for my late response. I used some time to do research work on oauth2. This issue is caused by my inappropriate usage. It don't make any senses to use both enableOauthClient and client credential grant type.
Thank you very much.


Following enhancements for system diagnose

As a first initial PR, looks good to me. Following possible enhancements in later PR:

For system info, the list of arguments in the demo gif looks to be truncated due to the width of the terminal. Multi-line display might be more helpful.
For demo gif, it is clear that when everything is right, the command works pretty well. Let's make some fake broken scenarios, for example, let controller down, and show how the system-info will show.
For demo gif, another interesting thing is the upgrade process. What if the kubevela-controller is upgrading and there are two different pods running at the same time. Could the system info command show their differences?
For system info, showing CPU/Memory usage is cool, let's add the percentage usage if the resource limit is set. For example, memory: 456Mi (45.6%).
For system info, there are some logs could be potentially useful. But I haven't come up with the idea of how to handle the log information. Feel free to make discussions if you have any ideas on it.

As for the diagnose, similar to [2] above, let's make some bad situation and see how this command performs.
Originally posted by @Somefive in https://github.com/kubevela/kubevela/pull/4379#pullrequestreview-1054993716

refer to https://github.com/kubevela/kubevela/issues/3924

/cc @foursevenlove


2?

  PLEASE PLEASE PLEASE ANSWER QUICKLY. WE HAVE SUNK FAR TOO MUCH TIME INTO THIS AND WE NEED TO KNOW.

GOTO: https://github.com/dwyl/english-words/issues/77#issuecomment-627080383

I have never seen a github repo with so many sketchy people


F-Droid can't build

ref: https://gitlab.com/fdroid/fdroiddata/-/jobs/5362089442#L1173
/LE: fyi https://gitlab.com/fdroid/fdroiddata/-/commit/8bc2d4275aedf2e613f4faafab182d3f0a3dea94

Please reopen, now there's gms classes :(
I tried cleaning some more
      - sed -i -e '/com.google.gms/d' -e '/com.google.firebase/d' -e '/libs.google/d'
        -e '/libs.firebase/d' build.gradle.kts
      - sed -i -e '/libs.google.services/d' -e '/libs.firebase/d' ../build.gradle.kts
      - sed -i -e '/firebase/d' -e '/google-services/d' ../gradle/libs.versions.toml

but still
...
2023-10-27 15:39:52,542 DEBUG: Problem: found class 'com/google/android/gms/common/GooglePlayServicesIncorrectManifestValueException'
2023-10-27 15:39:52,544 DEBUG: Problem: found class 'com/google/android/gms/common/GooglePlayServicesMissingManifestValueException'
2023-10-27 15:39:52,545 DEBUG: Problem: found class 'com/google/android/gms/common/api/Status'
2023-10-27 15:39:52,545 DEBUG: Problem: found class 'com/google/android/gms/common/util/DynamiteApi'
2023-10-27 15:39:52,545 DEBUG: Problem: found class 'com/google/android/gms/common/GooglePlayServicesManifestException'
2023-10-27 15:39:52,546 DEBUG: Problem: found class 'com/google/android/gms/common/api/internal/LifecycleCallback'
2023-10-27 15:39:52,546 DEBUG: Problem: found class 'com/google/android/gms/common/api/Scope'
2023-10-27 15:39:52,547 DEBUG: Problem: found class 'com/google/android/gms/common/annotation/KeepName'
2023-10-27 15:39:52,548 DEBUG: Problem: found class 'com/google/android/gms/common/api/GoogleApiActivity'
2023-10-27 15:39:52,549 DEBUG: Problem: found class 'com/google/android/gms/dynamite/DynamiteModule'
2023-10-27 15:39:52,549 DEBUG: Problem: found class 'com/google/android/gms/common/internal/ReflectedParcelable'
2023-10-27 15:39:52,554 DEBUG: Problem: found class 'com/google/android/gms/auth/api/signin/GoogleSignInAccount'
2023-10-27 15:39:52,557 CRITICAL: Found 12 problems in build/com.zaneschepke.wireguardautotunnel/app/build/outputs/apk/fdroid/release/app-fdroid-release-unsigned.apk

Thank you. I will look into this

Which lib are you rebuilding exactly ? https://github.com/zaneschepke/wgtunnel/commit/2690ce29e12ae50ece65aef32d43bb869c5872a2
We need to rebuild the same for F-Droid: https://gitlab.com/fdroid/fdroiddata/-/jobs/6202532336#L1008

Which lib are you rebuilding exactly ? 2690ce2 how?
We need to rebuild the same for F-Droid: https://gitlab.com/fdroid/fdroiddata/-/jobs/6202532336#L1008
/LE: fyi https://gitlab.com/fdroid/fdroiddata/-/commit/a9fd00deb938436063bb9eaf1e317a2cc439c87a

I'm still in the process of getting the lib into Maven Central. The lib was failing to publish to jitpack because they do not support recursive cloning. This lib is currently being pulled into the app from here using GitHub Packages, but GitHub Packages ridiculously requires auth to get packages.
repositories {
        maven {
            name = "GitHubPackages"
            url = uri("https://maven.pkg.github.com/zaneschepke/wireguard-android")
            credentials {
                username = getLocalProperty(GITHUB_USER_VAR) ?: System.getenv(GITHUB_USER_VAR)
                password = getLocalProperty(GITHUB_TOKEN_VAR) ?: System.getenv(GITHUB_TOKEN_VAR)
            }
        }

ping us when it's live in Maven

ok, so looking at https://github.com/zaneschepke/wgtunnel/blob/3.3.8/gradle/libs.versions.toml#L85 I should find a tag 1.1.0 at https://github.com/zaneschepke/wireguard-android/tags ?
I'd like to build your fork https://github.com/zaneschepke/wgtunnel/blob/3.3.8/README.md#building but I need a point of reference

how do we skip all the publishing part of your fork
  - versionName: 3.3.8
    versionCode: 33800
    commit: ee8db0a859778308ca584486a6d551a341b5b365
    subdir: app
    sudo:
      - apt-get update
      - apt-get install -y gcc g++ maven
      - apt-get install -y openjdk-17-jdk-headless
      - update-alternatives --auto java
    gradle:
      - fdroid
    srclibs:
      - wireguard-android-zaneschepke@161b73a35e3a92d9e58eb807b4e633d5c43f0825
    prebuild:
      - sed -i -e '/com.google.gms/d' -e '/com.google.firebase/d' -e '/libs.androidx.core.google.shortcuts/d'
        build.gradle.kts
      - sed -i -e '/libs.google.services/d' -e '/libs.firebase/d' ../build.gradle.kts
      - sed -i -e '/maven {/,/^        }/d' ../settings.gradle.kts $$wireguard-android-zaneschepke$$/tunnel/build.gradle.kts
    build:
      - cd $$wireguard-android-zaneschepke$$
      - gradle publishToMavenLocal

fails with
...
> Task :tunnel:javaDocReleaseJar
> Task :tunnel:generateMetadataFileForReleasePublication
Could not write standard input to command 'gpg'.
java.io.IOException: Broken pipe
        at java.base/java.io.FileOutputStream.writeBytes(Native Method)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:349)
        at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
        at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
        at org.gradle.process.internal.streams.ExecOutputHandleRunner.writeBuffer(ExecOutputHandleRunner.java:98)
        at org.gradle.process.internal.streams.ExecOutputHandleRunner.forwardContent(ExecOutputHandleRunner.java:85)
        at org.gradle.process.internal.streams.ExecOutputHandleRunner.run(ExecOutputHandleRunner.java:64)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

> Task :tunnel:signReleasePublication FAILED
gpg: directory '/home/vagrant/.gnupg' created
gpg: keybox '/home/vagrant/.gnupg/pubring.kbx' created
gpg: no default secret key: No secret key
gpg: signing failed: No secret key

FAILURE: Build failed with an exception.

I think instead of building the forked lib it would be easier to just add these env vars to the pipeline
      GH_USER: ${{ secrets.GH_USER }}
      GH_TOKEN: ${{ secrets.GH_TOKEN }}

The lib is already built and published to github packages, it is just stupid github requires read packages personal access token to get the package. Any valid github user can get the package with a valid read packages personal access token.
I should have the lib on maven central soon which will solve this issue.

F-Droid needs to build, since it's not (yet) in one on the trusted repos: https://gitlab.com/fdroid/fdroidserver/-/blob/master/fdroidserver/scanner.py#L457-L475
anyway, I'll stop trying and wait for your maven publish then, sorry for the noise

Package is available on Maven Central now. This should fix the build issue. Thank you for reaching out!

Done, thanks https://gitlab.com/fdroid/fdroiddata/-/commit/a8c5fc537d563518f86484cd15c8dcb5094003b0

Can we reopen?
...
> Task :app:checkFdroidReleaseAarMetadata FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:checkFdroidReleaseAarMetadata'.
> Could not resolve all files for configuration ':app:fdroidReleaseRuntimeClasspath'.
   > Could not find com.zaneschepke:pin_lock_compose:1.0.3.
     Searched in the following locations:
       - file:/home/vagrant/.m2/repository/com/zaneschepke/pin_lock_compose/1.0.3/pin_lock_compose-1.0.3.pom
       - https://dl.google.com/dl/android/maven2/com/zaneschepke/pin_lock_compose/1.0.3/pin_lock_compose-1.0.3.pom
       - https://repo.maven.apache.org/maven2/com/zaneschepke/pin_lock_compose/1.0.3/pin_lock_compose-1.0.3.pom
     Required by:
         project :app

ref: https://mvnrepository.com/artifact/com.zaneschepke does not have it
ref: https://github.com/zaneschepke/wgtunnel/commit/5946d7c10d195e6ddefe12b0666b64955130ae16#diff-5625e3601fa0ad3a6a2824239e5a2fde71c149597d31394f9224a08c24be7b9dR14

The lib has been added to maven central. The app should be able to build now.

thanks https://gitlab.com/fdroid/fdroiddata/-/commit/289525dd8979a90d19516a62757ce8fe7710e261

Hmm, do reopen
https://gitlab.com/fdroid/fdroiddata/-/jobs/7445363327#L1194

Hmm, do reopen
https://gitlab.com/fdroid/fdroiddata/-/jobs/7445363327#L1194
/LE: fyi https://gitlab.com/fdroid/fdroiddata/-/commit/f25bc60d5e4b438e516bc85bc50d20a25274c09f

https://gitlab.com/fdroid/fdroiddata/-/merge_requests/15494


Error opening programmer constraints file for writing

To reproduce create a CPOG work with two scenarios (e.g. the attached one) and run SCENCO tool (any mode, no ABC). A "Programmer error"  window appears saying "Error on opening constraints file for writing".

danilovesky at 2014-08-01T22:20:38Z:


test-IAM-policy-in-policy-generator

Signed-off-by: Chunxi Luo<EMAIL_ADDRESS>@gparvin

So as this PR, if we just put objectDefinition section in the policy manifest policy-limitclusteradmin.yaml and move the rest higher-level info into policyGenerator.yaml, then we got the below result in the output:

apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  annotations:
    policy.open-cluster-management.io/categories: AC Access Control
    policy.open-cluster-management.io/controls: AC-3 Access Enforcement
    policy.open-cluster-management.io/standards: NIST SP 800-53
  labels:
    open-cluster-management.io/policy-set: openshift-hardening
  name: policy-limitclusteradmin
  namespace: policies
spec:
  disabled: false
  policy-templates:
  - objectDefinition:
      apiVersion: policy.open-cluster-management.io/v1
      kind: ConfigurationPolicy
      metadata:
        name: policy-limitclusteradmin
      spec:
        object-templates:
        - complianceType: musthave
          objectDefinition:
            apiVersion: policy.open-cluster-management.io/v1
            kind: IamPolicy
            metadata:
              name: policy-limitclusteradmin-example
            spec:
              maxClusterRoleBindingUsers: 5
              namespaceSelector:
                exclude:
                - kube-*
                - openshift-*
                include:
                - '*'
              remediationAction: inform
              severity: medium
        remediationAction: inform
        severity: medium

Current policy-generator + policy-collection will get the below result in the output, we can find policy-generator just treat the policy manifest policy-limitclusteradmin.yaml  as an objectDefinition section, that is the reason of nested policy-templates. If we need to keep the policy manifest policy-limitclusteradmin.yaml  as current format, then we should add an additional boolean flag in the policy section of policyGenerator.yaml to let policy-generator subtract objectDefinition part from the policy manifest, rather than read the whole manifest as an objectDefinition section.

Feel free to let me know which direction we should go.
apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  annotations:
    policy.open-cluster-management.io/categories: CM Configuration Management
    policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
    policy.open-cluster-management.io/standards: NIST SP 800-53
  labels:
    open-cluster-management.io/policy-set: openshift-hardening
  name: policy-limitclusteradmin
  namespace: policies
spec:
  disabled: false
  policy-templates:
  - objectDefinition:
      apiVersion: policy.open-cluster-management.io/v1
      kind: ConfigurationPolicy
      metadata:
        name: policy-limitclusteradmin
      spec:
        object-templates:
        - complianceType: musthave
          objectDefinition:
            apiVersion: policy.open-cluster-management.io/v1
            kind: Policy
            metadata:
              annotations:
                policy.open-cluster-management.io/categories: AC Access Control
                policy.open-cluster-management.io/controls: AC-3 Access Enforcement
                policy.open-cluster-management.io/standards: NIST SP 800-53
              name: policy-limitclusteradmin
            spec:
              disabled: false
              policy-templates:
              - objectDefinition:
                  apiVersion: policy.open-cluster-management.io/v1
                  kind: IamPolicy
                  metadata:
                    name: policy-limitclusteradmin-example
                  spec:
                    maxClusterRoleBindingUsers: 5
                    namespaceSelector:
                      exclude:
                      - kube-*
                      - openshift-*
                      include:
                      - '*'
                    remediationAction: inform
                    severity: medium
              remediationAction: inform
        remediationAction: enforce
        severity: medium

Having the config-policy-controller deploy an IamPolicy is convoluted (and probably wouldn't return the compliance status all the way back to the hub). The idea of the enhancement is to support policy types other than ConfigurationPolicy--this should be addressed by updating the generator code to support it.

Sorry if I'm misunderstanding your explanations. To me, I think there are a couple options that come to mind:

When a policy is given through policies[].manifests[].path in the generator manifest, if it's a Policy kind, extract the objectDefinition object and dump it into the policy (and probably validate first to make sure it's one of ConfigurationPolicy, IamPolicy, CertPolicy)
Create a new policies[].path that can validate and use a Policy kind as-is
I'm not sure there's much desire in allowing users to have an IamPolicy or CertPolicy manifest since I'm assuming those would probably already be wrapped in a Policy.

@mprahl @ycao56 Do you have any further feedback on this?

Sorry if I'm misunderstanding your explanations. To me, I think there are a couple options that come to mind:
* When a policy is given through `policies[].manifests[].path` in the generator manifest, if it's a Policy kind, extract the `objectDefinition` object and dump it into the policy (and probably validate first to make sure it's one of ConfigurationPolicy, IamPolicy, CertPolicy)

* Create a new `policies[].path` that can validate and use a Policy kind manifest as-is

I'm not sure there's much desire in allowing users to have an IamPolicy or CertPolicy manifest since I'm assuming those would probably already be wrapped in a Policy.

Both makes sense. The second approach might have higher priority if considering we are going to merge iam and cert policy into config policy.
Agree? @gparvin @dhaiducek @mprahl

My preference would be to have a new field under manifests called policyPath which can point to a YAML file of a non-root policy (e.g. ConfigurationPolicy, IAMPolicy, CertPolicy). Then the non-root policy is taken as is and added to the policy-templates array. There would then be validation to ensure that manifest[0].path and manifest[0].policyPath are not both set.
I believe this makes the user experience fairly consistent with the existing user experience.

I agree, @mprahl. I guess the question is: do we want to enable users to take an existing parent Policy manifest and use it as-is through the generator? The policy-collection has been based on those (and customers probably have them also at this point). Then that policy can be used in a set, modify placement, etc (though it's not really being "generated" any more).

I agree, @mprahl. I guess the question is: do we want to enable users to take an existing parent Policy manifest and use it as-is through the generator? The policy-collection has been based on those (and customers probably have them also at this point). Then that policy can be used in a set, modify placement, etc (though it's not really being "generated" any more).

It's a good point. I think we don't want to support that since the Policy Generator can generate policy sets that contains policies that are not generated by the Policy Generator. In my opinion, any root policy not generated by the Policy Generator should be deployed outside of the Policy Generator.

the Policy Generator can already generate policy sets that contains policies that are not generated by the Policy Generator

It can? I think I missed that...

Oh! You just provide the name of the policy. Sorry--yes, in that case I 100% agree.

Thanks for the clarification. Close this issue and open new PR based on feedback.

@dhaiducek @mprahl @gparvin
So for Gus user case here
https://github.com/stolostron/backlog/issues/19466#issuecomment-1047886710
https://github.com/stolostron/policy-collection/blob/main/policygenerator/policy-sets/community/openshift-hardening/input-admin/policy-limitclusteradmin.yaml
It seems like root policy which should be directly added here
https://github.com/stolostron/policy-generator-plugin/blob/7f358d41b64d2295b44756dd11d27cf8dff828e4/internal/plugin.go#L839
rather than put into policyTemplates
Just confirm we will not support this case.

closed as we have decided to not support the root/full policy in the generator


tf doc face model convert to caffe got empty model

Platform (like ubuntu 16.04/win10):
ubuntu 16.04
Python version:
3
Source framework with version (like Tensorflow 1.4.1 with GPU):
tensorflow 1.2 cpu
Destination framework with version (like CNTK 2.3 with GPU):
caffe cpu
Pre-trained model path (webpath or webdisk path):
https://drive.google.com/open?id=1nBmggq4uixcI2AtucGu5nsa28dJRwkIz
https://pan.baidu.com/s/1RadsPNjcdzBDYcdoN5BAhQ
Running scripts:
mmconvert -sf tensorflow -in ./ckpt-320000.meta -iw ./ckpt-320000 --dstNodeName keep_prob -df caffe -om doc_face

I am getting empty caffemodel and prototxt as well. No error is shown.
Platform (like ubuntu 16.04):
ubuntu 16.04
Python version: 2.7
Source framework with version (like Tensorflow 1.5.0 with GPU):
Destination framework with version (caffe):
caffe cpu
Pre-trained model path (webpath or webdisk path):
https://drive.google.com/file/d/1mvxA9r9K1sydBEfVWk090f4Pdhg15YQD/view
Trrying to convert re3 from tensorflow to caffe.
re3 model
Running scripts:
mmconvert -sf tensorflow -in model.ckpt-260946.meta -iw model.ckpt-260946 --dstNodeName re3/lstm2/rnn/lstm_cell/w_o_diag -df caffe -om tf_re3
Output:
Parse file [model.ckpt-260946.meta] with binary format successfully.
Tensorflow model file [model.ckpt-260946.meta] loaded successfully.
Tensorflow checkpoint file [model.ckpt-260946] loaded successfully. [102] variables loaded.
IR network structure is saved as [263dfc99a4d54114a0604081af2552c7.json].
IR network structure is saved as [263dfc99a4d54114a0604081af2552c7.pb].
IR weights are saved as [263dfc99a4d54114a0604081af2552c7.npy].
Parse file [263dfc99a4d54114a0604081af2552c7.pb] with binary format successfully.
Target network code snippet is saved as [263dfc99a4d54114a0604081af2552c7.py].
Target weights are saved as [263dfc99a4d54114a0604081af2552c7.npy].
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1221 12:13:33.664634 5082 net.cpp:53] Initializing net from parameters:
state {
phase: TRAIN
level: 0
}
I1221 12:13:33.664669 5082 net.cpp:257] Network initialization done.
W1221 12:13:33.664810 5082 _caffe.cpp:139] DEPRECATION WARNING - deprecated use of Python interface
W1221 12:13:33.664818 5082 _caffe.cpp:140] Use this instead (with the named "weights" parameter):
W1221 12:13:33.664822 5082 _caffe.cpp:142] Net('tf_re3.prototxt', 1, weights='tf_re3.caffemodel')
I1221 12:13:33.664860 5082 net.cpp:53] Initializing net from parameters:
state {
phase: TEST
level: 0
}
I1221 12:13:33.664867 5082 net.cpp:257] Network initialization done.
Caffe model files are saved as [tf_re3.prototxt] and [tf_re3.caffemodel], generated by [263dfc99a4d54114a0604081af2552c7.py] and [263dfc99a4d54114a0604081af2552c7.npy].

my output is almost the same as yours

i found the reason. some ops are not supported yet by the mmdnn


Add redis-namespace dependency

so that the sidekiq redis connection client builder does not silently explode.
Sadly, this Sidekiq.logger.error message doesn't make it to log output in time
to be helpful :crying_cat_face: (on JRuby <IP_ADDRESS>, at least)
https://github.com/mperham/sidekiq/blob/0f285f05e4c0e7969a0b8ccb8e736608dcebfa6a/lib/sidekiq/redis_connection.rb#L47-L48

Maybe $stdout.sync = true can make that appear?
Can you add a bit more context here as to what redis-namespace does and how it fixes the issue?

iiuc redis-namespace is a lazy dependency, about which the log message is meant to politely inform, but we never got the log message, and instead I had to find out by bisecting with $stderr.puts :tada:

@meatballhat could it be that it was still in the flush buffer and $stderr.flush would make it show? if this is the case that would be a nice patch for sidekiq upstream. Could you perhaps see if you can make the message show?


Add soundfile compatibility backend

As a part of the "sox" backend sunset plan (#903), we add a "soundfile" backend that is compatible with the "sox_io" backend.
This PR contains;

The implementation of the new "soundfile" backend.
The function and global flag to switch the behavior of "soundfile" backend.
No new public backend is added. We provide a switch to change the interface/behavior of "soundfile" backend.
Test for the new backend.

The default behavior of "soundfile" backend is not changed. The users who want to opt-in the new "soundfile" interface can do so by torchaudio.backend.utils.use_soundfile_legacy_interface(False)
torchaudio.backend.utils.use_soundfile_legacy_interface = False

before changing the backend to "soundfile".
In 0.8.0 release, the "soundfile" backend will use this interface by default, and users can still use the legacy one with torchaudio.backend.utils.use_soundfile_legacy_interface(True). In 0.9.0, the legacy interface is removed and use_soundfile_legacy_interface function becomes no-op, (and eventually removed from the code base.)
TODO:

[ ] Update #903

In 0.8.0 release, the "soundfile" backend will use this interface by default, and users can still use the legacy one with torchaudio.backend.utils.use_soundfile_legacy_interface(True). In 0.9.0, the legacy interface is removed and use_soundfile_legacy_interface function becomes no-op, (and eventually removed from the code base.)

I'll edit the description to reflect the correct interface:
torchaudio.backend.utils.use_soundfile_legacy_interface = False

In 0.8.0 release, the "soundfile" backend will use this interface by default, and users can still use the legacy one with torchaudio.backend.utils.use_soundfile_legacy_interface(True). In 0.9.0, the legacy interface is removed and use_soundfile_legacy_interface function becomes no-op, (and eventually removed from the code base.)

I'll edit the description to reflect the correct interface:
torchaudio.backend.utils.use_soundfile_legacy_interface = False

It's torchaudio.USE_SOUNDFILE_LEGACY_INTERFACE.