[Feature][API] REST API

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
Is there a roadmap for supporting REST API?
Use case
I would like to use an external workflow automator in conjunction with Dolphin to initiate work on a specific usecase and then return the result to the workflow for further processing.
Related issues
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Do you mean integrating workflow related pages into other systems?

May PyDolphinScheduler will help you.


PointCloud XYZRGBL RGB8 colour transformer does not show cloud

I have a pointcloud with XYZRGBL fields generated in PCL, whose display does not seem to work correctly in rviz. If I view the cloud in pcl_viewer I can see the following colour handlers:
List of available color handlers for actor out.pcd-0: [random](1) x(2) y(3) z(4) rgb(5) label(6)

The cloud (attached below) has specific RGB values for each label and is received as a PointCloud2 type. However, when displaying this cloud in rviz, if the RGB8 colour transformer is selected, nothing appears.  This is also true for the MONO8 transformer.
Selecting any of the other colour transformers shows that there are indeed points coming through. With the intensity transformer, using the label channel correctly displays the cloud, with different label regions having different colours. However, the rgb values given by the label display are different to the rgb values given in the original instantiation of the labelled cloud, so if the user has set specific rgb values for each point with a certain label, they will not be displayed with this method.
This is in rviz 1.12.13 (also on 1.12.16), on Ubuntu 16.04. I'd like to help fix this but there are no error or warning messages which I can use to point me to where the problem might lie.
You can check this by starting a roscore, running rviz, adding a PointCloud2 display subscribed to cloud_pcd, and then running
rosrun pcl_ros pcd_to_pointcloud out.pcd 1

With the following pcd file:
out.pcd.zip
You will be able to see messages coming in, but because the default transformer is rgb8, you will not see anything.

having same issue after 2 years

@jediofgever, you are cordially invited to debug this issue and provide a fixing PR.


Axios and webpack are not working properly

We have an issue with axios which is bundled using webpack.
We are importing axios with import axios from 'axios' so nothing unusual. In attached screenshot you can see part of a source code which shows xhr adapter initialization. The problem is that function does not take config as entry point, instead it is using some y magic variable (y.data, y.headers and etc).
Any ideas where is the problem here? We are using axios v0.15.3 and webpack 1.14.0.

have you resolve it now ? i get this issue now , what can I do?

We can close the issue now, we figured it out. The problem was with build step - our code was minified twice and that caused the problem.


Implement bit slice logic in Compile.hs

This is a little tricky, as we use different representations depending on the size of a bit vector (BitUint or uXX) depending on its width. Bit slicing may involve conversion between these representations

Bit vector concatenation is also not implemented for the same reason.

Implemented in bc343f7073b629aa3019cde34c4b32ce6e04624c


Update data-lake-analytics-documentation-backlog.md

Hi @mwinkle, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

It looks like you're working at Microsoft (mwinkle). If you're full-time, we DON'T require a contribution license agreement.

If you are a vendor, DO please sign the electronic contribution license agreement. It will take 2 minutes and there's no faxing! https://cla.azure.com.

TTYL, AZPRBOT;

@mwinkle
Thanks, Matt!


Httpfs add

testCreateFileAsUser() kept failing quite reliably for me. I removed that one test and committed the rest. Feel free to debug that test and submit a new pull request.


Add check command

To check the project against current conventions and best practices. Why current and not "current for the version of SnarkyJS used"...to encourage consistency in the ecosystem.
Codify the current conventions and automatically check the current dir for:

[ ] expected dir structure & files
[ ] .prettierrc exists & follows current format (and code passes?)
[ ] lint config file exists & follows current rules (and code passes?)
[ ] package-lock.json exists in Git (b/c it's required by the npm ci command used in CI templates)
[ ] tests exist (and pass?)
[ ] test coverage above x%? or function call coverage at 100%?

Could add a CI file for this too. Write this in JS and it can be used by both the CLI & CI workflow.
For failures, we might want to give a link to a conventions.md (to the proper section header within it) which explains 1.) the expected convention, and 2.) why.

[ ] Add --fixor as a separate command to automatically update the user's project to current recommendations for Prettier/Eslint/etc configs, ignore files, etc.

To discuss. We may not take this approach

Closing to clean up. Not realistic that we prioritize this anytime soon. Need for it hasn't been seen in practice yet


Update UserDefaults.swift

And even the build failed because of this change

I never tried to build it, man. It's just that the var felt unnecessary. Best of luck in your development ☭


Make opening tabs more snappy

http://www.giphy.com/gifs/3o7bu7UxuUMTIn5xNm

Quick usage note:

Open a new tab
Type 'test' + ENTER to go to search page
Press back - end up with a neverending 'connecting...' title + animation in the tab

It should probably disable the back button or at least not set the UI as if it's trying to load a page

@bgrins

Quick usage note:

Open a new tab
Type 'test' + ENTER to go to search page
Press back - end up with a neverending 'connecting...' title + animation in the tab

It should probably disable the back button or at least not set the UI as if it's trying to load a page

Good catch! I'll fix this.

@bgrins Addressed your comments in the "Post:" commits.


How can I run tslint for entire directory on Windows?

Hi,
I have a directory structure like:
./checkout/src/random_dir/.ts
./checkout/test/random_dir/.ts
I'm trying to run tslint on Windows with './checkout_TEST/{,**/}*.ts' as an argument (and other combinations), but unless I specifically run it for test dir, it always seems to pick up only .ts files in src directory. It totally ignores test.
Anyone has any pointers?
Thanks!

The CLI accepts standard node globs as strings. You should be able to use tslint './checkout/**/*.ts'.

Hello,
unfortunately, it does not work - that was one of the first things I tried. When I do tslint './checkout/**/*.ts'  it again finds everything in the src directory (recursively), but nothing in test folder at all.
When I do tslint './checkout/test/**/*.ts' , it find issues in test directory but nothing at all in src directory.
I have also tried tslint './checkout/**/**/*.ts'  - which again finds only issues in src directory but not in test.
No other configuration is changed between the runs.
Thanks in advance for any pointers, these would be very appreciated. Please let me know if providing further details about the directory structure and/or output files would help.
Eva

Hi Eva,
This seems like the same issue described here in #1091. Single quotes aren't recognized by cmd.exe, so that's most likely what's causing you issues. You could try switching to PowerShell, which I believe allows the use of single quotes as quotes. Hope this helps!

Hi JKillian,
unfortunately I already run this from powershell :( (using Invoke-Expression)
The exact command being executed from a powershell script is:
tslint './checkout_TEST/**/**/*.ts' --format json --config ./tslint.json --out TsLint_out_TEST_1.0.22779.json
Eva

I'm not sure how the command-line parsing library we use works, but I'd try specifying the name of the file  last:
tslint --format json --config ./tslint.json --out TsLint_out_TEST_1.0.22779.jso './checkout_TEST/**/*.ts'

I tried that, does not help :( Only issues from src directory are found.

Hi,
can anyone help with this?

Make sure you include single quotes when specifying the glob pattern at the command line / terminal. This is implied in the previous comments but wasn't obvious for me at first.


Main class not found

2020-04-19 [22:30:39] [Server thread/ERROR]: Main class for plugin RPGLike not found

Just fixed!
Please have in mind that it's still a WIP


No craft recipe for portable batteries and charger in book

But it showing this page

Just realized this last night. I'll fix it next update.
Until then you can of course use the wiki to get the recipes.

Keeping the issue open so I don't forget about it.


feature(reset): refactor the reset service to reset the cluster state completely

What type of PR is this?
/area simulator
/kind feature
/kind cleanup

What this PR does / why we need it:
Currently, reset service tries to remove all supported resources.
But, those who use other resources want to reset the cluster completely; including all unsupported resources.
(+ Especially, before running Scenario, we wanna clean up all resources to make sure no resources that affect scheduling in the cluster.)

And, besides those enhancements, we also can remove DeleteCollection method from all resource services.
So, it partly contributes to https://github.com/kubernetes-sigs/kube-scheduler-simulator/issues/139.
Which issue(s) this PR fixes:

Fixes #218
Special notes for your reviewer:
/label tide/merge-method-squash

@196Ikuchil rebased. 🙏


Electron 126

Fixes user config issues and supports the latest spec.
@lneir @VikasShashidhar Please review and merge.

@lneir @VikasShashidhar Please review some more changes I've made to accommodate our requirements.
CC: @KiranNiranjan

@lneir I've made the necessary changes. Please review.
I've also changed all the callbacks to follow the es6 arrow syntax in the config.js file to keep it consistent across the board.

lgtm


Programatically control the visibility of configurations/settings

At the moment, we can dynamically control almost every UI contribution declared in package.json using  a when clause, like commands, views, menus, and welcome. A glaring exception are the configuration contributions.
This request is to support a when attribute on configuration properties to control whether or not they are visible on the settings editors, i.e., if the property shows up on the visual setting editor and on the content assist for the associated JSON editor.
Ideally, other than visibility, the behavior  of "hidden" configurations should be the same as "non-hidden" configurations, including notification, manipulation via the workspace.getConfiguration API, and "when clause" support via the config. prefix.
There are several reasons for this request:

Support the "feature-flag" pattern.

With clever usage of non-declared configurations and custom context keys, we can dynamic hide and show the UI contributions of features that we only want specific users (beta testers, design partners, ...) to see. Unfortunately there is no way to do that with the configurations associated with these features, so we have to either declare and show the configurations or not declare them - the former makes all users aware of the special features while the latter hinders the experience of users who know how to enable such features.

Support configurations that depend on other configurations

Because the when clause can refer to other configurations, we could then control the visibility of dependent configurations.

Support configurations that are environment specific.

Similar to number 2, but hiding features that are not applicable to the current environment - OS, desktop vs browser, VSCode version, ...

Completeness

The when clause is extremely powerful, well documented, and widely used in VSCode. At least to my naive eyes, I can't see a reason not to support it for configurations.
This feature request has been already suggested at least once:
https://github.com/microsoft/vscode/issues/130500

This feature request is now a candidate for our backlog. The community has 60 days to upvote the issue. If it receives 20 upvotes we will move it to our backlog. If not, we will close it. To learn more about how we handle feature requests, please see our documentation.
Happy Coding!

:slightly_smiling_face: This feature request received a sufficient number of community upvotes and we moved it to our backlog. To learn more about how we handle feature requests, please see our documentation.
Happy Coding!

We closed this issue because we don't plan to address it in the foreseeable future. If you disagree and feel that this issue is crucial: we are happy to listen and to reconsider.
If you wonder what we are up to, please see our roadmap and issue reporting guidelines.
Thanks for your understanding, and happy coding!

Hi, I'm confused. Was this moved to the backlog or was it dismissed? Thanks for your help.

Hi,
In 8 days this this feature request got 40 upvotes from the community. Moreover, some of the "upvoters" reached out to me to mention that they are found of at least one of the reasons I've mentioned on the description - and all agree particularly with the "completeness" one.
Would it be possible to understand the reasoning for closing a clearly desired feature?
Perhaps more importantly, is there anything I can do to help opening it again? Gather more votes? Try to implement it myself?
Thank you so much for the great work on VSCode. The only reason I am insisting here is because I really like what you are doing - I wouldn't care if I didn't ;-)

Very much appreciated for creating the feature request and providing the proper use case. The reason for closing this issue is that (as mentioned in the above comment), it is unlikely that we will plan and work on this feature in near future due to our priorities. In such case, we would like to keep this closed as out of scope instead of having this issue open and thereby giving false promises. It can also happen that priorities might change and there can be more traction to this feature and we would definitely come back and address it. Thanks for understanding.


strings.TrimRight unexpected output

What version of Go are you using (go version)?

$ go version
go version go1.15 linux/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env

What did you do?

https://play.golang.org/p/N87az0K_2by
package main

import (
	"fmt"
	"strings"
)

func main() {
<EMAIL_ADDRESS>"@xx.com"))

}

What did you expect to see?
abc
What did you see instead?
ab

In TrimRight, the second parameter is a cutset: TrimRight(s string, cutset string), i.e. a set of individual characters to be trimmed.
Passing<EMAIL_ADDRESS>doesn't mean "trim right at @xx.com", it means "trim all @ x, c, o, and m characters from the right". That's why your c gets trimmed, too.
Working as intended, closing this.

By the way, you might be looking for strings.TrimSuffix.


fix: good redirect after agent public login

Correctif
Zones impactées : ./middleware.ts.
Détails :

The usePathServer hook has been removed.
login : The middleware is now responsible for adding the pathFrom param when navigating to the "espace agent" page
logout : referer is directly used from the route to determine the pathFrom param

#1220

I can't figure how to reproduce any other trouble with the button.

fix le hook coté server
nouveau layout du header avec uniquement le search et les logos
éviter les client component si possible

I see. Nice job spotting this bug.

Have you tested it extensively ?

Also, you talked about using the callback from OpenIDConnect? Have you investigated it ?

@rmonnier9 I love what you did with the referrer. Do you think you can replicate it in order to move this component from a client component using usePathname into a server component ?
https://github.com/annuaire-entreprises-data-gouv-fr/site/blob/6977405015ec45cf116ba9a61b5f646d997f1c0b/components-ui/button-pro-connect/index.tsx#L1

Also, you talked about using the callback from OpenIDConnect? Have you investigated it ?

Yes, but in fact, it would have been a way to replace the data we save in session cookie between the initial call and the callback.
And actually, I think it was something particular in an other API I was using.

@rmonnier9 I love what you did with the referrer. Do you think you can replicate it in order to move this component from a client component using usePathname into a server component ?
https://github.com/annuaire-entreprises-data-gouv-fr/site/blob/6977405015ec45cf116ba9a61b5f646d997f1c0b/components-ui/button-pro-connect/index.tsx#L1
There is a challenge :

button is either called from here https://github.com/annuaire-entreprises-data-gouv-fr/site/blob/6977405015ec45cf116ba9a61b5f646d997f1c0b/components/espace-agent-components/agent-wall/index.tsx#L1
or from the landing page

So:

When calling from AgentWall you want the pathFrom to be the current page
When callfing from LP you want pathFrom to be the referrer

Regarding the browser history, the list of previous pages is not accessible due to security reasons.
As an alternative, I found a solution: calling an intermediate API route that redirects to the correct page while adding the pathFrom parameter.

Becoming really nice. I think we can use an even simpler implem : pages/api/auth/agent-connect/redirect.ts  with the following ButtonProConnect :
const ButtonProConnect: React.FC<IProps> = ({
  shouldRedirectToReferer=false
  event = 'BTN_DEFAULT',
}) => {

	const currentPath = usePathname();
	// it seems we can access the referrer from client https://developer.mozilla.org/fr/docs/Web/API/Document/referrer
	const referrer = document?.referrer;
 	cont pathFrom = shouldRedirectToReferer ? referrer : currentPath;

 	return (
    <form action="/api/auth/agent-connect/login" method="get">
      {pathFrom && (
        <input readOnly hidden aria-hidden name="pathFrom" value={pathFrom} />
      )}
	...

/lp/agent-public?pa

Brilliant. It will be so simple with this !

Becoming really nice. I think we can use an even simpler implem : pages/api/auth/agent-connect/redirect.ts with the following ButtonProConnect :
const ButtonProConnect: React.FC<IProps> = ({
  shouldRedirectToReferer=false
  event = 'BTN_DEFAULT',
}) => {

	const currentPath = usePathname();
	// it seems we can access the referrer from client https://developer.mozilla.org/fr/docs/Web/API/Document/referrer
	const referrer = document?.referrer;
 	cont pathFrom = shouldRedirectToReferer ? referrer : currentPath;

 	return (
    <form action="/api/auth/agent-connect/login" method="get">
      {pathFrom && (
        <input readOnly hidden aria-hidden name="pathFrom" value={pathFrom} />
      )}
	...

With this implem we completly get rid of most pathFrom in url params. Ithink we should still offer the possibility to create a /lp/agent-public?pathFrom=XXX but it is no longer necessary to use it
Also, referrer has to be compared with your new BASE_URL to avoid redirect to a different origin

Brilliant ! I will be really really simple with this.

Test e2e fail looks like noise. The tests checks wether footer has any broken links. So we cannot moke this part. But the error is not relevant here. Looks like a temporary downtime from remote website


Update python client header

@geigerj
@bjwatson
@tbetbetbe

@saicheems Do you know why the build is breaking?

Forgot to update the baseline, should succeed after my push just now

Cool, thanks.

LGTM


Test failures on windows

$ systeminfo   | findstr /B /C:"OS Name" /C:"OS Version"
OS Name:                   Microsoft Windows 10 Enterprise
OS Version:                10.0.19042 N/A Build 19042
$ raku --version
Welcome to Rakudo™ v2022.02-96-g6d04cc395b.
Implementing the Raku® Programming Language v6.d.
Built on MoarVM version 2022.02-36-ga1a45e57e.
$ zef test .
===> Testing: Log::Timeline:ver<0.4>
Aborting due to test failure: Log::Timeline:ver<0.4> (use --force-test to override)
[Log::Timeline]     # Failed test 'Can make another connection to the server'
[Log::Timeline]     # at t\output-socket.t line 123
[Log::Timeline]     # connection refused
[Log::Timeline] A react block:
[Log::Timeline]   in block <unit> at t\output-socket.t line 128
[Log::Timeline] Died because of the exception:
[Log::Timeline]     An operation first awaited:
[Log::Timeline]       in block  at t\output-socket.t line 137
[Log::Timeline]       in sub subtest at C:\raku\share\perl6\core\sources\2D4A7CA10695CD2B374573413D261A660E282E2A (Test) line 437
[Log::Timeline]       in sub subtest at C:\raku\share\perl6\core\sources\2D4A7CA10695CD2B374573413D261A660E282E2A (Test) line 425
[Log::Timeline]       in block <unit> at t\output-socket.t line 122
[Log::Timeline]     Died with the exception:
[Log::Timeline]         Cannot write to a closed socket
[Log::Timeline]           in block  at t\output-socket.t line 137
[Log::Timeline]           in sub subtest at C:\raku\share\perl6\core\sources\2D4A7CA10695CD2B374573413D261A660E282E2A (Test) line 437
[Log::Timeline]           in sub subtest at C:\raku\share\perl6\core\sources\2D4A7CA10695CD2B374573413D261A660E282E2A (Test) line 425
[Log::Timeline]           in block <unit> at t\output-socket.t line 122
===> Testing [FAIL]: Log::Timeline:ver<0.4>

Here's a golfed case that fails on windows complaining about not being able to write to a closed socket:
use Log::Timeline;

my constant $TEST_PORT = 19893;
PROCESS::<$LOG-TIMELINE-OUTPUT> = Log::Timeline::Output::Socket.new(port => $TEST_PORT);

my $conn;
$conn = await IO::Socket::Async.connect('localhost', $TEST_PORT);
dd $conn.WHERE;
$conn.close;

# THIS BLOCK IS NEEDED
{ $conn = await IO::Socket::Async.connect('localhost', $TEST_PORT) },
dd $conn.WHERE; # number changes

await $conn.print("not even JSON\n");

Ping - still failing on windows with raku 2022.07; Please let me know if I can provide further diagnostics.

THIS BLOCK IS NEEDED
{ $conn = await IO::Socket::Async.connect('localhost', $TEST_PORT) },
dd $conn.WHERE; # number changes

Is that , after the block meant to be a ;? As it is, the block never executes, and then the error is correct in the golf since there is no new $conn assigned.
Also, $conn.WHERE will change whenever the GC moves the object, you'd want $conn.WHICH if trying to determine object identity.

With those updates, the golf is no longer broken. The test, however, still fails.

Trying to regolf, but only down to 88 lines so far. Almost seems to need the subtest around the Async connect to get the error.

-subtest 'Handles lack of JSON input' => {
+#subtest 'Handles lack of JSON input' => {
+{

A golf-in-progress with this change goes from the cannot write to a closed socket to MoarVM panic: non-AsyncTask fetched from eventloop active work list

I think the test might be suffering from the same thing you pinged me on the golf. if I put a $conn.WHICH after each time $conn is set, it's still the same after the second one (in the subtest).

You can see this with the following diff to the original test file:
diff --git a/t/output-socket.rakutest b/t/output-socket.rakutest
index 11f3164..7b10e3e 100644
--- a/t/output-socket.rakutest
+++ b/t/output-socket.rakutest
@@ -34,6 +34,8 @@ lives-ok
         { $conn = await IO::Socket::Async.connect('localhost', $TEST_PORT) },
         'Can connect to the socket server';

+diag $conn.WHICH;
+
 my $received = '';
 react {
     whenever $conn.Supply {
@@ -123,6 +125,7 @@ subtest 'Handles lack of JSON input' => {
     lives-ok
             { $conn = await IO::Socket::Async.connect('localhost', $TEST_PORT) },
             'Can make another connection to the server';
+    diag $conn.WHICH;
     @handshake = ();
     my $timed-out = False;

These show the same two .WHICH results in the diag output of the stream.

On my M2 mac, when I run the modified test, the diag output shows two different $conn objects (windows shows the same one)

Does the lives-ok test there fail?

If so, then $conn is left as the previous (closed) connection. But the root cause of the issue is that connection failing.

Does the lives-ok test there fail?

no, works fine on the m2

If so, then $conn is left as the previous (closed) connection. But the root cause of the issue is that connection failing.

ok, will try to investigate.

Looks like the Windows CI is clean, perhaps this issue is gone?

Tracking down issues I'm having on windows getting the dependencies installed - please leave this open until I can verify, thanks.


Adding scenario around how we handle unexpected failures

Adding scenario around how we handle unexpected failures

@JackStromberg : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Any particular reason you want to explicitly call this out? Given ACI is a serverless service, I don't want to add anything related to the underlying infra unless absolutely necessary. We aim to have an SLA for a user's containers to be running, and this should account for issues related to the infra, without the user having to worry about it.
@dlepow thoughts?

@dkkapur - I agree this gets into infrastructure detail that might not be needed (or that could change). But it might help to link to SLA somewhere if we don't already?

I added this discussion to a pending doc work item around improving positioning of ACI as a serverless service in overview and landing page content. I think we can help make @JackStromberg's  point clearer as part of that work. So @dkkapur I recommend closing this PR. Thx.

#please-close


nf-validation bug

Hello,
I am using the<EMAIL_ADDRESS>plugin to validate the parameters of my pipeline and I am running into an issue.
I get this warning when running my pipeline:
WARN: Access to undefined parameter `monochromeLogs` -- Initialise it to a default value eg. `params.monochromeLogs = some_value`

To fix it I changed all of the places in my pipeline that I defined monochrome_logs to monochromeLogs and then I got this warning:
WARN: Access to undefined parameter `monochrome_logs` -- Initialise it to a default value eg. `params.monochrome_logs = some_value`

Additionally, when my colleague runs the pipeline it throws this error:
WARN: Access to undefined parameter `validationFailUnrecognizedParams` -- Initialise it to a default value eg. `params.validationFailUnrecognizedParams = some_value`

But again there is not validationFailUnrecognizedParams defined in the pipeline anywhere. Only validationFailUnrecognisedParams (with an s as opposed to a z)
I also tried to use the<EMAIL_ADDRESS>plugin, but I get the same issue. The template for the pipeline is the nf-core v2.14.1 template.

Hi this occurs because nf-validation is trying to use these parameters. The warning is thrown by Nextflow because there is no default for that parameter. You can fix this by simply adding these parameters to your nextflow.config file and setting their defaults to false. This has been fixed in nf-schema since that plugin uses configuration options instead of parameters.
Mind that this plugin is deprecated and I advice to migrate to nf-schema if you have the opportunity to do so :)

Closing this issue as I will archive this repo again. If you need any further help, please post on the forum :)
https://community.seqera.io/c/nextflow/5


zulip: 4.0.0 -> 5.0.0

Motivation for this change
Zulip 5.0 fixes multiple security issues. See:
https://blog.zulip.org/2020/04/01/zulip-desktop-5-0-0-security-release/
(cherry picked from commit 1586f2851e32834bf7033cb4bd30f6e5bcbfc356)
Reason: Zulip < 5.0.0 no longer works with the Zulip server---if you run
it with an up to date Zulip server you get an error message requesting
you upgrade to 5.0 and the app will otherwise not be functional.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Closure size before:
$ nix path-info -S /nix/store/d2ifz858d5459rpvlk2m9qrldc5qsdhn-zulip-4.0.0
/nix/store/d2ifz858d5459rpvlk2m9qrldc5qsdhn-zulip-4.0.0<PHONE_NUMBER>

Closure size after:
$ nix path-info -S /nix/store/7vfz0i3k5nnh1gcqsi1gc2s58is1w5x8-zulip-5.0.0
/nix/store/7vfz0i3k5nnh1gcqsi1gc2s58is1w5x8-zulip-5.0.0<PHONE_NUMBER>

Built, the GUI seems to work.
Not sure if it the program or our version of the package, but on closing  errors were returned about electron.

Not sure if it the program or our version of the package

AFAICT this is the program, and there have been electron errors for me for previous versions as well.


stack overflow iterating on a list

Description
A simple recursive iteration on a long list causes a stack overflow error.
I have seen this problem already reported, but I would like to add another case, especially given that it heavily affects the package that I am trying to port to eta.
Steps to Reproduce
git clone https://github.com/Quid2/flat
cd flat
git checkout d66cc7fc6880b50d81730fb9c075f7681889064f
etlas run listTest
Exception in thread "main" java.lang.StackOverflowError
at flat.data.flat.encoder.Prim.$L$wa15$1(Prim.hs:316)
at flat.data.flat.encoder.Prim.$La17$1(Prim.hs)
at flat.data.flat.encoder.Prim.eBoolF(Prim.hs:316)
at flat.data.flat.encoder.Prim$eBoolF.apply2V(Prim.hs:316)
at eta.runtime.apply.PAPSlow.apply(PAPSlow.java:71)
at eta.runtime.apply.PAPSlow.apply1V(PAPSlow.java:197)
at eta.runtime.thunk.Thunk.apply1V(Thunk.java:156)
at flat.data.flat.Instances$sat$215.apply1V(Instances.hs)
at eta.runtime.thunk.Thunk.apply1V(Thunk.java:156)
at flat.data.flat.Instances$sat$215.apply1V(Instances.hs)
at eta.runtime.thunk.Thunk.apply1V(Thunk.java:156)
at flat.data.flat.Instances$sat$215.apply1V(Instances.hs)
....
The code is simply:
import Data.Flat
longBools = replicate 1000000 True
main = print $ flat longBools
And what the code does is to iterate on the list encoding one boolean at the time.

Current Eta & Etlas version:
eta-0.8.6b2.
etlas version <IP_ADDRESS>
compiled using version <IP_ADDRESS> of the etlas-cabal library

@tittoassini By chance do you use some form of continuation passing style in implementing flat? If so, you'll need to add on a trampoline.
For now, can you let me know whether this works?
import Data.Flat
import Data.Function (trampoline)

longBools = replicate 1000000 True

main = print $ trampoline $ flat longBools

If that doesn't work, I'll see if I can assist you in finding the right location inside the definition of flat or its dependencies that you'll need to add it in.

I tried that, but no joy.
I also added a top level trampolineIO but it made no difference:
longBools = replicate 1000000 False
main = trampolineIO $ do
print $ (trampoline $ flat longBools)
The instance of Flat for [a] was generated by generics but I have rewritten
it explicitly so that I could add a 'trampoline' on encode:
instance {-# OVERLAPPABLE #-} Flat a => Flat [a] where
encode = trampoline . encodeListWith encode
...
{-# INLINE encodeListWith #-}
-- |Encode as a List
encodeListWith :: (t -> Encoding) -> [t] -> Encoding
encodeListWith enc = go
where
go [] = eFalse
go (x : xs) = eTrue <> enc x <> go xs
But I still get a stack overflow:
Exception in thread "main" java.lang.StackOverflowError
at flat.data.flat.encoder.Prim.$La17$1(Prim.hs)
at flat.data.flat.encoder.Prim.eBoolF(Prim.hs:317)
at flat.data.flat.encoder.Prim$eBoolF.apply2V(Prim.hs:317)
at eta.runtime.apply.PAPSlow.apply(PAPSlow.java:71)
at eta.runtime.apply.PAPSlow.apply1V(PAPSlow.java:197)
at eta.runtime.thunk.Thunk.apply1V(Thunk.java:156)
at flat.data.flat.Instances$sat$152.apply1V(Instances.hs)
at eta.runtime.thunk.Thunk.apply1V(Thunk.java:156)
...
By the way, reading your blog, I got the impression that 'trampoline' would
solve the stack overflow problem on all computations in its context so if
the trampolineIO in main does not fix it then there is no way of fixing it
by applying trampoline on any of the subcomputations, is this correct?
On Tue, 13 Nov 2018 at 21:59, Rahul Muttineni<EMAIL_ADDRESS>wrote:

@tittoassini https://github.com/tittoassini By chance do you use some
form of continuation passing style in implementing flat? If so, you'll
need to add on a trampoline.
For now, can you let me know whether this works?
import Data.Flatimport Data.Function (trampoline)
longBools = replicate 1000000 True
main = print $ trampoline $ flat longBools
If that doesn't work, I'll see if I can assist you in finding the right
location inside the definition of flat or its dependencies that you'll
need to add it in.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/typelead/eta/issues/901#issuecomment-438434328, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABYwRRDJgmz-by-By33Dt876XK2J2mDks5uuzKpgaJpZM4YWe2E
.

--
Dr. Pasqualino "Titto" Assini
http://networkpolitics.svbtle.com
http://quid2.org/


xen_4_8: init at 4.8.1

This commit adds the xen_4_8 package to be used instead of
xen (currently at 4.5.5):

Add packages xen_4_8, xen_4_8-slim and xen_4_8-light
Add packages qemu_xen_4_8 and qemu_xen_4_8-light to be used
with xen_4_8-slim and xen_4_8-light respectively.
Add systemd to buildInputs of xen (it is required by oxenstored)
Adapt xen service to work with the new version of xen
Use xen-init-dom0 to initlilise dom0 in xen-store
Currently, the virtualisation.xen.stored option is ignored
if xen 4.8 is used

Motivation for this change
This introduces Xen 4.8.1, which is the most current version, along with a few needed changes and a few fixes.
To use the new xen, use these configuration options.
virtualisation.xen.package = pkgs.xen_4_8;
virtualisation.xen.qemu-package = pkgs.xen_4_8;
virtualisation.xen.qemu = "${pkgs.xen_4_8}/lib/xen/bin/qemu-system-i386";

xen_4_8-slim may be used with this configuration:
virtualisation.xen.package = pkgs.xen_4_8-slim;
virtualisation.xen.qemu-package = pkgs.qemu_xen_4_8;
virtualisation.xen.qemu = "${pkgs.qemu_xen_4_8}/bin/qemu-system-i386";

Finally, xen_4_8-light may be used with this configuration:
virtualisation.xen.package = pkgs.xen_4_8-light;
virtualisation.xen.qemu-package = pkgs.qemu_xen_4_8-light;
virtualisation.xen.qemu = "${pkgs.qemu_xen_4_8-light}/bin/qemu-system-i386";

One notable limitation of this PR is that the virtualisation.xen.stored is ignored with xen 4.8, which I hope to address in the future.
Furthermore, there are a few statements in xen-dom0.nix and generic.nix that are conditional on the xen version, which is not ideal. Please let me know if you would like it to be handled differently.
Also, the package refers to the fork https://github.com/michalpalka/xen-libhvm of the xen-libhvm repo, which needed to be adapted to work with xen 4.8. This might equally be solved using a patch.
Please let me know if you have any feedback.
Things done

[ ] Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] Linux

[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

Awesome! Thanks! Will test on my machines in the next couple of days.
Problems I see without testing:

You introduce tabs in xen-dom0.nix (and I can now assume your tabstop = 8 :) )
The systemd patch is ugly. It potentially slows down complete system
boot by 10s. Is it upstream somewhere? Can we get it from upstream?
If not, how do they solve the same problem upstream?
Can 4.8's oxenstored build without systemd dependency? (A problem for
SLNOS which uses s6 init).

Hi!

Problems I see without testing:

You introduce tabs in xen-dom0.nix (and I can now assume your tabstop = 8 :) )

That's an oversight–I'll fix that and amend the commit

The systemd patch is ugly. It potentially slows down complete system
boot by 10s. Is it upstream somewhere? Can we get it from upstream?
If not, how do they solve the same problem upstream?

It does not slow the boot by 10s, because systemd continues as soon as it gets the notification from oxenstored. Interestingly, as I just found out, the patch is not needed anymore as I cannot reproduce the problematic behaviour with systemd 233. It seems that there was a problem with systemd 232, which led to the notification being dropped. Given that, it seems that the scripts in xen are correct, and the patch can be dropped–I'll drop it from the commit.

Can 4.8's oxenstored build without systemd dependency? (A problem for
SLNOS which uses s6 init).

Yes, oxenstored was previously built without an explicit systemd dependency. As a result, systemd was included when the xen package was compiled, but not xen-slim or xen-light. However, if oxenstored is built without systemd, it cannot be used with systemd services, so systemd needs to be somehow included conditionally.

I have just amended the commit removing the tabs and the race condition workaround. I have not done anything about the systemd dependency, and I am not sure how to continue. Should the packages have a useSystemd flag?

As a result, systemd was included when the xen package was compiled, but not xen-slim or xen-light.

Eh? Why? Systemd wasn't even referenced, IIRC.

I have just amended the commit removing the tabs and the race condition workaround. I have not done anything about the systemd dependency, and I am not sure how to continue. Should the packages have a useSystemd flag?

Nah, don't bother, I'll do that after I test it with s6 later.
LGTM. Thanks again!

Eh? Why? Systemd wasn't even referenced, IIRC.
It seems that systemd was propagated from one of the other dependencies, and just picked up by the configure script.

@grahamc can you merge this?

I would like to add security patches against this version of xen. Should I amend this PR, or create a new one?

I would add security patches on top here (but as separate commits, no squashing).

I have added the commit with security patches, and rebased everything.

michalpalka<EMAIL_ADDRESS>writes:

Is the idea to use ipxe packaged separately?

Yes, to make xen-light even lighter.

Actually, this code has since been dropped from the PR–see comment below, which I reproduce here:

No, I just fetched this branch and its still there.
Also the tabs in xen-dom0.nix are still there.

Is the idea to use ipxe packaged separately?

Yes, to make xen-light even lighter.
I'll create an issue.

Actually, this code has since been dropped from the PR–see comment below, which I reproduce here:

No, I just fetched this branch and its still there.
It seems that I have dropped the reference to the patch, but not the patch itself–sorry for that, I will fix it.
Also the tabs in xen-dom0.nix are still there.
That's right–I'll fix it too.

I have pushed new versions of commits, which hopefully address all concerns.

It seems that I have dropped the reference to the patch, but not the patch itself–sorry for that, I will fix it.

Actually, the reference to the patch was also there. I must have pushed old (unamended) commits by mistake. Thanks for the vigilance.

Thanks a lot @Mic92 @oxij @0xABAB @joachifm for reviewing and merging!
It would be great if some of you could also take a look at #25724, which adds pvgrub that is useful to have with xen.


EPUB an extra  is being added to the spine in package.opf and causing a validation issue

Before proceeding, check to make sure there isn’t an existing issue for this bug.

[X] I have searched the existing issues and determined this is a new bug.

Steps to reproduce

Start a new Quire project
Run quire build and then quire epub to create the EPUB file
Run the EPUB through epubcheck with java -jar path/to/epubcheck.jar patt/to/epubjs.epub

Actual behavior
An extra <itemref> is being added to the spine in package.opf, that doesn't resolve to an actual page and so is causing a validation error. It's the first item in the list and doesn't correspond to anything I can see in the manifest.json file.
<itemref idref="id_" linear="no"/>

@fchasen is this maybe meant to be a cover page/file, and so potentially related to issue #715?
Expected behavior
There should be this extra item or it should resolve to a page/resource.
Version numbers
quire-cli 1.0.0-rc.6
quire-11ty 1.0.0-rc.8
node 18.12.1
What browsers are you seeing the problem on?
No response
Relevant Terminal/Shell output
No response
Supporting Information
No response

Thanks, removed!


Disable PodSecurityPolicies with helm chart - ArgoCD

Hello everyone,
I am deploying stackrox-secured-cluster-services and I want to disable PSP.
I am installing the helm chart, through argo. Kubernetes version is 1.28
Is there a way to disable them without forking the Chart?
Thank you

Hello @makemesteaks,
The helm chart is implemented in a way that auto detects wether a cluster still has the PSP API resource. See this part.
If this is for some reason not working in your situation with Argo you can force this value to be false by setting it explicitly e. g. adding following to the helm call.
--set system.enablePodSecurityPolicies=false

Hope this helps you solve your issue :)

Thank you for the answers,
@johannes94 can I pass that as a flag to the values as well such as in the example below?
system:
   enablePodSecurityPolicies: false

Yes, that should work as well.


Failure on Django 1.7+ with django.db.models.fields.related.RelatedManager

We're trying to upgrade from Django 1.6 to 1.7 (and then to 1.8).
My tests fail with:
  File "/home/roger/.virtualenvs/fdw/local/lib/python2.7/site-packages/import_export/resources.py", line 383, in import_data
    real_dry_run)
  File "/home/roger/.virtualenvs/fdw/local/lib/python2.7/site-packages/import_export/resources.py", line 294, in get_diff
    v2 = self.export_field(field, current) if current else ""
  File "/home/roger/.virtualenvs/fdw/local/lib/python2.7/site-packages/import_export/resources.py", line 416, in export_field
    return field.export(obj)
  File "/home/roger/.virtualenvs/fdw/local/lib/python2.7/site-packages/import_export/fields.py", line 101, in export
    value = self.get_value(obj)
  File "/home/roger/.virtualenvs/fdw/local/lib/python2.7/site-packages/import_export/fields.py", line 86, in get_value
    value = value()
  File "/home/roger/.virtualenvs/fdw/local/lib/python2.7/site-packages/django/db/models/fields/related.py", line 693, in __call__
    manager = getattr(self.model, kwargs.pop('manager'))
KeyError: u'manager'

I can see that 0dfb4b9d8288bf47e56465f9269f96f56aea5c55 first tried to solve this using:
       # Manyrelatedmanagers are callable in Django >= 1.7 but we don't want
       # to call them
       if callable(value) and not value.through:
          value = value()

And that this was quickly upgraded by 2e2dc84aea199393549f21d312185cb88487eb4b to:
        if callable(value) and not hasattr(value, 'through'):
            value = value()

This still doesn't work for me. I have a model with a straightforward foreign key. It is still represented as a django.db.models.fields.related.RelatedManager but it isn't a ManyToMany and it doesn't have a through attribute and consequently it gets called here. That results in the following Django code being called:
def create_foreign_related_manager(superclass, rel_field, rel_model):
    class RelatedManager(superclass):
        def __init__(self, instance):
            super(RelatedManager, self).__init__()
            self.instance = instance
            self.core_filters = {rel_field.name: instance}
            self.model = rel_model

        def __call__(self, **kwargs):
            # We use **kwargs rather than a kwarg argument to enforce the
            # `manager='manager_name'` syntax.
            manager = getattr(self.model, kwargs.pop('manager'))

We aren't passing any arguments and the kwargs is an empty dict and so the pop fails.
If we don't want to call Managers, why don't we just test for them:
from django.db.models.manager import Manager

        # RelatedManager and ManyRelatedManager classes are callable in
        # Django >= 1.7 but we don't want to call them
        if callable(value) and not isinstance(value, Manager):
            value = value()
        return value

@rhunwicks does your proposed solution works? if so, can you send a pull request?
thanks!

It does fix the error I was getting. I'm trying to reproduce it in the django-import-export test suite so I can provide a test as well as a fix, but I haven't managed it yet.

Hi @rhunwicks
I noticed today when running tox that test_get_diff_with_callable_related_manager test fails with all django versions except 1.8.
this is stacktrace:
======================================================================
ERROR: test_get_diff_with_callable_related_manager (core.tests.resources_tests.ModelResourceTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bmihelac/dev/django/dev/django-import-export/tests/core/tests/resources_tests.py", line 205, in test_get_diff_with_callable_related_manager
    diff = resource.get_diff(author2, author)
  File "/Users/bmihelac/dev/django/dev/django-import-export/tests/../import_export/resources.py", line 293, in get_diff
    v1 = self.export_field(field, original) if original else ""
  File "/Users/bmihelac/dev/django/dev/django-import-export/tests/../import_export/resources.py", line 416, in export_field
    return field.export(obj)
  File "/Users/bmihelac/dev/django/dev/django-import-export/tests/../import_export/fields.py", line 105, in export
    return self.widget.render(value)
  File "/Users/bmihelac/dev/django/dev/django-import-export/tests/../import_export/widgets.py", line 36, in render
    return force_text(value)
  File "/Users/bmihelac/dev/django/dev/django-import-export/.tox/py27-1.7/lib/python2.7/site-packages/django/utils/encoding.py", line 85, in force_text
    s = six.text_type(s)
  File "/Users/bmihelac/dev/django/dev/django-import-export/.tox/py27-1.7/lib/python2.7/site-packages/django/db/models/manager.py", line 81, in __str__
    in opts.concrete_managers + opts.abstract_managers
StopIteration

I look at it. Thanks for the heads up.
On Fri, Sep 11, 2015 at 5:17 PM, Bojan Mihelac<EMAIL_ADDRESS>wrote:

Reopened #311
https://github.com/django-import-export/django-import-export/issues/311.
—
Reply to this email directly or view it on GitHub
https://github.com/django-import-export/django-import-export/issues/311#event-406968905
.

I am having hard time to see what AuthorResource in tests should export  for books. It fails for Django <=1.7 and for later versions it returns name of manager, ie core.Book.None. I have a feeling that it should be used with some kind of widget to get some meaningful output.
Can you help me @rhunwicks?
For now I skipped this test so travis does not complain.

My problem wasn't that I wanted to export anything - it is that imports were failing because Field.get_value() was calling the Manager. I.e. in 1.7+ where Managers are callable. I assume that this is the reason that the check for through was added. However, that check only works for ManyToMany models. I had an import failing for a ForeignKey - it didn't match the check, so the Manager was being called, which generated an error. I wanted to change the check so that it matched ForeignKey and ManyToMany fields, and I wanted to create a test that demonstrated the error that I was having before I made the change.
Hope that helps?

Yes it helps, thanks.
So i guess that we should test ForeignKey and ManyToMany fields in some ModelResource (if they are not already tested) so we can remove this failing test with previous django releases.


[v2] added opentelemetry metrics

@wtmaxim  This PR has issues i guess. I was not able to get any success locally. Had to rewrite this code, and add in OTEL exporter instead of Prometheus export. It seems to be working now. Can you maybe test this as well? It's merged to V2. Try to run the docker compose within the /compose folder. For now i have added only metrics. For traces, planning to add Jaegar.

@iammukeshm
Oh, I'm sorry. I did start Docker and everything was working perfectly. I will be even more careful in the future

@wtmaxim you were able to see the metrics coming into prometheus 9090? I was not able to, that's why had to change a couple of things.


Can't access disconnected regions with ASM

https://github.com/Azure/azure-sdk-for-python/blob/master/azure-servicemanagement-legacy/azure/servicemanagement/constants.py#L27 defines one specific hostname for API access. This effectively blocks use of the SDK for disconnected regions with their own API endpoints, like China East & China North.
The publishsettings file includes a management endpoint reference ( /PublishData/PublishProfile/Subscription/ServiceManagementUrl ), that could be used instead (like the nodejs tools); or a general mapping could be made of Azure regions to endpoints, e.g. 'West US' => 'management.core.windows.net', 'China North' => 'management.core.chinacloudapi.cn'.
As it stands though, I don't see any possible way to access disconnected regions.

You should be able to override that by passing a different value for the host parameter when you instantiate your service class:
https://github.com/Azure/azure-sdk-for-python/blob/master/azure-servicemanagement-legacy/azure/servicemanagement/servicemanagementservice.py#L67

Are you suggesting that every app that consumes this SDK and supports disconnected regions should provide a method of overriding the SDK?

Hi @bear454 !
@huguesv was just answered to your sentence:

I don't see any possible way to access disconnected regions

This is possible, since you can specify the host in the __init__ method :)
Could you be more specific and describe what you expected?
Thank you!

I will move forward with an override in azurectl, if this project chooses not to natively support disconnected regions, but what I expected is for this project to maintain parity with azure SDKs for other languages/environments:

https://github.com/Azure/azure-xplat-cli/blob/896b1bae02e56c5a732358b665ebf3bab2f9d7b7/lib/util/profile/publishSettings.js#L70
https://github.com/Azure/azure-sdk-for-go/blob/master/management/publishSettings.go#L70
https://github.com/Azure/azure-xplat-cli#azure-cli-with-china-cloud

We had already thought about a way to create a ASM client instance directly from the publish profile. We have an issue with requests, which do not handle in-memory certificate currently (your Go link shows clearly that they use in-memory certificate)
We chose that it's a too serious security issue to write the certificate in a temp file automatically. But the code to extract it is available:
https://github.com/Azure/azure-sdk-for-python/blob/master/azure-servicemanagement-legacy/azure/servicemanagement/publishsettings.py
Knowing that, tell me what could help you or submit a PR expanding this to something useful for you.

Of course! It's no problem to contribute a pull request; I just wanted to establish the issue first. ;-)

We've elected to handle this locally in the app, and explicitly send an endpoint to the SDK.


Log header is printed once for each application instance

Feature request
NLog version: 4.4.5
Platform:  .Net 4.0 / .Net 4.5 /
Current NLog config (xml or C#, if relevant)
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://www.nlog-project.org/schemas/NLog.xsd NLog.xsd"
      autoReload="true"
      throwExceptions="false"
      internalLogLevel="Off" internalLogFile="c:\temp\nlog-internal.log">

    <variable name="LogHome" value="${specialfolder:folder=CommonApplicationData}/Bosch/Video Stitcher/log/Wizard"/>
    <variable name="AppLogPath" value="${LogHome}/BVS.log"/>
    <variable name="LayoutFormat" value="${date:format=yyyy-MM-dd HH\:mm\:ss.fff} [${threadid}] ${level:uppercase=true} ${callsite} ${message} ${exception:format=ToString}" />
    <variable name="EndProgram" value="|-------------------------------------------|${newline}${newline}${newline}${newline}${newline}${newline}"/>

    <targets>
      <target name="MyLogfile"
              xsi:type="File"
              fileName="${AppLogPath}"
              archiveAboveSize="1024000"
              header="${date:format=yyyy-MM-dd HH\:mm\:ss.fff}${newline}"
              layout="${LayoutFormat}"
              footer="${date:format=yyyy-MM-dd HH\:mm\:ss.fff}${EndProgram}"
              archiveNumbering="DateAndSequence"
              archiveDateFormat="yyyy-MM-dd"
              maxArchiveFiles="500"
              encoding="utf-8" />
    </targets>
    <rules>
      <logger name="*" minlevel="Info" writeTo="MyLogfile" />
    </rules>
  </nlog>

What is the current result?
Similar to #1713, the header is printed only at the start of a log file, even if my application shuts down and starts again.
What is the expected result?
I'd like headers to appear everytime my application starts, so I can tell in the log file which logs belong to each run of my application.

Thanks for reporting!

l think we should introduce 2 options

headerOnStart
footerOnEnd

Has someone a better name proposal? ;)

Just to add info, I actually need the headerOnStart because if my application crashes (we can only hope) the footer is never printed.
If the footer was printed even if the app crashes I would not have requested this feature, not sure this is possible though.

What should happen on LoggingConfiguration-Reload (CloseTarget / InitializeTarget) ?
The header-logic is mostly for the CSV-layout. Why not have a log-message at startup like "Starting..." ?
Alternative solutions could be that you activated ArchiveOldFileOnStartup
An alternative approach could be to make a target-wrapper that injects an artificial LogEventInfo on first Log-Event (With an custom Layout to generate the message for the artifical LogEventInfo).

I'd like to have the option like "headerOnStart" too.
Regarding to

Why not manually write a log-message at startup like "Starting..." ?

this is not an elegant solution because such a logic produces the content as below:
----------------- Logging started on 2018-11-16 12:23:55.8735 ------------------
2018-11-16 12:23:55.8395 | Common | Info | Main form is opened |
----------------- Logging finished on 2018-11-16 12:24:10.4863 -----------------
2018-11-16 12:24:22.0540 | Common | Info | Main form is opened |

and is not what it should be.

I agree that this option "headerOnStart" could turn out to be very useful. Right now I'm writing the header manually because in our application we need to track when the app starts up.
Has this feature been introduced finally?

Nope. Feel free to send a PR! :)

@d3ns1ty Make sure to read https://github.com/NLog/NLog/issues/2119#issuecomment-302490558 before you start implementing a solution.

What should happen on LoggingConfiguration-Reload (CloseTarget / InitializeTarget) ?

I think in such one should leave an appropriate message in the log target, something as "The configuration have been reloaded", for example.

@zdens
I think the question is, will the headerOnStart printed on  CloseTarget / InitializeTarget, and if not, how to remember that?

Especially when a new instance of the target is created on reload :)
Sent from my Sony Xperia
---- Julian Verdurmen wrote ----
@zdenshttps://github.com/zdens
I think the question is, will the headerOnStart printed on CloseTarget / InitializeTarget, and if not, how to remember that?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/NLog/NLog/issues/2119#issuecomment-480328085, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AK-fnLXiSoR7VJzqeLJlUAYcK5V3OCkjks5vd3HZgaJpZM4NfGM-.

I think the question is, will the headerOnStart printed on CloseTarget / InitializeTarget, and if not, how to remember that?

I think no, and remember it logfactory as static (name target -> bool). What do you think @snakefoot ?

As long it is done in a target wrapper and not polluting the file-target even more with random logic then it is fine with me
Sent from my Sony Xperia
---- Julian Verdurmen wrote ----
I think the question is, will the headerOnStart printed on CloseTarget / InitializeTarget, and if not, how to remember that?
I think no, and remember it logfactory as static (name target -> bool). What do you think @snakefoothttps://github.com/snakefoot ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/NLog/NLog/issues/2119#issuecomment-480397631, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AK-fnAMl9KjLZKLaa5kivVUzj6xAOFvvks5vd6XkgaJpZM4NfGM-.

Would it be logic to have it available in other targets else than the file target? Database, event log maybe?

Anything where you need a starting and stopping event.
Sent from my Sony Xperia
---- Julian Verdurmen wrote ----
Would it be logic to have it available in other targets else than the file target? Database, event log maybe?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/NLog/NLog/issues/2119#issuecomment-480407491, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AK-fnJFG_H8D7p6uW1Gpyzm9IhcCvx-hks5vd65KgaJpZM4NfGM-.

But remember a good library doesn't try to do everything. But should be good at its core stuff while being open and extensible. Careful with not adding everything but the kitchen sink to the NLog core
Sent from my Sony Xperia
---- Julian Verdurmen wrote ----
Would it be logic to have it available in other targets else than the file target? Database, event log maybe?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/NLog/NLog/issues/2119#issuecomment-480407491, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AK-fnJFG_H8D7p6uW1Gpyzm9IhcCvx-hks5vd65KgaJpZM4NfGM-.

True, but I think this is a nice feature and suitable for NLog. If implemented as wrapper, it's modular :+1:

Are you going to implement this feature? @304NotModified

@mwagenfuehr Think you are very welcome to create a PullRequest and implement this feature.

@mwagenfuehr this isn't on the roadmap. PR is accepted :) (when having enough tests)

@304NotModified  @snakefoot
Maybe we have workaround for this issue? How is it cached or determined that the header has already been written? Maybe there is an API for clearing this cache?

This also doesn't work if the layout has been modified.

@RomanSoloweow You are welcome to provide a pull-request with your ideas.Make sure to read https://github.com/NLog/NLog/issues/2119#issuecomment-302490558 before you start implementing a solution.

NLog v5.2.6 has been released - https://www.nuget.org/packages/NLog/5.2.6
Where NLog FileTarget has the new setting WriteHeaderWhenInitialFileNotEmpty. Thanks to @RomanSoloweow


About first stage proxy code training

hello，I want to ask about the details about the first stage proxy code training. From the beginning training the cross entropy loss will round 6.9 (-log(1/1024)), and after training what the value of CE loss will lead to the results of baseline

Also want to get more details on this; And for the decoder finetuning, are you (1) only finetune VQGAN decoder (2) or finetune the Transformer and VQGAN decoder?

Hi,
Thank you all for your interests in our work and patience. We have released the two-stage training code and feel free to check it.


updated

updated

Looks like a lot of work you did there friend 👍   What exactly is your code and what's it doing?


Fail to return sum of priority [segment_tree.py ]

From the code: https://github.com/openai/baselines/blob/master/baselines/common/segment_tree.py
The "sum" method in SumSegmentTree class should return arr[start] + ... + arr[end] but, seems it returns arr[start] +... + arr[end-1]
def sum(self, start=0, end=None):
    """Returns arr[start] + ... + arr[end]"""
    return super(SumSegmentTree, self).reduce(start, end)

[my test code]
capacity = 16
ss_tree = SumSegmentTree(capacity)
ss_tree[0] = 3
ss_tree[1] = 2
ss_tree[2] = 5
ss_tree[3] = 4
print('sum of elements:', ss_tree.sum(start=0, end=3))
[result]
sum of elements: 10

It seems to be a typo in code comments (which reappears several times). Judging from implementation, functionality is supposed to mimic slicing logic: start <= idx < end
def reduce(self, start=0, end=None):
        if end is None:
            end = self._capacity
        if end < 0:
            end += self._capacity
        end -= 1
        return self._reduce_helper(start, end, 1, 0, self._capacity - 1)

Calling sum with no arguments executes tree-wise operation sum(0, self._capacity - 1) for example.

Then,  I think below code:
(from: https://github.com/openai/baselines/blob/master/baselines/deepq/replay_buffer.py)

has to be fixed:
len(self._storage - 1) -> len(self._storage)
right?

Yeah, you are right.
Either len(self._storage) or calling with no arguments.
That's a very old bug, judging from opened issues. Goes back to year 2017.
https://github.com/openai/baselines/issues?q=prioritized
https://github.com/openai/baselines/issues?q=is%3Aissue+is%3Aopen+tree
I think some developer claimed to fix it almost immediately, but his brunch probably got lost in time.
If you are interested in implementing prioritised replay for your own project, good news is segment_tree seems to be implemented correctly (besides the thing that executing reduce method on empty tree causes infinite recursion).
This issue https://github.com/openai/baselines/issues/1158 gives some useful insights about segment_tree

Ah, okay I got it. thank you for your attention!


Need a format in Calc for currency values (i.e., with currency symbol and correct number of decimal places)

Probably should use the locale settings for this.

After the fixes for #41 this is now working as intended.


Improving ReactiveSocket abstraction.

Today we have the following primary interfaces that application writers interact with:
ReactiveSocket
Primary abstraction for the multiplexed, full duplex connection following the protocol.
ConnectionSetupHandler
Used only at the server when a new connection is accepted.
RequestHandler
Used only at the server to handle various interaction models requested by the client.
Problem
There are a few issues/lack of feature with the above abstraction and current implementations:

Client has no way of accepting requests from the server (this isn't currently implemented and I can imagine using ConnectionSetupHandler and RequestHandler for this purpose)
RequestHandler and ReactiveSocket have very similar methods (primarily around various request interaction models) and hence is confusing as to why they are two different contracts.
Client does not have a way to accept leases from the server. (this is also a lack of feature today but not impossible with the current abstractions)

Proposal
We can replace the 3 interfaces with two interfaces
ReactiveSocket
Keep the current interface but use it both as a way to request and respond to requests on a ReactiveSocket. This means for every connection there will be two ReactiveSocket instances on each end.
ReactiveSocketFactory (May be a better name)
On either end (client/server) there will be a factory which may look something like the below.
public interface ReactiveSocketFactory {

    /**
     * {@code ReactiveSocket} is a symmetric protocol where everything that a client can do can be done on the server
     * and vice-versa. So, every transport connection is associated with two {@code ReactiveSocket} instances, one for
     * responding to requests from the peer and another for sending requests to the peer.
     * This method is the converter that accepts a remote {@code ReactiveSocket} which is used to send requests to the
     * peer and returns a local {@code ReactiveSocket} that responds to requests from the peer.
     *
     * @param requestingSocket Socket used to send request to the peer.
     *
     * @return {@code ReactiveSocket} that is used to accept and respond to request from the peer.
     */
    ReactiveSocket accept(ReactiveSocket requestingSocket);
}

I am not yet sure, but this may look different at client than at the server due to the nature of replying to or sending a ConnectionSetupFrame. Anyways, this will be a replacement for ConnectionSetupHandler and either generify or provide for client side usecases.
This change will simplify the API a bit and also provide symmetric APIs for both client and server.

Used only at the server to handle various interaction models requested by the client.

The terminology is a little off here. It is used on the "requester" side of the relationship. Since this protocol is bi-directional, it can be used on either client or server.

Client has no way of accepting requests from the server

Well then that should be fixed. It used to be possible. A client had an overload that took a RequestHandler. Here is old code showing that: https://github.com/ReactiveSocket/reactivesocket-java/blob/v0.0.4/src/main/java/io/reactivesocket/ReactiveSocket.java#L110

ReactiveSocket accept(ReactiveSocket requestingSocket);

I don't understand that signature. You return the same type as is passed in. Is it the same one? A different one?
It used to be very clear ... a server was created to accept connections, or a client connected to a server.

There are two aspects to this change, let us discuss this one at a time.
Remove RequestHandler in favor of ReactiveSocket
RequestHandler and ReactiveSocket interfaces are identical when it comes to the different interaction models i.e.
    Publisher<Void> fireAndForget(final Payload payload);

    Publisher<Payload> requestResponse(final Payload payload);

    Publisher<Payload> requestStream(final Payload payload);

    Publisher<Payload> requestSubscription(final Payload payload);

    Publisher<Payload> requestChannel(final Publisher<Payload> payloads);

    Publisher<Void> metadataPush(final Payload payload);

So, one may think of RequestHandler as an implementation of ReactiveSocket.
(This of course is assuming that we do not have methods like onRequestReady, for which there is a separate issue.)
Thus, it appears to me that the presence of RequestHandler isn't really useful. Assuming that we don’t need it, ConnectionSetupHandler would look like:
public interface ConnectionSetupHandler {
    ReactiveSocket apply(ConnectionSetupPayload setupPayload, ReactiveSocket reactiveSocket) throws SetupException;
}

The above is what I was calling as ReactiveSocketFactory in my initial proposal, removing ConnectionSetupPayload which was a mistake.
Let us park the second part of the proposal related to ReactiveSocketFactory for a while and see if this change makes sense at first.
PS: Me and @stevegury were discussing offline that having start() method isn't also totally useful but we can save that for another issue.

I really like this proposal for its elegance.
Does this allow two interesting use cases, or if not, what would be different?

a simple no serialization ReactiveSocket.  This seems great for testing and mocking purposes.  And the same RequestHandler.Builder pattern could be used for these ReactiveSockets.
trivial proxying of an existing ReactiveSocket over another transport.  Especially interesting when a middle service can bridge two transports or networks.

I really like this proposal for its elegance.
🎉 😄

I think the usability from a server end will roughly be the same just with less concepts (no RequestHandler). The benefit on the client side will be that it will be symmetric with the server, so again less concepts.
Both the usecases you mentioned I believe can be achieved today.

I meant that today if I have a RequestHandler that implements my business logic, I can't use it anywhere that wants a ReactiveSocket without creating something like a Local Transport ReactiveSocket and going through serialization to frames etc.  This is good and bad.
Likewise currently if I have a ReactiveSocket, to expose it over another transport I need to make a RequestHandler to wrap it.

@yschimke aah .. correct! I didn't think about that but it is a nice consequence of this change 👍

Talking more with @benjchristensen about it, a couple of points he brought up.

The signature for requestChannel is different deliberately, and allows the first payload to be peeked at for things like routing logic.
ReactiveSocket has additional methods that don't make sense when implementing a RequestHandler e.g. start, shutdown, on*

The cases I like can be pretty easily supported with adapters e.g. RequestHandler.fromOtherReactiveSocket(rs) and ReactiveSocket.buildLocal(requestHandler)

Well then that should be fixed. It used to be possible. A client had an overload that took a RequestHandler. Here is old code showing that: https://github.com/ReactiveSocket/reactivesocket-java/blob/v0.0.4/src/main/java/io/reactivesocket/ReactiveSocket.java#L110

This isn't too bad, its just the simpler helper methods of DefaultRequestSocket
DefaultRequestSocket.fromClientConnection(DuplexConnection, ConnectionSetupPayload, RequestHandler, Consumer)
and
DefaultRequestSocket.fromClientConnection(DuplexConnection, ConnectionSetupPayload, Consumer)
I think real world usage will dictate whether we support 1-2 line connection building for both cases in each ReactiveSocketConnector impl, or you need to drop back to creating the DuplexConnection, passing it into fromClientConnection.

Thanks for the input @benjchristensen @yschimke . Since @robertroeser and @stevegury are OOO I will wait for them to express their thoughts on this.

Keep the current interface but use it both as a way to request and respond to requests on a ReactiveSocket. This means for every connection there will be two ReactiveSocket instances on each end.

I don't like this approach because it confuses the semantics of Requester and Responder.
Just because they happen to have Publisher for both input and output, that does not mean Requester and Responder are semantically the same. They are asymmetric in a few ways:
a) Developers will only ever implement the Responder interface, not the Requester, as the Requester APIs are always concretely provided by the reactivesocket-java library.
b) The Requester APIs are exposed to be invoked by developers, but Responder APIs are enveloped inside the state machine and invoked by reactivesocket-java upon receiving frames.
c) Practical usage of the interfaces resulted in the channel signature differing between Requester and Responder (not just name, but parameter signature) so that it provided an "initial payload". Without this it is difficult and costly to make routing decisions to which code a request should be handled with. Forcing Requester and Responder to be the same negatively affects usability.
I strongly thing that Requester and Responder should be treated differently, have different names, and allow their signatures to diverge as necessary, despite being similar. They are similar because they are the duals of each other, but they are not the same.

Client has no way of accepting requests from the server (this isn't currently implemented and I can imagine using ConnectionSetupHandler and RequestHandler for this purpose)
Client does not have a way to accept leases from the server. (this is also a lack of feature today but not impossible with the current abstractions)

Both of these used to work and was quite elegant. Why not just revert to the prior design? What was the reason for moving away from it?

Just because they happen to have Publisher for both input and output, that does not mean Requester and Responder are semantically the same.

That very clearly is not the reason for this change. The reason is that they both deal with same interactions (FnF, Request-Response, ....), one requests and another responds with same signature. To me separating out the first Payload for requestChannel is not convincing enough reason for them to have different interfaces, YMMV.

About separating out the first Payload for requestChannel, I would like to be able to access the Payload before subscribing. Clients may want to engage different channel behaviors, and should be able to specify it cleanly; I am running into this problem while writing the TCK. Right now, I have to define a handler that has to call request(n) and then decide what response to give based on the first payload received, which does not seem as elegant as it should.

@NiteshKant
Yes, I think it would make for cleaner handler code if we could get the first payload in channel like we can in requestStream/Subscription/Response.

Ok so you are suggesting that this method in ReactiveSocket
Publisher<Payload> requestChannel(final Publisher<Payload> payloads);

can be changed to
Publisher<Payload> requestChannel(final Payload first, final Publisher<Payload> payloads);

It is interesting because a channel is symmetric and the need to peek to the first payload while sending a request may also apply to the received payload stream (returned Publisher here).
I wonder if this can be achieved by providing a utility like:
public static <T> void peekFirst(Publisher<T> source, BiConsumer<T, Publisher<T>> consumer)

which basically removes the need of this peculiarity in the main interface.

That is inefficient as it requires either subscribing twice, or multicasting.

That very clearly is not the reason for this change.

But that's what this change looks to be pursuing, as I see no benefit of ever having the same interface shared for both. No developer is ever implementing the Requester side, only the Responder side.
I am really struggling to see the real benefits of this change, only negatives.


Obscure error when the trying to compile for x86-uwp triplet and UWP support is not installed in Visual Studio

When trying to install the zlib port for x86-uwp, I got this failure:
pegua@LAPTOP-TO4SAKLB MINGW64 /c/src/vcpkg (master)
$ ./vcpkg.exe --triplet x86-uwp install zlib
The following packages will be built and installed:
    zlib[core]:x86-uwp
Starting package 1/1: zlib:x86-uwp
Building package zlib[core]:x86-uwp...
-- Using cached C:/src/vcpkg/downloads/zlib1211.tar.gz
-- Applying patch C:/src/vcpkg/ports/zlib/cmake_dont_build_more_than_needed.patch
-- Applying patch failed. This is expected if this patch was previously applied.
-- Configuring x86-uwp
CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:56 (message):
    Command failed: ninja;-v
    Working Directory: C:/src/vcpkg/buildtrees/zlib/x86-uwp-rel/vcpkg-parallel-configure
    See logs for more information:
      C:\src\vcpkg\buildtrees\zlib\config-x86-uwp-out.log

Call Stack (most recent call first):
  scripts/cmake/vcpkg_configure_cmake.cmake:259 (vcpkg_execute_required_process)
  ports/zlib/portfile.cmake:19 (vcpkg_configure_cmake)
  scripts/ports.cmake:71 (include)

Error: Building package zlib:x86-uwp failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with `.\vcpkg update`, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
  Package: zlib:x86-uwp
  Vcpkg version: 2018.11.23-nohash

Additionally, attach any relevant sections from the log files above.

pegua@LAPTOP-TO4SAKLB MINGW64 /c/src/vcpkg (master)

After inspecting the log in C:\src\vcpkg\buildtrees\zlib\config-x86-uwp-out.log, I understood that the problem was that UWP support was not installed in Visual Studio.
I wonder if there could be a way to get a message a bit more clear, as x86-uwp is a default triplet, and such an obscure message could confuse users that do not have a lot of experience with vcpkg.

This issue hasn’t been updated in a year; if it is still an issue, please reopen.

This issue hasn’t been updated in a year; if it is still an issue, please reopen.

Hi @PhoebeHui, I think this is still an issue. However, even if I am the original author of the issue, I can not reopen it.


Add output filepath option

Added a parameter to let users output the minimum requirements to a filepath
Added unit and integration test for this

Codecov Report

Merging #12 (25ecea8) into main (d94b85a) will decrease coverage by 1.56%.
The diff coverage is 73.91%.

@@            Coverage Diff             @@
##             main      #12      +/-   ##
==========================================
- Coverage   95.90%   94.33%   -1.57%
==========================================
  Files           7        8       +1
  Lines         244      265      +21
==========================================
+ Hits          234      250      +16
- Misses         10       15       +5

Impacted Files
Coverage Δ

minimum_dependency_generator/main.py
28.57% <0.00%> (-2.20%)
:arrow_down:

...pendency_generator/minimum_dependency_generator.py
95.53% <60.00%> (-3.50%)
:arrow_down:

minimum_dependency_generator/tests/test_utils.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d94b85a...25ecea8. Read the comment docs.


Struct data time string as ISO format.

Zero of UNIX epoch is represented to follow string as ISO Date Format.
1980-01-01T00:00:00.000000+00:00
On this update, the function to get the string which is ISO Date Format is added as new module.
Planed function prototype is follow.
def get_iso_date(
    year: int = 1980,
    month: int = 1,
    day: int = 1,
    hour: int = 0,
    minutes: int = 0,
    second: int = 0,
    millisecond: int = 0,
    time_zone_hour: int = 0,
    time_zone_second: int = 0
    ) -> str:

Create two modules to make it realized this functionality.
First module: "format_iso_date"
Convert date time value from human friendly format to format structured by type string.
{
    "year": {"year": 2023, "month": 4, "day": 1},
    "hour": {"hour": 4, "minute": 51, "second": 30, "micro": 123},
    "zone": {"hour": 9, "minute": 0},
}

Date time value as following format is returned.
{
    "year": {"year": "2023", "month": "04", "day": "01"},
    "hour": {
        "hour": "04",
        "minute": "51",
        "second": "30",
        "micro": "000123",
    },
    "zone": {"hour": "09", "minute": "00"},
}

Second module: "create_iso_date"
You can get date time value choose from following three types.

Date time string represented to ISO date format.

"2023-04-01T04:51:30.000123+09:00"

Date time object in Python default.

datetime(2023, 4, 1, 4, 51, 30, 123, tzinfo=timezone(timedelta(seconds=32400)))

Date time as UNIX epoch, it’s object of class "Decimal".

Decimal('1680292290.000123')


Hessian produces wrong results, but works if I add a perturbation

I have the following function I would like the Hessian of:
def rbf(x1, x2, l):
    diff = x1/l - x2/l
    dist = torch.linalg.norm(diff)
    return torch.exp(-dist * dist)

Taking the Hessian with functorch.jacfwd and functorch.grad seems to get me an incorrect result on a naive example. According to my derivation by hand, this should return 2.0 / l**2 * torch.eye(16). If I add a small number to one of the terms, the Hessian function returns the correct result:
jacfwdgrad_rbf = jacfwd(grad(rbf, argnums=0), argnums=1)  # doing d^2f/dx1dx2

a = torch.randn(16)
l = 1.0

print(torch.allclose(jacfwdgrad_rbf(a, a, l), 2/l/l * torch.eye(16)))  # prints False

print(torch.allclose(jacfwdgrad_rbf(a, a+1e-8, l), 2/l/l * torch.eye(16)))  # prints True

Is there some bug that means the former check doesn't work, or is there something I'm missing?

Quick update: using jacrev in place of jacfwd produces a matrix of nans when I don't perturb and works as expected when I do. Setting torch.autograd.set_detect_anomaly(True) gives a traceback for where there might be issues, and here's what came out:
/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/torch/autograd/__init__.py:300: UserWarning: There is a performance drop because we have not yet implemented the batching rule for aten::any. Please file us an issue on GitHub so that we can prioritize its implementation. (Triggered internally at /opt/conda/conda-bld/pytorch_1666642969563/work/aten/src/ATen/functorch/BatchedFallback.cpp:82.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/torch/autograd/__init__.py:300: UserWarning: Error detected in DivBackward0. Traceback of forward call that caused the error:
  File "/home/tawe141/mfip/scripts/test_bad_hess.py", line 25, in <module>
    print(torch.allclose(jacrevgrad_rbf(a, a, l), 2/l/l * torch.eye(16)))  # matrix of nans
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/eager_transforms.py", line 459, in wrapper_fn
    vjp_out = _vjp_with_argnums(func, *args, argnums=argnums, has_aux=has_aux)
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/vmap.py", line 35, in fn
    return f(*args, **kwargs)
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/eager_transforms.py", line 289, in _vjp_with_argnums
    primals_out = func(*primals)
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/eager_transforms.py", line 1241, in wrapper
    results = grad_and_value(func, argnums, has_aux=has_aux)(*args, **kwargs)
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/vmap.py", line 35, in fn
    return f(*args, **kwargs)
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/eager_transforms.py", line 1133, in wrapper
    flat_grad_input = _autograd_grad(flat_outputs, flat_diff_args, create_graph=True)
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/eager_transforms.py", line 113, in _autograd_grad
    grad_inputs = torch.autograd.grad(diff_outputs, inputs, grad_outputs,
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/torch/autograd/__init__.py", line 300, in grad
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/torch/fx/traceback.py", line 57, in format_stack
    return traceback.format_stack()
 (Triggered internally at /opt/conda/conda-bld/pytorch_1666642969563/work/torch/csrc/autograd/python_anomaly_mode.cpp:114.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/torch/autograd/__init__.py:300: UserWarning:

Previous calculation was induced by DivBackward0. Traceback of forward call that induced the previous calculation:
  File "/home/tawe141/mfip/scripts/test_bad_hess.py", line 25, in <module>
    print(torch.allclose(jacrevgrad_rbf(a, a, l), 2/l/l * torch.eye(16)))  # matrix of nans
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/eager_transforms.py", line 459, in wrapper_fn
    vjp_out = _vjp_with_argnums(func, *args, argnums=argnums, has_aux=has_aux)
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/vmap.py", line 35, in fn
    return f(*args, **kwargs)
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/eager_transforms.py", line 289, in _vjp_with_argnums
    primals_out = func(*primals)
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/eager_transforms.py", line 1241, in wrapper
    results = grad_and_value(func, argnums, has_aux=has_aux)(*args, **kwargs)
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/vmap.py", line 35, in fn
    return f(*args, **kwargs)
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/eager_transforms.py", line 1111, in wrapper
    output = func(*args, **kwargs)
  File "/home/tawe141/mfip/scripts/test_bad_hess.py", line 7, in rbf
    diff = x1 / l - x2 / l
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/torch/fx/traceback.py", line 57, in format_stack
    return traceback.format_stack()
 (Triggered internally at /opt/conda/conda-bld/pytorch_1666642969563/work/torch/csrc/autograd/python_anomaly_mode.cpp:121.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "/home/tawe141/mfip/scripts/test_bad_hess.py", line 25, in <module>
    print(torch.allclose(jacrevgrad_rbf(a, a, l), 2/l/l * torch.eye(16)))  # matrix of nans
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/eager_transforms.py", line 474, in wrapper_fn
    results = vmap(vjp_fn)(basis)
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/vmap.py", line 362, in wrapped
    return _flat_vmap(
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/vmap.py", line 35, in fn
    return f(*args, **kwargs)
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/vmap.py", line 489, in _flat_vmap
    batched_outputs = func(*batched_inputs, **kwargs)
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/eager_transforms.py", line 323, in wrapper
    result = _autograd_grad(flat_primals_out, flat_diff_primals, flat_cotangents,
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/functorch/_src/eager_transforms.py", line 113, in _autograd_grad
    grad_inputs = torch.autograd.grad(diff_outputs, inputs, grad_outputs,
  File "/home/tawe141/miniconda3/envs/mfip/lib/python3.10/site-packages/torch/autograd/__init__.py", line 300, in grad
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: vmap: It looks like you're calling .item() on a Tensor. We don't support vmap over calling .item() on a Tensor, please try to rewrite what you're doing with other operations. If error is occurring somewhere inside PyTorch internals, please file a bug report.

I think the problem is because norm is not differentiable at 0. IIRC, PyTorch returns sub gradient for norm at 0 but has trouble with higher orders.
x = torch.zeros(3, dtype=torch.double, requires_grad=True)
o = torch.linalg.norm(x)
gx = torch.autograd.grad(o, x, retain_graph=True, create_graph=True)[0]
gxx = torch.autograd.grad(gx, x, grad_outputs=torch.ones_like(gx))
print(gx)
print(gxx)

Output
tensor([0., 0., 0.], dtype=torch.float64, grad_fn=<MulBackward0>)
(tensor([nan, nan, nan], dtype=torch.float64),)

Pinging @lezcano will have a better understanding of the same.
Regarding, the failure with torch.autograd.set_detect_anomaly(True), you are right that it detects nan's but unfortunately it fails with incorrect message and it is tracked at https://github.com/pytorch/functorch/issues/1049

Thanks @kshitij12345 , I came up with a workaround that didn't use norm and it works great.

Kshiteej is right on the money on the second derivative of linalg.norm. As a small note, you may want to use linalg.vector_norm as it has cleaner semantics than linalg.norm.
At any rate, we could fix this second derivative issue, but given that the OP found a work-around, I'm leaning towards tagging this as low priority.


refactor contribution guideline

Description
Rough draft for updated contribution guidelines. @SirSpidey Probably need you to clean it up, please, after @ocofaigh has confirmed this is the approach we want.

Release required?

[ ] No release
[ ] Patch release (x.x.X)
[ ] Minor release (x.X.x)
[ ] Major release (X.x.x)

Release notes content

Run the pipeline
If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.
Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:
/run pipeline

Checklist for reviewers

[ ] If relevant, a test for the change is included or updated with this PR.
[ ] If relevant, documentation for the change is included or updated with this PR.

For mergers

Use a conventional commit message to set the release level. Follow the guidelines.
Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
Use the Squash and merge option.

replaced by #65


Collection of some N* issues...

          For the record: while investigating N* in this channel,

I find out something interesting about N(1440)
Roper resonance – solution to the fifty year puzzle
Originally posted by @shenvitor in https://github.com/ComPWA/gluex-nstar/issues/17#issuecomment-2180594314

"Since nothing has been heard from them since the 1960's, we declare them to be dead."
https://pdglive.lbl.gov/Particle.action?init=0&node=B128&home=BXXX005


Objects: Missed a bar

Ref: https://github.com/BonsaiDen/JavaScript-Garden/pull/373
This definitely looks weird. I have no idea what I'm doing.
Should probably change the entire example. Ideas welcome! 😅

I would suggest count instead of numberOfTimes


Bug: Non-linear, very long index restore durations (python 3.10.12, usearch==2.16.0)

Describe the bug
With larger usearch index sizes, restore times become impractically long. For a range of different index sizes, restore durations range from ~10s for an 11GB / 6m embedding index up to 45m (!) for a 32GB / 20m embedding index. This happens with memory mapping on or off (i.e. view=True or view=False). During the entire load time, 1 cpu core is pegged out at 100%. After being loaded, index appears to behave normally.
We are running this on an ec2 instance with 64GB of ram, so the entire index should fit very comfortably in memory even with memory_map turned off. The index files are being loaded from ephemeral SSDs attached to the ec2 instance, so disk read time should not be a major factor.
We are running this inside of docker (ECS), however we have not experienced similar file load issues with other software (we use a variety of python and non-python libraries that involve loading large files from this same storage, regularly >= 100GB) so it seems unlikely to be something at the OS/docker level 🤷  (the ECS task has access to the full amount of memory)
Steps to reproduce
import usearch.index
index = usearch.index.Index.restore(path_to_index, view=True) # also happens with view=False

The index was built with usearch using all defaults, then saved to disk via index.save(index_path). Once loaded, index functions normally.
Expected behavior
We would expect a somewhat linear-ish relationship between index size / embedding count and load times.
Thank you for such an awesome project, we have fallen in love with usearch and hope we can figure this one out, which is currently blocking us from using it!
USearch version
v2.16.0
Operating System
Ubuntu 22.04 (dockerized ECS)
Hardware architecture
x86
Which interface are you using?
Python bindings
Contact Details
<EMAIL_ADDRESS>Are you open to being tagged as a contributor?

[x] I am open to being mentioned in the project .git history as a contributor

Is there an existing issue for this?

[X] I have searched the existing issues

Code of Conduct

[X] I agree to follow this project's Code of Conduct

An example index info for what we're building:
usearch.Index(ScalarKind.BF16 x 768, MetricKind.IP, multi: False, connectivity: 16, expansion: 128 & 64, 6,738,822 vectors in 5 levels, haswell hardware acceleration)

This one took ~10s to load, another one with ~20m vectors took 45 minutes.

@kennon, interesting, looking into it!

@ashvardanian awesome, thanks! I don’t want to post a public url but if you drop me an email I can send you a link to the index files we are trying to load. Let me know if there is any more information I can provide, thanks!


Stale data read in subsequent requests after an HTTP GET that contains a Response Body

I have code that first attempts to issue an HTTP HEAD request to a url. If the HEAD request fails (either connection refused or non 2xx status code, I retry the request as a normal HTTP GET.
However, I have one URL for which the HTTP HEAD request returns an HTTP/1.1 400 that contains a body. (I realize this is against the http RFC which states "the server MUST NOT return a message-body in the response" https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html - but I think we can handle this case).
Using OkHttp3 (same exact repro with OkHttp2), the HEAD request correctly indicates the return code of 400 on the HEAD request. However, after closing that connection and opening a /new/ connection as an HTTP GET, that Response fails to parse the StatusLine. Instead, the String that is parsed as the status line of request #2 (the GET request), is actually the body of request #1, the HEAD request.
I've ensured that the HEAD request connection is closed before issuing the GET request. I've also disabled caching on the OkHttp client. But nothing is resolving this.
To help debug, here's some same curl requests for the url in question
curl of the url with the HEAD request:
$ curl -X HEAD -i http://wtmd-ice.streamguys1.com/wtmd.m3u
HTTP/1.0 400 Bad Request
Content-Type: text/html

unknown request

And a curl of the GET request:
# curl -X GET -i http://wtmd-ice.streamguys1.com/wtmd.m3u
HTTP/1.0 200 OK
Content-Type: audio/x-mpegurl

http://wtmd-ice.streamguys1.com:8000/wtmd

The exact exception thrown trying to parse the StatusLine for the GET (second) request is:
java.net.ProtocolException: Unexpected status line: unknown request

In essence, the simplified code looks like:
Request request = createRequestBuilder(url).head().build();
Response response = okhttpclient.newCall(request).execute();
if (response.code() < 200 || response.code() > 299) {
    response.close();
    request = createRequestBuilder(url).build();
    response = okhttpclient.newCall(request).execute(); // <- it's this line that's throwing the ProtocolException
}

And the OkHttp logging output:
HEAD request:
01-08 13:42:48.576 27221 27300 I okhttp3.OkHttpClient: --> HEAD http://wtmd-ice.streamguys1.com/wtmd.m3u http/1.1
01-08 13:42:48.576 27221 27300 I okhttp3.OkHttpClient: Cache-Control: no-cache
01-08 13:42:48.576 27221 27300 I okhttp3.OkHttpClient: User-Agent: <my user agent>
01-08 13:42:48.576 27221 27300 I okhttp3.OkHttpClient: --> END HEAD
01-08 13:42:48.940 27221 27300 I okhttp3.OkHttpClient: <-- 400 Bad Request http://wtmd-ice.streamguys1.com/wtmd.m3u (363ms)
01-08 13:42:48.940 27221 27300 I okhttp3.OkHttpClient: Content-Type: text/html
01-08 13:42:48.941 27221 27300 I okhttp3.OkHttpClient: <-- END HTTP

And the subsequent GET request:
01-08 13:42:48.941 27221 27300 I okhttp3.OkHttpClient: --> GET http://wtmd-ice.streamguys1.com/wtmd.m3u http/1.1
01-08 13:42:48.941 27221 27300 I okhttp3.OkHttpClient: Cache-Control: no-cache
01-08 13:42:48.941 27221 27300 I okhttp3.OkHttpClient: User-Agent: <my user agent>
01-08 13:42:48.941 27221 27300 I okhttp3.OkHttpClient: --> END GET
01-08 13:42:48.947 27221 27300 I okhttp3.OkHttpClient: <-- HTTP FAILED: java.net.ProtocolException: Unexpected status line: unknown request

I haven't yet, but I will and will report back when I hear back.
Clearing the connection pool works:
okhttpclient.connectionPool().evictAll();

I also found another workaround, via source inspection(!)
requestBuilder.addHeader("Connection", "close");

Personally, I think I'm going to go with the second workaround, so that these bad HEAD requests can be isolated from a latency standpoint as well, and not dump the entire cached connection pool, which might be reused on a subsequent (GET) request.
So I have a workaround, but what else does this bug uncover? Is it concerning that, even in this "error" case on connection 1, that that still has a downstream impact on subsequent connections. In other words, it seems like state and/or data is not properly being cleared and resources recycled across connections.

Yeah, that's sort of the problem with HTTP connection pooling. It's rarely a problem in practice because webservers usually fix their bugs.

But even after the connection is explicitly closed, why does (what i'm assuming is) some part of the connection's socket's buffer remain viable? Shouldn't an explicit close on a connection clear all of the underlying resources?
Even if it's a bug on the webserver, can't we do better than that and proactively catch this case in okhttp? I'm assuming this particular webserver is backed by a whole streaming content platform provided as a web offering to many customers (https://streamguys.com), so I would assume this is actually a systemic problem that all consumers of at least this one web platform will have.
It's just really concerning to me that resources like this are not managed cleanly, which makes any given request actually non-deterministic: a HEAD followed by a GET will fail while a single GET or a GET followed by a GET will succeed.
What does an actual fix look like in okhttp?

I don’t think there’s any satisfying fix for this in OkHttp, or in any HTTP client to be completely honest.
We could use available() to preview if the stream has unexpected bytes before issuing a request. This would add an extra system call to all HTTP requests, most of which aren’t going to be fighting broken webservers. And it wouldn’t necessarily work reliably.


Modified XMLToSARIF.sh script to support call parasoft-report-transformer

I think we should add a .gitattributes file to indicate that *.sh file using LF as line separator since I can't directly run this script after I pull it from GitHub.