[SPARK-18609][SQL][WIP]Fix when CTE with Join between two table with same column

What changes were proposed in this pull request?
After optimize the SQL with CTE and Join between two tables with the same column name,  there will throw a exception when the Physical Plan executed.
SQL:
>spark.sql("CREATE TABLE p1 (col STRING)"
>spark.sql("CREATE TABLE p2 (col STRING) "
>spark.sql("set spark.sql.crossJoin.enabled = true")
>spark.sql("WITH CTE AS (SELECT s2.col as col FROM p1 CROSS JOIN (SELECT e.col as col FROM p2 E) s2) SELECT T1.col as c1, T2.col as c2 FROM CTE T1 CROSS JOIN CTE T2")

before fixed the SQL explained:
== Physical Plan ==
*Project [col#123 AS c1#104, col#126 AS c2#105]
+- CartesianProduct
   :- CartesianProduct
   :  :- HiveTableScan MetastoreRelation default, p1
   :  +- HiveTableScan [col#123], MetastoreRelation default, p2, E
   +- *!Project [col#123 AS col#126]
      +- CartesianProduct
         :- HiveTableScan MetastoreRelation default, p1
         +- HiveTableScan MetastoreRelation default, p2, E

when execute the Physical Plan above, it will throw exception because of there is no attribute for col#123 to bind( the rules ColumnPruning/RemoveAliasOnlyProject are the root cause):
org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, tree: col#123 	at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:56) 	at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:88) 	at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:87) 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286) 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286) 	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69) 	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:285) 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:291) 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:291) 	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:328) 	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:186) 	at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildren(TreeNode.scala:326) 	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:291) 	at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:275) 	at org.apache.spark.sql.catalyst.expressions.BindReferences$.bindReference(BoundAttribute.scala:87) 	at org.apache.spark.sql.execution.ProjectExec$$anonfun$4.apply(basicPhysicalOperators.scala:61) 	at org.apache.spark.sql.execution.ProjectExec$$anonfun$4.apply(basicPhysicalOperators.scala:60) 	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) 	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) 	at scala.collection.immutable.List.foreach(List.scala:381) 	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
after fixed the SQL explained:
== Physical Plan ==
*Project [col#12 AS c1#0, col#12 AS c2#1]
+- CartesianProduct
   :- CartesianProduct
   :  :- HiveTableScan MetastoreRelation default, p1
   :  +- HiveTableScan [col#12], MetastoreRelation default, p2
   +- CartesianProduct
      :- HiveTableScan MetastoreRelation default, p1
      +- HiveTableScan [col#12], MetastoreRelation default, p2

and the result is
+---+---+
| c1| c2|
+---+---+
+---+---+

How was this patch tested?
unit test added

Test build #70021 has started for PR 16255 at commit a85db5e.

Test build #70021 has finished for PR 16255 at commit a85db5e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70021/
Test FAILed.

Merged build finished. Test FAILed.

@windpiger I am not sure that this is an analysis problem. I suspect that optimizer is messing up the query. Could you pinpoint the rule that causes the issue? Tip: in the spark shell (./build/sbt sparkShell) use sc.setLogLevel("TRACE"), and that will print all rules being applied.

Test build #70048 has started for PR 16255 at commit f8d602a.

ok, the optimizer rule apply as follows:

before optimizer:

Project [col#7 AS c1#4, col#10 AS c2#5]
+- Join Cross
:- Project [col#6 AS col#7]
:  +- Join Cross
:     :- MetastoreRelation default, p1
:     +- Project [col#9 AS col#6]
:        +- MetastoreRelation default, p2
+- Project [col#6 AS col#10]
+- Join Cross
:- MetastoreRelation default, p1
+- Project [col#9 AS col#6]
+- MetastoreRelation default, p2

after first ColumnPruning:

Project [col#7 AS c1#4, col#10 AS c2#5]
+- Join Cross
:- Project [col#6 AS col#7]
:  +- Join Cross
:     :- Project
:     :  +- MetastoreRelation default, p1
:     +- Project [col#9 AS col#6]
:        +- MetastoreRelation default, p2
+- Project [col#6 AS col#10]
+- Join Cross
:- Project
:  +- MetastoreRelation default, p1
+- Project [col#9 AS col#6]
+- MetastoreRelation default, p2

after first RemoveAliasOnlyProject:

Project [col#6 AS c1#4, col#10 AS c2#5]
+- Join Cross
:- Join Cross
:  :- Project
:  :  +- MetastoreRelation default, p1
:  +- Project [col#9 AS col#6]
:     +- MetastoreRelation default, p2
+- Project [col#6 AS col#10]
+- Join Cross
:- Project
:  +- MetastoreRelation default, p1
+- Project [col#9 AS col#6]
+- MetastoreRelation default, p2
`

after second ColumnPruning:

Project [col#6 AS c1#4, col#10 AS c2#5]
+- Join Cross
:- Join Cross
:  :- Project
:  :  +- MetastoreRelation default, p1
:  +- Project [col#9 AS col#6]
:     +- MetastoreRelation default, p2
+- Project [col#6 AS col#10]
+- Join Cross
:- Project
:  +- MetastoreRelation default, p1
+- Project [col#9 AS col#6]
+- MetastoreRelation default, p2

after second RemoveAliasOnlyProject:

Project [col#9 AS c1#4, col#10 AS c2#5]
+- Join Cross
:- Join Cross
:  :- Project
:  :  +- MetastoreRelation default, p1
:  +- MetastoreRelation default, p2
+- !Project [col#9 AS col#10]
+- Join Cross
:- Project
:  +- MetastoreRelation default, p1
+- Project [col#9 AS col#6]
+- MetastoreRelation default, p2

after third ColumnPruning:

Project [col#9 AS c1#4, col#10 AS c2#5]
+- Join Cross
:- Join Cross
:  :- Project
:  :  +- MetastoreRelation default, p1
:  +- MetastoreRelation default, p2
+- !Project [col#9 AS col#10]
+- Join Cross
:- Project
:  +- MetastoreRelation default, p1
+- Project
+- Project
+- MetastoreRelation default, p2

here

+- !Project [col#9 AS col#10]
+- Join Cross
:- Project
:  +- MetastoreRelation default, p1
+- Project
+- Project
+- MetastoreRelation default, p2

col#9 has no attribute to bind when do BindRefference

after CollapseProject:

Project [col#9 AS c1#4, col#10 AS c2#5]
+- Join Cross
:- Join Cross
:  :- Project
:  :  +- MetastoreRelation default, p1
:  +- MetastoreRelation default, p2
+- !Project [col#9 AS col#10]
+- Join Cross
:- Project
:  +- MetastoreRelation default, p1
+- Project
+- MetastoreRelation default, p2

when optimizer finished, the logic plan is

Project [col#9 AS c1#4, col#10 AS c2#5]
+- Join Cross
:- Join Cross
:  :- Project
:  :  +- MetastoreRelation default, p1
:  +- MetastoreRelation default, p2
+- !Project [col#9 AS col#10]
+- Join Cross
:- Project
:  +- MetastoreRelation default, p1
+- Project
+- MetastoreRelation default, p2
`

Test build #70048 has finished for PR 16255 at commit f8d602a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70048/
Test PASSed.

@hvanhovell please help to review this, thanks a lot!

Test build #70114 has started for PR 16255 at commit 0413f9d.

Test build #70114 has finished for PR 16255 at commit 0413f9d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70114/
Test PASSed.


add .direnv to default exclude argument

PR Checklist

[x] A description of the changes is added to the description of this PR.
[x] If there is a related issue, make sure it is linked to this PR.
[ ] If you've fixed a bug or added code that should be tested, add tests!
[x] Documentation in docs is updated

Description of changes

The default list of excludes makes me think that we could benefit from ignoring all files that are in .gitignore, similarly to what black does.
If you think that it makes sense, I'd be happy to create an issue for it.

The default list of excludes makes me think that we could benefit from ignoring all files that are in .gitignore, similarly to what black does.
If you think that it makes sense, I'd be happy to create an issue for it.

@mkniewallner That definitely makes sense :) Would have also prevented the related issue from popping up indeed.


Support diverging inclusive gateway

Problem you would like to solve
The diverging inclusive gateway (i.e. OR gateway) is added to Zeebe in https://github.com/camunda/zeebe/pull/9747 as an external contribution. Of course, all the Camunda Platform 8 need to be aligned and support the feature.
If the Modeler doesn't support modeling the inclusive gateway for Camunda Platform 8, then our customers cannot use this new feature.
Proposed solution
Please support the diverging inclusive gateway for Camunda Platform 8. The behavior is specified in the PR.
At a minimum:

a gateway can be configured to the Inclusive Gateway type
Conditions can be configured for outgoing sequence flows of the inclusive gateway
a Default outgoing sequence flow can be chosen for the inclusive sequence flow

Nice to have:

validations are added for:

an outgoing sequence flow of an inclusive gateway doesn't have a condition and the inclusive gateway has more than 1 outgoing sequence flow
there are multiple sequence flows targeting the inclusive gateway

Alternatives considered

Customers can update the XML by hand
Customers can use combinations of XOR and AND gateways to rebuild the OR gateway logic.

Additional context

https://github.com/camunda/product-hub/issues/214

PS: The Camunda Cloud label should probably be updated to Camunda Platform 8

TODO

[x] add inclusive gateway to replacement options (camunda-bpmn-js)

https://github.com/camunda/camunda-bpmn-js/pull/162

[x] show Condition group in properties panel if sequence flow source is inclusive gateway (bpmn-js-properties-panel)

https://github.com/bpmn-io/bpmn-js-properties-panel/pull/756

[x] add lint rules (bpmnlint-plugin-camunda-compat)

[x] inclusive gateway only supported in 8.1
[x] inclusive gateway must not have more than one incoming sequence flow ("converging OR gateway not supported")
[x] if inclusive gateway has more than one outgoing sequence flow each must have condition (i.e. Conditions are required, unless the OR gateway is not diverging)

@korthout to reiterate on what you say:

Forking OR gateway is supported, converging OR gateway is NOT supported

That means:

We allow modeling of OR gateway (replace)
We add dedicated lint rules:

OR gateway only supported in 8.1
OR gateway must not have more than one incoming sequence flow ("converging OR gateway not supported")

@nikku That sounds great.
Did you also consider a lint rule for:

OR gateway with more than one outgoing sequence flow must have a Condition for each of these flows (i.e. Conditions are required, unless the OR gateway is not diverging)

@korthout Added as a rule.

Thanks @skayliu for bringing attention to this.
The linting rules were actually the reason that camunda/zeebe#10310 was discovered. But I forgot to document it. Sorry about that.

may be the lint rule shold make some adjust

As you've identified, lets fix this in Zeebe, and not workaround in our lint rules.

@philippfromme please double-check but I believe with https://github.com/camunda/camunda-modeler/pull/3150 merged this can be closed?

@nikku Yes, we can finally close this issue. 🎉


[request]: aws-sig-auth - Support for omit_session_token for iotdevicegateway

A note for the community

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue, please leave a comment

Tell us about your request
I would like support for omit_session_token here: https://github.com/awslabs/smithy-rs/blob/ab0306b2a178ccd236b217c06de684172b667f70/aws/rust-runtime/aws-sig-auth/src/signer.rs#L88-L96 and here: https://github.com/awslabs/aws-sdk-rust/blob/63fc7ab18c9fdd7b6f981d5d1bf223ced0225e05/sdk/aws-sig-auth/src/signer.rs#L88-L96
And obviously in the underlying implementation in aws-sigv4
Tell us about the problem you're trying to solve.
It is currently not possible to create a presigned SigV4 request for certain services, due to the fact that they need to exclude the X-Amz-Security-Token from the query string used for calculating the signature.
As per the documentation note here: http://docs.aws.amazon.com/general/latest/gr/sigv4-add-signature-to-request.html.

When you add the X-Amz-Security-Token parameter to the query string, some services require that
you include this parameter in the canonical (signed) request. For other services, you add this parameter
at the end, after you calculate the signature. For details, see the API reference documentation for that
service.

The only service i have yet to encounter is iotdevicegateway, but judging from AWS's documentation note it appears as if there could be more.
Are you currently working around this issue?
We are currently using a patched version of rusoto for this, which is the last thing blocking us from migrating fully to aws-sdk-rust
Additional context
This was also part of the original feature request for AWS IoT in https://github.com/awslabs/aws-sdk-rust/issues/115, but seems to have been missed.

ah thanks! We tried to figure out why this was actually needed but couldn't reproduce it. Presigning was the missing link. We'll get this fixed (and would also happily accept a PR)

I am a bit unsure if the aws_sdk_iotdataplane client uses Sigv4 presigned connections, but if it does this might be the underlying issue behind https://github.com/awslabs/aws-sdk-rust/issues/468#issuecomment-1055389607 as well?

Awesome! That definitely makes it a bit easier :+1:

currently aws_sdk_iotdataplane doesn't model any presigned operations, but that is also not a huge lift I think. There is actually no universal model of what operations can be presigned, so generating presigned operations is accomplished by manually augmenting the model: https://github.com/awslabs/smithy-rs/blob/main/aws/sdk-codegen/src/main/kotlin/software/amazon/smithy/rustsdk/AwsPresigningDecorator.kt#L63-L80
We would also need to add a configuration to the PresignedTrait to mark operations as omitting the X-Amz-Security-Token (In the meantime though, you can always work around the issue by manually using aws-sig-auth)

(In the meantime though, you can always work around the issue by manually using aws-sig-auth once it supports omit_session_token)

Yeah, that is exactly what i am doing now:
let region_provider = RegionProviderChain::default_provider().or_else("eu-west-1");
let credentials_provider = CredentialsProviderChain::default_provider().await;
let region = region_provider
    .region()
    .await
    .ok_or(anyhow!("No region?"))?;

let config = aws_config::from_env().region(region_provider).load().await;
let iot_client = aws_sdk_iot::Client::new(&config);

let endpoint_address = iot_client
    .describe_endpoint()
    .endpoint_type(String::from("iot:Data-ATS"))
    .send()
    .await?
    .endpoint_address
    .ok_or(anyhow!("No endpoint address?"))?;

let port = 443;

let request_config = RequestConfig {
    request_ts: SystemTime::now(),
    region: &SigningRegion::from(region),
    service: &SigningService::from_static("iotdevicegateway"),
    payload_override: None,
};

let uri = Uri::builder()
    .scheme("wss")
    .authority(format!("{}:{}", endpoint_address, port))
    .path_and_query("/mqtt")
    .build()?;

let mut request = Request::builder()
    .method("GET")
    .uri(uri)
    .body(SdkBody::empty())?;

let mut operation_config = OperationSigningConfig::default_config();
operation_config.signature_type = HttpSignatureType::HttpRequestQueryParams;
operation_config.expires_in = Some(Duration::from_secs(86400));

// FIXME: Currently missing! (https://github.com/awslabs/aws-sdk-rust/issues/477)
// operation_config.signing_options.omit_session_token = true;

SigV4Signer::new()
    .sign(
        &operation_config,
        &request_config,
        &credentials_provider.provide_credentials().await?,
        &mut request,
    )
    .map_err(|e| anyhow!("{:?}", e))?;

let mut opts = MqttOptions::new(
    format!("bbctl-{}", uuid::Uuid::new_v4()),
    request.uri().to_string(),
    port,
);

you might be able to work around it by dropping the header, signing, then re-adding the header

@itsnottoolate you are more than welcome to see if you can make a PR tackling this 👍

Any updates on this issue?

Hey @MathiasKoch, We don't have any updates to share on this issue as we haven't gotten around to looking at it. We prioritize issues based on the number of upvotes they receive (counted in 👍s on the initial issue description.) If you'd like us to get to this issue sooner, encourage people to upvote it. You can check here to see which issues are currently our highest priority.

This was fixed by https://github.com/awslabs/smithy-rs/pull/2473


Fixed "File not found in working directory" error.

When you try to run the file from the command line, PyGame throws an error saying that the assets folder does not exist. This fixes that issue by changing the working directory to the current file being executed.

Thanks


10958: Set metadata so cloud attachments download properly

When using cloud storage, now instead of rendering a file named 20200729-9391-lk1wg1 in the browser, it downloads a file named fakemission7718-responses-2020-07-30-0403.csv as expected.

LGTM. Do existing Operations need to be migrated?

I experimented on staging and I don't think we can improve the behavior for legacy Operations without writing more complex code to change filenames and metadata on S3 (the new code I wrote causes fog to save the data correctly when uploading the file initially).
It doesn't seem worthwhile since this only affects CEJP and Staging, but I will mention to Dottie in case she wants it.

Good reasoning!
On Fri, Jul 31, 2020 at 12:04 PM Kevin Cooper<EMAIL_ADDRESS>wrote:

I experimented on staging and I don't think we can improve the behavior
for legacy Operations without writing more complex code to change filenames
and metadata on S3 (the new code I wrote causes fog to save the data
correctly when uploading the file initially).
It doesn't seem worthwhile since this only affects CEJP and Staging, but I
will mention to Dottie in case she wants it.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/thecartercenter/nemo/pull/751#issuecomment-667200132,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFLO4HSJOKE5X4E3H4ERILR6LTPFANCNFSM4PMX2MJA
.

--
Tom Smyth
Worker-Owner, Sassafras Tech Collective
Specializing in innovative, usable tech for social change
sassafras.coop · @sassafrastech
Pronouns: he/him


tests: add dcerpc test per #3109

Previous PR: https://github.com/OISF/suricata-verify/pull/799
Changes since v2:

sid checks
added readme
uncommented second rule
min version check for 6.0 as 5 is near EOL and it seems complicated to fix there

Merged in #852


Antispamtrigger error.

http://imgur.com/1LEn9Py

Fixed. I also added a default-off 'debug' option that prevents spamming the 'reducing scores' message. (If you want to see that, you'll need to turn it on).
Next time, please copy the trace and paste it into the comment box, then either indent each line with four spaces or put 3 backticks (```) on the lines immediately before and after the trace. it's way more readable; you can preview formatting by clicking the preview tab in the comment box. If you don't want to put the entire trace into the comment, then you can put it in a gist and link to it.


keep the last tooltip display after clicking the end.

I want to keep the last tooltip display after clicking the end.

You can customize the behavior.
Please check #868.


[Suggestion] Remove the action "Delete Environment…" from production context menu

OS: All
Build Version: 20200730.10
Repro steps:

Create a Static Web App.
Right click the "Production" node.
Select "Delete Environment...".
Check the result.

Actual:
Pop up an error "Cannot delete the production environment directly".

Suggestion:
Remove the action "Delete Environment…" from production context menu.

I want to keep the error because it could be confusing to users as to why the delete environment context menu item isn't there.  However we should remove "Report an issue" from the error dialog.


input folder missing

Thank you for sharing the repo. The input folder is missing --> /input/vnlicenseplate/

Yes it is. You have to create it yourself. A folder contains testing
images. Github doesnt allow me to upload that big data
On Fri, Mar 17, 2023, 06:13 LarousseKhosravi @.***>
wrote:

Thank you for sharing the repo. The input folder is missing -->
/input/vnlicenseplate/
—
Reply to this email directly, view it on GitHub
https://github.com/mrzaizai2k/License-Plate-Recognition-YOLOv7-and-CNN/issues/3,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJYP3L5O4N5EL2Y5K2PUF5TW4ONDLANCNFSM6AAAAAAV524LNQ
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***
com>

Thank you so much for the reply. It is resolved now. I am closing the issue.

You are welcome.
On Fri, Mar 17, 2023, 12:09 LarousseKhosravi @.***>
wrote:

Thank you so much for the reply. It is resolved now. I am closing the
issue.
—
Reply to this email directly, view it on GitHub
https://github.com/mrzaizai2k/License-Plate-Recognition-YOLOv7-and-CNN/issues/3#issuecomment-1473139638,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJYP3L76YH7ZBNYQDNRNTN3W4PWYTANCNFSM6AAAAAAV524LNQ
.
You are receiving this because you commented.Message ID:
<mrzaizai2k/License-Plate-Recognition-YOLOv7-and-CNN/issues/3/1473139638@
github.com>


Set Shipping Filter To User's Country by Default

On the store page, the shipping filter should default to the country of the user's first saved address, or the country set in the Settings/General tab if they have no saved addresses.
This is a sub-task from #80.

In a discussion with @morebrownies , we decided the default Ships To should be (Any country), otherwise the default view of the store is a filtered result set. The thinking was it's less than ideal if for someone to see all the listings in a store, they would need to clear some filter(s).

Ok. I'll update #80 to reflect that.


Fix/check tests under Mono

Some tests are excluded for Mono, because they are failing (Linux); See https://github.com/NLog/NLog/pull/944
Check the unit tests and try too fix them, or give a good reason why the unit test or code is wrong.

I would take a look at this, if no objections.


native-ext

Native messaging, already built into the browsers, is what gives unfettered access to the system--native-ext is just allowing Node.js to be used more easily in this capacity.
Yes, there are risks with installing native messaging apps (which requires their own separate install), but there are risks when installing any native app. The availability of highways expose users to the danger of accidents, and safety should be built into the system as much as possible, but users choose to accept the risks for the benefits they can get. "With power comes great responsibility"... While the statement is true, it doesn't mean that having power--whether as a user or developer--is therefore undesirable. :-)
If one wants to build the likes of a file browser, WebExtensions might never allow this otherwise...
I was probably a bit premature in offering consideration of native-ext specifically though, as I haven't yet used it myself. I followed the procedure of implementing my own Native messaging add-on with pkg (FWIW, I did find https://github.com/jdiamond/chrome-native-messaging very helpful in this regard). It just seems wasteful to be bundling Node with each add-on and I was glad someone provided a mechanism for allowing back JavaScript-based access to XUL-like power, not to mention without the apparent need to rebuild every time one changes one's Node code.

Native messaging, already built into the browsers, is what gives unfettered access to the system--native-ext is just allowing Node.js to be used more easily in this capacity.

No it doesn't, it can only send and receive messages, not run code directly.
This is not the same as just running a native app: you're potentially opening any website to run any code if an extension somehow XSS itself.


Fixed the managed policy permission link

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Thanks for providing this update - much appreciated!


Nacelle yaw control

I am trying to set the nacelle yaw control for the 15MW floating wind turbine.
I am struggling a bit to understand what each of the parameters in the yaw rate control stands for.
Can you tell me where can I find an explanation for the yaw control, and how to set the slow and fast low pass filters?
Thanks in advance.

Hi,
You can review the input file descriptions here: https://github.com/NREL/ROSCO/blob/51629665d22f92770f861daeebcebd47ab32abb1/Test_Cases/IEA-15-240-RWT-UMaineSemi/DISCON-UMaineSemi.IN#L96
Our yaw controller is based on the one presented in this article: https://iopscience.iop.org/article/10.1088/1742-6596/1037/3/032011
Please follow up in this repository, if you have additional questions.  The ROSCO_Toolbox repo is depreciated.
Best, Dan


How does the error budget calculation work?

slo:sli_error:ratio_rate30d is calculated as
sum_over_time(slo:sli_error:ratio_rate5m[30d])
/
count_over_time(slo:sli_error:ratio_rate5m[30d])

slo:sli_error:ratio_rate30d is then used to calculate slo:period_burn_rate:ratio and slo:period_error_budget_remaining:ratio.
If I'm reading that right, doesn't that mean we are taking all the 5 minute windows and averaging them together with equal weight? If it were truly events-based, then each window should be weighted by the number of requests within that window, but this seems like we are averaging an average. Or am I misunderstanding how these aggregations work?

Hi @jimethn
Those are optimized rules for 30d windows that lots of prometheus installations end dying there (rule timeout, expensive queries...), that's why I did that, you get the less precision metric instead of not getting anything or killing Prometheus.
If you want the real value and make the expensive queries you can use --disable-optimized-rules

Makes sense, thank you!


[WEB] Make pages responsive to zoom in and out - resize proportionally

Hi team,
the objective would be to make the page zoom in and out proportionally responsive to it.
The description can be seen here.
This is 100%

This 175%:

On this scenario on Chrome you can see the "And" text on the blue square comes up. It would be great the text format the same way.
Also if you get it to 200% the text "info" from "Covid-info" goes down:

Probably thats not something a person is doing all the time for sure, but in it seems a good practice/quality thing to have it (e.g. you can do it in linkedin page without login and you can see the proper resize and downsize of the page).
Also when you do the zoom out you see some differences:

Also i noticed now that we have a scroll on the page to see "View Community Postings" text.
Is this something we can achieve with this responsive page?
thanks ,
Tiago

I don't think zooming in and out is part of the responsive development. If we were to accumulate this use case we'll have to spend a long time trying to develop for every screen size and zoom %.

Seems to me that this is already responsive text sizing when I try to resize the viewport size
I can understand the "art direction" part of the design, similar to responsive images and how they resize across different screens, but for zooming I'm not sure if that will work as well, in this case I think maybe it can be adjusted with the container size around the title... this also depends on the type of units that are used, %, px, em / rems... etc.

@snowiesuet @robinv85 , thanks for the feedback.
I understand your points of view here. Probably for MVP can be too much, but something for sure to take in consideration for future, and break down this in multiple tasks. What do you think?
At this stage, and because we want to launch quickly i would say to try to put it better in the Home Page (fix size and images), in parallel we can do the work for other screens, just to see if there isn't a major layout issue with the zooming.
Meanwhile to try to make even more clear i did the bellow video with linkedin.
I selected part of a text that goes up and down a litle bit when you do a zoom, but most of the page grow proportionally, the images, the text (do not jump to another paragraph).
linkedin.zip
@robinv85, @snowiesuet Please give your thoughts on this?
Am i asking too much for this phase and maybe for later? or is this a level of detail in design not to care about because.. why? Probably i can be missing some context also.
Thanks,
Tiago

Hi guys,
so whats the verdict ? :) @robinv85 , @snowiesuet

I think this would be good for future release but for now I don't think it's a priority, because this doesn't just affect homepage, it'll affect all pages.

Yeah this is definitely not a priority right now... we use rems in lots of places so I think that would probably already be a pretty good part of making everything zoomable? But plenty of other things that are more high priority

Perfectly agree with you @robinv85 and @snowiesuet on this that at this moment we want to launch this can be taken care in the future release. For sure we can just double check with other PMs, but i think there will be a common agreement on this. Thanks for your feedback ;)

@tiagocostasilva , based on this thread, I'm moving this down the priority list for MVP-Release 1. Feel free to remove the MVP-Release 1 label altogether if you'd like.

I'm closing this - it's old, low priority and the resolution is unclear. If we get feedback from a user we can revisit.


Fix flaky test by resolving change note accepting empty timestamps

This provides a fix for a test that fails in about 10% of Publishing API builds. The test relies on random data and thus exposes an issue in the code:
  1) Message queue publishing puts the correct message on the queue
     Failure/Error: expect(errors).to eql([])

       expected: []
            got: [{:failed_attribute=>"TypeV4", :fragment=>"#/details/change_history/0/public_timestamp", :message=>"T...76f-aca8114b1247#", :schema=>#<Addressable::URI:0x16c60 URI:a4923dd9-f21e-5f89-b76f-aca8114b1247#>}]

       (compared using eql?)

       Diff:
       @@ -1,6 +1,11 @@
       -[]
       +[{:failed_attribute=>"TypeV4",
       +  :fragment=>"#/details/change_history/0/public_timestamp",
       +  :message=>
       +   "The property '#/details/change_history/0/public_timestamp' of type null did not match the following type: string in schema a4923dd9-f21e-5f89-b76f-aca8114b1247#",
       +  :schema=>
       +   #<Addressable::URI:0x16c60 URI:a4923dd9-f21e-5f89-b76f-aca8114b1247#>}]

     # ./spec/integration/message_queue_publishing_spec.rb:36:in `block in ensure_message_queue_payload_validates_against_notification_schema'
     # ./spec/integration/message_queue_publishing_spec.rb:30:in `ensure_message_queue_payload_validates_against_notification_schema'
     # ./spec/integration/message_queue_publishing_spec.rb:17:in `block (3 levels) in <top (required)>'
     # ./spec/spec_helper.rb:85:in `block (3 levels) in <top (required)>'
     # ./spec/spec_helper.rb:85:in `block (2 levels) in <top (required)>'

The cause of this issue is that Publishing API will present change notes to the frontend and message queue when a change note does not have a timestamp. This is invalid as the schema dictates that a change history will have a timestamp.
The fix for this is two-fold:

Accept that in the draft process we can have change notes without a timestamp and filter them out of data sent to frontend
At the point of publishing set any change notes that are missing a timestamp.

I also had to refactor some funky tests (that I wrote 🤦) to be able to implement it.
In practice this issue seems to have caused a very limited impact on the Publishing API with only about 40 documents existing with invalid change notes. This is likely because the change note feature of Publishing API is rather niche and not used by Whitehall and other big Publishing apps.
More info in the commits and definitely easiest reviewed commit by commit.
⚠ This repo is Continuously Deployed: make sure you follow the guidance ⚠

Thanks for looking at this! I actually created a card for this a few weeks ago if you want some trello satisfaction.
Fix for second bug here looks good. There's something about the first fix that feels instinctively off to me - may well be the case that this is the best way, have tried to explain what doesn't quite sit right with me above.

Thanks @BeckaL I'm not too concerned as their drafts so will plan to go ahead without.


Add connector for Pravega

https://www.pravega.io/
Similar to Kafka connector, it will be great to add a connector for Pravega.
Some info on pravega vs Kafka

@mcvsubbu @npawar and I had an initial discussion around this topic but our understanding of Pravega is a bit weak to propose a complete solution. Flavio, it will be great if you can drive this discussion and guide us in the right direction.

@fpj

There is some prior work we did to extend support for other streaming systems (this was done keeping Kinesis in mind).
#5359 Extend offset support for other streams

Thanks for starting this issue, @kishoreg. I'm wondering what would be the best way to approach the implementation of a connector. I started by looking at the spi interfaces, and in particular at the StreamLevelConsumer. Here is the reason.
Pravega has as a few different APIs, the main one being the event stream API. it also has a batch API that enables unordered reads for parallelism, but let's focus on the event stream API for now. A Pravega stream comprises a set of parallel segments, and that set can change over time according to a scaling policy. We do not expose the complexity of dealing with the set of segments changing, and let the readers in a group coordinate internally the assignment of segments, respecting order.
Given that segments aren't clearly exposed in the event stream API, the StreamLevelConsumer interface seems to provide the right level of abstraction, except for the commit call. We don't really provide the ability to commit per reader, like Kafka provides the ability to commit per consumer. Pravega reader groups instead produce checkpoints, which are consistent collections of offsets for segments currently being read. The application sees checkpoints.
Our approach to recording positions of one or more streams is consequently more coarse-grained and coordinated across the group. I wanted to understand how I can introduce checkpoints given that the interface expects a commit implementation.
I'd love to get some input and hopefully some ideas.

@mcvsubbu @npawar @KKcorps thoughts?

A Pravega stream comprises a set of parallel segments, and that set can change over time according to a scaling policy - is "set of segments" like partitions within the stream? Can we use them to instead add a PartitionLevelConsumer implementation?
The reason I ask is because we have not been steering away from StreamLevelConsumer for all practical purposes, because it is unfriendly for rebalance, capacity changes.
Is there any doc where i can read the detailed design of Pravega stream APIs and particularly the checkpointing?

Hi @npawar,  I missed your message, sorry about that.

is "set of segments" like partitions within the stream?

yes, but we do not expose segments individually. a group of readers coordinate among themselves to assign enabled segments. keep in mind that the set of segments in a pravega stream can change over time because of stream scaling.

Can we use them to instead add a PartitionLevelConsumer implementation?

We can't with the stream API because of the reason I stated above. The Batch API, which does not follow any order of segments and simply gives iterators for a stream, can be used, is more aligned with what you are asking for, but it is not event driven. The iterators will read the events up to a point in the stream.

Is there any doc where i can read the detailed design of Pravega stream APIs and particularly the checkpointing?

You may want to explore the documentation here:
https://pravega.io/docs/latest/
and in particular the "Developing Pravega Applications" section. You may also want to check the javadocs:
https://pravega.io/docs/latest/javadoc/clients/io/pravega/client/stream/ReaderGroup.html#initiateCheckpoint-java.lang.String-java.util.concurrent.ScheduledExecutorService-
The best example currently for the use of checkpoints is Apache Flink.


Make the lab work

Overview
This makes the lab work.
All tools should now be functional.
The front-end still requires some TLC to smooth out the experience.
Checklist
~- [ ] Styleguide updated, if necessary~
~- [ ] Swagger specification updated, if necessary~
~- [ ] Symlinks from new migrations present or corrected for any new migrations~
Notes
It's important that after you make a change to the parameters that you hit the 'Apply Parameters' button. Otherwise it wont work right. This is one of the rough edges.
Testing Instructions

Make sure you have least one project with ingested scenes
Go to the lab, try out the tools with the ingested scene project
The first render of the tiles may take longer than expected, you may hit cache memory errors

I'm going to merge this to get it into staging for testing and potentially deploying. If there are revisions we need to make we can add separate issues/tasks for them. But it looked pretty good to me in testing locally.


Use FQCN in playbooks

Use FQCN as recommended by ansible-lint.
Signed-off-by: Arnaud Patard<EMAIL_ADDRESS>

hm. no zuul in the check list :( will have to check that later...

Ansible zuul instance was replaced with a different instance. it might be linked to that.

@ssbarnea any idea of what needs to be done to get zuul back ? (hm. I guess we should rather discuss that elsewhere as zuul not working as nothing to do with this PR)


gRPC apps in Kubernetes scale correctly using gRPC client-side load balancing

Some load balancers don't work effectively with gRPC. L4 (transport) load balancers operate at a connection level, by distributing TCP connections across endpoints. Because gRPC calls happen on a single connection, all load gets sent to one server.
An example of this issue is Kubernetes. If a pod has multiple replicas, the gRPC connection will be made to one pod replica, and all calls will go to that pod replica. gRPC with default Kubernetes load balancing is broken.
There are two options to effectively load balance gRPC:

Client-side load balancing
L7 (application) proxy load balancing. For example, YARP and Envoy will distribute gRPC calls across replicas.

Proxy load balancing works today with gRPC in .NET. A lot of people will choose this option. However a proxy introduces complexity and performance penalties. Many customers have asked for support for client-side load balancing.
Other first-class gRPC implementations support client-side load balancing (grpc-go, grpc-java, C++). grpc-dotnet would aim to provide equivalent functionality.
Note: Changes to HttpClient maybe required to properly support client-side load balancing.
Reasons for client-side load balancing:

A must have feature to support load balancing with Kubernetes replicas
Provide a high-performance, proxyless alternative to using YARP/Envoy
To maintain feature parity with other first-class gRPC implementations
First-party customer demand

@JamesNK as this is parented by a Theme, perhaps it should be an Epic (or be under a gRPC epic)

Same parent as https://github.com/dotnet/core/issues/5493 perhaps.

@JamesNK any requirements on the client networking stack here?

@JamesNK any requirements on the client networking stack here?

I'm pretty sure there will be. Post 5.0 release I'll talk to @karelz about it

@JamesNK it would be good to now gain clarity here on libraries asks.

@danmosemsft I've created an issue about the big System.Net.Http feature required here - https://github.com/dotnet/runtime/issues/45246
Will discuss with @karelz @scalablecory @geoffkizer.

Bulk closing .NET 6 epics and user stories. If you think this issue was closed in error, please reopen the issue and update it accordingly.


Congratulations! Amazing script!

I will be teaching some children how to write their first words and this script is exactly what I was looking for.
Thank you so much!
Quick question: How to change the pen or ink color from black (current color) to white?
Thanks

You can set stroke style before stroke(), i.e., https://github.com/ChenYuHo/handwriting.js/blob/6d41109d3293295a7d7678d0cb97f483d28f47ec/handwriting.canvas.js#L93 and https://github.com/ChenYuHo/handwriting.js/blob/6d41109d3293295a7d7678d0cb97f483d28f47ec/handwriting.canvas.js#L140
e.g.,
this.cxt.strokeStyle = '#ff0000'; // make the following stroke red
this.cxt.stroke();

You may also add a color selector and set this.cxt.strokeStyle so that one can control the color of every stroke.

@ChenYuHo you did it again!
LOVE IT
I works beautifully


time: TestTicker timeouts after CL 232298

(Issue forked from #42237.)
panic: test timed out after 15m0s

goroutine 219 [running]:
testing.(*M).startAlarm.func1()
	C:/workdir/go/src/testing/testing.go:1680 +0xe6
created by time.goFunc
	C:/workdir/go/src/time/sleep.go:167 +0x4b

goroutine 1 [chan receive, 14 minutes]:
testing.(*T).Run(0xc000058000, 0x4cb903, 0xa, 0x545b78, 0x368a01)
	C:/workdir/go/src/testing/testing.go:1219 +0x2da
testing.runTests.func1(0xc000048f00)
	C:/workdir/go/src/testing/testing.go:1491 +0x7f
testing.tRunner(0xc000048f00, 0xc000133de0)
	C:/workdir/go/src/testing/testing.go:1173 +0xef
testing.runTests(0xc000004558, 0x674680, 0x6f, 0x6f, 0xbfe1343f8f278ecc, 0xd18c5b20e5, 0x678480, 0x4cca88)
	C:/workdir/go/src/testing/testing.go:1489 +0x310
testing.(*M).Run(0xc00001e100, 0x0)
	C:/workdir/go/src/testing/testing.go:1397 +0x1f5
main.main()
	_testmain.go:377 +0x145

goroutine 31 [chan receive, 14 minutes]:
testing.(*T).Parallel(0xc00019d680)
	C:/workdir/go/src/testing/testing.go:1039 +0x119
time_test.TestParseYday(0xc00019d680)
	C:/workdir/go/src/time/format_test.go:762 +0x52
testing.tRunner(0xc00019d680, 0x545ae8)
	C:/workdir/go/src/testing/testing.go:1173 +0xef
created by testing.(*T).Run
	C:/workdir/go/src/testing/testing.go:1218 +0x2b3

goroutine 231 [chan receive, 14 minutes]:
time_test.TestTicker(0xc000058000)
	C:/workdir/go/src/time/tick_test.go:45 +0xc5
testing.tRunner(0xc000058000, 0x545b78)
	C:/workdir/go/src/testing/testing.go:1173 +0xef
created by testing.(*T).Run
	C:/workdir/go/src/testing/testing.go:1218 +0x2b3
FAIL	time	900.029s

2020-11-05T19:22:02-370682a/windows-amd64-longtest
2020-11-05T16:47:45-a19a4dc/windows-amd64-longtest
2020-11-05T15:16:57-34c0969/windows-amd64-longtest
2020-11-02T21:08:14-4fcb506/windows-amd64-longtest
2020-11-02T15:23:43-d5388e2/linux-amd64-longtest
2020-11-02T05:29:59-f2ee58b/windows-amd64-longtest
2020-11-02T03:03:16-0387bed/windows-amd64-longtest
2020-10-30T18:06:13-6d087c8/windows-amd64-longtest
2020-10-30T16:29:11-1af388f/windows-amd64-longtest
2020-10-30T15:25:49-e1faebe/linux-386-longtest
2020-10-29T20:50:02-38d1ec8/windows-amd64-longtest
2020-10-29T19:06:32-5cc43c5/windows-amd64-longtest
2020-10-29T03:52:23-3089ef6/windows-amd64-longtest
2020-10-28T00:48:22-49b017f/windows-amd64-longtest
2020-10-28T00:25:05-40d1ec5/windows-amd64-longtest
2020-10-27T22:57:36-b4b0144/windows-amd64-longtest
CC @prattmic @ChrisHines @aclements @ianlancetaylor

This may be Windows-specific, if #42237 is sufficient to account for the failures observed on OpenBSD and NetBSD.

Oop, nope. There are two linux failures in the mix there too.

One of these failures was after http://golang.org/cl/267257, so it seems that's not the fix.
2020-11-06T23:59:04-f7ef5ca/windows-amd64-longtest

@laboger may have an easy reproducer for this on linux/amd64 (and linux/ppc64le) in #42402.

Sounds like #42402 was fixed by CL 267257, so unfortunately we don't seem to have a lead on a reliable reproducer.

Still ongoing, as expected:
2020-11-30T03:18:36-e5da18d/windows-amd64-longtest
2020-11-25T04:48:23-750b372/windows-amd64-longtest
2020-11-25T02:18:15-f6dcc97/windows-amd64-longtest
2020-11-24T19:53:18-65dcd15/linux-386-longtest
2020-11-19T04:02:59-5ba1c3f/windows-amd64-longtest
2020-11-17T15:10:45-0968d2d/linux-amd64-longtest
2020-11-14T17:24:37-92c732e/windows-amd64-longtest
2020-11-13T22:01:37-86954d5/linux-amd64-longtest
2020-11-13T15:00:23-35455ff/netbsd-386-9_0
2020-11-12T14:38:38-c167635/windows-amd64-longtest
2020-11-06T23:59:04-f7ef5ca/windows-amd64-longtest
There are also a few TestTeardown timeouts, which I suspect are related:
2020-11-28T03:50:50-358d354/windows-amd64-longtest
2020-11-02T05:29:59-f2ee58b/windows-amd64-longtest
2020-10-30T16:29:11-1af388f/windows-amd64-longtest
2020-10-29T19:06:32-5cc43c5/windows-amd64-longtest

The root cause is a timer in TestCheckRuntimeTimerOverflow, where we explictly set t.when = 0: https://cs.opensource.google/go/go/+/master:src/time/internal_test.go;l=67;bpv=0;bpt=1.
This timer is on top of the heap, and in turn, pp.timer0When = 0, and then checkTimers will decide there are no timers at all, thus missing any timers further down the heap.
I'm still looking into when we broke this, because as I recall, we've treated timer0When == 0 as "no timers" for a while.
As an aside, @mknyszek notes that we may want timer0When's sentinel value to be maxWhen to make this more obvious (regardless if it fixes the issue directly).

[5.158222300 P 2] M 4 modtimer: modify t 0xc0000661e0 when<PHONE_NUMBER>854775807 nextwhen 0 P 2
        0xec3288 [time.resetTimer+0x68 C:/workdir/go/src/runtime/time.go:611]
        0xec3244 [time.resetTimer+0x24 C:/workdir/go/src/runtime/time.go:224]
        0xef58c4 [time.CheckRuntimeTimerOverflow.func1+0x64 C:/workdir/go/src/time/internal_test.go:67]
        0xef56bb [time.CheckRuntimeTimerOverflow+0xfb C:/workdir/go/src/time/internal_test.go:78]
        0xfe5cee [time_test.TestOverflowRuntimeTimer+0x4e C:/workdir/go/src/time/sleep_test.go:482]
        0xf4de2e [testing.tRunner+0xee C:/workdir/go/src/testing/testing.go:1195]
        0xec6100 [runtime.goexit+0x0 C:/workdir/go/src/runtime/asm_amd64.s:1367]
...
[5.158222300 P 2] M 4 cleantimers: modified t 0xc0000661e0 when<PHONE_NUMBER>854775807 nextwhen 0
...
[5.158222300 P 2] P 2 dodeltimer0: t 0xc0000661e0
...
[5.158222300 P 2] P 2 doaddtimer: t 0xc0000661e0 when 0 now<PHONE_NUMBER>100 until -7187394008100 len(pp.timers) 0
... (when = 0 at top of heap)
[5.158222300 P 2] P 2 doaddtimer: t 0xc000066378 when<PHONE_NUMBER>100 now<PHONE_NUMBER>100 until 20000000 len(pp.timers) 1
        0xeacabc [runtime.addtimer+0xbc C:/workdir/go/src/runtime/time.go:260]
        0xec3191 [time.startTimer+0x31 C:/workdir/go/src/runtime/time.go:207]
        0xeeaa64 [time.NewTicker+0xe4 C:/workdir/go/src/time/tick.go:39]
        0xfe6c89 [time_test.TestTicker+0x349 C:/workdir/go/src/time/tick_test.go:42]
        0xf4de2e [testing.tRunner+0xee C:/workdir/go/src/testing/testing.go:1195]
        0xec6100 [runtime.goexit+0x0 C:/workdir/go/src/runtime/asm_amd64.s:1367]
[5.158222300 P 2] P 2 doaddtimer done: t 0xc000066378 len(pp.timers) 2 timer0When 0 numTimers 2
... (added new timer, timer0When still 0)
[5.158222300 P 2] M 4 checkTimers: P 2
[5.158222300 P 2] M 4 checkTimers: no timers: timer0When 0 timerModifiedEarliest 0
... (no timers!)

Bisect shows that this was broken by golang.org/cl/258303 (cc @ianlancetaylor). If I understand correctly, previously this happened:

TestCheckRuntimeTimerOverflow timer sets nextwhen = 0 in modtimer, and increments adjustTimers.
checkTimers sees adjustTimers != 0, adjusts timer, and then immediately runs the when = 0 timer, removing it from the heap.

Now we've only updated timerModifiedEarliest. Since timer0When and timerModifiedEarliestare 0,checkTimerswill never run thewhen = 0` timer, causing the stuckness.
Now, how to fix? The simple fix is to change https://cs.opensource.google/go/go/+/master:src/time/internal_test.go;l=67;bpv=1;bpt=0 to set when to 1 instead of 0. The overflow test still does it's job, the only question is could a timer ever legitimately get when = 0? I don't think so, but I may be missing something.


jsk ast fails with an error in certain cases

Summary

Running the following produces an error:
&jsk ast
def h(*, a):
    print(a)

Traceback:
Traceback (most recent call last):
  File "C:\Users\**\AppData\Local\Programs\Python\Python310\lib\site-packages\jishaku\features\python.py", line 413, in jsk_ast
    text = create_tree(argument.content, use_ansi=Flags.use_ansi(ctx))
  File "C:\Users\**\AppData\Local\Programs\Python\Python310\lib\site-packages\jishaku\repl\disassembly.py", line 202, in create_tree
    return '\n'.join(format_ast_node(user_code, use_ansi=use_ansi))
  File "C:\Users\**\AppData\Local\Programs\Python\Python310\lib\site-packages\jishaku\repl\disassembly.py", line 187, in format_ast_node
    yield from format_ast_block(
  File "C:\Users\**\AppData\Local\Programs\Python\Python310\lib\site-packages\jishaku\repl\disassembly.py", line 163, in format_ast_block
    for child_index, description in enumerate(format_ast_node(item, level=level + 1, use_ansi=use_ansi)):
  File "C:\Users\**\AppData\Local\Programs\Python\Python310\lib\site-packages\jishaku\repl\disassembly.py", line 187, in format_ast_node
    yield from format_ast_block(
  File "C:\Users\**\AppData\Local\Programs\Python\Python310\lib\site-packages\jishaku\repl\disassembly.py", line 163, in format_ast_block
    for child_index, description in enumerate(format_ast_node(item, level=level + 1, use_ansi=use_ansi)):
  File "C:\Users\**\AppData\Local\Programs\Python\Python310\lib\site-packages\jishaku\repl\disassembly.py", line 187, in format_ast_node
    yield from format_ast_block(
  File "C:\Users\**\AppData\Local\Programs\Python\Python310\lib\site-packages\jishaku\repl\disassembly.py", line 163, in format_ast_block
    for child_index, description in enumerate(format_ast_node(item, level=level + 1, use_ansi=use_ansi)):
AttributeError: 'NoneType' object has no attribute '_fields'

Reproduction steps

Just run the snippet above. You can also try it with any function that has a keyword-only argument without a default value.
Expected results

Ast tree prints out correctly.
Actual results

A traceback is sent to me by the bot and no ast is sent.
Checklist

[x] I have updated discord.py and jishaku to the latest available versions and have confirmed that this issue is still present
[x] I have searched the open issues for duplicates
[x] I have shown the entire traceback, if possible
[x] I have removed my token from display, if visible

System information

$ py -3.10 -m discord -v
- Python v3.10.4-final
- discord.py v2.1.0-final
- aiohttp v3.8.1
- system info: Windows 10 10.0.19044

$ py -3.10 -m pip show jishaku
Name: jishaku
Version: 2.5.1
Summary: A discord.py extension including useful tools for bot development and debugging.
Home-page: https://github.com/Gorialis/jishaku
Author: Devon (Gorialis) R
Author-email:
License: MIT
Location: c:\users\**\appdata\local\programs\python\python310\lib\site-packages
Requires: braceexpand, click, import-expression, line-profiler
Required-by:

Further Notes
This bug was discovered while trying to debug some bot code I wrote in hy. I had written a custom hy_ast command that parsed the ast tree and sent the tree to jishaku.repl.disassembly.format_ast_node. I initally thought it was an issue with my code, but I was able to replicate the bug with some python source.
The hy version the erroring code above is (defn h [* a] (print a)). Running ast.unparse(...) on the ast tree produced by hy gives the exact same python code as the example in the reproduction steps.
The following is the output of ast.dump(ast.parse(code), indent=4) for two snippets:
def h(*, a):
    print(a)

gives
Module(
    body=[
        FunctionDef(
            name='h',
            args=arguments(
                posonlyargs=[],
                args=[],
                kwonlyargs=[
                    arg(arg='a')],
                kw_defaults=[
                    None],
                defaults=[]),
            body=[
                Expr(
                    value=Call(
                        func=Name(id='print', ctx=Load()),
                        args=[
                            Name(id='a', ctx=Load())],
                        keywords=[]))],
            decorator_list=[])],
    type_ignores=[])

and
def h(*, a=None):
    print(a)

gives
Module(
    body=[
        FunctionDef(
            name='h',
            args=arguments(
                posonlyargs=[],
                args=[],
                kwonlyargs=[
                    arg(arg='a')],
                kw_defaults=[
                    Constant(value=None)],
                defaults=[]),
            body=[
                Expr(
                    value=Call(
                        func=Name(id='print', ctx=Load()),
                        args=[
                            Name(id='a', ctx=Load())],
                        keywords=[]))],
            decorator_list=[])],
    type_ignores=[])

It seems like the only difference is in the kw_defaults list on the FunctionDef args attribute.

I can also reproduce this:

Traceback

Traceback (most recent call last):
  File "/home/penumbra/.cache/pypoetry/virtualenvs/mipha-kl4lG4Iw-py3.10/lib/python3.10/site-packages/jishaku/features/python.py", line 414, in jsk_ast
    text = create_tree(argument.content, use_ansi=Flags.use_ansi(ctx))
  File "/home/penumbra/.cache/pypoetry/virtualenvs/mipha-kl4lG4Iw-py3.10/lib/python3.10/site-packages/jishaku/repl/disassembly.py", line 202, in create_tree
    return '\n'.join(format_ast_node(user_code, use_ansi=use_ansi))
  File "/home/penumbra/.cache/pypoetry/virtualenvs/mipha-kl4lG4Iw-py3.10/lib/python3.10/site-packages/jishaku/repl/disassembly.py", line 187, in format_ast_node
    yield from format_ast_block(
  File "/home/penumbra/.cache/pypoetry/virtualenvs/mipha-kl4lG4Iw-py3.10/lib/python3.10/site-packages/jishaku/repl/disassembly.py", line 163, in format_ast_block
    for child_index, description in enumerate(format_ast_node(item, level=level + 1, use_ansi=use_ansi)):
  File "/home/penumbra/.cache/pypoetry/virtualenvs/mipha-kl4lG4Iw-py3.10/lib/python3.10/site-packages/jishaku/repl/disassembly.py", line 187, in format_ast_node
    yield from format_ast_block(
  File "/home/penumbra/.cache/pypoetry/virtualenvs/mipha-kl4lG4Iw-py3.10/lib/python3.10/site-packages/jishaku/repl/disassembly.py", line 163, in format_ast_block
    for child_index, description in enumerate(format_ast_node(item, level=level + 1, use_ansi=use_ansi)):
  File "/home/penumbra/.cache/pypoetry/virtualenvs/mipha-kl4lG4Iw-py3.10/lib/python3.10/site-packages/jishaku/repl/disassembly.py", line 187, in format_ast_node
    yield from format_ast_block(
  File "/home/penumbra/.cache/pypoetry/virtualenvs/mipha-kl4lG4Iw-py3.10/lib/python3.10/site-packages/jishaku/repl/disassembly.py", line 163, in format_ast_block
    for child_index, description in enumerate(format_ast_node(item, level=level + 1, use_ansi=use_ansi)):
AttributeError: 'NoneType' object has no attribute '_fields'

System details are:
$ python -m discord -v
- Python v3.10.8-final
- discord.py v2.2.0-alpha
    - discord.py metadata: v2.2.0
- aiohttp v3.8.3
- system info: Linux 5.10.0-15-amd64 #1 SMP Debian 5.10.120-1 (2022-06-09)

$ pip show jishaku
Name: jishaku
Version: 2.5.1a460+g7777777.master
Summary: A discord.py extension including useful tools for bot development and debugging.
Home-page: https://github.com/Gorialis/jishaku
Author: Devon (Gorialis) R
Author-email:
License: MIT
Location: /home/penumbra/.cache/pypoetry/virtualenvs/mipha-kl4lG4Iw-py3.10/lib/python3.10/site-packages
Requires: braceexpand, click, import-expression, tabulate
Required-by:

Fixed in 167b9ca281e7ddbf2f9bf2a1784c522967259b9f


allow configuration

https://github.com/twosigma/beakerx/issues/6343
@GrabResolver(name='bintray', root='http://dl.bintray.com/6thsolution/apexnlp'),

make the equiv of this for %classpath add mvn

%classpath add repo ...

Maybe a different subcommand than add?  The existing add commands both add things to the classpath, while this does not.  %classpath config resolver?

good point.


Not allowed to use GPT-4o with enterprise account

Version
v1.50.0 (vscode: 1.95.3)
Describe the bug
Use the edit code feature, specify GPT-4o as model. I get a dialog box with "Upgrade to cody pro". I'm logged into my enterprise account user.
Expected behavior
I can use GPT-4o (I'm allowed to use it from pycharm using same account)
Additional context
No response

Hello @olejorgenb
As an enterprise user, please open up a support ticket via the enterprise support channel.
So, we can better coerce such issues into the dedicated channel.
Thank you


AWS Lambda Node 8.10 Runtime Support

It looks like the xray sdk instruments the handler with the assumption that the handler will follow the callback pattern required in node 4 and 6. With node 8, async functions are available (and encouraged in the node 8 on lambda blog post), but including the sdk without wrapping the async handler fails.
Working:
exports.handler = async() => ({statusCode: 200});

HTTP/1.1 200 OK

Broken:
require('aws-xray-sdk');

exports.handler = async() => ({statusCode: 200});

HTTP/1.1 502 Bad Gateway

{
    "message": "Internal server error"
}

Workaround:
require('aws-xray-sdk');

exports.handler = (event, context, callback) => {
  run()
    .then((res) => callback(null, res))
    .catch((err) => callback(err));
};

async function run() {
  return {statusCode: 200};
}

HTTP/1.1 200 OK

There's actually no server-side stack trace. The lambda code executes correctly, but passes a promise instead of a response object back to API Gateway which doesn't know how to handle it. The API Gateway logs (once you figure out how to enable them) says:
Execution failed due to configuration error: Malformed Lambda proxy response

We will do some testing around the new Lambda node runtime and update the thread soon. One thing to confirm is that for the 502 you simply did require('aws-xray-sdk'); with no other code that could be traced correct?

The three examples I posted are the only code I wrote. I'm using serverless and serverless-webpack to build and deploy, so there is some transformation that happens. It does occure to me that the serverless sentry plugin was also involved which might wrap it. Under those conditions, require('aws-xray-sdk') does appear to be the only variable across tests. That said, I'll try to test it tonight with the sentry plugin and webpack disabled.

Thank you for the information. We will also do some testing around raw Lambda function to start with.

If you guys need simple reproducible steps, please refer #29.

Another report https://github.com/aws/aws-xray-sdk-node/issues/29 for the same issue. I successfully reproduced the issue using the hello world example Lambda provides.

It looks like the problem happens on CLS which the SDK has direct dependency on.
Here is a repro:
Repro index.js
require('continuation-local-storage');

exports.handler = async (event) => {
    return 'Hello from Lambda!';
};

Test event result:
Response:
null

The working index.js
//require('continuation-local-storage');

exports.handler = async (event) => {
    return 'Hello from Lambda!';
};

Test event result:
Response:
"Hello from Lambda!"

So it looks like the new async handler provided by Lambda node 8.10 runtime will have issues if your application code has indirect dependency on CLS.
We will contact Lambda team for more insights. Ideally simply importing CLS should not affect what the async handler returns.

after more investigation, it seems depending package async-listener in CLS makes this behavior.
more specific re-pro code:
module.exports.foo = async (event) =>
  require('async-listener');
  return 3;
}

The async-listener patches core modules of Node.js, somehow it breaks lambda node.js 8.10 runtime.

Okay. here some code goes:
async-listener do monkey-patch to native Promise, and change global.Promise to wrapped version of promise they have.
so it causes the issue. interesting thing is:
'use strict';

// below function results `null`
module.exports.hello = async (event) => {
  const p = new Promise((resolve) => setTimeout(resolve, 1000 * 10));

  // Backup original promise
  const originalPromise = global.Promise;

  require('async-listener');
  // Now promise is patched by async-listener

  console.log('Promise is same before?', global.Promise === originalPromise); // prints false, funny!

  // Promise detection is broken also:
  const isPromise = Promise.resolve(p) === p; // surprise!
  console.log('isPromise?', isPromise); // false. wait, what????? we expected true!

  return 3;
};

'use strict';

// surprise! we can get 3!
module.exports.hello = async (event) => {
  // Backup original promise
  const originalPromise = global.Promise;

  require('async-listener');
  // Now promise is patched by async-listener

  // Simply restore original Promise
  global.Promise = originalPromise;

  return 3;
};

Okay. Finally i found why lambda returns null.
Here goes:

when requires aws-xray-sdk-core package from user land, it requires continuation-local-storage package internally
continuation-local-storage package requires async-listener package internally
async-listener patches global.Promise object
Lambda runtime fails Promise checking (see below screenshot)

According to Line 287~290 (end of invoke function) on/var/runtime/node_modules/awslambda/index.js
it uses response instanceof Promise to check given object is instance of Promise object.
but on step (3), global.Promise was mutated by async-listener. so it results false and lambda runtime treats this response as non-promise object.
so for now below two code will have same effect:
'use strict';

module.exports.hello = async (event) => {
  require('async-listener'); // patches global.Promise object
  return 3;
};

'use strict';

const BbPromise = require('bluebird');

module.exports.hello = (event) => BbPromise.resolve(3); // returns instance of bluebird promise

so to Lambda team:
Please use thenable object check instead of using response of Promise like below:
if (response && typeof response.then === 'function') { // given response is promise-like object
  response.then(context.succeed);
  response.catch(context.fail);
}

Please note Node.js project has thenable test suite, so if you need some tests please refer here:
https://github.com/nodejs/node/tree/db1087c9757c31a82c50a1eba368d8cba95b57d0/deps/v8/test/webkit/fast/js

Hi Mooyoul,
Thank you for the detailed deep dive. We have reached out to the Lambda team and will be following up on this ASAP.
Sorry for the trouble,
Sandra

Hello @awssandra , do you have a timeline for those changes?

@murilodag
We are continuing to prioritize this fix with the Lambda team. Please stay tuned for updates, we'll update this thread accordingly when we have more news.
Sorry for the trouble,
Sandra

I've also experienced this issue when returning a Bluebird promise in AWS lambda using the node 8.10 runtime, thanks for the explanation @mooyoul 👍

We are facing this problem too!

This can be fixed from xray sdk also if you just find an alternative for CLS. i mean whole problem is that CLS override global Promise with bluebird
@awssandra How about just finding an alternative of CLS and publish new version of aws-xray-sdk? this seems getting dragged

Any update on this issue?

@awssandra can we get an ETA on this, as this is critical bug i feel like and prevents people from fully using xray.

so for now, there's only one option that we can do:
Use callback. yeah, maybe it sounds ugly. but will work without having any issues.
For example:
const BbPromise = require('bluebird');

module.exports.foo = wrap(async (event) => {
  return 3;
});

function wrap(handler) {
  return (event, context, callback) => {
    BbPromise.resolve(handler(event, context)).asCallback(callback);
  };
}

Will this ever get fixed? A lot of people are using node 8 runtime and don't want to use callbacks.

I was hit by simply returning bluebird promise and receiving mysterious null value, along with API Gateway generic Internal Server Error.
Looks exactly like what @mooyoul has described in detail.
Is there a ticket or smth. on this issue on some public Lambda team tracker?

Hi everyone,
We're moving on this with the Lambda team as we speak, Thank you for all of your patience. We'll update as soon as we can.

Hi, any update?

Hi everyone,
Lambda has finished rolling out the fix. Please let us know of any further issues. Thank you again for all your patience.

Just tested this in eu-west-1  works like a charm! Now I can return bluebird promises!
Here's snippet I used to test
'use strict';

const bbPromise = global.Promise = require('bluebird');

module.exports.handler = async (event, context) => {

  return new bbPromise((resolve, reject)=>{

    let response =  {
      statusCode: 200,
      body: JSON.stringify({
        message: 'If you see this, you returned bluebird promise from async function upstream to Lambda core!',
      }),
    };

    setTimeout(()=>{
      resolve(response);
    }, 3000);

  });

};

How to test? I don't see any new version..

@Enase
The Lambda 8.10 runtime was updated under the hood. All existing Lambda 8.10 functions have been updated behind the scenes.


Realm crashes when trying to add an object

How frequently does the bug occur?
Sometimes
Description
We're seeing some crashes in production caused by adding objects to Realm. While these seem quite rare they also cause the crash every single time the app is launched for affected users, so effectively their app is completely stuck and they have to do a reinstall to recover from this.
Stacktrace & log output
Crashed: com.apple.main-thread
0  libsystem_kernel.dylib         0x7964 __pthread_kill + 8
1  libsystem_pthread.dylib        0x7378 pthread_kill + 268
2  libsystem_c.dylib              0x1ff50 abort + 164
3  Realm                          0x30b904 realm::util::terminate(char const*, char const*, long) + 10
4  Realm                          0x30bbcc _hidden#17674_ + 712
5  Realm                          0x30bd8c realm::util::terminate(char const*, char const*, long, std::initializer_list<realm::util::Printable>&&) + 328
6  Realm                          0x5e5b4 void realm::util::terminate_with_info<char const&, unsigned long&>(char const*, int, char const*, char const*, char const&, unsigned long&) + 51 (__hidden#746_:51)
7  Realm                          0x16d168 realm::ArrayBlob::replace(unsigned long, unsigned long, char const*, unsigned long, bool) + 552
8  Realm                          0x16e1a0 realm::ArraySmallBlobs::insert(unsigned long, realm::BinaryData, bool) + 140
9  Realm                          0x1841d0 _hidden#14257_ + 8508
10 Realm                          0x18388c _hidden#14251_ + 6136
11 Realm                          0x2e23d4 realm::TableClusterTree::for_each_and_every_column(realm::util::FunctionRef<bool (realm::ColKey)>) const + 84
12 Realm                          0x17ed80 realm::Cluster::insert_row(unsigned long, realm::ObjKey, std::__1::vector<realm::FieldValue, std::__1::allocator<realm::FieldValue> > const&) + 208
13 Realm                          0x180cf4 realm::Cluster::insert(realm::ObjKey, std::__1::vector<realm::FieldValue, std::__1::allocator<realm::FieldValue> > const&, realm::ClusterNode::State&) + 240
14 Realm                          0x18fa38 _hidden#14352_ + 668
15 Realm                          0x18aba4 realm::ClusterNodeInner::insert(realm::ObjKey, std::__1::vector<realm::FieldValue, std::__1::allocator<realm::FieldValue> > const&, realm::ClusterNode::State&) + 252
16 Realm                          0x18e430 realm::ClusterTree::insert_fast(realm::ObjKey, std::__1::vector<realm::FieldValue, std::__1::allocator<realm::FieldValue> > const&, realm::ClusterNode::State&) + 48
17 Realm                          0x18e70c realm::ClusterTree::insert(realm::ObjKey, std::__1::vector<realm::FieldValue, std::__1::allocator<realm::FieldValue> > const&) + 164
18 Realm                          0x2e1f24 realm::TableClusterTree::insert(realm::ObjKey, std::__1::vector<realm::FieldValue, std::__1::allocator<realm::FieldValue> > const&) + 52
19 Realm                          0xc008 realm::Object realm::Object::create<objc_object* __strong, RLMAccessorContext>(RLMAccessorContext&, std::__1::shared_ptr<realm::Realm> const&, realm::ObjectSchema const&, objc_object* __strong, realm::CreatePolicy, realm::ObjKey, realm::Obj*) + 317 (__hidden#677_:317)
20 Realm                          0xb70c RLMAccessorContext::createObject(objc_object*, realm::CreatePolicy, bool, realm::ObjKey) + 1093 (__hidden#651_:1093)
21 Realm                          0x5d0f0 RLMAddObjectToRealm + 139 (__hidden#5141_:139)
22 RealmSwift                     0x6124c Realm.add<A>(_:update:) + 432 (__hidden#6932_:432)
23 *
24 *
25 *
26 *
27 *
28 RealmSwift                     0x60a28 Realm.write<A>(withoutNotifying:_:) + 255 (__hidden#6932_:255)
29 *
30 *
31 *
32 *
33 *
34 libdispatch.dylib              0x2914 _dispatch_call_block_and_release + 32
35 libdispatch.dylib              0x4660 _dispatch_client_callout + 20
36 libdispatch.dylib              0x12b60 _dispatch_main_queue_callback_4CF + 944
37 CoreFoundation                 0x51cd4 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 16
38 CoreFoundation                 0xbeac __CFRunLoopRun + 2540
39 CoreFoundation                 0x1f3b8 CFRunLoopRunSpecific + 600
40 GraphicsServices               0x138c GSEventRunModal + 164
41 UIKitCore                      0x5196a8 -[UIApplication _run] + 1100
42 UIKitCore                      0x2987f4 UIApplicationMain + 2092
43 *
44 ???                            0x1015a5a24 (Missing)

Can you reproduce the bug?
Not yet
Reproduction Steps
No response
Version
v10.18.0
What SDK flavour are you using?
Local Database only
Are you using encryption?
No, not using encryption
Platform OS and version(s)
iOS, 14.2.0, 15.1.0
Build environment
Xcode version: 13.1
Dependency manager and version: Carthage 0.38.0

Hello @anlaital ,
Could you share more details on how you are using the Realm?
Specifically - configuration, threading, objects schemas and transactions?
Anything else you think can help to understand the issue?

Ping @anlaital

@bmunkholm unfortunately we've not been able to isolate the issue, so I have no further details to share at this time. 😔

For the looks of the stack trace seems it could be that the object you are adding, may have some corrupted data. Do you have any code from the App start up which could be adding some object to the database?


ISSUE 29: Implement a delete logic in /history

For now, focus on:

Removing any unnecessary data from the context that could interfere with the fights' logic.
Create a new button for each fight entry, to delete it.
Create a new popup confirmation message, that will have the options yes or no, to confirm the fight deletion.

After thinking about it for a while, I came to the conclusion that is not necessary to add that delete fight confirmation for now, it's better to invest my time improving other things like handling the first time usage of the app (With all the historyEntries and profileData clear)


Inconsistent inventory preserving in some cases??

I am not sure why, but during playing with Scout on a heavily modded Fabric client the satchels and pouches just tend to freak out and revert in time their inventories, deleting or adding stacks of items according to a state they randomly memorized. Sometimes things change just visually and you can click on an empty slot to get the item that was supposed to be stored there pretty recently. Sometimes I open Backpack from Inmis mod and the fake inventory in the pouches becomes real, duplicating items, and deleting some really valuable ones. I can't rely on the pouches to store my valuable information at all!
Here's my modlist:    (FABRIC 1.20.1) modlist.txt

I can also confirm it can happen on 1.19's version, so this is a longstanding issue.

Can confirm, I'm also having this issue on fabric 1.20.1 in a server environment, data loss seems to happen especially commonly upon leaving/rejoining the server, but I've also seen item desyncs occur when changing dimensions, as well as when switching from survival to creative and vice versa.


chore: Assets

📚 $\text{Stack Overview}$
Pulls submitted in this stack:

#109 👈
main

This comment was automatically generated by st.


Week4 ndvi

@nanaboamah89 I only made some small edits. I think it looks good and should merge so we can consider week 4 in its entirety. I also think we should put the final images in so people can check theirs looks the same.
@caitlinadams @andrewdhicks @jcrattz  any thoughts about inline code vs screenshots? Do you think we are out of the screenshot zone now? It certainly looks neater this way.
PS ignore the failed docs check, the week 3 content hasn't been pulled from master yet.

Hi @caitlinadams , I have updated the changes, the only concern was the time for loading the data, the whole notebook take about 15 mins to run in which about 97% of the time is taken at the loading of the data.

I ran the notebook again, this time it took less than 2 mins to execute, using the same area.


Should closeTab icon be hidden when highlightModifiedTabs is true?

Using this config:
"workbench.editor.tabCloseButton": "off",
"workbench.editor.highlightModifiedTabs": true,

- dirty files still take space for closing button (and shifting all tabs).
It makes sense that it worked that way when the only option to show modified state of files was the icon.
Should it take into account highlighting option and not change tab width now?

@misolori this seems broken now with latest:

Idea was to hide the dirty indicator in this case.

Fixed:

This should also hide the dirty state when workbench.editor.highlightModifiedTabs is set to true. Right now it only hides it when workbench.editor.tabCloseButton is off.

Here's all of the states to verify:
When workbench.editor.highlightModifiedTabs is true:

When workbench.editor.tabCloseButton is off:

When workbench.editor.highlightModifiedTabs is on and workbench.editor.tabCloseButton is off:

@bpasero @usernamehw let me know if I missed anything

I see both the dot and the bar with
  "workbench.editor.tabCloseButton": "right",
  "workbench.editor.highlightModifiedTabs": true

Actually maybe thats the desired behaviour?


:bar_chart: Migrate monkeypox to ETL

Migrate monkeypox from https://github.com/owid/monkeypox to ETL. There are minimal changes to the data (I've used our own population estimates which changed some metrics on the 5-th decimal place).
The repository is still updated by script in CRON. The next step will be implementing new ExternalStep that would be committing CSV to the repo.
The PR also adds detection when date is used instead of year and modifies the dataset by adding yearIsDay to all columns before inserting it to MySQL.
Explorer in staging.
TODO after merging

[ ] Update CSV link in explorer to https://catalog.ourworldindata.org/explorers/who/latest/monkeypox/monkeypox.csv

btw, don't fully understand why I had to commit anything in https://github.com/owid/etl/pull/3083/commits/c8f1b5e0f440aeb464aedc6c83932482f8419d81. I got a CONFLICT for some reason.


[BUG] The Microsoft.Web functionAppConfig.deployment.storage storageType enum should be renamed to functionsDeploymentStorageType.

API Spec link
https://github.com/Azure/azure-rest-api-specs/blob/main/specification/web/resource-manager/Microsoft.Web/stable/2023-12-01/CommonDefinitions.json
API Spec version
2023-12-01
Describe the bug
In the 2023-12-01 Microsoft.Web API, I introduced the functionAppConfig property bag to specify configuration for a Flex Consumption function app. In the deployment section, I defined a new enum type named storageType (https://github.com/Azure/azure-rest-api-specs/blob/main/specification/web/resource-manager/Microsoft.Web/stable/2023-12-01/CommonDefinitions.json#L3619-L3630). However, there was an existing enum with this name in ArcConfiguration (https://github.com/Azure/azure-rest-api-specs/blob/main/specification/web/resource-manager/Microsoft.Web/stable/2023-12-01/KubeEnvironments.json#L384-L395). To fix this issue, we need to rename the storageType enum in the deployment section of functionAppConfig to functionsDeploymentStorageType.
Please note that this issue was caught when requesting new SDKs for the 2023-12-01 API version, so this is not a breaking change yet. I will ask the other SDKs to wait until this change is completed.
Expected behavior
N/A
Actual behavior
N/A
Reproduction Steps
N/A
Environment
N/A

This issue has been fixed for the 2023-12-01 API version. The change has been merged to the main branch.


Bundled sqlite3-3.7.9 is not compatible with Ubuntu 14.04's GDAL

I'm having some issues with gdal on a 14.04 based image running a GeoDjango app with dokku.
It complains about sqlite3_column_database_name not being defined. I compared the symbols from the system wide libsqlite3 inside the container and the symbols from the bundled sqlite3 and such function is missing:
root@2fdfab92c524:/# objdump -TR /app/.heroku/python/lib/libsqlite3.so.0.8.6 | grep sqlite3_column_database_name
root@2fdfab92c524:/# objdump -TR /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6 | grep sqlite3_column_database_name
0000000000010860 g    DF .text<PHONE_NUMBER>000011  Base        sqlite3_column_database_name
0000000000010880 g    DF .text<PHONE_NUMBER>000011  Base        sqlite3_column_database_name16

Looking at the buildpack sources seems like it bundles sqlite-3.7.9 while Ubuntu 14.04 ships sqlite-3.8.6. So probably upgrading the bundled sqlite3 will fix the issue.
Best Regards,

@michaelshobbs I wrongly bugged the gdal maintainers in Ubuntu and you can find details on:
https://bugs.launchpad.net/ubuntu/+source/gdal/+bug/1406679

strongly second bumping up the version of Sqlite to the latest

Bumping the version doesn't solve the problem. Even bumped (as in my fork) to 3.8.6, the column metadata APIs (such as sqlite3_column_table_name and sqlite3_column_database_name won't be included.
The fix should be adding the -DSQLITE_ENABLE_COLUMN_METADATA flag.


Graduate NB2KG

We have been discussing throughout differents PRs that we should graduate NB2KG, and below are some of the suggested options:

Graduate NB2KG as a sub-project under Jupyter Enterprise Gateway
Incorporate the NB2KG functionality to Jupyter Notebook (e.g. jupyter notebook --gateway_url http://foo:8888

Let's use this issue for discussing any other alternatives and/or suggestions in a centralized place

We discussed moving the nb2kg code to https://github.com/jupyter-incubator/nb2kg during the Jupyter dev call today. No one there had any objections. I can take this ticket to move the code sometime this week or next.

As for merging it into notebook itself, two possible approaches come to mind:

Send the PR to notebook with a hefty description as to why, what it's enabling, and have a discussion there with code in hand.
Get people more familiar and comfortable with nb2kg as its own incubator project, state from the beginning the goal is to ultimately get it into notebook, work out any kinks there, and then send the PR to notebook down the line.

I don't have a read on people's thoughts on nb2kg functionality directly in notebook at the moment to know which is better.

That would be great - thanks @parente.
Although I'd love to have nb2kg embedded in Notebook - as @lresende mentions - I believe it would be viewed as opening the floodgates for a number of other server extensions (i.e., where do you stop?) even though it would be fairly trivial and lightweight to bolt it into NB. I'm curious if that avenue was discussed today.
Once moved, I think we should publish an update to pypi (and conda forge?).

I found jupyter/notebook#3006 about adding configuration overrides to support custom handlers in notebook. I am delaying creating the jupyter-incubator/nb2kg repo to see how that PR shakes out. If you'd like the incubator project created ASAP however, give a shout.

Thanks Peter - sorry for the delayed response.
Since the jupyter/notebook#3006 PR falls short on actually merging NB2KG into the Notebook repo, I'm inclined to move forward with NB2KG's promotion to an incubator for the following reasons.

Merging into Notebook will require further, and perhaps extensive, discussion and I think we need to publish a 0.0.2 release sooner.
There may be some other items that come up that warrant changes that would be good to get in before the merge since we'd want NB2KG to be virtually complete as post-merge changes will require more time to get published.

One area that I'd like to see improved is richer error messaging back to the client (from the gateway).  While the gateways (kernel and enterprise) raise detailed exceptions (that are logged on the gateway server), only the error code seems to be propagated.  I'm unfamiliar with HTTP error messaging and haven't actually looked into it, so I'm uncertain there's more that can be done, but I think improvement in that area could go a long way to helping our gateway clients.
I'd love to hear others' opinions.

I went ahead and created https://github.com/jupyter-incubator/nb2kg. If it turns out the extension is not needed one day, removing the repo can be a form of graduation.
I used the git filter-branch command to extract the nb2kg subfolder with its full history from this project and pushed it as the master branch in the new one. I invited all of the collaborators from jupyter-incubator/enterprise_gateway to then new project with equivalent permissions. I also added the original collaborators from the kernel_gateway_demos project. A nb2kg package was never pushed to pypi, so there's no permissions to configure there.
I'm going to submit a PR here momentarily that removes the nb2kg code from this repo and leaves a simple README in its place pointing to the new location.

Awesome, thank you @parente!
I'll update README.md since I'd like to add reference to enterprise gateway anyway.


Created a contribution.md file

@BatoolMM  fixes #125
The contribution.md file creates a short guide to new contributors, that are either new to open source or new to OSCSA website.
i created a contribution.md file

Brilliant, thank you for you hard work. Can you change the format of the headers in the Markdown by removing ===.

Many Thanks!

@all-contributors add @jesutobi for documentation

Brilliant, thank you for you hard work. Can you change the format of the headers in the Markdown by removing ===.

Many Thanks!

You are welcome, i would change the format now


[WIP] IPAM controller implementation

Code is not covered by test and is not tested well, but main flows should work.

The anonnotaion is not as defined in the doc
yours  {"my-pool":{"name":"my-pool","subnet":"<IP_ADDRESS>/16","startIP":"<IP_ADDRESS>","endIP":"<IP_ADDRESS>","gateway":"<IP_ADDRESS>"}}
and in the doc it without the name key. Why do we need the name key anyway...

The offset code is adding addition ip to the block
for example for this config
     "pools": {
         "my-pool": {
             "subnet": "<IP_ADDRESS>/29",
             "perNodeBlockSize": 2,
             "gateway": "<IP_ADDRESS>"
         }

each node get range of 3 ip
node 1: start="<IP_ADDRESS>" end="<IP_ADDRESS>"
node 2: start="<IP_ADDRESS>" end="<IP_ADDRESS>"
I0503 22:16:59.739984  628030 allocator.go:117] "allocator/pool=my-pool: range allocated" controller="node" controllerGroup="" controllerKind="Node" Node="worker-node01" namespace="" name="worker-node01" reconcileID=684b8ca2-d904-47fa-905c-8f735c12872a node="worker-node01" start="<IP_ADDRESS>" end="<IP_ADDRESS>"
I0503 22:16:59.766752  628030 node.go:104] "node object updated" controller="node" controllerGroup="" controllerKind="Node" Node="worker-node01" namespace="" name="worker-node01" reconcileID=684b8ca2-d904-47fa-905c-8f735c12872a
I0503 22:16:59.767120  628030 allocator.go:117] "allocator/pool=my-pool: range allocated" controller="node" controllerGroup="" controllerKind="Node" Node="worker-node02" namespace="" name="worker-node02" reconcileID=0d051c99-8076-48f4-8cc0-be72ce99af8c node="worker-node02" start="<IP_ADDRESS>" end="<IP_ADDRESS>"

when the configmap get deleted I see this message [1] in a loop.  restart the controller solves it.
[1] -
E0503 22:21:50.096217  628030 configmap.go:58] "failed to get ConfigMap object from the cache" err="ConfigMap "nvidia-k8s-ipam-config" not found" controller="configmap" controllerGroup="" controllerKind="ConfigMap" ConfigMap="default/nvidia-k8s-ipam-config" namespace="default" name="nvidia-k8s-ipam-config" reconcileID=221b2639-634e-45bc-aa01-474d33e84108

with this config  (blockSize is 4 but I mean 5 see +1 shift issue above)
{
"pools": {
"my-pool": {
"subnet": "<IP_ADDRESS>/28",
"perNodeBlockSize": 4,
"gateway": "<IP_ADDRESS>"
}
}
}
see below [1]  "worker-node02" got <IP_ADDRESS> which is broadcast address (start="<IP_ADDRESS>" end="<IP_ADDRESS>")
[1]
571eb2f46d8 node="master-node" start="<IP_ADDRESS>" end="<IP_ADDRESS>"
I0503 22:27:54.316226  630910 node.go:104] "node object updated" controller="node" controllerGroup="" controllerKind="Node" Node="master-node" namespace="" name="master-node" reconcileID=bf39f805-1ae8-48e4-87ac-7571eb2f46d8
I0503 22:27:54.316576  630910 allocator.go:117] "allocator/pool=my-pool: range allocated" controller="node" controllerGroup="" controllerKind="Node" Node="worker-node01" namespace="" name="worker-node01" reconcileID=5ee200ff-adb6-41c5-8d0a-a9ad7c7f2824 node="worker-node01" start="<IP_ADDRESS>" end="<IP_ADDRESS>"
I0503 22:27:54.347169  630910 node.go:104] "node object updated" controller="node" controllerGroup="" controllerKind="Node" Node="worker-node01" namespace="" name="worker-node01" reconcileID=5ee200ff-adb6-41c5-8d0a-a9ad7c7f2824
I0503 22:27:54.347544  630910 allocator.go:117] "allocator/pool=my-pool: range allocated" controller="node" controllerGroup="" controllerKind="Node" Node="worker-node02" namespace="" name="worker-node02" reconcileID=f7c6dd88-31f1-4f54-ab6b-1f7b5a8708b9 node="worker-node02" start="<IP_ADDRESS>" end="<IP_ADDRESS>"

when the configmap get deleted I see this message [1] in a loop. restart the controller solves it.
[1] - E0503 22:21:50.096217 628030 configmap.go:58] "failed to get ConfigMap object from the cache" err="ConfigMap "nvidia-k8s-ipam-config" not found" controller="configmap" controllerGroup="" controllerKind="ConfigMap" ConfigMap="default/nvidia-k8s-ipam-config" namespace="default" name="nvidia-k8s-ipam-config" reconcileID=221b2639-634e-45bc-aa01-474d33e84108

So when creating again the configmap the error stoped which is good. We can just open issue to see how to handle configmap delete as this is not critical

we should clean annotation from nodes when configmap is invalid.
scenario:

create valid config
annotation on all nodes
delete config
create invalid config
annotation should be removed (current code keep valid configmap annotation)

we should clean annotation from nodes when configmap is invalid.

CNI relies on valid configuration for all CNI calls  CMD Add and CMD Del
we can keep the "last valid" annotation on node
or we need to write a caching mechanism in CNI to save it from the add call.

PR re-based, testing is in progress

we should clean annotation from nodes when configmap is invalid. scenario:
1. create valid config

2. annotation on all nodes

3. delete config

4. create invalid config

5. annotation should be removed (current code keep valid configmap annotation)

I updated the controller behavior. Now the controller explicitly handles ConfigMap removal  - simply waits for ConfigMap recreation.
Current behavior is following: in case if there is no ConfigMap or if it contains invalid configuration then controller will simply keep all existing annotations for nodes and wait for ConfigMap update/creation. I think this is the safest behavior but we can change it later.


[themes] Incorrect syntax highlighting after destructured function arguments with a default value in JS

VSCode Version: 1.15.1
OS Version: macOS 10.12.6 and Linux Mint 18

Steps to Reproduce:

Paste this into a JS file:

const fn = ({
	one = 'one',
} = {}) => {
	const something = 'hello';
	return something;
};

Note that const and return keywords are not highlighted as reserved keywords:

Reproduces without extensions: Yes
Notes: Only occurs when the destructured object is formatted over multiple lines. Looks fine when the object is on a single line:

This issue seems similar and possibly related to https://github.com/Microsoft/vscode/issues/3008

Thanks for the report. This is a duplicate of https://github.com/Microsoft/TypeScript-TmLanguage/issues/499
Closing as upstream


Doesn't always add  and  tags

Request Reduce does not always remove the original scripts and css links and add the RequestReduceContent path.
Sometimes it runs and only the css appears reduced, sometimes it has all been reduced and sometimes nothing is reduced - even though new files are being created within RequestReduceContent.
Using MVC3/.NET4 on IIS7.5 with no changes to web.config (setup using NuGet).

There are a few reasons where this can happen. Is the page where you are seeing this publicly accessible? It would be very helpful to see this even without RequestReduce plugged in in order to further diagnose or explain this issue.
It can take RequestReduce anywhere from a second to a few minutes to complete all of its reductions. This depends on the amount of css/javascript you have and the number of background images your css contains. As it completes its reductions, subsequent reductions, subsequent requests will begin to see the reduced content. It is certainly possible for RequestReduce to have completed reducing the CSS but still be reducing Javascript. During this time, users should see the reduced css but the original javascript until RequestReduce completes the javascript reductions.
This also be impacted by near future expiring javascript. Currently, RequestReduce will generate reduced files for near future expiring javascript, but these will never be seen by the user. RequestReduce will flag the near future expiring url after it generates the reduction and the next time a request comes in, it will generate another reduction ignoring this url. At any rate, it is possible to take a few page views before all content is properly reduced and the user sees only the reduced content.
All of this can be further impacted if you are running RequestReduce in a web farm environment. Especially if you are not synchronizing via sql server.
Now, once the reductions are final accross each server, you should see consistent output and it should only change if you flush the cached content or if you change the css/javascript urls in the html. So if for some reason you have cache busting  querystring parameters in your css or javascript or some other aspect of the url that changes on each request, RequestReduce considers each complete url and queues any urls that it has never processed and then show the original unaltered content to the user until that complete url has been reduced and is requested again.

Thanks Matt - it's good to know how it works in more detail. After deploying to Azure it started working immediatly once it had been CNAMEd to the same domain as the SSL certificate I was using (preventing the SSL errors in browsers) - is there any reason why this would be the case?

Hi Jamie.  I'm especially gladto hear that it works on Azure. I have never tested it there and wondered if there might be odd permissions issues or other issues with the disk writing that RequestReduce does. I had another user that had problems with AppHarbor but we got those resolved.
The SSL cert issue actually makes perfect sense. RequestReduce processes your CSS and Javascript files by downloading them via http(or https if they are https links). So if you did not have a valid cert and RequestReduce was getting invalid or missing cert errors, the reductions would fail. Basically, if you can't navigate to your css and javascript links in a browser without error, neither can RequestReduce.
Just about an hour ago, I posted a new wiki page with troubleshooting and debugging tips for these exact kind of issues at https://github.com/mwrock/RequestReduce/wiki/RequestReduce-is-not-working.-I-don't-see-any-spriting-or-minification.-How-can-I-troubleshoot-this%3F. I wish that was available earlier when you encountered these problems.
Please create another issue or ping me on twitter (@mwrockx).


Consider re-ordering processing of setActivePlayers value argument

Currently arguments to setActivePlayers are processed in the following order:

value
player
others
all

Source →
This is “safe” in the sense that each argument that applies to a larger set of players follows the previous smaller argument, so if an all is passed, all players will definitely be in that stage. (And others and player are complementary.)
However, this order is not so convenient when using the value argument, because it prevents you combining value and other arguments. In particular for the following scenario:
setActivePlayers({ all: 'A', value: { [id]: 'B' } })

Reading this, I would like to have all players in stage A except for id who is in stage B. Kind of like a player–others set-up, but for an arbitrary player.
Given that value is in any case a more “advanced” use, I’d like to propose it being applied last inside setActivePlayers to allow for this kind of override behaviour. I’ll open a PR as it’s a super simple code change, but wanted to open this issue for a more thorough discussion.

Sounds good to me.