Spark 3.0 support

/question
Spark 3.0-preview is ready and official release targets to next Q1.
https://spark.apache.org/news/spark-3.0.0-preview.html
Some of the features maybe not be supported in spark-operator. Trying to understand the deployment cycle. When should we start dev work in spark-operator?

Ref https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/693#issuecomment-553707420

Some of the most notable features in Spark 3.0 relevant to Kubernetes are:

Pod templating support: this doesn't matter much for users using the Spark Operator because they define pod templates in the SparkApplication spec.
Kerberos support: ref #306.
An option to use a hostPath volume for the scratch space (work dir).
An option to keep the executor pods after a job is completed.

Both 3 and 4 are just a matter of adding a new optional field in SparkApplicationSpec and set the corresponding Spark config option, which is straightforward.

HI Yinan, I will take 3 and 4 and I already have changes ready for them.

Hi @liyinan926
For 2, changes are available as part of this PR.

Thanks @Jeffwan!
@abhisrao can you create a PR to upstream your change?

Sorry for the delay in responding @liyinan926  We had to make the changes on fork branch as there is discussion going on internally about CLA.. Will update when there is some progress on the CLA aspect.

Can we get a 3.0 drop of the operator?

Is it possible to use the operator with spark 3.0 ?

@liyinan926 @Jeffwan @gaocegege  PodTemplate is extremely helpful. We run spark on serverless kubernetes(virtual kubelet). Link to https://issues.apache.org/jira/browse/SPARK-31173

@ringtail Yeah I think so. Thus I'd appreciate it if we could migrate to podTemplate.

@ringtail @gaocegege What's the motivation to use podTemplate for spark-operator users?
As I understand, Spark users who want podTemplate is because using spark conf to set volumes, envs is kind of tedious when user use spark-submit.
Most spark-operator users are familiar with k8s. I am thinking spark-operator has mutation webhook to inject all pods needed envs, pvs information to pod directly in a native k8s way. Not sure if user still want to use pod-template. Can this meet the requirement?   Could you help clarify the use case? If there's something missing, then I think we should catch up the support.

@ringtail @gaocegege What's the motivation to use podTemplate for spark-operator users?
As I understand, Spark users who want podTemplate is because using spark conf to set volumes, envs is kind of tedious when user use spark-submit.
Most spark-operator users are familiar with k8s. I am thinking spark-operator has mutation webhook to inject all pods needed envs, pvs information to pod directly in a native k8s way. Not sure if user still want to use pod-template. Can this meet the requirement? Could you help clarify the use case? If there's something missing, then I think we should catch up the support.

For us. Pod creation performance is the point of concern. https://issues.apache.org/jira/browse/SPARK-31173

For us. Pod creation performance is the point of concern. https://issues.apache.org/jira/browse/SPARK-31173

More precisely, I think webhook performance drags down the overall performance ?

Put performance penalty of using the webhook aside, I don't see a need to support pod template as a first-class citizen in the operator that already has the declarative API for pod config. We can definitely have an option to internally convert certain fields in a SparkApplicationSpec into a pod template file and use that for Spark 3.0 instead of the webhook. However, we don't have a plan to add a field with type PodTemplateSpec.

Put performance penalty of using the webhook aside, I don't see a need to support pod template as a first-class citizen in the operator that already has the declarative API for pod config. We can definitely have an option to internally convert certain fields in a SparkApplicationSpec into a pod template file and use that for Spark 3.0 instead of the webhook. However, we don't have a plan to add a field with type PodTemplateSpec.

Yes. I agree this design of CRD spec.

Any insight into the cause of the performance penalty?  Is it reproducible on GKE?

Any insight into the cause of the performance penalty? Is it reproducible on GKE?

You can try to disable webhook

Hi, is there a dockerfile for Spark3.0 spark-operator? I followed the one for 2.4.5 https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/spark-docker/Dockerfile but got error when submitting the Spark app on GKE.

@jiayue-zhang it looks like there's no spark 3.0 image: https://console.cloud.google.com/gcr/images/spark-operator/GLOBAL/spark?gcrImageListsize=30
I couldn't find the repo where these images are being built.

Hi, is there a dockerfile for Spark3.0 spark-operator? I followed the one for 2.4.5 https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/spark-docker/Dockerfile (by replacing the base image with spark3.0.0) but got error when submitting the Spark app on GKE.

Could you provide the error log?

@cmontemuino That's right. But since spark-operator has developed features on top of Spark 3.0, I think developers have the image and just not officially released it yet. So I'm asking the Dockerfile that I can make a 3.0 image on my side to work on it because we want to try Spark 3.0 in GCP.
@ringtail Log 1:
...
org.apache.spark#spark-avro_2.11 added as a dependency
org.influxdb#influxdb-java added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-4df89fb7-c826-4ac2-af03-530802ffb429;1.0
confs: [default]
Exception in thread "main" java.io.FileNotFoundException: /opt/spark/.ivy2/cache/resolved-org.apache.spark-spark-submit-parent-4df89fb7-c826-4ac2-af03-530802ffb429-1.0.xml (No such file or directory)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:270)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
	at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorWriter.write(XmlModuleDescriptorWriter.java:70)
	at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorWriter.write(XmlModuleDescriptorWriter.java:62)
	at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.toIvyFile(DefaultModuleDescriptor.java:563)
	at org.apache.ivy.core.cache.DefaultResolutionCacheManager.saveResolvedModuleDescriptor(DefaultResolutionCacheManager.java:176)
	at org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:245)
	at org.apache.ivy.Ivy.resolve(Ivy.java:523)
	at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1387)
	at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:54)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

It seems to be related to loading extra jars. So I commented out my spark.jars.packages in yaml to see more logs. This time I got:
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/base/Preconditions
	at org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:328)
	at org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:341)
	at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:423)
	at org.apache.spark.deploy.SparkHadoopUtil$.newConfiguration(SparkHadoopUtil.scala:426)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$2(SparkSubmit.scala:342)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:342)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.google.common.base.Preconditions
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 14 more

This lead me to think that my Dockerfile might not be right. I use our own spark:v2.4.5-gcs image for 2.4.5 following https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/spark-docker/Dockerfile. Then I use similar dockerfile to build 3.0.0 except the ARG SPARK_IMAGE but got error. This is my dockerfile for 3.0.0:
ARG SPARK_IMAGE=gcr.io/our-project/spark:v3.0.0-rc1
FROM ${SPARK_IMAGE}

# Setup dependencies for Google Cloud Storage access.
RUN rm $SPARK_HOME/jars/guava-14.0.1.jar
ADD https://repo1.maven.org/maven2/com/google/guava/guava/23.0/guava-23.0.jar $SPARK_HOME/jars
ADD https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop2-2.0.1.jar $SPARK_HOME/jars

ENTRYPOINT ["/opt/entrypoint.sh"]

The reason I'm using gcs-connector-hadoop2-2.0.1.jar not gcs-connector-latest-hadoop2.jar is because we need use hadoop 2.7 which is by default used by Spark. More discussion is in https://github.com/GoogleCloudDataproc/hadoop-connectors/issues/323 but I don't think the error here is related to that.
My yaml file is same as the one I used for spark2.4.5. Some highlights are:
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
spec:
    image: "gcr.io/our-repo/spark:v3.0.0-rc1-gcs"
    sparkVersion: "3.0.0"
    sparkConf:
        "spark.jars.packages": "org.apache.spark:spark-avro_2.11:2.4.5,org.influxdb:influxdb-java:2.7"

@cmontemuino That's right. But since spark-operator has developed features on top of Spark 3.0, I think developers have the image and just not officially released it yet. So I'm asking the Dockerfile that I can make a 3.0 image on my side to work on it because we want to try Spark 3.0 in GCP.
@ringtail Log 1:
...
org.apache.spark#spark-avro_2.11 added as a dependency
org.influxdb#influxdb-java added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-4df89fb7-c826-4ac2-af03-530802ffb429;1.0
confs: [default]
Exception in thread "main" java.io.FileNotFoundException: /opt/spark/.ivy2/cache/resolved-org.apache.spark-spark-submit-parent-4df89fb7-c826-4ac2-af03-530802ffb429-1.0.xml (No such file or directory)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:270)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
	at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorWriter.write(XmlModuleDescriptorWriter.java:70)
	at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorWriter.write(XmlModuleDescriptorWriter.java:62)
	at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.toIvyFile(DefaultModuleDescriptor.java:563)
	at org.apache.ivy.core.cache.DefaultResolutionCacheManager.saveResolvedModuleDescriptor(DefaultResolutionCacheManager.java:176)
	at org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:245)
	at org.apache.ivy.Ivy.resolve(Ivy.java:523)
	at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1387)
	at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:54)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

It seems to be related to loading extra jars. So I commented out my spark.jars.packages in yaml to see more logs. This time I got:
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/base/Preconditions
	at org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:328)
	at org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:341)
	at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:423)
	at org.apache.spark.deploy.SparkHadoopUtil$.newConfiguration(SparkHadoopUtil.scala:426)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$2(SparkSubmit.scala:342)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:342)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.google.common.base.Preconditions
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 14 more

This lead me to think that my Dockerfile might not be right. I use our own spark:v2.4.5-gcs image for 2.4.5 following https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/spark-docker/Dockerfile. Then I use similar dockerfile to build 3.0.0 except the ARG SPARK_IMAGE but got error. This is my dockerfile for 3.0.0:
ARG SPARK_IMAGE=gcr.io/our-project/spark:v3.0.0-rc1
FROM ${SPARK_IMAGE}

# Setup dependencies for Google Cloud Storage access.
RUN rm $SPARK_HOME/jars/guava-14.0.1.jar
ADD https://repo1.maven.org/maven2/com/google/guava/guava/23.0/guava-23.0.jar $SPARK_HOME/jars
ADD https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop2-2.0.1.jar $SPARK_HOME/jars

ENTRYPOINT ["/opt/entrypoint.sh"]

The reason I'm using gcs-connector-hadoop2-2.0.1.jar not gcs-connector-latest-hadoop2.jar is because we need use hadoop 2.7 which is by default used by Spark. More discussion is in GoogleCloudDataproc/hadoop-connectors#323 but I don't think the error here is related to that.
My yaml file is same as the one I used for spark2.4.5. Some highlights are:
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
spec:
    image: "gcr.io/our-repo/spark:v3.0.0-rc1-gcs"
    sparkVersion: "3.0.0"
    sparkConf:
        "spark.jars.packages": "org.apache.spark:spark-avro_2.11:2.4.5,org.influxdb:influxdb-java:2.7"

I am not sure about the error. But if you want to create a spark base image. You can refer to this repo. (https://github.com/AliyunContainerService/spark/blob/alibabacloud-v2.4.5/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile)

@jiayue-zhang, looking at the logs you've provided when working with spark 3, please bear in mind that Spark 3.0 is compiled against 2.12. If you include a package compiled against 2.11 it should fail (e.g., org.apache.spark#spark-avro_2.11).
An alternative to what @ringtail is suggesting would be downloading the official preview2 package and use the docker-image-tool. If plain spark 3 is fine with you, then that's all what you need.
This is minimal script of what we currently do for building a raw spark 3 image:
SPARK_VERSION=3.0.0-preview2
SPARK_PACKAGE=spark-${SPARK_VERSION}-bin-hadoop3.2.tgz
SPARK_SOURCE_DIR=${SPARK_PACKAGE%.tgz}
SPARK_UID=${SPARK_UID:-0}  # this SPARK_UID is new in the image that in with preview2

wget "https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${SPARK_PACKAGE}"
tar xvzf "${SPARK_PACKAGE}"

cd "${SPARK_SOURCE_DIR}"
BUILD_OPTIONS=(-r "$YOUR_DOCKER_REPO" -t "$SOME_TAG" -u "$SPARK_UID")

./bin/docker-image-tool.sh "${BUILD_OPTIONS[@]}" build

We currently don't have an operator image based on Spark 3.0.0 yet, since it's not officially out yet. Like what @cmontemuino suggested, you could build a vanila Spark 3.0.0-preview2 Spark image using docker-image-tool.sh, and then build a custom operator image based on that Spark image. When installing using the Helm chart, you can specify the operator image you want to use.

it's officially here. https://spark.apache.org/releases/spark-release-3-0-0.html

Will create a new release based on Spark 3.0 soon. Stay tuned. Thanks.

I have just built and pushed the following images:
gcr.io/spark-operator/spark:v3.0.0
gcr.io/spark-operator/spark-py:v3.0.0
gcr.io/spark-operator/spark-operator:v1beta2-1.1.2-3.0.0

@liyinan926
What's the support plan? Do we want to maintain 2.x and 3.x in different branches and cut release separately? Or move to 3.x?

The short term plan is to switch the master branch to be based on 3.x and maintain 2.x in a separate branch. The long term plan is to move to 3.x completely.

I have just built and pushed the following images:
gcr.io/spark-operator/spark:v3.0.0
gcr.io/spark-operator/spark-py:v3.0.0
gcr.io/spark-operator/spark-operator:v1beta2-1.1.2-3.0.0

@liyinan926, could You release Java 11 based images as well?
Thanks

Yes java 11 please

We now support in the latest release pretty much all the new config options and enhancements introduced in Spark 3.0.0 except for pod template files, which don't make much sense in the context of the operator. The master branch has also been switched to be based on Spark 3.0.


IGAPP-655: Move search to web-ts

This pull request belongs to an issue on our bugtracker.
You can find it there by looking for an issue with the key which is mentioned in the title of this pull request.
It starts with the keyword IGAPP.

Tested


put BoltDB behind AttrStore interface

Overview
This PR creates AttrStoreGenerator and AttrStore interfaces, and moves the boltdb dependency out to an implementation of these interfaces.
We should probably add an in-memory implementation to use in testing. Currently, the tests have been modified to use the boltdb implementation.
Fixes #1096
Pull request checklist

[x] I have read the contributing guide.
[x] I have agreed to the Contributor License Agreement.
[x] I have updated the documentation.
[x] I have resolved any merge conflicts.
[x] I have included tests that cover my changes.
[x] All new and existing tests pass.

Code review checklist
This is the checklist that the reviewer will follow while reviewing your pull request. You do not need to do anything with this checklist, but be aware of what the reviewer will be looking for.

[ ] Ensure that any changes to external docs have been included in this pull request.
[ ] If the changes require that minor/major versions need to be updated, tag the PR appropriately.
[ ] Ensure the new code is properly commented and follows Idiomatic Go.
[ ] Check that tests have been written and that they cover the new functionality.
[ ] Run tests and ensure they pass.
[ ] Build and run the code, performing any applicable integration testing.

This looks good for the most part. My only criticism is that maybe AttrStoreGenerator adds some complexity for questionable benefit.
I took a stab at refactoring that interface to instead be a func(string) AttrStore (NewAttrStore) that is passed into the places that AttrStoreGenerator is passed in your branch. That code is here:
https://github.com/codysoyland/pilosa/commit/f9efa6c57928597fb3d0aa62aa757fa0562d338d
I think this is a little cleaner and reduces LOC, but I didn't want to push to your branch before getting your eyes on it. Let me know if you think this is a better approach.

Thanks @codysoyland
This approach looks good to me. Feel free to push to that branch.


Missing end of test lab connection

Should the end of test lab integration be added to the dashboard?

like the other milestones

This is not so important because all warning stops in a few days anyway, but you might like to consider it.

If anything is done about this suggestion, then I suggest to only make an addition to the text in the table of milestones, not to add an extra graphics flag.


Add [Main Market Africa mainmarketafrica.com] #issuer GDXPCE…OCQQFW

Address: GDXPCE6QLDAYKL7OKHKR4UEB35HGQ5AWLFJ7SMPP4BIIVHB32OOCQQFW
Tags: issuer
Title: Main Market Africa  Domain: mainmarketafrica.com
Requested by @mainmarketafrica

Main Market Africa is a premium membership business to business platform which match-makes member businesses according to the details of their profiles. This Digital Marketing process ensures that member businesses receive most relevant continuous updated buy or sell orders for their products or services. This platform has a self-driven/self-funding mechanism for its premium members powered by the mix of network activity, recycling of membership subscription fees, integration of digital currencies and smart contracts.

This is not flagged, what are you wanting to be done?
@mainmarketafrica

Thank you for your response,
We would like to be added to the stellarterm.com public directory manually.
Also our logo on the stellarterm.com needs to be updated with the copy
which we have uploaded on github.
Thanks and Regards,
Larry.
On Wed, Oct 12, 2022 at 11:15 PM CryptoHobbyist @.***>
wrote:

This is not flagged, what are you wanting to be done?
@mainmarketafrica https://github.com/mainmarketafrica
—
Reply to this email directly, view it on GitHub
https://github.com/stellar-expert/public-directory/issues/67#issuecomment-1276528585,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AX5LK53VCCC5AIZMXAC6TRLWC32LHANCNFSM54FHNJGQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

You will have to contact Stellar Term themselves about what criteria they use to list projects.
<EMAIL_ADDRESS>

Thanks for the reference.🙏
On Sun, Oct 16, 2022 at 9:11 PM CryptoHobbyist @.***>
wrote:

You will have to contact Stellar Term themselves about what criteria they
use to list projects.
@.***
—
Reply to this email directly, view it on GitHub
https://github.com/stellar-expert/public-directory/issues/67#issuecomment-1279994569,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AX5LK57P4XJ2KHTSTBCAJYDWDQO27ANCNFSM54FHNJGQ
.
You are receiving this because you were mentioned.Message ID:
@.***>


CODEC-300 - Simplify assertion

Coverage remained the same at 94.643% when pulling 9a262ca0f8885e07b599d4b94e8e979c0bd805f2 on arturobernalg:feature/CODEC-300 into a7c5af95a731b2a3a1baa6454f4c63ae30f719ec on apache:master.


Adding response to connectivity service creation

Hi Karthik,
Can you verify that the answer is consistant?
{
"_connConstraint": {
"serviceLayer": [
"OCH"
],
"serviceType": "POINT_TO_POINT_CONNECTIVITY"
},
"_connection": [
"http://localhost:8080/restconf/config/Context/_connection/concs1/"
],
"_servicePort": [
{
"_serviceEndPoint": "http://<IP_ADDRESS>:8080/restconf/config/Context/_serviceEndPoint/se1",
"direction": "BIDIRECTIONAL",
"localId": "sp1",
"role": "SYMMETRIC",
"serviceLayer": "OCH"
},
{
"_serviceEndPoint": "http://<IP_ADDRESS>:8080/restconf/config/Context/_serviceEndPoint/se2",
"direction": "BIDIRECTIONAL",
"localId": "sp2",
"role": "SYMMETRIC",
"serviceLayer": "OCH"
}
],
"uuid": "cs1"
}
While connection looks like:
{'_connectionPort': [{'_connectionEndPoint': {'_clientNodeEdgePoint': ['http://localhost:8080/restconf/config/Context/_topology/_node/node1/_ownedNodeEdgePoint/pe14/'], 'uuid': 'cep1'}, 'localId': 'cp1'}, {'_connectionEndPoint': {'_clientNodeEdgePoint': ['http://localhost:8080/restconf/config/Context/_topology/_node/node4/_ownedNodeEdgePoint/pe42/'], 'uuid': 'cep2'}, 'localId': 'cp2'}, {'_connectionEndPoint': {'_clientNodeEdgePoint': ['http://localhost:8080/restconf/config/Context/_topology/_node/node2/_ownedNodeEdgePoint/pe21/'], 'uuid': 'cep3'}, 'localId': 'cp3'}], 'direction': 'BIDIRECTIONAL', 'layerProtocolName': 'OCH', 'uuid': 'concs1'}
Can you please comment?
Thanks

The response looks fine.


KTH-50 - Create Docker Compose for local development

This PR intoduces:

A Makefile for simplifying local data generation and Docker containers management.
A new way of running Twiga locally with Docker Compose.
A specific Docker image for the DB, with the installation of pgvector.
5 new env-vars for DB (needed for the DB, although they are not critical).
Code simplification

This PR aims to simplify the local development process by encapsulating all affected services in a Docker Compose. Now, DB no longer depends on a PostgreSQL running in ones computer, creating an specific image for it and for running it in the compose. This change should:

Fix pgvector error that some colleagues were having (mostly windows-based computers).
Close #68

I have updated the GETTING_STARTED.md with the new way of running Twiga. Much simpler than before.
How to test?

[ ] Recreate your local environment completely. For doing this you might need to execute in a SQL terminal the following commands:

DROP SCHEMA public CASCADE;
CREATE SCHEMA public;

[ ] Follow the steps of GETTING_STARTED.md. Everything should run smoothly 👍

cc @jurmy24 @fredygerman @louhelhir

Hi @alvaro-mazcu. Great work!
I have a few comments/questions:

We have a lot of environment variables already, and all info in the first 5 env variables below are contained within the 6th and can be extracted using the urlparse() function.

DATABASE_USER=$YOUR_USER
DATABASE_PASSWORD=$YOUR_PASSWORD
DATABASE_SERVER_NAME=db
DATABASE_PORT=5432
DATABASE_NAME=$YOUR_DB_NAME
DATABASE_URL=postgresql+asyncpg://$YOUR_USER:$YOUR_PASSWORD@db:5432/$YOUR_DB_NAME

I see you use these directly in the docker-compose. Is there a way to extract them from the DATABASE_URL instead to avoid the duplication?

Is it more reasonable to store the docker-compose file within the docker folder under docker/dev? Let me know what you think.

Do you think it makes more sense that this is the default way to run twiga as a new contributor and have that as the main approach in the documentation? (and then just say that if you want to run it without docker contact us or something)

Try to use the commit conventions where you write keyword(scope): short description \n LINEAR-TAG

run pre-commit run --all-files to make sure your code is properly formatted and that it passes all the tests. If you haven't used pre-commit before you can check the CONTRIBUTING.md file.

Also, thanks for moving the get_database_url() file into utils. It makes more sense to have it there.

We have a lot of environment variables already, and all info in the first 5 env variables below are contained within the 6th and can be extracted using the urlparse() function.
DATABASE_USER=$YOUR_USER
DATABASE_PASSWORD=$YOUR_PASSWORD
DATABASE_SERVER_NAME=db
DATABASE_PORT=5432
DATABASE_NAME=$YOUR_DB_NAME
DATABASE_URL=postgresql+asyncpg://$YOUR_USER:$YOUR_PASSWORD@db:5432/$YOUR_DB_NAME
I see you use these directly in the docker-compose. Is there a way to extract them from the DATABASE_URL instead to avoid the duplication?

Yep, i think there's a way. They are not extremely important (I was testing with them and I include them into the PR, but I can delete them)

Is it more reasonable to store the docker-compose file within the docker folder under docker/dev? Let me know what you think.

It's ok, i placed it in the root for convenience. The difference will be at the time of running the compose: docker-compose -f /path/to/your/docker-compose.yml commandvs docker-compose up. I don't really care much about it, so it's up to you 😇

Is it more reasonable to store the docker-compose file within the docker folder under docker/dev? Let me know what you think.

It's ok, i placed it in the root for convenience. The difference will be at the time of running the compose: docker-compose -f /path/to/your/docker-compose.yml commandvs docker-compose up. I don't really care much about it, so it's up to you 😇

I think it's cleaner to put it in the docker folder in whatever specific place makes most sense. Then you can just modify the docker compose command in the docs.

Do you think it makes more sense that this is the default way to run twiga as a new contributor and have that as the main approach in the documentation? (and then just say that if you want to run it without docker contact us or something)

I think so, yeah. It's just 2-3 commands, much less than before, and replicable in all types of hardware :)

Great, then we do it that way.


Rob: Watch nutrinext\w*

Rob requests the watch of the watch_keyword nutrinext\w*. See the Metasmoke search here and the Stack Exchange search here.

https://stackoverflow.com/q/50071969/563532

nutrinext\w* has been seen in 0 true positives, 0 false positives, and 0 NAAs.

Coverage remained the same at 65.747% when pulling f90454f2ada0491bbbf1440333067c51b0ff4424 on auto-blacklist-1524880074.771267 into 5f36d526d64e29dfb60435b7936a3267d90e89fb on master.

!!/approve


add shortcut for existing full-screen edit mode (esc for example)

it would be great if we can exit the full-screen edit view by clicking esc key
@antonpod

https://github.com/wix-incubator/quix/commit/99dc730596cb74a5ced11e3a25da5a41ff63c306


Why not platform?

Just a surface-level addition to "Why not platform?" to make the explanation a bit less of of a text-dump, addressing #114. :balloon:

I know you're trying to help and I appreciate the PR. However, some of those objections are a little dated depending on which version of Platform you're talking about. Some of the more recent problems are things like the bundled Stack having a known, serious bug and the Platform maintainers not pushing out a fixed version until the next GHC release.
I'd rather leave the section as-is. Thank you for the contribution though!


iOS landscape orientation camera preview not following screen orientation

Describe the bug
I need to force user to landscape mode when taking pictures. Using https://github.com/yamill/react-native-orientation to lock into landscape mode but the camera preview is not following the screen orientation
To Reproduce
Steps to reproduce the behavior:
Add Orientation.lockToLandscapeLeft()
Expected behavior
Camera preview should show base on the orientation
Screenshots

Smartphone (please complete the following information):

Device: iPhone 6s, iPhone 15 pro max
Version: 13.0.0

+1 Same issue with me.

Try out v14.0.0 that was just released.
In addition, please lock the UI unless you know what you are doing, and instead use the new onOrientationChange event handler to counter-rotate UI elements if needed (see the example project):


How to eval with the human proxy agents?

How to eval with the human proxy agents? It seems only the script agents are available for evaluation.

The evaluation code with the human proxy agents is pretty tricky (and perhaps dirty)... that's why we have not put the code here. The human proxy agents are from the human_aware_rl repo, which corresponds to the paper On the Utility of Learning about Humans for Human-AI Coordination.
Unfortunately, the provided human proxy agents in  human_aware_rl repo is only supported under tensorflow (or tensorflow 2), which is different from our repo (pytorch). Also there is an inconsistency problem of other packages such as gym. Briefly, the tricky stuff we did for evaluation is running our agents and the human proxy agents in different python environments, while transmitting the actions through ZeroMQ. And you can see why i describe the evaluation process as "tricky" now... 🤣

Thanks for explanation😂


[QUESTION] Getting Exception While Trying to Port SKSwapChainPanel to WinUI 3

 Hi Friends,
I am using SKXamlCanvas into my current project but It has some performance issue. When I add lots of graphical shapes to canvas, zooming and panning started getting extremely slow so, I looked around and found out that the SKSwapChainPanel uses OpenGL which could solve my problem. As of now, SKSwapChainPanel is only available in UWP so, I decided to port it to WinUI3 my self. I downloaded latest code and removed comments from AngleSwapChainPanel.cs & SKSwapChainPanel.cs. I am able to compile/build fine but as soon as I add generated SkiaSharp.dll & SkiaSharp.Views.Windows.dll files to my Test project, I get following exception (see attached image below). I tried to clean rebuild many times without any luck. I also tried to compiled with x64 both projects as well.
I am using Visual Studio 2022, MSIX 1.0.
Can somebody please let me know what I am missing here? Please let me know if you want me to upload my project solutions here.
Thanks In Advance,
Amit Parmar

HI All,
Its been a long time since I post this question. Can somebody please help me figure out the issue?
Right now, we are using SkXmlnCanvas for our drawing application which is very slow so, we have to use some kind of GPU acceleration control for our application.
Thanks,
Amit

+1 on this

Duplicate of #1893

I am working on this. The ANGLE library needs to be rebuilt for WinUI and sadly it is not super easy as we are like 3 layers away. I am trying 2 approaches to both fix ANGLE and to make changes in the vcpkg tooling to support this.


quickstart: download needed?

On the quickstart part of the README, should we remove item 1 (download), since I think the pip install means that a manual download is not needed?

@danielskatz I think I left that in because we needed to clone the repo to get tutorials to the local machine.
Maybe we should split the tutorial to a different repo and merge it into the main repo as a submodule. What do you think ?

I think this makes sense.

The parsl_demos repo already has the tutorial material broken out. Please reopen if this is still an issue.


when topic does not exist, call topicRouteTable.remove

1 nameserver, 1 broker, 1 producer
producer send messages to a topic, when broker is down, producer still send heartbeat to broker
because
org.apache.rocketmq.client.impl.factory.MQClientInstance#cleanOfflineBroker
does not work

Coverage decreased (-0.06%) to 50.582% when pulling b5d0f45469c35e32362a68e64ea24a0e351995e9 on rushsky518:topic_not_exist into f8f6fbe4aa7f5dee937e688322628c366b12a552 on apache:develop.


Problems with JSX

I am trying to add time-picker in my Reactjs app.
But input attribute "data-enable-time=true" is not valid in JSX syntax. True in quotes - not working. What i should do?

Hello,
While that is strange, you could always pass in options with an object


Model::all() not scoped but Model::create() is... on the same model

Hi, I'm having an odd issue with scoping queries...
My User model contains:
namespace Acme\User;

use LaravelBook\Ardent\Ardent;
use AuraIsHere\LaravelMultiTenant\Traits\TenantScopedModelTrait;

class User extends Ardent {

    use TenantScopedModelTrait;

    ...

I have a custom 'before' filter called company, which I use in my routes file:
Route::group(array('prefix' => 'users', 'before' => 'company'), function()
{
    Route::get('/', 'UsersController@index');
    Route::get('create', 'UsersController@create');
    Route::post('create', 'UsersController@store');
    ...
});

Within the company filter, I get the current Company ID from the session and scope with:
TenantScope::addTenant('company_id', Session::get('company_id') );

My UsersController@store method looks like:
$user = new User();
$user->name = 'Joe';
$user->save();

...and as I would expect, creates a user called 'Joe' with the company_id value automagically set to 1.
However, my UsersController@index method contains:
$users = User::all();

...and it returns every user, instead of just users with a company_id of 1.
Curiously, if I var_dump TenantScope::getTenants() on that same index method, I get:
array(1) { ["company_id"]=> string(1) "1" }

...which to me would imply that the tenant has been set correctly?
Any input would be greatly appreciated. This is an awesome package and I hope I've just overlooked something very stupid!

I think that it is somehow related to LaravelBook\Ardent\Ardent.
This is how  the scope is made:
public static function bootTenantScopedModelTrait()
{
	$tenantScope = App::make("AuraIsHere\LaravelMultiTenant\TenantScope");

	// Add the global scope that will handle all operations except create()
	static::addGlobalScope($tenantScope);

	// Add an observer that will automatically add the tenant id when create()-ing
	static::creating(function (Model $model) use ($tenantScope)
	{
		$tenantScope->creating($model);
	});
}

Perhaps addGlobalScope isn't working as supposed to because of Ardent.
Can you use the default model and give it a shot?

@joeserg Any luck with this? I agree that it's likely an issue with Ardent. please report  back with your findings. Thanks!


Add unit tests for io.helidon.common.http.SetCookie

I've analysed your codebase and noticed that io.helidon.common.http.SetCookie is not fully tested.
I've written some tests for the methods in this class with the help of Diffblue Cover.
Hopefully, these tests will help you detect any regressions caused by future code changes. If you would find it useful to have additional tests written for this repository, I would be more than happy to look at other classes that you consider important.

There seem to be some compilation errors related to your new test.
You need to sign the OCA for us to accept your contribution, see https://www.oracle.com/technetwork/community/oca-486395.html.
The new file you are adding is missing copyright header, see other files.

There seem to be some compilation errors related to your new test.
You need to sign the OCA for us to accept your contribution, see https://www.oracle.com/technetwork/community/oca-486395.html.
The new file you are adding is missing copyright header, see other files.

I have sent a signed copy of the CLA to<EMAIL_ADDRESS>and I have added the copyright header to this pull request

Thanks. @delabassee can you let me know when you've processed the OCA ? Thanks.

The copyright year should just be 2019.
The test cannot compile right now because you are using the JUnit4 API, we use JUnit5.
You need to use org.junit.jupiter.api.Test instead of org.junit.test and org.junit.jupiter.api.Assertions Instead of org.junit.Assert.

@romain-grecourt this OCA has been reviewed and approved!

@romain-grecourt please wait before merging!

@delabassee OK.
@Braavos96 the test failures are interesting, it probably has to do with the wercker environment.
org.opentest4j.AssertionFailedError: expected: <foo=bar; Expires=Thu, 2 May 2019 10:28:42 +0000> but was: <foo=bar; Expires=Thu, 2 May 2019 10:28:42 GMT>
	at io.helidon.common.http.SetCookieTest.testExpireswithInstant(SetCookieTest.java:42)

testToString1  Time elapsed: 0.031 sec  <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <foo=bar; Expires=Thu, 2 May 2019 10:28:42 +0000; Max-Age=60000; Domain=http://; Path=baz; Secure; HttpOnly> but was: <foo=bar; Expires=Thu, 2 May 2019 10:28:42 GMT; Max-Age=60000; Domain=http://; Path=baz; Secure; HttpOnly>
	at io.helidon.common.http.SetCookieTest.testToString1(SetCookieTest.java:64)

testToString2  Time elapsed: 0.001 sec  <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <foo=bar; Expires=Thu, 2 May 2019 10:28:42 +0000> but was: <foo=bar; Expires=Thu, 2 May 2019 10:28:42 GMT>
	at io.helidon.common.http.SetCookieTest.testToString2(SetCookieTest.java:78)

How about saving the ZonedDateTime instance and formatting the same way as SetCookie.toString.
ZonedDateTime expires = ZonedDateTime.now();
expires.format(DateTimeFormatter.RFC_1123_DATE_TIME);

Hi, Apologies for the late response.
@romain-grecourt thank you for the tip. I used a set date/time as the test would then be more reliable.
@delabassee I have emailed a new copy of the CLA

Hi @romain-grecourt and @delabassee just following up to check if you would like me to make any more changes to the PR?

@delabassee Is the new OCA processed?

Hi @romain-grecourt and @delabassee just following up on this PR :)

Hi, I checked with @delabassee and it seems that the OCA has not been received. Can you please resend it ?. Thanks.

Hi @delabassee I have resent you the CLA to the email address<EMAIL_ADDRESS>

Hi @delabassee would you mind confirming that you have received the CLA please

It seems we had some process issues with this PR.
We currently do have a unit test that was created as part of other changes (WebClient).
I am closing this PR


LocalFile cache confusion

Unfortunately I cannot share my wordpress instance, however let me explain my findings.
I have multiple pages that reference the same image in different sizes.
in fetchNodeHtmlImageMediaItemNodes we can see that the imageNode.id is saved to the cache
    if (imageNode) {
      // save any fetched media items in our global media item cache
      _store.default.dispatch.imageNodes.pushNodeMeta({
        sourceUrl: htmlImgSrc,
        id: imageNode.id,
        modifiedGmt: imageNode.modifiedGmt
      }); // match is the html string of the img tag

In my case the imageNode is a MediaItem
  ...
  "remoteFile": {
    "id": "f61fe9db-337d-5f25-8ac0-35a9a8a9f501"
  },
  "localFile": {
    "id": "f61fe9db-337d-5f25-8ac0-35a9a8a9f501"
  },
  "parent": null,
  "internal": {
    "contentDigest": "2001149f43ce77ed8758db8595e7e8fd",
    "type": "WpMediaItem",
    "counter": 77,
    "owner": "gatsby-source-wordpress-experimental"
  },
  "type": "MediaItem",
  "nodeType": "MediaItem",
  "children": []

However the pushNodeMeta call also causes the nodeMetaByUrl lookup table to be set / overwritten.
    pushNodeMeta(state, {
      id,
      sourceUrl,
      modifiedGmt
    }) {
      state.nodeIds.push(id);
      state.nodeMetaByUrl[(0, _fetchReferencedMediaItems.stripImageSizesFromUrl)(sourceUrl)] = {
        id,
        modifiedGmt
      };
      return state;
    }

However in getFileNodeByMediaItemNode we pull the node meta out and treat it as if it was a localFile
const getFileNodeByMediaItemNode = async ({
  mediaItemNode,
  helpers
}) => {
  const {
    sourceUrl,
    modifiedGmt,
    mediaItemUrl,
    databaseId
  } = mediaItemNode;
  const fileUrl = sourceUrl || mediaItemUrl;

  if (!fileUrl) {
    helpers.reporter.warn((0, _formatLogMessage.formatLogMessage)(`Couldn't find source url for media item #${databaseId}`));
    return null;
  }

===>   const existingNodeMeta = getFileNodeMetaBySourceUrl(fileUrl);

  if ( // if we already have this image
  existingNodeMeta && existingNodeMeta.id && // and it hasn't been modified
  existingNodeMeta.modifiedGmt === modifiedGmt) {
    const node = await helpers.getNode(existingNodeMeta.id);
    return node;
  }

  return null;
};

So in my case the pushNodeMeta gets called multiple times, the first few instances with the correct remoteFileNode id and url coming from createRemoteMediaItemNode
 // so we can touch this node next time
  // and so we can easily access the id by source url later

  _store.default.dispatch.imageNodes.pushNodeMeta({
    id: remoteFileNode.id,
    sourceUrl: mediaItemUrl,
    modifiedGmt
  });

However, a bit later we get the call with a resized version of the url (coming from fetchNodeHtmlImageMediaItemNodes). Now pushNodeMeta gets called with the id of the imageNode.  Since both urls are normalized (stripImageSizesFromUrl) the cache in nodeMetaByUrl gets overwritten and it now points to an ImageNode rather than a localFile.
Now every call to getFileNodeByMediaItemNode will result in the ImageNode being returned rather than the FileNode.
In the end the build fails with
The "path" argument must be one of type string, Buffer, or URL. Received type undefined

  Error: TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be one of ty  pe string, Buffer, or URL. Received type undefined

called from replaceNodeHtmlImages when going through htmlMatchesWithImageResizes because
      if (imageResize) {
        imgOptions.fluid = imageResize
        ReactGatsbyImage = React.createElement(Img, imgOptions, null)
      } else {
        const { fileNode } = matchResize

        const fileName = `${fileNode.internal.contentDigest}/${fileNode.base}`

the fileNode here is no longer a real fileNode but the imageNode.

anyone home?

Hey @rburgst I was out on vacation the last couple weeks. Thanks for this detailed description of the problem, this is really helpful. So to reproduce I need to use the same image in 2 different fields but set a different size for each field?

Yes, in my case the default (no specific size) gets handled first and leads to the correct file node being stored in the cache. Then inside an html tag the same image with a different size spec gets referenced and leads the previous filenode cache entry to be overwritten with a mediaItem node. I have created a patch that solves the problem for me but am not 100% sure if the solution is a good one. Will post the MR today.

This is a really hard one to reproduce! Is there any way you'd be able to email me your /graphql API url? (tyler@gatsbyjs.com).
If that's not possible can you share more specifically in what kinds of fields these images are stored in and what the file path & name is for each? That might help me figure out how to reproduce. If I can reproduce then I can write a test for this which would be nice. Your code looks good but it's hard to merge it without being able to reproduce the problem.
In the last couple releases I also fixed some other caching issues around images which might (but probably doesn't) fix this issue so It's probably worth trying latest.
Thanks for your patience and clear description with this!

I will try with the latest master version and let you know. Unfortunately our server is behind VPNs.

It seems the bug still exists in latest master. Will try and analyze again / rebase my patch

I am fairly certain the bug is caused by a custom ACF with HTML content that contains html such as this:

ok, I did some more digging.
The problem appears if I have 2 pages that virtually have the identical content (I have 2 translations of the same page). Both have a HTML ACF which references the image

To me it looks like we have a race condition, we first store the filenode in state.nodeMetaByUrl["http://dockerhost:8000/wp-content/uploads/2018/05/imagename-300x163-1.jpg"] called from createRemoteMediaItemNode
callstack
```
pushNodeMeta(), image-nodes.js:32
createRemoteMediaItemNode(), create-remote-media-item-node.js:219
Async call from async function
createRemoteMediaItemNode(), create-remote-media-item-node.js:108
retryPromise(), fetch-referenced-media-items.js:147
anonymous(), fetch-referenced-media-items.js:65
run(), index.js:249
_tryToStartAnother(), index.js:197
anonymous(), index.js:263
add(), index.js:244
pushPromiseOntoRetryQueue(), fetch-referenced-media-items.js:56
createMediaItemNode(), fetch-referenced-media-items.js:132
processTicksAndRejections(), task_queues.js:93
Async call from async function
createMediaItemNode(), fetch-referenced-media-items.js:116
anonymous(), fetch-referenced-media-items.js:348
retryPromise(), fetch-referenced-media-items.js:348
pushPromiseOntoRetryQueue(), fetch-referenced-media-items.js:56
fetchMediaItemsBySourceUrl(), fetch-referenced-media-items.js:305
Async call from async function
fetchMediaItemsBySourceUrl(), fetch-referenced-media-items.js:285
fetchReferencedMediaItemsAndCreateNodes(), fetch-referenced-media-items.js:498
fetchNodeHtmlImageMediaItemNodes(), process-node.js:185
Async call from async function
fetchNodeHtmlImageMediaItemNodes(), process-node.js:152
replaceNodeHtmlImages(), process-node.js:415
processNodeString(), process-node.js:631
processNode(), process-node.js:667

a bit later we overwrite this entry with an imageNode called from `fetchNodeHtmlImageMediaItemNodes`

if (imageNode) {
  // save any fetched media items in our global media item cache
  _store.default.dispatch.imageNodes.pushNodeMeta({
    sourceUrl: htmlImgSrc,
    id: imageNode.id,
    modifiedGmt: imageNode.modifiedGmt
  }); // match is the html string of the img tag

callstack

pushNodeMeta(), image-nodes.js:32
fetchNodeHtmlImageMediaItemNodes(), process-node.js:256
Async call from async function
fetchNodeHtmlImageMediaItemNodes(), process-node.js:152
replaceNodeHtmlImages(), process-node.js:415
processNodeString(), process-node.js:631
processNode(), process-node.js:667

now `state.nodeMetaByUrl["http://dockerhost:8000/wp-content/uploads/2018/05/imagename-300x163-1.jpg"]` points to an `imageNode`.

then we enter `createRemoteMediaItemNode` a 2nd time (this time because the `img` tag also had a `wp-image-9076767` tag (probably)). Now `getFileNodeByMediaItemNode` will return the `imageNode` rather than the filenode which will cause the problem.

IMHO in https://github.com/gatsbyjs/gatsby-source-wordpress-experimental/blob/master/src/steps/source-nodes/create-nodes/process-node.js#L263 we must push the localFile node rather than the imageNode

I agree, just want to make sure I can add a test in for this. Thanks again!


Google OpenID Closed

Is there a plan to add support for the updated Google Auth service?

No, I don't use this library (or PHP at all) anymore, so I have no plans to update it anymore.


Token Binding Support in TLS extension

I need token binding support in TLS and it is not implemented in the BCJSSE.But in the extensions, it has been recognized, So will it be implemented?

What do you mean here: "But in the extensions, it has been recognized"?
Assuming you are talking about https://tools.ietf.org/html/draft-ietf-tokbind-protocol-16, It's on our TODO list, but that list is long, and this is still a draft specification.
Also, since there doesn't appear to be a JSSE API for it yet, the functionality could initially only be available in BCJSSE through a BC-specific extension API.
If you meant channel bindings, we have support already for tls-unique via an extension interface in BCJSSE.

Thanks for the quick response, I meant the Token Binding specification which is still in the draft version."But in the extensions, it has been recognized"- In the extension type class  token binding extension id has been written
public static final int DRAFT_token_binding = 24;
I know there is no Java implementation for this and only nginx , apache httpd server and chrome have support for this.But I am a student and need this support for a project(I am running in a short timeframe).
Earlier I just want to know, you have already started to work on this or not and your reasoning is fair and square regarding it.However, I need this support, If I am able to write the support for token binding in BC (through BC-specific extension API and in your conventions), Is it possible to send a PR to BC? I know we can send PR for small issues but I am not sure whether public can send PR for implementations.

Yes we accept PRs for new implementations. Please ensure you understand the terms of the BC license, and have permission from any actual or potential rights-holders (of the software). Also please understand that it can take significant time for us to review contributions.

Thanks, I'll look into license and permissions.Time is not a problem I just want everyone to benefit from it.

I started implementing this in client side and having some trouble with getting the negotiated values to the user. In the token binding implementation, we have to provide the user with negotiated key parameter and EKM.Currently, I am saving the key parameter in security parameters of Tls client protocol.But this cannot be accessed from the socket, So how should I do this?
I think if I finish this I could send an early PR with client-side implementation.

Hi,
I have sent an initial PR to add the token binding extension(client side), can you have look on that?


[SPARK-44426][SQL] Optimize adaptive skew join for ExistenceJoin

What changes were proposed in this pull request?
Supports automatic processing of data skews of the ExistenceJoin type, and only for left-table skews.
Why are the changes needed?
Automatic processing of data skews of the ExistenceJoin type is not currently supported and if data skew occurs, the application executes very slowly.

Does this PR introduce any user-facing change?
Yes. It will automatically handle data skew in the left table.
How was this patch tested?
New UT.

Could you help me to review this PR?
gently ping @cloud-fan
Thanks.

cc @ulysses-you


Add final (really!) Functions sample for Firebase, for analytics

Failures are not in functions. Merging.


maxRotation and other missing in documentation

Some config commands are not present in the official documentation.
Expected Behavior
Documentation up to date.
Current Behavior
minRotation, maxRotation and others missing in documentation.
Possible Solution
Steps to Reproduce (for bugs)
https://www.chartjs.org/docs/latest/axes/styling.html#tick-configuration
Context
Environment

These options are described for cartesian axes. If it's not supposed to work for radial scales, then I guess we can close this issue.

These settings are documented as linked above


WebView

I'm trying to figure out how to use the WebViewElement functionality to create a webview from F#.
Ideally I'd like to do this in react style. I've found a react wrapper for Electron but I'm not sure how to import it to achieve this functionality. I've worked successfully with the fable-react samples.
Right now I'm putting the <webview> in html and ignoring it in my render code.

Hi @aolney! Looking at the Webview sample it seems you need to import the default member of "webview", so something like this should work:
open Fable.Core.JsInterop
let WebView = importDefault<obj> "react-electron-webview"

Then you can use Fable.Helpers.React.from WebView to create a React component as in the material-ui sample.
I hope this helps!

You are legend, thanks!

Ups, sorry! Actually you have to import the webview as a React.ComponentClass. I edited my code above ☝
Please tell me if you need help for anything else. And when finishing your project, sharing the code or writing a post about your experience is much appreciated! 😄

The above worked but I'm stuck on a related issue. The webview does not want to expand to fill its enclosing div. I can edit height in the running page through developer tools and that solves the problem, but I'm unable to set the height in code. If I do this:
R.div [ Style [ GridArea "2 / 1 / 2 / 1"; CSSProp.Width (U2.Case2 "100%"); CSSProp.Height (U2.Case2 "100%");  ] ] [

            R.from WebView
                !!["src" => "http://stackoverflow.com/questions/21103622/auto-resize-image-in-css-flexbox-layout-and-keeping-aspect-ratio";
                     "style" => "height: 100%;";
                  ][]
    ]

it compiles but runs with an empty page (root div is never replaced).
If I replace "style" => "height: 100%;"; with "height" => "100%"; then I get a height property in the dom but it is not part of a style element so nothing happens.
Could you please explain how to add CSS properties when using the R.from construct?
BTW I modified your electron hello world example by including vscode debugging and a webpack watch task

Nice, thanks for extending the sample! :+1:
Sorry, I don't know exactly how this WebView component works so I cannot tell for sure. Maybe you can put the CSSProp list directly in the props object passed to WebView?
R.from WebView !![
    "src" => "http://stackoverflow.com/questions/21103622/auto-resize-image-in-css-flexbox-layout-and-keeping-aspect-ratio";
    "style" => [CSSProp.Height "100%"]
][]

If this doesn't work, could you post the JS you want to generate? Cheers!

Brilliant!


Some Modifications on Distributed Rendering

When I used distributed rendering, I found no difference from normal rendering.
So I debugged the source code and found some problems in the official package. After I modified some of the source code, the map was displayed properly. I don't know if this is the best way to modify it, but it worked. The package I used are "geowave-hbase-0.9.8-apache.jar" and "geowave-geoserver-0.9.8-apache.jar".
Here are some of my modifications.
1.The file "\extensions\adapters\vector\src\main\java\applicationContext.xml" was lost in the package "geowave-geoserver-0.9.8-apache.jar". So distributed rendering doesn't work.
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>
<bean id="getMapCallback" class="mil.nga.giat.geowave.adapter.vector.render.DistributedRenderCallback">
<constructor-arg ref="wms" />
</bean>
</beans>

2.Class "mil.nga.giat.geowave.adapter.vector.render.DistributedRenderCallback", Method "beforeLayer".
I think this should be "style.getTransformation() instanceof ProcessFunction ". The condition "style instanceof ProcessFunction" is always false.
// if (style instanceof ProcessFunction && style.getTransformation() != null
if (style != null && style.getTransformation() != null
	&& (((ProcessFunction) style.getTransformation()).getName() != null)
	&& ((ProcessFunction) style.getTransformation()).getName().equals(
	 DistributedRenderProcess.PROCESS_NAME))

3.Class "mil.nga.giat.geowave.adapter.vector.render.InternalDistributedRenderProcess", Rendering is repeated here.
if (features != null) {
	final SimpleFeatureIterator it = features.features();
	if (it.hasNext()) {
		final SimpleFeature resultFeature = it.next();  //features.features().next();

4.Class "mil.nga.giat.geowave.adapter.vector.plugin.GeoWaveFeatureSource", Added code
@Override
protected boolean canRetype() {
	return true;
}

5.Class "mil.nga.giat.geowave.datastore.hbase.operations.HBaseOperations",Method "aggregateServerSide", Added a line of code to set InternalAdapterId.
if (readerParams.getAggregation().getLeft() != null) {
  if (readerParams.getAggregation().getRight() instanceof CommonIndexAggregation) {...}
  else if (readerParams.getAggregation().getLeft() instanceof InternalDataAdapterWrapper) {
      InternalDataAdapterWrapper wrapper = ...
      requestBuilder.setAdapter(ByteString.copyFrom(URLClassloaderUtils.toBinary(wrapper.getAdapter())));
      requestBuilder.setInternalAdapterId(ByteString.copyFrom(ByteArrayUtils.shortToByteArray(wrapper
							.getInternalAdapterId()))); // added
  }
  else {...}
}

6.Class "mil.nga.giat.geowave.datastore.hbase.coprocessors.AggregationEndpoint", Method"aggregate".
All of adapters will be displayed if not set internalAdapterId.
if (request.hasAdapter()) {
	final byte[] adapterBytes = request.getAdapter().toByteArray();
	dataAdapter = (DataAdapter) URLClassloaderUtils.fromBinary(adapterBytes);
}
//else if (request.hasInternalAdapterId()) {
if (request.hasInternalAdapterId()) {
	final byte[] adapterIdBytes = request.getInternalAdapterId().toByteArray();
	internalAdapterId = ByteArrayUtils.byteArrayToShort(adapterIdBytes);
}

7.Class "mil.nga.giat.geowave.adapter.vector.render.DistributedRenderAggregation", Method "initRenderer". The time here should be milliseconds, while "options.getMaxRenderTime()" is in seconds.
//formatOptions.put("timeout", options.getMaxRenderTime());
formatOptions.put("timeout", options.getMaxRenderTime() * 1000);

I have 2.12 million polygon features, the map display takes 7 seconds. It seems that everything is all right. But when the data exceeds 10 million, the region servers dead. This is the next problem to be solved. Can someone help me？ @rfecher

Thats great that you were able to work through it. I have it working in some deployments but our integration tests are checking correctness when using this SLD. The challenge is ensuring it is in fact rendering it distributed.  I'll have to double check the current IT and look through your suggestions when I have a chance.  Thanks for the feedback!
Re: the region servers dying on 10 million polys, I'd be happy to try to help - is there anything in the region server's logs that might be relevant?

Yes, there are some warnings in the log, but I don't know what's wrong.

2019-03-11 14:30:51,504 WARN
[RpcServer.default.FPBQ.Fifo.handler=193,queue=13,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"ExecService(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$CoprocessorServiceRequest)","starttimems":1552285653329,"responsesize":7327,"method":"ExecService","param":"coprocessorService=AggregationService:aggregate","processingtimems":198175,"client":"<IP_ADDRESS>:29659","queuetimems":0,"class":"HRegionServer"}

2019-03-11 14:31:05,217 WARN
[RpcServer.default.FPBQ.Fifo.handler=197,queue=17,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"ExecService(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$CoprocessorServiceRequest)","starttimems":1552285667533,"responsesize":12533,"method":"ExecService","param":"coprocessorService=AggregationService:aggregate","processingtimems":197684,"client":"<IP_ADDRESS>:29659","queuetimems":0,"class":"HRegionServer"}

2019-03-11 14:32:06,951 INFO
[LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=521.11 MB, freeSize=27.44 MB, max=548.55 MB, blockCount=8179, accesses=39755881, hits=22147798, hitRatio=55.71%, , cachingAccesses=20851029, cachingHits=19913807, cachingHitsRatio=95.51%, evictions=44830, evicted=906365, evictedPerRun=20.21782288645996

one thing I can suggest trying is adding more splits.  When you setup your geowave index you can choose numPartitions which will automatically give you more randomized splits and may allow for a single region to complete the aggregation in sufficient time (although there will be more regions so the total time will likely be similar).  One other thing I had considered was to enable aggregation either within the coprocessor or within spark.  Would aggregating (in this case distributed rendering) in spark be a good option in your case?

Thank you for your suggestions, I'll try to use aggregation in spark.

no prob - if you have a contribution you think would be generally useful (distributed rendering in spark could definitely qualify), PRs are always appreciated 😃

@rfecher @hw4270
Hello,
I was trying to read more about distributed rendering for days but , can not find any info anywhere except this issue keeps popping up.
Can you please give some info about it like :
1- There is an DistributedRendering.sld in examples folder, but how do we use it ? Is it even relevant ?
2- Is distributed rendering only applicable to Accumulo and HBase ? For ex Cassandra and Redis not available for distributed rendering ?
3- Do we need to install some additional jars / applications on data nodes to make distributed rendering work ?
4- How does Geoserver tells nodes to render data and how do nodes understand it ? (Any client apps on nodes ?)
5- Can you supply some example project / data for me to work on and understand it ?
Best regards and sorry for the inconvenience

yes, that example SLD is how you'd use it through GeoServer (in particular it is enabled through the <Transformation> element defining a geowave:DistributedRender render transform)
yes, although it will "work" with any data store but the only time the processing is truly distributed is when you have server-side processing which is the case for HBase (through Coprocessors) and Accumulo (through Iterators) only at the moment
yes, both Accumulo and HBase have a concept of a distributed classpath and a particular geowave jar needs to be available (the jars are appropriately marked geowave-accumulo and geowave-hbase in the downloads section).  We also have RPMs for each which automatically place the appropriate jar on the distributed file system.  Theer should be detail instructions available.
No "special" client apps on nodes other than of course what is already in the stack for the datastore you're using (ie. HBase region server or Accumulo tablet server).  GeoWave has a built-in concept of an Aggregation which takes advantage of server-side processing and can be anything from computing counts or bounding boxes for a given query to distributed rendering.  The render transform above triggers geowave to use a distributed rendering aggregation.
No
The example Distributed Rendering SLD should suffice. Take the Vector Demo as an example but instead of using the SubsamplePoints SLD use the Distributed Rendering SLD.


Consider using groupby instead of alternative approaches to increase speed

          We aren't using DataFrame.first(), we're using an identically named method from GroupBy:

https://pandas.pydata.org/docs/reference/api/pandas.core.groupby.DataFrameGroupBy.first.html#pandas.core.groupby.DataFrameGroupBy.first
I would recommend again to use the groupby approach, it is around 5x faster than the current code:
In [19]: %%timeit
    ...: df = df2.loc[:, ["uc_n"]].drop_duplicates(keep="first")
    ...: df = df.merge(
    ...:     df2.loc[:, ["uc_n", "uc_desc"]]
    ...:     .drop_duplicates(keep="first")
    ...:     .dropna(),
    ...:     how='left',
    ...:     )
    ...:
2.57 ms ± 62.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [20]: %timeit df2.groupby('uc_n', sort=False).first()
414 µs ± 7.53 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

Originally posted by @siddharth-krishna in https://github.com/etsap-TIMES/xl2times/pull/179#discussion_r1489513450

Let's get related changes from #177 first...


Datagrid column sorting changing information is not read by screen reader

Describe the bug
When the column header is clicked, the sorting changing information is not read by screen reader
How to reproduce
Steps to reproduce the behavior:

Go to 'stackblitz clarity datagrid simple sample'
Run a screen reader such as VoiceOver for MacOS or NVDA for Windows
Click on datagrid 'name' column header
The sorting of name column is changed, but nothing read by the screen reader

Expected behavior
The sorting changing information should be read by the screen reader.
Versions
Clarity project:

[ ] Clarity Core
[x] Clarity Angular/UI

Clarity version:

[ ] v3.x
[ ] v4.x
[x] v5.x

Framework:

[x] Angular
[ ] React
[ ] Vue
[ ] Other:

Framework version:
Angular 11

We had such type of functionality a few versions ago, but remove it because it was the introduction a lot of complexity to the UI and did not always was working as people expected.
As a workaround, I could suggest using the aria-live attribute to announce the change. Our policy for v5 was that Aria Live events will be a developer's responsibility - for the reason mentioned above.


Quadrat.framework: No such file or directory

You wrote iOS 7.0+, but embedded frameworks are supported by iOS 8+. Please change ReadMe until you support 7.0+. If you try build project with iOS 7 target, error occurred Quadrat.framework: No such file or directory.

Thank you. I have updated readme. You can still use das-quadrat on iOS 7 if you add source code files directly into your project.

It's a good practice to add readme how to install it via git submodule for < iOS 8.


Timesheet - 15/04/2021 - Nikhil Taneja

Task 1
Reviewed timesheets
Proof of Work

Commented "Approved" for approved timesheets or left other comments to ask for additional Infos to the timesheet issues dated 14/04/2021
https://github.com/thenewboston-developers/Contributor-Payments/issues

Time Spent
1 hour

Task 2
Audit contributor payments for 12/04/2021

Checked sent payment amounts were equal to worked hours x hourly rate.
Checked payments made to the correct account.
Checked payments were not missed.

Proof of Work

https://drive.google.com/file/d/1kONAyyQ5Rk4nItm76N5QU9vklCmH7QX9/view

Time Spent
0.5 hour

Task 3

Added my view on increasing hourly rate of auditors
Added section "Ideal day for an auditor might involve doing tasks like"

Proof of Work

https://docs.google.com/document/d/11qWPCVpcJyH1eeRstjLwBsyBS_kA5m1V40boakWleMk/edit

Time Spent
0.5 hour

Task 4

Minor discussion of Discord with Nikhil, Chuy, and Muhammad regarding Audit.
Suggested keeping payment records in https://github.com/thenewboston-developers/Contributor-Payments/tree/master/payments for public access

Proof of Work

Chats can be verified by Audit team in #All channel

Time Spent
0.5 hour

Total Time Spent
2.5 hours
Account Number
e9c5acac0806aca6ba2c0ade74d93ec4f9a89d8743fa477c52ce9b7817dcad95

Approved


Update Dockerfile add node-conf

Fix : https://github.com/OT-CONTAINER-KIT/redis/issues/34

@drivebyer
Right now I have stop pushing for the exporter image since I don't have access to that repo so action have been failing.


AngularFire version is ahead

Even though it appears to have already been published, AngularFire is back down to v0.9.0, which makes bower install fail.
However, this means that previous changes should be regressed.
Maybe the README should encourage users to download the latest release?
Maybe the README should encourage users to download the latest release? Cloning non-functional versions of the code is confusing.


voange rebrand

Change to Vonage is fine, but does this need to keep the Counter and Weather stuff?

It's boilerplate stuff that's shoved in by the CLI when you generate the projects. I don't mind ripping it out but technically it puts the project in a different end state than spoken to by the blog post.


Root password updated

Root password updated does not work anymore.. :(

What OS did you selected? Can you post an image or module logs to give us a hint?

I have tested with debian and ubuntu. Logs > Module logs no any logs there..
There is screenshoots of how does looks created VMs:

Root password updated not updating.... :(
I have tested with new ppk , debian 9 and debian 10. Whmcs installed on debian 9. Version newest.
IP filled OK, Username not changed to root, Root password NOT updated.
Please help...

P.S.
I get error when clicked on Update stats (server admin).
scaleway_updatestats
Request
Server ID: (.....................................)
SSH Key length:
PuTTY-User-Key-File-2: ssh-rsa
Encryption: none
Comment: Scaleway
Public-Lines: 12
AAAAB3NzaC1yc2EAAAABJQAAAgEAi(....
...............................................)
Response
Net_SSH2 => failed to login to server via SSH2! => unable to establish connection


Standardize the use of setter types

The newer version of ESLint and TypeScript plugins enforce this rule by default. This is a preemptive change so that we can support the latest tools. Adds setter types where they were missing from the code.

Codecov Report

Merging #6708 into dev will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               dev     #6708   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           15        15
  Lines          671       671
=========================================
  Hits           671       671

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 438819e...eb1c86d. Read the comment docs.


fix(.travis.yml): run the docker-build target on all branches

This PR allows us to test Dockerfile changes in PRs
Fixes #169

Note: https://github.com/arschles/deis-builder/commit/ca94ea26ce2f85ee3c96c09d202e56ff6d4ec872 was made because it fixes an issue in the Dockerfile, which this PR caught


refactor: Refactored pyrovision.datasets module

Following up on #94, this PR introduces the following modifications:

refactors the datasets module into: utils (general utilities functions), video_utils (utilities specific to videos), openfire (everything related to OpenFireDataset), wildfire (everything related to WildFireDataset)
reflected these changes in unittests

Once this PR is merged (which keeps every single class/function), we'll need to reassess which part are relevant to the library I think (fire labeler might not be really relevant here, perhaps in a repo dedicated to data labelling?)
Any feedback is welcome!

The link validity check failure isn't related, and will be fixed once #96 is merged. Regarding the unittest failure, I have seen this one of other PRs before, I'm not sure about its origin. It's seems like a random edge case because it doesn't always pop, and I'm not able to reproduce it locally.


added nc4cache.c to list of doxygen input files, fixed many doxygen warnings

Fixes #2186.
Add a missing code file to the doxygen build.
Part of #2329
Also fixes many doxygen warnings.

This work is included in #2336, so I will close this PR.


Dependencies not available for HDAnalyzeR package installation

When attempting to install the HDAnalyzeR package, we encounter the following error:
ERROR: dependencies ‘clusterProfiler’, ‘org.Hs.eg.db’, ‘ReactomePA’ are not available for package ‘HDAnalyzeR’
Before the ERROR there is the following warning: Timeout of 60 seconds was reached.
This issue appears to be related to the availability of certain dependencies required by HDAnalyzeR. These dependencies are crucial for the package's functionality but are not being installed automatically, leading to the installation failure.

Increasing the Timeout
First, try increasing the timeout to ensure that all dependencies have enough time to download and install, especially if they are large packages or if the network connection is slow.
options(timeout = 1200)  # Set timeout to 1200 seconds (20 minutes)

devtools::install_github("HDA1472/DA_RPackage")

Installing Dependencies Manually
If increasing the timeout does not resolve the issue, you can manually install the required dependencies from Bioconductor (or CRAN) before installing HDAnalyzeR. Here are the steps:

Increase Timeout:

options(timeout = 1200)  # Set timeout to 1200 seconds (20 minutes)

Install Bioconductor Manager:

if (!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager")

Install Dependencies:

BiocManager::install(c("clusterProfiler", "org.Hs.eg.db", "ReactomePA"))

Install HDAnalyzeR:

devtools::install_github("HDA1472/DA_RPackage")

By following these steps, you should be able to resolve the dependency issues and successfully install the HDAnalyzeR package.


Add plugin-specific setup sub-directory

It's not that nice having the plugin together with the operator setup. The most notable situation is if an operator doesn't want to use the RegisterOnStartup setting to preserve their private key. In this case, it's not practical to change the config or have two copies of the repo.
This PR adds its own setup/ sub-directory to the operator plugin, with a separate config file, .env and so on.

I think the only missing change is docs-related. Going to do that soon, but opening for reviews already


Update Azure file feature gates for 1.20 beta

Docs for https://github.com/kubernetes/enhancements/issues/1885
/assign @msau42

/milestone 1.20
/assign @SomtochiAma
/sig storage
/hold no k/k pr

related k/k PR: https://github.com/kubernetes/kubernetes/pull/96293

Hey @andyzhangx  please change the base to dev-1.20 branch

Hey @andyzhangx please change the base to dev-1.20 branch

already created a new PR: https://github.com/kubernetes/website/pull/24940
@annajung shall we also merge this PR?

Hi @andyzhangx, no need to create a new PR. I believe you have an edit option on the top right side and once you click on it, you should be able to change the base to master.

Hi @andyzhangx, no need to create a new PR. I believe you have an edit option on the top right side and once you click on it, you should be able to change the base from master to dev-1.20.

these two branches has too many diff, I would like to create new PR. And what about master branch, do we still maintain this branch?

Sure, if easier, you can close out this PR and keep https://github.com/kubernetes/website/pull/24940
master is being maintained but all work related to the release needs to be part of dev-1.20 branch. The dev-1.20 branch will be merged to master during release day.


Update Electron - MSAL Node sample to use latest version of Electron

This PR:

Addresses #4631
Updates the communication method between Main and Renderer process to use the Electron contextBridge in the preload.ts script to give the Renderer process controlled and secured access to the Node APIs the Main process has access to
Delegates authentication window creation and management to the AuthProvider class
Refactors CustomFileProtocolListener into CustomProtocolListener since the internals now register a string protocol instead of a file protocol to listen for the auth code and avoid the protocol sending a file and causing a navigation error as a side effect of a successful authorization request.

@sameerag it does seem to be backward compatible from my own testing.


[dev-tool] Better JavaScript sample support

This PR works towards clearing up some faulty assumptions that dev-tool has made about samples and makes it a bit more robust to certain constructions of sample files. It also improves the user experience of generating samples in a few highly-requested ways. The high-level goal is to make dev-tool:

more strongly enforcing of our node support matrix, so it will now reject certain ECMAScript syntaxes that are not supported in ECMAScript 2019 without downleveling or at all (bigint literal, ?., ??, ??=, ||=, &&=, number literal separators, static initializers, static class fields, and private identifiers).
enforcing of simple import/export structures that can easily convert to/from CommonJS (so named export declarations are not allowed, but the export modifier on function/const/var/let declarations is).
more helpful when presenting errors about unsupportable syntax.
respecting of nested module structures, with the capability of converting exported declarations into CommonJS exports.
more careful when generating samples for beta packages. It will now append beta signifiers to titles, URL fragments, folder names, etc. (generating samples for @azure/template v1.0.0-beta.12 will now yield a folder named v1-beta instead of v1.

It is not possible to simply use CommonJS modules in the typescript compiler configuration, as the compiler works very hard to make sure that the semantics between the two are the same, but we need to make a JavaScript file that is both accurate and presentable (readable and without strange downleveling artefacts or tslib usage).
Implementation details:
We use the TypeScript API to look for top-level statement nodes that have an ExportKeyword modifier. These are exported name declarations. Note that this doesn't include export declarations, which have their own syntax nodes and no ExportKeyword modifier. We store the exported names and then insert an assignment node at the end of the file that assigns module.exports to an object that exports all the symbols we detected.
Separately, we also now have a simple object that encodes unsupported ECMAScript versions and their runtime syntaxes that must be avoided. Previously, dev-tool has simply been rendering whatever syntax is used in the TypeScript files into the JavaScript files, but this causes problems with very new syntax such as optional chains, logical operator shorthand assignments, and certain strange class features like static initializer blocks. Dev-tool will now simply produce an error for these cases and explain that only syntax up to ES2019 is supported. This will prevent developers from using syntax that will crash any of our supported Node versions. When Node 12 reaches EOL, we can simply remove these checks.
When presenting errors, dev-tool samples publish will now use the TypeScript diagnostic engine to provide a high quality error message (that shows the unsupportable syntax and its exact location in the file).
CC @bterlson, @xirzec, @deyaaeldeen : your review is appreciated.
Closes #15140
Closes #15724
Closes #16713
Closes #18035

I haven't had the time to stare deeply into the changes from a code correctness perspective, but I really like the idea of dev-tool + samples helping to enforce everything working when copy/pasted into our lowest supported runtime. 👍

@ckairen FYI


Simple bidi integration

This is an alternative solution to https://github.com/bevyengine/bevy/pull/4672
Objective

short: unicode bidi support for text and text2d
When text is presented in horizontal lines, most scripts display characters from left to right. However, there are several scripts (such as Arabic or Hebrew) where the natural ordering of horizontal text in display is from right to left. If all of the text has a uniform horizontal direction, then the ordering of the display text is unambiguous. However, because these right-to-left scripts use digits that are written from left to right, the text is actually bidirectional: a mixture of right-to-left and left-to-right text. In addition to digits, embedded words from English and other scripts are also written from left to right, also producing bidirectional text. Without a clear specification, ambiguities can arise in determining the ordering of the displayed characters when the horizontal direction of the text is not uniform. This is solved by the Unicode Bidirectional Algorithm.

Solution

using the unicode-bidi crate and allowing for bidi correction in the Text component

Hmm. I'm having a bit of trouble piecing together how this might look in practice. Could you put together a simple bidirectional_text.rs example?

It should be completely transparent to the user - for example the text example: I just changed the String to some characters with RTL:

and the result is a rendered text with corrected directions (well - if the font would support those characters ... ):

Standard examples with RTL/Bidi text and new font containing hebrew chars.

I would like to avoid the risk of having sections not be in sync with bidi_corrected. Would it make sense to remove pub from sections (and adding accessors), and update directly bidi_corrected  when updating sections?

This makes working with sections less convenient and might trigger the bidi correction way too often (every time a text or style is changed - even if it is changed several times in one update). I'm actually not sure if moving the bidi correction into the Text is the best way - to me it seems more in line with the ECS to move the corrected sections into an additional component and have the text_system react to text changes with creating the bidi sections once per update (as in the first approach). Maybe we can make that less visible (and therefore confusing) for the user?

this PR seems nearly ready
and hope it get merged it the next release

This makes working with sections less convenient and might trigger the bidi correction way too often (every time a text or style is changed - even if it is changed several times in one update). I'm actually not sure if moving the bidi correction into the Text is the best way - to me it seems more in line with the ECS to move the corrected sections into an additional component and have the text_system react to text changes with creating the bidi sections once per update (as in the first approach). Maybe we can make that less visible (and therefore confusing) for the user?

What about adding a boolean flag to indicate whether the bidi correction is up-to-date?
Would add some more complexity, but then we could check the flag if it's up-to-date, if yes we just use the cached value, if no we recompute the correction. When the text is edited or newly created we set the flag to false.
That way we could minimize the number of corrections applied and only apply them when the text is actually rendered

This makes working with sections less convenient and might trigger the bidi correction way too often (every time a text or style is changed - even if it is changed several times in one update). I'm actually not sure if moving the bidi correction into the Text is the best way - to me it seems more in line with the ECS to move the corrected sections into an additional component and have the text_system react to text changes with creating the bidi sections once per update (as in the first approach). Maybe we can make that less visible (and therefore confusing) for the user?

What about adding a boolean flag to indicate whether the bidi correction is up-to-date? Would add some more complexity, but then we could check the flag if it's up-to-date, if yes we just use the cached value, if no we recompute the correction. When the text is edited or newly created we set the flag to false. That way we could minimize the number of corrections applied and only apply them when the text is actually rendered

On a second thought, it would be better to make the cached value an Option instead of introducing a boolean flag, then we can simply set it to None when the text changes and ensure that the previously cached value is not used in that case

@TimJentzsch Thank you for your input - maybe you could have a look at the alternative implementation https://github.com/bevyengine/bevy/pull/4672 ? I would really like some feedback :)

Overall good quality on the code, I realize I need to read up on the details of bidi in general to feel able to review this properly. While this looks straightforward in its implementation, I think I prefer #4672 to this one.
I do like the .add_section() builder pattern. Maybe we should add it as a separate PR?

Reading up on Bidi is certainly interesting, but luckily all the heavy lifting is done in the dependency ;-)
Adding the builder pattern in the other PR would be easy - or we just add it as a separate PR afterwards.

or we just add it as a separate PR afterwards.

This would be my preference. We're already in the process of improving the UX around similar issues, see this converastion on discord

Backlog cleanup: closing, as cosmic-text already implements unicode bidirectional, and this is coming in 0.15. See also  #15592.


Implement an exhaustive API for Reaper

I would like to implement an exhaustive API for Reaper.
The current state of the art is:

Some of the exposed REST methods are implemented, mostly in the Cluster and RepairSchedule resources, but not all of the optional parameters are available.
A PR #3 is open to add an initial support for the RepairRun resource, again not all of the methods and optional parameters are implemented.

Here is my proposal:

Complete the existing resources by adding the missing methods and/or parameters, re-using the material in #3 for RepairRun methods.
Add support for missing resources, and mostly: Snapshot and Node.
Add support for authentication.
Add tests for new methods reusing the current (and excellent) test environment using Docker.

These things can of course be done little by little, and by different developers.
But I think it would be important to agree on the desired API first.
Indeed, some of the existing API methods and types might need to be renamed or re-engineered to comply with things like:

consistent naming of types, and methods and parameters
consistent mapping of REST endpoints to Go methods
how to expose optional parameters
consistent behavior (consistent error messages, etc.)

Also, I think it is important to settle on a common code related to HTTP plumbery, so that new methods can be implemented faster and more reliably. This will likely require some refactoring of the existing HTTP-related functions.
I took the initiative to bootstrap a design document with a proposal for a revised API:
https://docs.google.com/document/d/1xau9Lr9BsctZBgpoFnWJmmkFz5c885O8MhqkGWMca1I/edit?usp=sharing
Feel free to add comments there or here.
If/when we agree on the proposed changes, I can start working on this ASAP and provide implementations for the most important resources as a first step: Login, Ping, Cluster, RepairRun and RepairSchedule.
┆Issue is synchronized with this Jira Task by Unito

I'm struggling with one major thing: Why? Is there even a ticket which explains why these are needed?
The reaper-client-go was only ever intended to be used for testing of reaper-operator features. None of these enhancements make a difference in that sense.

Why?

Maybe for nothing :-) I just thought it would be cool for Reaper to have an official Go client, but maybe I'm overthinking all of this.

None of these enhancements make a difference in that sense.

We would need at least a minimal RepairRun and RepairSchedule support in order to implement https://github.com/k8ssandra/k8ssandra/issues/580.

Yes, RepairRun / RepairSchedule (like #3 started) would be necessary to test that scheduling was done properly. But otherwise, I personally think Reaper is something we wish the user would never have to worry about or even think about.

From the K8ssandra perspective I do think we want Reaper to be something that users don't have to think too much about, so I agree with you there @burmanm
Looking at it a bit more broadly though it does seem like a useful piece for the broader Reaper community, which has plenty of bulk outside of K8ssandra. I'm also thinking that this could prove fairly useful if there were to be a K8ssandra UI/API/client that could leverage this as a good clean API integration point.
There is however a lot here, we'd need to think about how to prioritize things and see what might have the most value in the K8ssandra context.
Super nice doc @adutra thanks for putting the thought and time into this idea.

We would need at least a minimal RepairRun and RepairSchedule support in order to implement k8ssandra/k8ssandra#580.

Correcting myself: we don't need repair schedules currently, only repair runs. I listed the 3 calls that need replacement here: https://github.com/k8ssandra/k8ssandra/issues/580#issuecomment-852061668.
With that in mind, I think we could start by providing just the RepairRun resource implementation. Others can definitely wait.

As discussed today with the team, I'm deprecating the Google doc mentioned above in favor of a PR to discuss the API design: #5 .


Allow custom positioning of overlays

Allow the provision of a position function for overlays via DI.
Let's search for a more idiomatic solution in the next round together with work related to #675.

Fixed via #691


getLoader SHOULD return null if loader does not exist

Actually, it generates an error : Cannot read property 'loader' of undefined
loaderService.getLoader('someUnititilizedloader');

PR created : https://github.com/biig-io/ngx-smart-loader/pull/11

Done with #12 👌


Add a back to the home page Navigation for the logo at the footer

Is your feature request related to a problem? Please describe.
The logo in the footer is currently unresponsive. I would like to change that
Describe the solution you'd like
Add navigation to the logo so that it takes the user back to the top of home page
Describe alternatives you've considered
No response
Additional context
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

@AshutoshDash1999 please check this issue

Add navigation to the logo so that it takes the user back to the top of the home page

It's already there. We have to click-to-top button.


PRPL server and differential serving

Are you working on allowing serving of different builds based on the browser's abilities? I currently use prpl-server and it's an important part of our system.
What's the current go with open-wc?

When prpl-server was conceived the front-end world looked quite different. Not all browsers were evergreen, and feature support in browsers varied wildly. Es modules were not yet a thing, there was not yet a <script type="module"> and <script nomodule> trick to load different versions and we were still hoping that we could serve unbundled code over HTTP2.
For better or worse, times have changed. Aside from IE11, browsers are evergreen and support is very good on average. Unbundled serving over HTTP2 has been disproven (https://philipwalton.com/articles/using-native-javascript-modules-in-production-today/ is a good read) and we now have better tools available for differential serving. <script type="module"> / <script nomodule> but also <link rel="prefetch"> which enabled runtime feature detection.
Since IE11 is becoming less important, our default build config just builds for modern browsers. We think that in the current landscape, one build is good enough to serve these browsers. There isn't benefit from serving different builds to different modern browsers, that energy is better spent optimizing other parts. This may change in the future.
For rollup and webpack we also have a "compatibility" build config, which creates a separate build output for legacy browsers (mostly IE11 at this point). This uses runtime feature detection to decide which bundle to load. We've set it up in such a way that there is minimal impact on load times, you can read more about that here https://github.com/open-wc/open-wc/tree/master/packages/polyfills-loader
Serverside feature detection is theoretically faster, but I think it is quite a hassle for a lot of people to get this set up. You can no longer us a static file server, you need to add some extra behavior in your server. I am not sure if the benefits are large enough to justify that.

This makes so much sense... Please please please and please write this
somewhere prominent in the documentation -- both for open-wc and, why not,
prpl-server.
By looking at the polymer site, it looks like PRPL is still totally a thing.
On Sun, 2 Feb 2020, 5:29 pm Lars den Bakker<EMAIL_ADDRESS>wrote:

When prpl-server was conceived the front-end world looked quite different.
Not all browsers were evergreen, and feature support in browsers varied
wildly. Es modules were not yet a thing, there was not yet a

We could add this as a “Does open-wc support PRPL?” in the FAQ

I humbly recommend adding this very explanation to the "publishing" section
https://open-wc.org/publishing/#manual-setup
This is where sobs like me will expect to find it!
On Sun, 2 Feb 2020 at 19:28, Pascal Schilp<EMAIL_ADDRESS>wrote:

We could add this as a “Does open-wc support PRPL?” in the FAQ
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/open-wc/open-wc/issues/1298?email_source=notifications&email_token=AAQHWXSTOIUA3LW3WFIGW4TRA2U4LA5CNFSM4KOW4OFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKRUPYQ#issuecomment-581126114,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAQHWXUJS73ZBRY3YXIIBYTRA2U4LANCNFSM4KOW4OFA
.

The entire docs on building, deploying/shipping apps and components is in need of a general restructuring / expansion soon :)


Option to disable formatting on console logs [FEATURE SUGGESTION]

I have a suggestion:

Provide a flag to disable formatting on messages
When reading logs in Elasticsearch, Kibana, we can see the color tags in of the logs. The formatting doesn't help us there however the other functionality (timestamps, message types, etc.) do help us. To clean up logs, I'd like to remove any color, styling etc. settings and I have made a rudimentary branch where setting disableFormatting=true allows different conditional blocks to trigger where no coloring, boldness etc. is applied to messages.

Please let me know what you think.

Hello @catonice,
This was implemented as { highlight: Boolean } option for LoggerConsole constructor, see updated docs. Please upgrade to 2.1.0 and let me know if this is solved
Feel free to close it in case if the issue is solved on your end.


MRG, ENH: Speed up examples

Tries to speed up our two slowest examples:
    - ../tutorials/source-modeling/plot_forward.py: 115.75 sec    448.3 MB
    - ../tutorials/source-modeling/plot_mne_solutions.py: 96.71 sec    383.4 MB

In the first one, if we use fewer sources it's faster. In the second one, just showing the loose inverse seems like enough (no need to compute and show fixed and free).

I am not too enclined to sacrifice pedagogy for CI speed. Users will copy paste from tutorials without realizing they don't use good default or will complain that tutorials use precomputed stuff that they don't know how to produce with their own data

Okay we can live with the fwd being slow then, and maybe I'll redo the mne_solutions to still show free and fixed, but not all four variants for each (the loose eloreta is the slow one anyway)


WMSID: 6661

Add here your WMS ID
General requirements

Information in the workshop is adequate and updated
Code is correct and working
Links are correct
Make sure you enter the help email link in your manifest.json
Please make sure WMS URLs are updated as needed after your PR is approved

Checklist - Refer to the QA document for the complete list
Please confirm that the following is completed before submitting your PR

[ ] All filenames are lower case (including folders, images, files, etc.)
[ ] Filenames are descriptive
[ ] Your workshop folder structure should be similar to the one used in the sample workshop (https://github.com/oracle-livelabs/common/tree/main/sample-livelabs-templates/sample-workshop)
[ ] Are you using multiple versions (desktop/, sandbox/, tenancy/)? Make sure that each of them contains a manifest.json and an index.html
[ ] Image references in markdown contain an alternate text

Update to OIRI workshop : images have been changed from firefox to chrome

Confirmed. PR is for #6661 and not 9762
@ibalasun please address the remaining items from @arabellayao request above

Please remove .DS hidden files

Hi! I am closing this PR now. Please fix the following issues suggested by Madhu and me, and create a new PR when you finish the changes.

Please remove .DS hidden files
Images’ names should be descriptive instead of step numbers
All images should have descriptive alternative texts in the Markdown files for accessibility
Please refer to the self QA form for details and let me know if you have any questions. Thanks!


Missing tabs for indent level when assign boolean statement to a variable

Describe the bug
Missing tabs for indent level when assign two or more boolean expressions to a variable
Erroneous code
static void	test(int a, int b, int c)
{
	a = c != 0 || c != 5;
}

Additional infos

OS: 20.04.4 LTS
python --version: 3.8.10
norminette -v: 3.3.51

Additional context
Error: TOO_FEW_TAB          (line:  15, col:  19):	Missing tabs for indent level

fixed in 3.3.52


Will it be updated to 1.17.1?

I have this issue and its very annoying but everytime i try to run minecraft with this mod it crashes

I'm not maintaining this project any further. However, all the code is ported to CaffeineMC/phosphor-fabric which serves as a replacement.