JDK9 support

Hello, thank you for your great tool. Looks like it wasn't updated for long and it doesn't support JDK9. On start of my webapp I got the following:
>> jHades - scanning classpath for overlapping jars:

Mar 02, 2018 5:37:56 PM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class org.jhades.JHadesServletListener
java.lang.NoClassDefFoundError: sun/misc/Launcher
	at org.jhades.model.ClazzLoaderFactory.createBootstrapClassLoader(ClazzLoaderFactory.java:34)
	at org.jhades.service.ClasspathScanner.findAllClassLoaders(ClasspathScanner.java:236)
	at org.jhades.service.ClasspathScanner.findAllClassLoaders(ClasspathScanner.java:233)
	at org.jhades.service.ClasspathScanner.findAllClassLoaders(ClasspathScanner.java:233)
	at org.jhades.service.ClasspathScanner.findAllClasspathEntries(ClasspathScanner.java:68)
	at org.jhades.service.ClasspathScanner.findAllClasspathResources(ClasspathScanner.java:107)
	at org.jhades.service.ClasspathScanner.findOverlappingJars(ClasspathScanner.java:262)
	at org.jhades.JHades.overlappingJarsReport(JHades.java:189)
	at org.jhades.JHadesServletListener.runJHades(JHadesServletListener.java:35)
	at org.jhades.JHadesServletListener.contextInitialized(JHadesServletListener.java:27)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4939)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5434)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
	at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.lang.ClassNotFoundException: sun.misc.Launcher
	at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1702)
	at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1547)
	... 20 more

Mar 02, 2018 5:37:56 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext

P.S.  http://jhades.org  seems not working anymore

@jhades this has just come up on https://jira.opendaylight.org/browse/INFRAUTILS-52 by @skitt ...
Do you have any plans to look into and fix this? Or, if someone were to raise a PR with a fix, would you merge and push a new release to Maven Central? @stokito would you consider contributing to this open source project with a Pull Request which fixes this?

Hi @vorburger I can try to find some hot fix but actually I think that from practical point of view this not needed. Let me explain what I mean. The only one way to fix dependency hell is to engage microservices:

Division by domain and deployment cycle: it's ok for the Admin Panel to use some legacy Tapestry 4 while main processing server using latest Guava i.e. it's ok to update/redeploy the admin panel rarely while the Processing Server often and keep it up to date. Also different domains has different load/usage profiles and all of them sharing the same CPU cache which is some nature bottle neck.
When you have two libraries and both uses transitively another lib but with different mayor version  (aka ambient dependency) then you got a serious dependency hell. If your service is small then it's unlikely happens.
If you writing a micro service then just use the most popular framework i.e. Spring Boot. In fact SB is a collection of right versions of different libraries.  Just update SB itself.

 You can use JHades to fix biggest problems even before migrating to JDK9.

Maybe not now but eventually you will face the problems again (i.e. business will be changed and you need fully replace part of your system) and you'll anyway rewrite your system to microservices.

@stokito thanks for your thoughts. I want it anyway... :smile:
@jhades I've looked into this, and unless I'm missing something realized this is far less trivial than I thought... while one could perhaps replace the use of sun.misc.Launcher & sun.misc.URLClassPath in org.jhades.model.ClazzLoaderFactory by java.lang.ClassLoader either getSytemClassLoader() or the Java 9 only getPlatformClassLoader() to avoid the ClassNotFoundException shown above, that is actually the least of all problems...
The real issue is that in Java 9+ the application classloader is no longer an URLClassLoader - but JHades is pretty much built on the assumption of an URLClassLoader ...

While there may be a way to signficantly adapt JHades to fix this, I've instead opted to build something quite like it on top classgraph, another existing library which is very convenient for any sort of classpath scanning in general (which duplicate detection is basically an application of) - see https://github.com/classgraph/classgraph/issues/256.

ClassGraph has not only support for JDK9+, but support for an extremely wide array of different classpath specification mechanisms. Maybe it makes sense to rebase JHades on top of ClassGraph, for maximum compatibility with a wide range of projects? Feel free to test that out.
https://github.com/classgraph/classgraph/wiki/Classpath-Specification-Mechanisms
At @vorburger's suggestion, I added this mechanism to ClassGraph to find duplicated classfiles (or drop ".classFilesOnly()" if you want to find all duplicated resources):
new ClassGraph().scan().getAllResources().classFilesOnly().findDuplicatePaths()
@vorburger has a list of common duplicated resource names that should be ignored.

@vorburger has a list of common duplicated resource names that should be ignored.

see the ClasspathHellDuplicatesChecker in OpenDaylight's infrautils
http://blog2.vorburger.ch/2019/02/how-to-find-duplicate-classes-and.html


Suggestion: Use absolute urls when referencing issues on this repo

When a new commit is pushed to this repo, Lincoln often links to related issues by relative urls like this #564.
However, it causes problems to fsharp/fsharp and FCS because when code transfering is done, a lot of issues there get closed automatically. It's wrong and difficult to track which issues have been closed mistakenly. Moreover, the relative urls also create bogus issue references on fsharp/fsharp and FCS, which makes it harder to examine issues in those repos.
An occurrence of this problem happened a few days ago when F# 4.0 code flows to FCS.
I suppose if we use absolute urls like this  https://github.com/Microsoft/visualfsharp/issues/564 when referencing issues in pushed commits, above-mentioned issues no longer apply.
Please help to give clean history to fsharp/fsharp and FCS. Thanks.

Good feedback, makes sense. We'll make sure to do that in the future!

I've started doing this with recently-merged cleanup PRs.  Note that GH still renders it as a short link:

But in the raw commit it's the full URL

Confirm that it works fine (see https://github.com/fsharp/FSharp.Compiler.Service/commit/ae910eec18f00727834d8fa1c1254c5aaff06cb7).
Thanks.


Constantly Reestablishing Connections in Cluster Mode

Expected Behavior
Creating a cluster client using pretty much default settings should not overwhelm Redis with constant barrage of new connections.
redis.NewClusterClient(&redis.ClusterOptions{
    Addrs: []string{redisAddr},
    TLSConfig: &tls.Config{},
})

Current Behavior
Occasionally, at times completely unrelated to system load/traffic, we are seeing connections being constantly re-established to one of the cluster nodes in our Redis cluster. We are using ElastiCache Redis in cluster mode with TLS enabled, and there seems to be no trigger we can find for this behavior. We also do not see any relevant logs in our service's systemd output in journalctl, other than
redis_writer:85 {}        Error with write attempt: context deadline exceeded

which seems more like a symptom of an overloaded Redis cluster node rather than a cause.
When this issue happens, running CLIENT LIST on the affected Redis node shows age=0 or age=1 for all connections every time, which reinforces that connections are being dropped constantly for some reason. New connections plummet on other shards in the Redis cluster, and are all concentrated on one.
New Connections (Cloudwatch)

Current Connections (Cloudwatch)

In the example Cloudwatch graphs above we can also see that the issue can move between Redis cluster shards. As you can see, we're currently running with a 4-shard cluster, where each shard has 1 replica.
Restarting our service does not address this problem, and to address it we basically need to do a hard reset (completely stop the clients for a while, then start them up again).
We've reached out to AWS support and they have found no issues with our ElastiCache Redis cluster on their end. Additionally, there are no ElastiCache events happening at the time this issue is triggered.
Possible Solution
In this issue I'm mainly hoping to get insight into how I could better troubleshoot this issue and/or if there are additional client options we can use to try and mitigate this worst case scenario (i.e. rate limiting the creation of new connections in the cluster client) in absence of a root-cause fix.
My main questions are:

Is there a way for me to gather more data that would be helpful for the Redis/go-redis experts here?
Is there a way for us to rate-limit the creation of new connections in the ClusterClient to keep things from getting too out of control if this does continue to occur?
Has anyone else encountered a similar issue with Cluster mode, whether or not it was with ElastiCache Redis?

Steps to Reproduce
The description of our environment/service implementation below, as well as the snippet of our NewClusterClient call at the beginning of this issue, provide a fairly complete summary of how we're using both go-redis and ElastiCache Redis. We've not been able to consistently trigger this issue since it often happens when we're not load testing, and are mainly looking for answers for some of our questions above.
Context (Environment)
We're running a service that has a simple algorithm for claiming work from a Redis set, doing something with it, and then cleaning it up from Redis. In a nutshell, the algorithm is as follows:

SRANDMEMBER pending 10 - grab up to 10 random items from the pool of available work
ZADD in_progress <current_timestamp> <grabbed_item> for each of our items we got in the previous step
Any work items we weren't able to ZADD have been claimed by some other instance of the service, skip them
Once we're done with a work item, SREM pending <grabbed_item>
Periodically ZREMRANGEBYSCORE in_progress -inf <5_seconds_ago> so that claimed items aren't claimed forever

Currently we run this algorithm on 6 EC2 instances, each running one service. Since each instance has 4 CPU cores, go-redis is calculating a max connection pool size of 20 for our ClusterClient. Each service has 20 goroutines performing this algorithm, and each goroutine sleeps 10ms between each invocation of the algorithm.
At a steady state with no load on the system (just a handful of heartbeat jobs being added to pending every minute) we see a maximum of ~8% EngineCPUUtilization on each Redis shard, and 1-5 new connections/minute. Overall, pretty relaxed. When this issue has triggered recently, it's happened from this steady state, not during load tests.
Our service is running on EC2 instances running Ubuntu 18.04 (Bionic), and we have tried using github.com/go-redis/redis/v8 v8.0.0 and github.com/go-redis/redis/v8 v8.11.2 - both have run into this issue.
As mentioned earlier, we're currently running with a 4-shard ElastiCache Redis cluster with TLS enabled, where each shard has 1 replica.
Detailed Description
N/A
Possible Implementation
N/A

Thanks for the detailed issue :+1: We recently received https://github.com/go-redis/redis/discussions/1910, but it looks like a different problem.

redis_writer:85 {}        Error with write attempt: context deadline exceeded

This does not look like a go-redis log. We don't use redis_writer prefix anywhere, so it could be a good idea to figure out what happens here. context deadline exceeded suggests a timeout which can cause the behavior you mentioned.

Restarting our service does not address this problem, and to address it we basically need to do a hard reset (completely stop the clients for a while, then start them up again).

What is the difference between restarting a service and restarting a client?

Is there a way for me to gather more data that would be helpful for the Redis/go-redis experts here?

I would add a log here to figure out why connections are constantly closed:
func (p *ConnPool) Remove(ctx context.Context, cn *Conn, reason error) {
	+log.Printf("closing bad conn: %s", reason)
	p.removeConnWithLock(cn)
	p.freeTurn()
	_ = p.closeConn(cn)
}

Is there a way for us to rate-limit the creation of new connections in the ClusterClient to keep things from getting too out of control if this does continue to occur?

No, but problem is not with creation of new connections, but with closing of existing connections.

Has anyone else encountered a similar issue with Cluster mode, whether or not it was with ElastiCache Redis?

I would check Read/Write timeouts - anything less than time.Second is not very practical. Also context.Context deadline should be reasonably high.

@vmihailenco Thanks for the quick response!

I would add a log here to figure out why connections are constantly closed

Is there a viable hook/other way for us to inject the log you've recommended without having to duplicate the go-redis library and modify it? It's not entirely out of the question for us, but generally our builds that we promote happen on a remote build server and start from a clean state, so we would have to copypaste the whole go-redis library and no longer reference it directly using go modules to do this.

This does not look like a go-redis log.

I would check Read/Write timeouts - anything less than time.Second is not very practical. Also context.Context deadline should be reasonably high.

redis_writer is one of our files, and that log is essentially just what we print if the context.Context we passed into the SADD to pending on the writer side timed out. Each individual attempt has a timeout of 100ms, which sounds like it may be too low from what you're saying. We wanted to set an aggressively low timeout on this system but perhaps we were too brazen in our choice of timeout. Any other timeouts on the Redis client itself would be the default values, since we aren't overriding anything other than the Addrs and TLSConfig.

What is the difference between restarting a service and restarting a client?

They're essentially the same thing. However, something about having all clients shut down at the same time gets us back into a good state sometimes, whereas doing a rolling restart did not.

P.S. are there instructions for copying and modifying go-redis/redis in an existing project? Just cloning it to a subdirectory doesn't quite work, as Go doesn't like a nested project with its own go.mod/go.sum.

@vmihailenco Another question - are there any other places we can add logs to see when connections are being closed for reasons that aren't bad conns? I deployed versions of our services with a fork adding your recommended log line and although we haven't been able to reproduce the issue where we have a constant stream of new connections yet, we also don't see this closing bad conn log at all despite seeing small amount of new connections without a corresponding increase in total connections

You can also duplicate the log message at https://github.com/go-redis/redis/blob/master/redis.go#L265.

despite seeing small amount of new connections without a corresponding increase in total connections

PubSub bypasses the pool completely - see https://github.com/go-redis/redis/blob/master/pubsub.go#L145 and https://github.com/go-redis/redis/blob/master/redis.go#L651-L654

Is it worth also logging in the else clause there?
	if isBadConn(err, false) {
		c.connPool.Remove(ctx, cn, err)
	} else {
		c.connPool.Put(ctx, cn)
	}

I'm not positive what that case is responsible for.

Put returns healthy connections to the pool.

@vmihailenco This behavior fortunately started again on one of our cluster nodes this weekend! The added log I see is giving the following reason for constantly closing connections:
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:10 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:10 closing bad conn: context deadline exceeded
Oct 04 11:49:11 <hostname_redacted> <service_redacted>[25105]: 2021/10/04 11:49:11 closing bad conn: context deadline exceeded

I'm open to the possibility that increasing our timeout is the move here, but if my hypothesis is true it still seems bad that this can happen in the first place.

Around the time the latest event started, we see a smaller amount of I/O timeout logs starting as well. Hard to know which is the chicken and which is the egg though. Maybe this is helpful (i/o timeouts vs context deadline exceeded, logarithmic scale)?

Those specifically look like:
2021/10/02 15:21:46 closing bad conn: read tcp <ip_redacted>:42792-><ip_redacted>:6379: i/o timeout

and are present both for read tcp and write tcp

Try to upgrade to v8.11.4. May be it will help.
But overall go-redis does the right thing by closing connections on context deadline and/or timeouts, because such connections conns can/will receive unsolicited responses. Nothing to improve here.

That makes a lot of sense 😄
Are there any other parameters on the Redis client we can tune other than our per-transaction context deadline that could help mitigate reconnect storms like this? We're currently using a fork of 8.11.4

Hi @enjmusic I am wondering if you managed to get to the bottom of this issue. There are multiple things we have in common here. Running redis cluster on ElasticCache, Redis CPU stuck at 100% due to tens of thousand connections despite a very small pool and only able to recover by stopping all workers. We are also using context timeouts and pipeline commands.
@vmihailenco what is a reasonable high context timeline? We have this at 1s and ReadTimeout at 750ms but we do not want to have any lower to this.
We are running v8.11.3
Any advice would be greatly appreciated
Thank you

Hi @pedrocunha ! Thanks for asking, it's good to know we're not the only ones. We were not able to resolve this issue through a variety of mitigation measures, including (not all at once):

Increasing all timeouts to 1 second or greater
Lowering pool size
Reducing # of goroutines exercising our Redis claim logic
Moving all operations out of MULTI/EXEC pipelines
Completely disabling idle connection reaping
Fuzzing the interval at which each individual cluster node's Client cleans idle connections
Changing the instance type of the ElastiCache Redis nodes
Rebalancing/increasing shard count for the ElastiCache Redis cluster

We are seeing the issue a lot less often after

Reducing # of goroutines exercising our Redis claim logic
Lowering pool size

but I don't think it's a permanent fix, just a consequence of there being less potential resources to overload an individual ElastiCache Redis cluster node. We still have nothing to blame for when this 100% due to tens of thousand connections despite a very small pool happens. It seems to trigger randomly, often at times when the system is not even at its peak load.

Thanks for the reply! You listed a few things that we wanted to try too but now I feel less encouraged we will get to the bottom of it.
Couple of questions:

What is your dial timeout?
Do you also see a spike on CurrentConnections?
What does your traffic look like? We do about ~2 million GET requests per node per min (pipeline with several GETs) and this issue only happens at peak randomly during the week around 5-6pm BST.
Have you considered not using redis cluster at all?
Have you tried using SLOWLOG?

We are seeing this issue on a cache.r6g.xlarge at less than 20% CPU and 67% memory usage. I am also suspicious of us storing big objects as our NetworkBytesOut can be ~500MB/min.

Similar deal here:

10-30% CPU usage, but very low (only .15%) memory usage
Not sure if I remember/still have data for CurrentConnections, but NewConnections always ends up at around 15k for the shard in question
We're using the default DialTimeout
We tried using SLOWLOG from the AWS Console and didn't see anything interesting
Our alternative to using Redis Cluster would probably be similar to something we did in a legacy system with much more traffic where each node is a regular ElastiCache primary/replica pair. We haven't tried that solution here as we didn't want to commit to handling our own sharding and sharding configuration again
As far as maximums go for different byte rates, I see ReplicationBytes at ~200MB/min and NetworkBytesOut at ~900MB/min. Usually our payloads we're storing/fetching/deleting are around 1-2KB in size but that can vary
The busiest shards (that happen to hold more keys than others because randomness ain't perfect) see ~650k SET-type commands/min and ~180k GET-type commands/min at maximum

We encountered the same problem. Many new connections will be created suddenly at an unfixed point in time. According to the error log, the I/O timeout is displayed, so go-redis creates many new connections, which causes the redis  server load to be too high, but it still fails to find out why this happening, because the timing of such problems is completely random and the system's load bottleneck is not reached at all

Glad we aren't alone on this @klakekent. This has been an extremely frustrating issue for us. We implemented a rate limiter and it doesn't fully mitigate this issue. We have now just upgraded to redis 6.2 this morning and we are hoping to see if it helps.

@pedrocunha  And i also contact with AWS support to check this issue and they told me the Elasticache and EC2 found nothing wrong, so I think the redis client should try to check what happened. @vmihailenco

It sounds like there are three of us all experiencing the same issue now. @vmihailenco would it be possible to reopen this issue for further investigation? When we contacted AWS support, similar to @klakekent, we also were told that they count absolutely nothing wrong on their end.

@enjmusic It is best to open a new issue with summary of what is happening and what have been tried so far.

@vmihailenco @klakekent @pedrocunha I've opened a new issue for this problem https://github.com/go-redis/redis/issues/2046


odo create --s2i flag shows Warning message on 4.6 cluster

/kind bug

What versions of software are you using?

Steps:

Copy the file from https://github.com/openshift/odo/tree/master/tests/examples/source/nodejs to the working directory
$ odo create --s2i nodejs

Validation
Warning: nodejs is not fully supported by odo, and it is not guaranteed to work
 ✓  Validating component [4s]

Please use `odo push` command to create the component with source deployed

However there is no warning message reflects on 4.5
$ odo create --s2i nodejs
Validation
 ✓  Validating component [1s]

Please use `odo push` command to create the component with source deployed

Actual behavior
Shows warning message Warning: nodejs is not fully supported by odo, and it is not guaranteed to work on 4.6
Expected behavior
Functionality of component create should not depend on cluster.
Any logs, error output, etc?

So, I dug little bit more into the issue and the point of hitting warning message is the image difference on 4.5 and 4.6 . The latest image on 4.6 (registry.redhat.io/ubi8/nodejs-12) is unsupported whereas on 4.5 (registry.redhat.io/rhscl/nodejs-12-rhel7) has supported one according to https://github.com/openshift/odo-init-image/blob/master/language-scripts/image-mappings.json
Logs:
Creating component on 4.6 cluster and image description
$ odo project create blah
 ✓  Project 'blah' is ready for use ✓  New project created and now using project: blah

$ odo create --s2i nodejs --project blah
Validation
Warning: nodejs is not fully supported by odo, and it is not guaranteed to work
 ✓  Validating component [817ms]

Please use `odo push` command to create the component with source deployed

$ oc -n openshift get is nodejs -o yaml
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/display-name: Node.js
    openshift.io/image.dockerRepositoryCheck: "2020-09-23T06:14:48Z"
    samples.operator.openshift.io/version: 4.6.0-0.nightly-2020-09-22-213802
  labels:
    samples.operator.openshift.io/managed: "true"
  name: nodejs
  namespace: openshift
spec:
  lookupPolicy:
    local: false
  tags:
  - annotations:
      [...]
  - annotations:
      description: |-
        Build and run Node.js applications on UBI. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-nodejs-container/blob/master/12/README.md.

        WARNING: By selecting this tag, your application will automatically update to use the latest version of Node.js available on OpenShift, including major version updates.
      iconClass: icon-nodejs
      openshift.io/display-name: Node.js (Latest)
      openshift.io/provider-display-name: Red Hat, Inc.
      sampleRepo: https://github.com/sclorg/nodejs-ex.git
      supports: nodejs
      tags: builder,nodejs
    from:
      kind: ImageStreamTag
      name: 12-ubi8
    importPolicy: {}
    name: latest
    referencePolicy:
      type: Local
status:
  dockerImageRepository: image-registry.openshift-image-registry.svc:5000/openshift/nodejs
  tags:
  [...]
  - items:
    - created: "2020-09-23T06:14:48Z"
      dockerImageReference: registry.redhat.io/ubi8/nodejs-12@sha256:1175006938852dff3ac21cdcab0939b65e0499a6ba0c4d1586869b3f31465f3f
      generation: 2
      image: sha256:1175006938852dff3ac21cdcab0939b65e0499a6ba0c4d1586869b3f31465f3f
    tag: latest

Creating component on 4.5 cluster and image description
$ odo project create blah
 ✓  Project 'blah' is ready for use
 ✓  New project created and now using project: blah

$ odo create --s2i nodejs --project blah
Validation
 ✓  Validating component [724ms]

Please use `odo push` command to create the component with source deployed

$ oc -n openshift get is nodejs -o yaml
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/display-name: Node.js
    openshift.io/image.dockerRepositoryCheck: "2020-09-23T07:55:01Z"
    samples.operator.openshift.io/version: 4.5.0-0.ci-2020-09-22-154858
  labels:
    samples.operator.openshift.io/managed: "true"
  name: nodejs
  namespace: openshift
  selfLink: /apis/image.openshift.io/v1/namespaces/openshift/imagestreams/nodejs
spec:
  lookupPolicy:
    local: false
  tags:
  - annotations:
      [...]
  - annotations:
      description: |-
        Build and run Node.js applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-nodejs-container/blob/master/12/README.md.

        WARNING: By selecting this tag, your application will automatically update to use the latest version of Node.js available on OpenShift, including major version updates.
      iconClass: icon-nodejs
      openshift.io/display-name: Node.js (Latest)
      openshift.io/provider-display-name: Red Hat, Inc.
      sampleRepo: https://github.com/sclorg/nodejs-ex.git
      supports: nodejs
      tags: builder,nodejs
    from:
      kind: ImageStreamTag
      name: "12"
    importPolicy: {}
    name: latest
    referencePolicy:
      type: Local
status:
  dockerImageRepository: image-registry.openshift-image-registry.svc:5000/openshift/nodejs
  tags:
  - items:
    [...]
  - items:
    - created: "2020-09-23T07:55:01Z"
      dockerImageReference: registry.redhat.io/rhscl/nodejs-12-rhel7@sha256:2bfd71b41a1591a1ea19513cb92e5c1f43d235bbdf15b114b8fddd5521e99482
      generation: 2
      image: sha256:2bfd71b41a1591a1ea19513cb92e5c1f43d235bbdf15b114b8fddd5521e99482
    tag: latest

@kadel @girishramnani Should we add the latest image pointed in 4.6 as supported? Please let me know your suggestions.

@kadel @girishramnani Should we add the latest image pointed in 4.6 as supported? Please let me know your suggestions.

yes, we need to test odo with registry.redhat.io/ubi8/nodejs-12 and if everything works properly add it to supported images.

I have tested odo with some of the command on nodejs image registry.redhat.io/ubi8/nodejs-12 and everything looks good to me so far. I will be sending PR to add this image as supported.
Commands verified on image registry.redhat.io/ubi8/nodejs-12

devfile commands
$ oc version
   Client Version: 4.4.0-0.ci-2020-02-26-215902
   Server Version: 4.6.0-0.nightly-2020-10-01-041253
   Kubernetes Version: v1.19.0+beb741b$ odo create nodejs
   Validation
   ✓  Checking devfile existence [458972ns]
   ✓  Creating a devfile component from registry: DefaultDevfileRegistry [373044ns]
   ✓  Validating devfile component [229417ns]
   Please use `odo push` command to create the component with source deployed$ odo url create --host console-openshift-console.apps.rhamilto46.devcluster.openshift.com --port 3000 --ingress
   ✓  URL nodejs-3000 created for component: nodejs
   To apply the URL configuration changes, please use `odo push`$ odo push
   Validation
   ✓  Validating the devfile [87065ns]
   Creating Kubernetes resources for component nodejs
   ✓  Waiting for component to start [37s]
   Applying URL changes
   ✓  URL http-3000: http://http-3000-nodejs-blah.apps.rhamilto46.devcluster.openshift.com/ created
   ✓  URL nodejs-3000: http://nodejs-3000.console-openshift-console.apps.rhamilto46.devcluster.openshift.com/ created
   Syncing to component nodejs
   ✓  Checking files for pushing [1ms]
   ✓  Syncing files to the component [6s]
   Executing devfile commands for component nodejs
   ✓  Executing install command "npm install" [6s]
   ✓  Executing run command "npm start" [3s]
   Pushing devfile component nodejs
   ✓  Changes successfully pushed to component$ odo watch
   Waiting for something to change in /Users/pkumari/go/src/github.com/openshift/odo/testdevfile/project
   File /Users/pkumari/go/src/github.com/openshift/odo/testdevfile/project/server.js changed
   Pushing files...
   Validation
   ✓  Validating the devfile [7ms]
   Creating Kubernetes resources for component nodejs
   ✓  Waiting for component to start [288ms]
   Applying URL changes
   ✓  URLs are synced with the cluster, no changes are required.
   Syncing to component nodejs
   ✓  Checking file changes for pushing [3ms]
   ✓  Syncing files to the component [7s]
   Executing devfile commands for component nodejs
   ✓  Executing install command "npm install" [4s]
   ✓  Executing run command "npm start" [3s]
   Waiting for something to change in /Users/pkumari/go/src/github.com/openshift/odo/testdevfile/project$ odo push --debug
   Validation
   ✓  Validating the devfile [58561ns]
   [...] ✓  Changes successfully pushed to component$ odo log --debug
   time="2020-10-01T19:51:23Z" level=info msg="create process:devrun"
   time="2020-10-01T19:51:23Z" level=info msg="create process:debugrun" [...]ODO_COMMAND_DEBUG is npm run debug
   Changing directory to /project$ odo describe
   Component Name: nodejs
   Type: nodejs
   Environment Variables:
   · PROJECTS_ROOT=/project
   · PROJECT_SOURCE=/project
   · DEBUG_PORT=5858 [...]

Devfile registry and s2i commands also works fine.

@prietyc123 what you did is not correct and you are not testing registry.redhat.io/ubi8/nodejs-12 ❗ ❗ ❗
You are using Devfile not s2i.
You need to use --s2i.

You need to use --s2i.

@kadel Though I have not mentioned all the commands that I verifies. Yes, For s2i I have used --s2i on 4.6 cluster where the default latest image is ubi8/nodejs-12 and it works fine with --s2i too. Here in comment I have only mentioned the devfile commands however I verified with lots of scenarios.

Would it add value if there was a script that would test the various scenarios automatically for a provided image name? For example,
$ script.sh --s2i nodejs some.registry.io/nodejs:sometag
$ script.sh --s2i java some.registry.io/some-java-framework:sometag

$ script.sh --devfile nodejs some.registry.io/nodejs:sometag
$ script.sh --devfile java some.registry.io/some-java-framework:sometag

I'm asking this because if I understand correctly, we are soon going to focus on growing odo horizontally (add support for more frameworks) than just growing it vertically (adding more features.)
WDYT @kadel ?

@kadel Though I have not mentioned all the commands that I verifies. Yes, For s2i I have used --s2i on 4.6 cluster where the default latest image is ubi8/nodejs-12 and it works fine with --s2i too

and debugiung worked with ubi8/nodejs-12 and default init image ?

@prietyc123

Here in comment I have only mentioned the devfile commands however I verified with lots of scenarios.

The problem that those commands are completely unrelated to this issue. When you are using this sequence of commands it doesn't use s2i builder images. So you can't say that you tested ubi8/nodejs-12 with those commands.
@dharmit

Would it add value if there was a script that would test the various scenarios automatically for a provided image name? For example,
$ script.sh --s2i nodejs some.registry.io/nodejs:sometag
$ script.sh --s2i java some.registry.io/some-java-framework:sometag

$ script.sh --devfile nodejs some.registry.io/nodejs:sometag
$ script.sh --devfile java some.registry.io/some-java-framework:sometag

I'm asking this because if I understand correctly, we are soon going to focus on growing odo horizontally (add support for more frameworks) than just growing it vertically (adding more features.)

I see the same confusing as with Priti's comment. Devfile components doesn't use s2i builder images.
 script.sh --devfile nodejs some.registry.io/nodejs:sometag this command wouldn't make any sense.

I see the same confusing as with Priti's comment. Devfile components doesn't use s2i builder images.
 script.sh --devfile nodejs some.registry.io/nodejs:sometag this command wouldn't make any sense.

Ah. This is likely a better question for the devfile-registry repo? My intention here is for us to be able to have a framework that would make adding support for new component types easier and more automated.

AFAIU I have added the image ubi8/nodejs-12as supported via PR https://github.com/openshift/odo/pull/4070 and openshift/odo-init-image#72 . However not sure why it fails on odo-init-image travis CI claiming the image as unsupported.
Failure : https://travis-ci.com/github/openshift/odo-init-image/jobs/394059776#L18974-L18983 explains the image as unsupported.
@girishramnani @kadel I am bit lost at this point. Could you please let me know what am I missing.

Tests for ubi8/nodejs-12 is failing on prow due to context content verification within dc environment variable ODO_S2I_SRC_BIN_PATH . However I tried debugging the failure and observed those contents within ODO_S2I_DEPLOYMENT_DIR. This is really a weird behavior but reason could be some change in the image which has not been handled in odo properly. @kadel is looking into it further. Till then we are blocked on this issue.
Detailed failure log: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_odo/4070/pull-ci-openshift-odo-master-v4.5-integration-e2e/1323912242453811200#1:build-log.txt%3A1405


world-parameters

Would it be possible to add support for world-parameters?
Thanks

Any chances of having this published to the public npm registry?

meanwhile this hack worked for me:
`  let runOptions = [];
runOptions.push('--world-parameters');
runOptions.push(JSON.stringify({
"test": "value"
}));
return gulp.src(globs, {
read: false
})
.pipe(cucumber({
'format': format,
'runOptions': runOptions
}))...`


Android: Black screen on back press

Bug description:
There is an edge case where while in fullscreen mode on android, the parent webview is removed then the black screen (video background) is still visible.
In my case i was using react-navigation and had a stack view that opens a webview, when back is pressed the stack closes and removes the webview before the video is hidden, see screenshot.
its worth noting that it is intermittent so might be related to react-navigation but the fact remains that if this case happens you get stuck with a black screen.
To Reproduce:
remove the webview while fullscreen is active
Expected behavior:
no black screen remains and app functions as expected
Screenshots/Videos:

https://user-images.githubusercontent.com/207225/78639067-9ef8c400-78ad-11ea-81ff-b0634cf7d861.gif
Environment:

OS: Android
OS version: 6+
react-native version:0.61.3
react-native-webview version:9.1.0

Same issue
I use react-navigation 4.3.7
Even without video, just html page

this pr fixes the issue,
https://github.com/react-native-community/react-native-webview/pull/1298

+1


Question: How can I create a stored view of sorted documents? (CDC - Change Data Capture)

How can I create a view of sorted documents?
Use Case:
I have key based pagination, and I'm sorting docs based on an attribute.
For the sorting to work correctly I have to Inner Join the collection, one without sorting and one with sorting applied.
for c in collection
    filter c._key >= '7613128'
    for d in collection
        filter c._key == d._key
        sort d.attribute DESC
        limit 20
return d

It has impacts on performance and takes more effort to code.
Is there a feature available like Views in MySQL?

At the moment there is no materialized view functionality in ArangoDB (nor does it exist in MySQL).
That means the query will be re-evaluated every time it is executed.
With the default indexes the query performance will not be very good, given the full collection scan on the outer collection. It may be sped up by using indexes though.
Can you show the query execution plan for the exact query you are running? Plus which indexes exist on the collections used in the query. Thanks!

Hi,
any news on this? or can we close it?

@jsteemann you're right, maybe something like this: MATERIALIZED VIEWS WITH MYSQL
http://www.fromdual.com/mysql-materialized-views
Open sourcing Databus: LinkedIn's low latency change data capture system
https://engineering.linkedin.com/data-replication/open-sourcing-databus-linkedins-low-latency-change-data-capture-system
@dothebart please label it as a feature request

Change Data Capture (CDC) tracks data changes (usually close to realtime). In MySQL, the easiest and probably most efficient way to track data changes is to use binary logs.
https://www.percona.com/blog/2016/09/13/mysql-cdc-streaming-binary-logs-and-asynchronous-triggers/

Flexviews is a unique open source project. Flexviews is a materialized views implementation for MySQL. It includes a simple API that is used to create materialized views and to refresh them. The advantage of using Flexviews is that the materialized views are incrementally refreshed, that is, the views are updated efficiently by using special logs which record the changes to database tables. Flexviews includes tools which create and maintain these logs. The views created by Flexviews include support for JOINs and for all major aggregation functions.
https://github.com/greenlion/swanhart-tools/tree/master/flexviews

A Materialized View (MV) replaces a SQL multi-table-view (or query) with a new table that holds all data permutations
MV's are used to improve performance, and are preferable to replication where problem is due to an inefficient query plan
http://www.materialized.info/

Are there any updates on this?  Materialized views derived from an AQL would be an incredibly powerful feature both for performance and reduced query complexity.

If the Java driver's support for WAL / CDC gets properly integrated, I could create materialized view support using that.


[CONTROLLER]optimize log

This PR is for:

Server

Fixes Optimizing Log Content
Steps to reproduce the bug

debug vtap interfaces without vtap info

Changes to fix the bug

add error log if vtap not found when generate vtap interfaces data
add mysql error info in domain update api
change recorder cache refresh interval to 1h
add cloud state info log in recorder

Affected branches

main

Checklist

[ ] Added unit test to verify the fix.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.zhengya seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Add jpverkamp/advent-of-code

Repository URL
https://github.com/jpverkamp/advent-of-code
Crate name
aoc2024
Toolchain version
No response

You need to remove the trailing slash at the end of the URL: https://github.com/jpverkamp/advent-of-code

I suppose that is what the template shows. Any idea if editing it will trigger a re-run or if I need to create a new issue? (I can wait a bit to see if it works).

Looks like you resubmitted yours, I'll do the same.
Thanks!


polish gregorian - default and the only one calendar translated from en

Hi. I want to say "Thank you" for this library, and from my side, I want to add Polish locales. If you are open to that, merge to your base code :) Thanks!

Thank you.
I appreciate your collaboration.


error occur in JukeboxItem.swift, line 116

I meet a strange error just happened a special url.
Here are three mp3 url waitting for play.
[
"http://dingdong.sosys.cn:8080/uploads/course/107/0/audio.mp3",
"http://dingdong.sosys.cn:8080/uploads/course/107/1/audio.mp3",
"http://dingdong.sosys.cn:8080/uploads/course/107/2/audio.mp3"]
error only occurs when jukebox loaditem which has the second urlString("http://dingdong.sosys.cn:8080/uploads/course/107/1/audio.mp3").
Really ,I have no idea about this, could anyone who has some idea can guide me a way to figure this out?

new version released, check if the crash is still reproducing


Fixing Hydrology for #421

This issue is for a PR to align the hydrology model with the changes requested in #421 and being implemented in #441

Closed by merge of #445


fix: update copyright to Dragonfly Authors

Signed-off-by: lowzj<EMAIL_ADDRESS>
Ⅰ. Describe what this PR did
Update copyright from Alibaba Group to Dragonfly Authors.
Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

LGTM, but wait for the CI.

Codecov Report

Merging #368 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #368   +/-   ##
=======================================
  Coverage   58.73%   58.73%
=======================================
  Files          36       36
  Lines        1854     1854
=======================================
  Hits         1089     1089
  Misses        723      723
  Partials       42       42

Impacted Files
Coverage Δ

dfdaemon/util/netutil.go
13.33% <ø> (ø)
:arrow_up:

dfdaemon/muxconf/muxconf.go
100% <ø> (ø)
:arrow_up:

cmd/dfget/app/root.go
61.87% <ø> (ø)
:arrow_up:

dfdaemon/util/urlutil.go
100% <ø> (ø)
:arrow_up:

cmd/dfget/app/gen_doc.go
59.09% <ø> (ø)
:arrow_up:

cmd/dfget/app/version.go
33.33% <ø> (ø)
:arrow_up:

dfdaemon/global/global.go
92.3% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bb09fd6...7d8dc67. Read the comment docs.


Re-create chart when symbol change

OK to merge ~

Changes Unknown when pulling 936fed1a2a0efb5febe78a93e9adbbfbac7604ba on qingweibinary:qingwei/update_fn_dynamically into ** on binary-com:master**.

OK to merge


Add more most specific host benchmarks

Please provide a description of this PR:
Related to #41610
Adding more benchmarks for MostSpecificHostMatch:

matching and non-matching cases
mixed specific and wildcard cases
multiple wildcard match cases

Hi @sschepens. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@howardjohn adding more benchmarks related to the other PR

/ok-to-test

@sschepens: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
e592e82291477b27782ff2805abc22d69d6efaf0
link
true
/test release-notes

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@sschepens: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
e592e82291477b27782ff2805abc22d69d6efaf0
link
true
/test release-notes

lint_istio
e592e82291477b27782ff2805abc22d69d6efaf0
link
true
/test lint

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@sschepens: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
e592e82291477b27782ff2805abc22d69d6efaf0
link
true
/test release-notes

lint_istio
e592e82291477b27782ff2805abc22d69d6efaf0
link
true
/test lint

gencheck_istio
e592e82291477b27782ff2805abc22d69d6efaf0
link
true
/test gencheck

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@sschepens: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

lint_istio
e592e82291477b27782ff2805abc22d69d6efaf0
link
true
/test lint

gencheck_istio
e592e82291477b27782ff2805abc22d69d6efaf0
link
true
/test gencheck

release-notes_istio
5ac95135784173f7c8f7fd341c5f496b5373f2bc
link
true
/test release-notes

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest


Merge diff from cloudant fork

This PR brings in three commits made directly to the cloudant fork of fabric after the last significant chunk of merge work.

+1 after proplist fix


Namespace for custom objects not included in serialised config

This is not a contribution.
System information.

TensorFlow version: 2.6.0

Describe the problem.
Custom object serialisation is inconsistent when using tf.keras.utils.custom_object_scope as compared to tf.keras.utils.register_keras_serializable. custom_object_scope is not able to use namespaces for different custom objects, which risks name collisions.
Describe the current behavior.
import tensorflow as tf

class Serialisable:
    def __init__(self, arg1, arg2):
        self.arg1 = arg1
        self.arg2 = arg2
    def get_config(self):
        return {'arg1': self.arg1, 'arg2': self.arg2}

custom_objects = {
    'custom_package>Serialisable': Serialisable,
}
serialisable = Serialisable('arg1', 2)

with tf.keras.utils.custom_object_scope(custom_objects):
  serialised = tf.keras.utils.serialize_keras_object(serialisable)

# note that the class_name field does not contain the namespace
serialised
>>> {'class_name': 'Serialisable', 'config': {'arg1': 'arg1', 'arg2': 2}}

with tf.keras.utils.custom_object_scope(custom_objects):
    deserialised = tf.keras.utils.deserialize_keras_object(serialised) # throws ValueError

Describe the expected behavior.
# compare to what happens with tf.keras.utils.register_keras_serializable

tf.keras.utils.register_keras_serializable(package='custom_package')(Serialisable)

# class_name now contains the namespace information
serialised_after_registration = tf.keras.utils.serialize_keras_object(serialisable)
>>> {'class_name': 'custom_package>Serialisable', 'config': {'arg1': 'arg1', 'arg2': 2}}

# and deserialisation works as expected
with tf.keras.utils.custom_object_scope(custom_objects):
    deserialised_after_registration = tf.keras.utils.deserialize_keras_object(serialised)

So what I would like is that the names provided in the custom_object_scope keys be used when serialising, rather than just the object name.
Source code / logs.
I believe that this discrepancy happens because the custom object context has two objects tracking the registered objects and names. These map from name->object and from object->name, however they are not kept in sync. register_keras_serializable updates both, however custom_object_scope only updates one of the mappings. This means that there is a miss when looking up the name in the other mapping during serialisation. Either custom object scope should be changed to update both mappings, or I think it would likely be less error prone to only maintain one mapping, and derive both views from one source of truth to avoid any future divergences.
Aside: get_registered_name is called twice during serialisation, and the second looks redundant.

@peteboothroyd Sorry for the late response!
Could you please check this link to know more info to pass the custom objects and let us know if it helps ?
Thank you!

@sushreebarsa thanks, I don't think that addresses the issue I raised though. I've just checked and this still reproduces on 2.11

Hello, Thank you for reporting an issue.
We're currently in the process of migrating the new Keras 3 code base from keras-team/keras-core to keras-team/keras.
Consequently, This issue may not be relevant to the Keras 3 code base. After the migration is successfully completed, feel free to reopen this issue at keras-team/keras if you believe it remains relevant to the Keras 3 code base.
If instead this issue is a bug or security issue in legacy tf.keras, you can instead report a new issue at keras-team/tf-keras, which hosts the TensorFlow-only, legacy version of Keras.
To know more about Keras 3, please take a look at https://keras.io/keras_core/announcement/. Thank you!


namespace error in spatial/explog-quaternion.hpp

https://github.com/stack-of-tasks/pinocchio/blob/11e195bb5ae57bb0d3010642608cf836620bf14e/src/spatial/explog-quaternion.hpp#L191
This line cause compilation error when I include this file :
/opt/openrobots/include/pinocchio/spatial/explog-quaternion.hpp: In function ‘void pinocchio::quaternion::Jlog3(const Eigen::QuaternionBase<Derived>&, const Eigen::MatrixBase<U>&)’:
/opt/openrobots/include/pinocchio/spatial/explog-quaternion.hpp:183:7: error: ‘Jlog3’ is not a member of ‘pinocchio’
       pinocchio::Jlog3(t,w,Jlog);
       ^
/opt/openrobots/include/pinocchio/spatial/explog-quaternion.hpp:183:7: note: suggested alternative:
/opt/openrobots/include/pinocchio/spatial/explog-quaternion.hpp:175:10: note:   ‘pinocchio::quaternion::Jlog3’
     void Jlog3(const Eigen::QuaternionBase<QuaternionLike> & quat,

It is solved by either changing it to pinocchio::quaternion::Jlog3 or just Jlog3.

I think the issue is coming from your inclusions.
Can you provide a minimal example showing your issue?

You should include `#include "pinocchio/spatial/explog.hpp" to get access to the signatures of exp and log functions with Quaternion inputs.

You should include `#include "pinocchio/spatial/explog.hpp" to get access to the signatures of exp and log functions with Quaternion inputs.

I see pinocchio/spatial/explog.hpp includes pinocchio/spatial/explog-quaternion.hpp at the end. So the user should never include pinocchio/spatial/explog-quaternion.hpp directly, they should only care about pinocchio/spatial/explog.hpp.
I think this is not intuitive and the error made by @pFernbach is natural. If explog-quaternion.hpp is not meant to be employed by the user, I suggest to take one of the following actions:

Removing explog-quaternion.hpp and putting all its code into explog.hpp
Renaming explog-quaternion.hpp to explog-quaternion.hxx and possibly putting the forward declarations in explog.hpp
Performing a more thorough reorganization, by doing the same as above but also moving the code in explog.hpp to a file called explog.hxx

I favor this last hypothesis. @jcarpent do you agree? This is minor, but it is quick to do and it will improve the organization, preventing errors like this one.

@gabrielebndn I do agree on the third option which allows a perfect clean of this part of the coder.
On my side, I have already performed a deeper splitting with log.{hpp,hxx} and exp.{hpp,hxx}.
This is on a branch which currently under development.
Before submitting this PR, I have a quick fix in mind which consists of adding these lines in the explog-quaternion.hpp file:
#ifndef __spatial_explog_hpp__
  #include "pinocchio/spatial/explog.hpp"
#endif

@gabrielebndn Do you agree?

@gabrielebndn If you are interested to implement the complete splitting, you can go ahead if you want.

Before submitting this PR, I have a quick fix in mind which consists of adding these lines in the explog-quaternion.hpp file:
#ifndef __spatial_explog_hpp__
  #include "pinocchio/spatial/explog.hpp"
#endif

@gabrielebndn Do you agree?

I think the #ifndef is not strictly needed, the include guard inside explog.hpp should be enough.
Also, I don't think the fix itself is needed. The problem at hand is solved by including the correct header, that is explog.hpp instead of explog-quaternion.hpp
I am not sure the third proposal is clear. I do not want a complete split. What I propose is:
explog.hpp:
<forward explog declarations>

<forward explog quaternion declarations>

#include pinocchio/spatial/explog.hxx

#include pinocchio/spatial/explog-quaternion.hxx

So that the user could still do #include <pinocchio/spatial/explog.hpp> to have everything.
Also, I do not see the reason of splitting log and exp in two different headers.

Also, I do not see the reason of splitting log and exp in two different headers.

To simplify maintenance and to not have hundred of line of codes in the same file.

Solved by #863


UNION ALL with ORDER BY results are inconsistent

Describe the bug
The results for a UNION ALL that specifies an ORDER BY is sometimes sorted incorrectly.

Note
I bisected to the first bad commit being 5fc91cc8fbb56b6d2a32e66f8b327a871f7d15ac via #5661

To Reproduce
Using datafusion-cli (version 22.0) execute;
WITH
  m0(time, tag0, f64) AS (VALUES<PHONE_NUMBER>000000000, 'val00', 10.1),<PHONE_NUMBER>000000000, 'val00', 21.2),<PHONE_NUMBER>000000000, 'val00', 11.2),<PHONE_NUMBER>000000000, 'val00', 19.2),<PHONE_NUMBER>000000000, 'val01', 11.3),<PHONE_NUMBER>000000000, 'val02', 10.4),<PHONE_NUMBER>000000000, 'val00', 18.9)),

  m1(time, tag0, f64) AS (VALUES<PHONE_NUMBER>000000000, 'val00', 100.5),<PHONE_NUMBER>000000000, 'val00', 200.6),<PHONE_NUMBER>000000000, 'val01', 101.7)),

  t AS (
    SELECT 'm0' as "iox::measurement", tag0, 0::timestamp as time, COUNT(f64), SUM(f64), stddev(f64) FROM m0 GROUP BY 1, 2, 3),

  u AS (
    SELECT 'm1' as "iox::measurement", tag0, 0::timestamp as time, COUNT(f64), SUM(f64), stddev(f64) FROM m1 GROUP BY 1, 2, 3)
SELECT * FROM t
UNION ALL
SELECT * FROM u
ORDER BY 1, 2, 3;

Expected behavior
Output should be sorted on the first 3 columns:
+------------------+-------+---------------------+------------+----------+-------------------+
| iox::measurement | tag0  | time                | COUNT(f64) | SUM(f64) | STDDEV(f64)       |
+------------------+-------+---------------------+------------+----------+-------------------+
| m0               | val00 | 1970-01-01T00:00:00 | 5          | 80.6     | 5.085961069453836 |
| m0               | val01 | 1970-01-01T00:00:00 | 1          | 11.3     |                   |
| m0               | val02 | 1970-01-01T00:00:00 | 1          | 10.4     |                   |
| m1               | val00 | 1970-01-01T00:00:00 | 2          | 301.1    | 70.7813887967734  |
| m1               | val01 | 1970-01-01T00:00:00 | 1          | 101.7    |                   |
+------------------+-------+---------------------+------------+----------+-------------------+

Actual behavior
Output is sometimes inconsistent, noting that the sort order is incorrect:
+------------------+-------+---------------------+------------+----------+-------------------+
| iox::measurement | tag0  | time                | COUNT(f64) | SUM(f64) | STDDEV(f64)       |
+------------------+-------+---------------------+------------+----------+-------------------+
| m0               | val00 | 1970-01-01T00:00:00 | 5          | 80.6     | 5.085961069453836 |
| m0               | val01 | 1970-01-01T00:00:00 | 1          | 11.3     |                   |
| m1               | val00 | 1970-01-01T00:00:00 | 2          | 301.1    | 70.7813887967734  |
| m0               | val02 | 1970-01-01T00:00:00 | 1          | 10.4     |                   |
| m1               | val01 | 1970-01-01T00:00:00 | 1          | 101.7    |                   |
+------------------+-------+---------------------+------------+----------+-------------------+

Additional context
DataFusion versions 18, 19, 20 and 21 work consistently after numerous executions of the same query
The following also works on 22
WITH
  m0(time, tag0, f64) AS (VALUES<PHONE_NUMBER>000000000, 'val00', 10.1),<PHONE_NUMBER>000000000, 'val00', 21.2),<PHONE_NUMBER>000000000, 'val00', 11.2),<PHONE_NUMBER>000000000, 'val00', 19.2),<PHONE_NUMBER>000000000, 'val01', 11.3),<PHONE_NUMBER>000000000, 'val02', 10.4),<PHONE_NUMBER>000000000, 'val00', 18.9)),
  m1(time, tag0, f64) AS (VALUES<PHONE_NUMBER>000000000, 'val00', 100.5),<PHONE_NUMBER>000000000, 'val00', 200.6),<PHONE_NUMBER>000000000, 'val01', 101.7)),
  t AS (SELECT 'm0' as "iox::measurement", tag0, 0::timestamp as time, COUNT(f64), SUM(f64), stddev(f64) FROM m0 GROUP BY 1, 2, 3),
  u AS (SELECT 'm1' as "iox::measurement", tag0, 0::timestamp as time, COUNT(f64), SUM(f64), stddev(f64) FROM m1 GROUP BY 1, 2, 3),
-- add another CTE
  v AS (
  SELECT * FROM t
  UNION ALL
  SELECT * FROM u)
SELECT * FROM v
ORDER BY 1, 2, 3;

I'm working on this.

I couldn't get the datafusion-cli to reproduce, but I think it largely depends on the plan and the target partitions / CPU count. So here's a code test that reproduces the issue on my system as well:
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_issue5970_mini() -> Result<()> {
    let config = SessionConfig::new().with_target_partitions(2).with_repartition_sorts(true);
    let ctx = SessionContext::with_config(config);
    let sql = "
WITH
    m0(t) AS (
        VALUES (0), (1), (2)),

    m1(t) AS (
        VALUES (0), (1)),

    u AS (
        SELECT 0 as m, t FROM m0 GROUP BY 1, 2),

    v AS (
        SELECT 1 as m, t FROM m1 GROUP BY 1, 2)
SELECT * FROM u
UNION ALL
SELECT * FROM v
ORDER BY 1, 2;
    ";

    // check phys. plan
    let dataframe = ctx.sql(sql).await.unwrap();
    let plan = dataframe.into_optimized_plan().unwrap();
    let plan = ctx.state().create_physical_plan(&plan).await.unwrap();
    let expected = vec![
        "SortPreservingMergeExec: [m@0 ASC NULLS LAST,t@1 ASC NULLS LAST]",
        "  UnionExec",
        "    SortExec: expr=[m@0 ASC NULLS LAST,t@1 ASC NULLS LAST]",
        "      ProjectionExec: expr=[Int64(0)@0 as m, t@1 as t]",
        "        AggregateExec: mode=FinalPartitioned, gby=[Int64(0)@0 as Int64(0), t@1 as t], aggr=[]",
        "          CoalesceBatchesExec: target_batch_size=8192",
        "            RepartitionExec: partitioning=Hash([Column { name: \"Int64(0)\", index: 0 }, Column { name: \"t\", index: 1 }], 2), input_partitions=2",
        "              AggregateExec: mode=Partial, gby=[0 as Int64(0), t@0 as t], aggr=[]",
        "                RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1",
        "                  ProjectionExec: expr=[column1@0 as t]",
        "                    ValuesExec",
        "    SortExec: expr=[m@0 ASC NULLS LAST,t@1 ASC NULLS LAST]",
        "      ProjectionExec: expr=[Int64(1)@0 as m, t@1 as t]",
        "        AggregateExec: mode=FinalPartitioned, gby=[Int64(1)@0 as Int64(1), t@1 as t], aggr=[]",
        "          CoalesceBatchesExec: target_batch_size=8192",
        "            RepartitionExec: partitioning=Hash([Column { name: \"Int64(1)\", index: 0 }, Column { name: \"t\", index: 1 }], 2), input_partitions=2",
        "              AggregateExec: mode=Partial, gby=[1 as Int64(1), t@0 as t], aggr=[]",
        "                RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1",
        "                  ProjectionExec: expr=[column1@0 as t]",
        "                    ValuesExec",
    ];
    let formatted = displayable(plan.as_ref()).indent().to_string();
    let actual: Vec<&str> = formatted.trim().lines().collect();
    assert_eq!(
        expected, actual,
        "\n\nexpected:\n\n{expected:#?}\nactual:\n\n{actual:#?}\n\n"
    );

    // sometimes it "just works"
    for i in 0..10 {
        println!("run: {i}");
        let actual = execute_to_batches(&ctx, sql).await;
        let expected = vec![
            "+---+---+",
            "| m | t |",
            "+---+---+",
            "| 0 | 0 |",
            "| 0 | 1 |",
            "| 0 | 2 |",
            "| 1 | 0 |",
            "| 1 | 1 |",
            "+---+---+",
        ];
        assert_batches_eq!(expected, &actual);
    }
    Ok(())
}

results in:
expected:

[
    "+---+---+",
    "| m | t |",
    "+---+---+",
    "| 0 | 0 |",
    "| 0 | 1 |",
    "| 0 | 2 |",
    "| 1 | 0 |",
    "| 1 | 1 |",
    "+---+---+",
]
actual:

[
    "+---+---+",
    "| m | t |",
    "+---+---+",
    "| 0 | 0 |",
    "| 0 | 2 |",
    "| 1 | 0 |",
    "| 0 | 1 |",
    "| 1 | 1 |",
    "+---+---+",
]

To me, the phys. plan looks fine. I guess either SortExec or SortPreservingMergeExec are buggy. The issue was bisected to #5661 which makes use of SortPreservingMergeExec so my current hypothesis is that this is the buggy one.

So the plan is seemingly right but actual wrong:
The two SortExecs in the above plan have two partitions each (= 4 in total). They are hash-partitioned because this is what you get from the group-by operation.
The UnionExec thinks it is "partition aware" (whatever that means):
https://github.com/apache/arrow-datafusion/blob/fcd8b899e2a62f798413c536f47078289ece9d05/datafusion/core/src/physical_plan/union.rs#L157-L168
As a consequence, it only emits 2 output partitions (instead of 4) and concats the input data:
https://github.com/apache/arrow-datafusion/blob/fcd8b899e2a62f798413c536f47078289ece9d05/datafusion/core/src/physical_plan/union.rs#L276-L292
Hence, it no longer preserve sorting. However the EnforceSorting optimizer pass thinks it can push SortExec safely through UnionExec:
[2023-04-13T15:26:06Z TRACE datafusion::physical_plan::planner] Optimized physical plan by EnforceDistribution:
    SortPreservingMergeExec: [m@0 ASC NULLS LAST,t@1 ASC NULLS LAST]
      SortExec: expr=[m@0 ASC NULLS LAST,t@1 ASC NULLS LAST]
        UnionExec
          ProjectionExec: expr=[Int64(0)@0 as m, t@1 as t]
            AggregateExec: mode=FinalPartitioned, gby=[Int64(0)@0 as Int64(0), t@1 as t], aggr=[]
              RepartitionExec: partitioning=Hash([Column { name: "Int64(0)", index: 0 }, Column { name: "t", index: 1 }], 2), input_partitions=2
                AggregateExec: mode=Partial, gby=[0 as Int64(0), t@0 as t], aggr=[]
                  RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
                    ProjectionExec: expr=[column1@0 as t]
                      ValuesExec
          ProjectionExec: expr=[Int64(1)@0 as m, t@1 as t]
            AggregateExec: mode=FinalPartitioned, gby=[Int64(1)@0 as Int64(1), t@1 as t], aggr=[]
              RepartitionExec: partitioning=Hash([Column { name: "Int64(1)", index: 0 }, Column { name: "t", index: 1 }], 2), input_partitions=2
                AggregateExec: mode=Partial, gby=[1 as Int64(1), t@0 as t], aggr=[]
                  RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
                    ProjectionExec: expr=[column1@0 as t]
                      ValuesExec

[2023-04-13T15:26:06Z TRACE datafusion::physical_plan::planner] Optimized physical plan by EnforceSorting:
    SortPreservingMergeExec: [m@0 ASC NULLS LAST,t@1 ASC NULLS LAST]
      UnionExec
        SortExec: expr=[m@0 ASC NULLS LAST,t@1 ASC NULLS LAST]
          ProjectionExec: expr=[Int64(0)@0 as m, t@1 as t]
            AggregateExec: mode=FinalPartitioned, gby=[Int64(0)@0 as Int64(0), t@1 as t], aggr=[]
              RepartitionExec: partitioning=Hash([Column { name: "Int64(0)", index: 0 }, Column { name: "t", index: 1 }], 2), input_partitions=2
                AggregateExec: mode=Partial, gby=[0 as Int64(0), t@0 as t], aggr=[]
                  RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
                    ProjectionExec: expr=[column1@0 as t]
                      ValuesExec
        SortExec: expr=[m@0 ASC NULLS LAST,t@1 ASC NULLS LAST]
          ProjectionExec: expr=[Int64(1)@0 as m, t@1 as t]
            AggregateExec: mode=FinalPartitioned, gby=[Int64(1)@0 as Int64(1), t@1 as t], aggr=[]
              RepartitionExec: partitioning=Hash([Column { name: "Int64(1)", index: 0 }, Column { name: "t", index: 1 }], 2), input_partitions=2
                AggregateExec: mode=Partial, gby=[1 as Int64(1), t@0 as t], aggr=[]
                  RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
                    ProjectionExec: expr=[column1@0 as t]
                      ValuesExec

Now we could either blame EnforceSorting for performing the wrong optimization or UnionExec for having some life on its own while seemingly being a really simple node (who would have expected that it concats input batches from different input partitions?!). I would argue that UnionExec should NEVER modify its inputs but just be a plain, simple node that forwards its inputs w/o messing up sorting (or any other property).
CC @mustafasrepo who authored #5661 (= making EnforceSorting smarter) and @mingmwang who taught UnionExec to be "partition aware" in #4043.

I would argue that UnionExec should NEVER modify its inputs but just be a plain, simple node that forwards its inputs w/o messing up sorting (or any other property).

I agree with this sentiment -- we already have RepartitionExec that concatenates batches from different streams

I would argue that UnionExec should NEVER modify its inputs but just be a plain, simple node that forwards its inputs w/o messing up sorting (or any other property).

I agree with this sentiment -- we already have RepartitionExec that concatenates batches from different streams

Does it mean that the UnionExec will return the inputs sequentially (i.e. concatenated) or will it potentially interleave the inputs, whilst maintaining the ordering?

I would argue that UnionExec should NEVER modify its inputs but just be a plain, simple node that forwards its inputs w/o messing up sorting (or any other property).

I agree with this sentiment -- we already have RepartitionExec that concatenates batches from different streams

Does it mean that the UnionExec will return the inputs sequentially (i.e. concatenated) or will it potentially interleave the inputs, whilst maintaining the ordering?

So what it SHOULD do (IMHO) and what it also does in the majority of the cases (i.e. when the partition_aware = false):
---
# Plan:
UnionExec:
  children:
    - SomeChild:
        output_partitions:
          - [batch111, batch112]
          - [batch121, batch122]
    - SomeChild:
        output_partitions:
          - [batch211, batch212]
          - [batch221, batch222]

---
# Equivalent pseudo-plan
UnionExec:
  output_partitions:
    - [batch111, batch112]
    - [batch121, batch122]
    - [batch211, batch212]
    - [batch221, batch222]

However what it actually does with partition_aware = true:
---
UnionExec:
  output_partitions:
    - CombinedRecordBatchStream:
      - [batch111, batch112]
      - [batch211, batch212]
    - CombinedRecordBatchStream:
      - [batch121, batch122]
      - [batch221, batch222]

---
# May yield (if lucky):
UnionExec:
  output_partitions:
    - [batch111, batch112, batch211, batch212]
    - [batch121, batch122, batch221, batch222]

---
# May also yield (if not so lucky):
UnionExec:
  output_partitions:
    - [batch111, batch211, batch112, batch212]
    - [batch221, batch222, batch121, batch122]

The exact logical of CombinedRecordBatchStream  can be found here:
https://github.com/apache/arrow-datafusion/blob/fcd8b899e2a62f798413c536f47078289ece9d05/datafusion/core/src/physical_plan/union.rs#L364-L408
This shuffling obviously confuses the SortPreservingMergeExec logic because it assumes that the inputs are sorted.

My suggestion is making the partition-aware optimization a configure option, but do not remove the code.
By default, we turn it off. So that the UnionExec just like a plain Union and do not keep the partition info.

My suggestion is making the partition-aware optimization a configure option, but do not remove the code.
By default, we turn it off. So that the UnionExec just like a plain Union and do not keep the partition info.

But that also means that if a user turns it on, DataFusion may produce incorrect results. I'm not sure this is a desired feature then.


backup operator: fixed incorrect backup file naming convention.

When limiting the number of backups, if the revision would move to needing another digit, the logic would never select a new entry as _ is > to 9 in ascii terms.
Note that this is backwards incompatible as now files receive a correct padding and old revisions will still be incorrectly handled.

@etcd-bot ok to test

@kapouille it looks like we missed this line in the tests
https://github.com/coreos/etcd-operator/blob/86211e9f3bc32cd97187c7ee9beb32be60264fed/test/e2e/e2eslow/backup_restore_test.go#L55


Message server should support error streams

Description
currently our endpoint function are defined to return []byte as response to requested message,
It would be better if we could return []byte, error instead to signal that there has been an error on receiving end

will implement this as part of #2507


Recoup some real-estate in the header/jumbotron of the helping interface

There's a lot of blank space in the header/jumbotron area of the helping interface that requires scrolling past to get to the specimen records. Redesign this to fill more real-estate & reduce the need for scrolling. This will have to accommodate situations when there is or is not a photo and/or signature.

Search box added so this ticket is no longer relevant.

Search box added so this ticket is no longer relevant.


chore(Java): Sonatype requires User Token

Issue #, if available: Sonatype requires User Token
As of Friday 6/21, Sonatype requires maven releases to use token-based authentication, instead of username and password authentication.
Description of changes:

Use Sonatype token-based authentication to publish to Maven
Rename WIP GitHub Workflows to release, to make it clear they are not used to release, but ideally would be used in the future.

Squash/merge commit message, if applicable:
chore(Java): Sonatype requires User Token

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

CFN changes were successfully deployed.


sh: 0: getcwd() failed: No such file or directory

When I try to use the behat plugin, I have this error message :

sh: 0: getcwd() failed: No such file or directory
fatal: Could not get current working directory: Aucun fichier ou dossier de ce type

Failed to clone remote git repository.
Exception: Could not create a working copy.

This is probably related to #881.

Hello there, just discovering PHPCI but i'm stuck exactly there on a fresh installation. Have you had any progress on #881 ?


(De-)serialization of CircuitOperations

Follow up for #3891. Part of #3634.
This PR adds the CircuitOp(De)serializer classes and updates existing behavior to handle them appropriately, including:

Addition of Op(De)serializer superclasses for operation (de-)serializers
Backwards-compatible conversion of GateOp(De)serializer fields to non-public, with public properties
Updates and deprecations for names made awkward by this change (e.g. gate_type -> internal_type)

Changes not included in this PR:

Add CircuitOperation (de-)serializers to the default QCS gateset

TODO: replace newly-deprecated fields in tests and add deprecation tests for these fields.

TODO in this PR: replace newly-deprecated fields in tests and add deprecation tests for these fields.

This is now done for tests caught by check/pytest-changed-files. May still hit some bumps if other uses of these fields are floating around the repo somewhere.

Pinging @dstrain115  and @maffoo for review.

Apologies for the review spam, everyone - had some troubles merging master into this PR. It should be resolved now - feel free to unsubscribe from this if it's still spamming you.


[New Feature] 3D Real-time Brush Synchronization; Worth it to be added?

I implemented a synchronized brush for stacked images that shows real-time segmentation of axial, coronal, and sagittal views. It goes like this:

This inherits BaseBrushTool of latest version, but my source code needs further refactoring to comply with cornerstoneTools' contributing guidelines.
If this tool(I named it as 'SyncBrush') is worth to be added as new feature or 3rd party implementation, then I can organize above and do PR under current contributing guidelines. I'm curious about your comments.
Sincerely.

Looks awesome, and it would be very useful for me. But i don't know if it should be added as a feature in this repo, since it is probably quite a niche tool.
Could you link to your code, i would love to have a look :)

Maybe I should maintain my own repository and plugin-ize this SyncBrush in the near future.

I finished plugin-izing this tool. Refer to this issue


KeGroup trait implementation

Currently KeGroup is implemented on what represents the public key of a curve, the RistrettoPoint for ristretto255, ProjectivePoint for P-256 and MontgomaryPoint for X25519. When I switched to higher-level implementation for #250 I realized that we potentially don't even need a custom trait anymore.
elliptic_curve::PublicKey requires a type that implements Curve and ProjectiveArithmetic for example.
We can't actually use those because the current ristretto255 implementation we use doesn't implement any of these traits. So instead I'm proposing the following:

Change KeGroup to not be implemented on the public key, but on a trait that is supposed to implement Curve and ProjectiveArithmetic. We introduce two associated types for a public and secret key.
We can't provide a blanket implement of KeGroup for anything that satisfies the right trait, because we lack specialization to also provide a custom implementation for RistrettoPoint, but we can provide a simple template as an example.
In the future, when curve25519-dalek implements the necessary traits, we can do a blanket implementation and basically remove all curve dependencies and associated crate features. Unless Rust stabilizes specialization first, lol.

This idea can also be implemented for voprf.
This is a proposal, I am planning to do the implementation as soon as this is approved.

Hmm, interesting! So if I am understanding correctly, the main benefit for doing this would be that the consumer of the library doesn't need to provide definitions for our current custom traits KeGroup and OprfGroup, right?
One worry that I have is that we are coupling closely to how elliptic_curve and curve25519_dalek are doing things, and if in future versions they change, this might result in this library having to become more complex. Whereas if we are OK with defining the custom trait like how we are already doing, then we can maybe keep those future changes a bit more isolated...
I also don't expect that most consumers of this library will really need to worry about providing their own definitions of KeGroup and OprfGroup, since they will likely just use the existing supported ones we have (p256, ristretto255, x25519), so the pain point seems to be minor to me. Please correct me if I am wrong on this though!

Hmm, interesting! So if I am understanding correctly, the main benefit for doing this would be that the consumer of the library doesn't need to provide definitions for our current custom traits KeGroup and OprfGroup, right?

I was actually more concerned about us having to provide implementations. This way we are up-streaming the whole implementation, they probably have better testing and hopefully less errors. Certainly there are more eyes and users on elliptic-curve then on opaque-ke.

One worry that I have is that we are coupling closely to how elliptic_curve and curve25519_dalek are doing things, and if in future versions they change, this might result in this library having to become more complex. Whereas if we are OK with defining the custom trait like how we are already doing, then we can maybe keep those future changes a bit more isolated...

I'm not really worried about these libraries changing, they still follow the same RFCs and specs we follow. If you mean their API, I believe we should still use the same wrappers we currently have, just internally, so the code really shouldn't change much.

I also don't expect that most consumers of this library will really need to worry about providing their own definitions of KeGroup and OprfGroup, since they will likely just use the existing supported ones we have (p256, ristretto255, x25519), so the pain point seems to be minor to me. Please correct me if I am wrong on this though!

I agree, I was more hoping to upstream our work then users work. But it's definitely nice to be able to just get all kinds of implementations for free: p256 and p384 and hopefully in the future p521 and decaf448.
See RustCrypto/elliptic-curves#495 for upstream hash to curve and RustCrypto/elliptic-curves#489 for p384 arithmetic implementation that will be available soon.

Another thought: being able to just depend on traits means we won't need to expose any features anymore either, I think that's a nice bonus.

I did some rework based on our discussion here:  novifinancial/voprf#49.


GPU info not displayed on Windows

Description

[x] Does this issue still occur in the master branch? (Required if issue)

It seems like the GPU part errored out, so no info is printing. Relevant log from -vv:
+ get_gpu
+ case $os in
++ wmic path Win32_VideoController get caption
C:\Users\yzgyy\scoop\apps\neofetch\current\neofetch: line 2381: /dev/fd/62: No such file or directory
+ gpu=
++ wmic path Win32_VideoController get caption

Minimal reproducible:
yzgyy@YGY-AORUS MINGW64 ~/Projects/neofetch (master)
$ while read -r line; do
while read -r line; do
> echo $line
echo $line
> done < <(wmic path Win32_VideoController get caption)
done < <(wmic path Win32_VideoController get caption)
wmic path Win32_VideoController get caption
bash: /dev/fd/62: No such file or directory
wmic path Win32_VideoController get caption

So, looks like a bug in MINGW64, as described in:

https://github.com/bats-core/bats-core/issues/256
https://github.com/rupa/z/issues/278

Neofetch version
7.1.0, also Github latest (master, https://github.com/dylanaraps/neofetch/tree/cc39539c55c612ca8cfcd313cc8ccb84d89b0f82).
Screenshot

Config file
Default/ N/A

I can confirm #1799 fixes the issue for me on Windows. However, this change was not pushed or ported to the latest 7.1.0 release. I would appreciate a new tag + an update to the scoop repository.


Update FlagD to support multiple Kubernetes syncs

resolved with this PR, also adds behaviour for using multiple sync providers simultaneously


CHAPTER 8. FOLLOWERS

Dear Miguel ,
class User(UserMixin, db.Model):
...
followed = db.relationship(
'User', secondary=followers,
primaryjoin=(followers.c.follower_id == id),
secondaryjoin=(followers.c.followed_id == id),
backref=db.backref('followers', lazy='dynamic'), lazy='dynamic')
followers.c.follower_id == id    (id) What their source and why not use self.id

there is no self in this context, that is only available when you are in a method. The id in this context refers to the Column instance.

thank you :)


Attempt to perform arithmetic on a nil value (field 'y')

See https://github.com/giann/sirocco/issues/2. When I paste multiple lines of codes into the REPL of croissant, I got the same problem.
Problem exists on macOS 12.4 with Terminal/iTerm2/VSCode.
lua: /usr/local/lib/sirocco/prompt.lua:355: attempt to perform arithmetic on a nil value (field 'y')
stack traceback:
	/usr/local/lib/sirocco/prompt.lua:355: in function 'croissant.luaprompt.update'
	/usr/local/lib/sirocco/prompt.lua:431: in function 'croissant.luaprompt.ask'
	/usr/local/lib/croissant/repl.lua:22: in function 'croissant.repl'
	/usr/local/bin/croissant:52: in local 'route'
	/usr/local/bin/croissant:56: in main chunk
	[C]: in ?

iTerm2
# rachel @ ZhengPro in ~[22:55:25]
$ echo -e "\033[6n" | cat

# rachel @ ZhengPro in ~[22:55:28]
$ ;1R

# rachel @ ZhengPro in ~[22:56:01]
$ echo $TERM
xterm-256color

Terminal
# rachel @ ZhengPro in ~ [23:02:56]
$ echo -e "\033[6n" | cat

^[[5;1R%
# rachel @ ZhengPro in ~ [23:02:58]
$ 1R

# rachel @ ZhengPro in ~ [23:02:58]
$ echo $TERM
xterm-256color

https://user-images.githubusercontent.com/5410705/173788217-e32a10cf-ae5d-48c0-8503-67eb9268004b.mov

Many terminals won't respond to Prompt.escapeCodes.user7 control question until multiple lines of codes been completely entered.
https://github.com/giann/sirocco/blob/b2af2d336e808e763b424d2ea42e6a2c2b4aa24d/sirocco/prompt.lua#L407-L411


Numeric overflow when using preparedQuery.

Here is my example:
BigInteger number = new BigInteger("123456789123456789123456789");
Numeric numeric = Numeric.create(number);
String insertNumeric = "INSERT INTO test (number) VALUES ($1 / 10)";
client.preparedQuery(insertNumeric, Tuple.of(numeric), ar2 -> {});

Will insert 208066126.
But I try to ues postgresql PREPARE, it's OK:
PREPARE insert_test(numeric) AS INSERT INTO test (number) VALUES ($1 / 10);
EXECUTE insert_test(123456789123456789123456789);

Here is my sample project, just run ./gradlew run and see select * from test.test(Don't know why select does not show anything in output log):
reactive-pg-client-numeric-test.zip

I can reproduce it and find I can get the right result when using the SQL
INSERT INTO test (number) VALUES ($1::NUMERIC / 10).
The problem is when using with a math function such as division in the example, the server will return the metadata telling the value of $1 is INT4 type which should indeed be NUMERIC but postgres client does not know about this, then regards the value as a INT4 and truncates the actual numeric value, so you get a incorrect value here.
I think we need to investigate how we can handle these cases in a better way.


test: test new setup headless brnach

temporary PR to check some changes to setup-headless-action

works


Update the lost sentence

The following PR will adjust the sentence shown when you lose the game with one more appropriate with the typical gamer.

@liguori Would it be possible to merge this pull request?

We o fetapàààà


Discussion: General approach to component serialization

As of today (ECS 6.9) components are serialized using JSON and JSON envelopes (JSON serialized inside JSON strings). That worked well for a small scale with few entities and simple scenes. But the performance limitations of using JSON play against the potential of the platform.
The general approach was to use JSON, now there are some components that are known for sure that need a special serialization to optimize speed: Transform is a clear example. Since transform can be mapped 1-1 with a memory struct of floats, it makes sense that the serialization of the component is only a copy of the memory of the struct.
That may not be the case for components with complex structures, like Material, which has strings on it and other resources (or sub structs) like textures. The memory mapping is non viable.
For this problem, there are some options that need to be considered and evaluated:

Not all components may have the same serialization mechanism, it will enable some components to have their optimizations like the Transform component.
If we open the door to custom serialization mechanisms, then the implementation may become too complex and susceptible to segmentation in implementations, which may be a risk in the long run, considering maintenance costs.
Scene creators may define their own components. That assumption may lead us to a schema-based serialization and versioning, backwards compatibility must be taken into account.
flat/protocol buffers may be used as a schema-based serialization, but it incurs in operative costs and external tooling.
Implementation of serializers/deserializers must take into consideration performance constraints in C# and TS (speed and memory allocations)

The objective of this conversation is to evaluate options and the expected result is an ADR
Open questions:

are flatbuffers backwards compatible in case of removal of a field? if not, what is the path to be taken?

@leanmendoza could you transfer the analysis and decision to this issue?

Transfered from https://github.com/decentraland/sdk/issues/241
This pair, definitions and serialization, can live together from a parent definition, for example, a schema that auto generates both, the serialization and its type.
A point to be remembered is each component can have its own serialization, even a JSON one.
The proposal option are:

Using a JSON schema that generate a versionable append only component

Details
A versionable JSON schema that an append-only component
Types
Primitives
Its representation is a standard and the CPU understands it.

Int8 signed (big or little-endian)
Int16 signed (big or little-endian)
Int32 signed (big or little-endian)
Int64 signed (big or little-endian)
Float32 (IEEE-754 single-precision floating-point)
Float64 (IEEE-754 double-precision floating point)

Compound only from primitives and fixed types
Here we can introduce the recursive condition, all primitives types
are fixed in their lengths, but if we have a struct with all fixed
length fields, the struct is also a fixed length one.

Fixed struct with
Fixed array

The type of item can be any type, even an array

Other supported types (non-fixed-length)

Non-fixed-length arrays: they write the count of items

The type of item can be any type, even an array

String with UTF8 encode

If a component is a fixed-length struct, it can be mapped 1-1 in memory.
Question: what about the padding?
Approach and examples
Directory tree
The components are placed in a directory components and each component has its
own directory named in pascal case e.g.BoxShape. In the component directory,
there will be the version of the component. E.g.:

components/

components/Transform

v1.json

component/BoxShape

v1.json
v2.json

Maintenance (strict similar )
The only maintenance rule is each future version has to contain the previous one.
We could add a field at the end of the root struct, but we can not remove one. If we
wish to support optional fields, we can create a new type that stores the state
of existence of the mentioned field. Optional types break the fixed-length and
non-fixed-length components cannot be optimally copied to the struct in the languages
that be able to do it.
We neither can modify a child struct.
-- What about if a fixed-length struct in v1.json mutates to a non-fixed length in v2.json?
Well, the first part of v2.json is v1.json and we append fields at the end. With this rule,
the first part of v2.json is a fixed-length one.
Hypothetic case:

We have a flag in a BoxShape called isTriggered that indicates if a BoxShape was clicked.
This flag was introduced in components/BoxShape/v1.json, at the begging of its life ECS 7.0.0.
In ECS 7.3.0 we want to add the feature that indicates with what button was clicked.
It sounds seductive to change the isTriggered from a boolean to a struct that has the
flag and the button field. WE CAN NOT DO THIS.
Instead, we add a field at the end with isTriggeredV2 or a triggeredButton. In this case,
we'll create isTriggeredV2 as a struct with two fields: flag: boolean and button: int32.

So far if the Renderer sends a BoxShapeV2 to a scene, the scene will be able to read it,
no matter if the scene was compiled with ECS 7.0.0 or ECS 7.3.0.
The same occurs on the opposite side if the renderer has the implementation of BoxShapeV2 as
the last updated version also can get a BoxShapeV1 from the ECS 7.0.0 scene.
Generation
The Transform component
{
  "schema": {
    "type": "struct",
    "fields": {
      "position": {
        "type": "struct",
        "fields": {
          "x": { "type": "float32" },
          "y": { "type": "float32" },
          "z": { "type": "float32" }
        }
      },
      "rotation": {
        "type": "struct",
        "fields": {
          "x": { "type": "float32" },
          "y": { "type": "float32" },
          "z": { "type": "float32" },
          "w": { "type": "float32" }
        }
      },
      "scale": {
        "type": "struct",
        "fields": {
          "x": { "type": "float32" },
          "y": { "type": "float32" },
          "z": { "type": "float32" }
        }
      }
    }
  },
  "metadata": {
    "componentId": 1
  }
}

Protocol Buffer

Details
Protocol Buffer
https://developers.google.com/protocol-buffers

Flat Buffer

Details
Flat Buffer
https://google.github.io/flatbuffers/

Research
To give a real comparison of these approaches, they were tested two recurrent components: Material and Transform. Material is a full optional field component and Transform is a component that probably never changes and it'll be sent a lot.
The scenario was:

Test were running in their typescript version (it'll be where the code of scene is executed)
The number chosen was arbitrary and bigger enough for the test.
The based code is https://github.com/decentraland/ecs/tree/1c4bb4c9fbacf9b673790ed3dcaa5b9d66a25fe1

Results:
https://docs.google.com/spreadsheets/d/1RGFkudVUMlOkNaQZFYAr0qEvVlrULmWp_F1fGBw0pwM/edit?usp=sharing

Expand tables
| Serialization of 12,000 full filled materials | Serialization of 12,000 empty materials
-- | -- | --
Type of serialization | Time [ms] | Size [KB] | Time [ms] | Size [KB]
EcsType | 54.63 | 2784 | - | -
EcsType with optional | 92.17 | 2592 | 12.74 | 192
FlatBuffer | 88.44 | 12288 | 103.2 | 12288
Protocol Buffer | 55.23 | 2988 | 50.57 | 48
| Deserialization of 12,000 full-filled materials
-- | --
Type of serialization | Time [ms] | Size [KB]
EcsType | 150.26 | 2784
EcsType with optional | 134.78 | 2592
FlatBuffer | 185.66 | 12288
Protocol Buffer | 65.14 | 2988
| Serialization of 12,000  Transforms
-- | --
Type of serialization | Time [ms] | Size [KB]
EcsType (raw) | 5.57 | 480
FlatBuffer | 8552.9 | 12288
Protocol Buffer | 99.52 | 420
  |   |
  | Deserialization of 12,000  Transform
Type of serialization | Time [ms] | Size [KB]
EcsType (raw) | 14.6 | 480
FlatBuffer | 37.45 | 12288
Protocol Buffer | 16.49 | 420

Expand charts

Short analysis
Flatbuffer will be discarded in this analysis because it seems to be not compatible with having an external byte builder buffer, its optimization is more oriented to a C++ implementation and an ecosystem with all Flattbuffer. Another inconvenience is its code-generator for typescript, it uses classes and these would introduce extra retyping work, ECS 7.0.0 is a strictly data-oriented paradigm.
Protobuffer demonstrated a powerful performance in complex types, and raw EcsType is in the same level, but for the Transform case, raw is around 20x faster for serialization.
For the maintenance, Protobuffer has a clear advantage. Since all their fields are naturally optional, the messages can be changed completely. EcsType approach doesn't have, and for each version it'd be necessary to send the previous data.
Conclusion

For fast serialization and immutable&fixed-length types, the best option is to serialize directly in memory (like transform).
For the rest of the components, Protocol Buffer seems to be the best option.

Moved to https://adr.decentraland.org/adr/ADR-123