Initial MDS implementation

Store each connection in a shared map
Capture special IP address, redirect to MDS handler
MDS handler takes 4 tuple as input.
Response is JSON containing identity

Also has an example Golang library implementation to add HTTP middleware that extracts the identity.
Istio tests: https://github.com/istio/istio/pull/44536 (blocked by this PR, of course)

Store each connection in a shared map Have we thought about how this might conflict with eBPF-based + map-based conntracking?
Most eBPF CNIs will need to do something like this, and it will work almost exactly the same way (just in kernelspace).

Sorry I'm missing how this would benefit users since we said we want to keep ztunnel small and lean.  MDS - is this google cloud metadata service or more generic?

It has nothing to do with Google - this allows users applications to find the peer identity. Today they can do this with XFCC header, but that is HTTP based which we cannot do in ztunnel. This provides a generalized way.
Note this is needed for sandwiching as well

It has nothing to do with Google - this allows users applications to find the peer identity. Today they can do this with XFCC header, but that is HTTP based which we cannot do in ztunnel. This provides a generalized way.
Note this is needed for sandwiching as well

Thanks, btw, what does MDS stand for?
I thought we aren't using sandwich approach at all?

On Thu, Apr 27, 2023 at 7:29 AM Eitan Yarmush @.***>
wrote:

@.**** commented on this pull request.
In go-metadata/istio.go
https://github.com/istio/ztunnel/pull/504#discussion_r1179250379:

"net/url"

"time"
+)

+// ConnectionMetadata returns the metadata about a connection
+type ConnectionMetadata struct {

Identity string json:"identity"
+}

+// metadataClient provides a default client for all metadata requests
+var metadataClient = http.Client{

Timeout: time.Second,
+}

+// metadataContextKey is the key the metadata handler will store connection metadata in
+var metadataContextKey = &contextKey{}
I agree that I don't love implicit context storage, but this is what
go-grpc uses all over the place, so in this instance probably better to
follow the same pattern

The pattern may be fine in some cases. That's not my concern - this library
should not be a framework, just expose the raw command.
It will likely be combined with checks for XFCC, ProxyProtocol, etc - there
are plenty of auth schemes an app may support, and
plenty of frameworks that set the identity in some way.

MDS is 'metadata server' AFAIK, all cloud vendors have one. Not sure who
was first.
This has little to do with sandwich - any user application may need to know
the identity of the peer.
For example we may have authz to allow Alice to talk to Bob - but Bob still
wants to know who's talking to.
On Thu, Apr 27, 2023 at 6:02 PM Lin Sun @.***> wrote:

It has nothing to do with Google - this allows users applications to find
the peer identity. Today they can do this with XFCC header, but that is
HTTP based which we cannot do in ztunnel. This provides a generalized way.
Note this is needed for sandwiching as well
Thanks, btw, what does MDS stand for?
I thought we aren't using sandwich approach at all?
—
Reply to this email directly, view it on GitHub
https://github.com/istio/ztunnel/pull/504#issuecomment-1526834424, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAUR2XX36RPONSNSU7QFZDXDMJJFANCNFSM6AAAAAAXLK5ZTQ
.
You are receiving this because you commented.Message ID:
@.***>

Interesting - will ztunnel break the platform MDS ? I think CNI needs to
probably exclude the range ( most platforms seem to be using
same /24).
But ztunnel MDS itself needs to be intercepted, how else can the pod
communicate with it ?
On Thu, Apr 27, 2023 at 7:45 PM John Howard @.***>
wrote:

@.**** commented on this pull request.
In src/proxy/outbound.rs
https://github.com/istio/ztunnel/pull/504#discussion_r1179878713:

     orig_dst_addr: SocketAddr,

     block_passthrough: bool,
 ) -> Result<(), Error> {

   let remote_ip = remote_addr.ip();

   if orig_dst_addr.ip() == METADATA_SERVER_IP {

I am not sure how we can opt out? since ztunnel will capture all traffic
—
Reply to this email directly, view it on GitHub
https://github.com/istio/ztunnel/pull/504#discussion_r1179878713, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAUR2VDSEHDRTXDW5TOCPTXDMVMHANCNFSM6AAAAAAXLK5ZTQ
.
You are receiving this because you commented.Message ID:
@.***>

Hmm we did fix platform MDS somehow and have a test for it. Let me see how
we did it
On Thu, Apr 27, 2023, 8:30 PM Costin Manolache @.***>
wrote:

Interesting - will ztunnel break the platform MDS ? I think CNI needs to
probably exclude the range ( most platforms seem to be using
same /24).
But ztunnel MDS itself needs to be intercepted, how else can the pod
communicate with it ?
On Thu, Apr 27, 2023 at 7:45 PM John Howard @.***>
wrote:

@.**** commented on this pull request.
In src/proxy/outbound.rs
https://github.com/istio/ztunnel/pull/504#discussion_r1179878713:

orig_dst_addr: SocketAddr,
block_passthrough: bool,
) -> Result<(), Error> {

let remote_ip = remote_addr.ip();
if orig_dst_addr.ip() == METADATA_SERVER_IP {

I am not sure how we can opt out? since ztunnel will capture all traffic
—
Reply to this email directly, view it on GitHub
https://github.com/istio/ztunnel/pull/504#discussion_r1179878713, or
unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AAAUR2VDSEHDRTXDW5TOCPTXDMVMHANCNFSM6AAAAAAXLK5ZTQ
.
You are receiving this because you commented.Message ID:
@.***>

—
Reply to this email directly, view it on GitHub
https://github.com/istio/ztunnel/pull/504#issuecomment-1526934027, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEYGXJOBNZ25SLBRUKS4MTXDM2UPANCNFSM6AAAAAAXLK5ZTQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

MDS is 'metadata server' AFAIK, all cloud vendors have one. Not sure who was first. This has little to do with sandwich - any user application may need to know the identity of the peer. For example we may have authz to allow Alice to talk to Bob - but Bob still wants to know who's talking to.
…

Thanks, thinking loud here - could this be something opted in so it doesn't impact users don't use MDS.

MDS is 'metadata server' AFAIK, all cloud vendors have one. Not sure who was first. This has little to do with sandwich - any user application may need to know the identity of the peer. For example we may have authz to allow Alice to talk to Bob - but Bob still wants to know who's talking to.
…

Thanks, thinking loud here - could this be something opted in so it doesn't impact users don't use MDS.

This doesn't impact anyone if they don't use it, t hey have to explicitly call the metadata server for it to do anything

The reason for the hardcoded IP is pretty simple: in an app, how do you
make the http request ? You need a DNS name or IP.
Many MDS servers are also DNS servers - there is a plan to allow running a
DNS proxy in ztunnel. Also what hostname would you
use - and how would CNI know that some VIP is actually the MDS address ?
All MDS servers I know rely on a hardcoded 'well known' address in that
range. We can also use a special 150xx port, but if we
want ztunnel to be a resolver it'll need to listen on 53 ( yes, we can do
some complicated redirection to use a different port for DNS - but
why not use the simplest solution ). If we are using this pattern - I would
like to see a good reason why we should follow
a different pattern, and what would that be - can't think of a simpler or
cleaner solution.
Also this whole thing doesn't need a fancy iptables or eBPF or k8s - CNI
can simply add a route and add a second
address with the well-known IP on the ztunnel pod.
It will also work on VMs and non-K8S environments - even in cases where no
iptables are possible ( ztunnel can work without
any CNI or iptables, by setting SOCKS_PROXY env variable with most
applications ).
On Fri, Apr 28, 2023 at 3:42 PM Ben Leggett @.***>
wrote:

@.**** commented on this pull request.
In src/proxy/metadata.rs
https://github.com/istio/ztunnel/pull/504#discussion_r1180864757:

fn drop(&mut self) {

   trace!(tuple=?self.tup, "forgetting connection");

   let mut cm = self.ct.conns.lock().unwrap();

   cm.remove(&self.tup);

}
+}

+#[derive(Clone, Debug, Serialize)]
+pub struct ConnectionMetadata {

#[serde(default)]
identity: Option,
+}

+/// METADATA_SERVER_IP provides the well-known metadata server IP.
+/// This is captured by the redirection.
+pub const METADATA_SERVER_IP: IpAddr = IpAddr::V4(Ipv4Addr::new(169, 254, 169, 111));
Is it any simpler to have it in the CNI layer instead of in ztunnel though?
Long term I think so, since the CNI (ours or someone else's) will be
handling/redirecting the traffic based on intended dest anyway, and we
already have the concept of directing packets to different well-known
ztunnel listener ports and handling them differently as a result (for
purposes of early exit/handling bypass in eBPF/iptables redirect logic, for
instance)
So far we have leaned towards keeping the CNI simpler so there is a better
chance we can replace it with better approaches (and rust is easier than
iptables/eBPF)
Absolutely a fan of keeping CNI simpler, but sending captured traffic
based on intended dest to a given ztunnel listener is what CNI is for -
that's what it does best. It's how we do DNS redir for instance. CNI redir
already does special-casing around certain dest ports (or whether src is
host ip, etc etc) for purposes of data path optimization - it's already
handling this stuff.
Feels logically consistent to maintain that pattern here, and it has the
added benefit of making it very easy for the IP to be changed or redefined
without ztunnel caring (or us needing to add a flag to ztunnel to let
people define what they want).
—
Reply to this email directly, view it on GitHub
https://github.com/istio/ztunnel/pull/504#discussion_r1180864757, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAUR2XZUNHVJ75L7CON3MLXDRBXDANCNFSM6AAAAAAXLK5ZTQ
.
You are receiving this because you commented.Message ID:
@.***>

On Fri, Apr 28, 2023 at 12:49 PM Louis Ryan @.***>
wrote:

@.**** commented on this pull request.
In go-metadata/istio.go
https://github.com/istio/ztunnel/pull/504#discussion_r1180762108:

"context"

"encoding/json"
"fmt"
"io"
"net"
"net/http"
"net/url"
"os"
"time"
+)

+// ConnectionMetadata returns the metadata about a connection
+type ConnectionMetadata struct {

// Identity provides the identity of the peer.
// Example: spiffe://cluster.local/ns/a/sa/b.
Identity string json:"identity"

Can we open a tracking issue to capture the other information about the
identity that we would like to expose here
Over time I would expect to see us add either a field or a new API that
provided details from the public key of the peer

And probably some of the telemetry or labels and other info we got from the
PTR-DS. I agree in time we'll likely add more,
but the first step is to get identity in and validate.
I would also expose a lookup using only the peer IP - the 4-tuple is only
really needed if ztunnel can't impersonate the original
source IP.

I think the interesting question is what are we trying to prove to the
caller about the veracity of the identity we just provided. Options include:

How can the caller know that the MDS is reliably acting on its behalf.
Clearly theres a certain amount of infrastructural trust seeing as this is
coming from the 'network' as perceived by the app. One option here if for
the MDS to offer a "who am I" API. This is a bit like SDS but now its an
implicit part of the infrastructure. Response to the call should either be
the current public cert or some shorthand of it. Variant, can prove that
the infra has the private key which is shielded from the app by signing
some random bytes passed to the API which can then be verified with the
public key. Likely want to deliver a shorthand in addition to x509 to
shield users from parsing its gory details.

I think this may be overkill and defeat the purpose. If the app is willing
to do all this - and link libraries and deal with certs - they are probably
better off
just using a proxyless library. With sidecars they don't get any guarantee
either, nor with most CNI.
It's also not the caller that needs to be assured - what is it going to do
? The network admin, who installs the CNI and ztunnel needs to be sure
all packets go to ztunnel, but that can be exposed using telemetry.

How can the caller know that the infrastructure actually performed a
secure handshake with the identified peer? This is harder and of
debateable value. If we provide proof that infrastructure already has the
public and private keys as described in 1 above then the infra is already
capable of impersonating the application for whatever purposes it sees fit.
We could do something like a 'signed ping'
 Client generates random bytes and encrypts with public key of peer (opaque to local ztunnel)
 Client app calls 'secure ping' passing the encrypted bytes
 local ztunnel simply forwards encrypted bytes to the identified peer
 peer ztunnel decrypts random byes with private key and returns their signature
 local ztunnel forwars message signature to client
 Client compares message signature to locally computed hash

This proves that there is something between the client and the peer that
has access to the private key associated with the identity. The client
doesn't know who or what that is necessarily

That seems for super-advanced users - it may be something for a specialized
ztunnel to support for specialized apps with libraries that can take
advantage of it,
but maybe not something to have in OSS and the simple/clean implementation
we have ? It is a cool idea.

Message ID: @.***>

The reason for the hardcoded IP is pretty simple: in an app, how do you make the http request ? You need a DNS name or IP. Many MDS servers are also DNS servers - there is a plan to allow running a DNS proxy in ztunnel. Also what hostname would you use - and how would CNI know that some VIP is actually the MDS address ?

I'm suggesting if you have a need to map an arbitrary environmental IP of your choice to always get redirected to a listener in ztunnel, it's pretty simple to do that with a single iptables rule without ztunnel needing to care what that IP is, or needing to hardcode it in ztunnel. We could pick a fixed IP, anyone who wanted to do something different could change it, ztunnel wouldn't care either way.

All MDS servers I know rely on a hardcoded 'well known' address in that range.
MDS clients do, I'm not sure the MDS server being added to ztunnel does.

We can also use a special 150xx port, but if we want ztunnel to be a resolver it'll need to listen on 53 ( yes, we can do some complicated redirection to use a different port for DNS - but why not use the simplest solution ).

ztunnel already listens on 15053 for this purpose, I believe. We have iptables rules that rewrite UDP on port 53 to 15053 within ztunnel's netns.

If we are using this pattern - I would like to see a good reason why we should follow a different pattern, and what would that be - can't think of a simpler or cleaner solution. Also this whole thing doesn't need a fancy iptables or eBPF or k8s - CNI can simply add a route and add a second address with the well-known IP on the ztunnel pod. It will also work on VMs and non-K8S environments - even in cases where no iptables are possible ( ztunnel can work without any CNI or iptables, by setting SOCKS_PROXY env variable with most applications ).

I'm proposing we use the pattern already in use - ztunnel exposes well-known ports, and CNI redirects traffic, based on (variously) dest IP, dest port, and dest packet type to those ports - for this as well. A fixed IP within ztunnel isn't a requirement for this, and it feels weird to add it here when nothing else currently in ztunnel works like that.

On Fri, Apr 28, 2023 at 6:18 PM Ben Leggett @.***>
wrote:

The reason for the hardcoded IP is pretty simple: in an app, how do you
make the http request ? You need a DNS name or IP. Many MDS servers are
also DNS servers - there is a plan to allow running a DNS proxy in ztunnel.
Also what hostname would you use - and how would CNI know that some VIP is
actually the MDS address ?
I'm suggesting if you have a need to map an arbitrary environmental IP of
your choice to always get redirected to a listener in ztunnel, it's pretty
simple to do that with a single iptables rule without ztunnel needing to
care what that IP is, or needing to hardcode it in ztunnel. We could pick a
fixed IP, anyone who wanted to do something different could change it,
ztunnel wouldn't care either way.

I see, you are right, that would work too.
However since ztunnel is intercepting all egress anyways - and istio-cni is
extremely complex and hopefully will become part of the native CNI,
I would rather keep it in ztunnel and simplify the requirement on the CNI
to 'just redirect all egress from the app to ztunnel'.
All MDS servers I know rely on a hardcoded 'well known' address in that

range.
MDS clients do, I'm not sure the MDS server being added to ztunnel does.
We can also use a special 150xx port, but if we want ztunnel to be a
resolver it'll need to listen on 53 ( yes, we can do some complicated
redirection to use a different port for DNS - but why not use the simplest
solution ).
ztunnel already listens on 15053 for this purpose, I believe. We have
iptables rules that rewrite UDP on port 53 to 15053 within ztunnel's netns.

Unless someone added DNS - I think agent does that. There is an option to
also listen on 53 and not require iptable rules. Just because
it is possible to use iptables - it doesn't mean we should ( and the rules
for 53 require raw and some very ugly stuff - UDP is far more
complicated )

If we are using this pattern - I would like to see a good reason why we
should follow a different pattern, and what would that be - can't think of
a simpler or cleaner solution. Also this whole thing doesn't need a fancy
iptables or eBPF or k8s - CNI can simply add a route and add a second
address with the well-known IP on the ztunnel pod. It will also work on VMs
and non-K8S environments - even in cases where no iptables are possible (
ztunnel can work without any CNI or iptables, by setting SOCKS_PROXY env
variable with most applications ).
I'm proposing we use the pattern already in use - ztunnel exposes
well-known ports, and CNI redirects traffic, based on (variously) dest IP,
dest port, and dest packet type to those ports - for this as well. A fixed
IP within ztunnel isn't a requirement for this, and it feels weird to add
it here when nothing else currently in ztunnel works like that.

Another pattern already in use is CNI redirects all traffic - and ztunnel
routes it wherever it needs to go. I would rather reduce complexity in CNI.
I also mentioned we have the SOCKS mode that is independent of iptables (
works without any ) - with the current method it would work
just fine. And if we ever add a HTTP/1.1 CONNECT proxy ( for the few apps
that support HTTPS_PROXY and not SOCKS ) - it would work too.
Message ID: @.***>

PR needs rebase.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Looking back at this:

It probably needs a new name, since it's unrelated to https://github.com/istio/istio/issues/43937
This is functionally identical to SPIFFE/SPIRE self-identity-lookup in a lot of ways, and if you were using SPIRE with Istio this would be a redundant mechanism. That's not necessarily an objection, but it concerns me a bit.

Lets drop this for now to keep things minimal.


libnd4j: confusion_matrix op: switch output dtype config from iarg to darg

https://github.com/eclipse/deeplearning4j/blob/master/libnd4j/include/ops/declarable/generic/parity_ops/confusion_matrix.cpp#L69-L76
ATM can't configure dtype without also configuring the number of classes.
If the datatype was set via a darg instead of an iarg, this wouldn't be a problem.

Implemented in my branch. Will be merged later.


UI - Error messages are not within the table header

v0.51.0
Steps to Reproduce:

Create 2 load balancer with same exposed port
Find the container that errors out due to port conflict

Result: The error message is overlapping with the table.

Expected Result: The error message not overlap with the table.

Version - master 3.20
Verified fixed


Fixed joystick POV values not being set properly for devices that are…

This fixes an issue where joystick/controller POV values were not set correctly for devices that are accessed via DirectInput.
See: https://github.com/SFML/SFML/pull/1634#issuecomment-667706226
DirectInput provides a combined value for both X and Y hat directions, this value has to be translated back to the separate X and Y values that SFML exposes through its API. Previously the implementation erroneously treated X and Y hat axes as separate objects which could only work if the device reported the same hat twice as distinct objects. This was the case with some devices but not all causing this bug to not always be reproducible.
This fix was tested with both a joystick and a controller (the same one mentioned in the linked comment).

Tested this PR with the joystick example with my controlled Torid Speedlink. It's got two modes, XInput and DirectInput (dunno which is which), and both appear to work fine.
(Just the second sees the back triggers as on-off buttons instead of analog input. But I think that's a controller thing.)
https://i.imgur.com/aiD9ebg.gif
https://i.imgur.com/WlakuyF.gif


Patch com.snowplowanalytics.factotum/factfile/jsonschema/1-0-0

#317

Merged into final release PR, closing


Decouple Streaming Use-Case from File IO Abstractions

Is your feature request related to a problem or challenge?
Currently we accommodate streaming workloads within DataFusion by overloading the file IO abstractions.
This is not always a very good fit and results in a number of workarounds:

Providing PartitionedFile information to methods that write new files via FileSinkConfig - https://docs.rs/datafusion/latest/datafusion/datasource/physical_plan/struct.FileSinkConfig.html#structfield.file_groups
Passing unbounded_input to parallel write logic that really shouldn't care - https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/src/datasource/file_format/write/orchestration.rs#L79
Special logic to handle reading files that aren't regular files - https://github.com/apache/arrow-datafusion/pull/7282#discussion_r1294019448
The somewhat confusing notion of an "infinite" file - https://docs.rs/datafusion/latest/datafusion/datasource/physical_plan/struct.FileScanConfig.html#structfield.infinite_source
An ObjectStore::append method that doesn't naturally fit with the design goals of the project - https://github.com/apache/arrow-rs/pull/4978/files#diff-fbf0cdd67af2b24703a831d28c932bfe0535234e3b745d3a5437b9c0bc881ddeR89
Schema inference within ListingTable currently assumes files are finite
ObjectMeta is passed around in plans with the implicit assumption that files are not changing underneath
And more...

As DataFusion gets more sophisticated about handling catalogs, reading/writing partitioned data, caching data, this overloading is getting more and more arcane and hard to reason about, and I think it is overdue we do something to address it.
Describe the solution you'd like
I would like to separate the notions of FileSink and FileScan from a StreamSink and StreamSource, this would allow abstractions that better fit their respective use-cases.
In particular

FileSink and FileScan can focus on reading/writing partitioned immutable files following standard big data practices
StreamSink and StreamSource can focus on reading/writing CSV / JSON (/ Avro) data from streaming sources

Not only would this simplify the current code, but would also expand the streaming support in DataFusion

Allows for more efficient non-blocking IO, as linux FIFO's support poll(2) (unlike general files)
Potential future integrations with data streaming systems such as Kafka, etc...

Describe alternatives you've considered
No response
Additional context
No response

I would like to separate the notions of FileSink and FileScan from a StreamSink and StreamSource, this would allow abstractions that better fit their respective use-cases.

Sounds like a great idea to me.

+1 from me as well. The more we finely optimize batch/streaming reads and writes the more the specific implementations ideally will diverge a bit. I felt this when working on parallelizing writes as some code that work great for batch caused inescapable infinite loops for infinite sources, requiring adding special cases to the code. Separating the two cases will allow both use cases to optimize appropriately without harming the other use case.

I created a POC of ripping append semantics out of ListingTable in https://github.com/apache/arrow-datafusion/pull/8017 and the good news is this is very achievable :smile:

I, too, am in favor of this split; the implementation of the StreamSink methodology stands to significantly enhance the unified approach, maintaining the integrity of features on both ends. At the moment, my agenda is filled with high-priority tasks, but I am available to assist with code reviews and provide guidance on the overall structure of the code. Additionally, I will contribute some tests to the codebase to verify that we are preserving the streaming functionality. I believe this will also serve to optimize the Datafusion use cases. Thank you for your hard work, @tustvold, @alamb, and @devinjdangelo

Whilst discussing potential API designs with @alamb the question came up of whether the existing TableProvider abstractions are now sufficiently mature that we could just use them directly. In particular the combination of TableProvider::insert_into and TableProviderFactory I think may cover all the necessary extension points for people to implement streaming workloads as they deem fit.
I tried to show what this might look like in https://github.com/apache/arrow-datafusion/pull/8021 PTAL.
If this would be sufficient for the streaming use-cases, perhaps we can avoid needing to define further abstractions, and can simply remove the streaming support from ListingTable proper, i.e. a more polished version of #8017?

There is a lot of related discussion on https://github.com/apache/arrow-datafusion/pull/8021#issuecomment-1802589931

I wrote an initial design document, @alamb and @tustvold , can you check this out?

I wrote an initial design document, @alamb and @tustvold , can you check this out? I have tried to cover all use cases while making it easier to develop the batch mode feature. The document allows commenting, so please share your thoughts 😀

Thanks @metesynnada  -- I will do so, likely later today or tomorrow

Thank you for writing this up, before getting into specific questions about the design, I think it would help to articulate what the objectives here are? Some possibilities might be:

To replicate the functionality currently provided by ListingTable
To facilitate the addition of new streaming sources beyond FIFO files

From my reading of the document, the proposed abstractions appear to largely mirror the equivalent TableProvider abstractions used in #8021, whilst also quite closely fitting to the current FIFO file mechanism based around serialized byte streams. This makes me suspect that neither of the above is quite the vision here?
On a more concrete level, it occurs to me that if StreamStoreRegistry were moved under StreamingTableFactory, all of this functionality would be encapsulated under one "extension point". Perhaps this might provide a mechanism to iterate and evolve these APIs incrementally, e.g. in datafusion-contrib or similar, without needing to front-load the design effort? This would allow new streaming sources to be added, and the abstractions evolved as necessary, all in one place? I for one anticipate Kafka, and certainly Kinesis, will require some non-trivial iteration on these APIs to accommodate their particular quirks. Just a suggestion, but it might not only allow us to make progress on this ticket quicker, but also yield a better development experience for ongoing work on the streaming functionality.

I have read the document and I really like it. Thank you @metesynnada.
Use Case
My understanding is that the goal of this exercise is not primarily about the specifics of decoupling append/FIFO from ListingTable, but the more general goal of easily using DataFusion to build systems with functionality similar to Apache Flink, as described in https://github.com/apache/arrow-datafusion/issues/4285
In particular the goal is to make it easy to

Read data from "streaming" sources
Write data to "streaming" sinks

Where "streaming" means the data is NOT composed of a fixed set of immutable objects (as in object storage), but rather some number of "streams" which can be appended / consumed. Some examples are FIFO files, Kafka, RabbitMQ, Kinesis, etc.
One of the major challenges in trying to implement such streaming systems today is that ListingTable, as its name implies, is implemented assuming the data is being read or written to an object store, with both the benefits and limitation of the object_store  API.
Design Feedback
I really like the idea to create a parallel StreamingTable to ListingTable that is backed by a different API than object store and that can be used to build such streaming usecases. It much better matches:

The differences in reading from a streaming vs immutable discrete objects in object store
The difference in writing data out in a streaming fashion compared to immutable discrete objects in object store

Requests / ideas
As we expand DataFusion in this streaming direction, I would like to request we take this opportunity to define some more crate boundaries so the ListingTable is not so tightly integrated / intertwined, and likewise StreamingTable is not so so intertwined -- perhaps we can aim to end up with three new crates datafusion_listing_table datafusion_streaming_table and datafusion_data_format
Next steps:
Would it be possible to create a PoC / proposal with the basic APIs and make sure they fit together and into the rest of DataFusion? This is likely to be a large change, so I think getting the skeleton in place and then filling out the details in subsequent PRs (rather than one massive one) would be my preferred process.

I think we can probably sketch out enough of this StreamingTable API to migrate the existing FIFO support pretty quickly. I would be happy to help / review quickly

I would like to request that we don't overload the file format abstractions for streaming use-cases as that is part of the confusion I am trying to eliminate. There isn't a great of logic that can be shared anyway

I agree that starting with a basic StreamingTable abstraction, and copying over only API features from the currentListingTable that are needed (and not replicating APIs until necessary) makes a lot of sense

I would like to request that we don't overload the file format abstractions for streaming use-cases as that is part of the confusion I am trying to eliminate.

Do you mean the FileFormat trait?  At the moment, it seems like it is tightly tied to ObjectStore both in the infer methods  and create_writer_physical_plan via FileSinkConfig so I agree it is not clear how to make this easily fit into the streaming design
My understanding that the StreamingTable would an analgous trait (like StreamFormat perhaps).

My understanding that the StreamingTable would an analgous trait (like StreamFormat perhaps).

Exactly.

I think we can probably sketch out enough of this StreamingTable API to migrate the existing FIFO support pretty quickly. I would be happy to help / review quickly

Thank you 🚀
We will start implementing this next week to get the core APIs and the functionality in place. Your help in fine-tuning the design through reviews during this process will be much appreciated. @metesynnada and I scoped out the work and it seems the implementation should go relatively smoothly.

What do people think about perhaps polishing up https://github.com/apache/arrow-datafusion/pull/8021 into a v0.1 for StreamingTable? I believe most of the functionality is there, and I would be happy to do this.
Just thinking about ways I could help get this over the line...

Thank you @tustvold  -- I think that would be very helpful assuming @metesynnada  / @ozankabak  are not already working on it.

We started working on it and hope to get things ready for review this week. We will appreciate your help on reviewing it. Thank you @alamb and @tustvold

Is there any value in starting small with something based off https://github.com/apache/arrow-datafusion/pull/8021? I'm confident I could have something ready for review this morning?

I want to avoid any possible overlap between the progress so far on our end and your time. Given that we will have something out this week, I think we can best utilize your time if you carefully review the upcoming PRs and help us optimize the design and code through that process. Once we have the basics out, there will also be some important work to remove leftover code in the ListingTable end to optimize it fully for bounded files, your help will be crucial there too.

My time is immaterial, I just want to get this over the line. If it all gets replaced in 2 weeks, I would be more than happy.
As it stands now though, from my perspective, we have had a largely functionally complete PR that preserves the current functionality for weeks, and was signed off by @metesynnada - https://github.com/apache/arrow-datafusion/issues/7994#issuecomment-1788838912 2 weeks ago. I am therefore I think pretty understandably frustrated at the prospect of yet further delay for reasons that haven't been articulated.
I would very much like to just get this over the line so that the work in DF and IOx that is blocked on it can move forward. If that means doing work that will be thrown away, so be it, it has mostly already been done.

Dear @tustvold,
Just like you, we also would like to get to a state where everyone can move forward, as we have been negatively impacted by this situation. I would like to remind you that we did not create the situation we are all in. The breaking arrow-rs change, which is the root cause of the time pressure, was done without us knowing. We were only notified of it after your merged it.
The fact that we agreed on the general aspects of the approach to adopt in terms of the roadmap does not mean we committed to a very specific timeframe. If we are to discuss frustrations, we can contribute quite a few talking points. I don't want to go down that path, as it will not help remedy the situation. I invite you to consider doing the same.
Whatever happened, happened. We are all active contributors to this project, and regardless of the situation we should focus on finding productive ways to work together. @alamb and I engaged in such a discussion last week to chart a path forward, and we are acting in good faith to get to a resolution as quickly as possible. The first step was to come up with a proposal, and we did in a timely manner. @alamb took a detailed look at it, and it seems we are on the same page w.r.t. to the general approach within it. Now, we are in the process of implementing it. I have said before that we will have something out this week, and @alamb will help us with reviews and suggestions to move the ball forward quickly when it is out (#8151). We are making progress as a community with a focus on moving forward. Please, I request from you to not exacerbate the situation needlessly. If you would like to help, we do welcome your help -- I already mentioned how.
Datafusion is a great project with a huge role to play in the data ecosystem if we are able to form and maintain a cohesive community around it. I hope we don't lose sight of this due to temporary roadblocks like the one we are going through right now.

I have now updated https://github.com/apache/arrow-datafusion/pull/8021 inline with the design proposal, if this is acceptable I would very much like to consider moving forwards with it as a stopgap. I leave the rest with you

Thank you, I appreciate you sharing your reflection publicly. I am glad that we are able to come together and collaborate towards a solution. @metesynnada and I are reviewing #8021 to see how we can use it as a base to accelerate the work towards implementing the proposal. We will circle back shortly.


[FEATURE] [FEATURE] Implement Demographics Questionnaire

Re: University Monday 2022/10/10 / 2022-10-12
2.3 Research Platform: This will be phase 1 of this feature and we will move on to phase 2 in January. This is the rough idea:
After Onboarding and a NetID login present the user with another onboarding panel that asks if the user is interested in participating in Research Projects? (Must be >= 4 privacy and NetID login)
If No then go to the home panel and if Yes then show a Demographics Questionnaire.
The Demographics Questionnaire results will be stored in Core BB. We will need a JSON structure that will have a Title, Short Description and then multiple questions that will all have checkbox entries and each question may have two short descriptions, one before the beginning and one after the checkboxes.
We will need a Panel and Favs widget that have tabs for Research Invites and Research Projects being participated in.
In the users Profile Panel we will need a new panel for the demographic questions from onboarding. This may have new questions also and will also need the ability to "Opt Out" if participating or "Opt In" (same as onboarding) if not.
There will be a Core BB permission that is needed in order to create a Groups BB group that will be marked as a Research Group.
A Group BB research Group will use the Groups BB "request to join questionnaire" that we already have.
In the Groups BB Group UI we will need to be able to "Send an invite" (Notification) to all users who meet some demographic requirements.
If a user gets an invite and accepts it, then we will show the Groups BB questions and after completion do a request to join.
2.4 Groups UI and Groups BB:
2.4.1 Add "Research Group" attribute.
2.4.2 Notifications group by group
Re: University Monday 2022/10/10 / 2022-10-13
1.2 Research Feature
1.2.1 I attach the questions, they may change. We will use checkboxes only for phase 1 and some will be single selection within a question and others multiple checks. In this set of questions, only 8 is multiple choice.
Re: University Monday 2022/10/10 / 2022-10-13
3 Misho/Mladen, another bit of info on the Research feature. In the Group we will have another entry for all users called "Notifications". This will yield a panel similar to Group Updates when clicking the Notification icon at the top of the home panel, but it will have an additional checkbox called "Override my notification preferences for this group". See image attached.
Re: University Monday 2022/10/17 / 2022-10-18
6 Misho, please start implementing the Research feature then.
6.1. "Demographics Questionnaire: I can handle this. It would be nice to get some UI mockups but this is not obligatory (i.e. I could create it myself)." - Do it
6.2. "Group Invites / Notifications / Research Groups. I would suggest Dobromir or Todor to handle this as they are much more familiar with the Groups than me. I had been working in Groups quite occasionally on some small pieces of functionality." - Yes OK please direct as and when one of them has time and we have what we need from the backend so they can implement it.
Linked to rokwire/illinois-app#2465

Re: University Monday 2022/10/17 / 2022-10-20
3 Misho, Research. Great start.
3.1 Can we make the popup in onboarding a panel so it looks more like everything else in onboarding?
3.2 The popup/panel wording should be "Participate in research" with Yes and No buttons
3.3 In the Profile,
3.3.1 Can we have a check box for "Participate in research" which is checked based on the user state.
3.3.2 If 3.3.1 is checked then another entry called "Research questionnaire"
3.3.3 If it was checked and then becomes unchecked, a popup to say "You have decided to no longer participate in research and will clear all my research questionnaire information. Do you want to stop participating?" Yes/No
3.4 In the questionnaire, the question on ethnic towards the end should allow for more than one entry to be checked, like I am arabic and irish (ha ha)


why it's showing me wrong?

according to video saw it mustn't show 10 and 30,i did it once and it works but now it doesn't, what should i do?
i think i must know how does it work in behind the scenes
prices = [10,20,30]
total = 0
for price in prices:
total += price
print(f'total: {total}')
it showed me 10 30 60
it mustn't show me 10 and 30

prices = [10,20,30]
total = 0
for price in prices:
total += price   #Your Problem is in simple index problem
print(f'total: {total}')
Now when you run it doesn't  show the error because due to indexing problem now you see first your total price is 0 at start
when price element comes 10 it get added to total which is 0 then it get 10. after price value change to 20 and total value is 10 we get new total value as 30 then 60

what do you mean indexing problem?

your print command was inside the for loop which leads to answer 10 and 30
I have given space but it is not showing I will add the program in above so that you can understand

Do you understand the answer .

Do you understand the answer .

Yes yes,thank you:)


fix: update version constraints for TPG 5 compatibility

The version must be set to < 6.0. Using < 5.0.0 as the constraint would lead to compatibility issues with version 5.1.0.

Hi @apeabody,
I've completed the changes and would appreciate if you could take some time to review my pull request. I believe your feedback will be invaluable, and I want to ensure everything aligns with our code standards before proceeding with the merge.
Please let me know if you have any initial thoughts or if there's anything specific you'd like me to address.
Thanks in advance for your time and expertise!
Best regards,
Felipe

Let's also use fix: for the PR subject so this will trigger a release and be included in the release notes.

Let's also use fix: for the PR subject so this will trigger a release and be included in the release notes.

Thank you, @apeabody, for the detailed explanation and suggested changes. I've now applied them to the code. Please take a look when you have a moment.

Thanks for the contribution @fveas!
Can you also remove the 4.0 version constraint (all of line 21) in the example so the CI test won't be version constrained: https://github.com/GoogleCloudPlatform/terraform-google-secret-manager/blob/main/examples/simple/versions.tf#L21
Let's also wider the google and google-beta version constraints in https://github.com/GoogleCloudPlatform/terraform-google-secret-manager/blob/main/test/setup/versions.tf#L22 from ~> to >= so they can run with TPG 5 as well.
Thanks!

Hi @fveas - Did you have an opportunity to make these changes as well?  Thanks!

Thanks for the contribution @fveas!
Can you also remove the 4.0 version constraint (all of line 21) in the example so the CI test won't be version constrained: https://github.com/GoogleCloudPlatform/terraform-google-secret-manager/blob/main/examples/simple/versions.tf#L21
Let's also wider the google and google-beta version constraints in https://github.com/GoogleCloudPlatform/terraform-google-secret-manager/blob/main/test/setup/versions.tf#L22 from ~> to >= so they can run with TPG 5 as well.
Thanks!

Hi @fveas - Did you have an opportunity to make these changes as well? Thanks!

Hi @apeabody, I apologize for the oversight; I misunderstood the initial request. I've now made the requested changes. Can you please review them and let me know if everything looks good? Thanks for your patience and guidance.

Hi @apeabody, I apologize for the oversight; I misunderstood the initial request. I've now made the requested changes. For context, this is my first contribution on GitHub, so I'm still getting familiar with the process. Can you please review the changes and let me know if everything looks good? Thank you for your patience and guidance.

Hi @fveas - No worries at all. I'm only seeing a single file currently changed in this pull request, perhaps they didn't get included in you commit, etc?  You can see all the changes in the PR here: https://github.com/GoogleCloudPlatform/terraform-google-secret-manager/pull/34/files Thanks!

/gcbrun

Breaking change to secret_manager_secret in https://github.com/hashicorp/terraform-provider-google/releases/tag/v5.0.0

Breaking change to secret_manager_secret in https://github.com/hashicorp/terraform-provider-google/releases/tag/v5.0.0
secretmanager: removed automatic field in google_secret_manager_secret resource (https://github.com/hashicorp/terraform-provider-google/pull/15859)

So, this change will actually need to be a breaking change fix(TPG>=5)!: and incorporate the above breaking change in the provider secret_manager_secret resource.


[Q] Plugin "analyze" max bitrate of service ...

Can the analyzer plugin extract the maximum bitrate of a service ?
By maximum I mean the maximum bitrate since the start of the analysis.
Thanks very much.

This information is not available. All reported bitrates are average bitrates for the duration of the analysis.
Note that "the maximum bitrate since the start of the analysis" is a bit ambiguous since it suggests that a bitrate is an instantaneous value. A bitrate is always over a certain amount of time (bits / duration). Assuming that we split the analysis time in slices of 1 second, 100 ms, 10 ms (your choice), you would like to get the maximum value of all bitrates over all these slices. It would then require an additional parameter, the pseudo-instantaneous slice duration.


Integration not functional: TypeError: 'HassLogger' object is not callable

Version of the custom_component
0.0.15
Configuration
Using the integration... no yaml
Describe the bug
Integration loaded at boot but after some time, I am getting an error and no way to have the integration up and running...
Debug log

2023-06-15 22:23:23.503 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry <email_removed> for porscheconnect
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 387, in async_setup
    result = await component.async_setup_entry(hass, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/porscheconnect/__init__.py", line 94, in async_setup_entry
    access_tokens = await controller.getAllTokens()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pyporscheconnectapi/client.py", line 32, in getAllTokens
    return await self._connection.getAllTokens()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pyporscheconnectapi/connection.py", line 195, in getAllTokens
    token = await self._requestToken(application)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pyporscheconnectapi/connection.py", line 202, in _requestToken
    await self._login()
  File "/usr/local/lib/python3.11/site-packages/pyporscheconnectapi/connection.py", line 118, in _login
    _LOGGER("We already have a code in session, skip login")
TypeError: 'HassLogger' object is not callable

Same issue here with 0.0.15

same issue for me.
Home Assistant Core
Error setting up<EMAIL_ADDRESS>for porscheconnect
07:56:20 – (ERROR) Porsche Connect (custom integration)
Unexpected error fetching porscheconnect data: 'HassLogger' object is not callable
07:51:20 – (ERROR) Porsche Connect (custom integration) - message first occurred at 15 June 2023 at 23:08:13 and shows up 98 times
Error requesting porscheconnect data: Cannot connect to host api.porsche.com:443 ssl:default [Try again]
15 June 2023 at 22:48:06 – (ERROR) Porsche Connect (custom integration) - message first occurred at 15 June 2023 at 22:27:47 and shows up 2 times
Waiting on integrations to complete setup: onvif, device_tracker.porscheconnect
15 June 2023 at 22:07:32 – (WARNING) bootstrap.py

I restarted HA this evening on one instance out of two and everything went fine for the integration...  I can even reload the integration without error...
but on the other hand, the same integration for the same account/car on the other instance (same HA version, OS and so on) is still giving me an error...  no way to reload it...  So I decided to restart this instance as well, and the integration went up also without problem...  So it seems that something is happening to have this error message and no way to reload the integration without restarting HA...  I will keep one of my instance in debug mode for this integration to hopefully trap the root cause of this issue...

Here is the result of the debug.... Seems to work ok and then for any reason the hasslogger issue is appearing every 5 minutes...
 2023-06-17 00:04:47.882 DEBUG (MainThread) [custom_components.porscheconnect] Finished fetching porscheconnect data in 1.740 seconds (success: True) 2023-06-17 00:09:48.890 DEBUG (MainThread) [custom_components.porscheconnect] Finished fetching porscheconnect data in 1.748 seconds (success: True) 2023-06-17 00:14:50.022 DEBUG (MainThread) [custom_components.porscheconnect] Finished fetching porscheconnect data in 1.879 seconds (success: True) 2023-06-17 00:19:52.032 DEBUG (MainThread) [custom_components.porscheconnect] Finished fetching porscheconnect data in 1.890 seconds (success: True) 2023-06-17 00:24:53.944 DEBUG (MainThread) [custom_components.porscheconnect] Finished fetching porscheconnect data in 1.802 seconds (success: True) 2023-06-17 00:29:53.455 ERROR (MainThread) [custom_components.porscheconnect] Unexpected error fetching porscheconnect data: 'HassLogger' object is not callable Traceback (most recent call last):   File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 283, in _async_refresh     self.data = await self._async_update_data()                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/config/custom_components/porscheconnect/__init__.py", line 256, in _async_update_data     vdata = await self._update_data_for_vehicle(vehicle)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/config/custom_components/porscheconnect/__init__.py", line 146, in _update_data_for_vehicle     services = await self.controller.getServices(vin)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/usr/local/lib/python3.11/site-packages/pyporscheconnectapi/client.py", line 484, in getServices     data = await self._connection.get(            ^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/usr/local/lib/python3.11/site-packages/pyporscheconnectapi/connection.py", line 233, in get     headers = await self._createhead(application)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/usr/local/lib/python3.11/site-packages/pyporscheconnectapi/connection.py", line 287, in _createhead     token = await self._requestToken(             ^^^^^^^^^^^^^^^^^^^^^^^^^   File "/usr/local/lib/python3.11/site-packages/pyporscheconnectapi/connection.py", line 202, in _requestToken     await self._login()   File "/usr/local/lib/python3.11/site-packages/pyporscheconnectapi/connection.py", line 118, in _login     _LOGGER("We already have a code in session, skip login") TypeError: 'HassLogger' object is not callable 2023-06-17 00:29:53.463 DEBUG (MainThread) [custom_components.porscheconnect] Finished fetching porscheconnect data in 0.321 seconds (success: False) 2023-06-17 00:34:53.406 ERROR (MainThread) [custom_components.porscheconnect] Unexpected error fetching porscheconnect data: 'HassLogger' object is not callable Traceback (most recent call last):   File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 283, in _async_refresh     self.data = await self._async_update_data()                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/config/custom_components/porscheconnect/__init__.py", line 256, in _async_update_data     vdata = await self._update_data_for_vehicle(vehicle)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/config/custom_components/porscheconnect/__init__.py", line 146, in _update_data_for_vehicle     services = await self.controller.getServices(vin)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/usr/local/lib/python3.11/site-packages/pyporscheconnectapi/client.py", line 484, in getServices     data = await self._connection.get(            ^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/usr/local/lib/python3.11/site-packages/pyporscheconnectapi/connection.py", line 233, in get     headers = await self._createhead(application)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/usr/local/lib/python3.11/site-packages/pyporscheconnectapi/connection.py", line 287, in _createhead     token = await self._requestToken(             ^^^^^^^^^^^^^^^^^^^^^^^^^   File "/usr/local/lib/python3.11/site-packages/pyporscheconnectapi/connection.py", line 202, in _requestToken     await self._login()   File "/usr/local/lib/python3.11/site-packages/pyporscheconnectapi/connection.py", line 118, in _login     _LOGGER("We already have a code in session, skip login") TypeError: 'HassLogger' object is not callable

The root cause for this is that the auth backend sometimes presents a captcha, we are trying to figure out how to get around that, might be problematic though.

This error - when happens during inital setup - looks to block the creation of the devices (at least I assume why no devices have been created).  Should I just keep reloading the integration to try to get it to succeed?  I wonder if there is a way to route the HTML provided for CAPTCHA to the HA UI for the user to interact with?

Thinking loudly .....
@CJNE & HA Porsche fans,
Not sure if you or someone have reached out to Porsche directly asking them to make some sort of change on their endpoint to allow our HA to connect on a more IoT fashion way rather than on a normal end user one.
Rationale would be that HA is a community (according to HA analytics) of more than 250k people around the globe, with many of them being Porsche's clients. Unfortunately we cannot see how many of us have the custom component installed.
Additionally we could (each of us) email Porsche to a given centralized email address asking for that IoT style connectivity feature to be enabled. At the end of the day, more and more brands are enabling this type of IoT oriented type of connections.
As I said, thinking loudly
Thanks 👍

I pulled the code down for this client to see if I could help diagnose the issue and although I've not got a solution, I might have made a change that has made my integration work at least a little better (its been connected for an hour which it hadn't done before).
I modified locally the custom_components/porscheconnect/__init__.py file and changed the country and locale to gb in case the reason it was kicking out to capcha because the country didn't match the query location... oddly after changing these, I've managed to connect.
Like I say, not a solution, but maybe interesting... will update if it remains connected or if it disconnects and gets into the failure state again

I just tried the same thing then tried AU for Australia which is where I reside on the off chance that Porsche geo block or something.  No dice - can't even get the integration to load still

I wonder if its the same underlying issue as described here - https://github.com/evcc-io/evcc/issues/9274
No solution in sight, but could be related to refreshing the access token and their API perhaps changing recently

Between bits of work I’ve been looking at this issue and couldn’t make sense of why it would run reliably all day from a terminal but not when inside HASS… today, while looking around through the packages this component is built from I worked it out… and was the 2nd person to do so!
This library builds on top of @CJNE s other library https://github.com/CJNE/pyporscheconnectapi which has recently (2 versions ago) had a fix for a typo in the line of code that is crashing here, there is currently an MR waiting to bump this version up but until that happens you can manually fix your instance by upping the version of that library to be 0.1.5 on this line https://github.com/CJNE/ha-porscheconnect/blob/main/custom_components/porscheconnect/manifest.json#L10
I’ve been getting updates for over two hours now… but hopefully this is now as stable as the local terminal-running version.

@tonycoupland Thank you for your fix...  I implemented it yesterday night and 9 hours later, the integration is still up and running... This seems to have fixed the issue...  Thanks !

@tonycoupland , thanks also for the proposal, I made the change and is working fine now for more than 24h

Sorry for the lack of updates, the PR has been merged and a new release is out.


UR CODE WAS COPIED FOR SELL NOW

http://manaai.cn/aicodes_detail3.html?id=25
A group of contemptible thieves has clone ur code, replace ur license with theirs and peddling it on their "WEBSITE".

they even charge 10rmb for downloading the code@ruinmessi


commit.sync is not a function when using Vuex Modules

In Vuex MyModule it fails:
myAction({ commit }) {
    return commit.sync({type: MY_ACTION}); // this fails, commit.sync is not a function
}

Workaround:
Inside RootModule: add your mutation and call the mutation from your sub Module
mutations: {
        [MY_ACTION]: () => {
          this.store.commit('myModule/' + MY_ACTION)
    }
},

Inside MyModule, call the mutation from the Root Module, this will trigger Logux Sync
import Store from "../../store";

myAction({ commit }) {
    return Store.store.commit.sync({type: MY_ACTION});
}

ps: other things might be needed depending on your use case, the main idea is that you move things to the Root Module and delegate back to the ones in your sub Module, until this issue is fixed.

@cosminv6u I just published fix in v0.8.0-beta.1. Can you try to check it out?
npm i @logux/vuex@next

Seems to work fine. I see the Action name is now composed of Namespace/MutationName.

Seems to work fine. I see the Action name is now composed of Namespace/MutationName.

Logux improves every day. Thanks for your help.
I'll publish the release this weekends.


feat(quantic): standalone search box

Related to SFINT-4014.
Added a quantic standalone searchbox.
Added an example standalone searchbox component to test it.
Modified the searchbox and standalone searchbox to use a lightning-button-icon.

Tested locally ✔
Tested in a build-your-own community ✔
Tested in Napili wrapped in an Aura component ✔

https://user-images.githubusercontent.com/2488155/129917444-ec9cf224-4f5d-4bcf-b125-104881c83fa8.mp4


Initial fixes for dimmer_rf

Removed the lastbinary and lastbrightness because the weren't being used.
Added isReceivedValue to denote whether we got this from serial (RF) or if it was set by the ESP.  This keeps us from setting a value if it was already set by RF.
Changed int to float for brightness so when dividing it we also get a float.
Changed division and multiplication to be by 100

Thanks for taking a look at this for me, I appreciate the contribution 👍


Draft: Remove sudo

sudo is not needed as this script is executed as root.

This PR can be ignored if PR #94 is implemented (there could be a conflict).

Not needed anymore.


Something is going wrong with nutrition edit

What

Something is going wrong with nutrition edit
Could it be async upload ? Something else ?

Could be the async upload
Could be that the user has an old version of the app
Could be the fact that for some products ("wasa fibres" if I remember) the nutrition edit didn't work, like it was locked as already provided officially by the producer
Could be a problem with the switch between "100g" and "portion"
Now that we switched to "offline mode" it's harder to check if the changes are pushed to the server database
Of course we could work on the matter with the users, if they are kind enough to cooperate

Now that we switched to "offline mode" it's harder to check if the changes are pushed to the server database

I am not sure about this, but if we could use sentry to record the failed tasks, or at least log it if users are kind enough to share, we could know if there was a problem with the offline mode data syncing.

@AshAman999 Failed tasks are indeed a problem, as we update the data locally while not being able to update the server accordingly.
Bouncing on #3018: that would mean that we should be able to apply to local data what a background task is supposed to change for a product.
Instead of saying:

I apply that change locally, and I plan to apply the same change to the server whenever possible.

Maybe we should say:

I plan to apply that change to the server whenever possible, and meanwhile I apply all the pending changes to the local product.

The difference being that if for some reason we cannot update the server (let's say we try 5 times), we can decide to dismiss that specific change locally. Or not. A bit tricky, isn't it?

The difference being that if for some reason we cannot update the server (let's say we try 5 times), we can decide to dismiss that specific change locally. Or not. A bit tricky, isn't it?

@monsieurtanuki
think this is something similar to what Instagram or LinkedIn does, I think we could do a try catch there, if a task fails in catch statement we could do a fresh call to server and put that product in the localdatabae

we could do a fresh call to server and put that product in the localdatabae

And forget about this task? And in case the server is down, you dismiss all the tasks?

We do retry it like 5 times , first after 30 seconds, then after 5 mins, then 30 mins, then an hour, then 3 increasing exponentially

Then my suggestion of preventing the user refresh while there are pending tasks is not compatible, right?
Btw I'm not convinced with the "only 5 times": when number 5 failed, what happens with the local data?
I'm not convinced either with the 3h delay: potential source of data collision.
Not easy indeed.

cc @anshpathania7

@stephanegigandet says "spotted an issue with the nutrition edit, maybe a known bug : if i switch to serving, it removes all values, except the energy in kcal"

We need to ask the user whether it's fixed


op

Operating system and version:
nvm debug output:

nvm ls output:

How did you install nvm?

What steps did you perform?
What happened?
What did you expect to happen?
Is there anything in any of your profile files that modifies the PATH?

If you are having installation issues, or getting "N/A", what does curl -I --compressed -v https://nodejs.org/dist/ print out?

I am so sorry. I clicked on it by mistake. Please forgive me.


translation i18n fixes

I add the eu translation and also the fix for running the i18n script, which should use the @plone/scripts dependency.

Thanks!


Update technical-limitations.md to include max group count

https://jira.cfops.it/browse/AUTH-3489

Howdy @kennyj42, it looks like we have an updated value of 300 for group count per org.
Does that mean we can close out this PR?

Yea, we can delete this PR


Update daily limit from 500 to 1000 in the notification

Context: https://valora-app.slack.com/archives/C0134SBQTQR/p1623879963023400

I think this was already done here: https://github.com/valora-inc/wallet/commit/31c44cfc2429046ce9bb11f28936303457c8b257
I tested locally, and it's using 1000 as limit
@jeanregisser I think we can close this

Hey @aleksey-nov  , We have verified this issue using latest Test flight build V 1.19.0 and Android internal build V1.19.0 & updated status in Gsuite.
GSuite Link: https://docs.google.com/spreadsheets/d/1bWi6smdTMClreG96sGjYxb2lee9Ja6eT4-Lm7eXm1gM/edit#gid=1462650076
Verified On Devices: OnePlus 7t (11.0) , iPhone 6+ (12.5.4), Redmi note 8 (10.0), iPhone 12 (14.7.1)
CC: @jeanregisser

https://valora.testquality.com requirement R82.

Hey @MuckT We have verified this issue using latest Test flight build V 1.19.0 and Android internal build V1.19.0 & updated status in Test Quality Tool
Test Quality Tool link:
Android (9.0):

https://valora.testquality.com/project/12128/plan/21816/run/150277/result/2485115

Verified On Devices: Redmi Note 8 (9.0)
CC: @dievazqu


The IBM API Connect contain fail to download

Expected behavior
Actual behavior
Information about the Issue
Steps to reproduce the behavior

...
...

No information provided or steps to replicate - Closing 😞


Fix command description

"by scanning by comparing" -> "by scanning and comparing"

Thanks. Merged :)


编译环境相关参数

Describe the bug
A clear and concise description of what the bug is.
m1 mac 使用 node js v20.13.1或 yarn 1.22.22无法编译，需要提供编译相关环境参数
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Device information (please complete the following information):

OS Name: [e.g. Windows, macOS, Linux...]
OS Version: [e.g. 11, 10.14...]

Screenshots & Attachments
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.

修改后yarn install 报错。报错内容如下：
yarn install
yarn install v1.22.22
warning ../../../../package.json: No license field
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/5] 🔍  Validating package.json...
[2/5] 🔍  Resolving packages...
[3/5] 🚚  Fetching packages...
[4/5] 🔗  Linking dependencies...
warning " ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning "electron-builder ><EMAIL_ADDRESS>has unmet peer dependency<EMAIL_ADDRESS>[5/5] 🔨  Building fresh packages...
[1/5] ⢀ better-sqlite3
[-/5] ⢀ waiting...
[3/5] ⢀ esbuild
[-/5] ⢀ waiting...
error VutronMusic/node_modules/electron: Command failed.
Exit code: 1
Command: node install.js
Arguments:
Directory: VutronMusic/node_modules/electron
Output:
RequestError
at ClientRequest. (/VutronMusic/node_modules/got/dist/source/core/index.js:970:111)
at Object.onceWrapper (node:events:634:26)
at ClientRequest.emit (node:events:531:35)
at origin.emit (VutronMusic/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)
at TLSSocket.socketErrorListener (node:_http_client:500:9)
at TLSSocket.emit (node:events:519:28)
at emitErrorNT (node:internal/streams/destroy:169:8)
at emitErrorCloseNT (node:internal/streams/destroy:128:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)AggregateError [ETIMEDOUT]:

error VutronMusic/node_modules/electron: Command failed.：这表明在安装 Electron 时发生了错误。
RequestError 和 AggregateError [ETIMEDOUT]：这个错误通常表示网络请求超时，可能是由于网络连接不稳定或无法访问某个依赖项的服务器。
你看看是不是自己没有设置electron源

设置后就好了，这是我build出来的arm64的包，fork了一份放在release里了https://github.com/doubledeft/VutronMusic/releases

最后提一下，之前版本里的store/player.ts里还有听歌记录的功能，这个功能会导致封号的风险，现在最新版本暂时把这个功能屏蔽了，建议看看你打包的那个版本里有没有屏蔽哦。。

更新了，更新后的打包传上去了


Clarify that metadata has no namespace

Closes #1352
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Today I learned.


Enable using hugemem partial nodes

@johrstrom I noticed in another pr we set min-num-cores to 4, should this be the case for all hugemem or can we just remove this field?

I had to find out the minimums through brute force. There are different minimums for different node type + cluster combinations. Here's a table in case you don't see the slack message with the same

cluster
node_type
minimum

owens
hugemem
4

pitzer
largemem
24

pitzer
hugemem
20

Ok. I was wondering about largemem too. From what you just sent it looks like we should be making changes to those as well.


ODE Benchmark Error

Looks like we may have introduced some bugs. The ODE benchmarks do not run under Pycket:
can't find module #%foreign for '_int8
  while executing: _int8
  in module: unsafe

This looks like a change in racket to me.
On Sat, Feb 21, 2015, 6:38 PM Spenser Bauman<EMAIL_ADDRESS>wrote:

Looks like we may have introduced some bugs. The ODE benchmarks do not run
under Pycket:
can't find module #%foreign for '_int8
while executing: _int8
in module: unsafe
—
Reply to this email directly or view it on GitHub
https://github.com/samth/pycket/issues/156.

I saw this error as early as December or January iirc.
It define natively came with a Racket upgrade to (then) <IP_ADDRESS>
--
Tobias Pape
sent from a mobile device

Am 22.02.2015 um 01:45 schrieb Sam Tobin-Hochstadt<EMAIL_ADDRESS>This looks like a change in racket to me.
On Sat, Feb 21, 2015, 6:38 PM Spenser Bauman<EMAIL_ADDRESS>wrote:

Looks like we may have introduced some bugs. The ODE benchmarks do not run
under Pycket:
can't find module #%foreign for '_int8
while executing: _int8
in module: unsafe
—
Reply to this email directly or view it on GitHub
https://github.com/samth/pycket/issues/156.

—
Reply to this email directly or view it on GitHub.

What exact command line (preferably for pycket, not rebench) produces this error?

@samth You can see this error by running pycket-c ode.rkt without any additional arguments.

This seems to have been a case of dated JSON files.

@krono If this is still an issue for you, let me know and I will reopen, but please try nuking your JSON files first.

It's ok, just wanted to report that I encounered that a while ago


New version: GeometryBasics v0.1.2

Registering package: GeometryBasics
Repository: https://github.com/SimonDanisch/GeometryBasics.jl
Created by: @SimonDanisch
Version: v0.1.2
Commit: f418f48c685c66288074cd4414159b05e0d0a838
Reviewed by: @SimonDanisch
Reference: https://github.com/SimonDanisch/GeometryBasics.jl/commit/f418f48c685c66288074cd4414159b05e0d0a838#commitcomment-33747558

TagBot tag

welp, i'll try again in a few.

TagBot tag

Cloning into '/tmp/975824888'...
remote: Invalid username or password.
fatal: Authentication failed for<EMAIL_ADDRESS>Creating release: Creating tag: git push: exit status 128

Huh, that's a new one. I tried cloning with that URL and it worked fine...

one more time before I give up: TagBot tag


Docs Support

Add Docs and Readme

Should we remove test run for nightly then given it's dependent on CxxWrap?

That sounds fine to me. I am wondering if testing nightly may be something we should all do less of; making compensatory changes in package code might obscure efforts to detect failures in backwards compatibility for Julia itself.

Can we review it and if it looks okay, merge it? I don't have write access

I don't have write access

Just gave you the maintain permission!


fix: load offer performance

:tada: This PR is included in version 0.15.3 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


switch to reusing workflows

Implemented this comment suggestion: using reusing workflow instead of calling another github-action.

This still doesn't solve the problem where a user isn't allowed to create the latest tag. See the workflow runs from any of the dependabot PRs (like this one for example).

I have added token here  maybe could solve the problem

I wasn't sure what that was about. Ok, let's try it.

Looks like we still might need to go with generated reusable workflows instead.
I don't fully understand the token business, but I feel like you were on the right track. Although, the secret token may be of more useful in the "retag-branch" step; previously the checkout action inherited the secrets.GITHUB_TOKEN and executed without error.

It's not a bug. Its a newer security precaution. Thanks for the research and explanation!


Add github_branch_exists()

Use it instead of get_branch_info() to avoid GitHub's API rate limiting.
Signed-off-by: Zack Cerza<EMAIL_ADDRESS>

Refer to this link for build results (access rights to CI server needed):
http://jenkins.ceph.com//job/teuthology-pull-requests/1508/
Tests passed for this pull request.


MicroSD card for PYNQ-Z2

Hi, my name is Vladimir.
Can you help me with small question connect with Xilinx Pynq-z2
I want to use it and bought MicroSDHC 16GB, but my windows or Ubuntu don't see such drive, only device but drive letter don't appear. Maybe you know the reason? Thanks

Hi Vladimir,
Could be a bad connection? Or write protection (there should be a tiny switch on the card)?
Not sure -- maybe try contacting the card vendor.


Doesnt work for sketch 45

It wasn't working for sketch 45 so i've rewritten it.
https://github.com/imranparvez/reLabel
I'm not a developer so the code might not be so great.

Hi -- please download my latest version and give it a try.  There were some reports of issues with version 45 but I addressed those in my latest update. Let me know if it's still not working for you.

Works fine on 45.2 for me with latest version of the plugin

Sorry yeah, i downloaded again to check it. It works. My bad.

I just downloaded the latest and it isn't working with symbols. It works with normal grouped elements.

Hi Tom, please download the Guidelines file and see if those symbols work (they work for me with Sketch 45.2).  It may be something about how your symbols are constructed.
https://github.com/kenmoore/sketch-relabel-button/raw/master/Relabel Button Guidelines.sketch

@kenmoore thanks for the follow-up. My button is working now. I followed the instructions for positioning text elements in symbols found in the Guidelines file - this fixed my problem. Thanks!

Super, glad you got it working!

Thanks!
On Thu, Aug 3, 2017 at 10:58 AM Ken Moore<EMAIL_ADDRESS>wrote:

Super, glad you got it working!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kenmoore/sketch-relabel-button/issues/35#issuecomment-320029125,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AASus-5oIM_kS186sXQ89vO92IvIom3Rks5sUfxGgaJpZM4OlTPn
.


publish@ci: fix npm publish

What is the current behavior?

What is the new behavior?

Why did you make this change?

Checklist

[ ] I'm not PR to the main branch.
[ ] Commit message was checked by commitlint.

Additional information

Codecov Report

Merging #69 (ededc76) into main (675cfb9) will not change coverage.
The diff coverage is n/a.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@           Coverage Diff           @@
##             main      #69   +/-   ##
=======================================
  Coverage   41.31%   41.31%
=======================================
  Files         113      113
  Lines        9638     9638
  Branches     1476     1476
=======================================
  Hits         3982     3982
  Misses       5226     5226
  Partials      430      430

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Resize, redraw, and disabling features

Current behaviour

CropService does not change size when adornedElement changes, it stays in same size and place
It's not possible to disable cropping

Changes

Redraw method in ShadeTool sets shading size to match with canvas
Added Redraw method in CropService to be able to redraw it with new size and position
Added SetEnabled method in CropService to be able to enabled and disable

CropToolControl changes

Image Stretch have changed to uniform
Added SizeChanged event for SourceImage to redraw CropService

thank you , i was looking for that , it's working great.


java.lang.NoSuchMethodError: 'void net.kyrptonaught.customportalapi.api.CustomPortalBuilder.registerPortal()'

Several users have reported this error that crashes their games even before the title screen.
https://mclo.gs/tU2XhJd#L2014

You can find our mod's code here.


relation with lighthouse ?

Can you please clarify how is this related to lighthouse, is it replacement / wrapper , why one should use this tool instead of just lighthouse ?
Thanks,
Vikram

Hi Vikram.
PWMetrics is a tool/wrapper for Lighthouse. It runs Lighthouse using chrome-launcher -> gets metrics results due to perf-config and either take a look at docs or read article we wrote to check all cool features we have.

Thanks @denar90 let me go through those and get back in case of further doubts.
Regards,
Vikram


streamalert - rules - duo - bypass codes

to: @ryandeivert
cc: @airbnb/streamalert-maintainers
size: small
Background

3 new rules for DUO admin logs that are centered around the creation of bypass codes.

Changes

3 new rules
Associated tests

Testing
python manage.py lambda test --processor all

StreamAlertCLI [INFO]: (31/31) Successful Tests
StreamAlertCLI [INFO]: (60/60) Alert Tests Passed
StreamAlertCLI [INFO]: Completed

Coverage remained the same at 94.715% when pulling d834436235c4980b3d22b3e2c439e8f025cf90bd on mimeframe-streamalert-rule-duo-bypass into 56f08a353e5cc0411bd75e6405f8747bd2183213 on master.

Coverage remained the same at 94.715% when pulling 3f003a6a101c59c8254a5173176ca3d28c1478b3 on mimeframe-streamalert-rule-duo-bypass into 56f08a353e5cc0411bd75e6405f8747bd2183213 on master.


"core" in distribution package names in .NET 5

The current distribution packaging guide includes packages with the "core" suffix: "aspnetcore-runtime-X.Y" and "aspnetcore-targeting-pack-X.Y". With the rename from ".NET Core" to ".NET" in the 5.0 timeframe, should these docs be updated to drop the "core"? The package names would become just "aspnet-runtime-X.Y" and "aspnet-targeting-pack-X.Y"?
Target framework
Check the .NET target framework(s) being used, and include the version number(s).

[x] .NET Core
[x] .NET 5
[ ] .NET Framework
[ ] .NET Standard

cc @tmds @dagood

As of 5.0 preview 7, the Microsoft deb/rpm aspnetcore packages still have "core" in the name:
aspnetcore-runtime-5.0.0-preview.7.20365.19-x64.deb
aspnetcore-targeting-pack-5.0.0-preview.7.20365.19.deb

It's also still called ASP.NET Core on the download site:

Included runtimes
.NET Runtime 5.0.0-preview.7.20364.11
ASP.NET Core Runtime 5.0.0-preview.7.20364.11
Desktop Runtime 5.0.0-preview.7.20364.11

@leecow @nakarnam @wtgodbe @NikolaMilosavljevic do you know if this is going to change for 5.0?
I'm not able to find any discussion about ASP.NET Core vs. the ".NET Core" => ".NET" rename.

There's no plan to change asp.net core to asp.net.
cc @glennc @DamianEdwards @LadyNaggaga

Thanks, that resolves my concerns.


code-gen: needs to support custom find_operation

Issue: aws-controllers-k8s/community#1827
Description of changes:
This change allow to implement custom find function. The name of the function could be written in generator.yaml as 'find_operation: ' similar to update_operation. And generated code for sdkFind() would call this custom find function. The find function itself could be written in hooks.go.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

/ok-to-test

/retest
/assign @jljaco @RedbackThomson


php配置和调试问题

php不能输出，只能显示终端

感谢您创建此问题！
然而，由于您没有使用英语，我们很难进行处理。如果可能，请您将此问题描述修改为英文。请您直接编辑此问题，不要提交回复或新问题。
请注意，机器翻译服务通常无法处理用于讨论软件问题所需的技术性语言，所以我们推荐您寻找一位熟悉相关技术的人来协助翻译。
如果您无法做到，没有关系。此问题已经被标记为需要翻译，等待社区成员志愿进行翻译。

Thanks for creating this issue!
Unfortunately, as the description is not in English, it's hard for us to work on. If possible, please edit the original issue to be in English.
In our experience, automatic translation services are not able to handle the technical language needed to effectively communicate about software issues, so we recommend finding a human translator who is familiar with our technology to help.
Otherwise, we have marked this as needing translation and a community member may be able to help out.

php不能输出，只能显示终端

我看你截图里面不是输出了hello world了吗

这是我朋友的页面，我想设置成这样的但是我不知道怎么做

Sorry, I don't understand the problem.

Sorry, I don't understand the problem.

After all, I don't think this issue is strongly relevant with vscode.


Cannot import FastifyPluginAsyncJsonSchemaToTs

Prerequisites

[X] I have written a descriptive issue title
[X] I have searched existing issues to ensure the bug has not already been reported

Fastify version
4.2.0
Plugin version
2.1.0
Node.js version
18.4.0
Operating system
Windows
Operating system version (i.e. 20.04, 11.3, 10)
21H2
Description
The FastifyPluginAsyncJsonSchemaToTs and FastifyPluginCallbackJsonSchemaToTs types added in v2.1.0 are missing in dist/index.d.ts file.
Steps to Reproduce

pnpm i @fastify/type-provider-json-schema-to-ts

import type { FastifyPluginAsyncJsonSchemaToTs } from '@fastify/type-provider-json-schema-to-ts'

=> Module '"@fastify/type-provider-json-schema-to-ts"' has no exported member 'FastifyPluginAsyncJsonSchemaToTs'.
Expected Behavior
No response

cc @RafaelGSS


Redundant warnings and potential confusion regarding updating repository cache for CE/EE edition

File: engine/installation/linux/ubuntu.md, CC @mstanleyjones
From: https://docs.docker.com/engine/installation/linux/ubuntu/#install-using-the-repository

Going to piggy back here to suggest the section below be changed to be "tab-sensitive".  Since the user is already tabbing to get Docker CE/EE specific instructions, instruction number 3 shouldn't require the user to remember to substitute docker-ce with docker-ee.

And right below instruction #3 is another tab-selection opportunity.  It'd be ideal to just carry the first tab selection all the way through...

The double warnings are to do with a big bug introduced by some changes to the CSS around admonitions. The problem exists all over the place and I'm not sure what to do about it.
Regarding the tabs, I've got a POC up that actually separates out the CE and EE installation docs, which should mitigate some of this overload. Your thoughts would be appreciated over there at #3350.


[App] Bug: scalar chart not finding logged values

Describe the bug
I'm trying to create a scalar chart from a logged value.

However, some scalar charts (e.g., the middle one from the top) seem not to find the corresponding values.

But the value is clearly logged, and visible in a simple scatter plot.

Expected behavior
I would expect to see the value of the logged value, not an error.
Operating System

OS: [e.g. iOS] MacOS
Browser [e.g. chrome, safari] Microsoft Edge
Version [e.g. 22]

wandb 0.12.9
logs created from the wandb integration with pytorch-lightening 1.5.5
python 3.7.9

Hi there @andreimargeloiu! Sorry about this, we'll look into it straight away. Can you link us the workspace where you're seeing this issue?

Run: https://wandb.ai/andreimargeloiu/low-data/runs/wvkt6mvz?workspace=user-andreimargeloiu

Here I'm detailing my use case and asking how to use wandb for it.
I'm training a model in my codebase and choosing the best model based on the validation loss. However, in the wandb UI I want to see the accuracy of the best model (note that the accuracy of the best model chosen on the validation loss can be smaller than the model with the best accuracy).
I can't simply use wandb.define_metric(my_metric, summary='max'), because, commonly, the model with the best validation loss doesn't necessarily have the best validation accuracy.
Current solution: After training, compute the accuracy of the best model, and save using wandb.log, giving a chart with only one data point saved. Then, I create a scalar chart that simply shows the value of
Can you suggest how to simplify this workflow?

Hi there @andreimargeloiu! Sorry for the late response; just got caught up post-holidays.
Based on your workflow, I think you can do the following:
For every run, do
wandb.define_metric('validation_loss', summary='min')
wandb.define_metric('validation_accuracy', summary='max')

This configuration will tell W&B to keep track of the minimum validation_loss as well as the maximum validation_accuracy for every single model you're training. Then, in your project workspace (the one with the large table with all your training runs), you can sort your models by the lowest validation_loss.min and you will be able to see that model's validation_accuracy.max as well.
I may have misunderstood what you're trying to do, so please let me know if this doesn't solve your use case. Please also let me know if this explanation is unclear in any way; I can elaborate further.
I also looked into the issue that's causing your scalar chart to not display your bestmodel_train/ metrics. This looks like it could be a backend issue on our side. Would you be able to share your full training script(s) that you used for this project? That would be immensely useful for our debugging.
Thanks in advance for your help, and sorry again for dropping off! I should have gotten back to you before the holidays.

@andreimargeloiu I'm currently investigating the issue and would like to inquire: Did you use wandb.Api() to update your run while it was running? If that's the case, we have a known issue that is likely to cause the problems you're experiencing.

@laxels, thank you for trying to help out.

Regarding my use-case: I don't think your solution would actually work... In pseudo-code, I'm trying to:

validation_accuracy = [array with validation accuracy at every epoch]
validation_loss = [array with validation loss at every epoch]
print(validation_accuracy[np.argmin(validation_loss)])

For example, I want:
validation_loss = [0.5, 0.4, 0.45]
validation_accuracy = [0.7, 0.8, 0.9]

print(validation_accuracy[np.argmin(validation_loss)])  # save 0.7 in wandb

Regarding issue: I am logging everything using pytorch-lightening function self.log

Specifically, my model has:

an attribute self.log_test_key, which is the wandb metric key I want to log
call self.log(f'{self.log_test_key}/balanced_accuracy', balanced_accuracy_score(y_true, y_pred)) from within def validation_step()

To compute these metrics and log to wandb I do:
model.log_test_key = 'bestmodel_valid'
trainer.test(model, dataloaders=data_module.val_dataloader(), ckpt_path=checkpoint_callback.best_model_path)

@laxels, thank you for trying to help out.

Regarding my use-case: I don't think your solution would actually work... In pseudo-code, I'm trying to:

validation_accuracy = [array with validation accuracy at every epoch]
validation_loss = [array with validation loss at every epoch]
print(validation_accuracy[np.argmin(validation_loss)])

For example, I want:
validation_loss = [0.5, 0.4, 0.45]
validation_accuracy = [0.7, 0.8, 0.9]

print(validation_accuracy[np.argmin(validation_loss)])  # save 0.8 in wandb

I see. So you want the accuracy for the step with the lowest loss. Unfortunately, we don't have a simpler solution for that use case at the moment.
We're looking into what's causing your values to not be logged properly. Would it be possible for you to share your full training script (with potentially sensitive data removed)? That would be immensely helpful for reproducing the bug on our end.

Sorry, I don't have the script at the point in time. I don't seem to experience the bug anymore, but I'm sure the bug is still there.


[#2229] feat(Gravitino Python client): client module initialization

What changes were proposed in this pull request?
i made a python client module to get the version data from gravitino jetty server
TBD:

[ ] requirements setup needs to be decided (poetry vs. requirements.txt)
[ ] configure the dependent packages in this project (i haven't set the required packages in this module)
[ ] automation test (GitHub Action config) needs to be configure

Why are the changes needed?
for https://github.com/datastrato/gravitino/issues/2229
Does this PR introduce any user-facing change?
no
How was this patch tested?

in order to test this module, you need to install pytest, requests packages first
after the packages is installed, you can simply run pytest in any directory inside gravitino/

@xunliu i think before the PR can be merged, there are somethings needs to be done:

requirements setup needs to be decided (poetry vs. requirements.txt)
configure the dependent packages in this project (i haven't set the required packages in this module)
automation test (GitHub Action config) needs to be configure

Also, since there are several task dependent on this task, would you please kindly help me to deal with the GitHub Action config? Many Thanks

@Lanznx I think you can add a license header to fix the GitHub CI pipeline.
 Caused by: org.apache.rat.api.RatException: Apache Rat audit failure - 7 unapproved licenses
            	See file:///home/runner/work/gravitino/gravitino/build/reports/rat/index.html

@coolderli thanks for reminding, should i add the following license header in every new file manually?
"""
Copyright 2023 Datastrato Pvt Ltd.
This software is licensed under the Apache License version 2.
"""

since comment format is different between python and java, i'm not sure whether this would work for the test

Should be Copyright 2024 on new files.

Should be Copyright 2024 on new files.

ok, @justinmclean  thanks for reminding
however, after randomly checked several files, i found that the year were all 2023
perhaps we need a CI pipeline to update year automatically?
https://github.com/datastrato/gravitino/blob/main/dev/docker/gravitino/Dockerfile
https://github.com/datastrato/gravitino/blob/main/dev/docker/gravitino/gravitino-dependency.sh
https://github.com/datastrato/gravitino/blob/main/web/.editorconfig

Files created last year will have "copyrght 2023" in them; files created this year need "copyrght 2024" in them. The year should not be updated in an existing file.

I see, thanks for explaining!

@SophieTech88, @diqiu50  Please help review this PR, thanks!

requirements setup needs to be decided (poetry vs. requirements.txt)

Which way is more popular nowadays? I've used requirements.txt but have no idea about poetry.

requirements setup needs to be decided (poetry vs. requirements.txt)

Which way is more popular nowadays? I've used requirements.txt but have no idea about poetry.

@jerryshao
Here I found some articles about poverty:
<EMAIL_ADDRESS>https://medium.com/@nasriadzlani/how-to-create-library-and-publish-on-pypi-python-33a8c1d3bf0b
And it was used in iceberg-python: https://github.com/apache/iceberg-python
Poerty is very useful for managing a complex project. But I think clinet-python may not be so complex. Just my thought.

hi @Lanznx Thank you for your contributions.
I thank you need adding a README.md file in the clients/client-python/ directory. to explain:

What Python runtime environment needs to be installed?
How to run this test case?
Or other we need to know things.

@SophieTech88, @diqiu50 Please help review this PR, thanks!

Should we add a python document in the docs instead of README.md?

requirements setup needs to be decided (poetry vs. requirements.txt)

Which way is more popular nowadays? I've used requirements.txt but have no idea about poetry.

@jerryshao
Here I found some articles about poverty<EMAIL_ADDRESS>https://medium.com/@nasriadzlani/how-to-create-library-and-publish-on-pypi-python-33a8c1d3bf0b
And it was used in iceberg-python: https://github.com/apache/iceberg-python
Poerty is very useful for managing a complex project. But I think clinet-python may not be so complex. Just my thought.

I think requirements.txt is more common so a new user can easily know how to install the dependencies.

Hi Brandon, thank you for preparing this PR. Currently I'm working on https://github.com/datastrato/gravitino/issues/2418, which is trying to simplify the metalake usage in different clients. One change is, adding "metalake" in client's configuration (together with the server-url), so that the metalake name doesn't need appear in subsequent API calls like getting tables, views, etc. If user wants to switch a metalake, he may need initialize another client instance. Your idea and comment is welcomed!

Hi, I've finish the refactor and some patches, please help with the review 🙌

Do we need to setup CI environment for Python? Also, seems like pytest is another command to run the test, can we integrate pytest with gradle, so that we can use gradle to build/test the python client module?

Do we need to setup CI environment for Python? Also, seems like pytest is another command to run the test, can we integrate pytest with gradle, so that we can use gradle to build/test the python client module?

@jerryshao , this is a good suggstion. I think that is doable. Can we create another issue to track and do that a little bit later? Some guys are waiting for this PR to start, so I don't want them to wait.

Do we need to setup CI environment for Python? Also, seems like pytest is another command to run the test, can we integrate pytest with gradle, so that we can use gradle to build/test the python client module?

@jerryshao , this is a good suggstion. I think that is doable. Can we create another issue to track and do that a little bit later? Some guys are waiting for this PR to start, so I don't want them to wait.

OK, let's do it in a separate PR.

Do we need to setup CI environment for Python? Also, seems like pytest is another command to run the test, can we integrate pytest with gradle, so that we can use gradle to build/test the python client module?

@jerryshao , this is a good suggstion. I think that is doable. Can we create another issue to track and do that a little bit later? Some guys are waiting for this PR to start, so I don't want them to wait.

OK, let's do it in a separate PR. @Lanznx @shaofengshi can you please create an issue to track this?

Sure, subtask #2518 is created


[Fix #12699] Support searching for .rubocop.yml in compliance with dot-config

Resolves #12699
This PR supports searching for .rubocop.yml in compliance with dot-config.
dot-config prioritizes project-specific configurations over user-specific configurations, with a search order that first looks for the .rubocop.yml within the project's directories, in accordance with dot-config standards, before considering user-specific configurations.

Before submitting the PR make sure the following are checked:

[x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
[x] Wrote good commit messages.
[x] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Ran bundle exec rake default. It executes all tests and runs RuboCop on its own code.
[x] Added an entry (file) to the changelog folder named {change_type}_{change_description}.md if the new code introduces user-observable changes. See changelog entry format for details.

Great work!

Shouldn't this use .config/rubocop.yml rather than .config/.rubocop.yml? The dot-config page itself lacks examples, but looking at the repositories it points to (https://github.com/ansible/ansible-lint, https://github.com/ansible/molecule), the files in .config seem to all themselves not be hidden. Similarly, in ~/.config/rubocop/config.yml only ~/.config is hidden, and rubocop/ and config.yml are not.

@mvz I get your point, but to me it seems it's nicer to stick to a consistent name of the configuration file, so that users won't get confused if it varied by by location.  But perhaps we can allow for both .rubocop.yml and rubocop/config.yml (for consistency with XDG, which this is extending to projects).

:bulb: A little late to the party but wanted to mention that the Runcom gem provides all of this support with minimal hassle.


SonarQube project and configure Quality Gate

unable to create storage in shell.azure.com, unable to create the storage, can anyone help me.
https://azuredevopslabs.com/labs/vstsextend/sonarqube/

Hi @nkorvi ,
Sorry for the delay in response. You should have Contributor access to your Azure Subscription to create Storage.
Thanks


Unexpected key(s) in state_dict: "up1.model.0.weight",

Unexpected key(s) in state_dict: "up1.model.0.weight", "up1.model.0.bias", "up1.model.1.pa0.weight", "up1.model.1.pa0.bias", "up1.model.1.pa1.0.weight", "up1.model.1.pa1.0.bias", "up1.model.2.ca0.weight", "up1.model.2.ca0.bias", "up1.model.2.ca1.weight", "up1.model.2.ca1.bias", "up1.relu.weight"
how to deal with this

First of all, thanks for your attention! You can simply change "torch.load(smoother_path)" into "torch.load(smoother_path, strict=False)". up1 is only used as regularization during training, and thus has nothing to do with inference pass.

I will fix and test the code later, but the workaround above can just do the trick.


Leaked Connection warning from OkHttp3

Using Azure SDK for Java 1.0 from http://adxsnapshots.azurewebsites.net
We're seeing the following warning show up in our output log after moving to the Azure SDK 1.0:
2016-03-03 14:57:06.973  WARN 10848 --- [ ConnectionPool] okhttp3.OkHttpClient                     : A connection to https://management.azure.com/ was leaked. Did you forget to close a response body?
I've attached a file with the output captured after setting the HttpLoggingInterceptor.Level to BODY.
OkHttp3_output.txt

obsolete - we'll most likely be moving away from OkHttp in the near future


Allow isMobile to exist in global space in AMD environment

The code currently adheres to an AMD environment, but breaks if the user tries to use isMobile outside of AMD's define context. This fix allows that.

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Long axis title truncated on barplot in Maps

Long axis title was truncated on barplot in map visualization.
It should read "181 Community repeated measures"

Changed to only show the total count, entity type was removed. Looks good


changing language/locale for each HTTP request (express)

Hi,
I am a little confused on how to set language per request using express. Here is a test code I have written that takes a long long time to load. The documentation is not so clear about this.
// route handler
function( req, res) {
  var locale = req.locale;
  var i18n   = req.i18n;
  if( req.user ) {  // passport.js user
    res.write( "logged in"  );
    i18n.setLng( req.user.locale, { fixLng: true }, function( enUS ) {
      res.write( locale );
      res.write( i18n.lng());
      res.write( i18n.t( "key" ) );
      res.end();
    });
  } else {
      res.write( locale );
      res.end();
  }
}

I only see the text "logged in" after a long time of trying to load the page. Please let me know how to fix this or if its a bug.
Thanks.

As this thread is almost an year old, I hope someone raised a feature request to replace '-' with '_' or any other character in locale filenames. If it is already there, kindly point me to the required settings.
@jamuhl do you have any update on it ?

sorry for that...but no PR landed for that feature yet.