Add phase 1 implementation of new storage spec

What this PR does / why we need it:
The new modelmesh serving method needs a new storage definition to unify the model storage for both single and multi-model serving on different framework server. The Phase 1 is to show what the new storage definition looks like without breaking any regression test. In this phase 1 implementation, we initially has S3 storage, but will add more storage support when the modelmesh agent able to support other storage types.
Design docs: https://docs.google.com/document/d/1rYNh93XNMRp8YaR56m-_zK3bqpZhhGKgpVbLPPsi-Ow/edit#
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1814
Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Release note:

/cc @animeshsingh @yuzisun @njhill

Here's my proposal on how to standardize:
The ComponentInterface.GetStorageUri() func is currently used only:

As part of the determination of whether the InferenceService is a (non-modelmesh) MMS
To set the storageUri annotation on the corresponding component's kube resources, to be picked up by the storage initializer injector

For (1) I'd propose to leave existing behaviour as-is so as not to break backwards compatibility (don't change impl).
For (2) I think we should make a change to set an annotation containing the "resolved" storage spec in json form instead of the storage uri. In this case that only a storageUri is provided, this would be converted to a storage spec first.
This gets passed to the storage initializer on the command line, I would propose to instead just set it as an environment variable (the json contents). Maybe the storageUri command line arg could continue to be supported, but I presume that it it's not used as a "standalone" utility?
The storage-type specific env vars / volumes from existing secrets are separately added to the storage initializer / agent via the CredentialBuilder.CreateSecretVolumeAndEnv() func. I propose we change this function to instead populate/update the shared/common storage secret, and then always mount just that secret to the agent / storage initializer. So no environment variables are set for this (we could consider one env var which points to the mounted storage secret dir). Note that we would probably need for this common storage secret to be per- service account so may need to think about the naming (couldn't be a single well-defined name in that case).
storage.py then does not need to have any knowledge of storageUris and would work only with the provided storage spec. The "key" would be read from the spec or otherwise default based on the type, and then this key would be used to look up a corresponding entry (file) in the mounted storage secret dir. The combination of the provided spec and the contents of that storage entry would then be used to configure the appropriate storage client (rather than env variables).
@Tomcli @yuzisun WDYT?

Here's my proposal on how to standardize:
The ComponentInterface.GetStorageUri() func is currently used only:

As part of the determination of whether the InferenceService is a (non-modelmesh) MMS
To set the storageUri annotation on the corresponding component's kube resources, to be picked up by the storage initializer injector

For (1) I'd propose to leave existing behaviour as-is so as not to break backwards compatibility (don't change impl). For (2) I think we should make a change to set an annotation containing the "resolved" storage spec in json form instead of the storage uri. In this case that only a storageUri is provided, this would be converted to a storage spec first.
This gets passed to the storage initializer on the command line, I would propose to instead just set it as an environment variable (the json contents). Maybe the storageUri command line arg could continue to be supported, but I presume that it it's not used as a "standalone" utility?
The storage-type specific env vars / volumes from existing secrets are separately added to the storage initializer / agent via the CredentialBuilder.CreateSecretVolumeAndEnv() func. I propose we change this function to instead populate/update the shared/common storage secret, and then always mount just that secret to the agent / storage initializer. So no environment variables are set for this (we could consider one env var which points to the mounted storage secret dir). Note that we would probably need for this common storage secret to be per- service account so may need to think about the naming (couldn't be a single well-defined name in that case).
storage.py then does not need to have any knowledge of storageUris and would work only with the provided storage spec. The "key" would be read from the spec or otherwise default based on the type, and then this key would be used to look up a corresponding entry (file) in the mounted storage secret dir. The combination of the provided spec and the contents of that storage entry would then be used to configure the appropriate storage client (rather than env variables).
@Tomcli @yuzisun WDYT?

Thanks @njhill
For (1) we didn't change the behavior of how isvc determine MMS. I only changed the implementation logic when the model needed the storage initializer.
For (2) Let the controller to create a common storage secret per service account is a good approach. This can help KServe to sync the current s3/gcs/azure storage secrets into a common storage secret. We originally planned to do this in phases 2-3 because it changes the implementation and inputs for storage.py.
So for (2) I have two suggestions,
A) I can implement what you have suggested above and let the storage initializer always mount on the common secret to get the credentials. It can be implemented in a way that's seamlessly for Kubernetes users. However, developers who are testing the KServe python server on their local machine need to be aware of the parameter changes in the python storage class.
B) If we don't want to change the current storage.py implementation, I can move all the new storage spec credential loading logics to the controller side. This means the controller will read the common secret to extra the storage type, access key, and secret access key. Then map them as environment variables or Uri the same way as the current storage.py. This way, no changes need to be done on the python server and the initializer script, but the controller will have storage-specific logic for parsing the common secret and mapping them to the current storageUri and secret format.
Both solution above can remove the secret:// schema since the controller is going to parse the secret. @njhill probably wants the (A) approach. @yuzisun What do you think?

@Tomcli maybe (B) is least invasive for a first pass (at least to support just S3 via the new config mechanism) and defers having to worry about the secret transfer part.
But (A) imo would be better destination since we'll then have consistency of how storage config is consumed by the storage initializer/puller, i.e. a single secret with a consistent schema which can work the same for multi and single model (and same code used by both). storageUri parsing will then only be done in the controller, as outlined in my prior comment.
Question for @yuzisun and others: Do we think it could be an issue supporting the existing secret formats via the controller copying them to the new common secret? I.e. from a security/permission pov since the controller's service account would require secret write access. It would only be done within the context of the same service account.

@njhill sorry for the late update. I addressed your comments and implemented with the (A) method where all the mapping logics are in the controller now. Since we are not modifying the python storage initializer for this PR, we are pushing all the environments variables as value for now. This will be changed to secret mounting in phase 2 when we bring in the modelmesh golang storage client.

@Tomcli I did a bit more refactoring but was unable to push to this branch so I pushed a commit to my own fork. The commit message has a summary of what's changed, PTAL!
https://github.com/Tomcli/kfserving/compare/storagespec...njhill:storagespec?expand=1

Hi @yuzisun, do you think we can merge this phase 1 implementation right after the 0.8 release? The AWS IAM integration is probably similar to GCP's metacontroller that can mutate the serving pod with credentials, so it should still work with the current set of annotations.
We are planning to have the phase 2 implementation that built on top of this with the new modelmesh pullman library that can also support GCP, HTTPS, AZURE, etc. We can probably ask users on different cloud providers to verify all the IAM integration during phase 2.

In phase 1 we mainly want to introduce the storageSpec API, the S3 support is more like a preview on what it can look like.

In phase 1 we mainly want to introduce the storageSpec API, the S3 support is more like a preview on what it can look like.

Thanks @Tomcli ! Going to merge after 0.8 is released, can you also help follow up with an e2e test for the new storage spec? we can install minio on the test cluster if needed.

Thanks @yuzisun, I added the e2e test and regenerated the python client sdk with the latest 0.8 release commit.

Awesome work @Tomcli ! Looking forward to see the next phase of storage API unification.
/lgtm
/approve

thanks @yuzisun and @njhill for all the reviews.

Hi, Which apiVersion of kind "TrainedModel" has PVC support for storageUri? Please provide example yaml file if available.


Binary caching using the GitHub Actions cache

This PR enables the use of the GitHub Actions cache for binary caching! You can see it in action here https://github.com/quyykk/vcpkg-gha-demo, by looking over the CI logs.
It can be used by passing x-gha as binary source to the vcpkg tool. It requires two environment variables as well:

ACTIONS_CACHE_URL: URL to the cache
ACTIONS_RUNTIME_TOKEN: Access token to the cache.

Feedback appreciated!
cc @lukka

why not using https://github.com/microsoft/vcpkg/blob/master/docs/users/binarycaching.md#github-packages ?

GitHub packages aren't really made to cache port builds. It would seem that I have to manually delete a cache that became unused? Sounds annoying. I also have to install mono on MacOS runners (I believe), and authenticate to the Nuget server. Two things I don't have to do with the GHA cache.

This is super cool, but we want to make sure this is an "allowed" use of the GitHub Actions cache before merging. We're going to try to meet with GitHub next year to confirm before moving forward. Thanks again!

Update: green light! We're excited about this feature and just need to assign an engineer on our side to fully drink it in and give it a good review :)

This is fantastic!

Thanks for the review! I have made the changes you requested. I will open a PR to the docs repo shortly.

Additionally, here are some changes that are important but not mandatory to land this PR:

We should add a switch like --x-binary-cache-github-actions which is identical to --binarysource=clear;x-gha,readwrite. The intent is to make it easier to set the binary caching backend to GitHub actions without having to learn how to use --binarysource=.... However, it is not intended to be used with --binarysource=... I.e., if --x-binary-cache-github-actions --binarysource=<source> is specified, then --binarysource=<source> wins.

Add x-gha as a valid default provider. In other words, users should be able to set the default provider to x-gha. I.e., --binarysource=clear;default,read is equivalent to --binarysource=clear;x-gha,read.

Update the docs to just say "Adds default providers"

Add x-gha as a valid default provider. In other words, users should be able to set the default provider to x-gha. I.e., --binarysource=clear;default,read is equivalent to --binarysource=clear;x-gha,read.

Doesn't default include files as well?  Shouldn't it be both then? (I think I'm misunderstand this)

Add x-gha as a valid default provider. In other words, users should be able to set the default provider to x-gha. I.e., --binarysource=clear;default,read is equivalent to --binarysource=clear;x-gha,read.

Doesn't default include files as well? Shouldn't it be both then? (I think I'm misunderstand this)
edit: Wouldn't that also trigger the error about missing environment variables for everyone? (unless I disable it when using the default provider)

Ok, I think I need to give this some more thought. Please disregard the default stuff for now since it isn't really that important.

My apologies, but I actually think we should remove the --x-binary-cache-github-actions.

lmao no worries

Could we automatically detect the github actions ci environment so that user don't have to enable it manually?

Yes, there is a environment variable that can be used to detect the github actions ci environment.

What is the scope of this cache? Just one action? All the actions in the same repository? All the repositories in the same organization? Is there any way for forked versions of a project to benefit from the cache?

What is the scope of this cache?

The repo. You'll find more details in github's documentation

Is there any way for forked versions of a project to benefit from the cache?

Not really, since it's per repo. (But any Actions run because of a PR can use the cache of the parent iirc)

Thank you for this PR, it works great 🔥 Very useful 👌

I want to ask if this Binary caching works the same way also in the Manifest mode.

Yes

Thx

And what about the --binarysource='clear;x-gha,readwrite' command-line option in the Manifest mode? Do I need to set up the VCPKG_BINARY_SOURCES before the Manifest mode?
Is this correct?: $env:VCPKG_BINARY_SOURCES = 'clear;x-gha,readwrite'

For an example how to use this see the example linked in the PR: https://github.com/quyykk/vcpkg-gha-demo

I saw it, there is used the Classic mode only, here

No? This is manifest mode

Ohh, sry, it contains the vcpkg.json in the root, I didn't note it. 🙃
So it means that I don't need to define the VCPKG_BINARY_SOURCES? Is this handled automatically because I don't see nothing else and there is nothing about Manifest mode in Binary caching documenation.

If you use the same code as in the example then yes (the line - uses: lukka/run-vcpkg@v11 does that for you)

I'm not using lukka/run-vcpkg, I don't want to for so simple thing.

Then you have to define VCPKG_BINARY_SOURCES and do all the steps mentioned in https://learn.microsoft.com/en-us/vcpkg/users/binarycaching#gha

Check this out: https://github.com/quyykk/vcpkg-gha-demo/blob/bbefbd0984ee2bce612c11cbc37b70afbdb797a1/.github/workflows/ci.yml

Thank you for the advice, I'm currently implementing it in my workflows, your advice will help me save a few minutes.
I don't call vcpkg install at all, I'm using -D CMAKE_TOOLCHAIN_FILE:FILEPATH="$env:VCPKG_INSTALLATION_ROOT/scripts/buildsystems/vcpkg.cmake"


OpenStack and services for BigData applications

KSM Performance on TripleO QS is much more efficient than the one which  was provided by instack-virt-setup ( even activated before deployment ). I guess explanation might be found here 👎
http://blog.oddbit.com/2016/05/19/connecting-another-vm-to-your-tripleo-qu/
TripleO QS works  pretty stable for me via 👎
$ bash quickstart.sh -R master  --config config/general_config/pacemaker.yml
--nodes  config/nodes/3ctl_1comp_2ceph.yml  $VIRTHOST

Addressng complain "Updated 2017/03/16: The --config option seems to be broken, using instead -e @~/deploy-config.yaml.". See README.rst 👎
Feature Configuration and Nodes
In previous versions of triple-quickstart a config file was used to determine
not only the features that would be enabled in tripleo and openstack but also
the number of nodes to be used. For instance "config/general_config/ha.yml" would
configure pacemaker and ensure three controller nodes were provisioned.  This type
of configuration is now deprecated but will still work through the Queens release.
The feature and node configuration have been seperated into two distinct
configuration files to allow users to explicity select the configuration known as
"feature sets" and the nodes to be provisioned.  The feature set configuration
can be found under tripleo-quickstart/config/general_config/ and the node
configuration can be found under tripleo-quickstart/config/nodes/


Feature:  dispatchers support monitor names and workspace names

Good day to you!
suggestions for:

Supporting for workspace names in:

# default named workspace `workspaceName` on `monitorName`
workspace=monitorName, name:workspaceName

usecase: to be able to startup Hyprland with default named workspaces.

supporting monitor names in moveworkspacetomonitor:

exec=hyprctl dispatch moveworkspacetomonitor "name:term DP-1"

usecase: workaround and addition to paragraph 1), but also to be able to move multiple workspaces at startup to monitor
Thank you!

both 1 and 2 are already supported.
multiple workspaces at startup are impossible, as Hyprland will close a workspace thats out of focus and empty.

my bad. thank you!

that's why i asked in first message about

sure

it works good.
Thank you!


Temporary workaround to get ResourceContent

This is an ugly workaround that could lead to path clashes. 2 different APIs must be provided to retrieve content, one relative to classpath root and the other relative to .kaoto metadata file

not enough it creates another duplicated xsd file in a subfolder:

not enough it creates another duplicated xsd file in a subfolder:

This specific issue is due to an issue in datamapper itself, see https://github.com/KaotoIO/kaoto-datamapper-integration/issues/194


car fsspec reference file system output

@d70-t what do you think about outputing an fsspec ReferenceFileSystem representation:
https://fsspec.github.io/kerchunk/spec.html
for the resulting car files? The motivation is to store the car files locally or AWS S3, e.g., and access them in Python while removing any need for an IPFS server or gateway.

That's a very neat idea!
Do you want to give it a try? -- Otherwise I might also have a try on it, as I've been playing around with building reference files for grib data a few days ago.

I may get there, but I would not be surprised if you beat me to it :-)

😬 ... there might be an issue though: the 2MiB block size limit which is necessary to protect users from being DoS-ed by bad peers on IPFS. TLDR: you can only verify the correctness of a block after computing the hash of the entire thing. If someone sends you 100GB, you'd have to accept it only to throw it away after recognizing that it's just garbage.
Anyways, as a consequence, one might have to split zarr chunks up into smaller blocks using e.g. Flexible Byte Layout (FBL) if it should be possible to transfer the blocks via IPFS. But as far as I understand the ReferenceFileSystem spec, it's not possible to assemble a fsspec-file from multiple parts of a (CAR-) file, which would be required to translate FBL into refs.
As long as zarr chunks would stay below the 1 MiB soft-limit or the 2 MiB hard-limit, FBLs wouldn't be required and ReferenceFileSystem should be simple to build.
Probably this isn't too bad though, as #1 could provide a nice way of packing the small 1MiB chunks into larger objects again, which would be pretty close to zarr sharding zarr-developers/zarr-python#877.

Yes, the block size limit is a constraint. For my use cases, chunks that are 256x256 or 64x64x64 in size would be OK. Yes, maybe we need sharding support in zarr before this could be used generally.

A sidenote that I mentioned to @thewtex, I think the second prototype from @jstriebel (https://github.com/zarr-developers/zarr-python/pull/947) since it doesn't touch the store itself may be testable with this store?!

@joshmoore, I'd say yes and no 😬 ... Technically, it should work on this store, as zarr would pass on "shards" to this store in stead of "chunks", but for the store it would look just the same.
However, problems would arise indirectly due to the increased size of shards when compared to chunks. This is at least for two (related) reasons:
verifying correctness
This store builds up a Merkle-DAG from the pieces (shards / chunks) provided to the store. Usually (e.g. without any inlining or splitting of objects), each of these pieces whould be hashed as one thing and put into the DAG. If correctness of the data should be verified, the entire piece has to be downloaded and the hash has to be computed again. If a reader only wants to obtain a smaller "chunk" from a larger "shard", the reader can't verify correctness of the chunk. The same problem would arise if any form of Merkle-Tree would be used as discussed in zarr-developers/zarr-python#392.
obtaining data from untrusted sources
If data should be received from untrusted sources (e.g. as common in p2p networks), verification of the pieces is a lot more important. For this reason, p2p networks usually place a hard upper limit on the size of individually hashed blocks. In case of bitswap (the IPFS network protocol), this is 2 MB. So it would be possible to encode larger blocks using ipldstore, but you would not be able to pass them around afterwards.

what could we do?
The main issue here is that hashes must be computed on a tree of smaller pieces in order to be able to verify parts of the array. This looks a lot like the "compression problem": we also want to compress only small chunks of the array in order to be able to read smaller parts. We then pack many of those parts into a larger shard, such that we can store multiple of those pieces together. If the shard would be arranged as a "little Merkle Tree" itself, we would be able to verify the correctness of individual chunks without having to download the entire shard.
option 1: hashing on chunk-level
This would require to have the hashes / CIDs / checksums of each "chunk" listed within some toplevel object, which could be the shard index. The toplevel object would then have to be hashed itself, such that there would be one single hash value for the entire shard, which can be used to verify the correctness of all the chunk-hashes. However, we maybe don't want to include the offset positions of the chunks within the shard as part of the toplevel hash, because then hashes would change if the order of chunks within the shard would be modified. Thus, a "verifyable shard" might conceptually look like:
#index section
root offset, root size, chunk 0 offset, chunk 0 size, chunk 1 offset, chunk 1 size...
#root section <- that's what makes up the toplevel hash
chunk 0 hash, chunk 1 hash, chunk 2 hash ... (in some deterministic order)
#chunk section
chunk 0 data, chunk 1 data, ...

A possible (maybe not ideal) format for such a "verifyable shard" would be something like CARv2. It's a bit redundant for this particular usecase, but maybe not too much...
option 2: hashing on arbitrary slices of the shard
Another option would be to slice up the shards again on arbitrary boundaries, build the same kind of Merkle-Tree as above on top of the slices and store that in a similar format as the "verifyable shard" above. Thats about what the "flexibly byte layout" does. An advantage of this would be, that it would work on arbitrary chunk sizes (even if they are larger than any limit of any transport mechanism). The disadvantage would be, that there'll be quite a lot of duplicated work. Also I'd guess that "double chunking" turns out to be quite bad for performance.

I don't yet have a clear picture which "route to sharding" would be best to get there. But I can't really see a way around hashing on chunk-level, if we want any kind of checksumming. In the ipldstore-world, sharding might appear naturally if parts of the trees are exported as CAR (see #1). While this doesn't help directly for other stores, this whole story might be a little hint that implementing sharding on store-level (or as a store-transformation layer) could be a better choice than implementing it on the array level.

I don't yet have a clear picture which "route to sharding" would be best to get there. But I can't really see a way around hashing on chunk-level, if we want any kind of checksumming. In the ipldstore-world, sharding might appear naturally if parts of the trees are exported as CAR (see #1). While this doesn't help directly for other stores, this whole story might be a little hint that implementing sharding on store-level (or as a store-transformation layer) could be a better choice than implementing it on the array level.

I completely agree on this, and I arrived at this conclusion, also! I am going to take a stab at implementing a SharedStore. :thought_balloon:

#4

Tried this brilliantness! Looks to be very close, but I get an error?
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Input In [58], in <cell line: 1>()
----> 1 reference_fs = car2reference_fs(f'{image_name}.zarr.car')
      2 print(reference_fs)

File ~/bin/mambaforge/envs/spatial-image/lib/python3.9/site-packages/ipldstore/car_reference_fs.py:59, in car2reference_fs(filename)
     57 def car2reference_fs(filename: str) -> Dict[str, Any]:
     58     with open(filename, "rb") as stream:
---> 59         refs = car2reference_fs_refs(stream, "{{a}}")
     60     return {"version": 1, "templates": {"a": filename}, "refs": refs}

File ~/bin/mambaforge/envs/spatial-image/lib/python3.9/site-packages/ipldstore/car_reference_fs.py:32, in car2reference_fs_refs(stream_or_bytes, stream_name)
     29 def car2reference_fs_refs(stream_or_bytes: StreamLike, stream_name: str) -> Dict[str, Any]:
     30     root, cbor_objects, object_locations = collect_tree_objects(stream_or_bytes)
---> 32     tree = dag_cbor.decode(cbor_objects[root])
     33     assert isinstance(tree, dict)
     34     sep = "/"

KeyError: CID('base58btc', 1, 'dag-cbor', '1220c834b4567f834677ddbcf77d1858120536d76ea9474e8264419268f29acbebe5')

Reproducible locally with https://github.com/thewtex/spatial-image-multiscale/blob/tifffile/examples/ConvertTiffFile.ipynb

😬 that's unfortunate...
I couldn't exactly reproduce your notebook (to_multiscale didn't return scales other than 0), however, I was able to build the references. I figured that this is due to the version of the multiformats package. There had been an issue about how CIDs have been compared to each other which was solved in hashberg-io/multiformats#2 and merged, however this has not yet been released to pypi...

There had been an issue about how CIDs have been compared to each other which was solved in hashberg-io/multiformats#2

oi !
Also, it seems we need the IPFS implementations to catch up so they can cat dag-cbor :-(

Also, it seems we need the IPFS implementations to catch up so they can cat dag-cbor :-(

yes, that's the big issue with the direct IPLD-approach... However, it's planned (ipfs/go-ipfs#8640) for the next release of go-ipfs to support raw blocks and CAR export from gateways. This would likely simplify the use of generic IPLD structures over gateways.
The other option probably would be to build additional unixfs trees on top of the IPLD raw-leafs (#3). Most of the data would not have to be duplicated, but still data would be available in both worlds... I'm really torn between "zarr on unixfs on IPLD" and "zarr on IPLD". The former seems to be more compatible whereas the latter seems to be more elegant (and may be more useful also as a StorageTransformer for other storage backends).
Anyways, to go forward on this issue, you could either try installing multiformats from git, or we could write out own keys in the cbor_objects-dict, which would be a workaround for the mentioned bug.

it's planned (ipfs/go-ipfs#8640) for the next release of go-ipfs to support raw blocks

Great! -- raw-blocks + fsspec on gateways could be a big performance bump!

try installing multiformats from git,

Worked! :tada: :taco:

I am going to take a stab at implementing a SharedStore.

https://github.com/thewtex/shardedstore :tada:

I feel like I should applaud but I'm a bit lost as to what for :)
Are you guys up for a pres., demo, or blog at some point?
cc: @MSanKeys963

Hmmm..... so what's not working? :smile:

Ok, so it's currently possible to create one large CAR of zarr-on-IPLD objects using ipldstore on a single machine. It's also possible to build a reference filesystem based off this CAR (if using the current HEAD of multiformats). So probably we should close this issue and if you think it helps, make some post / presentation out of it. It would basically be:

put some zarr into ipldstore and make a CAR out of it
make a reference filesystem out of it
open it using xr.open_zarr("reference::...")

On its own, it's more or less a complicated way of storing a zarr into a zip store (or any other single-file backend), but it helps in exploring :-)

What's not yet implemented, but has been discussed a bit here (probably out of scope of this issue) would be:

splitting the CAR into multiple cars (to use this for sharding)
use CARv2 instead of the reference-filesystem-json as s shard format (to keep checksums / CIDs visible)
use this for verification (for that to work, we might not want to use reference filesystem, but ipldstore instead, because by creating the reference filesystem, we basically have to strip off the checksums)
renumbering chunks (as suggested here) to facilitate splitting the cars

Hi,
I've been working on a similar issue behind the scene. I was searching for a way to

verify CARs of zarr files locally, especially those where one zarr file is splitted across CARs
access their contents directly

I basically combined all the great work you did and added a few extra lines to build car_referencer. So now you could do something like
car_referencer -c "carfiles.*.car" -p preffs.parquet -r ROOT-HASH

import xarray as xr
ds = xr.open_zarr("preffs:preffs.parquet")

Maybe it is helpful for you as well. Feedback welcome.


Unable to preload plugins

Unable to preload the following plugins:
libRenderer.so
libRustNative.so
libRenderer.so
sqlite3.so
The server is not listet in die list of servers in the game

They are normal errors while running in docker


Updates Python requirements

But... we don't use the flask?

I can try with only adding blinker, but if they add other dependencies we could have the same problem.


chore: release 0.20.3

TBR = @ofrobots (this is busted in prod)

Lgtm
On Sat, Jun 30, 2018, 9:37 PM Justin Beckwith<EMAIL_ADDRESS>wrote:

Merged #188 https://github.com/googleapis/nodejs-common/pull/188.
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
https://github.com/googleapis/nodejs-common/pull/188#event-1710161708,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAE0qfD-VOiFN0KOKa6qoI1ByEA0ChaJks5uCFIQgaJpZM4U-MfF
.


master/dev sync issues

Well, I've been creating and merging PRs, but they've been landing on the master branch and not the dev branch.
Should we set the default branch for this project to "dev" to help avoid this problem?

@jshrake before @Gobiner spins up on this project, can you set some policy around master vs. dev?  Maybe document your thoughts in the README?

I make release tarballs off of the HEAD of master (https://github.com/Oblong/homebrew-tools/blob/master/obi.rb#L4)
brew install obi --HEAD installs from the HEAD of dev (https://github.com/Oblong/homebrew-tools/blob/master/obi.rb#L6)
All work should land on dev. I announce internally when I push changes to dev, and that ask that dev users update. After a few weeks, if all is well, I'll merge to master and tag a release.

added that content to README, closing.


The constructor shorthand property gets assigned twice when explicitly declared inside the constructor body

monaco-editor version: the online version
Browser: Version 61.0.3163.100 (Official Build) (64-bit)
OS: Mac OS
Steps or JS usage snippet reproducing the issue:
The constructor shorthand property gets assigned twice if I explicitly assign this.form = form; in the constructor. Is this an expected behaviour?

Go to link

or write the following code in the TypeScript Playground:
class MyService {
    constructor(private form: HTMLFormElement) {
        this.form = form;
    }
}

the output will be
var MyService = /** @class */ (function () {
    function MyService(form) {
        this.form = form;
        this.form = form;
    }
    return MyService;
}());

Expected output:
var MyService = /** @class */ (function () {
    function MyService(form) {
        this.form = form;
    }
    return MyService;
}());

This is how the TypeScript compiler behaves. When you use constructor(private form: HTMLFormElement) the TS compiler will emit the assignment for you. So the line this.form = form is redundant.
Use:
class MyService {
    constructor(private form: HTMLFormElement) {
    }
}


stats: More StatName conversions

Description: Avoid using scope.counter(string) in Zookeeper and ExtAuthz, factoring out some infrastructure for looking up dynamically constructed stat-names for shared use. 6 more files to go and then #4980 can hopefuly be merged, addressing #4196.
Risk Level: low
Testing: //test/...
Docs Changes: n/a
Release Notes: n/a

/retest

@lizan I had previously thought this PR was all green; but now it's in this "waiting for status..." state. How do I trigger a re-test for this flavor of CI?

Any other comments? This is blocking the reviewability of https://github.com/envoyproxy/envoy/pull/7842 which I think is the last thing we need prior to reviewing #4980 and thus resolving #4196 .

Thanks @snowp -- @envoyproxy/senior-maintainers can I get another round of approval?


java.lang.UnsatisfiedLinkError: nativeOpen on 1.2 and Android SDK15

After upgrade to 1.2 observe the following exception on Android SDK 15 (emulator) with this code:
new Manager(new AndroidContext(context), Manager.DEFAULT_OPTIONS);
java.lang.UnsatisfiedLinkError: nativeOpen
    at com.couchbase.lite.internal.database.sqlite.SQLiteConnection.nativeOpen(Native Method)
    at com.couchbase.lite.internal.database.sqlite.SQLiteConnection.open(SQLiteConnection.java:194)
    at com.couchbase.lite.internal.database.sqlite.SQLiteConnection.open(SQLiteConnection.java:178)
    at com.couchbase.lite.internal.database.sqlite.SQLiteConnectionPool.openConnectionLocked(SQLiteConnectionPool.java:461)
    at com.couchbase.lite.internal.database.sqlite.SQLiteConnectionPool.open(SQLiteConnectionPool.java:191)
    at com.couchbase.lite.internal.database.sqlite.SQLiteConnectionPool.open(SQLiteConnectionPool.java:183)
    at com.couchbase.lite.internal.database.sqlite.SQLiteDatabase.openInner(SQLiteDatabase.java:814)
    at com.couchbase.lite.internal.database.sqlite.SQLiteDatabase.open(SQLiteDatabase.java:798)
    at com.couchbase.lite.internal.database.sqlite.SQLiteDatabase.openDatabase(SQLiteDatabase.java:693)
    at com.couchbase.lite.storage.SQLiteStorageEngineBase.open(SQLiteStorageEngineBase.java:62)
    at com.couchbase.lite.DatabaseUpgrade.importData(DatabaseUpgrade.java:81)
    at com.couchbase.lite.Manager.upgradeDatabase(Manager.java:830)
    at com.couchbase.lite.Manager.upgradeOldDatabaseFiles(Manager.java:808)
    at com.couchbase.lite.Manager.<init>(Manager.java:130)

Logcat
02-25 09:53:35.880 646-661/? W/Database: mustExist is true and db (cblite-test) does not exist
02-25 09:53:35.880 646-661/? D/dalvikvm: Trying to load lib /data/data/com.couchbase.lite.test/lib/libCouchbaseLiteJavaForestDB.so 0x4160eea0
02-25 09:53:35.920 646-661/? D/dalvikvm: Added shared lib /data/data/com.couchbase.lite.test/lib/libCouchbaseLiteJavaForestDB.so 0x4160eea0
02-25 09:53:35.941 646-661/? D/dalvikvm: No JNI_OnLoad found in /system/lib/libsqlite.so 0x4160eea0, skipping init
02-25 09:53:35.941 646-661/? D/dalvikvm: Trying to load lib /data/data/com.couchbase.lite.test/lib/libcbljavasqlitedefault.so 0x4160eea0
02-25 09:53:35.941 646-661/? E/Database: Cannot load native library
02-25 09:53:35.950 646-661/? D/dalvikvm: Trying to load lib /data/data/com.couchbase.lite.test/lib/libcbljavasqlitedefault.so 0x4160eea0
02-25 09:53:35.950 646-661/? E/Database: Cannot load native library
02-25 09:53:35.950 646-661/? W/dalvikvm: No implementation found for native Lcom/couchbase/lite/internal/database/sqlite/SQLiteConnection;.nativeOpen (Ljava/lang/String;ILjava/lang/String;ZZ)J
02-25 09:53:35.950 646-661/? I/TestRunner: failed: test12_AllDocumentsBySequence(com.couchbase.lite.ApiTest)
02-25 09:53:35.960 646-661/? I/TestRunner: ----- begin exception -----
02-25 09:53:36.001 646-661/? I/TestRunner: ----- end exception -----
02-25 09:53:36.030 646-649/? D/dalvikvm: GC_CONCURRENT freed 387K, 4% free 12933K/13383K, paused 5ms+2ms
02-25 09:53:36.030 646-661/? I/TestRunner: finished: test12_AllDocumentsBySequence(com.couchbase.lite.ApiTest)

StackTrace
Testing started at 9:53 AM ...
Target device: Nexus_4_API_15_arm [emulator-5554]
No apk changes detected since last installation, skipping installation of /Users/hideki/github/couchbase-lite-android/build/outputs/apk/couchbase-lite-android-debug-androidTest-unaligned.apk
Force stopping package: com.couchbase.lite.test
DEVICE SHELL COMMAND: am force-stop com.couchbase.lite.test
Running tests
Test running started
java.lang.UnsatisfiedLinkError: nativeOpen
at com.couchbase.lite.internal.database.sqlite.SQLiteConnection.nativeOpen(Native Method)
at com.couchbase.lite.internal.database.sqlite.SQLiteConnection.open(SQLiteConnection.java:194)
at com.couchbase.lite.internal.database.sqlite.SQLiteConnection.open(SQLiteConnection.java:178)
at com.couchbase.lite.internal.database.sqlite.SQLiteConnectionPool.openConnectionLocked(SQLiteConnectionPool.java:461)
at com.couchbase.lite.internal.database.sqlite.SQLiteConnectionPool.open(SQLiteConnectionPool.java:191)
at com.couchbase.lite.internal.database.sqlite.SQLiteConnectionPool.open(SQLiteConnectionPool.java:183)
at com.couchbase.lite.internal.database.sqlite.SQLiteDatabase.openInner(SQLiteDatabase.java:814)
at com.couchbase.lite.internal.database.sqlite.SQLiteDatabase.open(SQLiteDatabase.java:798)
at com.couchbase.lite.internal.database.sqlite.SQLiteDatabase.openDatabase(SQLiteDatabase.java:693)
at com.couchbase.lite.storage.SQLiteStorageEngineBase.open(SQLiteStorageEngineBase.java:62)
at com.couchbase.lite.store.SQLiteStore.open(SQLiteStore.java:183)
at com.couchbase.lite.Database.open(Database.java:1150)
at com.couchbase.lite.Manager.openDatabase(Manager.java:336)
at com.couchbase.lite.Manager.getDatabase(Manager.java:300)
at com.couchbase.lite.LiteTestCaseWithDB.ensureEmptyDatabase(LiteTestCaseWithDB.java:245)
at com.couchbase.lite.LiteTestCaseWithDB.startDatabase(LiteTestCaseWithDB.java:230)
at com.couchbase.lite.LiteTestCaseWithDB.setUp(LiteTestCaseWithDB.java:132)
at com.couchbase.lite.LiteTestCase.runBare(LiteTestCase.java:20)
at com.couchbase.lite.LiteTestCaseWithDB.runBare(LiteTestCaseWithDB.java:88)
at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:545)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1551)

Finish

libsqlite.so exists, is it does not have JNI_OnLoad() method?
libsqlite.so is sqlite not JNI.

I guessed on API 15, it couldn't load system sqlite.

https://github.com/couchbase/couchbase-lite-android/issues/434
As described in linked ticket, I guess Android 4.0.3 might try to load armeabi JNI library instead of armeabi-v7a.
We don't provide armeabi JNI for forestdb, it might explain why loading forestdb JNI, but not SQLite Jni.

Is there a workaround?

@filipesperandio: The workaround should be using an emulator that is 4.1+ / API 16+, as the bug is in Android 4.0.3 itself.
We implemented a workaround previously, but when we overhauled our storage code we didnt also re-implement the fix. This will get fixed in an upcoming patch release.

I am targeting devices running android 4.0.4 and got this very issue. If is there anything I can patch around to stick with couchbase I would love it, otherwise I think will have to reimplement my persistence layer with SQLite itself. =\

@filipesperandio try using the APK split technique. This should bypass the Android bug by only including native images for one platform per APK.

No much luck so far.
Tried this:
splits {
        abi {
            enable true
            reset()
            include 'armeabi-v7a', 'x86'
            universalApk false
        }
    }

But error still happens on that device. Should it be something slightly different from that? Not sure if I got the problem right. Is it an issue the way android 4.0.4 loads abis native code? (sort of)

Couchbase Lite Android JNI loads system sqlite3.so from Android, and call sqlite3_db_readonly() method which was added from 3.7.11. However, Android API 15 or earlier uses SQLite 3.7.4 or earlier. System.loadLibrary(<CBL JNI library name>) can identify unsatisfied method. Then throws Exception.
We Fixed this by commented out the lines to check if SQLite database is readonly.
PR: https://github.com/couchbase/couchbase-lite-java-native/pull/40
Note: I tried some solutions by checking if function pointer is null or checking SQLite version to identify sqlite3_db_readonly() exists. However, System.loadLibrary() throws UnsatisfiedLinkError if code calls this method.

Following is build from master branch. This includes the fix.
http://files.couchbase.com/maven2/com/couchbase/lite/couchbase-lite-android/0.0.0-605/

Cool. Thanks!
On Mar 3, 2016 9:20 PM, "Hideki Itakura"<EMAIL_ADDRESS>wrote:

Following is build from master branch. This includes the fix.
http://files.couchbase.com/maven2/com/couchbase/lite/couchbase-lite-android/0.0.0-605/
—
Reply to this email directly or view it on GitHub
https://github.com/couchbase/couchbase-lite-android/issues/792#issuecomment-192033466
.

Hello all. My first time posting an issue to this repository. Thanks for all your work so far on this library.
I'm still getting this crash on a Kindle Fire. I've attached the output of adb shell getprop in case it's helpful.
kindle_props.txt
I've tried several builds, including the latest from master, the 605 referenced above, and 1.2.1 (was doing final release testing with 1.2.0 when this came up).
The stack trace looks the same as the ones pasted above, but just in case I've attached mine as well.
stack_trace.txt
Thanks for any help!

Please disregard my previous comment. It turns out there was an issue in my build script with correctly copying the .so files into the APK. 1.2.1 is no longer crashing my Kindle.
Thanks anyway!


ShowFPS still shows Max KSP version even though it's been updated

I adopted this mod, and updated the netkan.
It properly shows in CKAn when showing compatible mods, but it still lists the Max KSP version as 1.1
I suspect this is because the old version had it hard-coded in the netkan, and that's been removed, as well as the new .version file doesn't have that field at all.
I can always push a new version with an updated .version file, but would rather not if I don't have to.
Let me know

I also added an epoch to it, hoping that will fix the issue, since the versioning changed

The epoch fixed it

Thanks for calling this out; the client shouldn't be doing that, but KSP-CKAN/CKAN#2131 got reverted along with KSP-CKAN/CKAN#1929 in KSP-CKAN/CKAN#2150...


Compilation differences found in tests between ballerina build and ballerina test commands

Description:
There are differences when compilation when running tests with ballerina build -a and ballerina test -a followed by building with the project with gradle.
Steps to reproduce:

Checkout https://github.com/ballerina-platform/module-ballerinax-aws.ses
Build the project with ./gradlew clean build. It will build and run tests correctly.
Then go to <module_home>/ses-ballerina directory and execute ballerina test -a. It will successfully run tests without any issue.
Now from the same directory location execute ballerina build -a. It will build successfully but fail the tests due to AWS Signature failure.
This is due to the line in https://github.com/ballerina-platform/module-ballerinax-aws.ses/blob/534cff6808bdc6a70f142804e90c3150f493a083/ses-ballerina/src/ses/ses.bal#L425 the character \n in variable canonicalRequest is not correctly converted to the newline character when that is called in line 427.

Resolved by #24964


Update EVP_KDF-X942-ASN1.pod

Replaced OSSL_KDF_PARAM_KEY with OSSL_KDF_PARAM_SECRET as that seems to be the intended value from the code (OSSL_KDF_PARAM_KEY is also supported but looks like a fallback).
Fixed name for OSSL_KDF_PARAM_X942_USE_KEYBITS.
CLA: trivial

Checklist

[x] documentation is added or updated

Sorry for the force-push, I had to fix an incorrect committer email.

ANS X9.42-2003 and RFC2631 both talks about a key not a secret, so I suspect this change is wrong.
@slontis any thoughts here?

The term is z.. Which is a shared secret.. Either name is fine since they are aliases..

ANS X9.42-2003 and RFC2631 both talk about a key not a secret, so I suspect this change is wrong. @slontis any thoughts here?

The same file itself actually uses OSSL_KDF_PARAM_SECRET in its example, so it's a bit confusing when the docs say you need to use OSSL_KDF_PARAM_KEY but the example does something else. Even though they are aliases in the source.

24 hours has passed since 'approval: done' was set, but as this PR has been updated in that time the label 'approval: ready to merge' is not being automatically set.  Please review the updates and set the label manually.

Merged, thanks for the contribution.
I was just making sure we wouldn't be using "secret" when the standard says "key".  The various KDFs are not consistent in their terminology :unamused:


Add config option to change the order of section panels in the sidebar

Adds a preview_inspector.sidebar_panels config option that allows for the reordering (or omitting) of section panels in the Lookbook sidebar.
config.lookbook.preview_inspector.sidebar_panels = [:pages, :previews]

The default value is [:previews, :pages].

Closed in favour of https://github.com/ViewComponent/lookbook/pull/556


Update new Activity Type filters in API

Adding new Action type filter to the API

Learn Build status updates of commit 0b2f607:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

CloudAppSecurityDocs/api-activities.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

@dcurwin
Can you review the proposed changes? IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team


Update app marketplace and installed-app view details

What's Changed

App Marketplace view now displays installed marketplace apps with an Installed label instead of not showing them at all.
Marketplace manifest now omits apps that have all display properties set to False
App Marketplace view now includes a breadcrumb
App Marketplace tiles and Installed Apps tiles now are more consistently sized
Installed Apps view now uses the marketplace headline for installed marketplace apps instead of the app-provided description.

Screenshots

TODO

[x] Explanation of Change(s)
[x] Added change log fragment(s) (for more information see the documentation)
[x] Attached Screenshots, Payload Example
n/a Unit, Integration Tests
n/a Documentation Updates (when adding/changing features)
n/a Example App Updates (when adding/changing features)
[x] Outline Remaining Work, Constraints from Design

Now also includes a number of icon updates in the marketplace manifest.


[ALLUXIO-2941] make createPath method in InodeTreeTest.java private.

https://alluxio.atlassian.net/browse/ALLUXIO-2941

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/18709/
Test PASSed.


Loss is 0 when policy (model) and reference models (model_ref) are the same

Hi,
I had a question about the implementation. When the policy and reference models are the same, will the loss not be 0?
https://github.com/huggingface/trl/blob/main/trl/trainer/dpo_trainer.py#L361
Or equation 7 in the paper?
Maybe I am missing something? Can you please help me understand it better?
In tutorials, since base model has a LoRA weights addon, I think this issue does not pop up.
But full fine-tuning of the base model will likely cause the issue.

Looks like we do the chosen rejection subtraction first
https://github.com/huggingface/trl/blob/main/trl/trainer/dpo_trainer.py#L355


ERROR: unable to download video data: HTTP Error 403: Forbidden

python3 Youtube_Vid_Downloder.py
2024-12-29 09:28:17.627 Python[14064:12807126] +[IMKClient subclass]: chose IMKClient_Modern
2024-12-29 09:28:38.062 Python[14064:12807126] The class 'NSOpenPanel' overrides the method identifier.  This method is implemented by class 'NSWindow'
[youtube] Extracting URL: https://www.youtube.com/watch?v=9yZA7Q-ZKHU
[youtube] 9yZA7Q-ZKHU: Downloading webpage
[youtube] 9yZA7Q-ZKHU: Downloading ios player API JSON
[youtube] 9yZA7Q-ZKHU: Downloading mweb player API JSON
[youtube] 9yZA7Q-ZKHU: Downloading m3u8 information
[info] 9yZA7Q-ZKHU: Downloading 1 format(s): 135+251
ERROR: unable to download video data: HTTP Error 403: Forbidden

Thank you for letting me know, I'll fix it as soon as possible.

I liked your application. Please add additional functionality to download list of videos.
Thanks and Happy holidays.

I have made some changes for now. I'll work on the application part as soon as possible.
Thank you.

Tested and confirmed to be working. Thank you!


Test util task migration v5

Solution for https://jira.arm.com/browse/PWRSOFT-2145
Last CI Tests: https://jenkins.oss.arm.com/view/Power/job/power/job/synthetic-tests/job/power-start-matrix-lisa-synthetics/381

juno-r0:  OneTaskCPUMigration[board=juno-r0/00:02:F7:00:58:F3]:test_util_task_migration:  FAILED 11/50 (22.0%)
juno-r2:  OneTaskCPUMigration[board=juno-r2/00:02:f7:00:67:99]:test_util_task_migration:  FAILED 7/50  (14.0%)
hikey960: OneTaskCPUMigration[board=hikey960/00:0e:c6:b9:54:d7]:test_util_task_migration: passed 50/50 (100.0%)
TC2:      OneTaskCPUMigration[board=tc2/00:02:F7:00:48:A4]:test_util_task_migration:      passed 50/50 (100.0%)
db845c:   OneTaskCPUMigration[board=db845c/00:e0:4c:75:10:0f]:test_util_task_migration:   passed 50/50 (100.0%)

Commit 8cf1198ad718 "lisa.tests.scheduler.load_tracking: Fix remaining issue for CPUMigrationBase::test_util_task_migration" brings the error rate for juno-r0 and juno-r2 furher down to:

juno-r0:  OneTaskCPUMigration[board=juno-r0/00:02:F7:00:58:F3]:test_util_task_migration:  FAILED 3/50 (6.0%)
juno-r2:  OneTaskCPUMigration[board=juno-r2/00:02:f7:00:67:99]:test_util_task_migration:  FAILED 2/50  (4.0%)

Related to the pelt_interpolate() implementation:
Currently, I'm not parsing the clock, put cpu_capacity as a third parameter.
diff --git a/lisa/pelt.py b/lisa/pelt.py
index 8a30146c077e..5cadf3eeb1a4 100644
--- a/lisa/pelt.py
+++ b/lisa/pelt.py
@@ -20,7 +20,7 @@ import math
 import pandas as pd
 import numpy as np

-from lisa.datautils import series_envelope_mean
+from lisa.datautils import series_envelope_mean, df_add_delta, df_extend_index

 PELT_WINDOW = 1024 * 1024 * 1e-9
 """
@@ -266,4 +266,34 @@ PELT_SCALE = 1024
     """
     return series_envelope_mean(util)

+def pelt_interpolate(util, interpolate_at, cpu_capacity):
+    util = df_extend_index(util, interpolate_at)
+
+    util['cpu'].fillna(method='ffill', inplace=True)
+    util = df_add_delta(util)
+    util['delta'] = util['delta'].shift()
+    util['prev_util'] = util['util'].shift()
+
+    util = util.dropna(subset=['cpu', 'delta'])
+
+    def compute_switch_phase_df(row):
+        timestamp = row.name
+        # Transform ftrace timestamp to PELT clock time.
+        # Only CPU capacity (and not frequency) is considered
+        # here. It has the bigger impact.
+        delta = row.delta * cpu_capacity/PELT_SCALE
+        last_update = timestamp - delta
+        prev_util = row.prev_util
+        window_shrink = 1e3
+        activations = pd.Series([1, 1], index=[last_update, timestamp])
+        simulated_phase_df = simulate_pelt(activations, init=prev_util,
+                                           window=PELT_WINDOW/window_shrink,
+                                           half_life=PELT_HALF_LIFE*window_shrink)
+        return simulated_phase_df.iloc[-1]
+
+    switch_loc = util['util'].isnull()
+    util.loc[switch_loc, 'util'] = util.loc[switch_loc].apply(compute_switch_phase_df, axis='columns')
+
+    return util['util']
+
 # vim :set tabstop=4 shiftwidth=4 textwidth=80 expandtab
diff --git a/lisa/tests/scheduler/load_tracking.py b/lisa/tests/scheduler/load_tracking.py
index c94d159a69b0..5afaa3fc3c60 100644
--- a/lisa/tests/scheduler/load_tracking.py
+++ b/lisa/tests/scheduler/load_tracking.py
@@ -29,14 +29,14 @@ from lisa.tests.base import (
 )
 from lisa.target import Target
 from lisa.utils import ArtifactPath, groupby, ExekallTaggable
-from lisa.datautils import series_mean, df_window, df_filter_task_ids, series_refit_index, df_split_signals, df_refit_index, df_add_delta, df_extend_index
+from lisa.datautils import series_mean, df_window, df_filter_task_ids, series_refit_index, df_split_signals, df_refit_index
 from lisa.wlgen.rta import RTA, Periodic, RTATask
 from lisa.trace import FtraceCollector, requires_events, may_use_events, MissingTraceEventError
 from lisa.analysis.load_tracking import LoadTrackingAnalysis
 from lisa.analysis.tasks import TaskState, TasksAnalysis
 from lisa.analysis.rta import RTAEventsAnalysis
 from lisa.analysis.frequency import FrequencyAnalysis
-from lisa.pelt import PELT_WINDOW, PELT_HALF_LIFE, PELT_SCALE, simulate_pelt, pelt_settling_time, kernel_util_mean
+from lisa.pelt import PELT_SCALE, simulate_pelt, pelt_settling_time, kernel_util_mean, pelt_interpolate

 UTIL_SCALE = PELT_SCALE

@@ -987,37 +987,9 @@ Time in seconds for util_avg to converge (i.e. ignored time)
                 for task, wlgen_task in self.rtapp_profile.items():
                     if cpu in wlgen_task.phases[phase-1].cpus:
                         df_state_clipped = df_window(df_state[task], (start, end), method='pre', clip_window=True)
-                        cpu_phase_df = df_extend_index(cpu_phase_df, df_state_clipped)
-
-                cpu_phase_df['cpu'].fillna(method='ffill', inplace=True)
-                cpu_phase_df = df_add_delta(cpu_phase_df)
-                cpu_phase_df['delta'] = cpu_phase_df['delta'].shift()
-                cpu_phase_df['prev_util'] = cpu_phase_df['util'].shift()
-
-                cpu_phase_df = cpu_phase_df.dropna(subset=['cpu', 'delta'])

                 cpu_capacity = self.plat_info['cpu-capacities']['rtapp'][cpu]
-
-                def compute_switch_phase_df(row):
-                    timestamp = row.name
-                    # Transform ftrace timestamp to PELT clock time.
-                    # Only CPU capacity (and not frequency) is considered
-                    # here. It has the bigger impact.
-                    delta = row.delta * cpu_capacity/PELT_SCALE
-                    last_update = timestamp - delta
-                    prev_util = row.prev_util
-                    window_shrink = 1e3
-                    activations = pd.Series([1, 1], index=[last_update, timestamp])
-                    simulated_phase_df = simulate_pelt(activations, init=prev_util,
-                                                       window=PELT_WINDOW/window_shrink,
-                                                       half_life=PELT_HALF_LIFE*window_shrink)
-                    util = simulated_phase_df.iloc[-1]
-                    return util
-
-                switch_loc = cpu_phase_df['util'].isnull()
-                cpu_phase_df.loc[switch_loc, 'util'] = cpu_phase_df.loc[switch_loc].apply(compute_switch_phase_df, axis='columns')
-
-                util = cpu_phase_df['util']
+                util = pelt_interpolate(cpu_phase_df, df_state_clipped, cpu_capacity)
                 cpu_util.setdefault(cpu, {})[phase] = kernel_util_mean(util, plat_info=self.plat_info)

         return cpu_util

+def pelt_interpolate(util, interpolate_at, cpu_capacity):

util should be a Series, not a dataframe, so the function becomes simpler and more generically applicable. It's the caller's responsibility to provide a pelt clock that goes with the entity represented by util.
For example, this function does not need to know what CPU the task is placed on, because that information feeds from the PELT clock, which is computed separately with something like that:
def simulate_pelt_clock(capacity, clock=None, scale=PELT_SCALE):
    """
    Simulate a PELT clock of an entity from the capacities of the CPU it's
    residing on.

    :param capacity: CPU capacity over time.
    :type capacity: pandas.Series

    :param clock: If provided, a series of timestamps at which the clock is to
        be observed. The returned :class:`pandas.Series` will provide the
        simulated clock values at these instants.
    :type clock: pandas.Series or None

    :param scale: Maximum value allowed for CPU capacity.
    :type scale: float
    """
    user_clock = clock is not None
    if user_clock:
        # Ensures the clock's index is the same as the clock
        clock = clock.copy(deep=False)
        clock.index = clock
    else:
        clock = capacity.index

    df = pd.DataFrame(
        dict(
            clock=clock,
            capacity=capacity,
        ),
    )
    # Needed for "time" interpolation
    df.index = pd.TimedeltaIndex(df.index, unit='s')

    # Time flows linearly between 2 samples of the clock
    df['clock'].interpolate(method='time', inplace=True)
    # Shift so that the capacity is aligned with the corresponding delta
    df['capacity'] = df['capacity'].fillna(method='ffill').shift()
    df['delta'] = df['clock'].diff()

    # Scale each delta independantly
    df['delta'] *= df['capacity'] / scale

    # Fill the NaN with the initial value for the cumsum() fold
    df['delta'].iat[0] = df['clock'].iloc[0]
    # Reverse df_add_delta() now that we scaled each delta
    df['new_clock'] = df['delta'].cumsum()

    # Back to Float64Index
    df.index = df.index.total_seconds()

    if user_clock:
        df = df.loc[clock]
    return df['new_clock']


Support for MIDI models in HARP.

Introduces basic support and display for Midi-to-Midi models along with a major cleanup of the underlying GUI components that control media display.

This should now be good to go. We just need to make sure we update the commit and branch for the pyharp submodule after https://github.com/TEAMuP-dev/pyharp/pull/15 is merged, and that we update the compatibility table in the README.

@xribene, @NathanPruyne, this branch has been rebased to the current main and can be merged if it all looks good to you guys. I also moved the undo/redo stuff to the MediaDisplayComponent class to decouple it from the model.

I'm currently having a failed build on Windows with:
C:\HARP\HARP\src\media\AudioDisplayComponent.h(100,1): error C4716: 'AudioDisplayComponent::getPlaybackPosition': must
return a value [C:\HARP\HARP\build-win\HARP.vcxproj]

Seems like the value isn't actually returned there explicitly (i.e. with "return"). I'll try to debug this and then pr a branch into this one, and also make sure it works on Linux with the fix.

Ok, that fix seems to have worked, and is in #192 . I think this is good to go!
Couple notes for improving the MIDI display in the future:

Upon file load we might want to jump the display down to where there's actual MIDI content so it's clear that it's loaded. It might not hurt to jump there for after processing too to clearly display the new content.
IMHO actual note names (e.g. "B3", "C4", "C#4", etc.) would be more useful for the y-axis labels than the MIDI note numbers.
Horizontal scroll should probably be limited to the length of the MIDI file. Maybe limit vertical scroll to the highest and lowest notes of the file too?

Nice catch @NathanPruyne. In terms of your comments on the MIDI display, I'm actually working on this right now in a separate branch (fc/pianoroll_improvements). I'm pretty much in agreement with what you're suggesting.


(Feature) Reorder contract executors: owners are first

If an executed contract has owner or getOwners getters, and addresses from the resulted array are in the list of wallet accounts, they will be first in the list in the Choose contract executor screen.

@vbaranov
Uncaught error in Chrome's console when attempt to execute method transfer from  contract 0x215b2ab35749e5a9f3efe890de602fb9844e842f, Sokol
Input data:

address 0xd7b7AFeCa35e32594e29504771aC847E2a803742
value  1

As a result NW can't execute method transfer


Fix react-native-keyboard-tracking-view when used in a mutli-page chat app

Context: You're building a chat app. It has multiple scrollviews. The first one is a list of people. When you tap on a person, it takes you to a messages page, where you can message that person.
Problem: react-native-keyboard-tracking-view chooses the first scrollview available. The first scrollview in this case is the list of people. But, if you were using react-native-keyboard-input-view and wanted to use it for an InputAccessoryView on the messages page, it will track the scrollview with the list of people, instead of the messages page.
Solution: requiresSameParentToManageView
requiresSameParentToManageView already exists, however it has a bug.
When requiresSameParentToManageScrollView is true, we're asking KeyboardTrackingViewManager to ignore all scroll views that do not share a superview with the <KeyboardTrackingView />. This is a way to isolate a more specific scroll view to track, than simply the first one it finds.
The line after checking requiresSameParentToManageScrollView doesn't ensure that requiresSameParentToManageScrollView is false, meaning that if requiresSameParentToManageScrollView is true, and it finds a scrollview that isn't the parent superview...it will just use that one, defeating the purpose of requiresSameParentToManageScrollView.
This PR ensures that if requiresSameParentToManageScrollView is true,  it ignores scroll views that don't share a superview.
Testing plan
Manually. I verified in my app that it now chooses the parent scrollview, rather than the first scrollview found during the breadth-first search

available in v5.4.4


feat: DocumentType (_type) has been deprecated in ElasticSearch 7

Trying to resolve #123 by not having DocumentType as required (can be blank).
https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html
Also improved the ability for getting debug-output into NLog InternalLogger (When enabled)
Also change the default-value from "logevent" to "_doc" as it seems to work better (also for Serilog)
Checklist

[x] My commits follow Conventional Commits
[ ] documentation is updated

When this PR has been merged, then I also think a small fix can be made, so one can use NLog JsonLayout directly (or another Json-related-Layout like EcsLayout)

Thanks for the merge. Have now created #125 as mentioned.


Mac's readlink compatibility

Mac users should do:
brew install coreutils

Then do:
export PATH=/usr/local/opt/coreutils/libexec/gnubin:$PATH

to get this command to work properly.

I constantly use readlink -f in linux, now I can use it on the Mac too! Thanks.


Revise homepage: Iteration 1

Hold a requirements capture session with DH, KC and TH and revise the page to reflect their first thoughts.
This is likely to require several iterations and therefore anticipate this will all change.

requirements capture is done, and home page stripped to minimum. implementing a crude version is next step

The current unstyled homepage satisfies the goal of the milestone.


Add boiler starts and other related sensors

The title says it all. I think in HA I could not see a disabled sensor for number of burner starts etc?
Is it useful ?

Multiple sensors added to RC300.
Will add them also for other devices.


Question: why use host to target api?

As a api gateway, when we add apis to it, we need one way to target specific api.
There're 2 way to target apis in kong for clients:

Proxy an API by its DNS value

# send a request
$ curl -i -X GET \
  --url http://localhost:8000/ \
  --header 'Host: mockbin.com'

Proxy an API by its request_path value

# send a request
$ curl -i -X GET \
  --url http://localhost:8000/status/200

Then one question comes to me that why clients need know host behind gateway? I think clients shouldn't know host behind gateway. In which Scenario, will we use the first way?
Thinks

when using multiple apis, each on their own domain? served by a single Kong cluster
the new 0.10 release allows more flexibility by combining http methods, hosts and request paths.


Changes to app not showing even after refresh.

I am following screencast on getting started from laravel-angular.io. I followed exact steps but for some reasons whenever i change the code in my source code, the change is not reflected in browser even after refresh and 're-serving'.
When i try to use gulp && gulp watch, i get an error
module.js:85
throw e;
^

SyntaxError: Error parsing C:\xampp\htdocs\laravel5-angular-material-starter\nod
e_modules\gulp\node_modules\orchestrator\node_modules\end-of-stream\package.json
: Unexpected token

Hi @yunus89
I just tried the setup and I didn't have this error..
Could you try removing your node_modules folder and installing it again?
Let me know if that fixes it for you

problem fixed


[feature request] react/jsx-newline

Hi,
I would find it interesting to provide a new option to the react/jsx-newline rule, so that there is a newline between JSX expressions ONLY if they are multiline.
I find that single line JSX expressions are easier to read and don't require a space between them, but multiline expressions can quickly become hard to read.
Would anyone be eager to help me implement this feature, as this would be the first rule I would contributing to the project?
Thanks in advance!

Can you give an example of a single-line component? I'm having trouble imagining when a component would be so simple that it's best expressed as a single line.

I realise now an example would have helped, here it is:
const MyComponent = () => {
  return (
    <div>
      <OneLineComponent />
      <AnotherOneLineComponent prop={prop} />

      <MultilineComponent
        prop1={prop1}
        prop2={prop2}
      />

      <OneLineComponent />
    </div>
  );
}

I was more talking about JSX expressions than components, sorry again for the confusion.

ahhh ok, thanks. That sounds like a great option to me.

Would anyone be up to help me tackle this?

I'm happy to review a PR, and give guidance if you have specific questions.

@TildaDares go for it! I think we'd want to keep the same behavior by default, but add a new option that ignored single-line jsx expressions.


Header stack is not handled in LLVM IR

For headers
header abc {     int<32> a;     bit<16> b; } struct Headers {     hdr h;     hu hu1;     abc[2] xyz; }
Generated LLVM IR type should have array of structs, instead it has only struct type(%struct.abc).
%struct.Headers = type { %struct.hdr, %struct.hu, %struct.abc }

This is a false issue. I did not compile with p4c-llbmv2-ss.


Add support for n-th roots, e.g. \sqrt[3]{2}?

This doesn't seem to be supported right now?
julia> texparse(raw"\sqrt[3]{2}")
ERROR: TeXParseError: end of string reached with unfinished argument_gatherer
at position 10 (string index 10)
\sqrt[3]{2}
         ^
Stack before
...

It is indeed not supported.
It is on my list, but since it requires the support for optional argument in commands, I will need to add some rules to the parser and it is always a bit tricky.


Standard deviation implementation on Measure-Object

PR Summary

This continues the work done by @dfinke on pull request #4969. It removes the rounding before displaying the standard deviation, and addresses comments by @iSazonov and @SteveL-MSFT about removing code due to different parameter sets. It passes Measure-Object.Tests.ps1.
PR Checklist

[x] PR has a meaningful title

Use the present tense and imperative mood when describing your changes

[x] Summarized changes
[x] Change is not breaking
[x] Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
[x] This PR is ready to merge and is not Work in Progress.

If the PR is work in progress, please add the prefix WIP: to the beginning of the title and remove the prefix when the PR is ready.

User-facing changes

[ ] Not Applicable
OR
[x] User-facing Documentation needed

[ ] Issue filed - Issue link:

Testing - New and feature

[ ] Not Applicable or can only be tested interactively
OR
[x] Make sure you've added a new test if existing tests do not effectively test the code changed

[ ] Add [feature] if the change is significant or affects feature tests

I have some questions about the implementation

should we implement "sample standard deviation" or "corrected sample standard deviation"? I'd expect first
should we implement "one pass" algorithm without store all values (more cpu load)? I'd vote for "one pass"
should we throw if StdDeviation=$true and Average=$false? I would consider the presence of individual parameters as a request for output values. These values are nullable. So if we set -Count we want to get the count in output. If we set -Average we want get the average in output. And so on. It doesn't affect what we do internally. So -Average can assume that we do count.

@tylerl0706 I suppose @vors is leading this. I believe he "translate" strange requests from maintainers and community 😄 If students have questions - feel free to ask - you are protected by Hack Illinois 😄

Need to do an experiment to prove this, but my gut feeling is that one-pass implementation will suffer considerably from floating point arithmetic.
I think that memory allocation is ok, if user explicitly asked for standard deviation information. PowerShell often operates with large in-memory collections and this place hardly can be a bottleneck.

My concern is that PowerShell is too eats in all subsystems. I think we should think about being more effective every time.
For the feature we haven't source scenario so we can only do general reasoning. What I have is thousands of objects for which I can count the deviation. Not too much. There's one more task out of millions of objects but this is a database and I can use SQL instead PowerShell.

I think original concerns about the arithmetic precision are invalid.
I run this code to calculate sd(1:100000)^2
$s2 = 0.0
$x = 0.0
$n = 100000
$data = 1..$n
foreach ($i in 1..$n) {
    $xPrev = $x
    $x = $xPrev + ($data[$i-1] - $xPrev) / $i
    if ($i -gt 1) {
        $s2 = ($i - 2) / ($i - 1) * $s2 + ($data[$i-1] - $xPrev)*($data[$i-1] - $xPrev) / $i
    }
}
$s2

The output is 833341666.666671 compare to the 833341667 that R produces (R implements the precision estimation better and rounds it).
So I agree with @iSazonov : we should implement the iterative approach.
I took formulas from wikipedia https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
@CloudyDino would you be interested in rewriting the algorithm to the iterative approach?

Yeah, I'd love to give this another try with the iterative method! A bit new to GitHub, would I just delete this branch of mine and make a new one to implement it? Or should I just make a new branch without deleting this one? @vors

@CloudyDino All you have to do is make changes to the branch you're already working off of and push them up. Github will notice that you made changes and those changes will be automatically reflected here 😄

Awesome! Yes, what @tylerl0706 said: push to the same brunch and PR will be updated. More details: https://stackoverflow.com/questions/9790448/how-to-update-a-pull-request-from-forked-repo

I haven't changed the tester file at all and I'm not sure if I'm failing the test if I wrote my code wrong or if the iterative method needs that extra wiggle room. I've been trying to figure this out for a bit. Am I failing anything besides the tests in PowerShell/test/powershell/Modules/Microsoft.PowerShell.Utility/Measure-Object.Tests.ps1 ?

@CloudyDino to find out what exactly failed, you can click the "Details" links next to the red crosses. And then dig thru logs a little bit. In our case we can see the following:
https://ci.appveyor.com/project/PowerShell/powershell/build/v6.1.0-preview.8336/tests
Expected .00000000000001 to be less than the actual value, but got 0.0280303239036055.

Since $testObject = 1,3,4 we really should not get that big of the difference in the methods. I think the implementation has a bug. Let me take a look.

One more request: can you, please, add a new test for something like 1..100 (should be ~ 29.01149), since we have a non-trivial algorithm it would be good to have more than 1 test for it.

This is getting closer to be mergeble imo. Can somebody else take a quick look?

@SteveL-MSFT Here we enhance a public structure - formally we need PowerShell Committee conclusion.

I realize that for some reason the tests started failing after commit c40ab47, I'm taking a look at what I did to see what went wrong.

@PowerShell/powershell-committee reviewed this, we are ok with adding a new parameter to this cmdlet.  The specific implementation details should be part of the regular PR review unless there is something controversial.

 I should have checked to see if everything still worked, but standard deviation wasn't being output due to that specific commit. Trying to figure out how to go about checking when we need to put StandardDeviation out.

@iSazonov @CloudyDino thank you! 🎉

I filled https://github.com/PowerShell/PowerShell-Docs/issues/2183 for the help follow-up.

So happy to see this get merged :) nice @CloudyDino!


feat: refactor module port with config from workspace

What type of PR is this?
/kind feature
What this PR does / why we need it:
refactor module port with config from workspace
Which issue(s) this PR fixes:

Fixes #683

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

45 of 62 (72.58%)  changed or added relevant lines in 1 file are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.2%) to 75.179%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pkg/modules/generators/workload/network/ports_generator.go
45
62
72.58%

Files with Coverage Reduction
New Missed Lines
%

pkg/modules/generators/workload/network/ports_generator.go
1
64.22%

Totals

Change from base Build<PHONE_NUMBER>:
0.2%

Covered Lines:
7675

Relevant Lines:
10209

💛  - Coveralls


Sniff::$unslashingSanitizingFunctions: add doubleval() and count()

While doubleval() is an alias of floatval() and shouldn't be used, for the purposes of the ValidatedSanitizedInput sniff, both functions should be recognized.
And as count() doesn't actually access the data in the variable, but only counts the number of elements, it is also safe to use without unslashing/sanitizing the variable beforehand.
Same goes for the sizeof() alias of count().
Includes unit tests.
Fixes #1659

@GaryJones Can I merge this ?

Yes - when I approved, the tests hadn't finished running.

Not sure if it can be added here or needs a separate issue:
strpos and stripos also do not have to have sanitized variables, since they do not modify either (and its just useless to sanitize e.g. the HTTP_HOST when its only used in strpos)

@kkmuffme This PR has been merged already. Please open an issue.


all int64s wrapped for specific JSON parsing

FYI the brittany formatter changed some of the files, LMK if its not a style you like.

How are you invoking this formatter? I don't see any kind of config file or make command.

How are you invoking this formatter? I don't see any kind of config file or make command.

It's the default formatter in hie-vscode

I went through and fixed the formatting. I also wrote custom parsers where necessary. The parser for HexString now handles, Null and parsers it as an empty hexstring. Since its defined for use within this package it seemed like a good idea as it saves writing a bunch of custom parsers. Let me know if this doesn't work and I'll dive in and do the custom parsers.

Found another problem with our encoding, we need to use Base64 strings instead.


It stopped working

e.g. proliferated

Thanks for reporting this.
I've fixed the issue and released a new version. Please update and test it out.


Relax Octokit version dependency

Octokit v5.0.0 was released. The only change is the drop of support for EOL Ruby's.
https://github.com/octokit/octokit.rb/releases/tag/v5.0.0

Release 0.11.1 has been pushed to RubyGems.


Bind two ports or interfaces at the same time

With blaze client builder cannot bind two ports or two interfaces at the same time. I could imagine this might be a more efficient use of resources in some cases, but maybe it's by design?

Do you mean blaze server builder?  You can run two in the same app, and share an executor between them.  The selector threads wouldn't be shared, but I'm not sure you'd want that anyway.  It could help keep one port responsive if the other is underwater.  I'd be interested to hear a concrete problem this is causing, but I think just running two is probably what you want in the general case.

@rossabaker
Then do I add two bindHttp one for each interface? It doesn't work when I try for example
     exitCode <- BlazeServerBuilder[F]
        .bindHttp(8080, "::1")
        .bindHttp(14015, "<IP_ADDRESS>")
        .withHttpApp(finalHttpApp)
        .serve

or
      exitCode <- BlazeServerBuilder[F]
        .bindHttp(8080, "127.0.0")
        .bindHttp(14015, "<IP_ADDRESS>")
        .withHttpApp(finalHttpApp)
        .serve
    } yield exitCode
  }.drain

It only seems to serve on one of my interfaces. You can reproduce by making the changes to below.
https://github.com/drocsid/https_example/blob/aab787f5ce12e621002f1e661a2972f63df351c8/src/main/scala/com/example/weatherdemo/WeatherdemoServer.scala#L50
Then maybe it's a limitation

We should probably document this, but what you need to do is something like this:
(this has not been compiled)
def twoServers(app: HttpApp[IO]): Stream[IO, ExitCode] = {
    val server1 = BlazeServerBuilder[IO]
        .bindHttp(8080, "<IP_ADDRESS>")
        .withHttpApp(app)
        .withoutBanner
        .serve
    val server2 = BlazeServerBuilder[IO]
        .bindHttp(8081, "<IP_ADDRESS>")
        .withHttpApp(app)
        .withoutBanner
        .serve

    fs2.Stream(server1, server2).parJoinUnbounded.as(ExitCode.Success)
}

As both servers built with .serve from BlazeServerBuilder will be Stream[F, ExitCode]. There are already merge, mergeHaltBoth, mergeHaltL and mergeHaltR for one to chose from.
So, server1.mergeHaltBoth(server2).compile.drain.as(ExitCode.Success) should do the trick.

There is also a .resource on the server builder, which is handy if your server depends on other resources.  And to keep the servers alive until termination, .use(_ => IO.never).  Something like (pseudocode):
(for {
  pool1 <- makePool
  pool2 <- makePool
  server1 <- BlazeBuilder.bindHttp(8080).withHttpApp(blah(pool1)).resource
  server2 <- BlazeBuilder.bindHttp(8081).withHttpApp(blah(pool2)).resource
} yield ()).use(_ => IO.never).as(ExitCode.Success)

Thank you. I will try against https://github.com/drocsid/https_example .  Regarding documentation, I will look and see if it looks appropriate and succinct in an existing area.

@drocsid Just to clarify on the builder. It returns a new value each time you call a method on it.
Which means that if you do builder.bindHttp(8080, "::1").bindHttp(14015, "<IP_ADDRESS>"), then the second bindHttp will overwrite the first one.

@IndiscriminateCoding has shared a jetty4s project, in which multiple servers are returned to reflect multiple bound ports.  This is natural in Jetty, which supports multiple connectors.  It has never been considered natural in blaze.  And I think the jetty4s use case might be better served if a Server had a NonEmptyList of ports.  But something to consider, now that another backend is doing it.

Milestoning to decide whether we want a NonEmptyList of ports before 1.0.


graphite documentation aggregation needs update

The docs show to use a regex which does not catch all of the "sum_95" "sum_99" metrics that stats.timers emits
https://github.com/etsy/statsd/blob/cluster_proxy/docs/graphite.md
perhaps
[max]
pattern = \.(max_..|max)$
xFilesFactor = 0.0
aggregationMethod = max

I wasnt looking at master nevermind..
\.upper(_\d+)?$

Should they all have the same regex as upper?

@tehlers320 could you explain your question?

I guess it is, should all of the timer metrics with 99th percentile also have the same regex as upper?
so for example count_99.wsp will get an average by graphite instead of a sum.

That really depends on what you want to do with the data. Average is a sensible default, but if you want to sum or max the counts over the time period, that's able to be your implementation choice


Firestore, JSON interop issue

I'm having an issue with JSON/Firestore interoperability.
I have a webapp that receives JSON POST requests:
{
   "name": "Peter",
   "age": 20
}

I have a struct that I use to unmarshal these payloads:
type User struct {
  Name string `json:"name,omitempty"`
  Age int     `json:"age,omitempty"`
}

Once unmarshal'ed  I may want to save the struct to Firestore. But to handle this properly I need a struct that has firestore:"" options. Otherwise the values gets saved with incorrect field names (Name and Age instead of name and age). How do we handle such use cases where JSON and Firestore support need to be used together?

Similarly what if I want to unmarshal a struct populated from Firestore into JSON?

You can have multiple key-value pairs in a tag (https://golang.org/pkg/reflect/#StructTag). So write
Name string `json:"name,omitempty" firestore:"name"`


update Fbe::FakeOctokit

@yegor256 I added the fake data to the Fbe::FakeOctokit for this PR https://github.com/zerocracy/judges-action/pull/286

@Suban05 thanks!

@yegor256 Can you release a new version, please?

@Suban05 🎉 Great work on your contribution! You've earned +5 points, bringing your total to +417. While we had to deduct some points for excess code and lack of review, we appreciate your effort. Keep the contributions coming and focus on quality and collaboration for even better results next time!


fix(data-change-target): roll back to use begin/end updates instead o…

…f buggy performBatchUpdates.
@stephanecopin please consider rolling back to using deprecated begin/end update approach instead of the performBatchUpdates which is probably needs to be fixed later given some existing radars and other reports from many devs.

Works for me, it's pretty buggy indeed


chore: fix a slip-up in readme

@aladdin-add Can you move the "Ignore comment" section under printing where it should be :)

https://github.com/kataw/kataw/pull/147


Remove unused forward declarations and atoms

:notebook_with_decorative_cover: Description
We have a lot of them that aren't used anywhere.

Thanks for approving. I decided to keep the atoms add, connect, list, and start, because they are generic enough to be useful in plugins.


Small fixes

Thank you @green-coder! The proposed change makes sense to me, it's more generic than log.SetOutput(os.Stderr) which is just a default. I noticed the PR after changing a bunch of things but I can cherry-pick your commits to https://github.com/saucelabs/sypl/pull/5


Suggestion: More fields

Since you guys are filtering the data I think you should provide more data outside, Polkachu did a tremendous work with those.
What is missing that I think is a must have:

gov link to the upgrade
block link to the explorer block (like mintscan etc)

Nice to have:

Crawl the body of the gov and extract the repo link if any or attempt to find the tag within github
Short guide to upgrade the node
Cosmo visor folder

This also mentioned here: #24


[sparsegplvm] added sparsegplvm and tests for minibatch sparsegplvm

Added tests for SparseGPLVM and SparseGPLVMMinibatch

Current coverage is 37.98%

Merging #322 into devel will increase coverage by +0.17% as of b0347c5

@@            devel    #322   diff @@
=====================================
  Files         166     166
  Stmts       15250   15228    -22
  Branches     2218    2215     -3
  Methods         0       0
=====================================
+ Hit          5767    5784    +17
+ Partial       232     222    -10
+ Missed       9251    9222    -29

Review entire Coverage Diff as of b0347c5

Powered by Codecov. Updated on successful CI builds.


python3.pkgs.lsprotocol: 2022.0.0a10 -> 2023.0.0a1

Description of changes
Python code in lsprotocol repo was moved into subdirectory https://github.com/microsoft/lsprotocol/tree/main/packages/python
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 224952 run on x86_64-linux 1

17 packages built:

cmake-language-server
cmake-language-server.dist
python310Packages.jedi-language-server
python310Packages.jedi-language-server.dist
python310Packages.lsprotocol
python310Packages.lsprotocol.dist
python310Packages.pygls
python310Packages.pygls.dist
python311Packages.jedi-language-server
python311Packages.jedi-language-server.dist
python311Packages.lsprotocol
python311Packages.lsprotocol.dist
python311Packages.pygls
python311Packages.pygls.dist
vimPlugins.YouCompleteMe
vscode-extensions.ms-python.python
ycmd


Spacekookie/tcp transport examples

Some minor changes used by the example added by this PR: https://github.com/ockam-network/website/pull/390

Merged as https://github.com/ockam-network/ockam/pull/1085


Readme update

Opening pr

@Exxonnnnnn review readme please

This was meant to be "off" not "of" lmao, "on the" works too

I know it was meant to be off, I'd say "on the" works better / more formal. Maybe a UK / American thing idk.


WIP: Add type parameter of event subtype for app.message listeners #796

Summary
This pull request shows a possible improvement for #796 use case - better support for message event subtypes in TypeScript. I'm still wondering if there may be a better solution than this one. I would love to get inputs and feedback from anyone who is interested in this topic.
Requirements (place an x in each [ ])

[x] I've read and understood the Contributing Guidelines and have done my best effort to follow them.
[x] I've read and agree to the Code of Conduct.

converted to draft for safety

As we are aiming to release v3.4.0 within a few days and this pull request needs more time, let me move this one to v3.5 milestone.

Let me close this long-lived but incomplete PR. If someone would like to reuse the code changes in the future, go ahead with it! 🚀


Add php version to valet links command

> valet links
+-------+-----+--------+--------------------+-----------------+
| Site  | SSL | Php    | URL                | Path            |
+-------+-----+--------+--------------------+-----------------+
| site1 |     | 8.0.17 | http://site1.local | D:\_Sites\site1 |
| site2 |     |        | http://site2.local | D:\_Sites\site2 |
+-------+-----+--------+--------------------+-----------------+

Thanks @damsfx I'll have a look at this this weekend

When specifying a site to use a specific version with valet use "8.1.8" --site="my-project", is there any way to check which php version those isolated sites are using?

@yCodeTech I've just discovered this post and your fork ... It's just great. 😍
I think this project is dead ... but your fork is top notch.


Issue with iPhone 6s

 not opening on iPhone 6S. Works fine on Android but nothing on iOS. Any suggestions please?!?

used the href="" to call the modal instead and this rectified the issue. Thanks for a great plugin ;)


updated charts to fit with new design

issue: #379

Looking good. There's a couple of small details still needed from the last list above (the ones unticked) and also:

[ ] Slightly more padding below the numbers (or, check they're vertically centered)
[ ] There's a hairline above the arrow in the bubble - I'll see if a white top border fixes it
[ ] The number in the bubble should be bold, styled like the chart headers

Will check code now


[PE-D][Tester D] Optional fields not wrapped in square brackets

Why not add n/NAME j/POSITION d/DEPARTMENT [p/PHONE] [e/EMAIL] [a/ADDRESS] [l/LEAVE] [t/TAG]… if in the command format the square bracket has been defined as an optional field?

Labels: type.DocumentationBug severity.Medium
original: TheSoggy/ped#5

Accepted, will fix with #157.

Fixed in #157.


Adding optional string array of topics to publish gitlab action

Hey, I just made a Pull Request!
Adding the ability to pass (an optional) array of strings that will be applied to the newly scaffolded repository as topic labels. This is something we had to build a custom scaffolder action for since all our repos are required to have certain topics applied to them. It seemed like this should be something that was possible in the built-in scaffolder actions instead of something exclusive to us.
:heavy_check_mark: Checklist

[x] A changeset describing the change and affected packages. (more info)
[x] All your commits have a Signed-off-by line in the message. (more info)

Not quite sure if / where to update documentation for this.
Similarly, looking at the accompanying tests, I'm not sure if / how I could mock a response to test this change.
Any guidance for these points would be very welcome.

This is ready for another review. 😀

@freben Will it be possible to get this reviewed again before the end of the month for Hacktoberfest? If not no biggie.


customComponent not support image or text as children

@mohamed-sultan Hi, please check this out https://github.com/KirillGudkov/in-app-message-example
images works fine


Spam click on element!!!

Hello everyone, do I have one such bug?
Many times I click on the element that I will be dragging, and next to it I have an element with the "draggable-mirror" class ...
Who knows what the problem might be? Help me please!

Delete "plugins: [Plugins.ResizeMirror]", and all good XD


Try setting calcMode to auto for mac

This works on Windows Excel and LibreOffice on Linux so don't know why it doesn't on Mac.
Resolves #51
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Works on Mac


Parsing time error

Thanks for the script!
Although, I cannot get it to work. Something to do with the system time?
./letsencrypt-expiring-certs -expire="date"
2016/03/23 16:50:35 parsing time "wo mrt 23 16:50:35 CET 2016" as "Mon Jan _2 15:04:05 MST 2006": cannot parse "wo mrt 23 16:50:35 CET 2016" as "Mon"
./letsencrypt-expiring-certs -expire="date --date='2 weeks'"
2016/03/23 16:50:47 parsing time "wo apr  6 17:50:47 CEST 2016" as "Mon Jan _2 15:04:05 MST 2006": cannot parse "wo apr  6 17:50:47 CEST 2016" as "Mon"
./letsencrypt-expiring-certs -expire="date --date '03/05/2016 11:28'"
2016/03/23 16:50:52 parsing time "za mrt  5 11:28:00 CET 2016" as "Mon Jan _2 15:04:05 MST 2006": cannot parse "za mrt  5 11:28:00 CET 2016" as "Mon"
uname -a
Linux vps 4.3.0-0.bpo.1-amd64 #1 SMP Debian 4.3.5-1~bpo8+1 (2016-02-23) x86_64 GNU/Linux
locale
LANG=nl_NL
LANGUAGE=
LC_CTYPE="nl_NL"
LC_NUMERIC="nl_NL"
LC_TIME="nl_NL"
LC_COLLATE="nl_NL"
LC_MONETARY="nl_NL"
LC_MESSAGES="nl_NL"
LC_PAPER="nl_NL"
LC_NAME="nl_NL"
LC_ADDRESS="nl_NL"
LC_TELEPHONE="nl_NL"
LC_MEASUREMENT="nl_NL"
LC_IDENTIFICATION="nl_NL"
LC_ALL=
cat /etc/issue.net
Debian GNU/Linux 8

Hey!
Could you try by using the following format: '%a %b %d %T %Z %Y'?
./letsencrypt-expiring-certs -expire="$(date +'%a %b %d %T %Z %Y' --date='2 weeks')"
./letsencrypt-expiring-certs -expire="$(date +'%a %b %d %T %Z %Y' --date='3/05/2016 11:28')"
and so on

Actually maybe it will be still wrong, because the weekday and month name will be different in you locale. Let me know, and if that's the case I will add another date format to parse.

Thanks for your quick reply. I tried it, but unfortunately still the same error:
./letsencrypt-expiring-certs -expire="$(date +'%a %b %d %T %Z %Y' --date='2 weeks')"
2016/03/23 17:24:06 parsing time "wo apr 06 18:24:06 CEST 2016" as "Mon Jan _2 15:04:05 MST 2006": cannot parse "wo apr 06 18:24:06 CEST 2016" as "Mon"
./letsencrypt-expiring-certs -expire="$(date +'%a %b %d %T %Z %Y' --date='3/05/2016 11:28')"
2016/03/23 17:24:12 parsing time "za mrt 05 11:28:00 CET 2016" as "Mon Jan _2 15:04:05 MST 2006": cannot parse "za mrt 05 11:28:00 CET 2016" as "Mon"
But... I found that lets encrypt-auto supports renew now of all the domains, just like your script.
I entered this in my crontab:
30 2 * * 1 /root/letsencrypt/letsencrypt-auto renew >> /var/log/le-renew.log
35 2 * * 1 /bin/systemctl reload nginx
See step 4 here: https://www.digitalocean.com/community/tutorials/how-to-secure-apache-with-let-s-encrypt-on-ubuntu-14-04
So, I think I don't need your script anymore, but I'm happy to keep helping here.

Thank you, good to know! :)


How many smart contract problems can Mythril detect?

Hello, I am a graduate student from Hohai University. I want to know how many smart contract problems Mythril can detect today. I haven't found any descriptions on the GitHub page or mythx. io website, so I'm asking you a question here. If I have omitted an existing description, please let me know. Sorry to disturb your life, I wish you a happy life.

Hi @xf97, the links below might be helpful.
Problems which mythril detects --> https://mythril-classic.readthedocs.io/en/master/module-list.html
Problems which MythX detects --> https://mythx.io/swc-coverage

Thank you for your reply, which has helped me, and I wish you a happy life!
At 2019-09-12 14:58:26, "Nikhil Parasaram"<EMAIL_ADDRESS>wrote:
Hi @xf97, the links below might be helpful.
Problems which mythril detects --> https://mythril-classic.readthedocs.io/en/master/module-list.html
Problems which MythX detects --> https://mythx.io/swc-coverage
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Excuse me, I haven't found any documents or papers describing Mythx and mythril detection methods and problem determination criteria. Could you tell me where I can get them?
At 2019-09-12 14:58:26, "Nikhil Parasaram"<EMAIL_ADDRESS>wrote:
Hi @xf97, the links below might be helpful.
Problems which mythril detects --> https://mythril-classic.readthedocs.io/en/master/module-list.html
Problems which MythX detects --> https://mythx.io/swc-coverage
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Mythril keeps changing dynamically, it currently doesn't have a paper which explains it's full working.
MythX is closed source and it uses some other tools too like harvey(https://arxiv.org/pdf/1905.06944.pdf, but this tool changed a lot from the time this paper was published), Maru.
I think it should be possible to go through the analysis modules to understand how Mythril detects various vulnerabilities(https://github.com/ConsenSys/mythril/tree/develop/mythril/analysis/modules), If you have any query regarding the working you can message us on discord or simply message here.

Hello, could I ask the difference between Mythril and Mythx?

MythX uses Mythril and other non open source tools like a static analysis tool- Maru and a greybox fuzzer- harvey, so it detects a wide range of vulnerabilities. You can look into here https://medium.com/consensys-diligence/the-tech-behind-mythx-smart-contract-security-analysis-32c849aedaef for more details

I will close this for now, as the question seems to be answered. If you have another question, feel free to open another issue.


Outgoing artifacts registered on custom configurations shouldn't be built by :assemble

Once registered, outgoing artifacts on custom configurations are part of :assemble dependencies. This makes it impossible for plugins to contribute artifacts that should only be built if other participants (tasks, projects etc..) depend on them.
Expected Behavior
Registering outgoing artifacts on custom configurations should not lead to added task dependencies for :assemble unless explicit dependencies are added.
Current Behavior
All outgoing artifacts of all configurations are built when running :assemble.
Context
Writing a plugin that registers several artifacts on several configurations that shouldn't be built unless they are explicitly depended on by consumers.
Steps to Reproduce
build.gradle.kts
plugins {
    base
}
val myTask by tasks.registering(Jar::class) {
    // ...
}
val myConf by configurations.registering {
    outgoing.artifact(myTask)
}

Run:
user@host $ ./gradlew assemble --console=verbose
> Task :myTask
> Task :assemble

BUILD SUCCESSFUL in 1s
1 actionable task: 1 executed

:myTask should not be executed.
Your Environment
Gradle 6.6 and 6.x

A workaround is to do the following:
configurations.archives {
    artifacts.clear()
}

But it feels a bit too much.
I tried other ways to more selectively remove artifacts from archives without success.
For example:
configurations.archives.artifacts.removeAll(configurations.myConf.outgoing.artifacts)

has no effect.

Closing as a duplicate of https://github.com/gradle/gradle/issues/6875


cadvisor processes

cAdvisor provides information about processes of each container. Could heapster get this information?

This sounds like out of scope for kubernetes monitoring.