Security Concerns in the Pinot Docker Image and Java Depedencies

Pinot is falling behind on image and dependency security. In addition to #10274, I have two other high-level concerns that are going to affect security conscious organizations ability to run Apache Pinot on production infrastructure.
I'm curious to hear the team's thoughts on the issues below.
docker image concerns
pinot-base image is currently based on the openjdk image which is "officially deprecated" and for "non-production builds". Our security software is noting vulnerabilities that would likely be fixed from changing the base image.
Pinot users should be able to solve image layer issues easily themselves by either building or pulling artifacts directly into an image of their choice. This responsibility does not fall entirely on Apache Pinot, but if pinot-base continues to use openjdk, I do believe non-production should be specified where ever the image is mentioned.
Apache Pinot Java Dependencies
Apache Pinot java dependencies themselves are probably the greater cause for concern. When pulling artifacts from the 0.12.0 release into our image, our security scan is flagging the CVEs listed below.
Of course the responsibility does not fall entirely on the maintainers of the open source project. I have created #10304 to get a feel for contributing. I do have concerns that some of these deps upgrades are non-trivial and will likely require intimate knowledge of the project, or at least more java knowledge than I currently possess.
15:47:14.189 I Image Vulnerabilities
15:47:14.189 I ID             	Severity	Package Name                  	Package Version
15:47:14.189 I CVE-2022-25168 	critical	org.apache.hadoop_hadoop-hdfs 	2.10.1
15:47:14.189 I CVE-2022-23305 	critical	log4j_log4j                   	1.2.17
15:47:14.189 I CVE-2021-37404 	critical	org.apache.hadoop_hadoop-hdfs 	2.10.1
15:47:14.189 I CVE-2020-9548  	critical	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-9547  	critical	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-9546  	critical	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-9493  	critical	log4j_log4j                   	1.2.17
15:47:14.189 I CVE-2020-8840  	critical	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2019-20330 	critical	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2019-17571 	critical	log4j_log4j                   	1.2.17
15:47:14.189 I CVE-2019-17531 	critical	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2019-16943 	critical	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2019-16942 	critical	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2018-7489  	critical	com.fasterxml.jackson.core_jackson-databind	2.4.0
15:47:14.189 I CVE-2019-20445 	critical	io.netty_netty                	3.10.6
15:47:14.189 I CVE-2019-20444 	critical	io.netty_netty                	3.10.6
15:47:14.189 I CVE-2022-26612 	critical	org.apache.hadoop_hadoop-common	2.10.1
15:47:14.189 I CVE-2022-25168 	critical	org.apache.hadoop_hadoop-common	2.10.1
15:47:14.189 I CVE-2021-37404 	critical	org.apache.hadoop_hadoop-common	2.10.1
15:47:14.189 I CVE-2022-23307 	high    	log4j_log4j                   	1.2.17
15:47:14.189 I CVE-2022-23302 	high    	log4j_log4j                   	1.2.17
15:47:14.189 I CVE-2021-33036 	high    	org.apache.hadoop_hadoop-hdfs 	2.10.1
15:47:14.189 I CVE-2021-25642 	high    	org.apache.hadoop_hadoop-hdfs 	2.10.1
15:47:14.189 I CVE-2020-11113 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-11112 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-11111 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-10969 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-10968 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-10673 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-10672 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2021-20190 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-36189 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-36188 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-36187 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-36186 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-36185 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-36184 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.189 I CVE-2020-36183 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-36182 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-36181 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-36180 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-36179 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-35728 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-35491 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-35491 	high    	com.fasterxml.jackson.core_jackson-databind	2.4.0
15:47:14.190 I CVE-2020-35490 	high    	com.fasterxml.jackson.core_jackson-databind	2.4.0
15:47:14.190 I CVE-2020-35490 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-24750 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-24616 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-14195 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-14062 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-14061 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-14060 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-11620 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-11619 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-10650 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-10650 	high    	com.fasterxml.jackson.core_jackson-databind	2.4.0
15:47:14.190 I CVE-2022-42004 	high    	com.fasterxml.jackson.core_jackson-databind	2.4.0
15:47:14.190 I CVE-2022-42004 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2022-42004 	high    	com.fasterxml.jackson.core_jackson-databind	2.12.7
15:47:14.190 I CVE-2022-42003 	high    	com.fasterxml.jackson.core_jackson-databind	2.4.0
15:47:14.190 I CVE-2022-42003 	high    	com.fasterxml.jackson.core_jackson-databind	2.12.7
15:47:14.190 I CVE-2022-42003 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2022-41881 	high    	io.netty_netty-all            	4.1.79
15:47:14.190 I CVE-2022-41881 	high    	io.netty_netty                	3.10.6
15:47:14.190 I CVE-2022-41881 	high    	io.netty_netty-codec          	4.1.79
15:47:14.190 I CVE-2021-37137 	high    	io.netty_netty                	3.10.6
15:47:14.190 I CVE-2021-37136 	high    	io.netty_netty                	3.10.6
15:47:14.190 I CVE-2020-36518 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.190 I CVE-2020-36518 	high    	com.fasterxml.jackson.core_jackson-databind	2.4.0
15:47:14.190 I CVE-2020-25649 	high    	com.fasterxml.jackson.core_jackson-databind	2.9.10
15:47:14.191 I CVE-2019-16869 	high    	io.netty_netty                	3.10.6
15:47:14.191 I CVE-2021-22573 	high    	com.google.oauth-client_google-oauth-client	1.31.0
15:47:14.191 I CVE-2022-45693 	high    	org.codehaus.jettison_jettison	1.1
15:47:14.191 I CVE-2022-45685 	high    	org.codehaus.jettison_jettison	1.1
15:47:14.191 I CVE-2022-40150 	high    	org.codehaus.jettison_jettison	1.1
15:47:14.191 I CVE-2022-3510  	high    	com.google.protobuf_protobuf-java	3.19.2
15:47:14.191 I CVE-2022-3509  	high    	com.google.protobuf_protobuf-java	3.19.2

Any traction on this?

Thanks for refreshing this up @npandeyaof. IMHO, we should address it quickly (at least the transitive dependencies). Any interest in contributing?

I did a quick scan and do see several of these had already been updated a while back @andscoop @npandeyaof

Thanks for refreshing this up @npandeyaof. IMHO, we should address it quickly (at least the transitive dependencies). Any interest in contributing?

Hey @mayankshriv  I would be interested. There are new cve's popping up regularly as well.

All the vulnerable dependencies are upgraded. Closing this ticket


Downgrades to old versions of required packages upon installation

The versions of the packages in requirements.txt are fixed with == .
This leads to the replacement of current versions of this packages with old ones upon installation of newspaper with pip install newspaper.
I would plead for using >= syntax for all required packages unless there is an important reason to use a specific version on single requirements. An nice example requirements.txt can be found here: https://github.com/scikit-image/scikit-image/blob/master/requirements.txt

Good idea, doing this now. (Originally done with == to ensure no breakage on current setups, but that clearly isn't working with many recent issues being around version numbers).

+1
I can add this quickly (current versions and up should work).

Go for it @yprez

@codelucas ping


"Open with OniVim2"

When I attempt to open a file with OniVim2, the app launches but the file isn't opened. Instead I'm left with a blank "untitled" document. Is this currently expected to work?
I'd love to be able to use Oni as my default editor for .txt and .svg files, even at this early stage.

Thanks for opening the issue and the gif, @iryan2 ! Can confirm this bug - we should be able to open that way.
It looks like we aren't handling the specific arguments from finder correctly in this case.

@bryphe I'll look into this. 👍

Awesome, thanks a lot for the help @KevinSjoberg !
Some ideas / things that might help:

Here's the steps to create a 'release' Onivim2.app build: https://onivim.github.io/docs/for-developers/building#release
With the esy create-release command, you'll get a _release/Onivim2.app in your oni2 folder - you can point the Open With dialog to that.

We'll likely need to add some logging to figure out why we aren't getting the commands we expect. It might help to log the arguments we get prior to parse (these are available via Sys.argv - we could log them out around here: https://github.com/onivim/oni2/blob/0a17c40a96d78b28807f432838a196453d3b247a/src/bin_editor/Oni2_editor.re#L88)
The tricky thing about these startup-issues is that we don't get to add parameters to generate a log file - so we could temporarily hard-code our logging to always log to a file for the purpose of the investigation: https://github.com/onivim/oni2/blob/0a17c40a96d78b28807f432838a196453d3b247a/src/Core/Cli.re#L109
That way, when we start the app via 'Open With', we could still get a log we could work with.
Hope that gives some ideas! Thanks again and let me know if you have questions

Oh, one other idea too - if we populate an LSItemContentTypes entry in our Info.plist - it'll help streamline the 'Open With ->' experience too:
https://developer.apple.com/library/archive/documentation/General/Reference/InfoPlistKeyReference/Articles/CoreFoundationKeys.html
We already specify a list of extensions + roles + types in our package.json: https://github.com/onivim/oni2/blob/0a17c40a96d78b28807f432838a196453d3b247a/package.json#L127
We could take that information, and then, in our release.js script (the script that handles esy create-release) - we could add it to our plist creation:
https://github.com/onivim/oni2/blob/0a17c40a96d78b28807f432838a196453d3b247a/scripts/release.js#L116

Adding this for reference, https://developer.apple.com/library/archive/documentation/Miscellaneous/Reference/UTIRef/Articles/System-DeclaredUniformTypeIdentifiers.html#//apple_ref/doc/uid/TP40009259-SW1.

Unfortunately, I have not managed to get Oni2 to open the selected file yet, looking at the logs, we don't seem to get any command line arguments at all when opening a file using Open With.
Excerpt taken from the logs
[INFO]  Oni2_editor Starting Onivim <IP_ADDRESS> (cc79c6fe)
[INFO]  Oni2_editor Starting with arguments 1
[INFO]  Oni2_editor Argument: /Users/kevinsjoberg/code/oni2/_release/Onivim2.app/Contents/MacOS/Oni2_editor

Does Mac OS handle it differently?

Is there a linked ticket in Revery that we can watch?

@zbaylin has actually just started working on this for us (integrating a hook in Revery for this API, and wiring it up into Onivim 2)

@bryphe thanks for taking it over. I completely dropped the ball on this one. I recently became a father of two, so I've been busy with life. 🙂
Is there anything I can do here or is it pretty much done?

@KevinSjoberg - no worries, my apologies - I think you were blocked and waiting on me. Thanks for exploring this and checking in.

I recently became a father of two, so I've been busy with life. 🙂

Congratulations, that's so exciting! That's the good kind of busy 😄

Is there anything I can do here or is it pretty much done?

I think this is pretty much done, now - is there anything left @zbaylin , or can we close this out?

@KevinSjoberg Congratulations from me as well!
@bryphe I think we're done with this, it works well for me. If someone has a bug with the implement they can open a separate issue.
Glad to have this in!

If someone has a bug with the implement they can open a separate issue.

Sounds good, I'll close this out.


✨ Add e2e test to install and validate a Helm chart on a workload cluster

What this PR does / why we need it: Adds a test spec to install any Helm chart on a workload cluster and make sure that it the HelmReleaseProxy ready condition is true and that the Helm release status is deployed.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #172

/assign @mboersma

/hold
Since I need to squash

@mboersma Responded to comments and also added a bit more to clean up the workload cluster after the test. Once you LGTM I can squash.

/lgtm

/hold cancel


RAIR-sync error - address info not found

This issue was due to wrong service name on Kubernetes configuration. Now it's updated to rair-sync.


App stuck after closing OSM log in popup

Low prio issue I noticed while testing on https://openstreetmap.us/iD/master/, it doesn't happen on osm.org, as users are already logged on there.
After doing some edits and hitting the save button, I get a log on popup for OSM. If I just close this popup without actually logging on, most of the iD buttons are still disabled, the Save button has no effect at all. To continue from here, I have to use the browser Refresh [F5] and recover my session afterwards.

yep that's actually #3546  - and I consider it high priority! :)


Zigbee2mqtt LXC installation error

Please verify that you have read and understood the guidelines.
yes
A clear and concise description of the issue.
Trying to install Zigbee2mqtt LXC from the community-scripts. Getting this error:
"2024-12-01 09:44:28 starting update
2024-12-01 09:44:28 start download http://download.proxmox.com/images/aplinfo-pve-8.dat.asc
2024-12-01 09:44:28 download finished: 200 OK
2024-12-01 09:44:28 start download http://download.proxmox.com/images/aplinfo-pve-8.dat.gz
2024-12-01 09:44:28 download finished: 200 OK
2024-12-01 09:44:28 signature verification: gpgv: Signature made Tue Nov 19 14:13:43 2024 CET
2024-12-01 09:44:28 signature verification: gpgv: using RSA key F4E136C67CDCE41AE6DE6FC81140AF8F639E0C39
2024-12-01 09:44:28 signature verification: gpgv: Good signature from "Proxmox Bookworm Release Key<EMAIL_ADDRESS>2024-12-01 09:44:28 update successful
2024-12-01 09:44:28 start download https://releases.turnkeylinux.org/pve/aplinfo.dat.asc
2024-12-01 09:44:28 download failed: 521 Unknown code
2024-12-01 09:44:28 update failed - no signature file '/var/lib/pve-manager/apl-info/pveam-releases.turnkeylinux.org.tmp.72641.asc'
What settings are you currently utilizing?

[X] Default Settings
[ ] Advanced Settings

Which Linux distribution are you employing?
Debian 12
If relevant, including screenshots or a code block can be helpful in clarifying the issue.
If you go on the website (https://releases.turnkeylinux.org/pve/aplinfo.dat.asc), you can see the site is down.

Please provide detailed steps to reproduce the issue.
No response

Which Script? All of the Outputs are Not from our Script 🤔

Which Script? All of the Outputs are Not from our Script 🤔

https://community-scripts.github.io/ProxmoxVE/scripts?id=zigbee2mqtt

Post in Verbose mode or Screenshots. Your messages do not come from us. Something is being done with Turnkey & this is currently unavailable.

It's working again! Thank you! :)


HKG CAN FD: some cars use different message to disable LFA

Ioniq 6 does not have message CAM_0x2a4 which we use to disable LFA.

[ ] Find new message on these cars.

Do we have any routes with the cars enabling LFA, have LFA steer, then disengage LFA by pressing the button?

While it's steering? I don't think so, how adeeb found the current message was to block everything the camera sends and send our own with all zeros, and that eventually disabled LFA and greyed out the toggle in settings. I made a similar change on the Ioniq 6 branch, but that user has been MIA for a while.

From https://github.com/commaai/opendbc/pull/927#discussion_r1306900803:

We noticed that on the Ioniq 6 HDA2, the Lane Safety setting still disappeared even when we didn't block the message to suppress LFA. We confirmed this with and without blocking this message. I suspect the EV6 HDA2 might have the same behavior, so the Lane Safety setting is probably affected by something other than this message.

Information about current message: https://github.com/commaai/openpilot/pull/29658
And new message is done with 2nd Gen Kona EV port: https://github.com/commaai/openpilot/pull/28725


fix: agent-list show zone info

这个 PR 实现什么功能/修复什么问题:
修正：agent-list需要展示该agent的zone信息
是否需要 backport 到之前的 release 分支:

release/2.10.0
release/2.11
release/2.12
release/2.13

/area region
/cc @zexi @wanyaoqi

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@3508996). Click here to learn what that means.
The diff coverage is 0%.

@@           Coverage Diff            @@
##             master   #3649   +/-   ##
========================================
  Coverage          ?   6.97%
========================================
  Files             ?     499
  Lines             ?   83734
  Branches          ?       0
========================================
  Hits              ?    5840
  Misses            ?   77399
  Partials          ?     495

Flag
Coverage Δ

#aFlag
6.97% <0%> (?)

Impacted Files
Coverage Δ

pkg/compute/models/baremetalagents.go
5.75% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3508996...ad58d41. Read the comment docs.

/LGTM

/lgtm
/approve


Implements SQL statement Prepare, Execute and Deallocate

Closes #278
Description
Implements SQL statement Prepare, Execute and Deallocate as part of Prepared Statement feature.
Is it a feature that change user experience?
No
Client output
psql (12.1, server 0.0.0)
Type "help" for help.

xxx=> create schema schema_name;
CREATE SCHEMA
xxx=> create table schema_name.table_name (col1 smallint, col2 smallint, col3 smallint);
CREATE TABLE
xxx=> prepare fooplan (smallint, smallint) as insert into schema_name.table_name values ($1, $2, $3);
PREPARE
xxx=> execute fooplan(1, 2, 3);
INSERT 0 1
xxx=> deallocate fooplan;
DEALLOCATE
xxx=> select * from schema_name.table_name;
 col1 | col2 | col3
------+------+------
    1 |    2 |    3
(1 row)

Hi @alex-dukhno , please help review this PR if you have time. Thanks.

OK, I see. I will try to move src/protocol/src/pgsql_types.rs associating code to pg_model.

Pull Request Test Coverage Report for Build 313749468

188 of 244 (77.05%)  changed or added relevant lines in 6 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.06%) to 89.05%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/pg_model/src/pg_types.rs
16
26
61.54%

src/node/src/query_engine/mod.rs
82
128
64.06%

Totals

Change from base Build 312735826:
0.06%

Covered Lines:
8027

Relevant Lines:
9014

💛  - Coveralls


feat: support for msg undelegate

Why do we need MsgUndelegate (and MsgDelegate) in the neutronjsplus?

Why do we need MsgUndelegate (and MsgDelegate) in the neutronjsplus? Neutron doesn't have staking module

well, MsgDelegate was already implemented before, hence I decided to add MsgUndelegate as well.
Should I remove both now? I will have to patch all integration tests though.

Why do we need MsgUndelegate (and MsgDelegate) in the neutronjsplus? Neutron doesn't have staking module

well, MsgDelegate was already implemented before, hence I decided to add MsgUndelegate as well.
Should I remove both now? I will have to patch all integration tests though.

Ideally yes

Superseded by #19


[🐞] No multiple actions can run simultaneously

Which component is affected?
Qwik City (routing)
Describe the bug
Doing:
Trying to run() multiple actions when a button is clicked.
      <button onClick$={() => {
        foo.run();
        bar.run();
      }}>Foo Bar</button>

Expect:
These actions have run.
Happened:
Randomly selected only one action has run.
Reproduction
https://github.com/genki/qwik-test/tree/no_multi_actions
Steps to reproduce
git clone the the above and npm run dev
The reproduction code is in src/router/index.tsx
Please note this expects the latest qwik is located ~/project/clone/qwik
System Info
System:
    OS: macOS 13.1
    CPU: (8) arm64 Apple M2
    Memory: 84.31 MB / 24.00 GB
    Shell: 3.5.1 - /opt/homebrew/bin/fish
  Binaries:
    Node: 19.7.0 - /opt/homebrew/bin/node
    Yarn: 1.22.19 - /opt/homebrew/bin/yarn
    npm: 9.5.0 - /opt/homebrew/bin/npm
  Browsers:
    Chrome: 110.0.5481.177
    Firefox: 110.0.1
    Safari: 16.2
  npmPackages:
    @builder.io/qwik: file:~/project/clone/qwik/packages/qwik => 0.21.0
    @builder.io/qwik-city: file:~/project/clone/qwik/packages/qwik-city => 0.5.3
    undici: ^5.20.0 => 5.20.0
    vite: 4.0.4 => 4.0.4

Additional Information
I faced this issue as well at other situation such that several actions run at the same timing.
For example, when one signal triggers several useTask$ at once and each of them runs different actions.

Here's workaround I found for this issue:
Define the function that serializes execution of actions.
let prevAction = Promise.resolve();
export const enqueueAction = async (action, ...args) => {
  await prevAction;
  prevAction = action.run(...args);
};

Then call it instead of run(...) like this:
  // foo.run(args)
  // bar.run(args)
  enqueuAction(foo, args)
  enqueuAction(bar, args)

man, I was going nuts with this issue
anyway, this solution seemed to work for me
useVisibleTask$(async () => {
    await firstAction.submit({ data });
    await secondAction.submit({ data });
  });

which is kinda simpler

I think this is a simple and elegant solution.
Thanks @AleksKislov for sharing.

man, I was going nuts with this issue
anyway, this solution seemed to work for me
useVisibleTask$(async () => {
    await firstAction.submit({ data });
    await secondAction.submit({ data });
  });

which is kinda simpler

But this is not "simultaneously", they are sequential.
I have no context about this discussion, tho.

But this is not "simultaneously", they are sequential.

well, yes, but at least it works for some cases.
But the thing that an action can interrupt other actions is a bummer


log switch_format, mipi_syst tests failing on intel_adsp_ace15_mtpm

Describe the bug
These testcases fail to build using xtensa toolchain on intel_adsp_ace15_mtpm platform.
tests/subsys/logging/log_switch_format/logging.log_switch_format.deferred
tests/subsys/logging/log_switch_format/logging.log_switch_format.immediate
tests/subsys/logging/log_switch_format/logging.log_switch_format.custom_output
tests/subsys/logging/log_syst/logging.log_syst.mipi_syst

To Reproduce
west build -p -b intel_adsp_ace15_mtpm tests/subsys/logging/log_switch_format/
Expected behavior
Testcase needs to pass
Impact
Failing testcase
Logs and console output
In file included from zephyr/lib/libc/newlib/libc-hooks.c:12:
zephyr/include/zephyr/posix/sys/stat.h:40:10: fatal error: 'sys/_timespec.h' file not found
#include <sys/_timespec.h>
         ^~~~~~~~~~~~~~~~~
1 error generated.

Environment (please complete the following information):

OS:  Linux
Toolchain: RI-2021.7
Commit SHA: 461a13c085de73aabdb5777247a96675472a73ad

@aasthagr Same error observed for logging.log_syst.mipi_syst, so I added it to this bug instead of making you file another one

This bug isn't resolved by the linked PR.


Transparent Water

Would it be possible to edit RGB values for underwater blocks to allow transparency?
something like on hit of water, raytrace again ignoring water?
my quick and dirty proof of concept code was this
set {_r} to {_r} * 0.5
set {_g} to {_g} * 0.7
set {_b} to {_b} + 100
if {_b} is more than 255:
set {_b} to 255
but it let me see differences between sand, clay and dirt (image is only 64x64 and has a heavy noise filter)
image was not taken using imageryAPI, but much slower concept skript

Hi kuraiskrap, we already do this for translucent blocks (like glass, cobwebs, iron bars and such). You can view the configured translucent blocks here: https://github.com/jensjeflensje/minecraft_imagery/blob/374732c2759fadd9919ffe8b680bba08deb03b74/src/main/java/dev/jensderuiter/minecraft_imagery/Constants.java#L72.
It's calculated somewhat similarly to your script, by using a TranslucentBlock's dye and applying that with a certain factor.
I understand why you would want the same for water, but I'm unsure if this should be done by default, as it will probably decrease performance. It could be implemented by making the list of translucent blocks configurable.
Would you be open to contributing? Given the fact that you seem to be doing some experiments of your own :)

unfortunately my java knowledge is.... untouched since a starter course like 11 or 12 years ago
I've really only familiar with skript at this point, super happy you support it :)
if i knew java id try and contribute, was helping with the other camera skript

I don't think you'll get good results from that and it does sound somewhat... inefficient. If I have time, I can add the list of blocks to the config, like how the block color palette currently works. I'm just not sure if that's going to be very soon 😅

managed to get a proof of concept working on my fork
water has transparency values of 0 0 0, so it makes no difference dye wise
it flips a flag when it hits water for the next raytrace to ignore fluids (technically causes issues with water over lava)
it then applies the RGB value changes i mentioned to the pixel avoiding the dye system i don't fully understand

Thanks for your experiments! I'll take a look at your fork to see how I can use this to actually integrate it into the library.

how about having the distance from the first water hit to the final depth hit contribute to a darker dye. This information is already available if you run a second ray under water and will increase the fidelity quite a bit


Support epmdless runtimes

Desktop mode should run without epmd by default via LIVEBOOK_EPMDLESS=true. We will change the runtimes to be of the format livebook-runtime-PORT (we need to figure out how Erlang finds the available port. is it just random?) and implement a custom -epmd_module (such as this one) that intercepts "livebook-runtime-" nodes and extracts the port out.  Then we can use -start_epmd false.
The Elixir standalone runtime on epmdless mode would also need to set -start_epmd false and -epmd_module(which implies we would need a mechanism to share the-epmd_moduleitself), perhaps by simply pointing to itsebin`.
This should improve the Livebook Desktop experience for those behind VPN and poorly configured machines.

Closing in favor of #2591.


Add support for token-based Auth

DCOS has an adminrouter to control public access
to the Marathon API.  This requires token-based authentication.
This simple change lets a user pass in an auth_token rather
than username and password for this use-case.
Fixes #148

Thanks!


Show url in tooltips for all links

We should be nice and humble and build trust by showing the url in a tooltip for each link in the Desktop Modeler (what's new, welcome page, element template documentation, inline doc, process definition after deployment).
This also prevents us from the very rare but possible scenario of someone providing a forked Modeler build with phishing links, especially when we may link to resources in Camunda SaaS apps in the future.

Quickly checked with another App to confirm that this is expected behavior:

This also prevents us from the very rare but possible scenario of someone providing a forked Modeler build with phishing links, especially when we may link to resources in Camunda SaaS apps in the future.

I'm not sure if it is possible to automate it, as we would need to use the title attribute. With a forked Modeler, you would be able to just add a valid URL as title and a fake site as href anyway, giving the users a false sense of security

@christian-konrad Please use an existing issue template for your report. Is this a bug, is this a feature?
As an alternative for the tooltip is to simply re-implement what browsers already got:

In my opinion it is less obtrusive and users already know this.

@marstamm Does electron provide the bottom-left link preview per default?

I did a quick research and it appears it does not. But it is easy to build, using basic web technologies.

If we think this is valuable, let us see if we can tackle it as a spring cleaning topic. I've tagged it accordingly.


Fix async saving to return a promise and work

Fixes #27

Coverage increased (+7.3%) to 33.182% when pulling 6cb1fbeaf8ba2bbfeb88513abef4b483a0bf68d0 on mitchellsimoens:27_fix into 817b27308be054cb2abb246302974e18e4f55473 on EpiphanySoft:master.

Coverage increased (+7.3%) to 33.182% when pulling 6cb1fbeaf8ba2bbfeb88513abef4b483a0bf68d0 on mitchellsimoens:27_fix into 817b27308be054cb2abb246302974e18e4f55473 on EpiphanySoft:master.

awesome thx - sorry I didn't notice this for a bit :p


Latex in abstract doesn't work

The abstract
abstract = "We present a strong solution of the board game pentago, computed using exhaustive parallel retrograde analysis in 4 hours on 98304 ($3 \times 2^{15}$) threads of NERSC's Cray Edison.  At $3.0 \times 10^{15}$ states, pentago is the largest divergent game solved to date by two orders of magnitude, and the only example of a nontrivial divergent game solved using retrograde analysis.  Unlike previous retrograde analyses, our computation was performed entirely in-core, writing only a small portion of the results to disk; an out-of-core implementation would have been much slower.  Symmetry was used to reduce branching factor and exploit instruction level parallelism.  Despite a theoretically embarrassingly parallel structure, asynchronous message passing was required to fit the computation into available RAM, causing latency problems on an older Cray machine.  All code and data for the project are open source, together with a website which combines database lookup and on-the-fly computation to interactively explore the strong solution."

fails to format \times correctly: it goes through to HTML as

We present a strong solution of the board game pentago, computed using exhaustive parallel retrograde analysis in 4 hours on 98304 ($3 	imes 2^{15}$) threads of NERSC's Cray Edison.  At $3.0 	imes 10^{15}$ states, pentago is the largest divergent game solved to date by two orders of magnitude, and the only example of a nontrivial divergent game solved using retrograde analysis.  Unlike previous retrograde analyses, our computation was performed entirely in-core, writing only a small portion of the results to disk; an out-of-core implementation would have been much slower.  Symmetry was used to reduce branching factor and exploit instruction level parallelism.  Despite a theoretically embarrassingly parallel structure, asynchronous message passing was required to fit the computation into available RAM, causing latency problems on an older Cray machine.  All code and data for the project are open source, together with a website which combines database lookup and on-the-fly computation to interactively explore the strong solution.

In short, $3 \times ... becomes $3<tab>imes ....  Is there an easy way to fix latex support in abstracts?

Apologies: using \\times (two backslashes) works.  I'll leave it up to those more knowledgeable to know whether this is the ideal solution, and whether to close the issue.  Thanks for the great theme!

Thanks for using the theme, let me know if you have any other feedback!
Double quotes (") or backslashes (e.g. LaTeX \times) occurring within the value of any TOML frontmatter parameter should be escaped with a backslash (\). For example, the symbol " and LaTeX text \times become \" and \\times, respectively. This is just related to how Hugo/TOML parse the frontmatter - you can refer to the TOML documentation for more info.
I'll add a note to the documentation to clarify this :smiley:


Add unit tests

This pull request adds unit tests, located in tests/. Testing is done via a small Xcode project, created solely for the purposes of these tests.

Is Xcode and XCTest the best way to test this? A Makefile-built binary that exits 0 or 1 called from CI seems like it might be simpler than pulling in Objective-C, etc.

Is Xcode and XCTest the best way to test this?

I don't know how to define "best" here, but I was able to create it quickly and pretty much mindlessly, which was my criteria.

A Makefile-built binary that exits 0 or 1 called from CI seems like it might be simpler than pulling in Objective-C, etc.

I don't know how to do this targeting the simulator. Tests with the simulator would  be useful in testing issues like #31. Also I get what you're saying at a high level, but I don't understand the argument about pulling in Objective-C, not in the context of basic tests for fishhook.
I don't have a strong interest in tests for this project, I wanted something that would give me feedback that the two-levels namespace changes in #34 work to some degree. For example, I realized CI could be setup once tests were added, but I have no current intentions of doing that work.

For a practice C++ project I did the other day, I used googletest. It's fairly easy to set up using Git submodules, CMake, and a Makefile. In fact, I copied most of the setup from facebook/xcbuild -- only about 60-100 lines total, all very clean, no crufty Xcode XML. Thanks for the awesome example, @grp!
I think I'd recommend against using XCTest as well, but the simulator argument is enough to convince me! XCTest makes that very easy to do.

Thanks for the feedback @grp, @modocache.
To summarize:

I'm not arguing for Xcode or XCTest
I'm not arguing against simpler Makefiles
Given my time and knowledge, using Xcode is the approach I am willing to do
XCTest could be seen as providing some amount of integration testing

If anyone feels strongly against this, I'm fine to throw it away once #34 is done. I can always bring it back quickly in the future if I need/want it.

A Makefile-built binary that exits 0 or 1 called from CI seems like it might be simpler than pulling in Objective-C, etc.

I don't know how to do this targeting the simulator.

Thanks to @lawrencelomax, this can be done via xcrun simctl spawn.

I was more talking philisophically than as a practical concern with adding
any kind of tests :) Although the simctl approach sounds promising.
Sent from my iPhone
On Sep 9, 2016, at 02:33, Dave Lee<EMAIL_ADDRESS>wrote:
A Makefile-built binary that exits 0 or 1 called from CI seems like it
might be simpler than pulling in Objective-C, etc.
I don't know how to do this targeting the simulator.
Thanks to @lawrencelomax https://github.com/lawrencelomax, this can be
done via xcrun simctl spawn.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/facebook/fishhook/pull/33#issuecomment-245694698, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAJ-PquXiPD859EJjrsjx2b4SOGnRDOMks5qoFTmgaJpZM4J3sf1
.

Maybe I'll create a with-tests branch and put this there.

I've been waiting this PR be solved for some time. And really wanna know whether this PR will be merged or not. 😆

@Draveness I plan to create a branch, probably master-with-tests and keep this there.

@Megra this pull request was originally a complement to #34, I wanted to make sure I didn't break behavior. I probably won't be updating this, except if done to support #34.


Add pip topic

Thank you for contributing! Please confirm this pull request meets the following requirements:

[x] I followed the contributing guidelines: https://github.com/github/explore/blob/master/CONTRIBUTING.md
[x] I have no affiliation with the project I am suggesting (as a maintainer, creator, contractor, or employee).

Which change are you proposing?

[ ] Suggesting edits to an existing topic or collection
[x] Curating a new topic or collection

Editing an existing topic or collection
I'm suggesting these edits to an existing topic or collection:

[ ] Image (and my file is *.png, square, dimensions 288x288)
[ ] Content (and my changes are in index.md)

Please replace this line with an explanation of why you think these changes should be made.

Curating a new topic or collection

[x] I've formatted my changes as a new folder directory, named for the topic or collection as it appears in the URL on GitHub (e.g. https://github.com/topics/[NAME] or https://github.com/collections/[NAME])
[x] My folder contains a *.png image (if applicable) and index.md
[x] All required fields in my index.md conform to the Style Guide and API docs: https://github.com/github/explore/tree/master/docs

pip is package manager for Python. Its topic is used by around 700 repositories. I also added pypi (package index), pip-package and pypi-package as aliases because they are also quite popular and related to pip topic.

Please note: we will close your PR without comment if you do not check the boxes above and provide ALL requested information.

Ok... Good job


Adding new blog posts breaks blog path

Hello. Thanks for the great starter theme. Clean and simple, just what I was hoping to find.
In setting it up for my own Netlify account, I found an error.
Deleting old blog posts and adding a new one breaks the path to the blog posts from the blog page, but not from the home page.
I created an example website to demonstrate the issue.
https://gatsby-starter-foundation.netlify.app/
I'm new to Gatsby, so though I will troubleshoot, I will probably move slow, so any help here will be greatly appreciated. TIA!

2 points to clarify. I made all the edits online with the Netlify CMS, nothing local. The new blog post is reachable from the home page, but the blog page adds a "blog/" to the URL between the base and the post's slug.


How to make Anchor working with DropdownMenu.Item?

Describe the bug
I try to make a simple dropdown menu with  component on solid-start
my first try code is
<DropdownMenu.Item>
  Login
</DropdownMenu.Item>

with this code it will not navigate to /login
the second try
<DropdownMenu.Item asChild>
  <As
      component={A}
      href="/login>Login
  </As>
</DropdownMenu.Item>

its still not navigate to /login
only this way will navigate me to /login
<DropdownMenu.Item asChild onSelect={() => navigate('/login')}>
  <As
    component={A}
    href="/login">Login
  </As>
</DropdownMenu.Item>

how to make anchor link work normally without onSelect?
ps. sorry if my issue gone wrong

OS: Ubuntu22.04(WSL2 on windows 10)
Browser Microsoft Edge
Version 115.0.1901.183 (

Additional context
Add any other context about the problem here.

Hey 👋 I tried to recreate the issue but it works fine for me.
Can you give more informations about what you want to achieve and more code please?
My test
https://codesandbox.io/p/sandbox/cold-bash-hgltfl?file=%2Fsrc%2FApp.tsx%3A6%2C24

not try native a href yet only have problem with A component from solid. I'll try and update result soon. thanks

It looks like closeOnSelect might have something to do with this issue.
Specifically, if you want to include a link as one of the dropdown menu items it seems like you can do this:
<DropdownMenu.Item asChild closeOnSelect={false}> // note: closeOnSelect={false}
  <As component={A} href={"/wherever"}>
    <DropdownMenu.ItemLabel>My link</DropdownMenu.ItemLabel>
  </As>
</DropdownMenu.Item>

Here's a link to a reproduction

I had a similar issue using Kobalte in an Astro project and @andrewrosss 's solution worked for me. Thanks.

Encountering the same issue using Kobalte in an Astro project. Solved with @andrewrosss  's solution.
Many thanks, Andrew!

i have try with native anchor element(a) like you sandbox but still no luck about it. i have use solid-start not solid-vitejs(not sure bot are same or not) below is code
both component="a" or component={A} will not navigate me to new url
only this way will navigate me to new url
dependency version information
"@kobalte/core": "^0.9.8",
"@solidjs/router": "^0.8.2",
"solid-js": "^1.7.8",
"solid-start": "^0.2.29"

What font is this?


Update README.md

fixing old issue that hasn't been resolved yet

I have no idea how from here we can link it to a previous issue. Also, dont know if linking the PR to the issue and resolving the issue will auto-merge to the main-branch.


Implement custom call_command

Description
Hello, what an interesting project! I'd find it quite useful for something I am working on, where I need to load management commands from outside apps.
One thing we heavily use at work is the call_command function, for running management commands from celery for example, etc. Right now with your solution that wouldn't work yet, as that function is not aware of the customisations (sadly the management command register cannot be easily overwritten)
Suggested Solution
Similar to how a custom implementation of execute_from_command_line must be imported, same should be possible for call_command, which would be aware of these custom commands
Terms

[X] I have read and followed the project's Code of Conduct and Contributing Guide.
[X] I have checked that this issue is not a duplicate.

I have tried this:
def call_command(command_name, *args, **options):
    if isinstance(command_name, BaseCommand):
        return base_call_command(command_name, *args, **options)

    if dotted_path := settings.PATHS.get(command_name):
        command_class = import_command_class(dotted_path)
    else:
        try:
            app_label, name = command_name.rsplit(".", 1)
        except ValueError:
            app_label, name = None, command_name

        command_class = load_command_class(name, app_label)

    command_name = command_class()

    return base_call_command(command_name, *args, **options)

And that should work for when commands are included PATHS or MODULES/SUBMODULES - won't work for ALIASES I think?
There are probably more edge cases though!
I would be happy to contribute if you point me towards anything I'm missing?


Price parameter of Entities\Transaction\TransactionItem should be allowed to be null

Describe the bug
The Price parameter of the Paddle\SDK\Entities\Transaction\TransactionItem.php should be allowed to be null because the API accepts either a price_id or a price definition: https://developer.paddle.com/build/invoices/create-issue-invoices
In that case, the constructor forces us to include the "Price" even if we already have a price_id.
Steps to reproduce
Create a new TransactionItem
Expected behavior
Be able to only provide the price_id
Code snippets
No response
PHP version
PHP 8.2
SDK version
1.20
API version
Paddle Version 1
Additional context
No response

Hello :wave: Thank you for reporting this issue.
When making a transaction create request, you should use Paddle\SDK\Resources\Transactions\Operations\CreateTransaction which supports items (from v1.4.0):

Paddle\SDK\Resources\Transactions\Operations\Create\TransactionCreateItem
Paddle\SDK\Resources\Transactions\Operations\Create\TransactionCreateItemWithPrice

Example providing only price_id :
new CreateTransaction(
    items: [
        new TransactionCreateItem('pri_01he5kxqey1k8ankgef29cj4bv', 1),
    ],
);

In v1.2.0 you could use Paddle\SDK\Entities\Transaction\TransactionCreateItem but is deprecated and replaced by Paddle\SDK\Resources\Transactions\Operations\Create\TransactionCreateItem in v1.4.0


Here's a new site I'd like you to consider blocking

Here's a new site I'd like you to consider blocking:
https://einthusan.tv
(please include any other relevant details)
https://einthusan.tv/movie/browse/?lang=hindi

I've really got to figure out a better way to add these. Sorry it took so long, but I've added it now. Let me know if it doesn't look fixed on some page, and I'll try again.


micro-events : Extract logging rate limiter from Job capturing aspect

All jobs should be posted to the EventBus, the rate limiter should be pluggable on the client side.

All events are logged for the Metrics Cather, rate limiting only affects passing on the whole SystemData event that clients can listen to if they like. Rate limiting can be dialed down via system.logging.max.per.hour.
Closing this issue for now.


Angie testing something (to delete)

Expected Behavior
Actual Behavior
Steps to Reproduce the Problem

Specifications

Version:
Platform:

Please provide the user's actual question or issue description. I need the details of the "Expected Behavior," "Actual Behavior," "Steps to Reproduce the Problem," and "Specifications" to provide a helpful response.
Once you provide the complete issue description, I can:

Identify if it's a feature request, issue, known issue, or question.
Provide relevant documentation links or workarounds.
Ask clarifying questions if needed.

Please provide the user's question or issue.  I need the actual content of the issue to provide a helpful and relevant response.

Thank you for reporting this issue! To help us understand the problem better, could you please provide the following information:

What is the expected behavior you are seeing?
What is the actual behavior you are experiencing?
What are the specific steps to reproduce the problem?
What is the version of Temporal you are using?
What is the platform you are running Temporal on?

Once you provide this information, we can investigate the issue and try to find a solution for you.


Move currencies load out of construct

it will be nice to have the currencies available in a static method, so we can integrate the same currency list elsewhere by using the Currency API rather than loading the file manually (which would cause path problems).
If we add this method, the include will be always relative to this module directory, no matter if the module is the /vendor/foo/bar/Money or /vendor/foo/bar/baz/Money, plus it is fully backwards compatible, so... why not?

@mathiasverraes ping :dancers:

@mathiasverraes I've been watching your discussion with @danizord in twitter about this PR, and I totally agree with your point that you should define the application currencies somewhere instead of using the ones provided by Money. But...
The use case we're working on is a module that integrates Money with doctrine and zf2, and we would like to provide in this case the list of all available currencies in Money. So if this module is working directly with money, it makes sense to it know the money currency limitations/possibilities. It doesn't necessarily means they'll be all used in the application, and i'm not sure this limitation is up to you to force. Having a feature doesn't mean it'll be even used. The application developer will know he doesn't accept all those currencies and will probably prefer to have a list of supported ones instead of having a full list of useless things.
Although there is a method in nextrelease that makes available to ask if some currency is available or not, there is no method in the current one that does that and that is the one we're using right now. I propose you to merge this and release 1.3. I still think this could be included in both, but maybe it could be removed for the next one.
That said, i'll say again that it would be nice to have this list available so we could integrate with the money form element in our module and it would be a fallback in case the developer doesn't specify the currency list.

Currency lists have been completely removed from Money. No more currency code checking. If you need to work with a set of currencies, you have to implement Currencies interface.

@sagikazarmark That is still valid for the current release.

@fabiocarneiro Indeed it is :)

The current master is likely to be renamed to 1.x and maintained for some time. Since this PR makes sense on that branch, I am merging it.
Thank you @fabiocarneiro

@sagikazarmark many thanks! Can you tag a release for it?

Will do as soon as I get some time. Please follow #109 for details.


software stop at second pin during profile setup

Issue description
Download the appimage, controlled the sha and positive match.
setup the profile but at the second screen when I place the confirmation pin (only numeric) the software does not advance.
Firefly version
1.3.0
OS details
Operating System: MXLinux Debian GNU/Linux 11 KDE Plasma Version: 5.20.5 KDE Frameworks Version: 5.78.0 Qt Version: 5.15.2 Kernel Version: 5.10.0-9-amd64 OS Type: 64-bit Processors: 4 × Intel® Core™ i7-6560U CPU @ 2.20GHz Memory: 15.2 GiB of RAM Graphics Processor: Mesa Intel® Iris® Graphics 540
Can the issue be reliably reproduced?
Yes
Steps to reproduce the issue

I follow the step
as i get to the second screen of the pin number
after I write the 6 pin for the second time
the button on the lower left side activate but as it pressed does not forward the program to the next step (I don't know what to expect)

...
Expected behaviour
To see the next step in the profile creation
Actual behaviour
Nothing happens
Errors
No error reported

Make sure to install gnome keyring

it worked


Fix punctuation

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (62b7ded) 100.00% compared to head (c74a861) 100.00%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@             Coverage Diff             @@
##                main      #169   +/-   ##
===========================================
  Coverage     100.00%   100.00%
  Complexity       554       554
===========================================
  Files             49        49
  Lines           1700      1700
===========================================
  Hits            1700      1700

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Remove title component and make it a utility class

We would like to delete the title component and create a font size utility class.
That way if a user wants to create a title like structure they can do:

title

But it'll also allow changing the font size of other parts of the UI too.
cc/ @michael-coker what do you think?

so @andresgalante with this utility we would include size, weight and line-height?

@andresgalante More commenting for React tracking. We will need to make a Text component for this. What all props/utilities are expected to be used by this? FontSize, Weight, LineHeight, margin?

Hi @dmiller9911 you are right. since the title component does more than just font size, I've changed it to just remove the margins and use utility classes instead


Revert "Merge pull request #23 from RedbackThomson/dummy-test"

This reverts commit 439182d46e99fe081f62bac2ed2597e8a4332de6, reversing
changes made to 88f85621f5c6064b1c685fc98e925ac511f4ab07.

/unapprove

/remove-approve

/assign @vijtrip2

/approve

/lgtm


Add a functional boxplot candela component

Ideally, I imagine that we'd assume that these would be precomputed and already part of the dataset. But that would mean that we should have a mechanism in the candela API to associate secondary attributes to a primary one (e.g. set one data column to be the mean, another to be its 95% confidence interval).
Or maybe it would be better to create an entirely separate functional boxplot component that only draws one line, as opposed to the line chart component that shows an arbitrary number of lines (but with no secondary attributes)?

I want to be careful about stuffing too many features into what should be a simple component. "Line chart" right now does more or less what is written on the tin. We should think carefully about where a line chart with error bars or line chart with confidence interval ought to live.

Agreed. I'd lean toward creating a separate functional boxplot component for a single line (not just because it's easier, but from a vis perspective, a line with that much detail should probably have its own distinct chart anyway).
But I'm wondering if this is symptomatic of a deeper problem. We've run across similar questions with lat/lon stuff, where two attributes have to go together, but we might want to accept multiple pairs of attributes.
I'll close this in favor of #175 to preserve the link in case this comes up again

Did this feature request come from a user? If so, we should keep it open (possibly changing the name, demanding a new line chart type that can handle quartiles, etc.). I don't think what this issue is asking for is the same thing as #175.

I learned the term "functional boxplot" today. Excellent! :mortar_board: :mortar_board: :mortar_board:

The new component should accept one (?) attribute for the center curve (e.g. mean, median), n pairs of attributes as envelopes (e.g. lower / upper bounds of confidence intervals), and n attributes as outlier curves
Or if we decide on a different standard for things like lat/lon coordinates (e.g. accept a single number tuple attribute instead of two number attributes), the n pairs should take a similar approach.

@alex-r-bigelow Using the same approach as geo locations depends on how these confidence pairs are normally encoded "in the wild" and if they are similar to how locations are encoded. I could also see Candela or Resonant Lab being able to compute the confidence intervals from raw data - I assume they are often computed specifically for the visualization in other tools like R.

True, we could calculate confidence intervals ourselves... but I tend to want to punt on any kind of derived data. I'd rather let the user do it in another tool where they'll have more control. I think we're running with the assumption in Resonant Lab that we can set expectations for the shape of the data, because we're planning on integrating with existing wrangling tools and / or writing our own.
I agree we should try to figure out the shape that these things take most frequently in the wild, and use that. But it's usually the case that someone has data that doesn't fit the format that we/most people expect/support, so reshaping is almost always going to be necessary anyway—I don't think it's worth trying to anticipate too much.


Compressing VarBinArray to ConstantArray loses info about offsets_ptype

Hi Vortex, I am not sure this is the desired behavior. For example, if we compress a LargeBinary or LargeUtf8 Arrow Array into Vortex's ConstantArray and then canonicalize it back, we will get Binary or Utf8 Arrow Array. This is because VarBinArray::from_iter always uses the u32 offsets builder:
https://github.com/spiraldb/vortex/blob/e75606de2624a9c5b73ee0176fb56582fad9aebe/vortex-array/src/array/varbin/mod.rs#L164
This can be reproduced by running the round_trip_arrow_compressed test. It is ignored but Arrow now supports comparing Structs:
https://github.com/spiraldb/vortex/blob/e75606de2624a9c5b73ee0176fb56582fad9aebe/bench-vortex/src/lib.rs#L264-L268
The taxi dataset has a field store_and_fwd_flag which is mostly N. It is reasonable for a ConstantArray to just use u32 offset but if we have a ChunkedArray where the first chunk is Constant and the second chunk is not, we may have inconsistent Arrow schema between output RecordBatches? (while this may be the problem of Arrow missing a logical type)

Hey @XinyuZeng , thanks for taking a look! You are correct that in general, while we do our best to preserve round-tripping between Arrow arrays and the nearest Vortex encoding, after compression we can't guarantee to give you back the exact same Arrow encoding you started with. For example, if you provided a StringViewArray or BinaryViewArray, we would still give you back a StringArray / BinaryArray.
This is because of the Vortex type system encodes logical types (e.g. "these bytes are Utf8 encoded") VS the Arrow physical encoding types. Each of our logical types has a blessed "canonical" encoding which can represent values of that type while being zero-copy to Arrow.
To provide an exact round-trip guarantee would require storing information somewhere about the original Arrow encoding, which I don't think is something we're considering right now.

Got it, thanks! Still wondering will this have a potential issue when connecting to DataFusion since they are using Arrow physical schema directly. For example, we always got Utf8 in schema (https://github.com/spiraldb/vortex/blob/3bef63b6db28c8c0f26654492619cf1f417dc23f/vortex-datafusion/src/datatype.rs#L67) but the canonical Array may be LargeUtf8 (https://github.com/spiraldb/vortex/blob/3bef63b6db28c8c0f26654492619cf1f417dc23f/vortex-array/src/canonical.rs#L270-L271). Maybe a corner case though, just accidentally think of.

This is indeed an issue. This particular case would be solved by #757. I don't recall why we didn't take the schema from the array. Likely needs to add a new trait i.e. ArrowDType and implement it for all our arrays to solve this issue. Then we could produce the true canonical dtype.

@a10y pointed out that we might have enough metadata in our files to produce arrow dtypes even if we could solve it for in memory arrays

Yep we're tracking that keenly and agree it will be likely to help here 😄


[WIP] Add introduction notebook

Adding a very basic introduction notebook. Going to add some more explanatory text and figures to make this easier to read in the coming days.
Feel free to make any suggestions to spice this up some or any other ideas you'd like to see.

Thanks for setting this up! I will edit it as we go, switching to a draft PR until it is ready for review.

Added some more words. I think this is good enough for now and we can expand as the library expands

Indeed that's good enough for
now, thanks! Merging.

Thanks buddy!


Routing bug - Default actions are mapped to wrong controller

Step to reproduce:
Add the following action to any MVC controller as follows:
		[HttpGet("{id}")]
		public IActionResult Get(string id)
		{
			return new JsonResult("xyz");
		}//public IActionResult Get(string id)

Here is the generated mapping:
	    app.UseMvc(routes =>
            {
                routes.MapRoute(
                    name: "default",
                    template: "{controller=Home}/{action=Index}/{id?}");
            });

Actual result:
The default action of any controller gets mapped to “/TodoItems/Get”, and client receives "xyz".
For an examples:
"/Home"	 =>    "/TodoItems/Get"

The only way to invoked “/Home” is using “/Home/Index”.

If client receives a 404, what leads you to conclude / is mapped to /TodoItems/Get? Suggest you provide a repro project and more information about the debugging you've done so far.

From the log output. When error code is not 404, the client receives content from wrong controller (Get).
Just add the following action to any MVC controller to reproduce the bug:
		[HttpGet("{id}")]
		public IActionResult Get(string id)
		{
			return new JsonResult("xyz");
		}//public IActionResult Get(string id)

Remove the above action, all back to normal.

Sorry, I missed the template in [HttpGet("{id}")]. That's an attribute route which will match every HTTP GET of the form http://localhost:5001/xyz. Suggest using attribute routes in your HomeController as well and adjusting their Order as you wish. Or change that template to something specific e.g. "/api/{id}".

@dougbu Yes, the cause is  [HttpGet("{id}")]. But the issue is that request gets routed to wrong controller (TodoItemsController). No attribute routes in HomeController. Even if HomeController or another controller uses attribute routes, default action should not be mapped to another controller (TodoItemsController).

It seems you would like the template "{id}" to be relative to /TodoItems instead of /. If that's correct, add [Route("/TodoItems")] on the TodoItemsController class.
Put another way, your conventional and attribute route templates currently overlap. You need to make some changes to separate them.

@dougbu You are right. Close it.


Protobuff support in Kafka automated testing

As a SDET
I want to produce a Google Protobuff Message to a Kafka topic automatically as a Zerocode step
So that I can continue my automation journey involving proto messages
As a SDET
I want to consume a Google Protobuff Message from a Kafka topic automatically as a Zerocode step
So that I can continue my automation journey involving proto messages
As a SDET
I want to produce/consume a Google Protobuff Message with a Generic type serializer/deserializer e.g. ByteArraySerDe
So that I can continue my automation journey involving proto messages easily without getting concerned about a specific type of SerDe
@respondsid

Duplicate of https://github.com/authorjapps/zerocode/issues/423


Create Monitor Plan API endpoint for Systems

endpoint that will accept a locId and return all systems for the specified location
endpoint uri
/api/monitor-plan-mgmt/locations/{loc-id}/systems

Ticket re reopened in order to be updated. The beginHour and  endHour was added.

Re tested - Passed

systemIdentifier field was added and systemType data now reflects more accurately its data in the UI wireframes

Re tested - Passed


fix: passing runner args in CLI

Summary
Passing runner args through CLI doesn’t work, because it’s always passed as a string which is then spread to individual characters. Bug introduced in #511.

Test plan
Pass custom config through the TEST_RUNNER_PATH env variable.

Good catch @honzahruby !

This PR has been released in Reassure v1.2.1 🎉


Download multiple images at once

I thought that the download progress could be sped up by doing what the title says. I have had good results so far, but sadly due to how Threading works we get a lot of overhead while checking what is already downloaded. I will move that into a new method soon. Wanted to create this pull request beforehand to see what you think of it.

Noticed an issue, it doesn't save the progress anymore, will probably fix it tomorrow

Hi! Thanks for the time to make this edit.
Unfortunately, I won't merge your Pull Request.
I have already considered the idea of using threads or async functions to download multiple cards simultaneously but YGOPPRODeck's API Guide states "If we find you are pulling a very high volume of images per second then your IP will be blacklisted and blocked.".
Most people will download all cards (over 12k requests) and since they doesn't specify what is considered a "very high volume of images" and I don't want to ban anyone from the website or the API I used a very low frequency to download cards (about 2 per second).
But you can still fork the repository and keep working on your own version without problems! Just remember to change UserAgent and put your name on the LICENSE
Now, if you are willing to go back and change your code, maybe a different approach would be better than using 3 threads on 3 lists.
Some download handler class that uses async calls, a queue and some Observer-like programming pattern would probably be more maintainable and readable.


intracellular targets with Ab tractability assessment

Workshop attendees have brought to our attention that intracellular proteins seem to be predicted to be tractable by Ab as per the outcome of our tractability pipeline.
One example is BRCA1 raised at a workshop in 2019. Chris Radoux (formerly at ChEMBL) has provided this explanation at the time:

As per release 20.04, BRCA1 is still deemed tractable by Ab:

The latest report of this behaviour was done on May 4th at a Sanofi training. I will try to get additional examples to BRCA1 so that we can investigate whether this issue can be addressed.

Thanks @deniseOme. The issue is on the tractability bucket assumption that GO provides correct cellular localisation. I added a screenshot on the rewrite where the data can be seen side-by-side.

We'll report back to the tractability team for them to think more carefully about this assumption. I'll close as it should require no more action in our side.


Update 'Requires at Least' to 'Requires WP' and some other minor stuff

Here's a new PR. I'm not sure about the PHPDoc changes but it did remove errors in PHPStorm.

Oops, I need to fix.


customskill endpoint 404 error but local test the endpoint is working(without add to customskill)

Hi I followed demo-python/code/azure-search-vector-image-python-sample.ipynb to create a customskill:
        customSkill_endpoint = "https://{myresource}.cognitiveservices.azure.com/computervision/retrieval:vectorizeImage"
        skillset_name = f"{index_name}-skillset"
        skill_uri = customSkill_endpoint

        skill = WebApiSkill(
            uri=skill_uri,
            inputs=[
                InputFieldMappingEntry(name="imageUrl", source="/document/metadata_storage_path"),
                InputFieldMappingEntry(name="recordId", source="/document/metadata_storage_name")
            ],
            outputs=[OutputFieldMappingEntry(name="vector", target_name="imageVector")],
        )

        skillset = SearchIndexerSkillset(
            name=skillset_name,
            description="Skillset to extract image vector",
            skills=[skill],
        )

        client = SearchIndexerClient(service_endpoint, AzureKeyCredential(key))
        client.create_or_update_skillset(skillset)
        print(f' {skillset.name} created')

        index_client = SearchIndexClient(
            endpoint=service_endpoint, credential=credential)
        fields = [
            SimpleField(name="id", type=SearchFieldDataType.String, key=True, sortable=True, filterable=True, facetable=True),
            SimpleField(name="imageUrl", type=SearchFieldDataType.String, retrievable=True),
            SearchableField(name="title", type=SearchFieldDataType.String, searchable=True, retrievable=True),
            SearchField(
                name="imageVector",
                type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
                searchable=True,
                vector_search_dimensions=1024,
                vector_search_configuration="my-vector-config",
            ),
        ]

        vector_search = VectorSearch(
            algorithm_configurations=[
                VectorSearchAlgorithmConfiguration(
                    name="my-vector-config",
                    kind="hnsw",
                    hnsw_parameters={
                        "m": 4,
                        "efConstruction": 400,
                        "efSearch": 1000,
                        "metric": "cosine"
                    }
                )
            ]
        )
        index = SearchIndex(name=index_name, fields=fields, vector_search=vector_search,)
        result = index_client.create_or_update_index(index)
        print(f' {result.name} created')

Then I go to azure search service and check my indexer, it gave me this error:
Web Api response status: 'NotFound', Web Api response details: '{"error":{"code":"404","message": "Resource not found"}}'
but I test the endpoint in local with python code, it works fine:
        def get_image_embeddings(imageUrl):
            cogSvcsEndpoint = "https://{my cognitive service resource}.cognitiveservices.azure.com/"
            cogSvcsApiKey = "{cogsvcsapikey}"
            url = f"{cogSvcsEndpoint}/computervision/retrieval:vectorizeImage"
            params = {
                "api-version": "2023-02-01-preview"
            }
            headers = {
                "Content-Type": "application/json",
                "Ocp-Apim-Subscription-Key": cogSvcsApiKey
            }
            data = {
                "url": imageUrl
            }
            response = requests.post(url, params=params, headers=headers, json=data)
            if response.status_code != 200:
                print(f"Error: {response.status_code}, {response.text}")
                response.raise_for_status()
            embeddings = response.json()["vector"]
            return embeddings

May I know what did I do wrongly for adding the customskill to azure cognitive search?

It looks like your Azure Function hasn't deployed correctly? Can you verify that you can call your deployed Function App via Postman?

It looks like your Azure Function hasn't deployed correctly? Can you verify that you can call your deployed Function App via Postman?

Hi. I managed to deploy the Function App and this time it works for me. But I noticed another error when the api processed over 300 images. it will throw error and show failed in acs indexer.
Same thing will happened if I just use normal python function like below:
def get_image_embeddings(imageUrl):
cogSvcsEndpoint = "https://{my cognitive service resource}.cognitiveservices.azure.com/"
cogSvcsApiKey = "{cogsvcsapikey}"
url = f"{cogSvcsEndpoint}/computervision/retrieval:vectorizeImage"
params = {
"api-version": "2023-02-01-preview"
}
headers = {
"Content-Type": "application/json",
"Ocp-Apim-Subscription-Key": cogSvcsApiKey
}
data = {
"url": imageUrl
}
response = requests.post(url, params=params, headers=headers, json=data)
if response.status_code != 200:
print(f"Error: {response.status_code}, {response.text}")
response.raise_for_status()
embeddings = response.json()["vector"]
return embeddings
May I check what could be the potential reason for that?
Many thanks in advance!

Hi @viciaky, I may need you to submit a support ticket so we can dive into the logs on what the root cause of this error code is. Thanks!

Hi @farzad528 . I submitted azure support ticket regarding to this. here is the TrackingID#2307270030002958
Let me know if this is the right support ticket you mentioned.


Missing semicolons

I know that semi-colons are not at required by any means in Javascript but it is still far more common to include them rather then to omit them.
Many Javascript apps require building and usually enforce a consistent style guide, not having the semi colon would make the builds fail.
Providing an option to include a semicolon would be the best for both worlds, if its not to complex to implement.

Perhaps check if selected line has a semicolon and if so, add one, if not, skip it.

this is a good suggestion, thanks for bringing it up. I am so used to coding in Javascript Standard Style that I did not think about this. I will add a package configuration option in the next update.

@Josh-Star and @wilmoore this should now be a configurable options as of v0.6.2. Check it out and let me know if you see any further issues


Import frameworks in extensions

These imports are missing from each category, this fixes them if you don't auto import them.

:+1:


User arn:aws:sts::XXXXXXXXX:assumed-role/sam-batch-test-SendEmailFunctionRole-XXXXXX/sam-batch-test-SendEmailFunction-1FHgDhZRZNoE' is not authorized to perform ses:SendEmail' on resource<EMAIL_ADDRESS>

I just ran the template as-is with an existing domain (that is registered and verified) and ran into a SES permissions issue.
User `arn:aws:sts::XXXXXXXXX:assumed-role/sam-batch-test-SendEmailFunctionRole-XXXXXX/sam-batch-test-SendEmailFunction-1FHgDhZRZNoE' is not authorized to perform `ses:SendEmail' on resource<EMAIL_ADDRESS>
I managed to get around the issue by appending permissions specific to the sender email, but I'm not sure if this is the best practice solution. Furthermore, it doesn't seem like providing permission for the SESIdentityName is necessary.
        - SESCrudPolicy:
            IdentityName: !Ref SESIdentityName
        - SESCrudPolicy: # This is what I added to resolve the issue
            IdentityName: !Ref SESSender # This is what I added to resolve the issue

Had the same problem and your fix worked for me.  Thanks!

Thanks for reporting this, apologies for delay in replying. I have since modified the README file to be explicit about the value that you specify for the  SESIdentityName to be same as SESSender.


Passwords and other secrets in server logs

The last merge (PR #282) extended the server side logging with display of request bodies.
While this is good for debugging issues in the communication between client and server it also means that the log may contain sensitive data like credentials or some such. See for example the user-specified data when creating a custom service.
On the other hand, the default server verbosity is low enough to not log anything.
On the third hand, one of the first things we may do when suspecting a communication issue will be to ask the user to bump the server verbosity (by patching the server deployment / pod) so that we can get detailed logs.
This needs at least a note in the documentation. Best placed wherever we document how to crank up server verbosity.
Maybe further separate the logging of request bodies from general logging.
I.e. control body logging with either a separate option, or place at an even higher verbosity level.

Probably outdated. Closing it.


[FX-2192] Show header info

This builds off the new grid component for layout in https://github.com/artsy/palette/pull/794
This is actually a weirdly complex conditional layout that becomes very easy with the new grid component.
No about, no viewing room

Has about, no viewing room

Has about and viewing room

No about, has viewing room

Everything stacks into single column on mobile

@mzikherman this should be good to go


Gather all logged runs in a standard date format

Type of change

[ ] New sample
[x] New feature
[x] Bug fix
[ ] Documentation

Summary
This PR paginates over the running_datastore datastore to collect all logged runs and stores runs with with the ISO-8601 timestamp. Tests are also returned with a stable date to verify proper month/year wrapping when gathering past weeks.
Closes #7, fixes #22, and resolved #13.
Requirements

[x] I’ve checked my submission against the Samples Checklist to ensure it complies with all standards
[x] I have ensured the changes I am contributing align with existing patterns and have tested and linted my code
[x] I've read and agree to the Code of Conduct

Thanks for the review @misscoded! 🚀


[CI]: Remove Ashish (Revolyssup) from reviewers in dependabot yml file

Current Behavior

We have Ashish Tiwari as a reviewer in dependabot yml file by default.
Desired Behavior

We need to remove him from dependabot.yml file.
Implementation

Acceptance Tests

Contributor Guides and Resources

📝 Meshery Adapters Spreadsheet
🛠 Meshery Build & Release Strategy
🙋🏾🙋🏼 Questions: Layer5 Discussion Forum and Layer5 Community Slack

#530 successfully completes this issue ticket.


[GH] Communication channels

As an open source project, we need to choose our communication channels. Having something like:
Communication
If you need help, use Stack Overflow. (Tag 'kong')
If you'd like to ask a general question, use Stack Overflow.
If you found a bug, open an issue.
If you have a feature request, open an issue.
If you want to contribute, submit a pull request.
Or/and use Gitter or/and use a freenode channel.

This is coming up soon, @jitachi is already building the page and it we'll be available next week most likely.

This has been partially implemented: http://getkong.org/contribute/
Waiting on designing and implementing the support page before closing it. @jitachi is on it.

Needs to be in the README too. Same text as I originally put (but specific to Kong)


Add FindTraceIDs to the spanstore.Reader interface

Requirement - what kind of business use case are you trying to solve?
Have an caching layer which retrieves full traces from cache.
Finding traces matching a criterion involves finding matching traceIDs and then retrieving them. These are separate concerns for both ES and C*.
I would like the interface to maintain these as separate concerns so that a caching layer may make decisions on where to retrieve traces from.
Problem - what in Jaeger blocks you from solving the requirement?
The current interface,
FindTraces(ctx context.Context, query *TraceQueryParameters) ([]*model.Trace, error)
is pretty coarse and doesn't allow for easily inserting a cache.

Proposal - what do you suggest to solve the problem or improve the existing situation?
Add a FindTraceIDs(ctx context.Context, query *TraceQueryParameters) ([]model.TraceID, error) to the spanstore.Reader interface

+1, always wanted it.
Don't quite agree with your stated use case, but where it's definitely useful is when splitting span storage (e.g. a KV store like Cassandra) and span indexing (e.g. Pinot).
One issue with adding it to the Reader is how the query service is going to use it. E.g. if it always does Find first, followed by Get, then it introduces unnecessary 2nd query to the database even when the request could've been resolved in one query. However, by having FindTraceIDs in the interface allows us to implement a decorator Reader that delegates Find to one storage and delegates Get to another.

Having FindTraceIDs in the interface allows us to implement a decorator Reader that implements Get by calling Find on one storage and then Get on another.

Precisely - this is exactly what I want to do!


Update Rails symbol syntax for 400 status

:bad_request is a symbol, and what Rails expects to be passed as status, while bad_request would have looked for a method or variable with that name.

Your finding it now included in the new active repo at httpstatuses


[dev] Commitment Verification Trait

Piltover will call both is_valid on the verifier contract with the DA fact, and also verify_attestation on the blobstream contracts in order to ensure a valid state update.
Since blobstream is not the only means by which an appchain can prove it's data commitment this should be a generic trait that verifies its input with a verify_attestation call.

Hey please can i be assigned this?

@Oshioke-Salaki sorry for the delay. This has been assigned to you.


RDB Loader: move steps from CLI arguments to storage target config

As they're target-specific, not pipeline-specific.
Or even better we can remove steps as VACUUM, ANALYZE etc entirely as "house-keeping" is not part of RDB Loader mission.
https://github.com/snowplow/snowplow/issues/3178

Migrated to https://github.com/snowplow/snowplow-rdb-loader/issues/20


doc/articles/wiki: move _ warning slightly higher at one point

Currently working myself through https://golang.org/doc/articles/wiki/ and got stuck at the first HTTP server example. It doesn't have any error handling but easily runs into a bad Go panic
2017/11/07 20:08:24 http: panic serving <IP_ADDRESS>:48718: runtime error: invalid memory address or nil pointer dereference
Took me some time to find out that I was missing the test.txt file. Created a solution comment before noticing that pull requests to are not allowed but don't want to go through the full contributor process right now just to add few lines to the tutorial page.
You may copy from my commit or write something else, but there should be a hint for others running into the same problem:
https://github.com/sewi-cpan/go/commit/c47c0178da16780f73403adddd6f8242cdbd1686

In my opinion no action is needed here, since it is clearly mentioned in the text:

Again, note the use of _ to ignore the error return value from loadPage. This is done here for simplicity and generally considered bad practice. We will attend to this later.

There is even a special section in the very same wiki document dedicated specifically to this issue (Handling non-existent pages)

Agree. May be we can change this text This is done here for simplicity and generally considered bad practice. to bold ? So that there are less chances of getting tripped.

In the text:

First, this function extracts the page title from r.URL.Path, the path component of the request URL. The Path is re-sliced with [len("/view/"):] to drop the leading "/view/" component of the request path. This is because the path will invariably begin with "/view/", which is not part of the page's title.
The function then loads the page data, formats the page with a string of simple HTML, and writes it to w, the http.ResponseWriter.
Again, note the use of _ to ignore the error return value from loadPage. This is done here for simplicity and generally considered bad practice. We will attend to this later.

Let's move the final ("Again...") paragraph above the first two.

The hint is ok, but the result of the "" could be a problem: The unintended reader easily runs into what looks like a completely unrelated error while trying the source until this point. The "" is perfectly ok at this place and also the reference to attend to this later.
I read the "_" warning. The consequences might be obvious for Go experts but not for newbies walking through the tutorial.
Another idea to reduce the risk of problems here: The first example (in "Data Structures") does create a "TestPage.txt", but the webserver example discussed here expects "test.txt". Why not just use the same name in both examples. Doing so would create the test file needed for part 2 in part 1.


CONTENT_DOWNLOAD_MISMATCH exception

Environment details

OS: Windows (firebase functions emulator)
Node.js version: v8.12.07s
npm version: 5.6.0
@google-cloud/storage version: 1.7.0 & 2.3.4

Steps to reproduce

Upload an image file using: file.save(mediaBuffer, {metadata: {contentType}, gzip: true})
Try downloading the img file using: storage.bucket(fromBucket).file(f.name).createReadStream()
Exception is thrown: code=CONTENT_DOWNLOAD_MISMATCH message=The downloaded data did not match the data from the server. To be sure the content is the same, you should download the file again., stack=Error: The downloaded data did not match the data from the server. To be sure the content is the same, you should download the file again.

First I tried it using firebase-admin, and when I saw that they use an old version(1.7), I tried taking latest @google-cloud and run it again, but still nothing works.
The only solution I could find was to set storage.bucket(fromBucket).file(f.name).createReadStream({validation: false});
but this is just a quick hack, there should not be any reason the validation should fail

I see that this issue might be a very long due.
I'm trying to use google vision api on GCS gzipped image and it failed.
I tried looking it up and found this answer from 2016:
https://stackoverflow.com/a/36684248/1861047

Turns out, there is a bug in Google's implementation of gzip compression on the request... I'm not sure if the API only applies gzip compression when the request size hits a certain threshold, or if the implementation bugs works for small requests, but not for large ones.

There was no issue if {metadata: {contentType}} wasn't supplied.

I'm glad that this is reproducible for you too. Is the problem in the nodejs module? or a problem with the backend?

So I tested it and as you said, when setting gzip "auto" it works. No need for validations anymore.
Anyways, I think that there's still a problem. When removing the gzip metadata from the image, it becomes unviewable so there's more to that than just the encoding. Now I have many photos on my storage that are "corrupt" this way so I can't download them without the "validation: false" flag. How can I fix it?
BTW, please note that gzip option is nowhere to be found other than in the code itself. I did try to look for it in the official docs, but couldn't find it. I looked at file.createWriteStream and in the links for the options object there, but no gzip option was provided. It does talk about contentEncoding, but not on the gzip flag itself

Now I have many photos on my storage that are "corrupt" this way so I can't download them without the "validation: false" flag. How can I fix it?

For those files, you should be able to fix existing objects by unsetting (stripping) the Content-Encoding: gzip metadata:
await file.setMetadata({
  contentEncoding: null // this unsets the 'gzip' encoding causing the issue
});

We're getting this error without any content-encoding or gzip settings:
Running the below code once works (creating the file), the second time gives the CONTENT_DOWNLOAD_MISMATCH error (when updating the file).
const storage = new Storage();
const bucket = storage.bucket(GCP_STORAGE_BUCKET_NAME);
const remoteFile = bucket.file('test.csv');

let fileContents = ''+ Date.now();

remoteFile.save(fileContents).then(
	() => remoteFile.download().then(console.log).catch(console.warn)
).catch(console.warn);

@jheeffer - I'm having the exact same problem. Have tried with different ways of getting the file, through .createReadStream or simply .download(), but with same results. Disabling the validation returns old data. It just as if, on upload, the file data is not changed - but the hashes are.
This is really starting to annoy me. And I'm simply trying to write to a JSON-file.

@jheeffer I tested with this script, and it didn't have any issues matching. Could you give it a run to see if you still get errors? If you can make any modifications to demonstrate the error, that would be helpful as well. Thanks!
'use strict'

const {Storage} = require('@google-cloud/storage')
const storage = new Storage()

const bucket = storage.bucket(GCP_STORAGE_BUCKET_NAME)
const file = bucket.file('test.csv')

async function test() {
  let fileContents = `${Date.now()}`
  console.log('saving: ', fileContents)
  await file.save(fileContents)
  const [contents] = await file.download({validation: false})
  console.log('downloaded:', contents.toString())
}

(async () => {
  await test()
  await test()
  await test()
  await test()
  await test()
})()

@danielsetreus do you have a script I can try?

@stephenplusplus I ran you rest and can confirm there is something strange...
exports.test = () => {
  (async () => {
    await test()
    await test()
    await test()
    await test()
    await test()
  })()

};

const test = async () => {
  const {Storage} = require('@google-cloud/storage');
  const storage = new Storage()
  const bucket = storage.bucket('spring-internals');
  const file = bucket.file('test.csv');

  let fileContents = `${Date.now()}`;
  console.log('saving: ', fileContents);
  await file.save(fileContents);
  const [contents] = await file.download({validation: false});
  console.log('downloaded:', contents.toString());
}

Results:
node -e 'require("./index.js").test()'
saving:<PHONE_NUMBER>636
saving:<PHONE_NUMBER>652
downloaded:<PHONE_NUMBER>636
saving:<PHONE_NUMBER>010
downloaded:<PHONE_NUMBER>636
saving:<PHONE_NUMBER>028
downloaded:<PHONE_NUMBER>028
saving:<PHONE_NUMBER>196
downloaded:<PHONE_NUMBER>028
saving:<PHONE_NUMBER>206
downloaded:<PHONE_NUMBER>196
saving:<PHONE_NUMBER>404
downloaded:<PHONE_NUMBER>196
saving:<PHONE_NUMBER>415
downloaded:<PHONE_NUMBER>404
saving:<PHONE_NUMBER>022
downloaded:<PHONE_NUMBER>404
saving:<PHONE_NUMBER>047
downloaded:<PHONE_NUMBER>047
downloaded:<PHONE_NUMBER>047

But the script also seems to not wait for itself properly...
The original script I had is pretty long, but in concept it's the same as above. Download the file to a local copy, write to that, and upload the result. But it fails already in the download with CONTENT_DOWNLOAD_MISSMATCH. So, either the hash is badly created on upload, or the hash calculation is wrong on download.
Have any of you tried setting a specific hash algoritm?

Got this working by really simplifying the script. Could it be that the local download changes some metadata about the file?
exports.writeEntry = async (message, attributes) => {
  const entry = constructEntry(message, attributes);
  let current = await getEventFile();
  console.log('Current: ', current);
  if(current.length === 0)
    current = '[]';

  current = JSON.parse(current);
  current.push(entry);

  console.log('Write', current);
  await writeEventFile(JSON.stringify(current));
};

const constructEntry = (message, attributes) => {
  return {
    message: message,
    severity: attributes.severity,
    timestamp: attributes.timestamp,
    readableTimestamp: Date(attributes.timestamp*1000).toString(),
    origin: attributes.origin
  };
};

const getEventFile = async () => {
  const content = await file.download();
  return content.toString();
};

const writeEventFile = async (data) => {
  await file.save(data, {
    contentType: 'application/json;',
    resumable: false,
  });
};

Thanks for that! It sent me down a few different tracks that ended up with this PR: #643. Feel free to chime in over there if you have any ideas about a solution.

If it helps anyone doing a search online, I had this error code and wasn't sure what was going wrong.
What I realised was happening was that I had my rate limiter doing 2 concurrent read/write operations on the same file.


Use not only in overleaf?

Hi, thanks for this, it is very helpful and convenient, I am wondering if it is possible to include other websites, not only overleaf? Maybe it is better to have a blank for users to add other webs?

Hi @XiangLiu-github,
First of all, thank you!
Unfortunately, to register the keybinds the extension uses the Ace* API. Thus, it would only work on other services that use this same editor.
Given that, and for simplicity, I would prefer keeping the scope of this project restricted to Overleaf. But if you do have a specific suggestion of such a service, do let me know and I will check how easy it would be to adapt it.
*: Ace editor is a JavaScript library that Overleaf uses. It implements the actual text/code editor part of it.

Thanks for the explanation! My university has a service that is totally identical to overleaf (possibly use the Ace API), https://tex.nju.edu.cn/project, but in another web and only for institutional login.
Is it that impossible to test that? or how do I know if it is Ace editor?
Anyway, thanks for your work!

OK, you can adapt it and test it by doing the following:

Clone or download the repository to a location of your choosing (note that if you delete it, you will delete the extension).
Edit the 'manifest.json' file: change the overleaf url to the one you linked.
Follow the instruction in Step 2 of this page on how to install and test an extension from source.

That should really be all. Please do let me know if it works!

Oh, and I don't know how chrome handles two extensions with the same name entry, so it might be wise to also change the name field too in the 'manifest.json' file if you're keeping the published version installed. Just to make sure that doesn't cause any problems...

Thanks! It worked for me! I will just use it then.


Trying to create tiles to zoom 20 (User error)

What I want is to render Michigan(ish, tile 8,11,5 according to the tool linked in the generate jobs readme) from zoom 15 to 20, as I can download up to zoom 14, so I'm trying to follow the instructions in USAGE.md.
I've been having some issues with the documentation so far, and I'll make a pull request later with suggested edits once I have something that actually works.
What I want right now is to confirm that I finally know what I'm doing before I waste even more time processing vector tiles I don't need.
What I expected to work:
docker-compose run \
  -e TILE_X=8 -e TILE_Y=11 -e TILE_Z=5 \
  -e JOB_ZOOM=14 -e MAX_ZOOM=20 \
  generate-jobs

It didn't work. (It looks like it just would have generated jobs for level 14?)
generate_world_jobs.sh doesn't take MAX_ZOOM as an argument. generate_jobs.py however, does, which is what I had read and made bad assumptions on.
So what I need to do is modify generate_world_jobs.sh to pass MAX_ZOOM,
...
+ readonly MAX_ZOOM=${MAX_ZOOM:-"14"}
...
- python generate_jobs.py pyramid "$TILE_X" "$TILE_Y" "$TILE_Z" --job-zoom="$JOB_ZOOM" > $jobs_file
+ python generate_jobs.py pyramid "$TILE_X" "$TILE_Y" "$TILE_Z" --job-zoom="$JOB_ZOOM" --max-zoom="$MAX_ZOOM" > $jobs_file
...

and run
docker-compose run \
  -e TILE_X=8 -e TILE_Y=11 -e TILE_Z=5 \
  -e JOB_ZOOM=15 -e MAX_ZOOM=20 \
  -e WORLD_JOB="false"
  generate-jobs

Should this work?
Also in my last try, the rabbitmq docker container restarted on it's own, once I got to the merge step. I was able to just loop through all the mbtiles in the AWS bucket, but any idea why the rabbitmq docker container restarted instead of just crashing, or better yet not crashing?

May I ask why you want to render vector tiles until z20? Are you including a lot of extra data at high zoom levels?
Are you running it on multiple machines or on a single one?
If you are running it on a single machine I suggest you use the local export.
docker-compose run \
   # bounding box of michigan
  -e BBOX="19.6875,40.97989806962015,20.390625,41.50857729743933" \
  -e MIN_ZOOM="14" \
  -e MAX_ZOOM="20" \
  export

If you are running on multiple machines we will further investigate.
Your approach is correct, the Docker container does not pass on max zoom it seems so adding it is good.
So you want to create jobs at zoom level 14 that will render down from z14 to z20 for all z15 descendants from your z5 tile 8/11/5.
python generate_jobs.py pyramid 8 11 5 --job-zoom=14 --max-zoom=20

This returns will return you 4^(15-5)= 1048576 jobs where each job renders 4^(20-14)=4096 tiles.
So you want to render 4 billion z15 tiles?
You get jobs like these. From a quick checks that seems correct.
{"id": "322a315fa30c4a5925a422e6ef80e719d24e0bea", "pyramid": {"tile": {"max_zoom": 20, "y": 11286, "min_zoom": 15, "x": 8296}, "bounds": {"west": -88.857421875, "north": 48.763431137917955, "south": 48.756188762805515, "east": -88.846435546875}}, "type": "pyramid"}

Also in my last try, the rabbitmq docker container restarted on it's own, once I got to the merge step. I was able to just loop through all the mbtiles in the AWS bucket, but any idea why the rabbitmq docker container restarted instead of just crashing, or better yet not crashing?

Never had a RabbitMQ crash so far. We were running it for several weeks.

@mtfurlan :   for the ZOOM > 14 topic ..
Probably you have  to modify the  [osm2vectortiles.tm2source/data.yml] (
https://github.com/osm2vectortiles/osm2vectortiles/blob/master/osm2vectortiles.tm2source/data.yml)  meta file ... because the current implementation  max zoom level = 14  ;
so you have to modify  and adapt every SQL WHERE filters ( near z(!scale_denominator!) );
for example
   WHERE z(!scale_denominator!) BETWEEN 13 AND 14

to
   WHERE z(!scale_denominator!) >= 13

@lukasmartinelli  I'm sorry, I just had a fundamentally wrong approach to my problem. I don't need the extra data, I just needed higher zoom and I was using the mapbox gl style wrong, so it was just requesting higher zoom tiles, so I thought "Well, better figure out how to make higher zoom tiles"
That was a lot of time I didn't need to spend.
I'll try to make a pull request for USAGE.md this evening.
Thanks for the quick response to my wrong approach.

@lukasmartinelli I'm sorry, I just had a fundamentally wrong approach to my problem. I don't need the extra data, I just needed higher zoom and I was using the mapbox gl style wrong, so it was just requesting higher zoom tiles, so I thought "Well, better figure out how to make higher zoom tiles"
That was a lot of time I didn't need to spend.
I'll try to make a pull request for USAGE.md this evening.
Thanks for the quick response to my wrong approach.

Coming from raster tiles this is completely understandable. The overzooming is like magic with vector tiles the first time you see it. We should perhaps educate people better in our repo and on the website.


Connections

connections between nodes
always align themselves straight -> in other words: vertically or horizontally
this means: if the two connected nodes are offset both vertically and horizontally, the connecting cable makes two changes in direction, each at an angle of 90 degrees
the angles should be slightly rounded
cables are light gray when the nodes are connected, but no condition has yet been fulfilled
as soon as the condition has been selected through which the following node is selected, the cable appears dark
In the middle of the cable there is a small, narrow box in which the condition is displayed
if you click on it, the condition in the right side bar can be changed
right-click on cable: delete option
Validation -> not everything should be connected to everything else #30 #29

I wonder whether a straight/90-Degree-Connection is actually better than the currently implemented curved connection in terms of

Connections "colliding" with / disappearing behind other nodes
General clarity of Connections and appealing design
Don't have strong opinions on it, it's probably easy to implement & test anyways. What do you think @PhilGarb

Also those two we decided against (for now):

In the middle of the cable there is a small, narrow box in which the condition is displayed
if you click on it, the condition in the right side bar can be changed

All the other points I agree on (except maybe the alignment -> see previous post).


Update README to link cennznet docs

cennznet docs are linked in main README, so that docs are scoped in cennznet folder, also  README created by polkadot is not polluted too much;
dockerhub-description worked well in other demo project, but is not working here, so get it removed; If we needed, we can manually update DockerHub homepage description;

@holygits @KarishmaBothara This PR is ready to review, are you happy with changes?


Update Guide Developer Front End

Looks good, although we need each commit signed off (see here).