Pytorch to tflite conversion

BUG=pytorch to tflite

@tingyan19 please review and merge this change

@tingyan19 please review and merge this change

@nyadla-sys there seems to be a failed test - do you mind taking a look? Thanks.

@tingyan19 Looks like it uses HiFi4_INTERNAL flag as part of xtensa/Pad.cc and this seems to be causing the issue
when we test in our local build for fusionf1/hifi4/hifi5 it works fine as it does not use HiFi4_Internal
And we are not sure  how to enable HiFi4_INTERNAL flag as part of our local build to reproduce the issue

@tingyan19 Looks like it uses HiFi4_INTERNAL flag as part of xtensa/Pad.cc and this seems to be causing the issue when we test in our local build for fusionf1/hifi4/hifi5 it works fine as it does not use HiFi4_Internal And we are not sure how to enable HiFi4_INTERNAL flag as part of our local build to reproduce the issue

Can you try using "TARGET_ARCH=hifi4_internal" in your command line (instead of " TARGET_ARCH=hifi4") for your local build?

@tingyan19 Looks like it uses HiFi4_INTERNAL flag as part of xtensa/Pad.cc and this seems to be causing the issue when we test in our local build for fusionf1/hifi4/hifi5 it works fine as it does not use HiFi4_Internal And we are not sure how to enable HiFi4_INTERNAL flag as part of our local build to reproduce the issue

Can you try using "TARGET_ARCH=hifi4_internal" in your command line (instead of " TARGET_ARCH=hifi4") for your local build?

I tried and getting below compilation issues
tensorflow/lite/micro/kernels/xtensa/add.cc:122:15: error: use of undeclared identifier 'xa_nn_elm_add_asym16sxasym16s_asym16s'; did you mean 'xa_nn_elm_add_asym8sxasym8s_asym8s'?
err = xa_nn_elm_add_asym16sxasym16s_asym16s(
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
xa_nn_elm_add_asym8sxasym8s_asym8s
tensorflow/lite/micro/tools/make/downloads/xa_nnlib_hifi4/include/nnlib/xa_nnlib_kernels_api.h:1754:8: note: 'xa_nn_elm_add_asym8sxasym8s_asym8s' declared here
WORD32 xa_nn_elm_add_asym8sxasym8s_asym8s(WORD8 * restrict p_out,
^
tensorflow/lite/micro/kernels/xtensa/add.cc:123:13: error: cannot initialize a parameter of type 'WORD8 *' (aka 'signed char *') with an rvalue of type 'short *'
tflite::micro::GetTensorData<int16_t>(output),
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tensorflow/lite/micro/tools/make/downloads/xa_nnlib_hifi4/include/nnlib/xa_nnlib_kernels_api.h:1754:64: note: passing argument to parameter 'p_out' here
WORD32 xa_nn_elm_add_asym8sxasym8s_asym8s(WORD8 * restrict p_out,
^
tensorflow/lite/micro/kernels/xtensa/leaky_relu.cc:83:21: error: use of undeclared identifier 'xa_nn_vec_leaky_relu_asym16s_asym16s'; did you mean 'xa_nn_vec_leaky_relu_asym8s_asym8s'?
int32_t err = xa_nn_vec_leaky_relu_asym16s_asym16s(

@tingyan19 Looks like it uses HiFi4_INTERNAL flag as part of xtensa/Pad.cc and this seems to be causing the issue when we test in our local build for fusionf1/hifi4/hifi5 it works fine as it does not use HiFi4_Internal And we are not sure how to enable HiFi4_INTERNAL flag as part of our local build to reproduce the issue

Can you try using "TARGET_ARCH=hifi4_internal" in your command line (instead of " TARGET_ARCH=hifi4") for your local build?

I tried and getting below compilation issues tensorflow/lite/micro/kernels/xtensa/add.cc:122:15: error: use of undeclared identifier 'xa_nn_elm_add_asym16sxasym16s_asym16s'; did you mean 'xa_nn_elm_add_asym8sxasym8s_asym8s'? err = xa_nn_elm_add_asym16sxasym16s_asym16s( ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ xa_nn_elm_add_asym8sxasym8s_asym8s tensorflow/lite/micro/tools/make/downloads/xa_nnlib_hifi4/include/nnlib/xa_nnlib_kernels_api.h:1754:8: note: 'xa_nn_elm_add_asym8sxasym8s_asym8s' declared here WORD32 xa_nn_elm_add_asym8sxasym8s_asym8s(WORD8 * restrict p_out, ^ tensorflow/lite/micro/kernels/xtensa/add.cc:123:13: error: cannot initialize a parameter of type 'WORD8 *' (aka 'signed char *') with an rvalue of type 'short *' tflite::micro::GetTensorData<int16_t>(output), ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ tensorflow/lite/micro/tools/make/downloads/xa_nnlib_hifi4/include/nnlib/xa_nnlib_kernels_api.h:1754:64: note: passing argument to parameter 'p_out' here WORD32 xa_nn_elm_add_asym8sxasym8s_asym8s(WORD8 * restrict p_out, ^ tensorflow/lite/micro/kernels/xtensa/leaky_relu.cc:83:21: error: use of undeclared identifier 'xa_nn_vec_leaky_relu_asym16s_asym16s'; did you mean 'xa_nn_vec_leaky_relu_asym8s_asym8s'? int32_t err = xa_nn_vec_leaky_relu_asym16s_asym16s(

Below is command I used
make -f tensorflow/lite/micro/tools/make/Makefile TARGET=xtensa OPTIMIZED_KERNEL_DIR=xtensa TARGET=xtensa TARGET_ARCH=hifi4_internal test_pytorch_to_tflite_test -j

@tingyan19 Looks like it uses HiFi4_INTERNAL flag as part of xtensa/Pad.cc and this seems to be causing the issue when we test in our local build for fusionf1/hifi4/hifi5 it works fine as it does not use HiFi4_Internal And we are not sure how to enable HiFi4_INTERNAL flag as part of our local build to reproduce the issue

Can you try using "TARGET_ARCH=hifi4_internal" in your command line (instead of " TARGET_ARCH=hifi4") for your local build?

I tried and getting below compilation issues tensorflow/lite/micro/kernels/xtensa/add.cc:122:15: error: use of undeclared identifier 'xa_nn_elm_add_asym16sxasym16s_asym16s'; did you mean 'xa_nn_elm_add_asym8sxasym8s_asym8s'? err = xa_nn_elm_add_asym16sxasym16s_asym16s( ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ xa_nn_elm_add_asym8sxasym8s_asym8s tensorflow/lite/micro/tools/make/downloads/xa_nnlib_hifi4/include/nnlib/xa_nnlib_kernels_api.h:1754:8: note: 'xa_nn_elm_add_asym8sxasym8s_asym8s' declared here WORD32 xa_nn_elm_add_asym8sxasym8s_asym8s(WORD8 * restrict p_out, ^ tensorflow/lite/micro/kernels/xtensa/add.cc:123:13: error: cannot initialize a parameter of type 'WORD8 *' (aka 'signed char *') with an rvalue of type 'short *' tflite::micro::GetTensorData<int16_t>(output), ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ tensorflow/lite/micro/tools/make/downloads/xa_nnlib_hifi4/include/nnlib/xa_nnlib_kernels_api.h:1754:64: note: passing argument to parameter 'p_out' here WORD32 xa_nn_elm_add_asym8sxasym8s_asym8s(WORD8 * restrict p_out, ^ tensorflow/lite/micro/kernels/xtensa/leaky_relu.cc:83:21: error: use of undeclared identifier 'xa_nn_vec_leaky_relu_asym16s_asym16s'; did you mean 'xa_nn_vec_leaky_relu_asym8s_asym8s'? int32_t err = xa_nn_vec_leaky_relu_asym16s_asym16s(

Below command I used make -f tensorflow/lite/micro/tools/make/Makefile TARGET=xtensa OPTIMIZED_KERNEL_DIR=xtensa TARGET=xtensa TARGET_ARCH=hifi4_internal test_pytorch_to_tflite_test -j

@nyadla-sys
I changed https://github.com/tensorflow/tflite-micro/blob/d817b58c601175bc6a503fa7f67fe3668bf53b21/tensorflow/lite/micro/kernels/xtensa/pad.cc#L239-L240 to
  TF_LITE_KERNEL_LOG(context, "Type %d not currently supported by Pad.",
                     input->type);

and (without using hifi4_internal) I got:
input->dims->size:4
input->dims->data[0]:1
input->dims->data[1]:3
input->dims->data[2]:224
input->dims->data[3]:224
input->type:9
input->bytes:150528
Type -387389208 not currently supported by Pad.
Node PAD (number 3) failed to invoke with status 1
Invoke failed
It seems like there is an invalid type number was produced. This should have nothing to do with hifi4_internal.
Can you try and see if you can reproduce this?

@tingyan19 Looks like it uses HiFi4_INTERNAL flag as part of xtensa/Pad.cc and this seems to be causing the issue when we test in our local build for fusionf1/hifi4/hifi5 it works fine as it does not use HiFi4_Internal And we are not sure how to enable HiFi4_INTERNAL flag as part of our local build to reproduce the issue

Can you try using "TARGET_ARCH=hifi4_internal" in your command line (instead of " TARGET_ARCH=hifi4") for your local build?

I tried and getting below compilation issues tensorflow/lite/micro/kernels/xtensa/add.cc:122:15: error: use of undeclared identifier 'xa_nn_elm_add_asym16sxasym16s_asym16s'; did you mean 'xa_nn_elm_add_asym8sxasym8s_asym8s'? err = xa_nn_elm_add_asym16sxasym16s_asym16s( ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ xa_nn_elm_add_asym8sxasym8s_asym8s tensorflow/lite/micro/tools/make/downloads/xa_nnlib_hifi4/include/nnlib/xa_nnlib_kernels_api.h:1754:8: note: 'xa_nn_elm_add_asym8sxasym8s_asym8s' declared here WORD32 xa_nn_elm_add_asym8sxasym8s_asym8s(WORD8 * restrict p_out, ^ tensorflow/lite/micro/kernels/xtensa/add.cc:123:13: error: cannot initialize a parameter of type 'WORD8 *' (aka 'signed char *') with an rvalue of type 'short *' tflite::micro::GetTensorData<int16_t>(output), ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ tensorflow/lite/micro/tools/make/downloads/xa_nnlib_hifi4/include/nnlib/xa_nnlib_kernels_api.h:1754:64: note: passing argument to parameter 'p_out' here WORD32 xa_nn_elm_add_asym8sxasym8s_asym8s(WORD8 * restrict p_out, ^ tensorflow/lite/micro/kernels/xtensa/leaky_relu.cc:83:21: error: use of undeclared identifier 'xa_nn_vec_leaky_relu_asym16s_asym16s'; did you mean 'xa_nn_vec_leaky_relu_asym8s_asym8s'? int32_t err = xa_nn_vec_leaky_relu_asym16s_asym16s(

Below command I used make -f tensorflow/lite/micro/tools/make/Makefile TARGET=xtensa OPTIMIZED_KERNEL_DIR=xtensa TARGET=xtensa TARGET_ARCH=hifi4_internal test_pytorch_to_tflite_test -j

@nyadla-sys
I changed
https://github.com/tensorflow/tflite-micro/blob/d817b58c601175bc6a503fa7f67fe3668bf53b21/tensorflow/lite/micro/kernels/xtensa/pad.cc#L239-L240
to
  TF_LITE_KERNEL_LOG(context, "Type %d not currently supported by Pad.",
                     input->type);

and (without using hifi4_internal) I got:
input->dims->size:4 input->dims->data[0]:1 input->dims->data[1]:3 input->dims->data[2]:224 input->dims->data[3]:224 input->type:9 input->bytes:150528 Type -387389208 not currently supported by Pad. Node PAD (number 3) failed to invoke with status 1 Invoke failed
It seems like there is an invalid type number was produced. This should have nothing to do with hifi4_internal.
Can you try and see if you can reproduce this?

@tingyan19 I tried on our local server with hifi4 and hifi5 core config and it works fine..
lets discuss more details in our calls


Monitoring integration

This is an issue to explore which monitoring products and platforms we might want to offer out-of-the-box integration for.
So far we have (in no particular order):

Prometheus
Icinga
Metronome
Zabbix
Nagios
Influxdb
SNMP, SMUX, AgentX

Feel free to add whatever it is you are using if it isn’t in the list already!
To elaborate a bit: we are planning to produce a set of metrics regarding both the Routinator’s health and the content of the RPKI cache it has and also provide some active service monitoring, probably based on an RTR client.

I would second SNMP implemented as AgentX. AgentX is a newer interface, replacing SMUX, and is standardized in RFC 2741 and is in many ways better than SMUX.

Given that the ticket was intended to be mostly exploratory, I’ll close it for now. We will evaluate adding SMTP support at some point, perhaps together with the other RPKI relying party software vendors.


Add animation complete block or delegate

This is a great lib. it would be great if you add an animation complete handler.

Added in version 0.4.0.


Add pycharm docs to setup Docker as Python interpreter

Description
Add Pycharm docs to setup Docker as Python interpreter
Development notes
Simple docs addition.
Checklist

[ x] Read the contributing guidelines
[ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
[x ] Updated the documentation to reflect the code changes
[ ] Added a description of this change and added my name to the list of supporting contributions in the RELEASE.md file
[ ] Added tests to cover my changes

Notice

[x] I acknowledge and agree that, by checking this box and clicking "Submit Pull Request":

I submit this contribution under the Apache 2.0 license and represent that I am entitled to do so on behalf of myself, my employer, or relevant third parties, as applicable.

I certify that (a) this contribution is my original creation and / or (b) to the extent it is not my original creation, I am authorised to submit this contribution on behalf of the original creator(s) or their licensees.

I certify that the use of this contribution as authorised by the Apache 2.0 license does not violate the intellectual property rights of anyone else.

LGTM, thank you once again for the contribution @moutai! BTW, we'd be happy if you add your name as a contributor in RELEASE.md (here) 😄

Hi @moutai  - Thank you for your contribution! We're holding a community event later this week which may be of interest to you.


Remove rel-eng directory

This directory was auto generated and merged when played tito.
Need to be removed.

:+1:

+1

+1


Dead code injection variables rename

Coverage decreased (-0.03%) to 98.641% when pulling 3970513a64dabb73816cd0c4e2dd03f9ed1d4e29 on dead-code-injection-variables-rename into c210bb433834acdfe6f7d3edfb787041ecc266b5 on master.


use a traitlets based spec and make interactive templates

add compatability layers to separate out pieces of the narrative into different units.

another big ass lift that removes pluggy and relies solely on traitlets. for better and definitely worse. its canonical.🤷🏻
we also have a compatability folder to defer discussion about internal machinery to a deeper place.


ci: setup everything

add .vale.ini to setup Vale tool

.gitignore
exclude imported Vale Packages

.yamllint.yaml
YAML validation, excluding imported Vale Packages

add .editorconfig to define files format

add .ecrc to validate files format against .editorconfig
but, excluding imported Vale Packages

Git hooks via pre-commit including:

typos
codespell
yamllint

added following items to GitHub Actions:

yamllint
vale
typos
codespell
markdownlint-cli2
editorconfig-checker

@alexvonme here is what I worked on and that could be reused for https://github.com/jargonLint/jargonLint

hey @jason3s I'm pretty proud of what I set up here
please review

hey @Jason3S I'm pretty proud of what I set up here
please review

@ccoVeille,
I'm not sure how I can help. codespell is not a project I have worked on.

Thanks, no problem.
I initially wanted to add cspell, but the fact this is still not available, I'm afraid I cannot consider it for now

https://github.com/streetsidesoftware/cspell/discussions/5751


I have run the code for many times, but the test result fails to reach 88, which is only about 87.6. Is there any detail that needs attention

I have run the code for many times, but the test result fails to reach 88, which is only about 87.6. Is there any detail that needs attention？

This is odd. Did you use the pretrained model from the data/checkpoints/SNLI folder ? If not, it is possible that different executions to train the model produce different weights with varying levels of accuracy. If you train the model multiple times, you should however reach something around 88% on average.


Sync folder, but skip one of the subfolders

I want to sync: "C:\Users\Matthew\AppData\Roaming\foobar2000" but skip the folder "wsh_data". How do I do this?
I'm sorry if this is really easy, but I couldn't find it on the documentation.

Filtering has it's own documentation page: https://rclone.org/filtering/
Try something like this: rclone sync --filter "- wsh_data/" foobar2000 someRemote:/

I'm going to close this now - please ask questions in the rclone forum - thanks!


HDDS-6970. EC: Ensure DatanodeAdminMonitor can handle EC containers during decommission

What changes were proposed in this pull request?
Extract a common interface for ContainerReplicaCount and ECContainerReplicaCount to allow DatanodeAdminMonitorImpl to handle both EC and non-EC containers.
https://issues.apache.org/jira/browse/HDDS-6970
How was this patch tested?
Full CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/2590660510

Thanks @sodonnel for the review.


Create filters to select certain splits/teams/etc?

What would it look like to do this without a Redux store?


Add speedup tool.

With this, it is fairly straightforward to produce data files that can be fed to gnuplot.  The scripts to do so are not part of this PR, but I tried it on my own machine and got this result:

Nice!

Would it make sense to sort the benchmark list to the right of the graph according to their speedup factor on Nmax (i.e., 8) cores? It is somewhat difficult to identify which of the benchmarks perform poorly...

My worry is that sorting them is a bad idea when we have a plot per compiler (at least the two MLKit variants, and maybe also MPL).  It will be very confusing if the colours don't match up.

In the meantime, here's a speedup plot for MPL.


Consider adding Cradle to the list

I'd like to request that you consider adding CRADLE to your repository.
CRADLE is a new framework designed to empower foundation agents towards General Computer Control (GCC). It interacts with software in a human-like manner, using screenshots as input and keyboard/mouse actions as output. CRADLE exhibits impressive performance and generalization across various software applications and even complex video games.
Here's some information about CRADLE:
CRADLE: Empowering Foundation Agents Towards General Computer Control
Website: https://baai-agents.github.io/Cradle/
Paper: https://arxiv.org/abs/2403.03186v3
Video(s): https://www.youtube.com/@CradleGCC
Code: https://github.com/BAAI-Agents/Cradle
TLDR: CRADLE is a modular LMM-powered framework enabling foundation agents to perform complex computer tasks by receiving input from screens and audio and outputting keyboard and mouse actions, similar to how humans interact with computers.
Key Features:
Unified Interface: CRADLE interacts with any software through screenshots as input and keyboard and mouse actions as output.
Modular Framework: Comprises six key modules: Information Gathering, Self-Reflection, Task Inference, Skill Curation, Action Planning, and Memory.
Generalizability: Demonstrates impressive performance across commercial video games (Red Dead Redemption 2, Stardew Valley, Dealer’s Life 2, Cities: Skylines), software applications (Chrome, Outlook, CapCut, Meitu, Feishu), and the comprehensive OSWorld benchmark.
I believe CRADLE's unique approach and impressive performance make it a valuable addition to your repository. Thank you for considering this request.

Thank you @XinrunXu for reaching out,
I have added your work!

Added.


Introduce kube-state-metrics, ReplicaSet dashboard

The telemetry pipeline leverages denormalized kubernetes data to
identify source and destinations pods.
This change introduces kube-state-metrics into the conduit install, and
a ReplicaSet Grafana dashboard to prove out this approach.
kube-state-metrics listens to the Kubernetes API, generates metrics
about the state of objects, and serves those metrics to Prometheus.
By joining kube-state-metrics with conduit-proxy metrics, we may be able
to normalize all condiut metrics data. This could include removal of
self-identifying information added at inject time, and also destination
info added at destination lookup time.
Relates to #741.
Signed-off-by: Andrew Seigner<EMAIL_ADDRESS>

Can you include some documentation about how this works? This is way over my head.

@brancz would love your feedback on the approach here, specifically:

We've added kube-state-metrics to the conduit namespace, to keep everything conduit-related self-contained. Any concern with this?

In order to calculate rates over time while also performing joins with kube-state-metrics, I found it necessary to introduce recording rules, as I was hitting binary expression must contain only scalar and instant vector types otherwise:
- record: request_total:irate30s
  expr: irate(request_total[30s])

(Note we won't necessarily ship with irate[30s], but for testing this branch it's easier to make the queries equivalent to what's already in the codebase.)
This approach based on:
https://stackoverflow.com/questions/45213745/prometheus-how-to-calculate-proportion-of-single-value-over-time?rq=1
https://www.robustperception.io/composing-range-vector-functions-in-promql/
Does this seem correct? I tried to minimize our recording rules usage, specifically avoiding joins, as I assume that would defeat the purpose of this whole exercise?

Using this new query pattern, here is the simplest query we're doing, Request Rate by ReplicaSet:
sum(request_total:irate30s{direction="inbound"} *
  on(namespace, pod) group_left (owner_kind, owner_name)
  kube_pod_owner{namespace="$namespace", owner_kind="ReplicaSet",
owner_name="$replica_set"}
) by (owner_name)

...where $replica_set is inserted by Grafana. Does this seem correct?

Expanding on the previous question, this pattern leads us to some relatively complex queries involving label_replace. Here is a success rate calculation for a given ReplicaSet against all its outbound ReplicaSet's:
sum(response_total:irate30s{direction="outbound", pod=~"$pod", classification="success"} *
  on(namespace, dst_pod) group_left (owner_kind, owner_name)
  label_replace(kube_pod_owner{namespace="$namespace", owner_kind="ReplicaSet"}, "dst_pod", "$1", "pod", "(.*)")
) by (owner_name)
/
sum(response_total:irate30s{direction="outbound", pod=~"$pod"} *
  on(namespace, dst_pod) group_left (owner_kind, owner_name)
  label_replace(kube_pod_owner{namespace="$namespace", owner_kind="ReplicaSet"}, "dst_pod", "$1", "pod", "(.*)")
) by (owner_name)

...where $pod is filled in via a Grafana template variable, defined by:
label_values(kube_pod_owner{namespace="$namespace", owner_kind="ReplicaSet", owner_name="$replica_set"}, pod)
Does this approach still seem reasonable or does this query complexity suggest we should rethink our approach?

Generally sounds good. More generally it would probably be good if you document the whole monitoring setup a bit more, as people who use Prometheus on Kubernetes are very likely already running kube-state-metrics as well, so it would be nice to enable people to do that.

I don't see a bool expression, I think what you're trying to do is possible without recording rules. Otherwise, using recording rules is a perfectly legitimate thing - that's what they're for. In fact with recording rules you should see a query performance improvement as recording rules pre-calculate the query and store it in a new time-series so when you then query that time-series created by the recording rule, it's like any other time-series you're reading, only that it can be based on a complex query. But as I said, I'm not sure you even need that here.

An example query that I run against my own cluster with services that expose the http_requests_total metric:
rate(http_requests_total[5m]) * on(namespace, pod) group_left(owner_kind, owner_name) kube_pod_owner

If I want more specific labels selectors of the http_requests_total metric I would add that in its selector, and if I wanted to restrict the namespace/owner-name then I would add it in the label selector of the kube_pod_owner one.

The equivalent of the above would be, which doesn't seem so bad

sum(rate(http_requests_total[5m]) * on(namespace, pod) group_left(owner_kind, owner_name) kube_pod_owner) by(owner_name, owner_kind, namespace)

For this one it would be worth having a recording rule, in order to have the dst pod accessible as easy as the pod. Basically a recording rule for:

kube_dst_pod_owner = label_replace(kube_pod_owner, "dst_pod", "$1", "pod", "(.*)")

Note that here you may want to label_replace even more so you can do Owner->Owner descriptions as a higher level service description rather than Pod->Pod.

Then the query is much simpler again:
sum(rate(http_requests_total{classification="success"}[5m]) * on(namespace, pod) group_left(owner_kind, owner_name) kube_dst_pod_owner) by(owner_name, owner_kind, namespace)
/
sum(rate(http_requests_total[5m]) * on(namespace, pod) group_left(owner_kind, owner_name) kube_dst_pod_owner) by(owner_name, owner_kind, namespace)

Note as I only have incoming requests in my application I haven't tried this query so I can't 100% say it will work, but I think you get the gist.
Sorry for the long comment, let me know if that makes things clearer.

Thanks for all the feedback @brancz! All your suggestions make sense, I'll incorporate them into the next commit.
One more question I'd like your thoughts on: How do you decide whether to join with kube-state-metrics via recording rules or at query time? I can see benefits either way. If we join via recording rules, it makes querying Prometheus (particularly adhoc queries) simpler (and probably faster?). On the other hand, the approach puts denormalized data into Prometheus, possibly affecting storage?

More comments in-line for my own reference:

More generally it would probably be good if you document the whole monitoring setup a bit more, as people who use Prometheus on Kubernetes are very likely already running kube-state-metrics as well, so it would be nice to enable people to do that.

Agreed! We already have some documentation in case people want to use their own Prometheus instance. We can definitely expand that to cover kube-state-metrics.

I don't see a bool expression, I think what you're trying to do is possible without recording rules.

You're right, I must have been using sum and rate incorrectly. We definitely want to use recording rules for optimizations in general. In this particular case, I just wanted to confirm what was possible without recording rules, thanks!

The equivalent of the above would be, which doesn't seem so bad

sum(rate(http_requests_total[5m]) * on(namespace, pod) group_left(owner_kind, owner_name)
kube_pod_owner) by(owner_name, owner_kind, namespace)

Makes sense. The examples I was providing were specific to Grafana, which I think ends up requiring slightly more complicated queries, as we are using lots of labels to display a specific metric per-graph. In our programmatic use of Grafana from the Conduit Controller, we'll build queries less with labels and more with grouping, like the examples you've provided.

For this one it would be worth having a recording rule, in order to have the dst pod accessible as easy as the pod.

Makes sense, I'll add recording rules to avoid a label_replace at query time.

One more question I'd like your thoughts on: How do you decide whether to join with kube-state-metrics via recording rules or at query time? I can see benefits either way. If we join via recording rules, it makes querying Prometheus (particularly adhoc queries) simpler (and probably faster?). On the other hand, the approach puts denormalized data into Prometheus, possibly affecting storage?

Rule of thumb is if it's not possible at query time, like the bool example above (but there are a few more) or if the query performance is not good enough, then a recording rule makes sense. In your case, you may also want to consider the additional number of time-series you are producing, as each result of a query of a recording rule is inserted into a (potentially new) time-series in the Prometheus TSDB. I'm saying this as I know the product of two metrics may create a large amount of time-series, however, as always don't optimize before measuring :slightly_smiling_face: .

Updated the Proposal section of #741 to identify the approach laid out in this PR.


How do we tie the services together?

We need to choose how to manage services in multiple repositories. Within the Phase 1 work, it only needs to work in a docker-compose. In the next phase, however, we will need to package them all in a helm chart and run that in a local KIND cluster, for instance.
Option 1 - dedicated repo
Keep the docker-compose.yaml in a separate repo (e.g. dev, glue, infrastructure etc). In Phase 2 we could keep the KIND cluster config here as well.
➕ services get to stay completely independent
➖ docker-compose is maintained separately to everything and will have to include relative paths
Option 2 - docker-compose in each service repo
Maintain a docker-compose.yaml in each repo. The service in the repo is built from the repo docker-compose is in, while other services' latest image will be pulled from ghcr.io.
➕ no separate repo, docker-compose can be run in the same repo as the service code, making it easier to develop the service itself
➖ changes to the setup will need to be replicated across all repositories, including the database the services are talking to
Option 3 - ???

I am personally leaning to the second option suggested by @chipe88, as the docker-compose files should not get too many changes in Phase 1 and Phase 2 should rely more on kubernetes and helm charts

Since this is a temporary solution, until we start using k8s & helm - does not really matter, I am fine with both.

as we plan to deploy this with helm soon anyway, might as well set up a helm-charts repo with github pages so that helm could pull the versioned charts directly from the repo.
This is what it would look like more or less: https://github.com/two-tauers/helm-charts
The only addition will be the e.g. infrastructure folder at the root that can contain a docker-compose file and maybe also the KIND config.
@danyloid @chipe88 sounds good?

https://github.com/flam-flam/helm-charts


Start/Stop/Restart from a tag terminal

Currently you can Start/Stop/Restart tasks under a tag but not in bulk.

Done in vite branch


Translate docs into Korean

Hi guys!
I would like to translate gulp docs into korean.
Since I saw translation docs in README, I thought it would be possible.
So, can I participate in the translation works?
If it's possible, When after translation I will put the PR.
Thanks!

Yep! Send a PR when you have it


Clicking on element that is asserted visible fails due to not being visible

What is your Test Scenario?
Trying to test clicking on a Google Maps marker. I've configured Google Maps to create markers with optimized: false which renders them as DOM elements. In my test, I verify that the element is visible first before attempting to click on it.
What is the Current behavior?
Test click fails with error "The element that matches the specified selector is not visible"
What is the Expected behavior?
The test should click on the element.
What is your web application and your TestCafe test code?

Navigate to the following URL from each remote browser.
Connect URL: http://<IP_ADDRESS>:55805/browser/connect
CONNECTED Chrome 72.0.3626 / Windows 10.0.0
 Running tests in:
 - Chrome 72.0.3626 / Windows 10.0.0

 Expose TestCafe bug
 ✖ It fails to click on a visible element

   1) The element that matches the specified selector is not visible.

      Browser: Chrome 72.0.3626 / Windows 10.0.0

          9 |
         10 |    const targetMarker = markers.nth(0);
         11 |
         12 |    await t
         13 |        .expect(targetMarker.visible).ok()
       > 14 |        .click(targetMarker)
         15 |        .expect(infoWindow.visible).ok();
         16 |});

         at Object.<anonymous> (/mnt/c/Users/demsu/work/hoodq/hq-sdk/tests/functional/isolated.ts:14:10)
         at <anonymous> (/mnt/c/Users/demsu/work/hoodq/hq-sdk/tests/functional/isolated.ts:7:71)
         at __awaiter (/mnt/c/Users/demsu/work/hoodq/hq-sdk/tests/functional/isolated.ts:3:12)
         at test (/mnt/c/Users/demsu/work/hoodq/hq-sdk/tests/functional/isolated.ts:6:58)
         at markeredfn (/mnt/c/Users/demsu/work/hoodq/hq-sdk/node_modules/testcafe/src/api/wrap-test-function.js:17:28)
         at <anonymous> (/mnt/c/Users/demsu/work/hoodq/hq-sdk/node_modules/testcafe/src/api/wrap-test-function.js:7:5)
         at fn (/mnt/c/Users/demsu/work/hoodq/hq-sdk/node_modules/testcafe/src/test-run/index.js:240:19)
         at TestRun._executeTestFn (/mnt/c/Users/demsu/work/hoodq/hq-sdk/node_modules/testcafe/src/test-run/index.js:236:38)
         at _executeTestFn (/mnt/c/Users/demsu/work/hoodq/hq-sdk/node_modules/testcafe/src/test-run/index.js:289:24)

 1/1 failed (21s)

Screenshots:

Steps to Reproduce:

Run the test.
Your Environment details:

testcafe version: 1.1.0
node.js version: 10.11.0
command-line arguments: testcafe remote path/to/test.ts
browser name and version: Chrome 72
platform and version: Chrome running on Windows 10; TestCafe running in WSL on Ubuntu 16.04
other:

Also relevant: this fails intermittently, though the vast majority of the time. It passes successfully maybe every 4th or 5th run.

I checked your example and found two causes which lead to the issue.

The searchable element is not in the viewport of the page. Since this website does not have usual scrolling, but has some kind of endless scrolling, the TestCafe cannot scroll to the searchable element automatically.
The search element (the first with img[src*='_school_marker.png']) can be rendered in any place of the page; it is not defined.
I think in this endless scrolling scenario, you need to process scrolling to the searchable element in your code.
I prepared a sample test that demonstrates how this behavior can be reached.

import { Selector } from 'testcafe';

fixture('Expose TestCafe bug')
    .page('http://hoodq-sdk-demo.s3-website-us-east-1.amazonaws.com/');

test('It fails to click on a visible element', async t => {
    const markers = Selector("img[src*='_school_marker.png']");

    const targetMarker = markers.nth(0);

    const top = parseInt(await targetMarker.parent().getStyleProperty('top'));
    const left = parseInt(await targetMarker.parent().getStyleProperty('left'));

    const dragNumber = 5;

    for (let i = 0; i < dragNumber; i++) {
        await t.drag('#map',  -parseInt(left / dragNumber), -parseInt(top / dragNumber));
    }

    await t
        .expect(targetMarker.visible).ok()
        .click(targetMarker)
});

This code is still not 100% reliable, however, I think it's a good starting point to solve the issue.

Thanks, this pointed me in the right direction! I settled on this for my solution (additional help from https://stackoverflow.com/questions/52472321/need-help-to-check-if-element-is-in-current-viewport-with-testcafe)
        const visibleMarkers = <Selector>Selector(() => {
            const windowHeight = window.innerHeight;
            const windowWidth  = window.innerWidth;
            const docWidth     = document.documentElement.clientWidth;
            const docHeight    = document.documentElement.clientHeight;

            return Array
                .from(document.querySelectorAll("img[src*='_school_marker.png']"))
                .filter((el: HTMLElement) => {
                    const boundValues = el.getBoundingClientRect();
                    return boundValues.bottom > 0 && boundValues.right > 0 &&
                        boundValues.left < (windowWidth || docWidth) &&
                        boundValues.top < (windowHeight || docHeight);
                });
        });
        const targetMarker = visibleMarkers.nth(0);


Decrypting encrypted key creates empty file

While attempting to deploy an encrypted key recently I ran into the problem that the key_convert.sh script would not create a usable decrypted key.
I ended up having to decrypt the key using openssl but was momentarily stumped as to why the script fails to do what it says it's doing.
I plan to investigate this on my own but if you (@cvidler) get to it before me feel free.

going to need some diagnostics about the issue to do anything about it.
add -ddd to the command line for extra debug output.


Make configmaps more granular and introduce secrets where needed

We need to check and cleanup the deployment charts for the apps, so that we make sure we:

don't put unused variables in the apps
don't have big configmaps, but specific ones only for a particular set of configurations
use kubernetes secrets for secret configuration items

Configuration has been segmented into separate configMaps for the separate sources. At the moment we don't use kubernetes secrets.

Configuration has been segmented into separate configMaps for the separate sources. At the moment we don't use kubernetes secrets.


Clearer labelling before/after

It would be good to use color coding on every label for before/after toggling, as sometimes it gets tricky to remember which is which :)

Thanks, I'll add labelling down the sides every 200px or so.


youtube-player "ready" event isn't fired.

Reproduction
9.2.4 version "ready" event isn't fired after rendering component
Also, I caught the error in the browser console:
Failed to execute 'postMessage' on 'DOMWindow': The target origin provided ('https://www.youtube.com') does not match the recipient window's origin ('https://domain').
Expected Behavior
The "ready" event should  fire after rendering YouTubePlayer component
Have the error in the browser console " Failed to execute 'postMessage' on 'DOMWindow': The target origin provided ('https://www.youtube.com') does not match the recipient window's origin ('https://domain')."
Environment
Angular: 9.1.9
CDK/Material: 9.2.2
Angular YouTubePlayer: ^9.2.2
Browser(s): all
Operating System: Windows

After updating youtube-player package to the latest version 10. 1.3  (ready) event is invoked successfully


Fix missing syslog items in config schema

Riak CS fails due to missing syslog configs if one attempt to enable syslog by log.syslog=on in riak-cs.conf.
This PR also includes a small refactoring commit on tests.
Error messages
% bin/riak-cs console
19:26:45.918 [error] Error generating configuration in phase apply_translations
19:26:45.918 [error] Translation for 'lager.handlers' expected to find setting 'log.syslog.ident' but was missing
Error generating config with cuttlefish
  run `riak-cs config generate -l debug` for more information.

@ksauzz Good catch. Could you write down here the workaround you showed in the customer ticket?

Workaround
Add lager configs into advanced.config including lager_syslog_backend like:
 {lager, [
          {handlers, [
                      {lager_syslog_backend, ["riak", local1, info]},
                      {lager_console_backend, info},
                      {lager_file_backend, [{file, "/var/log/riak-cs/error.log"},
                                            {level, error},
                                            {size, 10485760},
                                            {date, "$D0"},
                                            {count, 5}]},
                      {lager_file_backend, [{file, "/var/log/riak-cs/console.log"},
                                            {level, info},
                                            {size, 10485760},
                                            {date, "$D0"},
                                            {count, 5}]}
                     ]}
         ]},

app.config in Riak CS 1.5.x is a good example to set lager configs in advanced.config.

Ah, yes. Copy and pasting from 1.5 app.config is the best way.

@ksauzz Could you back port this to 2.0 branch as well? I think this is worth being backported.

@Basho-JIRA open a ticket for this

@kuenishi Will do.

@borshop merge

Will also be backported to 2.0.
[posted via JIRA by Kota Uenishi]

@borshop merge

Release note: Fix a configuration system bug where Riak CS could not start if log.syslog=on is set. See original pull request for workaround in 2.0.x.
[posted via JIRA by Kota Uenishi]


Execution failed for task ':app:mergeDexProdRelease'.

[x] I have read the Get Started - Installation section
[x] I have read and done the Get Started - Setup Android section
[x] I have read and done the Get Started - Setup iOS section
[x] I have already searched for the same problem

Environment

Technology
Version

Flutter version
3.0.1

Plugin version
-

Android version
-

iOS version
-

Xcode version
-

Description
when I try to create a build using android flavor then it failed and gives this error.
Error

* What went wrong:
Execution failed for task ':app:mergeDexProdRelease'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.Workers$ActionFacade
   > com.android.builder.dexing.DexArchiveMergerException: Error while merging dex archives:
     Learn how to resolve the issue at https://developer.android.com/studio/build/dependencies#duplicate_classes.
     Type com.pichillilorenzo.flutter_inappwebview.BuildConfig is defined multiple times: /home/user/Documents/Flutter_Projects/myproject/build/flutter_inappwebview_quill/.transforms/178f333172fee0932eea9714e74812c6/classes/classes.dex, /home/user/Documents/Flutter_Projects/myproject/build/flutter_inappwebview/.transforms/66f9bac8836a8f0fbfe12bfeaeaef088/classes/classes.dex

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org/

BUILD FAILED in 41s

Had the same error. I had flutter_quill inside my pubspec.yaml. Removing it fixed the problem. I'm guessing it's due to flutter_inappwebview_quill conflicting with flutter_inappwebview.

I'm currently facing the same issue, but removing flutter_quill from my dependencies is unfortunately not an option for me. @ngc1514 I agree that the issue lies with flutter_inappwebview_quill, which uses the exact same package name (com.pichillilorenzo.flutter_inappwebview) as the original flutter_inappwebview package.
@garv-shah would it be possible to change the flutter_inappwebview_quill package name so that it doesn't collide with the original package?

hi there! I've updated my repository with the name change, but seeing as flutter_quill_extensions doesn't have a GitHub repository anywhere, I'm not sure how to get that change to be updated on Flutter Quill itself.
Seeing as how flutter_inappwebview has had a major refactor since when I made the package, I think it would probably make more sense get Flutter Quill to depend on the official repository itself

@garv-shah thanks a lot for the quick package name change! I'll try to get in touch with the maintainers of flutter_quill_extensions and flutter_quill.

Seeing as how flutter_inappwebview has had a major refactor since when I made the package, I think it would probably make more sense get Flutter Quill to depend on the official repository itself
Does that mean that you didn't do any changes specifically for quill in your fork? If so, I totally agree.

Does that mean that you didn't do any changes specifically for quill in your fork?

Yup, exactly, it was merely a fix for a bug that this repository had with the Xcode 14/iOS 16 betas that prevented building which got fixed officially in this PR, and was meant to be temporary since it was blocking app building on the latest iOS version.
Here's the exact issue: https://github.com/singerdmx/flutter-quill/pull/883
Either way, it was just meant to be a temporary change since as predicted it took about 4 months for it to be fixed officially, so it's definitely long overdue a change back :D

@garv-shah thanks a lot for the clarification! 🙏


Add new assignment: hello-world

This PR adds a new assignment for ECMAScript: hello-world. It's a very simple exercise, copied from exercism.io JavaScript track. This exercise is configured before the existing one because it's simpler.
Additionally, it changes the way to compile ECMAScript and to run tests during development time, in Travis CI. Now, it uses Gulp tasks, same as individual exercises do, to compile and run tests.
There was a problem with traceur and jasmine-node. Compiling code with a traceur command and running tests with jasmine-node caused this error:
$ make test
\e[4mRunning tests for hello-world assignment\e[0m
Exception loading: /tmp/hello-world.jBtjpsXi1A/hello-world_test.spec.js
[ReferenceError: $traceurRuntime is not defined]
ReferenceError: $traceurRuntime is not defined
    at Object.<anonymous> (/tmp/hello-world.jBtjpsXi1A/hello-world.js:1:80)
    ...

traceur included a reference to $traceurRuntime even before the code could require the traceur-runtime.js file (where that variable is initialized). Google, Stack Overflow or Traceur documentation say nothing about this issue.
"use strict";
$traceurRuntime.options.symbols = true;
require('./traceur-runtime');
var HelloWorld = $traceurRuntime.initTailRecursiveFunction(function() {
  //...

There is another tool (npm module) to run tests, jasmine. It needs a JSON configuration file, where traceur-runtime.js can be configured to be loaded before running the tests. And it works for individual exercises, but it doesn't for all exercises testing them with the make command. make command compile code and move some files to temp folders. The only way to access them is through absolute paths. And jasmine tool doesn't work with absolute paths. So, jasmine can't be used.
Everything works with Gulp tasks for individual exercises. This PR provides some Gulp tasks, that can be run from the make command. These tasks reads temp folders to use from the command line and they are able to use those temp folders to get the code, compile it and run the tests.

Thanks! I think having the gulp tasks is an improvement.
The more I think about it, the more I am skeptical of actually vendoring the node modules. What do you think? How hard would it be to git ignore all the node modules and then make people run a command to install dependencies?
In the future, would you please submit new assignments as separate PRs? It's really hard to review pull requests when they contain different types of changes.

Thanks for merging the pull request. I wasn't sure this solution was the best, because it's quite different from the tools used with JavaScript in exercism.io, but in ECMAScript, code must be compiled and this extra step make the process a bit more complicated.
To ignore or not to ignore node modules
That's the question. There are arguments that support both, with pros and cons. An intermediate point of view seems to emerge: check in into the repository those modules you need to deploy your application, ignore those you need only for development. They recommend to check in production modules not to depend on third parties (a external module repository).
Personally, I prefer ignoring modules. Module repositories exist for a reason, to download modules, so use them!. But now that I've read other opinions, I like very much the intermediate approach: check in only production modules.
How hard would it be? It should be quite easy. Just a line in .gitignore and delete node_modules directory from the repository. Then, anybody will be able to install all dependencies with just one command npm install.
Single PR for each new assignment
My original idea was to do exactly that: one single PR for each new assignment. But, as I faced the problem with traceur and jasmine-node I thought it would be a good idea to make a PR with both the new assignment and the solution. If I'd made a PR only with the new assignment, tests would have failed. If I'd made a PR only with the solution, there wouldn't have been a problem to solve that justified the solution.
So, my apologies to mix several concerns in the same PR. I could have tried harder. I could have tried to explain the problem first, then PR a solution, and finally PR the new assignment. So, again, I have to thank you for accepting this PR. I can do better, I promise :smiley:

I don't know what the right answer is, but I lean towards having people do an npm install. It seems kind of silly to have thousands of files stored in this git repo, when the whole point is that one should be able to install dependencies with one command.


plugin functionality: read multiple fields during a single load

As it is now, to load multiple fields you'd need to load multiple json files. It would be much better in terms of performance (and convenience) to be able to add multiple fields-to-sample to the json file. The schema and model ingestion side should be easy, need to check on how to return multiple 3D image layers for napari. I think it's a simple as putting the image layers in a list, but need to confirm and try it out.

Confirmed that just adding another tuple to the return from yt_napari._reader.reader_function() will return a separate layer. So can easily read multiple fields and return an image layer for each (with layer name and metadata -- see #15 ).


[questions] get warnings after upgrade

Hi, after I upgraded eve-wspace I get some  "OneToOne is better than ManyToMany" warnings. They appear when I make my migration  (./manage.py migrate --fake) as well as when I start the server  (./manage.py runserver <IP_ADDRESS>:8000). How to solve this and how to migrate from a build 3 month ago?
Some additional: is there still a possibility to get real time help e.g. IRC?

https://github.com/evewspace/eve-wspace/pull/219#issuecomment-167875224

Besides this, I hope you're using ./manage.py runserver for testing/development purposes only. :smile:

Of course, I use it only for testing. I have now changed the models so that the warnings disappear. Pull request follows.


docs: fix broken link in README.md

Give an overview of the tasks completed
Fixed simple typo/broken link in README.md
Link to issue(s) that this PR closes
N/A

lfg sina


kpod_start

Starts one or more containers.
Signed-off-by: baude<EMAIL_ADDRESS>

Also needs kpod.1.md updated with start. readme.md transfer.md

LGTM  other then the question on failed attach.

Looks like you have some gofmt issues.

LGTM once tests green

@rh-atomic-bot r+

@bot retry

@baude could you rebase an push.  I don't seem to be able to retrigger tests

@rh-atomic-bot r+


make a databroker podman image

This works:
podman run -it -p 8000:8000 -e TILED_SINGLE_USER_API_KEY=secret -v ./example_config.yml:/deploy/config.yml databroker
Using configuration from /deploy/config.yml
INFO:     Started server process [1]
INFO:     Waiting for application startup.
OBJECT CACHE: Will use up to 10_095_076_147 bytes (15% of total physical RAM)
INFO:     Application startup complete.
INFO:     Uvicorn running on http://<IP_ADDRESS>:8000 (Press CTRL+C to quit)

In response to @dylanmcreynolds' suggestion above that we should avoid exposing the MongoDB to the host network, I have rolled https://github.com/bluesky/databroker/issues/735 into this. In retrospect I should have handled that first (in its own PR) before attempting this. But I hope this is still reviewable.

Added POST /documents to provide a way to get documents into MongoDB via the service
Modified the structure returned by GET /documents for consistency and alignment with a current conversation in event-model
Added a method to the Python databroker client post_document and refactored db.insert to use it internally. This is expected to incur a small performance penalty compared to direct insertion into MongoDB, but I feel confident that it is the correct thing to do. Separately, in production at facility scale, bluesky-kafka or other means can be used. But the basic built-in behavior should use the service, not reach around to an exposed database port.

@mrakitin Oh, interesting. I think the docker-compose.yml should be modified.
Instead of:
image: databroker

we should use:
build:
    context: .

This will tell dockercompose to build on first run, or any time you start docker-compose with the ---build flag.

Thanks for the fast review, all.
Yes, I meant to add a GHA workflow, same as we have for Tiled, but I forgot. Thanks for catching that @mrakitin.
I intended to change image to point to a tagged and built image on the GH container registry. @dylanmcreynolds’ suggestion would be slower on first build but more dynamic in a way that seems overall useful.

As for how often to build: for tiled we build on merges to main and on tags. We do not build on every commit. That seems to be the right “volume”. Most people will use tagged images but it will be possible to work from bleeding edge.

I switch to
build:
    context: .

and confirmed that it works as @dylanmcreynolds describes, in a clean environment. I also added a GH workflow. We'll find out once it's merged into main whether it works as expected. I copied it directly from the tiled repo and there is nothing repo-specific about it, so I expect it will work.

Thanks, Dan! I think in my message I rather meant building the image and tagging without push to make sure the integration test works (i.e. use the added workflow + set up the environment & run examples/write_and_read_back_data.py).


Data structure desgin documentation of STREAM GROUP

Search before asking

[X] I had searched in the issues and found no similar issues.

Motivation
We should update the data structure documentation of STREAM to include the encoding of stream groups.
Solution
@Yangsx-1 Could you help?
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

I'll do it later.

I submitted a PR at kvrocks-website. apache/kvrocks-website#222  @PragmaTwice


Suboptimal behavior when updating file without changing size

Currently if a file consists of multiple blocks and one of them is modified, LFS will relocate that block as well as all of the following blocks, even if they have not been modified (this can happen if the file consists of records of fixed size).
This results in number of writes scaling with file size, not with the size of updated data.
This should be optimized to avoid relocating unmodified blocks.

I have the same question, when append data to an exist 20kB file ,even it’s just 1 byte ,LFS will erase many blocks .This consumes too many sectors, and it may not even be as efficient as directly deleting the original sector to update data。
I

Hi @rojer, @muguou, thanks for creating an issue.
Yeah, this is a known issue, and something I'm currently working on. The CTZ skip-list is an inherently append-only data-structure, and random writes aren't performant.
The solution I'm working towards is to replace the CTZ skip-list with a more traditional B-tree data structure. It's a big change, so it will take some time to be ready.
The B-tree code itself is actually working at the moment, but it still needs the exact mapping to files, as well as testing and other considerations.

@geky thanks for confirming, and good to know that a solution is in the works.
meanwhile, is it true that the most efficient write operation on existing files would be an append?

Yes, append is very efficient, up until you hit bottlenecks with the block allocator.
Actually, at an average of $O(1)$ operations to append to CTZ skip-lists, they in theory outperform most filesystem's file data structures in terms of pure append. There may be some other niche use case for them in other filesystems, but they aren't the right solution for littlefs.

I would love for an option to turn off the cloning and instead make a temporary copy of the affected block(s) and upon commit/sync copy those blocks back over the originals keeping unchanged blocks CTZ list intact.
This does allow for power resets during this operation leaving a block in the potentially erased state cutting off the list completely though... One only looses one file, not the filesystem.

Here's a thought:
If creating a new block(s) to overwrite parts of a file is done in new block, then when closing or syncing file one could copy the parts not overwitten in file to last of new blocks and then submit a "plan" to metadata/log about the file, and the new block(s). Then start the replacing of the original blocks. If interrupted one can compare the contents of the planned overwrites in new blocks to the exiting file blocks to detect progress and continue. When finished the file is again valid and the metadata plan can be updated/invalidated and the temporary new blocks can be released. The filesystem is then still valid upon interruption although some "work" remains before the file affected can be read as the CTZ list would pass through blocks that are in the middle of being erased or updated. It would however speed up edits on large files by a lot. In fact, it may be prudent to only trigger this behavior on very large files.

I am using battery-supply and monitoring battery levels and remaining charge with a fuel gauge chip giving me some fore-warning to powerloss. I shut down file access and close files before any power is cut. This does not save me from powerloss during product disassembly but that is acceptable in my use-case as opening to disconnect battery requires taking system out of production/used state which also closes all files. For other use-cases I agree that it may not be very useful.


unshare: 'Permission not found'

Attempting to remove sharing from a file, by ID, returns a 'permission not found' error:
% drive stat -id 1yX...Rw

1yX...Rw
Filename                  Name of file
FileId                    1yX...Rw
Bytes                     0
Size                      0.00B
DirType                   file
VersionNumber             1825593
MimeType                  application/vnd.google-apps.document
Etag                      "fbe...5NA"
ModTime                   2015-11-20 03:06:43 +0000 UTC
LastViewedByMe            2015-11-20 02:55:18 +0000 UTC
Shared                    true
Owners                    My Name
LastModifyingUsername     My Name
OriginalFilename
Md5Checksum
Copyable                  true
Starred                   false
Viewed                    true
Trashed                   false
ViewersCanDownload        false

*
Name: My Name<EMAIL_ADDRESS>Role                 owner
AccountType          user
*

*
Name: Someone Else<EMAIL_ADDRESS>Role                 reader
AccountType          user
*

*
Name: myappsdomain.net <>
Role                 reader
AccountType          domain
*

% drive unshare -id 1yX...Rw
Revoke access for accountType: user for file(s):
+  Name of file

Proceed with the changes? [Y/n]:
Name of file: googleapi: Error 404: Permission not found: user, notFound

(This is for files on an Apps domain, if that makes a difference.)
Trying the various -type options has the same result. Isn't unshare missing an -emails flag?

Hello @bdlow thanks for reporting this and welcome to drive. I'll take a look at it and investigate more over the weekend. I agree unshare might be trippy, I've found it awkward too.

Thanks for reporting this issue, addressed by PR #486.
Closing it in lieu of a new release, please reopen if it persists.

Thanks @odeke-em ; seems to work well for --type user.
I've had no success with --type domain (stat show an empty email address for domain); neither of the following works: (file is the same as in the OP, now with the<EMAIL_ADDRESS>user removed):
% drive unshare --emails 'example.net' --type domain -id 1yX..Rw
no matches found!
% drive unshare --emails '' --type domain -id 1yX..Rw
no matches found!

Hey @bdlow you can only unshare permissions that are already set which is why you get the "no matches found" notice, however I think I have an idea of what is tripping out.
Would you mind showing me a stat sample and I'll take it from here, I think I have a hunch on what is happening. However, I'll be able to look at it maybe at the week's end as am trying to clear out the last assignments before the semester ends, plus other commitments.
Thanks.

@bdlow thank you for the patience, please get the latest from master. I created an issue that was a follow up of your comments and addressed it with PR https://github.com/odeke-em/drive/pull/504.

Excellent, it's working well now!
Thanks,
Ben
On 30 November 2015 at 06:02, Emmanuel T Odeke<EMAIL_ADDRESS>wrote:

@bdlow https://github.com/bdlow thank you for the patience, please get
the latest from master. I created an issue that was a follow up of your
comments and addressed it with PR #504
https://github.com/odeke-em/drive/pull/504.
—
Reply to this email directly or view it on GitHub
https://github.com/odeke-em/drive/issues/484#issuecomment-160450329.


Installation of new pytorch version via Anaconda prevents GUI from opening

I received a "TORCH CUDA version not installed/working" error in Cellpose and installed the newest version of Pytorch via Anaconda using this line:
conda install pytorch torchvision -c pytorch
When I open the user interface using the 'python -m cellpose' line, I get a traceback with this error as the final line:
'SystemError: initialization of _internal failed without raising an exception'
Any ideas on how to get Cellpose up and running again?

cellpose installs pytorch via pip, so switching to a conda installation might be the source of this issue.
You can (1) try to install the pytorch and torchvision packages via pip by calling pip install torch torchvision while in your activated cellpose environment. Or (2) wipe the existing and problematic conda env and reinstall it using the installation instruction on the readme/docs.
Let me know if that works!

I actually think this error may be caused by a conflict between numba (too old) and numpy (>=1.24.0), although pytorch could be the culprit too. I fixed this error on my local computer with pip install numba --upgrade

Hi all, I tried to install the pytorch and torch vision packages as well as update numba. Both attempts failed to load the UI- I will remove the problematic env and re-install through conda.
Thanks for your help!
From: Michael Rariden @.>
Date: Friday, May 5, 2023 at 8:31 PM
To: MouseLand/cellpose @.>
Cc: Naira Abou-Ghali @.>, Author @.>
Subject: [EXTERNAL] Re: [MouseLand/cellpose] Installation of new pytorch version via Anaconda prevents GUI from opening (Issue #699)
cellpose installs pytorch via pip, so switching to a conda installation might be the source of this issue.
You can (1) try to install the pytorch and torchvision packages via pip by calling pip install torch torchvision while in your activated cellpose environment. Or (2) wipe the existing and problematic conda env and reinstall it using the installation instruction on the readme/docs.
Let me know if that works!
—
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_MouseLand_cellpose_issues_699-23issuecomment-2D1536938769&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=dOu84OJAwgPwkbxp49Xym_Ja1mxEoMqkirmc6t9TKBs&m=SsfKzVAHj3qeKJffnZx9hL66oS0_saGM2RzTOKs__eTZjsf6o1a_JmtxOyN8Z-T0&s=Q8yi4qXus-HdfcT9LKGZpnlJFsdMT49iRRG_8uIe32U&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_A7S4QCP63HHMCAQXDHLZT23XEWLU3ANCNFSM6AAAAAAXU2P7AI&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=dOu84OJAwgPwkbxp49Xym_Ja1mxEoMqkirmc6t9TKBs&m=SsfKzVAHj3qeKJffnZx9hL66oS0_saGM2RzTOKs__eTZjsf6o1a_JmtxOyN8Z-T0&s=AaNvstTDG3ZOAUouDzc3sI10NxoMkxMBht6J1mpTOuU&e=.
You are receiving this because you authored the thread.Message ID: @.***>


Split plug phoenix

Coverage increased (+7.6%) to 97.248% when pulling d6bdc5c174da3287cec817d18d642740fe77f341 on split-plug-phoenix into d23a9085807788cc0edc191ce29ea17d27fe29cf on master.


drivers: virtualization: Map ivshmem-v2 sections individually

Recent changes to the arm64 MMU code (https://github.com/zephyrproject-rtos/zephyr/commit/c9b534c4ebca8c3d02bc7d783c79771f1409d096) mean that you can no longer map R/O memory as R/W.
Mapping R/W memory now causes a cache invalidation instruction (DC IVAC) that requires write permissions or else a fault is generated. Modify ivshmem-v2 to map each R/O and R/W section individually.
The eth_ivshmem driver assumed the ivshmem-v2 output sections would be mapped contiguously, which is no longer true. Modify eth_ivshmem to treat each output section independently.
Further discussion: https://github.com/zephyrproject-rtos/zephyr/pull/64779#issuecomment-1792980385
If these changes are accepted, could they please be backported to v3.5-branch?
CC: @carlocaione @xakep-amatop

Hey, just giving this a bump


[extracting][dataloss] Undo highlighting if extraction was cancelled.

This is a good idea. I will see what I can do.

This can now be undone the same way as other changes.


cuelang.org: CUE version that defines the spec is not linked to the site's CUE version

What page were you looking at?
https://cuelang.org/docs/reference/spec/
What version of the site were you looking at?
https://github.com/cue-lang/cuelang.org/commit/0f80600d1d2083d1e94276c13b532b71171ae12f
What did you do?
I tried to document where and what versions of CUE are encoded  in the cue-lang/cuelang.org repo; and what effects their upgrades have. In doing so I was reminded that:

The version of CUE that the site uses is constrained by site.cue.
The version of the spec that is published at https://cuelang.org/docs/reference/spec/ is constrained by the version of CUE selected by go.mod, which is upgraded manually through running go get cuelang.org/go@latest (or similar) whilst inside cue-lang/cuelang.org.

What did you expect?
That upgrading the version of CUE that the site uses should automatically pull in the spec from the upgraded version of CUE.
What did you see instead?
The published spec (harmlessly) doesn't include the trivial changes merged in 7ed1bcaa253fc9111fd61ee3713da11b1f2e464a.

This should follow an approach similar to https://review.gerrithub.io/c/cue-lang/cuelang.org/+/1194099. genspec could be updated to create a temporary module from which the spec details are copied. We don't need a separate module for this dependency.


Add configuration option to ignore DNS RCODE 3 as errors

Description

This pull request introduces a new configuration option ignore_dns_rcode3_error that allows users to specify whether DNS responses with RCODE 3 (Name Error) should be treated as errors or not.
Motivation and Context
In Kubernetes clusters using the ClusterFirst DNS policy, KubeDNS may return RCODE 3 for public domains, which are usually treated as errors. This behavior may lead to unnecessary alerts and logging, especially when users are aware of this behavior and do not consider it an error.
How Has This Been Tested?

The compiled image runs normally.
After setting the ignore_dns_rcode3_error configuration to true and running the dig nonexistentsubdomain.dns3test.com command, the absence of trace metrics in Kindling indicates that the configuration change is effective and DNS queries with an RCODE of 3 are no longer considered errors.

Please add TestCase for TestDnsProtocol() in network_analyzer_test


Heroku one click web-console setup

Simplifies things a bit more by adding a one click web-console setup.

Does the one-click web-console setup remove the color codes? If not, the log becomes an unreadable mess

Sadly not, idk how to exactly do that automatically :x

@xPaw The way the remove coloring works is by making another version of cheat.php without the color code changes at line 773. The only downside is that whenever you want changes to cheat.php to reflect to cheat_heroku.php, you have to merge cheat.php into cheat_heroku.php

@xPaw Up to you if you want to have the second commit of this PR Remove coloring (#1) .. @ALEEF02 just thought It'd be better to do since colors are broken for Heroku logs, if you don't want to do this, I'll reverse this bit and just keep the first commit.


Improve error handling

empty url -> strange message
bad vendor -> crash

required for
https://github.com/keboola/developer-portal-cli-v2/issues/9

fixed in  https://github.com/keboola/developer-portal-php-client/commit/1e105b2bea83a290d454e6f504db8c1d3df93f83


Frontend won't start in Fedora 22

The service starts, all good, but no web interface.
Oct 01 14:31:48 localhost.localdomain ether-proxy[1797]: 2015/10/01 14:31:48 Loading config: /home/user/ether-proxy/config.json
Oct 01 14:31:48 localhost.localdomain ether-proxy[1797]: 2015/10/01 14:31:48 Running with 6 threads
Oct 01 14:31:48 localhost.localdomain ether-proxy[1797]: 2015/10/01 14:31:48 Upstream: main => http://<IP_ADDRESS>:8545

Sorry. Firewall issues :D works great !


--blocks_per_json_file flag not working as expected

Using the following CSV, test.csv:
85c044897c2460,98.000000
3b31,45.000000

With the following command line invocation, assuming ./json exists:
./bazel-bin/gematria/datasets/convert_bhive_to_llvm_exegesis_input --json_output_dir=./json --bhive_csv=./test.csv --blocks_per_json_file=1

We get the following in ./json:
0.json  1.json  2.json

Note that we  should only get two files.
0.json:
[
  {
    "Hex": "85c044897c2460",
    "MemoryDefinitions": [
      {
        "Name": "MEM",
        "Size": 4096,
        "Value": 305419776
      }
    ],
    "MemoryMappings": [
      {
        "Address": 65536,
        "Value": "MEM"
      }
    ]
  }
]

1.json:
[
  {
    "Hex": "3b31",
    "MemoryDefinitions": [
      {
        "Name": "MEM",
        "Size": 4096,
        "Value": 305419776
      }
    ],
    "MemoryMappings": [
      {
        "Address": 65536,
        "Value": "MEM"
      }
    ]
  }
]

2.json:
[]

We see one of the blocks duplicated, the second block shows up twice, we get an extra file, and the extra file is empty. This needs to be fixed.

BB is not actually duplicated. I apparently just can't look at hex values.


Bottom navbar get hidden when scrolled down.

Describe the bug
Bottom nanvbar is hidden when scrolled down to the limit
This bug may have introduced here: https://github.com/codeforcauseorg/edu-client/pull/63
To Reproduce
Steps to reproduce the behavior:

Open application in development mode.
scroll down to bottom and bottom navabar will be disappeared once.

Environment (please complete the following information):

OS: Ubuntu
Browser: Google chrome

@Abhishek-kumar09 can I work on this?

Sure go on. Let me know if need some help. You can also seek some help from @ADRE9.

Sure go on. Let me know if need some help. You can also seek some help from @ADRE9.

Ya sure. And Thanks for assigning this to me I will try my best order to solve this.


Impossible to use string values in methods

Using the minutes() and hours() methods of class Rewieer\TaskSchedulerBundle\Task\Schedule, it is impossible to use a string as value. Their declaration requires it to be an int and thus prevents the use of the wrapper for such cases.
Current workaround is to use Cron\CronExpression's setPart().

Can you explain what kind of values are you trying to feed these methods ?

For instance, it is impossible to use minutes('1,2,3,5,8,13,21,34,55').
Moreover, I found that using an int 0 will be displayed likewise in getExpression(), but will act as an * while this is not the case when using string 0.
However, this rather seems to be a Cron/CronExpression problem as it happens even when bypassing the wrappers that anyway pass the values as is and setting the Cron directly;

For the minutes function indeed I didn't plan to use it with different precise minutes, so this is a use case to add.
So when using '0' it works as intended but 0 is replaced by * ? Is it really the getter fault, or did you use a setter function that assumes that 0 = * and therefore stored at it as * ?
I'm not much into CRON jobs these days so I don't have much time to investigate this more

I didn't investigate much more either, just enough to find it must be a Cron/CronExpression problem as no custom code was used here, only CronExpression->setPart(). I will submit them and link the thread to this one.
Here is the trace :

Create a new task extending AbstractScheduledTask and register it in the scheduler;

In the initialize method, call $schedule->getCron()->setPart(CronExpression::minutes, 0);

$schedule->getCron()->getExpression() will now return 0 * * * *
Call recurrency : * * * * *

OR

In the initialize method, call $schedule->getCron()->setPart(CronExpression::minutes, '0');

$schedule->getCron()->getExpression() will now return 0 * * * *
Call recurrency : 0 * * * *

For the use case, I already made a commit for it on my fork. However I don't know if the tests should be updated accordingly.
Packages versions :
dragonmantank/cron-expression : v2.2.0
symfony/symfony : v2.8.49
rewieer/taskschedulerbundle : 0.1.0

Indeed this seems to be directly related to the Cron library. We should wait to see if this is going to be fixed.
If unfortunately, it wont, we shall fix it ourselves.

is this still relevant?

@marcin-derlukiewicz It might be. Can you propose a PR ?

@Zasco you mentioned that you made some kind of fix can you link it?

@marcin-derlukiewicz I would assume that https://github.com/Zasco/TaskSchedulerBundle/commit/8aba436d9804a4a740b14212884d6b36a8c9141e is the commit you are looking for?

ok, so for this functionality there are 2 methods already:
you can use shrothand method $schedule->setPart() (without getCron()) like here https://github.com/rewieer/TaskSchedulerBundle/blob/eb6a27226b6dba8beb1febe88197b68f6830f6ba/Tests/Task/ScheduleTest.php#L97
or use setExpression for entire definition like this:
https://github.com/rewieer/TaskSchedulerBundle/blob/eb6a27226b6dba8beb1febe88197b68f6830f6ba/Tests/Task/ScheduleTest.php#L85

Sorry, I just noticed that there already is a fix for this. Just decline my PR if you want to go with the solution from @Zasco.

I think this is fixed now with #37


Create CloudWatch alarms to watch the available free space

Thanks for creating the issue. We will add this ability.


issue-tracker demo is error

As image above, Watchman is required to watch for changes.

install watchman by brew install watchman
OR
change package.json
-    "start": "yarn run relay; concurrently --kill-others --names \"react-scripts,relay\" \"react-scripts start\" \"yarn run relay --watch\"",
+    "start": "yarn run relay && react-scripts start",

Thanks! It works for me.


Deploy Documentation

Codecov Report

Merging #1 (96e76bf) into master (d0c82e9) will decrease coverage by 0.56%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master       #1      +/-   ##
==========================================
- Coverage   82.95%   82.38%   -0.57%
==========================================
  Files           5        5
  Lines         176      176
==========================================
- Hits          146      145       -1
- Misses         30       31       +1

Impacted Files
Coverage Δ

src/LaxZonedDateTimes.jl
79.61% <0.00%> (-0.98%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d0c82e9...96e76bf. Read the comment docs.


Top-Bar Link Color is White

By default, $topbar-link-color is white - which is really hard to see on the topbar which is also a light color.

Yeah, that was added in 6.0.5 for some reason. It comes from:
      color: #fff; }```

Yep, this has been fixed but hasn't landed in a release yet. We'll likely do 6.0.6 by the end of the week. 9bf0552ef5f737c1e209e9efd5c2e76b05afbdbe


Pull and assemble /html/external during build time

It's preferred that we pull these external resources in at build time versus keeping them in our repository as dependencies.

Using npm to pull in web dependencies during build.

#267 will be the packaging/bundling strategy after this release.


Cannot eval on COCO test-dev on Google Colab

I am trying to use YOLACT++ from Google Colab. Inference on images/videos works fine, but I can't evaluate on COCO test-dev.
I run this command:
python eval.py --trained_model=weights/yolact_plus_resnet50_54_800000.pth --dataset=coco2017_testdev_dataset
And I get:
Traceback (most recent call last):
 File "eval.py", line 1105, in <module>
  evaluate(net, dataset)
 File "eval.py", line 936, in evaluate
  img, gt, gt_masks, h, w, num_crowd = dataset.pull_item(image_idx)
 File "/content/yolact/data/coco.py", line 172, in pull_item
  if target.shape[0] == 0:
AttributeError: 'NoneType' object has no attribute 'shape'

If I change line 172 in coco.py to check if target is None I get another error:
Traceback (most recent call last):
  File "eval.py", line 1105, in <module>
    evaluate(net, dataset)
  File "eval.py", line 956, in evaluate
    prep_metrics(ap_data, preds, img, gt, gt_masks, h, w, num_crowd, dataset.ids[image_idx], detections)
  File "eval.py", line 390, in prep_metrics
    gt_boxes = torch.Tensor(gt[:, :4])
TypeError: 'NoneType' object is not subscriptable

I can get past this error by adding the argument --output_coco_json, but I don't think it's the right way to use the script. I think it's a bug but could also be a Colab related issue (or some error of mine).

@Maxinho96 I commented out ''if target.shape[0] == 0: ...''

@Maxinho96 I commented out ''if target.shape[0] == 0: ...''

Did it work? I get the second error.

this solves the first problem and generate two json file, but i meet another problem(those files are too large for my model output)

the mask_detections.json is 947M, it is impossible to upload it to the test server @dbolya

I have just realised that the second error is normal because normal evaluation is not supported with COCO test-dev because it has no ground truth, so you can only evaluate it with json file + test server.
BTW the first error still remains...

You can use --score_threshold=0.05 to reduce the file size. As for the first bug, that's an oversight on my part. For now you can comment it out. I'll try to get a fix out today.

i got the same error, how to solves?

i got the same error, how to solves?

Comment this line:
https://github.com/dbolya/yolact/blob/092554ad707c2749631dfe545c8a953b2b3f4a68/data/coco.py#L172

i got the same error, how to solves?

Comment these lines:
https://github.com/dbolya/yolact/blob/092554ad707c2749631dfe545c8a953b2b3f4a68/data/coco.py#L172-L174
and use --output_coco_json flag.

For me, I add another condition based on the previous one as below:
"if target.shape[0] == 0 and  self.has_gt:"

i got the same error, however, even though I use --output_coco_json flag and commented out ''if target.shape[0] == 0: ...''， I still get the second error:
Traceback (most recent call last):   File "eval.py", line 1106, in <module>     evaluate(net, dataset)   File "eval.py", line 937, in evaluate     img, gt, gt_masks, h, w, num_crowd = dataset.pull_item(image_idx) TypeError: cannot unpack non-iterable NoneType object

i got the same error, however, even though I use --output_coco_json flag and commented out ''if target.shape[0] == 0: ...''， I still get the second error:
i got the same error, however, even though I use --output_coco_json flag and commented out ''if target.shape[0] == 0: ...''， I still get the second error:

i got the same error, however, even though I use --output_coco_json flag and commented out ''if target.shape[0] == 0: ...''， I still get the second error:
Traceback (most recent call last):   File "eval.py", line 1106, in <module>     evaluate(net, dataset)   File "eval.py", line 937, in evaluate     img, gt, gt_masks, h, w, num_crowd = dataset.pull_item(image_idx) TypeError: cannot unpack non-iterable NoneType object


Add highlight component

Creates and renders a highlight component for the srcView component. A couple of things to note:

I believe this could work for the SearchResults use of highlight with some additions, but I haven't attempted that yet
One thing I feel a little uncertain about is the pulling of positional data from the DOM, in make_highlight (L65). It seems to be working fine, but I think if we're seeing the highlight end up in the wrong location, there may be some render lifecycle conditions where the final position isn't yet available. If that's the case, I think we'd need to move most of the make_highlight code into componentDidUpdate in srcView, and put positional data in state, pass down as props, etc.

Cool, yeah, I can do that. A couple questions:

This means we can remove the last line calculation from the Highlight component, correct?
The highlight logic for the Line component could be removed, but I think it would need to stay for LineNumber, right? I could look at refactoring it a bit though to determine the actual line number that should be highlighted, and pass that down as a prop, rather than passing highlight_start and highlight_end.

This means we can remove the "last line" calculation from the Highlight component, correct?

yep

but I think it would need to stay for LineNumber, right?

Hmm, yeah. I think it would be nice to have a highlighted prop on line number and have the owning component set that from the highlight, rather than have the line number check highlight. That sounds like what your proposing with the refactoring, yes?


Feature : warning for vecLib on Mac

Running examples/train_mnist.py resulted in unreasonable behaviour (exploding loss & low accuracy) as follows:
Masaki-no-MacBook-Pro:mnist omeometo$ python train_mnist.py
GPU: -1
# unit: 1000
# Minibatch-size: 100
# epoch: 20
epoch       main/loss   validation/main/loss  main/accuracy  validation/main/accuracy  elapsed_time
1           60.7941     84.565                0.52985        0.5502                    47.3283
2           235.41      484.166               0.532617       0.3074                    89.3148
3           587.228     244.073               0.55655        0.7105                    123.146
4           738.536     386.792               0.60925        0.6935                    154.823
5           1135.55     2467.39               0.6131         0.5234                    185.989
6           1712.49     916.105               0.60735        0.6731                    216.959
7           2311.43     3468.54               0.62715        0.5642                    248.193
8           2610.56     2006.75               0.633933       0.6788                    278.72
9           3358.19     1443.68               0.644533       0.723                     310.921
10          4369.74     2402.41               0.6331         0.6985                    342.471
11          4168.4      1578.52               0.661133       0.7946                    374.207
12          6144.09     2845.26               0.654467       0.7004                    405.498
13          6025.8      2245.78               0.66355        0.8174                    436.602
14          6675.77     7182.15               0.661317       0.6203                    467.895
15          7697.48     5312.16               0.66345        0.6928                    499.086
16          7775.43     5914.33               0.6727         0.737                     530.525
17          10321.6     6359.2                0.655067       0.7048                    561.185
18          9819.28     18361.4               0.680817       0.5946                    592.306
19          11861.1     13093.2               0.678433       0.6231                    623.2
20          12883.1     5928.4                0.67095        0.7723                    654.457

It seems to have something to do with vecLib on MacOS. Here is the result of numpy.show_config() :
blas_mkl_info:
  NOT AVAILABLE
blis_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
atlas_3_10_blas_threads_info:
  NOT AVAILABLE
atlas_3_10_blas_info:
  NOT AVAILABLE
atlas_blas_threads_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
blas_opt_info:
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
lapack_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
  NOT AVAILABLE
atlas_3_10_threads_info:
  NOT AVAILABLE
atlas_3_10_info:
  NOT AVAILABLE
atlas_threads_info:
  NOT AVAILABLE
atlas_info:
  NOT AVAILABLE
lapack_opt_info:
    extra_compile_args = ['-msse3']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

This seems to be the same problem as in this post. It would be desirable to have some warning message about using chainer with vecLib.

I also encountered the same problem before and took quite some time to track down the cause.
Displaying warning would be very helpful for users who are unfamiliar with numpy backends.


User story request: shared namespaces

As a user, I want containers to be able to share certain aspects of a namespace,  i.e. to have multiple containers in the same network namespace, that's isolated from the host namespace.
I believe such a feature is needed for the pod concept of k8s & friends to function.

This is covered in my simple-jail proposal here which allows a container's config to request that its jail is a child of some other container jail.


Implement default threadid callback for Unix (fixes #426)

There are also examples in the test file for Irix, Solaris, BeOS, and Netware; however I'm not familiar with these OSes so I haven't added fallbacks for these. Let me know if I should.

the openssl threading stuff is not great.  As @kaduk points out, the current API does not portably work for pthreads!  which is amazingly horrible.  The probably best work-around is to define a new API which is CRYPTO_threadid_set_pthread which takes a pthread_t.

CRYPTO_THREADID seems to be used everywhere though. If that truncates pthread_t we might have an issue.

How many uses of CRYPTO_THREADID could not be replaced by thread-local storage?

@kaduk, you're probably right and we should look at doing that.  Remove THREADID for the more friendly named TLS :)

Wouldn't we need to do that backwards compatibly? Code relying on the old thread safety APIs would break when dynamically linked against a TLS based openssl. I'm not sure if there's a way to move threadid-based functionality to tls-based functionality without breaking things. (But I'm not too familiar with the internals here so ICBW)

We are breaking backward compatibility with the 1.1 release.  Code will need to be recompiled.  We have been saying this for months.

For a new thread implementation, see #451

Sweet.

If it's possible to have all the thread ID stuff replaced with thread-local storage stuff, I'm all for it.
If we need to keep tracking the thread ID somewhere, I think this needs a bit more work than just provide a default Unix / pthread implementation.  Not to say that in itself is a bad thing, but it should go further.  For example, I'd very much like the current CRYPTO_THREADID structure tossed to hell and replaced with something like this (tentative and in the same style as in #451):
#if !defined(OPENSSL_THREADS || defined(CRYPTO_TDEBUG)
typedef int CRYPTO_THREADID;

CRYPTO_THREADID CRYPTO_THREADID_current()
{
    return 1;
}
#elif defined(OPENSSL_SYS_WINDOWS)
# error "Windows currently unimplemented"
#else
# include <pthread.h>
type pthread_t CRYPTO_THREADID;

CRYPTO_THREADID CRYPTO_THREADID_current()
{
    return pthread_self();
}
#endif

Other implementations added with more #elif sections...
Makes sense?

Would we be able to do that for all the OSes? We'll probably need a way for users of obscure OSes to hook into this, I guess (and what if they have threadids which may not fit in unsigned longs?)
Anyway, yes, this makes sense :smile: .

(and what if they have threadids which may not fit in unsigned longs?)

The appropriate typedef should take care of that, as I'm sure they all have a thread identity or pointer type of some sort.  The point is to use the CRYPTO_THREADID type consistently and not try to retrofit it into an unsigned long.

Yeah, but this can't be hooked into dynamically (unlike the current API), that's my point.
(I guess we'll just have to ask folks on obscure OSes to add their OS to this file?)

That's a good point. I'll ponder on that a little bit.

I'm starting to think that we should really provide a structure CRYPTO_THREAD_METHOD, with function pointers for thread ID, thread local storage and locking, and a few implementations for some well known thread platforms, such as pthreads and Windows threads, and have one of them being a default where it makes sense.  Of course, there's also going to be a non-threads method...
Then, all that would be needed would be to have a function that can be used to override the default implementation for the platform at hand, and voilà!, anyone can set up their own method and have that used instead.
Does this make sense?  If it is, I'd be happy to put it together...
(ping @ghedo , since you're doing this kind of stuff as well)

I removed the comment I just wrote...  the thought wasn't quite finished.

Worst case some platforms define CRYPTO_THREADID as a pointer and dynamically allocate whenever it's necessary.  Very hackish though.

Yeah, that's a sure memory leak waiting to happen...

@richsalz might want to close this as well.

Yes, I'll close this in favor of #451


Page-Scrolling is prevented on mobile devices

Description
I mobile devices I can not scroll any further when I reach the carousel. Probably the .preventDefault breaks the normal page scrolling if the user starts the touch on the carousel.
To Reproduce
Steps to reproduce the behavior:

Go to dev.agapegemeinde.de with a mobile device (iOS+Safari and Android+Firefox tested)
Scroll down the page until you reach "Gruppen und Aktionen" try to scroll further down (the carousel will always prevent it)

Expected behavior
A threshold of movement to the side before the scroll down is prevented
Tested on (Smartphone)

Device: iPhone SE (Safari), Samsung Galaxy S7 (Firefox v.88.1.4)

PreventDefault is disabled for touching devices check in v0.1.20. Please update the carousel to the latest version.


Updated RenderDataRectangleNode.HitTest to properly hit-test rounded rectangles

What does the pull request do?
Updates the RenderDataRectangleNode.HitTest logic to properly hit-test rounded rectangle scenarios based on what is rendered on-screen.
What is the current behavior?
Rounded rectangle hit-testing doesn't currently work right and RenderDataRectangleNode only does a full rectangular bounds hit-test.  This is very evident when you have something like a circular button, where the circle is provided by a Border or ContentPresenter with large CornerRadius.  In that scenario, the button can incorrectly be clicked when the mouse is in a corner of its rectangular bounds, even though it's well outside the visual circular bounds.
What is the updated/expected behavior with this PR?
The old hit-test logic is still used for non-rounded rectangles.  But new hit-test logic is used for rounded rectangles.
How was the solution implemented (if it's not obvious)?
A new RoundedRectContains method for rounded rectangle hit-testing was added to complement the Rect.ContainsExclusive method used for non-rounded rectangle hit-testing.
Breaking changes
None.
Fixed issues
Fixes #13784
Unit tests
No unit tests were added yet since the HeadlessPlatformRenderInterface.SupportsIndividualRoundRects property returns false, which prevents RenderDataRectangleNode from ever being used.
A unit test with the following code added to CompositorHitTestingTests passes if HeadlessPlatformRenderInterface.SupportsIndividualRoundRects is true.
   [Theory,
        // Corners
        InlineData(50, 50, false),
        InlineData(150, 50, false),
        InlineData(150, 150, false),
        InlineData(50, 150, false),
        // Indent 10px
        InlineData(60, 60, false),
        InlineData(140, 60, true),
        InlineData(140, 140, false),
        InlineData(60, 140, true),
        // Indent 17px
        InlineData(67, 67, false),
        InlineData(133, 67, true),
        InlineData(133, 133, true),
        InlineData(67, 133, true),
        // Center
        InlineData(100, 100, true),
    ]
    public void HitTest_Should_Return_Expected_Result_For_RoundedControls_At_Point(double x, double y, bool expectsHit)
    {
        using (var s = new CompositorTestServices(new Size(200, 200)))
        {
            var border = new ContentPresenter
            {
                Width = 100,
                Height = 100,
                Background = Brushes.Red,
                CornerRadius = new CornerRadius(60, 10, 50, 30),
                HorizontalAlignment = HorizontalAlignment.Center,
                VerticalAlignment = VerticalAlignment.Center
            };

            s.TopLevel.Content = border;

            if (expectsHit)
                s.AssertHitTest(new Point(x, y), null, border);
            else
                s.AssertHitTest(new Point(x, y), null);
        }
    }

@Gillibald - Good suggestion.  Do you want me to do it or you?

CI build looks good in my testing in our app.


Bug: HASS Agent crashing when try add GPU Load or Temperature sensors

Describe the bug
My HASS Agent crashes whenever I try to add either the GPU Load or GPU Temperature sensors, weirdly it is fine with the CPU Load & Clock sensors so I assume it has something to do with GPU drivers, I have an AMD RX 7900 XTX.
To Reproduce
Steps to reproduce the behaviour:
Click add sensor and select either GPU sensor then when hit "Store & Activate Sensors"
Expected behavior
The sensor to be added and the agent to remain open
Misc info (please complete the following information):

Windows build (ideally screenshot/info of winver.exe output):

Windows' UI language:  English (UK)
HASS.Agent version: 2022.14.0

Please check what's applicable (multiple answers possible):

[x] Installed via installer
[ ] Installed manually
[x] Problem occurs in HASS.Agent
[ ] Problem occurs in Satellite Service

Logs
Nothing in logs, just quits as if the process were killed and then starts normally on the next boot.

This is similar to my problem #386
For reference, I'm using an RX480

This is a duplicate of my problem #386 For reference, I'm using an RX480

Just saw your issue update specifying which sensors made it crash, mine works fine with lastactive and lastboot or any combination discluding the GPU monitoring, assuming its AMD driver conflicts then.

Having the same problem as you.  6800XT can't add GPU load or temperature without crashing the app.   Nothing shown in HASS agent logs, but the following are in the Event Viewer:
Application: HASS.Agent.exe CoreCLR Version: 6.0.2623.60508 .NET Version: 6.0.26 Description: The process was terminated due to an internal error in the .NET Runtime at IP 00007FFFD6CF3C89 (00007FFFD6B20000) with exit code c0000005.
Faulting application name: HASS.Agent.exe, version: 2<IP_ADDRESS>, time stamp: 0x63558578 Faulting module name: coreclr.dll, version: 6.0.2623.60508, time stamp: 0x656f6ab2 Exception code: 0xc0000005 Fault offset: 0x00000000001d3c89 Faulting process id: 0x1f24 Faulting application start time: 0x01da56ac2b7ad6b0 Faulting application path: C:\Users\matth\AppData\Roaming\LAB02 Research\HASS.Agent\HASS.Agent.exe Faulting module path: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.26\coreclr.dll Report Id: 266d03dc-afcd-46f2-9df4-73b02a63a2c9 Faulting package full name:  Faulting package-relative application ID:
Both OpenHardwareManager and Libre are able to display the information without issue.

same problem, with windows 11 and nvidia gtx 1080

Same issue on my end.
Nothing in the logs, even with extended logging activated.

Expand for Event Viewer details
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
	<System>
		<Provider Name="Application Error" Guid="{a0e9b465-b939-57d7-b27d-95d8e925ff57}" />
		<EventID>1000</EventID>
		<Version>0</Version>
		<Level>2</Level>
		<Task>100</Task>
		<Opcode>0</Opcode>
		<Keywords>0x8000000000000000</Keywords>
		<TimeCreated SystemTime="2024-02-19T08:46:43.5226074Z" />
		<EventRecordID>921</EventRecordID>
		<Correlation />
		<Execution ProcessID="12896" ThreadID="27284" />
		<Channel>Application</Channel>
		<Computer>REDACTED</Computer>
		<Security UserID="REDACTED" />
	</System>
	<EventData>
		<Data Name="AppName">HASS.Agent.exe</Data>
		<Data Name="AppVersion">2<IP_ADDRESS></Data>
		<Data Name="AppTimeStamp">63558578</Data>
		<Data Name="ModuleName">coreclr.dll</Data>
		<Data Name="ModuleVersion">6.0.322.12309</Data>
		<Data Name="ModuleTimeStamp">6216cac2</Data>
		<Data Name="ExceptionCode">c0000005</Data>
		<Data Name="FaultingOffset">00000000001cd70f</Data>
		<Data Name="ProcessId">0x36c0</Data>
		<Data Name="ProcessCreationTime">0x1da63101e4f8e34</Data>
		<Data Name="AppPath">C:\Users\REDACTED\AppData\Roaming\LAB02 Research\HASS.Agent\HASS.Agent.exe</Data>
		<Data Name="ModulePath">C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.3\coreclr.dll</Data>
		<Data Name="IntegratorReportId">9e551b18-4f77-4983-b0d4-f30b42d539b3</Data>
		<Data Name="PackageFullName">
		</Data>
		<Data Name="PackageRelativeAppId">
		</Data>
	</EventData>
</Event>

I can confirm this is still an issue as well.

This is also happening to me with both GPU sensors.

Confirm that adding the GPU Load sensor will cause the application to crash (RTX 4080)

Likewise in Windows 11 and using a RX 6800. Tried GPU Load and Temp

amd Ryzen 9 7950X3d and a nvidea rtx2090 cant get Hassagent to run for more then 10 seconds

is there any fix for this?

Not that I’m aware of.  Just unmonitor the gpu
On Sun, Aug 25, 2024 at 1:49 AM manalotobry @.***>
wrote:

is there any fix for this?
—
Reply to this email directly, view it on GitHub
https://github.com/LAB02-Research/HASS.Agent/issues/388#issuecomment-2308665827,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABHDREXGXNNJ3QTQGDRAN3ZTFV75AVCNFSM6AAAAABBVRYEWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBYGY3DKOBSG4
.
You are receiving this because you commented.Message ID:
@.***>

Same Issue here with a AMD Radeon RX 6700 XT with Adrenalin Version 24.7.3 on Windows 10


Prevent semantic downgrades in bin update

Includes:

[x] Changes to the cmd.getLatestVersion() function
[x] Tests of the cmd.getLatestVersion() function

LGTM. Added a minor nit comment.


Documentation

Is it possible to have documentation about other functions (like set_temperature)?

See http://evohome-client.readthedocs.org/en/latest/api2/temperature.html#set-a-zone-temperature
Any other bits that are obviously missing?