[Narwhal] improve resilience of nodes growing and following DAG

Currently, the limiting factor to reduce Narwhal round generation speed (hence reducing e2e Narwhal latency) is the resilience of nodes growing and following DAG. It is observed that once more Narwhal rounds are generated per sec, at some limit nodes start to fail behind in following the DAG, resulting in higher e2e latencies. We plan to make the following improvements to help nodes receiving certificate out of order or missing one or more of them to stay up-to-speed with growth of the DAG.
Propagating recently created certificates

[ ] Move Certificate handling from Core into Synchronizer, which allows concurrent processing of certificates and access to metadata.
[ ] When broadcasting created own certificates, each peer should be sent a bounded list of certificates that they may need, not just the most recently created own certificate.
[ ] Create a buffer to keep track of known certificates that have missing parents, and certificates only known by digests. Their dependencies should be tracked so they can be accepted efficiently when possible.

Improving certificates catchup throughput

[ ] Improve efficiency of certificate verifications during catchup, by verifying higher round certificates and only ensuring digests match for child certificates.
[ ] Flesh out details for chaining certificates from own origin.
[ ] Improve efficiency of certificate fetching by fetching from highest rounds of accepted certificates per origin, instead of trying to fill in certificates within gc_depth.
[ ] Pipeline certificate fetching after verifications are no longer a major factor in catchup time.

When broadcasting created own certificates, each peer should be sent a bounded list of certificates that they may need, not just the most recently created own certificate.

I assume we'll want to do that only to the nodes who haven't voted for the underlying header, right?

Improve efficiency of certificate verifications during catchup, by verifying higher round certificates and only ensuring digests match for child certificates.

To confirm that child certificate digests match we'll need to have previously the child certificates processed? Trying to see the difference with current approach.

I assume we'll want to do that only to the nodes who haven't voted for the underlying header, right?
Why is that the case? This should be done against all peers for created certificates that have not been ack'ed, with a max size bound e.g. 100, to avoid the recipient having to rely on the higher latency fetch process.

To confirm that child certificate digests match we'll need to have previously the child certificates processed? Trying to see the difference with current approach.

The difference is that the previous child certificates do not have to be verified, if their digests are included transitively from verified certificates. The caveat is fetch response served from these data may contain certificates with unverified signatures now, so the recipient of the fetch response needs to be able to handle that.


Implement action's event propagation

It's completed and available but there's no handler for distributed systems.


Support selection of SDKs for QPU Selection

In context of Algorithm > Implementation > NISQ Analyzer:
When starting a new analysis, the user should be able to select what SDKs/compilers should be considered for compilation and analysis in the pop-up.
In case the cloud provider is IBMQ, pytket and Qiskit should be selectable (and shown).
In case the cloud provider is Rigetti, pytket and Forest should be selectable (and shown).
Note: At least one of the presented SDKs has to be selected, otherwise the analysis can not start.
Based on the selection, the compilersToUse list should be set.

Solves #45


Is ev-emitter.js a dependcy for imagesloaded?

I get an error in browser console (EvEmitter is undefined) if I don't load ev-emitter.js in my page. It seems this is a dependency for using imagesloaded, but the readme doesn't say so, so just wondering. Also is there a way to use imagesloaded without ev-emitter.js? I prefer to have as few dependencies in my script as possible.

You can use imagesloaded.pkgd.js or imagesloaded.pkgd.min.js, which includes dependencies

Thanks. For some reason that had not fixed the error before, but I don't see error anymore.


[Paimon-734][Core]Support Timestamp LTZ

(Please specify the module before the PR name: [core] ... or [flink] ...)
Purpose
https://github.com/apache/incubator-paimon/issues/734
Tests
(List UT and IT cases to verify this change)
API and Format
(Does this change affect API or storage format)
Documentation
(Does this change introduce a new feature)

Hello, @hk-lrzy, is there any progress or problem?

Seems that you may not have time for this ticket, I am afraid that I should close this.


merge_mode="concat" behaving unexpectedly? (with the GPU backend)

I'm building a network using the new Graph model which takes two sequences as inputs, feeds them through an RNN, concatenates the final output vectors, and passes that on to a dense layer. I expected that the input size of dense layer should be twice the output dimensionality of the RNNs but this doesn't seem to be working.
Does anyone know what I may be doing wrong?
graph = Graph()
graph.add_input(name='peptide', ndim=3)
graph.add_input(name='mhc', ndim=3)

# RNN for peptide sequences
graph.add_node(JZS1(input_dim=20, output_dim=5), name="peptide_rnn", input="peptide")

# RNN for MHC sequences
graph.add_node(JZS1(input_dim=20, output_dim=5), name="mhc_rnn", input="mhc")

# concatenate last output of both RNNs and transform them into a lower dimensional space
graph.add_node(Dense(5 * 2, 16, activation="relu"), name="hidden", merge_mode="concat", inputs=("peptide_rnn", "mhc_rnn"))

# output prediction of affinity
graph.add_node(Dense(16, 1, activation="sigmoid"), name="affinity", input="hidden")

# weird that I have to name the output twice
graph.add_output(name='affinity_output', input='affinity')
graph.compile('rmsprop', {'affinity_output':'mse'})

The error that I get is:
Traceback (most recent call last):
  File "train-jsz-rnn.py", line 138, in <module>
    history = graph.fit({'peptide':padded_peptides, 'mhc':padded_mhc, 'affinity_output':log_target_values}, nb_epoch=1)
  File "/home/ubuntu/keras/keras/models.py", line 537, in fit
    validation_split=validation_split, val_f=val_f, val_ins=val_ins, shuffle=shuffle, metrics=metrics)
  File "/home/ubuntu/keras/keras/models.py", line 135, in _fit
    outs = f(*ins_batch)
  File "/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", line 588, in __call__
    self.fn.thunks[self.fn.position_of_error])
  File "/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", line 579, in __call__
    outputs = self.fn()
ValueError: dimension mismatch in args to gemm (128,5)x(10,16)->(128,16)
Apply node that caused the error: GpuDot22(GpuJoin.0, hidden_W)
Inputs shapes: [(128, 5), (10, 16)]
Inputs strides: [(5, 1), (16, 1)]
Inputs types: [CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, matrix)]

It seems like the two RNN outputs are not actually getting merged into a 10 dimensional vector, is that expected?
edit: This error goes away (and the model trains successfully) when using the Theano CPU backend.
edit 2.0: The error goes away even on the GPU if I switch the JZS1 RNN to an LSTM.
edit 3.0 Actually, the error persists with LSTMs but only if I'm using floatX=32. Current hypothesis: merging of input variables doesn't get handled correctly when calling an external matrix multiply routine.

This is an old Theano bug. Just upgrade to the latest version of Theano (get it from Github) and you'll be all set.

thanks


Failed to load resource

Small issue, but when I right-click to Inspect Element on the browser page, there's a Console error:
Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost:51004/public/jquery.min.map

Related to #66

I tried adding "jquery.min.map" (renamed from "jquery-2.1.3.min.map") and the uncompressed "jquery-2.1.3.js" to my /Library/WebServer/Documents folder (assuming that's where localhost is referencing on my Mac), but the Console error is still there..

"jquery.min.map" default should be placed into ~/Library/Application\ Support/Sublime\ Text\ 3/Packages/OmniMarkupPreviewer/public/ folder.
A more convenient solution can be: Download ZIP-Extract & Rename the folder to OmniMarkupPreviewer-Replace /Library/Application\ Support/Sublime\ Text\ 3/Packages/OmniMarkupPreviewer folder with the newer one

Oh, wow! Thanks so much. This is exactly what I was looking for.
I didn't know there was already a /public folder for OmniMarkupPreviewer but I followed your second recommendation (who doesn't love an upgrade??) and now there's no Console error, hah! :)


Public facing services exposed via nginx-ingress suffer from higher latencies

I have 2 public-facing services, handled by nginx-ingress-controller. Since injecting Conduit into one of the service deployments (the green one on the following chart), its avg response time (as reported by nginx to prometheus) keeps going up.

Looking at the Conduit stats in the dashboard for this service, I am not seeing this rise in response time.
Restarting the ingress controller pods made the response time go back down where it usually is, but it started growing again over time.
In short, the calls go from (nginx) -> (conduit proxy -> green service)
and conduit's stats don't reflect that latency increase, but nginx's do.

I'm curious if #434 could be related to this -- does the green service talk to MongoDB by any chance?
My theory is that latency increases as we leak connections.

Yes, it does talk to Mongo but not via Conduit anymore (I am excluding the Mongo port from outbound connections via --skip-outbound-ports).

I don't have any leads yet on to why the latency is increasing, but I think I do know why Conduit isn't reporting it: the http Event::StreamRequestOpen is triggered just before we serialize the request, and so any time since having accepted the request originally and waiting on DNS, connecting, and waiting in queue is not calculated in the latency.

So, we do collect telemetry on both client and server socket opens and closes, but it looks like those events don't get forward into prometheus, and so cannot be shown in the conduit dashboard. So, the next is: is it possible to collect stats from nginx and/or the service in regards to if they are creating/receiving many connections that then just sit idle, never used again?

I separated out the aspect of the telemetry not being accurate into #479 so that this issue can be about the latency being bad and/or increasing.

Hi @bourquep,
We're looking into reproducing this behavior right now. It would be helpful to know what protocols are being used in your case: Is it HTTP/1, HTTP/2, or TCP by chance?
Thanks!

Ingress traffic is HTTP/1 (that may be upgraded to 2). Intra-service comms are via gRPC.

Quick update:
In an attempt to recreate this, we've had services receiving traffic via the conduit proxy for about ~30 hours now and have not seen an increase in latency.
Screenshot below:

Right now, we've added nginx to route calls to the proxy. So far we haven't seen an increase in latency, though it's only been ~1 hour. We will keep this running for another day as well to see if anything changes and give another update.
This is the config we're using in an attempt to recreate the issue:
https://gist.github.com/franziskagoltz/67de000256122ba37d326657cbd8ef15
However, if you have a config that recreates this issue and are willing to share it with us, that might help us with debugging and speed up the process. If there's anything you'd be willing to share, we'd appreciate it. Thanks!

In my case, the main HTTP handler for the ingress traffic calls into a few internal services via gRPC and HTTP, all of them proxied by Conduit.
Could I be observing the sum of overhead introduced by each proxy traversal?
I will try this again once 0.3.1 is released, and I'll try to gather more data to help diagnose this issue.

@bourquep https://github.com/runconduit/conduit/releases/tag/v0.3.1 😸

What?!? Did I miss the announcement? Anyhow, cool! I'll give it a spin tomorrow!

The announcement is only about 30mins old 😄
And great, thanks! We'll update as well once the new test has been running for a while longer.

I've been running 0.3.1 in prod for about an hour, and so far I haven't observed the raise in ingress latency that I was experiencing initially. So far so good!

Wonderful, very glad to hear! Is it still latency still stable?
We've been running the test overnight and so far also have not been able to reproduce it either.

We've been running the test overnight and so far also have not been able to reproduce it either.

Did we reproduce it on 0.3.0?

Yes, I should have been more clear. We've been trying to reproduce it with 0.3.0 but without success.

Yup, still stable. Here's the same chart as I had initially posted when opening this issue, for the past 3 hours with Conduit 0.3.1 injected into the same services as initially:

So I'm not quite sure what the cause was, but anyhow it is fine now. I believe you can close this issue now. #479 will still be useful, but unrelated to the current issue.

Alright, I'll go ahead and close the issue. Please feel free to reopen it again, if you see the issue again.


feat(command-functions-list): remove hidden flag

- Summary
Added  command functions:list to the docs
- Test plan
nil
- Description for the changelog
functions:list command which is missing in the docs have been added
- A picture of a cute animal (not mandatory but encouraged)

Sorry could not keep up with you, Thanks for including it in the docs!


'[[' syntax works in bash, but is undefined in the default shell. '[[: not found' exception will raise in some cases.

Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
Types of changes
What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] Documentation Update (if none of the other choices apply)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[ ] I have create an issue on Doris's issues, and have described the bug/feature there in detail
[ ] Commit messages in my PR start with the related issues ID, like "#4071 Add pull request template to doris project"
[ ] Compiling and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works
[x] If this change need a document change, I have updated the document
[ ] Any dependent changes have been merged

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

Long time passed. And I find this PR isn't essential. Close it.
If you have some question, feel free to contact me by @jackwener  / reopen this PR.


Geo Areas eager loading

remove geo areas from eager loading for fetching commodity measures

 [TARIFF17] CTL00003565 GSP Duty Rate Query


Include helm charts in release

Partial implementation for https://github.com/istio/istio/issues/21861.

@ostromart: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

gencheck_istio
0d2f5a1a5d0e08552dfad2982291bfa48297a51e
link
/test gencheck_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ostromart: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

unit-tests_istio
47025ae0ec7e83f5ce320220d6b93b02f7501fef
link
/test unit-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

In response to a cherrypick label: new pull request created: #21911


Local development Node image for web-twig-demo

Rebuild your images using make build and then using make start you should make the whole dev environment up and running. This also solves the problem with the core-js polyfills (@adamkudrna).
Into the Node container are mounted the demo app and the web package.
Into the PHP container are mounted the demo app and the web-twig package.
Please let me know if all works fine.

Coverage: 96.209%. Remained the same when pulling caeec67fb87c3cc1457d95b4eb3aee25197c8c0f on chore/web-twig-demo-node-image into e4355586a1071899b365c1161a4a06f4215b0160 on main.

@adamkudrna yarn build is fixed now by https://github.com/lmc-eu/spirit-design-system/pull/814 and https://github.com/lmc-eu/spirit-design-system/pull/815. This branch was rebased and contains those fixes.

@adamkudrna Certificates: The docker dev is based on official Symfony docker support, so this link https://github.com/dunglas/symfony-docker/blob/main/docs/tls.md could help. Beware of the different names of the containers. I will add the command for this soon.


All services use the same network(external) but some service can't find the correct address of another address

Description
~/.local/share/nerdctl/1935db59/etchosts/default/b3571ed508bb30a1598eb9f782ff49d134bccbe500dcbf34ef15bfd41641a743/hosts have multiple lines of the same service

and dns results for different services are different which cause ping error （The service can not use the correct ip）

the correct is ：

Steps to reproduce the issue
I don't know how it happens
Describe the results you received and expected
All services use the same external network can communicate with each other。
What version of nerdctl are you using?
nerdctl version 0.20.0
Are you using a variant of nerdctl? (e.g., Rancher Desktop)
No response
Host information
No response

Hello, would you mind to give us the compose file you use to reproduce this issue?

create network
network  create  --subnet <IP_ADDRESS>/24 --gateway <IP_ADDRESS> test

create redis-1.yml and redis-2.yml

redis-1.yml
version: "3"
services:
  redis-1:
    image: redis:7.0.2
    container_name: redis-1
    command: ["redis-server","/usr/local/etc/redis/redis.conf"]
    ports:
      - "51710:6379"
    volumes:
      - /data/Dockerfile/test/conf-1:/usr/local/etc/redis
      - /data/Dockerfile/test/data-1:/data
    networks:
      test:
        aliases:
          - redis-1
networks:
  test:
    external: true

redis-2.yml
version: "3"
services:
  redis-2:
    image: redis:7.0.2
    container_name: redis-2
    command: ["redis-server","/usr/local/etc/redis/redis.conf"]
    ports:
      - "51720:6379"
    volumes:
      - /data/Dockerfile/test/conf-2:/usr/local/etc/redis
      - /data/Dockerfile/test/data-2:/data
    networks:
      test:
        aliases:
          - redis-2
networks:
  test:
    external: true

start redis-1

nerdctl compose -f redis-1 up -d

start redis-2 multiple
nerdctl compose -f redis-2 up -d
nerdctl compose -f redis-2 down
nerdctl compose -f redis-2 up -d
nerdctl compose -f redis-2 down
nerdctl compose -f redis-2 up -d

ping on redis-1
nerdctl exec -ti redis-1 bash
> apt update
> apt install iputils-ping
> ping redis-2

check
nerdctl inspect redis-1

cat /home/saas/.local/share/nerdctl/1935db59/etchosts/default/66e72d84cccad33b476875dc85823a5eb6a4c5050fc25cd2ef2dbe476cc92ede/hosts

check actual redis-2 ip

I have reproduced this issue.  It seems like DNS resolver resolve the redis-2 target as the first redis-2 container's address

It seems like every time execute nerdctl compose -f redis-2 down  the DNS record about redis-2 have not be cleanded

I can reproduce this by simply doing nerdctl run @Zheaoli
nerdctl run -d --name ng nginx:1.19-alpine
nerdctl --debug-full run --rm alpine:3.13
nerdctl --debug-full run --rm alpine:3.13
nerdctl exec ng cat /etc/hosts
# <nerdctl>
<IP_ADDRESS>	localhost localhost.localdomain
:1		localhost localhost.localdomain
<IP_ADDRESS>       localhost localhost.localdomain localhost localhost.localdomain localhost localhost.localdomain
:1              localhost localhost.localdomain localhost localhost.localdomain localhost localhost.localdomain
<IP_ADDRESS>      3ea616afbe2b alpine-3ea61
<IP_ADDRESS>      762787dc0f14 ng
<IP_ADDRESS>      871bc4b69381 alpine-871bc
# </nerdctl>


transport/resource API version on xDS integration tests

Description:
Currently, the transport/resource API version is set V2. I think that we should change them into V3. In addition, we should support multiple resource API versions(V2/V3) for xDS integration test. This is like downstream/upstream HTTP protocol or v4/v6 with TestWithParam.

I would be worried about combinatorial explosion if we change every test to be v2/v3 with a test param. We already have a dedicated integration test to validate the space of transport/resource versions in https://github.com/envoyproxy/envoy/blob/master/test/integration/api_version_integration_test.cc.
I would advise that we switch most tests over to v3 and retain one legacy integration test for v2, essentially a smoketest.

@Shikugawa further thoughts on this one?

@htuch I agree with your idea. For the moment, we should transfer the V2 testing to V3 by remaining one of V2 integration test.

nostale


Visuals of CLI output not good!

Dear goron,
The visuals of output or the results are not so good.

Ok naryal, I'll try to make the visuals better

Dear Goran,
I have already forked your project. I'll make the visuals way better. And, also remove the dependency of huepy.

Thanks @naryal2580 for code contribution.


Add missing UpdateKey parameter

This documentation seems to be missing the UpdateKey argument for FindSet.  I grabbed the copy from the Dynamics NAV page for the same function hoping they are identical.
Please let me know if you need anything else.

@brentmclark this parameter was obsoleted.
Maybe that's the reason why this was removed.
If this is readded, then it should mention that this parameter is obsoleted now.
https://learn.microsoft.com/en-us/dynamics365/business-central/dev-itpro/developer/methods-auto/record/record-findset-boolean-boolean-method#parameters

@pri-kise thanks for letting me know this argument was marked obsolete.  I'm happy to add a note about that, but am unsure of the format.  Can you point me toward another spot in the documentation where an argument is marked obsolete so I can be sure to get the format right?
Thanks!


Auto mode possible with broken gyroscope?

I.e. accelerometer and compass work fine. Diagnostics show that clearly! 👍🏻
Old device but i hope still may be useful. 😞

Excuse my blindness. found that

Glad you found the option! We don't do this automatically as you'd be surprised how many phones report that they don't have a particular sensor but actually do.


Outdated download link on https://prometheus.io/download/#alertmanager

What did you do?
Opened https://prometheus.io/download/#alertmanager
What did you expect to see?
Download link and checksums of released Alertmanager 0.20.0
What did you see instead? Under which circumstances?
Releases 0.19.0 and 0.20.0-rc.0

This would be a docs issue, @juliusv


Unable to start Browservice on Windows 10 20H2 with WSL2

I followed through all the compile steps but changed the things needed for Issue #34 and am having the Network service crashed, restarting service. error looping over and over

If you only changed the file viceplugins/retrojsvice/html/main.html, then this is most likely not related to the manual changes. Googling the error message, one lead I get that it might be an issue with Chromium on some locales. Which locale do you use in WSL, i.e. what is the output of command locale? Does it work if you run it using the C locale using command LANG=C.UTF-8 LC_ALL=C.UTF-8 release/bin/browservice?
Another thing we should check is whether this happens only for Browservice or all CEF programs. To check that, could you try to run the cefclient example browser following the instructions in this issue comment and see if it works or if it loops the same error.

Output of locale
LANG=C.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_PAPER="C.UTF-8"
LC_NAME="C.UTF-8"
LC_ADDRESS="C.UTF-8"
LC_TELEPHONE="C.UTF-8"
LC_MEASUREMENT="C.UTF-8"
LC_IDENTIFICATION="C.UTF-8"
LC_ALL=

It doesn't run with running LANG=C.UTF-8 LC_ALL=C.UTF-8 release/bin/browservice
I built the cefclient but am unable to find it

OK, so the issue is probably not with locales.
As for the issue with building cefclient, could you show the output of the build command make -j3 and see how far the path tests/cefclient/Release/cefclient exists and what is in the bottommost existing directory (e.g. if tests/cefclient/Release exists, what files are there)? Also, please confirm that you are using the CEF version downloaded by the download_cef.sh script of Browservice <IP_ADDRESS>.
When you get cefclient working, you probably don't have a working X server setup so you need to run it using Xvfb as the X server using the command xvfb-run tests/cefclient/Release/cefclient (of course we can't see the window, but you can see whether you get Network service crashed, restarting service. errors).

I'll try again today, did a reinstall to 21H1

Running the appimage does not work and extracting the appimage then running browservice.desktop doesn't work as well
https://asciinema.org/a/1yc4jZhlKU8z0NgnThXZeuPzU -- Running appimage
https://asciinema.org/a/BrIszYQuQgLZI6sxYOYatccxx -- Starting through the extracted thing

Running AppImage. The error message fuse: device not found, try 'modprobe fuse' first seems out of place. You would typically only get this on WSL1. Are you sure you are running WSL2? What is the output of the Linux command uname -a?
Extracting the AppImage and running the extracted version. I see two issues:

Do not extract it under /mnt/c. It might work with luck, but the Windows filesystem might do weird things to it. Try extracting it under /home.
Run AppRun and not browservice.desktop (the desktop file is not executable; it has executable permissions only because you extracted it under /mnt/c).

uname -a output: Linux Revision-PC 4.4.0-19041-Microsoft #488-Microsoft Mon Sep 01 13:43:00 PST 2020 x86_64 x86_64 x86_64 GNU/Linux
Extracting under home gives the same result:
https://asciinema.org/a/vzx9WURwG71wMCeCruuzSBkTx

That uname -a output seems to indicate that you are running WSL1, not WSL2. To double-check this, run wsl -l -v in Windows command prompt. The VERSION tab shows the WSL version. For WSL2 installation, the output should look like this:
C:\Users\topi>wsl -l -v
  NAME            STATE           VERSION
* Ubuntu-20.04    Running         2

If you are indeed running WSL1, follow the instructions to switch to WSL2.

Closing due to lack of activity. If you still have problems, open a new issue.


It still sends you to Slack

Challenge http://www.freecodecamp.com/challenges/waypoint-pair-program-on-bonfires has an issue.
It still sends you to Slack here...
"6. We have a special chat room for people ready to pair program. Go to our http://freecodecamp.slack.com/messages/letspair and type "Hello Pair Programmers!" "

Thank you for reporting this, but it has already been fixed in a recent pull request that I can't find because I'm on mobile.

If it's fixed, it's not deployed. I'm not seeing it. (The "Pair" buttons on the all Bonfire pages do point to Gitter, but not instruction #6 on this page.)

You're right, it's not in production yet. From what I understand, all development is done on the staging branch and systematically pushed to production from there. This specific issue was fixed in 7004e81b4a29b196a659bcb5cdcef30253b71d6f

Nine days ago...
On Jul 1, 2015 1:07 PM, "Anton Strömkvist"<EMAIL_ADDRESS>wrote:

You're right, it's not in production yet. From what I understand, all
development is done on the staging branch and systematically pushed to
production from there. This specific issue was fixed in 7004e81
https://github.com/FreeCodeCamp/freecodecamp/commit/7004e81b4a29b196a659bcb5cdcef30253b71d6f
—
Reply to this email directly or view it on GitHub
https://github.com/FreeCodeCamp/freecodecamp/issues/1175#issuecomment-117810443
.

it hasn't been merged yet. expect merging of fixes to happen early next week possibly sooner

Should be fixed in prod now, right?

I don't know if "fixed" is the right word. The entire section seems to be deleted. Not sure where it went.
The original URL now gives an error "404: We couldn't find a challenge with the name waypoint-pair-program-on-bonfires Please double check the name."
And then it redirects, somewhat weirdly, to "http://www.freecodecamp.com/challenges/waypoint-learn-how-script-tags-and-document-ready-work"


release: v1.9.2

Summary
Plan to release this patch for tomorrow

Test Plan
CI should pass

I also added a few Grit changes to the CHANGELOG under a "GritQL" section. Seeing we don't have a better section, I think it makes sense, but let me know if you prefer something else :)


Pause Resume request

Would be great to pause video recording temporary and then resume recording later (e.g, after 5-10 seconds)
For example we won't feed anything (audio or video) to encoder about 5-10 seconds
But there could be a problem with timestamps (e.g. we were recording 10 seconds but it during playback in video player it says 15 seconds)
I suggest subtract this not recording count milliseconds (the time we wasn't feed any data)
    protected long getPTSUs() {
		long result = System.nanoTime() / 1000L;
		// presentationTimeUs should be monotonic
		// otherwise muxer fail to write
		if (result < prevOutputPTSUs)
			result = (prevOutputPTSUs - result) + result;
		return result - notRecordingTimeUS;
    }

	public void setNotRecordingTimeUS(long notRecordingTimeUS) {
		this.notRecordingTimeUS = notRecordingTimeUS; // e.g. subtract 5 seconds (the time we wasn't feed any video/audio to encoders)
	}

	private long notRecordingTimeUS;

@anonym24 你的想法是对的 ，这样可以做到暂停录制 ， 但是它不能删除上一段 ，不过这个功能有必要加到VideoRecordCore 里面 。 如果有时间的话 ，


Extend CRUD operations for single data objects and reference with consistency level

Codecov Report
Base: 96.35% // Head: 96.31% // Decreases project coverage by -0.05% :warning:

Coverage data is based on head (14f05be) compared to base (3d66fb7).
Patch coverage: 90.11% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #234      +/-   ##
==========================================
- Coverage   96.35%   96.31%   -0.05%
==========================================
  Files          60       60
  Lines        7081     7185     +104
==========================================
+ Hits         6823     6920      +97
- Misses        258      265       +7

Flag
Coverage Δ

unittests
96.31% <90.11%> (-0.05%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

weaviate/client.py
98.30% <ø> (ø)

weaviate/wcs/crud_wcs.py
98.50% <ø> (ø)

integration/test_authentication.py
89.09% <57.89%> (-6.52%)
:arrow_down:

weaviate/connect/connection.py
90.05% <70.37%> (-3.66%)
:arrow_down:

weaviate/data/crud_data.py
91.47% <94.73%> (+2.69%)
:arrow_up:

test/connection/test_connection.py
91.89% <100.00%> (ø)

test/data/references/test_crud_references.py
100.00% <100.00%> (ø)

test/data/test_crud_data.py
99.23% <100.00%> (+0.05%)
:arrow_up:

test/test_client.py
100.00% <100.00%> (ø)

test/test_version.py
100.00% <100.00%> (ø)

... and 8 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

We released 3.12, so there won't be any more releases before weavaite v1.18 is out. Feel free to merge this when all issues are resolved :)
PS we can make a beta release of the client with the new features as soon as we have released the RC


Include at least QuickJS, Deno in JavaScript tests

Rationale: node and bun are not the only representatives of JavaScript runtimes.

Added deno. Implemented quickjs but it was very slow on my machine. Will not add it since it's not fast enough to compete. Take a look at the code:
https://github.com/jinyus/related_post_gen/blob/quickjs/js/quickjs.js

Will not add it since it's not fast enough to compete.

That's not the point. The point is empirical data. QuickJS is faster than Node.js, Deno, and Bun as a Native Messaging host.
You are running a single test. When you beging writing different tests you will get different results.
native_messaging_javascript_runtime_tests
[{
  "nm_qjs": 0.2468439999997615
}, {
  "nm_bun": 0.2530850000003726
}, {
  "nm_deno": 0.34589599999971704
}, {
  "nm_nodejs": 0.36726300000011924
}]

I don't get your point.
Quickjs might be faster in that I/O heavy test but it's slower in this computational workload.
If you can improve the code to make it faster, be my guest. It take 1.6s on my machine.
I guess I can add it to the repo but I won't clutter the charts with another js runtime, especially if it's slower.

I guess I can add it to the repo but I won't clutter the charts with another js runtime, especially if it's slower.

I thought this repository was about empirical data?
It doesn't matter which programming language or runtime is slower or faster. Using your own logic should you remove Bun? And leave only Node.js? How will QuickJS maintainer and the downstream txiki.js know how "slow" that runtime is cf. other JavaScript runtimes when the data is intentionally excluded from the graph?
You are only running one kind of test and extrapolating a general speed from that single test. That is hardly comprehensive. I suspect you will add other tests, in which case you have prematurely excluded QuickJS, and therefore won;t know if QuickJS is "faster" in those instances.
It's your repository. Do as you will. I will tell you this, I didn;t have an expectation about which JavaScript runtime was faster when running Native Messaging tests. I let the data speak for itself.

I'll point this out as well. While QuickJS might be slower than Deno, Node, Bun, et al. in some cases, QuickJS is the choice for embedding in WebAssembly/WASI applications because after strip the qjs executable is less than 1 MB. In general, developers are not trying to compile node, deno or bun to WASM for embedding into WASI as a JavaScript runtime at 90-100+ MB. So, there is a trade-off.

@jinyus Nit: Technically Node.js, Deno, Bun are not Languages. They are JavaScript programming language runtimes. I didn't file another issue, though I cn.


Improve core.gardener.cloud.CloudProfile validation

The following cases could be improved:

Prevent .spec.kubernetes.versions with duplicating version. Example of invalid spec section:

spec:
  kubernetes:
    versions:
    - expirationDate: "2020-04-05T08:00:00Z"
      version: 1.10.13
    - expirationDate: "2007-04-05T08:00:00Z"
      version: 1.10.13

Prevent .spec.regions with duplicating name. Example of invalid spec section:

spec:
  regions:
  - name: us-east-2
    zones:
    - name: us-east-2a
  - name: us-east-2
    zones:
    - name: bar

/assign


How to restore the last opened directories?

I am using WT 1.16 on Windows 10.0.19044.1889.
Though I choose "Open windows from a previous session" in settings, after restarting the WT, the tabs (of Command Prompt) are restored all to the same starting directory but not the previous opened directories.
Do I miss something in the setting?

Did you configure your shell to tell the Terminal what the CWD is/?
Refer to: Opening a tab or pane in the same directory in Windows Terminal

Oh it works after I followed the link.
Thank you @zadjii-msft


Enqueue never finishes if dispatcher fails

https://github.com/apollographql/apollo-android/blob/a1cd963f632e2a5d333b9d34e90107b9b5e6733f/apollo-runtime/src/main/java/com/apollographql/apollo/internal/RealApolloCall.java#L125-L146
Given that the second half of the function is not caught, any exception at that time will mean that the callback never finishes.
As an example, I added an interceptor within the http client that will throw an exception. The result is correctly logged, but the respective unit test never finishes.
Perhaps try wrapping everything within a try catch, or introduce a way to provide a reasonable timeout?

Hm, not sure if this is a right approach to throw an exception in the interceptor rather than just return an error response. What I agree with is we should probably catch errors for user provided interceptors, as we can't rely on bad behaviour :)

I'm going to close this issue as it's pretty old now and everything has changed since then. @AllanWang please let us know if you want us to reopen.


[Suggestion] Could we make org package lazy loaded?

Currently, the org package will be loaded when we enter the copilot-mode, but I only see it needed when we use PanelSolution.
The org package usually takes a long time to load.

Thanks for your suggestion. It was fixed in 15a2fbe.


Add Policy Definitions to Enterprise-Scale Landing Zones

Add custom policy definitions to Enterprise-Scale Landing Zones and automatically enforce necessary policies:

[x] Synapse (https://github.com/Azure/Enterprise-Scale/pull/756)
[x] Databricks (https://github.com/Azure/Enterprise-Scale/pull/757)
[x] Search (https://github.com/Azure/Enterprise-Scale/pull/758)
[x] Data Factory (https://github.com/Azure/Enterprise-Scale/pull/759)
[x] Stream Analytics (https://github.com/Azure/Enterprise-Scale/pull/760)
[x] IoTHub (https://github.com/Azure/Enterprise-Scale/pull/761)
[x] EventHub (https://github.com/Azure/Enterprise-Scale/pull/762)
[x] Data Explorer (https://github.com/Azure/Enterprise-Scale/pull/763)
[x] Cosmos DB (https://github.com/Azure/Enterprise-Scale/pull/764)
[x] HDInsight (https://github.com/Azure/Enterprise-Scale/pull/765)
[ ] Storage
[ ] Batch
[ ] Diagnostics
[ ] KeyVault
[ ] MariaDB
[ ] MySQL
[ ] PostgreSQL
[ ] PrivateDNSZoneGroups
[ ] Private Endpoint
[ ] SQL
[ ] SQL InstancePools
[ ] SQL Managed Instance
[ ] Storage

We will use dataPolicies.json for all data policies. the policies.json is shrinking every month as they are all moving to built-ins.


Train new base network. Assert performance.

This is a huge issue, which is detailed further in
https://docs.google.com/document/d/1mL2n6KFi2CAcTpMI3OXBnHFIkkfVWBDEtDmGGLRU4hs/edit

@qiminchen @kriegman : what is the latest estimate for when this ticket is all the way done? In other words: how is the production training going?

@kriegman @beijbom Oscar, I just had a meeting with David today, here is the summary of all the works that I've been doing recently. Please refer to Sheet for all quantitative results.
1. Production training is done
Training setup: EfficientNet - b0, Adam Optimizer, w/o augmentation, manually adjust the learning rate

sheet v2/v3/v4, column D-F: Accuracy of each test source and overall accuracy in the last row, along with the accuracy improvement compared with v1 and backend.
sheet v4 sources details, column J-L: precision, recall and F1-score of each test source.
sheet v4 labels details, column M-O: precision, recall and F1-score of each label in the test sources, scroll down to the last row for the overall weighted performance.

I am gonna start production training on ResNet101 and see how far it can go.
2. Other ML classification models
Besides the Logistic regression classifier, I also tried Logistic regression with class_weight, Linear SVM, Random Forest, and Gaussian Naive Bayes. Full quantitative results in sheets.

Logistic regression with class_weight

SGDClassifier.partial_fit() does not support class_weight but sample_weight, and sample_weight is directly computed by class_weight therefore is applied during training. This method does not perform as good as directly applying geometric augmentation but faster.

Linear SVM

Changing the SGDClassifier(loss='log', average=True) to SGDClassifier(loss='hinge', average=True) gives us a Linear SVM model, performs slightly better than logistic regression classifier but takes a bit longer to train.

Random Forest

elif clf_method == 'rf':
    clf = RandomForestClassifier(warm_start=True, n_estimators=30, n_jobs=-1)
    with open(os.path.join('/mnt/sda/features/gamma/effnetb0_rf_4eps',
                           source, 'i000000.features.json'), 'r') as f:
        xrf = json.load(f)
    yrf = list(np.load(os.path.join('/mnt/sda/features/gamma/effnetb0_rf_4eps',
                                    source, 'i000000.features.anns.npy')))
    yrf = [classes_dict[i] for i in yrf]

This is a bit tricky since the RandomForestClassifier does not support partial_fit() so warm_start=True to reuse the solution of the previous call to fit and add more estimators to the ensemble. Also, it requires all labels to be presented in each chunk of data so I ended up appending each chunk of data with a list of features that include all the labels of that source.
n_estimators: tuned with 10, 20, 30, 50 and 100, the performance didn't change a lot when n_estimators > 30, but it takes way much longer to train when using n_estimators=100.
max_features: tuned with sqrt(n_features), log2(n_features), and n_features, ended up using sqrt(n_features) because using all features only improves the performance marginally but takes way much longer to train.

Gaussian Naive Bayes

Performance is trivial...
Let me know if you have any questions and Oscar, the production net weight is ready.

One comment to add
class_weight was done to balance the number of samples per class. This
"upweights" the rare classes/downweights the common clases so their loss
per class are equally weighted.
This should be contrasted with our data augmentation strategy where we
augmented the rare classes and subsampled the larger classes, but didn't go
as far as "full balancing across classes."
Not revealed through the statistics is the impact on classification of
rarer classes as seen in through anecdotal exploration of the confusion
matrices. Basically, augmentation increased accuracy of rarer classes, but
at the price of lower accuracy for more prevalent classes. So, it wasn't an
unqualified win, but rather a tradeoff. Balancing wasn't as a effective as
augmentation at benefiting the rarer classes.
On Mon, Jun 8, 2020 at 7:18 PM Qimin Chen<EMAIL_ADDRESS>wrote:

@kriegman
https://urldefense.com/v3/__https://github.com/kriegman__;!!Mih3wA!RQcXrTylMia9cB14XS3tPuJ3_mTYFnp9a1tac7MB7pFLQiokgZqp6V3UQV0CEISv$
@beijbom
https://urldefense.com/v3/__https://github.com/beijbom__;!!Mih3wA!RQcXrTylMia9cB14XS3tPuJ3_mTYFnp9a1tac7MB7pFLQiokgZqp6V3UQSpi0Zsn$
Oscar, I just had a meeting with David today, here is the summary of all
the works that I've been doing recently. Please refer to Sheet
https://urldefense.com/v3/__https://docs.google.com/spreadsheets/d/15NxI4AYrADqBB6KBQ-sL_ruEEf8mT10j_bq4w0tVhqI/edit?usp=sharing__;!!Mih3wA!RQcXrTylMia9cB14XS3tPuJ3_mTYFnp9a1tac7MB7pFLQiokgZqp6V3UQYZbCW3p$
for all quantitative results.
1. Production training is done
Training setup: EfficientNet - b0, Adam Optimizer, w/o augmentation,
manually adjust the learning rate

sheet v2/v3/v4, column D-F: Accuracy of each test source and overall
accuracy in the last row, along with the accuracy improvement compared with
v1 and backend.
sheet v4 sources details, column J-L: precision, recall and F1-score
of each test source.
sheet v4 labels details, column M-O: precision, recall and F1-score
of each label in the test sources, scroll down to the last row for the
overall weighted performance.

I am gonna start production training on ResNet101 and see how far it can
go.
2. Other ML classification models
Besides the Logistic regression classifier, I also tried Logistic
regression with class_weight, Linear SVM, Random Forest, and Gaussian
Naive Bayes. Full quantitative results in sheets.

Logistic regression with class_weight

SGDClassifier.partial_fit() does not support class_weight
https://urldefense.com/v3/__https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html__;!!Mih3wA!RQcXrTylMia9cB14XS3tPuJ3_mTYFnp9a1tac7MB7pFLQiokgZqp6V3UQeKSw9tC$
but sample_weight
https://urldefense.com/v3/__https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_sample_weight.html__;!!Mih3wA!RQcXrTylMia9cB14XS3tPuJ3_mTYFnp9a1tac7MB7pFLQiokgZqp6V3UQVZSCwRy$,
and sample_weight is directly computed by class_weight therefore is
applied during training. This method does not perform as good as directly
applying geometric augmentation but faster.

Linear SVM

Changing the SGDClassifier(loss='log', average=True) to SGDClassifier(loss='hinge',
average=True) gives us a Linear SVM model, performs slightly better than
logistic regression classifier but takes a bit longer to train.

Random Forest
https://urldefense.com/v3/__https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html__;!!Mih3wA!RQcXrTylMia9cB14XS3tPuJ3_mTYFnp9a1tac7MB7pFLQiokgZqp6V3UQYlECHy5$

elif clf_method == 'rf':
clf = RandomForestClassifier(warm_start=True, n_estimators=30, n_jobs=-1)
with open(os.path.join('/mnt/sda/features/gamma/effnetb0_rf_4eps',
source, 'i000000.features.json'), 'r') as f:
xrf = json.load(f)
yrf = list(np.load(os.path.join('/mnt/sda/features/gamma/effnetb0_rf_4eps',
source, 'i000000.features.anns.npy')))
yrf = [classes_dict[i] for i in yrf]
This is a bit tricky since the RandomForestClassifier does not support
partial_fit() so warm_start=True to reuse the solution of the previous
call to fit and add more estimators to the ensemble. Also, it requires all
labels to be presented in each chunk of data so I ended up appending each
chunk of data with a list of features that include all the labels of that
source.
n_estimators: tuned with 10, 20, 30, 50 and 100, the performance didn't
change a lot when n_estimators > 30, but it takes way much longer to
train when using n_estimators=100.
max_features: tuned with sqrt(n_features), log2(n_features), and
n_features, ended up using sqrt(n_features) because using all features
only improves the performance marginally but takes way much longer to train.

Gaussian Naive Bayes
https://urldefense.com/v3/__https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html__;!!Mih3wA!RQcXrTylMia9cB14XS3tPuJ3_mTYFnp9a1tac7MB7pFLQiokgZqp6V3UQR-LcWII$

Performance is trivial...
Let me know if you have any questions and Oscar, the production net weight
is ready.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://urldefense.com/v3/__https://github.com/beijbom/coralnet/issues/234*issuecomment-640987607__;Iw!!Mih3wA!RQcXrTylMia9cB14XS3tPuJ3_mTYFnp9a1tac7MB7pFLQiokgZqp6V3UQcHu6Df0$,
or unsubscribe
https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABKA5AJKFADWZYPDMYM6SA3RVWLX3ANCNFSM4IMD3KXQ__;!!Mih3wA!RQcXrTylMia9cB14XS3tPuJ3_mTYFnp9a1tac7MB7pFLQiokgZqp6V3UQXIbbO6I$
.

Thanks @qiminchen . 0.37% boost compared to v1 isn't amazing, but you have done a great work building up to this and I don't think there are any obvious stones unturned. Looking forward to seeing what ResNet101 will do.
Can you share the training-loss curve? How many epochs did you end up training for?
Also: did you end up not using any data-augmentation at all? I thought we agreed on using V4: 2-1275 which has class-balance + DA?
I'd suggest you go ahead and replace the EfficientNet_b0 version we have in spacer with this one. Perhaps we can simply update the weight and voila -- we have a new version! (This is OK at this point since this hasn't been deployed yet). You could also rename the extractor it but I think efficientnet_b0_ver1 is as good of a name as any.

... In fact, after you update the weights, can you add a check-sum to the load_model method to verify that we are actually downloading and using the correct weights? If we don't have that, we could accidentally over-write the weights-file on s3 and suddenly have a different feature extractor without knowing it. Ugh.

We were hovering between V4: 1-1275 and V4: 2-1275 and I ended up using V4: 1-1275 setting because I didn't see many improvements but it takes longer for training with geometric augmentation.
As for the training, I ended up training the net with 22 epochs with a batch size of 156, and I logged the batch loss every 10,000 batches. I should've used tensorboard instead... The current best model was from epoch 21st.

Ok, thanks for sharing. I think those look good. I would probably have
deferred the first drop a bit later, but I don't think it makes a big
difference.
On Tue, Jun 9, 2020 at 1:36 AM Qimin Chen<EMAIL_ADDRESS>wrote:

We were hovering between V4: 1-1275 and V4: 2-1275 and I ended up using V4:
1-1275 setting because I didn't see many improvements but it takes longer
for training with geometric augmentation.
As for the training, I ended up training the net with 22 epochs with a
batch size of 156, and I logged the batch loss every 10,000 batches. I
should've used tensorboard instead... The current best model was from epoch
21st.
[image: production]
https://user-images.githubusercontent.com/45860234/84125283-30d99600-a9f1-11ea-8bc0-045b5cd874cc.png
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/234#issuecomment-641126514,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAITTFZMNCRRPD3CXK3DSHLRVXYCFANCNFSM4IMD3KXQ
.

@kriegman @beijbom hey guys, here are some updates about the performance of Logistic Regression and Multi-Layer Perceptron, Google Sheet for full quantitative results.
TL;DR
Multi-Layer Perceptron optimal parameters setting:

hidden_layer_size = (200,100) for # of total training patch > 10,000 and hidden_layer_size = (200) for # of total training patch <= 10,000
learning_rate=1e-4,
optimizer=Adam

Multi-Layer Perceptron performs better than Logistic Regression in both recall and F1-score with a considerable margin.
Multi-Layer Perceptron is faster than Logistic Regression during training.
Multi-Layer Perceptron performs better than Logistic Regression regardless of the relative source size: # of total training patch / # of class.
ResNet101 training is at epoch 6 now.

== > We may consider replacing Logistic Regression with Multi-Layer Perceptron.
Details

Multi-Layer Perceptron hyperparameters tuning

I've tried different setups of hidden_layer_size and learning_rate in 5 test sources with different source size (# of total training patches). s16 and s1420 have the largest and smallest source size respectively. By looking through different setups over 5 sources, it's pretty obvious:

1e-3 is too large and 1e-5 is too small
(200, 100) or (200) is preferable, no need to use more than 2 hidden layers

source
s16
s294
s367
s672
s1420

size
358,272
34,906
13,435
8,028
5,100

s16 - with a size of 358,272

Note: plots grouped by hidden_layer_size and learning_rate respectively for detailed comparison.

s294 - with a size of 34,906

s367 - with a size of 13,435

s672 - with a size of 8,028

s1420 - with a size of 5,100

Training time

Multi-Layer Perceptron is faster than Logistic Regression in most of the test sources. No tradeoff happened in training time.

Accuracy of 26 test sources

x-axis: the relative source size (# of total training patch / # of class), left to right -> small to large
y-axis: the accuracy

So, if I'm reading the spreadsheet correctly, the most important numbers
are:
CoralNet Beta (backend) accuracy: 71.36
New Production (Efficient Net B0)  with LR classifier accuracy: 76.57
New Production (Efficient Net B9)  with MLP classifier accuracy: 76.82
CoralNet Beta (backend) F1: ?? [Not in table]
New Production (Efficient Net B0)  with LR classifier F1: 75.02
New Production (Efficient Net B9)  with MLP classifier F1: 76.37
So, the improvement for accuracy is modest (0.25%), but is more significant
for F1 (1.35%).
But I don't understand your statement: "Multi-Layer Perceptron performs
better than Logistic Regression regardless of the relative source size: #
of total training patch / # of class."
The last bar plot shows LR being more accurate for some sources and MLP
more accurate for other sources. Am I missing something?
On Tue, Jun 16, 2020 at 6:44 PM Qimin Chen<EMAIL_ADDRESS>wrote:

@kriegman
https://urldefense.com/v3/__https://github.com/kriegman__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7dhEh8rhn$
@beijbom
https://urldefense.com/v3/__https://github.com/beijbom__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7dmyPP5K9$
hey guys, here are some updates about the performance of Logistic
Regression and Multi-Layer Perceptron, Google Sheet
https://urldefense.com/v3/__https://docs.google.com/spreadsheets/d/15NxI4AYrADqBB6KBQ-sL_ruEEf8mT10j_bq4w0tVhqI/edit?usp=sharing__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7dkIuhojV$
for full quantitative results.
TL;DR
Multi-Layer Perceptron optimal parameters setting:

hidden_layer_size = (200,100) for # of total training patch > 10,000
and hidden_layer_size = (200) for # of total training patch <= 10,000
learning_rate=1e-4,
optimizer=Adam

Multi-Layer Perceptron performs better than Logistic Regression in
both recall and F1-score with a considerable margin.
Multi-Layer Perceptron is faster than Logistic Regression during
training.
Multi-Layer Perceptron performs better than Logistic Regression
regardless of the relative source size: # of total training patch / #
of class.
ResNet101 training is at epoch 6 now.

== > We may consider replacing Logistic Regression with Multi-Layer
Perceptron.
Details

Multi-Layer Perceptron hyperparameters tuning

I've tried different setups of hidden_layer_size and learning_rate in 5
test sources with different source size (# of total training patches). s16
and s1420 have the largest and smallest source size respectively. By
looking through different setups over 5 sources, it's pretty obvious:

1e-3 is too large and 1e-5 is too small
(200, 100) or (200) is preferable, no need to use more than 2 hidden
layers

source s16 s294 s367 s672 s1420
size 358,272 34,906 13,435 8,028 5,100

s16 - with a size of 358,272

Note: plots grouped by hidden_layer_size and learning_rate respectively
for detailed comparison.
[image: s16_hls]
https://urldefense.com/v3/__https://user-images.githubusercontent.com/45860234/84842200-6128c200-aff9-11ea-8123-8ada25037fd5.png__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7dvhBIBBC$
[image: s16_lr]
https://urldefense.com/v3/__https://user-images.githubusercontent.com/45860234/84842232-7271ce80-aff9-11ea-9d8b-58d528281e6b.png__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7dmZ1ifCc$

s294 - with a size of 34,906

[image: s294_hls]
https://urldefense.com/v3/__https://user-images.githubusercontent.com/45860234/84842351-b95fc400-aff9-11ea-8db0-d184afffa119.png__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7dnyDlEnY$
[image: s294_lr]
https://urldefense.com/v3/__https://user-images.githubusercontent.com/45860234/84842357-bbc21e00-aff9-11ea-8b05-644a9ad399b3.png__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7dm2WwgTr$

s367 - with a size of 13,435

[image: s367_hls]
https://urldefense.com/v3/__https://user-images.githubusercontent.com/45860234/84843065-c5e51c00-affb-11ea-91df-60821c708e7b.png__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7dqVSxrs4$
[image: s367_lr]
https://urldefense.com/v3/__https://user-images.githubusercontent.com/45860234/84843066-c7164900-affb-11ea-84c9-9336175db46c.png__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7dtXLk6pr$

s672 - with a size of 8,028

[image: s672_hls]
https://urldefense.com/v3/__https://user-images.githubusercontent.com/45860234/84843125-e9a86200-affb-11ea-8b0c-6ef5965ec9c8.png__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7dhcHFCZk$
[image: s672_lr]
https://urldefense.com/v3/__https://user-images.githubusercontent.com/45860234/84843130-ed3be900-affb-11ea-965f-bfbea8a16200.png__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7dsvkaCu3$

s1420 - with a size of 5,100

[image: s1420_hls]
https://urldefense.com/v3/__https://user-images.githubusercontent.com/45860234/84843421-b0bcbd00-affc-11ea-8614-ebe1fa4f8d1d.png__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7drqSafEX$
[image: s1420_lr]
https://urldefense.com/v3/__https://user-images.githubusercontent.com/45860234/84843425-b1edea00-affc-11ea-975d-37e7d95fbe15.png__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7dg7BL4hC$

Training time

Multi-Layer Perceptron is faster than Logistic Regression in most of the
test sources. No tradeoff happened in training time.
[image: lr_mlp_runtime]
https://urldefense.com/v3/__https://user-images.githubusercontent.com/45860234/84844870-37bf6480-b000-11ea-820b-0ed8877d7692.png__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7dnBmYg96$

Accuracy of 26 test sources

x-axis: the relative source size (# of total training patch / # of
class), left to right -> small to large
y-axis: the accuracy

[image: lr_mlp_acc]
https://urldefense.com/v3/__https://user-images.githubusercontent.com/45860234/84844991-7fde8700-b000-11ea-9b26-0f30348e81ae.png__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7duuRFX7i$
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://urldefense.com/v3/__https://github.com/beijbom/coralnet/issues/234*issuecomment-645097544__;Iw!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7dnB6wMJH$,
or unsubscribe
https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABKA5APW4L5KDLMP5VH5YL3RXANWZANCNFSM4IMD3KXQ__;!!Mih3wA!SQVocWmtaZ-ZPBglASiSeH74pYOIQHMOcQXRhcTdUhtTANHUFMoMwSt7duO91PG-$
.

But I don't understand your statement: "Multi-Layer Perceptron performs
better than Logistic Regression regardless of the relative source size: #
of total training patch / # of class."
The last bar plot shows LR being more accurate for some sources and MLP
more accurate for other sources. Am I missing something?

You are right, Multi-Layer Perceptron performs better than Logistic Regression in most of the test sources, for s609, s843, s1579, 1559, s1070, Multi-Layer Perceptron is slightly worse.

@kriegman

But I don't understand your statement: "Multi-Layer Perceptron performs
better than Logistic Regression regardless of the relative source size: #
of total training patch / # of class."
The last bar plot shows LR being more accurate for some sources and MLP
more accurate for other sources. Am I missing something?

You are right, Multi-Layer Perceptron performs better than Logistic Regression in most of the test sources, for s609, s843, s1579, 1559, s1070, Multi-Layer Perceptron is slightly worse.

@qiminchen : what is the status on this ticket? Any outstanding work-items? Did resnet 101 finish? Did you want to add MLP to the spacer codebase? Did you end up writing up a report on your work? CC @kriegman .

yes, ResNet101 pretrained on V1 and trained for another 8 epochs works better than EfficientNetb0 as you could see in Sheet EfficiNetb0 vs. ResNet101. For a fair comparison, I also train EfficientNetbo pretrained on V1. The result will be available soon. And I will write up a report on EfficientNetb0 and ResNet101 on Logistic Regression and Multi-layer Perceptron very soon.

Okay, thanks. What is your timeline for the report? We will need to make a decision which one to use on the production server.

By Tues, EfficientNetbo pretrained on V1 would be available by then

The short answer from last monday was that ResNet50 had an absolute error
that was about 2% lower than EfficientNetB0.
But ResNet 50 is about half the speed and comparable to VGG16.
A month ago or so, Qimin was unsuccessful at training ResNet50 from
scratch, and so I asked him to initialize from 1-cycle optimizer version,
and that gave the boost.
So, he's gone back to EfficientNetB0 with the same strategy.
So, we'll need to make this decision of accuracy vs. speed.  [Hint; I'm
favoring accuracy].
Let's get a full report on this, make a decision, and get ready to deploy
it, presumably with MLP. But again, no commitment until we see the report.
D
On Sun, Aug 16, 2020 at 8:45 PM Oscar Beijbom<EMAIL_ADDRESS>wrote:

Okay, thanks. What is your timeline for the report? We will need to make a
decision which one to use on the production server.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://urldefense.com/v3/__https://github.com/beijbom/coralnet/issues/234*issuecomment-674640166__;Iw!!Mih3wA!WL3l1KOpkzJdYD-RjbdpwoGxjSns35WcEerCzazjQz3W_1iuIRxQHkyPKy9C6czS$,
or unsubscribe
https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABKA5AKUQ4RLE25VD73DQMTSBCRWTANCNFSM4IMD3KXQ__;!!Mih3wA!WL3l1KOpkzJdYD-RjbdpwoGxjSns35WcEerCzazjQz3W_1iuIRxQHkyPKzGJNVfZ$
.

@qiminchen : I'll close this issue now since the work is done.


Fix opening documents on Android

Summary
The library that we use to open documents implements MimeTypeMap.getFileExtensionFromUrl(url) on Android to detect the file extension which in some cases it fails.
As to why MimeTypeMap.getFileExtensionFromUrl(url) fails? It's expecting a properly formatted URL String, which if the filename has spaces or any other weird character it will just fail. so we will first encode it using URLEncoder.
Also the underlying library is concatenating the fileName and fileExtension so we are now removing the extension from the filename
Ticket Link
https://mattermost.atlassian.net/browse/MM-13355
Device Information
This PR was tested on: Android & iOS

Also the underlying library is concatenating the fileName and fileExtension so we are now removing the extension from the filename

What a weird way to do that. It would be nice if the library was written in a way that let us just tell it the type of the file if we already know it...

I think we need to update the library cause looks like they fixed it in master, but they haven't published an update to npm :S last commit was Oct 26 last npm update was 8 months ago

I think we need to update the library cause looks like they fixed it in master, but they haven't published an update to npm :S last commit was Oct 26 last npm update was 8 months ago.
https://github.com/philipphecht/react-native-doc-viewer/issues/124


Use dot based map

Technical task
We can settle on one dot based map and each rhythm/rhythm code can be associated with one specific dot (or one specific dot coordinate as is convenient for the implementer) on the map

As a .............

I Want .............

So That .............

Acceptance Criteria

Given .............

When .............
Then .............

Given .............

When .............
Then .............

Implementation Notes

Checklist:

[ ] I have defined the user story
[ ] I have defined some acceptance criteria
[ ] I have included any implementation notes
[ ] I have included all relevant links

This one https://upload.wikimedia.org/wikipedia/commons/a/a0/World_map_with_points.svg or the one from the .ai file attached in slack could be used

Does this replace #5 in that this should be our only map?

That's correct. As mentioned here on Slack https://codeyourfuture.slack.com/archives/C05SUQ3K5DY/p1696065119931789

@ManSangSin - As part of this ticket, I would ask that you make sure the icons appear in the right place.
In sprint planning we discussed PX CSS positioning or selecting SVG path elements as possible solutions for locating the dots. I leave it to you to decide

Need to have this thought through and communicated by Tuesday.


[RESTEASY-1974] Fix subset of checkstyle errors II - indentation

Set indentation to 3 spaces and remove tabs
Jira: https://issues.jboss.org/browse/RESTEASY-1974
Master PR: https://github.com/resteasy/Resteasy/pull/1725

Rebase done.

@tterem thanks


Drop rustc-serialize dependency

It has been deprecated: announcement.

It seems like data-encodings looks like a good candidate to replace it.
https://github.com/ia0/data-encoding


Update icons

Replace the current icons on the homepage with these from https://heroicons.com/

Remove and replace with the RSS icon...

Change the "paper airplane" to the mail icon...

Social media icons aren't centred properly...

the icons are in public/images/icon just replace them (with  the same names ) and that's it


Update sbt-coursier to 2.0.8

Updates io.get-coursier:sbt-coursier from 1.1.0-M9 to 2.0.8.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "io.get-coursier", artifactId = "sbt-coursier" } ]

labels: sbt-plugin-update, semver-major

Superseded by #146.


ISSUE-110 | Sync arrow position with Digital Crown orientation

Issue: https://github.com/Artificial-Pancreas/iAPS/issues/110

Changes

Used WKInterfaceDevice.current().crownOrientation to read the position of the Digital Crown
Changed the help arrow from pointing to the right (or left) to pointing down to avoid confusion when placed next to the Cancel button
Minor View cleanup

Notes

These are my first lines of code in Swift
Xcode Simulator has no toggle to simulate Digital Crown orientation

crownOrientation  == .left
crownOrientation  == .right

Thanks! Did you test this update also on different Watch sizes? The format, like for instance your spaces value, sometimes need special tweaking depending on Watch size.

Thanks! Did you test this update also on different Watch sizes? The format, like for instance your spaces value, sometimes need special tweaking depending on Watch size.

I tested it on my Ultra. I'll test in other watch simulators today.
What do you think about the shrunk "Cancel" button when the crown is on the left side? 🤔 I couldn't come up with a more elegant variant.

Thanks! Did you test this update also on different Watch sizes? The format, like for instance your spaces value, sometimes need special tweaking depending on Watch size.

What do you think about the shrunk "Cancel" button when the crown is on the left side? 🤔 I couldn't come up with a more elegant variant.

Removing the arrow or changing positions of the cancel button are two other alternatives...

I will merge it now. If we find issues, please be prepared fixing these later with new PRs!


Fix release script for gh-pages

My change to add pre-commit linting broke the gh-pages release script because it tries to run on the compiled files and blocks completion of the release. This flag bypasses that check on a release commit for gh-pages - something I had to work around in the 0.75.0 release - with this fix that will no longer be an issue.

Codecov Report

Merging #266 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #266   +/-   ##
=======================================
  Coverage   70.24%   70.24%
=======================================
  Files          46       46
  Lines         820      820
  Branches      189      189
=======================================
  Hits          576      576
  Misses        172      172
  Partials       72       72

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3d30528...76c46bc. Read the comment docs.

Deploy preview for gestalt ready!
Built with commit 76c46bc963002bb84258e47af5982656100c8d02
https://deploy-preview-266--gestalt.netlify.com


Stopwatch

This is my project where I have made a very simple Stopwatch using HTML, CSS , JS

I know how to make an interactive stop watch and have worked with the same previously. Please assign me this issue in GSSoC'23.

I can easily make an interactive stopwatch using HTML, CSS, JS.
Could you assign it to me?

Sir/Ma'am please assign me the issue.


[FEATURE] Fill out sql namespace for spec repo

The sql namespace on the spec repo is missing a few endpoints. We need these endpoints filled out in the spec so that we generate clients that interact with this plugin properly:

[ ] Make sure that the existing sql endpoints in the spec, if any, are correct
[ ] Fill out missing sql endpoints.

The Dev's Guide is a great place to start.

Catch All Triage - 1 2 3 4 5 6


⚠ Bofrost.be has degraded performance

In b5f1db8, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 806 ms

Resolved: Bofrost.be performance has improved in 099580f.


error adding binary sensor

Version of the custom_component

latest. also latest ha
Describe the bug
cant add binary sensor
Debug log
Error while setting up openhasp platform for binary_sensor
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 249, in _async_setup_platform
await asyncio.shield(task)
File "/config/custom_components/openhasp/binary_sensor.py", line 34, in async_setup_entry
[
File "/config/custom_components/openhasp/binary_sensor.py", line 35, in
HASPBinarySensor(
File "/config/custom_components/openhasp/binary_sensor.py", line 58, in init
self._attr_name = f"{name} binary_sensor {self._gpio}"
AttributeError: 'HASPBinarySensor' object has no attribute '_gpio'

I had this same issue. You can add the following line to the init function for the HASPBinarySensor class just prior to line 58 to resolve this error.
self._gpio = gpio
(This may not completely fix it, I'm having a separate error at the same time.)

Has this issue with the binary sensor been resolved ?
Didn't really notice that my sensors in HA was no longer available as I use MQTT in HA to trigger on GPIO  events from the plates

yes in ea2a04e049ddb59b5b0981d52560c8a902a185a6

Updated CC to ea2a04e -> All OK

yes in ea2a04e

excuse me but what should i do to get rid of this error? i have latest 0.6.6 installed....

excuse me but what should i do to get rid of this error? i have latest 0.6.6 installed....

openHASP CC 0.6.6 still has this binary sensor issue.
Switch to main branch, redownload and restart HA. Now your CC should be commit 45a4e44 and have working binary sensors in HA


Resource blocking doesn't work in Playwright

A reader of the Academy has reported that the lesson on resource blocking doesn't work in Playwright:

Looks like newCDPSession() now wants an argument - page. But it looked like it didn't work anyway?

This seems to be already fixed as const client = await page.context().newCDPSession(page);.


Update/Improve hooks

Run isort and black in the hooks and other small improvements in the hooks.
This should work in Windows, but I haven't tried it. It would be nice if someone could check.

@mr-tz

thoughts about changing from post- to pre-commit to avoid erroneous commits in the first place?

I think it is a good idea. It may help that we fix the offenses in the commit they are introduced and avoid the need for a fix offenses commit. This way we could also avoid code duplication by calling a script as the behavior of both hooks would be the same.
The argument to have it in a post-commit is that it doesn't block the commit, but in the meanwhile I have discovered the --no-verify option. This is even better than what we have now, as currently there is no way to skip running the post-commi hook (you can abort it but then you have to manually restore changes or edit manually the hook file).
@mr-tz @williballenthin should I add another commit to change from post- to pre-commit?

if it's a quick fix sure, don't burn too many cycles here 😉

@mr-tz

if it's a quick fix sure, don't burn too many cycles here 😉

It is: https://github.com/fireeye/capa/pull/213/commits/70e296759cef02793a5ce39230273f074e372aa0 😉 And we get a way to run the test locally without hooks as well (scripts/ci.sh) 🥳


riscv64 build error with 3.2.0

On riscv64 I'm getting:
../providers/implementations/ciphers/cipher_aes_ccm_hw_rv64i.inc: In function 'ccm_rv64i_zknd_zkne_initkey':
../providers/implementations/ciphers/cipher_aes_ccm_hw_rv64i.inc:20:27: warning: implicit declaration of function 'rv64i_zkne_set_encrypt_key' [-Wimplicit-function-declaration]
   20 |     AES_HW_CCM_SET_KEY_FN(rv64i_zkne_set_encrypt_key, rv64i_zkne_encrypt,
      |                           ^~~~~~~~~~~~~~~~~~~~~~~~~~
../providers/implementations/ciphers/cipher_aes_ccm_hw.c:21:5: note: in definition of macro 'AES_HW_CCM_SET_KEY_FN'
   21 |     fn_set_enc_key(key, keylen * 8, &actx->ccm.ks.ks);                         \
      |     ^~~~~~~~~~~~~~
../providers/implementations/ciphers/cipher_aes_ccm_hw_rv64i.inc:20:55: error: 'rv64i_zkne_encrypt' undeclared (first use in this function)
[...]

Full log at: https://buildd.debian.org/status/fetch.php?pkg=openssl&arch=riscv64&ver=3.2.0-1&stamp=1701034270&raw=0

What is the debian-riscv64 config target? How does it define asm_arch?
I do not see OPENSSL_CPUID_OBJ being defined in your build log which suggests the asm support is not being detected properly.
There is a discrepancy between the riscv64 detection here (where the declaration of the rv64i_zkne_set_encrypt_key function is) :
https://github.com/openssl/openssl/blob/6b7a11d8aa7abe50e6ebdd09a238e0a0df8cd228/include/crypto/aes_platform.h#L438-L444
and here:
https://github.com/openssl/openssl/blob/6b7a11d8aa7abe50e6ebdd09a238e0a0df8cd228/providers/implementations/ciphers/cipher_aes_ccm_hw.c#L64-L65
Where we determine what assembler support we have, i.e. the latter does not check for OPENSSL_CPUID_OBJ, which it probably should.

Possible fix in #22881.
@kroeckx could you try it out?
Although this will end up with the assembler implementation not being used on your platform because of the lack of OPENSSL_CPUID_OBJ which is perhaps not what you intended??

I added the "inherit from linxu64-riscv64" and it built fine. From my point of view this can be closed with no change.

I get the same build error but originating from cipher_aes_gcm_hw.c, I guess a fix in the same spirit as https://github.com/openssl/openssl/commit/ff279597692f9f19dca5b147944d3d96f2e109f8 is necessary here as well: https://github.com/openssl/openssl/blob/master/providers/implementations/ciphers/cipher_aes_gcm_hw.c#L145
?

I get the same build error but originating from cipher_aes_gcm_hw.c, I guess a fix in the same spirit as ff27959 is necessary here as well: https://github.com/openssl/openssl/blob/master/providers/implementations/ciphers/cipher_aes_gcm_hw.c#L145 ?

@marv I second this.
I'm no expert on this topic but this failure is preventing a new Rustup release from happening (https://github.com/rust-lang/rustup/pull/3653#issuecomment-1936981569).
Coming from https://github.com/alexcrichton/openssl-src-rs/issues/222#issuecomment-1937412946 I see that I used to have two different errors, and after upgrading to v3.2.1 the one regarding cipher_aes_ccm* is now gone, but unfortunately the one related to cipher_aes_gcm* is still there.

There are additional checks which need to be updated, I've submitted https://github.com/openssl/openssl/pull/23752 to adress the remaining compilation issues


Fixed: broken API to get theme details

#19
This issue is fixed in this PR.

Hi Thank you its working fine now

This is a breaking change so we need to update the major version instead of just patch.
Also do not update the version manually in the package json. I'll do it myself through the npm version command.
You do not need to take any action I've fixed it up.
closes #16

@NeekSandhu 👍🏼 better

<EMAIL_ADDRESS>is now live


Building TensorFlow Lite library with tensorflow ops failed

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): source
TensorFlow version: master
Python version: 3.5
Installed using virtualenv? pip? conda?: No
Bazel version (if compiling from source): 3.1.0
GCC/Compiler version (if compiling from source): 5.4.0
CUDA/cuDNN version: No
GPU model and memory: No

Describe the problem
I want to build a TensorFlow Lite C++ shared library that supports full tensorflow ops. I'm using cross-compile, buiding on PC and the target platform is arm64. But the build process failed.
Provide the exact sequence of commands / steps that you executed before running into the problem
Modify tensorflow/lite/BUILD to include "//tensorflow/lite/delegates/flex:delegate" in the last part: deps of tflite_cc_shared_object.
Then run:
bazel build --config=elinux_aarch64 --config=monolithic -c opt //tensorflow/lite:libtensorflowlite.so
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
ERROR: /home/xxx/tensorflow/tensorflow/core/lib/strings/BUILD:62:1: C++ compilation of rule '//tensorflow/core/lib/strings:proto_text_util' failed (Exit 1)
In file included from external/com_google_protobuf/src/google/protobuf/stubs/port.h:49,
                 from external/com_google_protobuf/src/google/protobuf/stubs/macros.h:34,
                 from external/com_google_protobuf/src/google/protobuf/stubs/logging.h:34,
                 from external/com_google_protobuf/src/google/protobuf/arena_impl.h:40,
                 from external/com_google_protobuf/src/google/protobuf/arena.h:53,
                 from ./tensorflow/core/platform/protobuf.h:33,
                 from ./tensorflow/core/lib/strings/proto_text_util.h:22,
                 from tensorflow/core/lib/strings/proto_text_util.cc:16:
external/com_google_protobuf/src/google/protobuf/port_def.inc:74:2: error: #error PROTOBUF_DEPRECATED was previously defined
 #error PROTOBUF_DEPRECATED was previously defined
  ^~~~~

The full log is too long and I omit the middle part... And the last part is:
Target //tensorflow/lite:libtensorflowlite.so failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 24.871s, Critical Path: 24.02s
INFO: 81 processes: 81 local.
FAILED: Build did NOT complete successfully

Could you help me analyze this? Many thanks.

Hi, would anyone please have a look :)


gson annotations(SerializedName(""), Expose) not retained after the class generation

annotations not retained in the generated class. If this is by design then how to achieve the gson serialization

This seems to be the same question as issue #189 -- you can refer to the discussion there for ideas.
For the sake of clarity: model specs aren't "real" Java objects, they're just shorthand for generating a table schema. The fields in the spec just generate Property declarations; the generated model class doesn't actually have any fields of its own, so there wouldn't be anywhere to copy the annotations to.
Hope that helps! Let us know if you have other questions.


Crash after migrate to AndroidX

java.lang.RuntimeException: Unable to get provider com.iceteck.silicompressorr.provider.GenericFileProvider: java.lang.ClassNotFoundException: Didn't find class "com.iceteck.silicompressorr.provider.GenericFileProvider"
getting crash after migrating to AndroidX

me too

@gauravkamble293 @zzsxxy Sory about that and thanks for your feedback. Please try v2.2.4

The issue still exists, I tried with 2.2.4

@gauravkamble293 @zzsxxy Sory about that and thanks for your feedback. Please try v2.2.4

Not working

Yep, still seeing this error on 2.2.4

Yep, still seeing this error on 2.2.4


[Discussion] Adding two extra custom fields

Hi @coughski,
Thanks for updating the package in the webstore to release the "show remaining minutes" feature.
I would like to discuss the following point with you:
What do you think of adding two extra custom fields?
At least for me, I would like to have two more. :)
Warmly,
Robert

@orschiro , thanks for contributing! If you submit a pull request for the 2 additional fields, I can add it.

@coughski
I will see what I can do!
Warmly,
Robert

It's also a interesting feature for me.

Dear @santagu,
Apologies but I won't provide any pull request as I am no longer using the extension.
Warmly,
~Robert


SNOW-1489361: Add support for GroupBy.all/any

Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

Fixes SNOW-1489361

Fill out the following pre-review checklist:

[ ] I am adding a new automated test(s) to verify correctness of my new code

[ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing

[ ] I am adding new logging messages
[ ] I am adding a new telemetry message
[ ] I am adding new credentials
[ ] I am adding a new dependency
[ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes.

Please describe how your code solves the related issue.

This PR adds support for GroupBy.all and GroupBy.any.

LGTM, might be worth doing a run on Jenkins (or manual) just to check that you have removed any/all from the unsupported APIs list if it is there

Good catch, I forgot to do that.


feat: add vmsProperty for referencing all VMs

:tada: This PR is included in version 1.3.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Not finding required dependencies (for Databricks ODBC connection)

Describe the bug
DBT spark not detecting dependencies correctly.
Runtime Error
  Credentials in profile "dbt_spark", target "development" invalid: Runtime Error
    odbc connection method requires additional dependencies.
    Install the additional required dependencies with `pip install dbt-spark[ODBC]`

Steps To Reproduce
I installed pip install "dbt-spark[ODBC]" successfully, and pip install pyodbc.
After that, i configured my profiles.yml as the following:
  outputs:
    dev:
      type: spark
      method: odbc
      driver: "/Library/simba/spark/lib/libsparkodbc_sbu.dylib"
      schema: [schema]
      host: [host]
      token: [token]
      cluster: [cluster]

  target: dev

After running dbt debug, I'm getting the following message:
Running with dbt=0.19.1
dbt version: 0.19.1
python version: 3.8.8
python path: /Users/user.name/.pyenv/versions/3.8.8/envs/dbt/bin/python3.8
os info: macOS-10.15.6-x86_64-i386-64bit
Using profiles.yml file at /Users/user.name/.dbt/profiles.yml
Using dbt_project.yml file at /Users/user.name/gousto/dbt_spark/dbt_spark/dbt_project.yml

Configuration:
  profiles.yml file [ERROR invalid]
  dbt_project.yml file [OK found and valid]

Profile loading failed for the following reason:
Runtime Error
  Credentials in profile "dbt_spark", target "development" invalid: Runtime Error
    odbc connection method requires additional dependencies.
    Install the additional required dependencies with `pip install dbt-spark[ODBC]`

Required dependencies:
 - git [OK found]

Expected behavior
I expected dbt debug to run without errors since all dependencies were already installed.
The output of dbt --version:
installed version: 0.19.1
   latest version: 0.19.1

Up to date!

Plugins:
  - bigquery: 0.19.1
  - snowflake: 0.19.1
  - redshift: 0.19.1
  - postgres: 0.19.1
  - spark: 0.19.1

The operating system you're using:
macOS 10.15.6 Catalina
The output of python --version:
Python 3.8.8

@giuliocavalcante Sorry you're running into this error! Are you installing dbt-spark in a virtual environment? Are you able to run pip freeze within that env, and confirm that pyodbc appears in the list of installed packages?

@jtcohen6 Yes, I'm running it in a pyenv.
pip freeze shows that i have pyodbc==4.0.30 installed.

Ok - if that error is being raised, it's because there's an ImportError when importing pyodbc:
https://github.com/fishtown-analytics/dbt-spark/blob/e84bd4f7b12279005b9401d8c2bdc769a5ff04b6/dbt/adapters/spark/connections.py#L19-L22
https://github.com/fishtown-analytics/dbt-spark/blob/e84bd4f7b12279005b9401d8c2bdc769a5ff04b6/dbt/adapters/spark/connections.py#L97-L103
Could you try importing pyodbc from within that pyenv, and see if you get any such error?

Issue solved by installing unixodbc (through brew install unixodbc). Apparently this is a necessary step when installing pyodbc via pip.
Cheers @jtcohen6.


Add workflow file through CLI

Added instructions on adding Github workflow file through CLI.

📚 Documentation preview 📚: https://ploomber-doc--64.org.readthedocs.build/en/64/

@neelasha23 don't forget to include responses and links to the observations. it's unclear to me if the were addressed or not

@neelasha23 don't forget to include responses and links to the observations. it's unclear to me if the were addressed or not

Have addressed all the comments. Most of the code has changed now so I didn't include the links of the previous deleted code.


Very slow?

Im running it on a server with 32 threads E5-2670, SSD and 128gb ram, windows server 2012 R2. Works fine but its SUPER SLOW. At best 2.7x speed, i even created a ramdisk for it to work on but its not many frames faster with it. Any idea how i can speed it up? Im trying to do 3 hour videos so id really like to speed it up..

I changed the outputAudioData from a numpy array to a python array, which lead to the script running about 5 times faster in my testing. It breaks the audiosmoothing feature between cuts, but I didn't really need that so I didn't really look too much into that. You could download it from here to take a look https://github.com/torsrex/jumpcutter/blob/master/jumpcutter.py It also adds some of the features from the pull requests on here.

I created a script for the https://github.com/torsrex/jumpcutter fork.
it splits the video into parts than uses jumpcutter.py on them and than merges them again.
This reduces the memory requirement @torsrex needs to whatever fits in your RAM.
Take a look https://github.com/Lamaun/jumpcutter/blob/master/jumpcutter.sh
The provided script wont work for Windows (help with a powershell script or whatever appreciated).

Awesome will look into these!


fix issue with outerjoin when there are repeated tables

This is a fix for https://github.com/toluaina/pgsync/issues/243, we notice that outer joins are disabled when any child has a filter applied, however, if a table is repeated then outerjoins should always be used when performing updates.

@toluaina is this fix to merge?


Move shared webview extension host code into platform

Re #10152
The webview extension host side code is mostly shared between Variable View, Plot view and Data viewer. The PR merges the extension-side/webview* code into platform/webviews as they can be reused by all webview related code.
With this change, the rest of the top folder src/webviews now only contains feature components, extension-side/feature-x and its webview counterpart webview-side/feature-x (though in difference naming conversion ;( )

The last PR will be moving src/webviews and src/standalone after this PR is merged.

[x] Pull request represents a single change (i.e. not fixing disparate/unrelated things in a single PR).
[x] Title summarizes what is changing.
[ ] Has a news entry file (remember to thank yourself!).
[ ] Appropriate comments and documentation strings in the code.
[ ] Has sufficient logging.
[ ] Has telemetry for feature-requests.
[ ] Unit tests & system/integration tests are added/updated.
[ ] Test plan is updated as appropriate.
[ ] package-lock.json has been regenerated by running npm install (if dependencies have changed).

Codecov Report

Merging #10529 (59b1a6c) into main (12f12a0) will decrease coverage by 0%.
The diff coverage is 97%.

@@          Coverage Diff           @@
##            main   #10529   +/-   ##
======================================
- Coverage     71%      71%   -1%
======================================
  Files        466      464    -2
  Lines      27909    27902    -7
  Branches    4699     4699
======================================
- Hits       19844    19821   -23
- Misses      6180     6202   +22
+ Partials    1885     1879    -6

Impacted Files
Coverage Δ

src/webviews/extension-side/variablesView/types.ts
100% <ø> (ø)

src/platform/webviews/webviewPanelProvider.ts
75% <93%> (ø)

src/platform/webviews/webview.ts
71% <100%> (ø)

src/platform/webviews/webviewHost.ts
80% <100%> (ø)

src/platform/webviews/webviewPanelHost.ts
93% <100%> (ø)

src/platform/webviews/webviewViewHost.ts
76% <100%> (ø)

src/platform/webviews/webviewViewProvider.ts
80% <100%> (ø)

src/standalone/serviceRegistry.node.ts
100% <100%> (ø)

...c/webviews/extension-side/dataviewer/dataViewer.ts
58% <100%> (ø)

...ebviews/extension-side/plotting/plotViewer.node.ts
20% <100%> (ø)

... and 14 more


Error when exclude-from-classmap points to non-existing folder

Hey, I'm having problem with installing Symfony\Process package.
It has the following line in composer.json:
    "autoload": {
        "psr-4": { "Symfony\\Component\\Process\\": "" },
        "exclude-from-classmap": [
            "/Tests/"
        ]
    },

It looks like the Composer distribution does not include this Test folder which causes the following error:
 [UnexpectedValueException]
  RecursiveDirectoryIterator::__construct(/Users/johannessiipola/Documents/Projects/searchwp-finnish-base-forms-new-local/app/public/wp-content/plugins/searchwp-finnish-base-forms/vendor/symfony
  /process//Tests/): failed to open dir: No such file or directory

I can prevent this error either by creating this folder or removing the line from the package's composer.json.

I have temporarily fixed this by adding the following to my Composer scripts to create this folder.
    "scripts": {
        "pre-update-cmd": [
            "mkdir -p vendor/symfony/Process/Tests"
        ]
    }

This error is caused by the exclude-from-classmap entries.
The last release (0.6.0) should fix this particular case.

Thanks @alessandromorelli
@joppuyo Reopen if v0.6 still failling.


fix: faucet correctly derives its account

Fix faucet not being able to send funds because its account didn't have any tokens.

Benchmark results
Metrics with a significant change:

batch_insert_into_append_only_tree_32_depth_ms (256): 132 (-69%)
batch_insert_into_append_only_tree_32_depth_hash_ms (256): 0.453 (-69%)
batch_insert_into_indexed_tree_20_depth_ms (256): 184 (-52%)
batch_insert_into_indexed_tree_20_depth_hash_ms (256): 0.486 (-53%)
avm_simulation_time_ms (Token:constructor): 123 (+33%)
avm_simulation_time_ms (Token:mint_public): 96.6 (+60%)
avm_simulation_time_ms (Token:_increase_public_balance): 14.8 (-82%)
avm_simulation_time_ms (FPC:pay_refund_with_shielded_rebate): 135 (+118%)
l2_block_processing_time_in_ms (4): 777 (-55%)

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Proof generation
Each column represents the number of threads used in proof generation.

Metric
1 threads
4 threads
16 threads
32 threads
64 threads

proof_construction_time_sha256_ms
5,725
1,545
706
742 (-2%)
769

proof_construction_time_sha256_30_ms
11,739
3,151
1,410 (+1%)
1,439
1,478 (+1%)

proof_construction_time_sha256_100_ms
43,809
12,045 (+1%)
5,482 (+1%)
5,453 (+1%)
5,391 (+1%)

proof_construction_time_poseidon_hash_ms
79.0 (+1%)
34.0
34.0
59.0 (+2%)
87.0 (+1%)

proof_construction_time_poseidon_hash_30_ms
1,518
419
202
221
269

proof_construction_time_poseidon_hash_100_ms
5,618
1,510
675 (-1%)
743 (+4%)
745 (-1%)

L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
4 txs
8 txs
16 txs

l1_rollup_calldata_size_in_bytes
708
708
708

l1_rollup_calldata_gas
6,576
6,592
6,576

l1_rollup_execution_gas
584,256
584,423
584,558

l2_block_processing_time_in_ms
:warning: 777 (-55%)
1,424 (+1%)
2,722 (-3%)

l2_block_building_time_in_ms
10,832
21,257 (+1%)
41,967 (-7%)

l2_block_rollup_simulation_time_in_ms
10,831
21,256 (+1%)
41,967 (-7%)

l2_block_public_tx_process_time_in_ms
9,061
19,265 (+1%)
39,836 (-8%)

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 8 txs.

Metric
3 blocks
5 blocks

node_history_sync_time_in_ms
7,082 (+1%)
9,923 (+1%)

node_database_size_in_bytes
12,759,120
16,916,560

pxe_database_size_in_bytes
16,254
26,813

Circuits stats
Stats on running time and I/O sizes collected for every kernel circuit run across all benchmarks.

Function
time_ms
bytecode_size_in_bytes

GasToken:_increase_public_balance
100 (+2%)
13,790

GasToken:set_portal
15.4 (+7%)
3,339

Token:constructor
:warning: 123 (+33%)
23,692

FPC:constructor
91.5 (+1%)
13,592

GasToken:mint_public
82.8 (+3%)
10,158

Token:mint_public
:warning: 96.6 (+60%)
19,034

Token:assert_minter_and_mint
62.6 (-3%)
12,925

AuthRegistry:set_authorized
46.4 (+4%)
7,812

FPC:prepare_fee
233 (-9%)
11,068

Token:transfer_public
46.0 (-14%)
44,012

FPC:pay_refund
75.9 (+7%)
15,961

Benchmarking:increment_balance
1,069 (-2%)
11,281

Token:_increase_public_balance
:warning: 14.8 (-82%)
15,006

FPC:pay_refund_with_shielded_rebate
:warning: 135 (+118%)
17,056

Public DB Access
Time to access various public DBs.

Function
time_ms

get-nullifier-index
0.153

Tree insertion stats
The duration to insert a fixed batch of leaves into each tree type.

Metric
1 leaves
16 leaves
64 leaves
128 leaves
256 leaves
512 leaves
1024 leaves

batch_insert_into_append_only_tree_16_depth_ms
10.4 (-2%)
16.7 (-2%)
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_count
16.8
31.7
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_ms
0.599 (-2%)
0.512 (-2%)
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_32_depth_ms
N/A
N/A
48.2
75.9 (-3%)
:warning: 132 (-69%)
246
471

batch_insert_into_append_only_tree_32_depth_hash_count
N/A
N/A
95.9
159
287
543
1,055

batch_insert_into_append_only_tree_32_depth_hash_ms
N/A
N/A
0.492
0.467 (-3%)
:warning: 0.453 (-69%)
0.445
0.438

batch_insert_into_indexed_tree_20_depth_ms
N/A
N/A
59.2
112
:warning: 184 (-52%)
353 (+1%)
700 (+1%)

batch_insert_into_indexed_tree_20_depth_hash_count
N/A
N/A
109
207
355
691
1,363

batch_insert_into_indexed_tree_20_depth_hash_ms
N/A
N/A
0.500 (-1%)
0.502
:warning: 0.486 (-53%)
0.478
0.482 (+1%)

batch_insert_into_indexed_tree_40_depth_ms
N/A
N/A
73.0 (-1%)
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_count
N/A
N/A
133
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_ms
N/A
N/A
0.519 (-1%)
N/A
N/A
N/A
N/A

Miscellaneous
Transaction sizes based on how many contract classes are registered in the tx.

Metric
0 registered classes
1 registered classes

tx_size_in_bytes
72,017
666,910

Transaction size based on fee payment method
| Metric |  |
| - |  |