/WFLY-10519 remove Hibernate JPA 2.1 + Hibernate ORM 5.1.x + Hibernate Search 5.5.8

https://issues.jboss.org/browse/WFLY-10519
I'll squash the commits after we pass the WF testsuite.

Good that the only failure is with org.jboss.as.test.integration.jpa.secondlevelcache.JPA2LCTestCase:
#testEntityCacheSameSession  	 #testEntityCacheSecondSession
`#testEvictEntityCache
Next step is understanding if this reflects a bug or a test needing changes.

Is Hibernate 5.3 backward compatibility work done ?
CCing @mjurc @simkam

@rsvoboda no, IMO, it is not yet done.  I created this PR early, so others can see progress of the effort, as well as getting more complete test runs.
A fix for https://hibernate.atlassian.net/browse/HHH-12651 is needed, we also are working on an application bytecode transformer (https://issues.jboss.org/browse/WFLY-10520) that at deployment time, will scan application bytecode for some simple issues and rewrite the bytecode, to work.  This might look something like https://github.com/scottmarlow/wildfly/blob/WFLY-10520_hibernatecompatabilitytransformer/jpa/subsystem/src/main/java/org/jboss/as/jpa/Hibernate51CompatibilityTransformer.java, which is work in progress.
The current plan is for applications to make a deployment descriptor change to enable the compatibility transformer that requires repackaging their "native Hibernate" based application, from reading https://github.com/hibernate/hibernate-orm/wiki/Migration-Guide---5.2 + https://github.com/hibernate/hibernate-orm/wiki/Migration-Guide---5.3, Hibernate native applications will most likely require some changes anyway (EE standard apps using JPA, shouldn't need changes.
Another question with Hibernate compatibility transformer, that we are trying to answer, is how deep should the transformer go into rewriting application bytecode?  For example, see question here about s.createQuery( "from A" ).setMaxResults( 0 ).list().  IMO, it might be best to not attempt to rewrite application bytecode too deeply, but instead either fix the setMaxResults(0) issue in Hibernate ORM 5.3 or by the user, making application changes directly.
We also need to see the org.jboss.as.test.integration.jpa.secondlevelcache.JPA2LCTestCase test pass (not sure why the second level cache is not caching as expected).  Until we understand why this test fails, we really won't know how it will be fixed.
Also, I should add that I think everyone is waiting on @n1hility to give his thumbs up on it being time for WF 14 to default to EE 8 (as well as remove EE 7 technologies).

@scottmarlow Adding the hold label as you indicated on mail. Let me know when it can be removed

I think that before switching default hibernate we should:

upgrade hibernate to at least 5.3.2 (fixes WFLY-10494)
merge compatibility transformer (WFLY-10520, including global switch)
perhaps wait for other fixes (https://issues.jboss.org/issues/?jql=project %3D WFWIP AND component %3D JPA)

What is the relation of this PR to https://issues.jboss.org/browse/EAP7-999 ? Seems weird to me that the RFE is still in kickoff with all PT in TODO

@msve I updated EAP7-999 to "dev in progress"

@simkam
Regarding your suggestion that we wait, until Hibernate ORM 5.3.2 is available and other ORM issues are resolved, we need WF14 to default to EE 8 now.

I think that before switching default hibernate we should:
upgrade hibernate to at least 5.3.2 (fixes WFLY-10494)

We can upgrade later to 5.3.2 and then 5.3.3, if there are further updates, we can upgrade further.

merge compatibility transformer (WFLY-10520, including global switch)

The transformer depends on 5.3.2, so we really need to wait for 5.3.2 before merging the transformer in.

perhaps wait for other fixes (https://issues.jboss.org/issues/?jql=project%20%3D%20WFWIP%20AND%20component%20%3D%20JPA)

When the fixes for these issues are available in a ORM 5.3.x release, we can uipgrade again.

FYI, we plan to release ORM 5.3.2.Final next week. It would be nice if this PR could be integrated at that point so that we could upgrade ORM without creating conflicts.

@kabir @scottmarlow We would like stick to process, so please don't merge this until PTs are in appropriate state. If possible I also prefer single PR containing all related changes (this + transformer + hibernate upgrade).

@simkam

@kabir @scottmarlow We would like stick to process, so please don't merge this until PTs are in appropriate state. If possible I also prefer single PR containing all related changes (this + transformer + hibernate upgrade).

I'm not sure of when the transformer changes will be complete enough.  Regarding the transformer, that is for the non-standard ORM native api, which is important but IMO, shouldn't block this PR.

@kabir @scottmarlow We're a month away from WildFly 14 feature freeze and we don't yet have any guarantees and plans on ensuring backwards compatibility for Hibernate. We don't have anything tangible we could assess the risks on either. Therefore this is in my opinion not ready for merge (besides other missing requirements).

retest this please.

as discussed in mail thread this can be merged, but please don't resolve RFE, there are still PTs to be done. Also please don't merge analysis document yet, we might need to update it.

Retest this please
@jamezp can you merge it when done (and don't resolve anything as per @simkam 's comment)


[NEW]: Looking for ALL and ANY skills to start a community-driven non-profit organization to promote Ethereum

Introduction
Talk a little about yourself:

Who you are: Vanja
What you do: Former TradFi researcher - now data/graphic designer and Ethereum user
How you got first into Ethereum: Saw the light, took the red pill
What brings you to ETHBerlin: Looking to build community and contribute to the ecosystem

Idea description

A community-driven non-profit who's mission is to promote Ethereum among marginalized communities and in parts of the world where adoption can achieve the greatest socio-economic impact.
Think of it as bringing together experts who love Ethereum combining their superpowers to do some good where it matters the most!
I've put together a brief website around this idea at www.ethcommons.org. This is not at all set in stone though!
Skillset
What skills do you need, or think you might need to implement the idea?
Everyone and anyone is welcome to join! The primary goal around doing this at the EthBerlin hackathon is to brainstorm, create inspiring conversations and put together a plan to move forward.
Communication
@vanja.eth:matrix.org

I might be interested! #82

Cool! Come find me at 19:00 at the team finding session for a chat! I'm very tall and my name is quite uncommon :)


adds util api functions. adds downstream lineage/relation api request …

…function. minor refactors

Coverage remained the same at 27.757% when pulling 9728ae46fd091677d2dbafb2d3824cb9a016c323 on theseyi:dataset-relationsips into 52d77b8f1735b339d35b81f34886f9e31841490c on linkedin:master.

Codecov Report

Merging #1266 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1266   +/-   ##
=======================================
  Coverage   16.43%   16.43%
=======================================
  Files          28       28
  Lines         645      645
  Branches      138      138
=======================================
  Hits          106      106
  Misses        539      539

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 52d77b8...9728ae4. Read the comment docs.


Feature Request - specify 2.1 & short text encoder description

I see that v2 was replaced with 2.1 entirely but it's still called v2 in the colab. I think to reduce confusion it should be specified as "v2.1".
Also, The changes made to the text encoder description are even more ominous now, I know people would very much appreciate a short description of what it's doing, in layman's terms, so they can decide if and for how long they should use it for their specific purposes.
Thanks Ben, your work is massively appreciated.

With the latest update, I separated the text_enc training from the UNet training, which helps give good results at low step count, but the text_enc steps should be kept low, no matter the dataset 350-500 steps is more than enough.


Merged Java PRs

This can't be merged directly, but should be about equivalent to everything currently out there merged together, and I'm looking to get a linter pass through on it, and be able to compare new PRs against it.

 Here is an overview of what got changed by this pull request:

Issues
======
+ Solved 1

Complexity increasing per file
==============================
- java/src/main/java/com/saucelabs/simplesauce/MacVersion.java  1

See the complete overview on Codacy

wow, it is lint free, I'm surprised.


Add support for running compute worker with other container engines

We merged these changes in the podman branch of this repository in order to run the automatic tests.
Here are the details of the original PR:
https://github.com/codalab/codabench/pull/763

@Didayolo Not sure what happened to my PR? Merged into another branch? I have a couple more commits I would like to push.

OK, I see you are running the CI, can I push my additional commits?

@Didayolo Not sure what happened to my PR? Merged into another branch? I have a couple more commits I would like to push.

@cjh1
Hi Chris, indeed that is a bit a confusing move. In the future, it may be a better practice to keep external contributions in the forks of the repository. If you do new commits to your branch in your repository, we can simply merge it to this new branch.

@Didayolo I just pushed two more commits to my branch.

We could have tested the feature locally on our test environment with "git fetch origin pull/763/head:podmanchris" and "git checkout podmanchris". We can also run our tests locally within django container launching the py.test command from it. I think this is a better workflow for next time.

Hi all,
I've been messing around with this this weekend and first I want to say Anne-Catherine's suggestion "git fetch origin pull/763/head:podmanchris" works fantastically and I believe that gives the effect we saw at first where the repo chris made called "podman" was effectively cloned to a local branch we had on the test VM (codabench-test.lri.fr).
Anyways the rest of my experience is encapsulated below:

Notes on PR: https://github.com/codalab/codabench/pull/772
First Steps:
create a queue
# podman_queue_cpu
BROKER_URL=pyamqp://82801eec-626e-4448-98ef-0340cf22ab17:c0b6d3d9-931f-4612-8289-b021fccff340@codabench-test.lri.fr:5672/740536a7-1183-4169-990f-9bc950e0c9cb

Copy necessary files from codabench repo to personal working directory and make necessary files:
# as user bbearce
cd /home/bbearce;
mkdir podman_pr_testing;
cp /home/codalab/codabench/podman /home/bbearce/podman_pr_testing/podman;
cp /home/codalab/codabench/compute_worker /home/bbearce/podman_pr_testing/compute_worker;
cp /home/codalab/codabench/Containerfile.compute_worker_podman /home/bbearce/podman_pr_testing/Containerfile.compute_worker_podman;
touch podman_pr_testing/.env;

Fill .env with this text:
BROKER_URL=pyamqp://82801eec-626e-4448-98ef-0340cf22ab17:c0b6d3d9-931f-4612-8289-b021fccff340@codabench-test.lri.fr:5672/740536a7-1183-4169-990f-9bc950e0c9cb
HOST_DIRECTORY=/codabench
CONTAINER_ENGINE_EXECUTABLE=podman

Build podman image
Create these files:

build_podman_container.sh
run_podman_container.sh

build_podman_container.sh
podman build -f Container.compute_worker_podman -t codalab/codabench_worker_podman .

This creates a container named as follows (localhost/codalab/codabench_worker_podman):

Note the extra "localhost"

bbearce@codabench-test-acl-220909:~/podman_pr_testing$ podman images
REPOSITORY                                 TAG     IMAGE ID      CREATED         SIZE
localhost/codalab/codabench_worker_podman  latest  3560a6e0c8c2  12 minutes ago  747 MB

Create worker user
sudo useradd worker

Contents of /etc/passwd:
...
dtran:x:1004:1004:,,,:/home/dtran:/bin/bash
acl:x:1005:1005::/home/acl:/bin/bash
worker:x:1006:1006::/home/worker:/bin/sh

run_podman_container.sh

Notice the "localhost"

# notice the "localhost" that is necessary
podman run -d \
  --env-file .env \
  --name compute_worker \
  --security-opt="label=disable" \
  --device /dev/fuse \
  --user worker \
  --restart unless-stopped \
  --log-opt max-size=50m \
  --log-opt max-file=3 \
  localhost/codalab/codabench_worker_podman

The container launches successfully and I get this in the logs:
bbearce@codabench-test-acl-220909:~/podman_pr_testing$ podman logs -f compute_worker

 -------------- compute-worker@767974418f5b v4.4.0 (cliffs)
--- ***** -----
-- ******* ---- Linux-5.10.0-17-cloud-amd64-x86_64-with-glibc2.34 2023-02-21 03:04:50
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:         __main__:0x7ff99a447130
- ** ---------- .> transport:   amqp://82801eec-626e-4448-98ef-0340cf22ab17:**@codabench-test.lri.fr:5672/740536a7-1183-4169-990f-9bc950e0c9cb
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> compute-worker   exchange=compute-worker(direct) key=compute-worker

Run submission on hello world challenges
When running a submission I get these logs:
[2023-02-21 03:06:13,526: INFO/ForkPoolWorker-1] Running pull for image: codalab/codalab-legacy:py3
time="2023-02-21T03:06:13Z" level=error msg="running `/usr/bin/newuidmap 16 0 1000 1 1 1 999 1000 1001 64535`: newuidmap: write to uid_map failed: Operation not permitted\n"
Error: cannot set up namespace using "/usr/bin/newuidmap": should have setuid or have filecaps setuid: exit status 1
[2023-02-21 03:06:13,589: INFO/ForkPoolWorker-1] Full command ['podman', 'pull', 'docker://codalab/codalab-legacy:py3']
[2023-02-21 03:06:13,589: INFO/ForkPoolWorker-1] Pull for image: codalab/codalab-legacy:py3 returned a non-zero exit code! Full command output ['podman', 'pull', 'docker://codalab/codalab-legacy:py3']
[2023-02-21 03:06:13,589: INFO/ForkPoolWorker-1] Updating submission @ https://codabench-test.lri.fr/api/submissions/563/ with data = {'status': 'Failed', 'status_details': 'Pull for codalab/codalab-legacy:py3 failed!', 'secret': 'afef73e2-9939-41f5-a804-9dc383e50eb9'}
[2023-02-21 03:06:13,612: INFO/ForkPoolWorker-1] Submission updated successfully!
[2023-02-21 03:06:13,612: INFO/ForkPoolWorker-1] Destroying submission temp dir: /codabench/tmpvvmn6l16
[2023-02-21 03:06:13,614: INFO/ForkPoolWorker-1] Task compute_worker_run[155e6621-6190-4480-a073-2d6872af80ca] succeeded in 0.24267210997641087s: None

I can shell into the container as so:
podman exec -it compute_worker bash

Once there I can simulate the same actions with (simulate_worker_actions.py):
# simulate_worker_actions.py
from subprocess import CalledProcessError, check_output

CONTAINER_ENGINE_EXECUTABLE = "podman"
image_name = "codalab/codalab-legacy:py3"

cmd = [CONTAINER_ENGINE_EXECUTABLE, 'pull', image_name]
# ['podman', 'pull', 'docker://codalab/codalab-legacy:py3']
container_engine_pull = check_output(cmd)

I get the following output:
>>> container_engine_pull = check_output(cmd)
ERRO[0000] running `/usr/bin/newuidmap 42 0 1000 1 1 1 999 1000 1001 64535`: newuidmap: write to uid_map failed: Operation not permitted
Error: cannot set up namespace using "/usr/bin/newuidmap": should have setuid or have filecaps setuid: exit status 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['podman', 'pull', 'codalab/codalab-legacy:py3']' returned non-zero exit status 125.
>>>

I believe there is something wrong the the created user "worker" that was made outside the image (on host server) and the one made in the Podmanfile (Dockerfile like file) "Container.compute_worker_podman". It seems the created worker user doesn't have the right permissions on the server to do a podman pull command.

@bbearce Can you confirm that when you built your podman image the branch included 3c27d79 ? This was added to fix an issue that looks the same as the one you are seeing.

I can confirm that I was NOT on that commit haha. I think we originally made this right branch and pulled your code before you added this. I deleted the branch we had and pulled the latest pr and voila!

@bbearce Great, I was hoping that was the case 😄. So the GPU version was added by @dtuantran ( I made two updates to get it working ), however, we will not be using it for deployment at NERSC as we have infrastructure that injects the correct drivers and libraries for the GPUs. So if testing that out is holding up this PR I would say we move it to a separate PR and get the CPU worker merged.

@bbearce You wouldn't expect to see any GPU processes unless there is code running that is using the GPU. In the invocation you show above there would be nothing running as you are just run bash and nvidia-smi.

Hi all,
Having a chance to review this with Tuan the issue was that I logged in originally as user "ubuntu". I used sudo su worker to change users to "worker". The problem is my terminal and login session are still saving environment variables tied to "ubuntu" even though I'm "worker" now. The solution is to completely log out and login from the beginning as "worker". This sets up the environment to be a true "worker" user.
Once I did that Tuan and Anne-Catherine reminded me I don't need to clone the codabench directory and build a GPU worker image. The design is so that users can clone our pre-built image for simplicity. After taking all of this into account, the new worker that was spun up did no need sudo rights and connected perfectly. It could see the GPU and we could run GPU submissions.
I feel confident now that this is fully tested and verified. My last note is we should mention this login situation in the wiki post PR acceptance.

Ready for merge!

Ready for merge!

@bbearce we have the following CircleCI test failing:
https://app.circleci.com/pipelines/github/codalab/codabench/925/workflows/e1f3cdc3-9bb3-41b8-9608-c9077f02fc7b/jobs/2604
_ ERROR collecting src/apps/competitions/tests/test_legacy_command_replacement.py _
ImportError while importing test module '/app/src/apps/competitions/tests/test_legacy_command_replacement.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/apps/competitions/tests/test_legacy_command_replacement.py:2: in <module>
    from docker.compute_worker.compute_worker import replace_legacy_metadata_command
E   ModuleNotFoundError: No module named 'docker'

@bbearce good catch, the CircleCI bug was due to a renaming of files.


update phusion base image to v0.11

See https://github.com/phusion/baseimage-docker/releases/tag/0.11

Thanks for that!

My pleasure! Thanks for the fantastic SDK :)


Overall Design

Is this pull request associated with an issue(s)?
No. In preparation for putting together the invited paper I wanted to start capturing the design.
Description
This PR contains the top-level design. My plan is to work top down starting with what the user sees, ending with inheritance diagrams. I've also left a wake of stub pages as I've been doing this PR. They need to be addressed in other PRs.

This is ready for review. There's sort of a lot of stuff in this PR, but just concentrate on the overview.rst file and terminology.rst; just about everything else is a stub to be filled in later. My next PR will move on the user API stub.

I'll try and figure out what's wrong with the .bib file tomorrow (may be it's empty). Other repos have licensed .bib files and the action doesn't fail.


'React/RCTBridgeModule.h' file not found

I am receiving a 'React/RCTBridgeModule.h' file not found error after installing.  I have tried installing manually and using react-native link.  I am on react native version 0.39.1 using react-native-vector-icons version 4.0.0.  The issue was observed on iOS.  I have not tested android.

Duplicate of https://github.com/oblador/react-native-vector-icons/issues/380


Moe/readme

Codecov Report

Merging #19 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #19   +/-   ##
=======================================
  Coverage   87.39%   87.39%
=======================================
  Files           8        8
  Lines         119      119
=======================================
  Hits          104      104
  Misses         15       15

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5d470c3...3844f95. Read the comment docs.


Snake game in Java

💥 Proposal
(A clear and concise description of what the proposal is.)
Kindly allow me to implement the 2D snake game in Java as a part of GSSOC'21.
Have you read the Contributing Guidelines on Pull Requests?
Yes, I have.
(Write your answer here.)

As I am a beginner, Can I start my contribution by working on this issue under GSSoC'21? Kindly assign it to me.


[r2.0.0-rc1] Converting to TFLite format: Invalid quantization params for op RESHAPE

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04.2 LTS
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
TensorFlow installed from (source or binary): Source
TensorFlow version (use command below): 2.0.0-rc1 commit 59bf33
Python version: 3.6.8
Bazel version (if compiling from source): 0.26.1
GCC/Compiler version (if compiling from source): 7.4.0
CUDA/cuDNN version: 10.0.130
GPU model and memory: NVidia 1080Ti / 11G

Describe the current behavior
Conversion of TF2.0 function containing softmax and reshape ops to TFLite format fails with the following RuntimeError. Source code of the program is listed in the Code section below.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-ec9775ede022> in <module>
----> 1 run()

~/mironov/hbtest/tflite_softmax_bug_v2.py in run()
     20   converter.optimizations = [tf.lite.Optimize.DEFAULT]
     21
---> 22   tflite_model = converter.convert()
     23   model_file = "/tmp/tflite_softmax_bug_v2.tflite"
     24   open(model_file, "wb").write(tflite_model)

/usr/local/lib/python3.6/dist-packages/tensorflow_core/lite/python/lite.py in convert(self)
    448     if self._is_calibration_quantize():
    449       result = self._calibrate_quantize_model(result, constants.FLOAT,
--> 450                                               constants.FLOAT)
    451
    452     return result

/usr/local/lib/python3.6/dist-packages/tensorflow_core/lite/python/lite.py in _calibrate_quantize_model(self, result, inference_input_type, inference_output_type)
    237     return calibrate_quantize.calibrate_and_quantize(
    238         self.representative_dataset.input_gen, inference_input_type,
--> 239         inference_output_type, allow_float)
    240
    241   def _get_base_converter_args(self):

/usr/local/lib/python3.6/dist-packages/tensorflow_core/lite/python/optimize/calibrator.py in calibrate_and_quantize(self, dataset_gen, input_type, output_type, allow_float)
     76     return self._calibrator.QuantizeModel(
     77         np.dtype(input_type.as_numpy_dtype()).num,
---> 78         np.dtype(output_type.as_numpy_dtype()).num, allow_float)

/usr/local/lib/python3.6/dist-packages/tensorflow_core/lite/python/optimize/tensorflow_lite_wrap_calibration_wrapper.py in QuantizeModel(self, input_py_type, output_py_type, allow_float)
    113
    114     def QuantizeModel(self, input_py_type, output_py_type, allow_float):
--> 115         return _tensorflow_lite_wrap_calibration_wrapper.CalibrationWrapper_QuantizeModel(self, input_py_type, output_py_type, allow_float)
    116 CalibrationWrapper_swigregister = _tensorflow_lite_wrap_calibration_wrapper.CalibrationWrapper_swigregister
    117 CalibrationWrapper_swigregister(CalibrationWrapper)

RuntimeError: Invalid quantization params for op RESHAPE at index 1 in subgraph 0

Describe the expected behavior
converter.convert() call finishes without errors
Code to reproduce the issue
File: tflite_softmax_bug_v2.py
import tensorflow as tf
import numpy as np

@tf.function(input_signature=[
    tf.TensorSpec(shape=[10], dtype=tf.float32, name='inp0')])
def model(inp0):
  res = tf.nn.softmax(inp0)
  res = tf.reshape(res, [1, 10])
  print(res.shape)
  return res

def run():
  def _representative_dataset_gen():
    for i in range(10):
      yield [np.random.random_integers(0, 9, size=[10]).astype('float32'),]

  cfunc = model.get_concrete_function()
  converter = tf.lite.TFLiteConverter.from_concrete_functions([cfunc])
  converter.representative_dataset = _representative_dataset_gen
  converter.optimizations = [tf.lite.Optimize.DEFAULT]

  tflite_model = converter.convert()
  model_file = "/tmp/tflite_softmax_bug_v2.tflite"
  open(model_file, "wb").write(tflite_model)

  interpreter = tf.lite.Interpreter(model_path=model_file)
  interpreter.allocate_tensors()

if __name__ == '__main__':
  run()

Other info / logs
N/A

The error can be also reproduced in  tensorflow nightly gpu version 1.15,
2019-09-26 09:57:52.194571: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:786] Optimization results for grappler item: graph_to_optimize
2019-09-26 09:57:52.194601: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788]   constant_folding: Graph size after: 4 nodes (-1), 3 edges (0), time = 1.698ms.
2019-09-26 09:57:52.194611: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788]   constant_folding: Graph size after: 4 nodes (0), 3 edges (0), time = 0.198ms.
Traceback (most recent call last):
  File "tflite_softmax_bug_v1.py", line 33, in <module>
    tflite_model = converter.convert()
  File "/home/hh1208-kang/venv_py2_tf_nightly/local/lib/python2.7/site-packages/tensorflow_core/lite/python/lite.py", line 993, in convert
    inference_output_type)
  File "/home/hh1208-kang/venv_py2_tf_nightly/local/lib/python2.7/site-packages/tensorflow_core/lite/python/lite.py", line 239, in _calibrate_quantize_model
    inference_output_type, allow_float)
  File "/home/hh1208-kang/venv_py2_tf_nightly/local/lib/python2.7/site-packages/tensorflow_core/lite/python/optimize/calibrator.py", line 78, in calibrate_and_quantize
    np.dtype(output_type.as_numpy_dtype()).num, allow_float)
  File "/home/hh1208-kang/venv_py2_tf_nightly/local/lib/python2.7/site-packages/tensorflow_core/lite/python/optimize/tensorflow_lite_wrap_calibration_wrapper.py", line 115, in QuantizeModel
    return _tensorflow_lite_wrap_calibration_wrapper.CalibrationWrapper_QuantizeModel(self, input_py_type, output_py_type, allow_float)
RuntimeError: Invalid quantization params for op RESHAPE at index 1 in subgraph 0

with this code.
import tensorflow as tf
import numpy as np

export_dir = 'test_saved_model'
builder = tf.saved_model.builder.SavedModelBuilder(export_dir)

ipts = tf.placeholder(dtype=tf.float32, shape=(10), name='x')
res = tf.nn.softmax(ipts)
res = tf.reshape(res, [1, 10])
print(res.shape)

def _representative_dataset_gen():
    for i in range(10):
      yield [np.random.random_integers(0, 9, size=[10]).astype('float32'),]

with tf.Session() as sess:
	sess.run(tf.global_variables_initializer())

	builder.add_meta_graph_and_variables(sess, ["test"])
	builder.save(as_text=True)

	converter = tf.lite.TFLiteConverter.from_session(sess, [ipts], [res])

	converter.representative_dataset = _representative_dataset_gen
	converter.optimizations = [tf.lite.Optimize.DEFAULT]

	tflite_model = converter.convert()

It seems that the converter try to "quantize" the input 1, the op RESHAPE, I guess.
Inputs {
  0: Softmax, float[10]
  1: shape, int32[2] = {1, 10}
}
Outputs {
  0: Reshape, float[1x10]
}
Options {
  new_shape
}

Firstly, the converter set the quantization parameters of the output tensor of SOFTMAX op. Let's name it as A. The quantization parameter of the tensor A is set to {scale 0.003906, zp -128.000000}. But min and max are not set.
When the converter tries to quantize the output tensor of the RESHAPE op, which follows the SOFTMAX op, it reads the tensor A as the input tensor. It tries to use the quantization parameters of tensor A, however, it fails since the min and max of the tensor A is not set.

I was currently working with modifying the tensorflow r1.14 version. Thus I'm not sure about other versions.
When I changed the following code
https://github.com/tensorflow/tensorflow/blob/be5eb917fdea8a99ec8d5eb9428f2e8866858560/tensorflow/lite/tools/optimize/quantize_model.cc#L550
  } else if (tensor_property.restriction) {
    const auto scale_and_zp = tensor_property.restricted_value;
    // Apply to output.
    output_tensor->quantization = absl::make_unique<QuantizationParametersT>();
    output_tensor->quantization->scale.push_back(scale_and_zp.first);
    output_tensor->quantization->zero_point.push_back(scale_and_zp.second);
    output_tensor->type = TensorType_INT8;
  } else {

as
  } else if (property.restriction_on_output) {
    const auto scale_and_zp = property.restricted_value_on_output;
    // Apply to output.
    output_tensor->quantization = absl::make_unique<QuantizationParametersT>();
    output_tensor->quantization->scale.push_back(scale_and_zp.first);
    output_tensor->quantization->zero_point.push_back(scale_and_zp.second);
    output_tensor->type = TensorType_INT8;

    const int input_index = op->inputs[property.input_indexes[0]];
    TensorT* input_tensor = subgraph->tensors[input_index].get();
    output_tensor->quantization->min.push_back(input_tensor->quantization->min[0]);
    output_tensor->quantization->max.push_back(input_tensor->quantization->max[0]);
  } else {

the converter completed the converting. However, I'm not sure this is safe or not.

Considering the tensorflow quantization specification (https://www.tensorflow.org/lite/performance/quantization_spec),
SOFTMAX
  Input 0:
    data_type  : int8
    range      : [-128, 127]
    granularity: per-tensor
  Output 0:
    data_type  : int8
    range      : [-128, 127]
    granularity: per-tensor
    restriction: (scale, zero_point) = (1.0 / 256.0, -128)

it seems better to use fixed min/max values as follows, rather than using the same min/max from the input tensor.
    const float scale = output_tensor->quantization->scale[0];
    const int32_t zp = output_tensor->quantization->zero_point[0];
    float min = (-128 - zp) * scale;
    float max = (127 - zp) * scale;
    output_tensor->quantization->min.push_back(min);
    output_tensor->quantization->max.push_back(max);
    printf("scale=%g, zero_point=%d, min=%g, max=%g\n",
    output_tensor->quantization->scale[0], output_tensor->quantization->zero_point[0],
    output_tensor->quantization->min[0], output_tensor->quantization->max[0]);

Just a little update.
I'm working with the v2.0.0, and I think that following one line modification seems better.
https://github.com/tensorflow/tensorflow/blob/64c3d382cadf7bbe8e7e99884bede8284ff67f56/tensorflow/lite/tools/optimize/quantize_model.cc#L528
    //output_tensor->quantization = absl::make_unique<QuantizationParametersT>();
    if (!output_tensor->quantization){
        output_tensor->quantization = absl::make_unique<QuantizationParametersT>();
    }

When I checked my model, it seems that tensors have min/max parameters of their own. So it seems enough to prevent the initialization of the quantization parameter, when it already had been initialized.

How is it going for now?

How is it going for now?
Continued in this issue
https://github.com/tensorflow/tensorflow/issues/33233

Any update about this issue? Reshape is very common operation in TFLite -- very strange to see that this problem is just skipped by Tensorflow Team about six months...

I have a similar problem, my model used MobileNet. When trying to optimize to int8 model under TF2.0, it raise:
return _tensorflow_lite_wrap_calibration_wrapper.CalibrationWrapper_QuantizeModel(self, input_py_type, output_py_type, allow_float)
RuntimeError: Invalid quantization params for op RESHAPE at index 22 in subgraph 0
After chaning to TF2.1, the error not rasized. And the output of tflite is correct.

We struggled in tensorflow-gpu 2.0.0 also with this bug. For us, the issue was resolved with an update from version 2.0.0 to 2.2.0.

@grwlf
Could you please try on latest stable version of tf 2.5 or 2.4.1 and let us know if this is still an issue.


Any reason why two KSCrash handlers are created?

There are two handlers
static const char* kThreadPrimary = "KSCrash Exception Handler (Primary)";
static const char* kThreadSecondary = "KSCrash Exception Handler (Secondary)";

https://github.com/kstenerud/KSCrash/blob/master/Source/KSCrash/Recording/Monitors/KSCrashMonitor_MachException.c#L50
Is it possible to have only one. This will reduce KSCrash overhead IMO.

pull request update: fix getdata() mask #429 fix it


PHPLIB-1228: Run legacy transaction tests on LB topologies

https://jira.mongodb.org/browse/PHPLIB-1228
https://jira.mongodb.org/browse/PHPLIB-1270
Patch build: https://spruce.mongodb.com/version/651c4fcd61837d018a6ef901/tasks

Note: the Ubuntu 20.04 test failures are all related to kms_kmip_server.py and Python 3.12 (see: Slack thread).
Will merge as-is and forgo a PHPLIB ticket for the build failure since this will ultimately be resolved in drivers-evergreen-tools.


Downlink enqueued after a TX and before RXs windows opening are not sent immediately

Describe the bug
When a downlink is enqueued after sending an uplink but before opening the RX1 and RX2 windows, the downlink is not sent immediately to the gateway, regardless of the RX1 delay, but will wait for a second uplink to send the downlink on the second RX1/RX2 windows.

To Reproduce

Configure RX1 delay with any value
Send an uplink
Quickly enqueue a downlink

Expected behavior
I expect the downlink to be sent to the RX1/RX2 windows relative to the first uplink.

Device Info (please complete the following information):

Environment (Production, Staging): production
Console Device ID: 4be56505-ca63-4440-8de7-114e2b22bb10
Device dev eui / app eui: ACACACACACACAC12/ACACACACACACAC12
How often is device sending data: Regularly
Region: EU868
Hotspots (names) used to transfer data: fast-hickory-raven/Real Smoke Nightingale

NOTE: do not post your device app_key

Yes this is right. The Router at that point as already made the decision to send (or not) the downlink.
The rx window (or rx delay) is then applied at the hotspot level.
For a downlink to be sent it needs to be queued before an uplink.

Yes this is right. The Router at that point as already made the decision to send (or not) the downlink.
The rx window (or rx delay) is then applied at the hotspot level.
For a downlink to be sent it needs to be queued before an uplink.

Hi, this is not how LoRaWAN network server usually work, and while an application should be able to cope with a downlink sent on the next uplink it usually allow optimizations that save power and network capacity.
Would you mind to check if that changing this behavior is a complicated thing on your side?
Thanks

Unfortunately it would be a major change as most of this lives on the Hotspot

I consider this a major bug as well. When the end-node is requesting something from it's application server (for instance, the current time) it now has no chance of receiving a reply, and will always have to send the request twice, and process the reply twice. Even if the RxDelay has been set to 15 seconds. I don't know of any other network server that has this design. Usually the network server will make the downlink decision later, at some fraction of the RxDelay.

Unfortunately it would be a major change as most of this lives on the Hotspot

Even with the upcoming gateway-rs switch? Can't a hotspot open a gRPC bidirectionnal streaming that the router can use to push the downlink to the hotspot, or close it after RxDelay if it has not received any downlink from the AS?


Add a bun example

WHY are these changes introduced?
This PR demonstrates Hydrogen running on Bun. To try it out:

cd examples/bun
bun install
bun run dev

Or run it in production mode with bun run start

WHAT is this pull request doing?

HOW to test your changes?

Post-merge steps

Checklist

[ ] I've read the Contributing Guidelines
[ ] I've considered possible cross-platform impacts (Mac, Linux, Windows)
[ ] I've added a changeset if this PR contains user-facing or noteworthy changes
[ ] I've added tests to cover my changes
[ ] I've added or updated the documentation

I think it's fine to close. Someone can find it with a search. It's not really an "official" example, but maybe helpful as a starting point for anyone that really wants to use bun.


fix: split cookies into multiple set-cookie headers

This fixes the issue noticed by @shirecoding in #113 , where multiple cookies cannot be set using the same Set-Cookie header

thx!

@BearToCode I've tested it temporarily as follows and confirm that ti works
thanks!
import { createTRPCHandle } from "trpc-sveltekit-monorepo/package/src/";

Glad that fixed the issue!
I didn't understand what you mean with merging:

import { createTRPCHandle } from "trpc-sveltekit-monorepo/package/src/";

Regarding test cases maybe is better if you ask the repo owner


Adding MeshMap keyboard shortcuts in the MeshMap docs.

Current Behavior
Currently, we don't have MeshMap keyboard shortcuts in the MeshMap docs.
Desired Behavior
We have to add the MeshMap keyboard shortcuts in the MeshMap docs.
location in meshmap of keyboard shortcuts.

Contributor Guide

🎨 Wireframes and designs for Layer5 site in Figma (open invite)

Adding missing "help wanted" and "good first issue" labels.

I would like to work on this issue

@abhijeetgauravm  i am facing a issue in setting up local env(
. May be docker target is missing in Makefile either cause i have docker installed and i have followed steps in contributing guidelines


BODY_PARTS_PAF_IDS

Hi @Daniil-Osokin,
I am trying to train on a custom dataset. I know you have probably answered this question a million times and I have read your responses 2x that and I am still confused about how properly set these ids.
Here are my custom ids:
BODY_PARTS_KPT_IDS = [[0, 1], [1, 2], [2, 3], [2, 4], [3, 4]]
BODY_PARTS_PAF_IDS = ([0,1], [1, 2], [2, 3], [2, 4], [3, 4])

What I don't understand is what the PAF_IDS correspond to? for example:
BODY_PARTS_KPT_IDS = [[1, 2], [1, 5], [2, 3], [3, 4], [5, 6], [6, 7], [1, 8], [8, 9], [9, 10], [1, 11],
                      [11, 12], [12, 13], [1, 0], [0, 14], [14, 16], [0, 15], [15, 17], [2, 16], [5, 17]]
BODY_PARTS_PAF_IDS = ([12, 13], [20, 21], [14, 15], [16, 17], [22, 23], [24, 25], [0, 1], [2, 3], [4, 5],
                      [6, 7], [8, 9], [10, 11], [28, 29], [30, 31], [34, 35], [32, 33], [36, 37], [18, 19], [26, 27])

looking at the last element in both ID lists: what does [5, 17] have to do with [26, 27]? There are not even 27 joints? I guess my question is once I have done the first part of training:
python train.py --train-images-folder <COCO_HOME>/train2017/ --prepared-train-labels prepared_train_annotation.pkl --val-labels val_subset.json --val-images-folder <COCO_HOME>/val2017/ --checkpoint-path <path_to>/mobilenet_sgd_68.848.pth.tar --from-mobilenet
What do I do to get my BODY_PARTS_PAF_IDS?
Sorry to ask, but I am really confused about this.

Hi! BODY_PARTS_PAF_IDS list defines indices of network output channels which encodes connection between corresponding keypoints pair (hope you have went through TRAIN-ON-CUSTOM-DATASET.md).

Hi! BODY_PARTS_PAF_IDS list defines indices of network output channels which encode connection between corresponding keypoints pair. So, to group keypoints with 5 and 17 ids in a person part, 26 and 27 channels of the second network output need to be used (hope you have went through TRAIN-ON-CUSTOM-DATASET.md).

Thank you! I will try.
One more question, I can make this a separate issue if that works.
I have been trying to evaluate my model trained with a custom dataset that consists of a entity with a more limited number of joints (only 5). I was able to modify the train code and train the model, but I can't evaluate? I am getting:
AssertionError: Results do not correspond to current coco set
Any idea why this may be happening? I changed:
https://github.com/Daniil-Osokin/lightweight-human-pose-estimation.pytorch/blob/master/val.py#L58 to
keypoints = [0] * 5 * 3
removed:
https://github.com/Daniil-Osokin/lightweight-human-pose-estimation.pytorch/blob/master/val.py#L64-L65
changed https://github.com/Daniil-Osokin/lightweight-human-pose-estimation.pytorch/blob/master/val.py#L73-L75 to:
            keypoints[position_id + 0] = cx             keypoints[position_id + 1] = cy             keypoints[position_id + 2] = visibility
Changed https://github.com/Daniil-Osokin/lightweight-human-pose-estimation.pytorch/blob/master/val.py#L73-L75 to
avg_heatmaps = np.zeros((height, width, 6), dtype=np.float32)# 19
avg_pafs = np.zeros((height, width, 10), dtype=np.float32)# 38

and changed  #https://github.com/Daniil-Osokin/lightweight-human-pose-estimation.pytorch/blob/master/val.py#L174 to:
PoseEstimationWithMobileNet(1, num_heatmaps=6, num_pafs=10)
But I am getting:
Traceback (most recent call last):
  File "val.py", line 216, in <module>
    evaluate(args.labels, args.output_name, args.images_folder, net, args.multiscale, args.visualize)
  File "val.py", line 198, in evaluate
    run_coco_eval(labels, output_name)
  File "val.py", line 21, in run_coco_eval
    coco_dt = coco_gt.loadRes(dt_file_path)
  File "C:\Users\kevin\Anaconda3\lib\site-packages\pycocotools\coco.py", line 327, in loadRes
    'Results do not correspond to current coco set'
AssertionError: Results do not correspond to current coco set

Is this due to the coco evaluator expecting 17 keypoints? Is there a way for me to get this to work with only the 5 kps? Do I keep all 17 and only use 5 of them (although it is different anatomy/kp than coco)

COCO evaluator works with custom annotation, but it has to be correctly formatted. I believe it is due to gold annotation has different format with predicted. Please, check COCO format description. Possibly categories in json contain more keypoints, than needed. You can make some minimal snippet for gold annotation and predicted to check that you have got the format right.

Hope, it helped.


Port the platinum element tests!

Part of https://github.com/dart-lang/polymer_elements/issues/1.
Relatively small number of elements, but it needs to be investigated how well these will even work via js interop. Porting the tests should be a good way to do this though :).

https://github.com/dart-lang/polymer_elements/tree/platinum-tests, unsurprisingly though there are issues with both platinum elements, especially in dartium. Going to put a hold on further work on this for now, and remove these elements completely in the short term.

Being that the tests have been ported I am going to close this issue, and created https://github.com/dart-lang/polymer_elements/issues/12 to track adding the elements back.


[STORM-1426] keep backtype.storm.tuple.AddressedTuple and delete duplicated backtype.storm.messaging.AddressedTuple

+1 Confirmed that back type.storm.messaging.AddressedTuple is not used.

+1 my bad.  Thanks for cleaning this up for me.  I guess I accidentally put two in somehow.


Programs Set-up-Admin-App - Sprint 4

Description:

Build the Screen using the style guide.
Integrate Screen with the Backend.

@askinss I want to work on this issue


golang-google-cicd to 0.0.20

Promote golang-google-cicd to version 0.0.20

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Switch to new custom syntax

change-type: patch

@balena-ci help

@balena-ci retest

@ab77 anything in particular I should know about this PR? As a syntax change, it seems benign.
The bigger issue is the failing tests. It looks like there is a problem with mountutils and/or drivelist packages. They are used by etcher-sdk, which now is on v9.1 while this project still is on v8.7.

@ab77 anything in particular I should know about this PR? As a syntax change, it seems benign.
The bigger issue is the failing tests. It looks like there is a problem with mountutils and/or drivelist packages. I suspect the problem is lack of attention to the code base. Those packages are used by etcher-sdk, which now is on v9.1 while this project still is on v8.7. These failures need to be sorted out first.

@kb2ma I was hoping that you would have more context as to why the tests are failing, I'll try bimping etcher-sdk and see if that passes..


Add support for Github pull request comments

Not to be merged yet, initial support for PHPCS is done... Need to add support for: a) Other tools, b) Other platforms (Bitbucket, etc).

FYI @mavimo for comparison with your Gitlab work.

It seems it also needs work on detecting which diff line it should actually comment on. At the moment it assumes the first line that matches is the one, but for lines like closing braces }, etc. that falls down.

See https://github.com/Block8/PHPCI/pull/803/files?diff=unified for an example.


'EditorGUILayout' does not contain a definition for 'BeginFoldoutHeaderGroup'

Unity 2018.4
'EditorGUILayout' does not contain a definition for 'BeginFoldoutHeaderGroup'
error CS0117: Library\PackageCache\com.dbrizov.naughtyattributes@ed0edab05e0503622f0b6c6b41061d5fb64e3ece\Scripts\Editor\Utility\NaughtyEditorGUI.cs(118,27)

Same here, Unity 2018.4.25f1

Oh, my bad. I merged a pull request for a foldout attribute. I didn't know the function used are not present in 2018.4. Just comment the code until I fix it. I will do it as soon as possible.

Fixed in 777dd313f4dc77cf8c5161f9352be680c2838d98


add support for Retryable Terraform Errors

Fixes #1195

@apeabody Could you also add where we saw those? I forgot if it was something common. For not common or repo specific flakiness, I suggest we pass in needed retries via WithRetryableTerraformErrors option.

@apeabody Could you also add where we saw those? I forgot if it was something common. For not common or repo specific flakiness, I suggest we pass in needed retries via WithRetryableTerraformErrors option.

We are seeing that when there is a project_id collision when project factory tries to create a new
"random" project with a project_id which has previously been used.  Most common with terraform-example-foundation, but could happen anywhere where a new project is created as part of the CI.

@apeabody Could you also add where we saw those? I forgot if it was something common. For not common or repo specific flakiness, I suggest we pass in needed retries via WithRetryableTerraformErrors option.

@bharathkkb We are seeing that when there is a project_id collision when project factory tries to create a new "random" project with a project_id which has previously been used. Most common with terraform-example-foundation, but could happen anywhere where a new project is created as part of the CI.

@apeabody @bharathkkb
This retry will not be able to fix the googleapi: Error 409: Requested entity already exists, alreadyExists.. error because this is an unrecoverable error.
in the case of projects, if we have a project creation failure, the ID of the project the configuration is trying to create will not change and it will fail again because the retry is done upon the current terraform state, not a clean one.


sampling error

When training is completed and sampling is performed, the following error occurs. How can I solve it?
please reply
be_great\great.py:162, in GReaT.sample(self, n_samples, start_col, start_col_dist, temperature, k, max_length, device)
160 # Convert tokens back to tabular data
161 text_data = _convert_tokens_to_text(tokens, self.tokenizer)
--> 162 df_gen = _convert_text_to_tabular_data(text_data, df_gen)
164 # Remove rows with flawed numerical values
165 for i_num_cols in self.num_cols:
be_great\great_utils.py:91, in _convert_text_to_tabular_data(text, df_gen)
89         values = f.strip().split(" is ")
90         if values[0] in columns and not td[values[0]]:
---> 91             td[values[0]] = [values[1]]
93     df_gen = pd.concat([df_gen, pd.DataFrame(td)], ignore_index=True, axis=0)
94 return df_gen
IndexError: list index out of range

Hi,
I'm sorry that you are experiencing issues with our framework.
In order to help you, we need more information about your script and data. Would it be possible to share a sample of your dataset or at least give us some info on it, such as feature names and dataset size (shape) + your code.

Thank you for the reply.
It is difficult to share the entire dataset because it is the dataset from the customer for the project, but the used feature names are as follows.

['Quality', 'Fob', 'Smls + Welded', 'Code', 'Material', 'Gauge', 'Thickness', 'Length', 'Order quantity', 'Unit weight', 'Unit price', 'Approval date']

And the used dataset is a mixture of category type and numerical type, the sample data is as follows.
Since the error does not always occur,
as in the following, if the number of data generated at once is adjusted to a small size, the error may not occur.
samples_025 = great.sample(100, temperature=0.25, k=10, device='cpu')

Also, Is there a case where the generated feature value comes out as a NaN value?
It seems to happen if there is a lack of learning, but I don't think it's certain.
Thank you for your reply!!
-----Original Message-----
From: "Vadim @.>
To: @.>;
Cc: @.>; @.>;
Sent: 2022-11-15 (화) 17:59:59 (GMT+09:00)
Subject: Re: [kathrinse/be_great] sampling error (Issue #4)
Hi,
I'm sorry that you are experiencing issues with our framework.
In order to help you, we need more information about your script and data. Would it be possible to share a sample of your dataset or at least give us some info on it, such as feature names and dataset size (shape) + your code.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Thank you for reporting the issue!
We will fix it, will keep you updated.

as in the following, if the number of data generated at once is adjusted to a small size, the error may not occur.

An interesting observation, you can try to lower the temperature, but it will lead to less diverse synthetic samples... In our experiments, we temperature=0.7 provided the right balance between diversity and stability.
Another question. For how many epochs have you trained GReaT?
@kathrinse, could you please have a look at that?

Thank you!
Please reply when the issue is resolved!!
To check creation, only 1 epoch was trained.
-----Original Message-----
From: "Vadim @.>
To: @.>;
Cc: @.>; @.>;
Sent: 2022-11-15 (화) 20:44:32 (GMT+09:00)
Subject: Re: [kathrinse/be_great] sampling error (Issue #4)
Thank you for reporting the issue!
We will fix it, will keep you updated.
as in the following, if the number of data generated at once is adjusted to a small size, the error may not occur.
An interesting observation, you can try to lower the temperature, but it will lead to less diverse synthetic samples... In our experiments, we temperature=0.7 provided the right balance between diversity and stability.
Another question. For how many epochs have you trained GReaT?
@kathrinse, could you please have a look at that?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

To check creation, only 1 epoch was trained.

That explains the weird error, 1 epoch is not enough for the fine-tuning step. I mean, the language model has to get used to your specific data structure. But I'm amazed that you could sample even after one epoch... We need to try it.
My recommendation is - please train GReaT further.
Of course, it depends on your problem, but I would at least train it for 40-50 epochs. And I'm sure you will get the most realistic synthetic samples for your dataset :)
And regarding the issue, we will introduce a more intuitive error message, so it will be clear that further training is needed or something wrong with the data.
Thank you for reporting it!

The new version of our package should provide a more user-friendly error message.


Android video playback audio works but no video

Hello,
Revisiting this extension after some inactivity and am currently trying to get mp4 video working on android. I have created a Screen that should be playing video. The problem is that the screen remains blank but the audio plays in the background.
Can you provide any guidance? Ideally I want to be able to render video on android onto a specific texture (in an Actor) so that I can put it in a stage and have other Actors rendered on top of the video.
Below is an excerpt of my Screen class:
`public VideoPlayer videoPlayer;
boolean videoLoaded = false;

@Override
public void show() {

    videoPlayer = VideoPlayerCreator.createVideoPlayer();
    videoPlayer.resize(Gdx.graphics.getWidth(), Gdx.graphics.getHeight());
    videoPlayer.setOnVideoSizeListener(new VideoPlayer.VideoSizeListener() {
        @Override
        public void onVideoSize(float v, float v2) {
            videoLoaded = true;
        }
    });

    try {
        videoPlayer.play(Gdx.files.absolute("/path/to/file.mp4"));

    } catch (FileNotFoundException e) {
        e.printStackTrace();
    }

    Gdx.gl.glEnable(GL20.GL_CULL_FACE);
    Gdx.gl.glCullFace(GL20.GL_BACK);

}

@Override
public void render(float delta) {

    Gdx.gl.glClearColor(0, 0, 0, 1);
    Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT);

    if(videoLoaded) {
        if (!videoPlayer.render()) { // As soon as the video is finished, we start the file again using the same player.
            try {
                videoLoaded = false;
                videoPlayer.play(Gdx.files.absolute("/path/to/file.mp4"));
            } catch (FileNotFoundException e) {
                e.printStackTrace();
            }
        }
    }

}`

After further investigation it seems that this problem is device specific. On some devices it can even play two concurrent videos with no problems but on some it is just blank screen with audio in the back ground.

In that case it can only be one of two issues: Either the device does not support the video codec used, or the device does not support the SurfaceTexture that is used for efficient video -> texture conversion

I see. So if I replace SurfaceTexture then that may do the trick?
Also I am currently trying to play videos consecutively by invoking videoPlayer.play(new file) when the current one is completed. The problem is that it causes the entire app to freeze for a little bit and then go back to normal as the next video comes in. How would you suggest I go about doing this in a way that does not freeze the UI thread? Would I be able to create another VideoPlayer object and load the next video (but not invoke mediaPlayer.start()) in a separate thread and then pass the new object to the Screen object?

Replacing SurfaceTexture is hardly possible. However, codecs are the most likely issue anyway, so I would first look into this (Try to see whether the android media player can play the file. If it does, the device does not support SurfaceTexture. If it does not play the video, it is a codec issue).
Gdx-video does not prevent you in making multiple players. However, the android mediaplayer that is used in the background does have some requirements on threading. I would check the android documentation to see what it states exactly about the threading (It's a long time since I implemented the code, so I can't remember the threading details on android anymore).

or the device does not support the SurfaceTexture that is used for efficient video -> texture conversion

Pardon, but how exactly may I handle this particular situation?

If that is the case, there is no easy/clear fix. The problem is that the only alternative would be using the desktop approach with ffmpeg. However, on a mobile phone the performance would instantly become an issue.

Thx for a quick answer!
Any Android update would not fix it? I'm 4.2.2.
If no, would you recommend interfacing native android MediaPlayer as necessity?

Android updates might fix it, given the fact that the video also doesn't work in a different player (try to play it in your device's gallery.
I would say interfacing with the native android MediaPlayer is a necessity, because, unless the video is very small and low quality, the phone's cpu would struggle with decoding, and the bandwidth for texture transfers would be troublesome too. By using the native MediaPlayer, you can use the GPU for decoding, and don't have the extra step of getting the raw image to the GPU.

I played it in a built-in android gallery, no errors, no crashes. When playing via gdx-video, there's only audio and a black screen. Tried also on an emulator of Nexus 5, same.

The emulator is known to not work with gdx-video on certain SDK versions.
However, the fact that the video is playing in the gallery just fine, makes me believe that it is indeed the SurfaceTexture that is not supported. It might be fixed by an android update, however I think it is unlikely.
Honestly, I don't know how to fix this. You might try a different codec for the video, but then again, if it is the SurfaceTexture, it probably won't make a difference.

Actually, I just see the example enabling culling. Could you try to see whether that might be hiding your texture?

https://developer.android.com/guide/topics/graphics/opengl.html
Do you mean this one?

// enable face culling feature
gl.glEnable(GL10.GL_CULL_FACE);

Yes. You could either disable culling, or try Gdx.gl.glCullFace(GL20.GL_FRONT_AND_BACK)

Tried both, separatly, and after that, combined one after another, didn't work.
Also, my log is writing this every render call:
06-18 12:01:09.142 16524-16579/com.quizit.udobox E/emuglGLESv2_enc: glDrawElements: no data bound to the command - ignoring
06-18 12:01:09.208 16524-16579/com.quizit.udobox W/GLConsumer: [SurfaceTexture-100-16524-1] bindTextureImage: clearing GL error: 0x502

Hmmz, don't know what's wrong because the glDrawElements should have data bound.
I will look into this further when I have some time (won't be before at least thursday)

For more details, It works on desktop project, on android one I tried .3gp, .webm and .mp4.
Thanks for help

batch.begin();
if (videoPlayerAndroid!=null && videoPlayerAndroid.isBuffered()) {
videoPlayerAndroid.render();
}

Could you try if the issue persists with the newest snapshot?


react-hot-loader: 4.0.0 -> 4.0.1

I am using mobx with next.js
Sometimes hot reloading fails with an error.
Open stacktrace

Cannot read property '0' of null
TypeError: Cannot read property '0' of null
    at bindDependencies (http://localhost:3000/_next/-/page/index.js:42990:28)
    at trackDerivedFunction (http://localhost:3000/_next/-/page/index.js:42972:5)
    at Reaction.webpackJsonp../node_modules/mobx/lib/mobx.module.js.Reaction.track (http://localhost:3000/_next/-/page/index.js:43164:22)
    at ProxyComponent.reactiveRender [as render] (http://localhost:3000/_next/-/page/index.js:39408:22)
    at finishClassComponent (http://localhost:3000/_next/-/main.js:19188:33)
    at updateClassComponent (http://localhost:3000/_next/-/main.js:19156:12)
    at beginWork (http://localhost:3000/_next/-/main.js:19777:16)
    at performUnitOfWork (http://localhost:3000/_next/-/main.js:22609:16)
    at workLoop (http://localhost:3000/_next/-/main.js:22638:26)
    at renderRoot (http://localhost:3000/_next/-/main.js:22669:9)

That issue was fixed in react-hot-loader 4.0.1

@timneutkens can we please bump react-hot-loader all the way up to v4.1.0? It adds support for React v16.3, currently console is clogged with errors like this when using static getDerivedStateFromProps:

#4171, not going to merge immediately. First landing #4156.


platform/pkt/machine: delete SSH key after provisionning

we defer the key deletion, no matter the result of the provisionning
Signed-off-by: Mathieu Tortuyaux<EMAIL_ADDRESS>How to use

run tests on packet platform

closes https://github.com/kinvolk/Flatcar/issues/236

There already is the Destroy() function of the flight which removes the key. Also, doing it after provisioning the machine is not wanted I guess?
The problem of left-over keys seems to be linked to terminating kola, and the solution is to add key removal to the ore packet gc command (see GC in platform/api/packet/api.go).


Show upload progress

There's a related ticket here:
https://github.com/BD2KGenomics/dcc-storage/issues/21

Moving to dcc-spinnaker-client


Add a Conjur provider test for a var with spaces

This commit adds a test to the Conjur provider to verify it's able to pull
variables with spaces in the name correctly
It also:

Updates the postgres container for conjur to use password authn for access
Fixes the volume mount in the dev container

What ticket does this PR close?
Connected to #1244
Checklists
Change log

[ ] The CHANGELOG has been updated, or
[x] This PR does not include user-facing changes and doesn't require a CHANGELOG update

Test coverage

[x] This PR includes new unit and integration tests to go with the code changes, or
[ ] The changes in this PR do not require tests

Documentation

[x] This PR does not require updating any documentation, or
[ ] Docs (e.g. READMEs) were updated in this PR, and/or there is a follow-on issue to update docs

(For releases only) Manual tests

[ ] Manually tested Keychain provider
[ ] Manually run Kubernetes-Conjur demo with a local Secretless Broker image build of your branch

@sgnn7 the PR is updated if you can re-review :)


Visualization: having a cmap_name in plot_results kwargs is restrictive.

While working on a recent assignment for 8250, I noticed that the cmap_name in the plot_results function kwargs is overly restrictive.
My use case:
Imagine a user wants to use a custom colormap to visualize model runs, with blues corresponding to one set of seeded runs, reds to another, and so on.
This is not possible in the current code, which only allows use of default matplotlib cmaps via their string name.

Yes this is true. The current visualization code is an early version. We can update the function to instead take in a function that takes in an index and return a color. Would that work for your use case?

six weeks later, I do not remember the full syntax, but your solution sounds non-standard...
One alternative approach you might consider for all your visualization functions is adding an optional matplotlib Axes object argument, ex. plot_func(...,ax=None). By default, the plotting functions would generate a new Axes object on which to plot data, but they would use the user's input ax if overriden.
This approach would make solving the original problem as simple as modifying the cmap_name arg to a more general cmap argument with expected type an actual Matplotlib colormap (or recognized string name). I could have then plotted my custom colormaps on the same Axes object by passing to the plot_results function multiple times for different sets of run data.
This approach is also standard in other plotting libraries, for instance Seaborn IIRC.

I see, that makes sense, thanks for the suggestion! I'll update this thread when I create the PR for the changes.


feat: multiple authorizers

Allows to accept token signed by one of multiple issuers.

building AuthorizationLayer with a vector or a slice of one or more authorizers
backward compatbility (though JwtAuthorizer::layer is deprecated, and replaced by into_layer())
tests

fixes: #14

Is this something being driven further? Solves my issue https://github.com/cduvray/jwt-authorizer/issues/30


image to pdf

Pull Request Template
script name -
Brief about script
It converts images inside a folder("/images") to a pdf file
Issue no.(must) -
Self Check(Tick After Making pull Request)

[x] This issue was assigned to me.
[ ] One Change in one Pull Request
[x] My file is in proper folder (Name of folder should be in lowercase with no space in between) (E.g. meet_schedular)
[ ] I am following clean code and Documentation and my code is well linted with flake8.
[ ] I have added README.md and requirements.txt (Include version numbers too e.g. pandas==0.0.1) with my script
[ ] Just including required dependencies in requirements.txt (Don't include Python version too)

If issue was not assigned to you Please don't make a PR. It will marked as invalid.

Hey first a raise a issue of this

Hey first a raise a issue of this

i have added the issue :)

Hey first a raise a issue of this

i have added the issue :)

now fix the linting errors

sorry for the linting error
its fixed now
:D

Hey first a raise a issue of this

i have added the issue :)

now fix the linting errors

Now add README.md to brief describe the purpose, how to use, and you're done :)

That's neat now :)

thank you :)


read function should only get AYON parameters

Is there an existing issue for this?

[x] I have searched the existing issues and added correct labels.

Description
Current Behavior
ayon_houdini.api.lib.read should get extra AYON parameters.
Houdini Nodes can include different type of parameters and some of them like ramp breaks the instances collection as copy.deepcopy fails.
Here's some bug report when it fails
cannot pickle 'SwigPyObject' object
Traceback (most recent call last):
  File "E:\Ynput\ayon-core\client\ayon_core\pipeline\create\context.py", line 1395, in reset_instances
    creator.collect_instances()
  File "E:\Ynput\ayon-houdini\client\ayon_houdini\api\plugin.py", line 193, in collect_instances
    node_data, self

  File "E:\Ynput\ayon-core\client\ayon_core\pipeline\create\structures.py", line 800, in from_existing
    instance_data = copy.deepcopy(instance_data)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\PROGRA~1\SIDEEF~1\HOUDIN~1.370\python311\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\PROGRA~1\SIDEEF~1\HOUDIN~1.370\python311\Lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\PROGRA~1\SIDEEF~1\HOUDIN~1.370\python311\Lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\PROGRA~1\SIDEEF~1\HOUDIN~1.370\python311\Lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\PROGRA~1\SIDEEF~1\HOUDIN~1.370\python311\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\PROGRA~1\SIDEEF~1\HOUDIN~1.370\python311\Lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\PROGRA~1\SIDEEF~1\HOUDIN~1.370\python311\Lib\copy.py", line 161, in deepcopy
    rv = reductor(4)
         ^^^^^^^^^^^
TypeError: cannot pickle 'SwigPyObject' object

when checking the instance data, it shown something like
'color_ramp': <hou.Ramp is_color=True num_keys=2 data=((t=0, rgb=(0, 0, 0)), (t=1, rgb=(1, 1, 1)))>,

Expected Behavior
read function should only get AYON parameters
Steps To Reproduce:

Create an HDA product
Add a ramp parameter
open the publisher tool or click refresh button,

Additional context:
Version
latest everything
What platform were you running when you found the bug?
Windows
You can also provide any additional information relevant to the issue.
Relevant log output:

Yes - for the containers we already did this, but we should also for creators. (It may also make them faster!)
So we would be getting way more explicit about the attributes we're expecting. However, currently those are quite a lot already - especially with backwards compatibility:
# creators
keys = [
    "folderPath",
    "task",
    "creator_identifier",
    "id",
    "AYON_productName",
    "productName",
    "creator_attributes",
    "publish_attributes",
    "variant",
    "active",
    # legacy attributes
    "asset",
    "subset"
]

And I might still be missing some?


Problem with spline derivatives in a Parametric Spline Reprsentation

When I construct a Parametric spline representation (Vadym Pasko), the spline works, but the derivative method seems broken. Please see code and error below. Are there any work arounds? Thanks for your help.
import numpy as np
import matplotlib.pyplot as plt
import scipy.interpolate as si

#####  CONTROL POINTS   #####
# Closed symmetric polygon

x = [6, -2, 4, 6, 8, 14, 6]
y = [-3, 2, 5, 0, 5, 2, -3]

xmin, xmax = min(x), max(x)
ymin, ymax = min(y), max(y)

n = len(x)
plotpoints = 100

#####  PARAMETRIC SPLINE REPRESENTATION  #####

# n is number of contol points

k = 3
knotspace = range(n)

# find knot vector

knots = si.InterpolatedUnivariateSpline(knotspace, knotspace, k=k).get_knots()
knots_full = np.concatenate(([knots[0]]*k, knots, [knots[-1]]*k))

# define tuples of knot vector, coefficient vector
# (control points coordinates) and spline degree

tckX = knots_full, x, k
tckY = knots_full, y, k

# construct spline functions
splineX = si.UnivariateSpline._from_tck(tckX)
splineY = si.UnivariateSpline._from_tck(tckY)

splineX.derivative()

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [7], in <cell line: 1>()
----> 1 splineX.derivative()

File ~/miniconda3/lib/python3.8/site-packages/scipy/interpolate/_fitpack2.py:513, in UnivariateSpline.derivative(self, n)
    469 def derivative(self, n=1):
    470     """
    471     Construct a new spline representing the derivative of this spline.
    472
   (...)
    511
    512     """
--> 513     tck = _fitpack_py.splder(self._eval_args, n)
    514     # if self.ext is 'const', derivative.ext will be 'zeros'
    515     ext = 1 if self.ext == 3 else self.ext

File ~/miniconda3/lib/python3.8/site-packages/scipy/interpolate/_fitpack_py.py:698, in splder(tck, n)
    696     return tck.derivative(n)
    697 else:
--> 698     return _impl.splder(tck, n)

File ~/miniconda3/lib/python3.8/site-packages/scipy/interpolate/_fitpack_impl.py:1211, in splder(tck, n)
   1207     raise ValueError(("Order of derivative (n = %r) must be <= "
   1208                       "order of spline (k = %r)") % (n, tck[2]))
   1210 # Extra axes for the trailing dims of the `c` array:
-> 1211 sh = (slice(None),) + ((None,)*len(c.shape[1:]))
   1213 with np.errstate(invalid='raise', divide='raise'):
   1214     try:

AttributeError: 'list' object has no attribute 'shape'

Hi @deburgess, thank you for reporting. There is no error here. You are using a private function _from_tck (indicated by the use of the leading _). Hence we don't provide the same convenience as public API functions and don't check for shape and type of parameters.
In this case x and y needs to be NumPy arrays to work.
I don't think there is anything to do here.


you can not use ariang because this browser does not meet

Brave browser latest version (Chrome 80)

My browser issue. close it


Can't able to play marker on path

Script was working fine before.
But from yesterday, it stops working.
Issue description:
On page load, a map marker is not playing on path and also that features of play, pause, reset, etc not working.
In console, getting below error.
Please insert valid location Array
Uncaught TypeError: Cannot read property 'play' of null
at TravelMarker.play (travel-marker.umd.js:969)
at play ((index):168)

Hi,
The problem was in getting the location array from Google Polyline. I changed the
line.getPath().j;

to
line.getPath().getArray();

Even examples are working fine now.

Yes. It's working fine now.
Thank you so much for your help.


How do you pass in the relative path to the file object?

#285 Wondering if there's a simplified way to pass in the relative path to a file object. I know that I can create a field file._relativePath, but it seems like this variable does not work by itself and I would have to receive it from chrome like this, http://stackoverflow.com/questions/3590058/does-html5-allow-drag-drop-upload-of-folders-or-a-folder-tree? This is less an issue with the project than a question of my experience but I'm wondering how would you be able to combine the two together to get the path for each file in relativePath? An example, link, or documentation beyond 285 to make this easier would really be appreciated.

You can get the relative path by file.path from ngf-drop dropped files.

Thank you. Your suggestion worked.


updated baf2sql to 2.9.0

updated baf2sql to 2.9.0 (should fix some "unknown compressor id" errors)

Closes #1891

This breaks BAF support on WINE apparently, but I tested with different WINE versions and was able to get it working. So I'll be updating WINE before deploying this patch to DockerHub. Probably to 7.0 once it's official (currently in release candidate stage). I'll hold off merging this until then to avoid making this test fail on other PRs.

/rebase


cicero-tui: 0.3.0 -> 0.4.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/eyeplum/cicero-tui/releases.
meta.description for cicero-tui is: "Unicode tool with a terminal user interface"
meta.homepage for cicero-tui is: "https://github.com/eyeplum/cicero-tui"
meta.changelog for cicero-tui is: ""
Updates performed

Rust version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.4.0 with grep in /nix/store/23p1cwh1cf8pbaw1pdgvivkmqjj90vqs-cicero-tui-0.4.0
found 0.4.0 in filename of file in /nix/store/23p1cwh1cf8pbaw1pdgvivkmqjj90vqs-cicero-tui-0.4.0

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
cicero-tui

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/23p1cwh1cf8pbaw1pdgvivkmqjj90vqs-cicero-tui-0.4.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A cicero-tui https://github.com/r-ryantm/nixpkgs/archive/42bd265b8e88d3007171c2b101a935a3c6044442.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/23p1cwh1cf8pbaw1pdgvivkmqjj90vqs-cicero-tui-0.4.0
ls -la /nix/store/23p1cwh1cf8pbaw1pdgvivkmqjj90vqs-cicero-tui-0.4.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

cicero-tui

Maintainer pings
cc @SCOTT-HAMILTON for testing.

Result of nixpkgs-review pr 191754 run on x86_64-linux 1

1 package built:

cicero-tui


Store the string lengths on the string tape

This PR adds a 32-bit header (as an unsigned int) to each string being written on the tape. For convenience, the strings are still null terminated.
Upsides:

Users can get the string length without computation.
We get to support strings with null characters in them (which is apparently something very important to some).

Downsides:

There is a small performance degradation (probably always less than 5%). Some of this degradation is due to the extra work we do (computing the string length, storing it). Some of it might also be related to the write patterns. That is, we write on the string tape the string, then once we are done, we go back at the start of the string to write the string length. That might not be best. It is also possible that the increase size of the tape is hurting us (we write more data and so there are more data stores and loads, more stress on the memory subsystem).
We increase the memory usage quite a bit... these 32-bit counters are not free.

Would solve issue https://github.com/lemire/simdjson/issues/87
See
https://github.com/lemire/simdjson/pull/89
For discussion.

cc @EgorBo @TkTech
I am especially interested in Egor's since he has a competing PR.

Note: This is not really a competing PR to @EgorBo's PR. It is best defined as a variant/revision.

@geofflangdale : given that this comes with measurable downsides, we should not merge this without proper consideration.

I tried a few things, but I think that the penalty we pay is probably not going to get resolved easily. It is small penalty, however (I expect it is always less than 5%).
Here are some numbers of my iMac (that machine is not configured for testing and there is quite a bit of noise in the measurements).  I don't want to assess my own work and its effects. I'd prefer to have an objective take on it from someone else.

I'll run the benchmarks in few hours to check.
Btw, I've added an alternative C# version - simple bindings for the native lib

C bindable glue (I am using singleheader stuff) - https://github.com/EgorBo/SimdJsonSharp/blob/master/src/BindingsForNativeLib/SimdJsonNative/bindings.cpp
C# wrappers - https://github.com/EgorBo/SimdJsonSharp/blob/master/src/BindingsForNativeLib/SimdJsonSharp.Bindings/Bindings.cs
Makefile

The lib is .NETStandard 2.0 - SimdJsonSharp.Bindings and contains native libs from Win64 and osx-64.

Checked on Windows (Core i7 8700) I see from 0.5% to 3.5% regression (avg for test data is about to 1%), the 3.5% value was for canada.json file but it significantly improves string conversion in C#.

@EgorBo That’s a helpful comment.

I have reworked the PR by moving some of the expensive logic in string processing to stage 1. Except for documents containing few strings, this is a net win and makes this PR faster than 'master'. The gain is not so small in some cases so this is worth consideration.
 More work in the domain of stage 1 fits well with the overall vision here - we will be able to speed this up with future SIMD improvements (e.g. AVX512 or SVE).
Storing the string lengths is also a substantial usability bump and resolves #40 (and #87).

 Before I merge this... I need people to be aware that...
We are now rejecting documents larger than 4GB. That's fixable later but we are introducing a hard limit on string lengths (4GB) and the best way that I can see to enforce this is to reject documents larger than 4GB (thus avoiding the issue entirely).

I am going to merge. We can sort out the 4GB issue later if someone is upset.


Update macbook conda

adding napari_cellseg3D to the conda
removing un-used packages
note that it works for m2, m3 chips too

@C-Achard in the docs after install, if would be really great to either say (1) how to launch the app, or (2) link to the page that does so! For example, I have an env built, and if I want to just run "inference," if i jump to that tab it doesn't tell me how to launch the app ;)

@C-Achard in the docs after install, if would be really great to either say (1) how to launch the app, or (2) link to the page that does so! For example, I have an env built, and if I want to just run "inference," if i jump to that tab it doesn't tell me how to launch the app ;) so I would link to: https://adaptivemotorcontrollab.github.io/CellSeg3d/welcome.html#usage

Will do, thanks ! :D

in docs @C-Achard you might consider adding a note on how to update an env, deactivate first, be in the root project folder, and run conda env update --f conda/napari_cellseg3d_m1.yml

Done !

tested on my M2 chip; all works (aside from I modified to keep "old name" napari-cellseg3d, but once 0.2.0 is merged this will work.


[DO-NOT-MERGE] use starlark.SimpleFilter to not touch the id annotations

Depending on https://github.com/kubernetes-sigs/kustomize/pull/4059
fixes: https://github.com/GoogleContainerTools/kpt/issues/2382

Relevant: https://github.com/kubernetes-sigs/kustomize/pull/3995
This will be the general way all function implementations need to be implemented

Superseded by #473


Mapping-collection: Add mappings for d5005-matrix-mult-orchestrated

This mapping will be used in the new demo screencast for FPGA plugin deployment in orchestrated mode.

Codecov Report

Merging #412 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #412   +/-   ##
=======================================
  Coverage   58.89%   58.89%
=======================================
  Files          26       26
  Lines        2428     2428
=======================================
  Hits         1430     1430
  Misses        940      940
  Partials       58       58

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8b4ca4a...9cdb9a1. Read the comment docs.


Curated but searchable/extensible dropdown menus

If you don't mind me popping in here, I'd like to table an alternative approach that came to mind when reading the motivating issue for this PR. It may be applicable/better suited to other scenarios.
Simplifying the drop-list is a good idea, but it doesn't necessarily require removing GUI access to the full suite. A '...' option could be appended to the menu - clicking this would present any remaining options; clicking again would collapse back to the initial subset. This approach is motivated by a quote from Alan Kay and can be taken further...
Interpolation methods that are parameterised could allow for tuning within the GUI. But, in a similar vein as above, this functionality wouldn't be presented to the user by default. A button ('...' or an icon of analogue dial) placed adjacent to the drop-list would expand/collapse the available parameters. If the selected method is non-parameterised, this button would be faded out.
On the informational side of things, a similarly behaving button (the classic 'information i') could toggle the display of explanatory notes and/or links. The content could be sourced from the corresponding docstring, or from some common resource shared by the docstring.

For each idea, feel free to take it or leave it - no worries either way.
If any take your fancy, let's talk implementation. I haven't worked with PyQt/Qt before - do you think these kind of features would be approachable for me to implement? At least the extensible drop-list to begin with.

A couple of misc. thoughts that personally motivate this approach:

Features that are hidden or inaccessible for a user are less likely to be tested 'in the wild'.

I've found that the topic of interpolation (and the methods involved) is often laced with subtlety. Allowing a user to surface some of the details involved could help them get a better grasp of how things work/which methods are appropriate for which situation.

Originally posted by @ambientnuance in https://github.com/napari/napari/issues/2552#issuecomment-841909363

An interesting idea @ambientnuance! In the past we have discussed searchable comboboxes where you could search more than what is displayed by default, e.g. so you could search for any matplotlib colormap, not just the ones we have pre-selected. I think @Czaki had done some work with that, but I can't find a reference to that code.
Let's keep the discussion in this new issue rather than in a merged PR. 😊

If I good remember the option of search is not included in the final version of PR.
Here is a sample implementation of the searchable combo box https://github.com/4DNucleome/PartSeg/blob/2129d17b53a0609d7e39c3af3750e6c1a23a7b86/package/PartSeg/common_gui/searchable_combo_box.py I'm not sure if it could be easy extended to requested version but it may help you.

Thanks for opening the new issue @jni, and @Czaki for the sample code.
Here is a modification demonstrating the expand/collapse drop-down list.
import sys
from PyQt5.QtCore import Qt
from PyQt5.QtWidgets import QApplication, QWidget, QComboBox, QCompleter, QVBoxLayout

class SearchCombBox(QComboBox):
    """
    ComboCox with completer for fast search in multiple options.
    The item list also has expandable functionality.
    """
    def __init__(self, parent=None):
        super().__init__(parent)
        # completer stuff
        self.setEditable(True)
        self.completer_object = QCompleter()
        self.completer_object.setCaseSensitivity(Qt.CaseInsensitive)
        self.setCompleter(self.completer_object)
        # initial items
        self.items = ['one', 'two', 'three', 'four', 'five', '...']
        self.addItems(self.items)
        # expansion items
        self.expansion_items = ['aye', 'bee', 'cee']
        self.is_expanded = False

    def addItem(self, *args):
        super().addItem(*args)
        self.completer_object.setModel(self.model())

    def addItems(self, *args):
        super().addItems(*args)
        self.completer_object.setModel(self.model())

class Example(QWidget):
    """
    Wrapper widget (comobo box on its own didn't want to play nice)
    """
    def __init__(self):
        super().__init__()
        # combo box
        self.combo_box = SearchCombBox(self)
        self.combo_box.activated[str].connect(self.box_selection)
        #                        ^ returns selected item as a string
        # layout
        v_layout = QVBoxLayout()
        v_layout.addWidget(self.combo_box)
        self.setContentsMargins(50, 50, 50, 50)
        self.setLayout(v_layout)

    def box_selection(self, txt: str):
        """
        Hook/callback function for handling list expansion.
        ^ (not sure what the terminology in Qt is for this)
        """
        box = self.combo_box
        if txt == '...':
            if not box.is_expanded:
                # expand
                box.addItems(box.expansion_items)
            else:
                # collapse
                end_index = len(box.items)
                for _ in box.expansion_items:
                    box.removeItem(end_index)

            box.is_expanded = not box.is_expanded
            box.showPopup()  # keeps the list open after an expansion or collapse

if __name__ == '__main__':
    app = QApplication([])
    example = Example()
    example.show()
    sys.exit(app.exec())

I'd certainly be happy to have a widget like this in the repo.  I'm still not too concerned about the interpolation case specifically: since that interpolation option is controlling a purely visual thing.  That is, if people could actually use the interpolated image for analysis, I'd be more interested in the subtle differences between the various interpolation modes...
but that's sort of beside the point here.  I like this widget!

Hi @ambientnuance, great to see you here after PyCon!
I think a general purpose widget like this would be great. We can start with a contribution here, though I'm wondering long term if we might pull it out to be it's own thing (like we're doing with our range slider and https://github.com/tlambert03/QtRangeSlider)
There might be some ideas in https://github.com/napari/napari/issues/268 that are related to this.
If you could add a screnshot/ movie of what you've got so far that could really help the discussion too. Thanks!!

@ambientnuance this is awesome! Here's a quick video for all...
https://user-images.githubusercontent.com/7307488/118655301-536b5600-b7e1-11eb-8dda-921e800b267c.mp4
I second what @sofroniewn said about having this available as its own widget outside of napari too, it's generally useful!

Since it seems we might have a few useful widgets coming in the future (beside the existing ones), would it make sense to have a package that bundles them? e.g. napari-widgets napari-qtwidgets

Rightio, I've put together a subclass of ComboBox that handles expansion, completion and a history of previous selections. It's called 'ExpandoBox'.
https://github.com/ambientnuance/pycuties (gif is laggy compared to actual)
https://pypi.org/project/pycuties/
Only a couple of minor quirks are left to tackle - hypothesis did a good job of uncovering edge-cases. It's a tool that I'd been meaning to wrangle with for a while, so this was a good sandbox project to play with it. I'm glad that it lived up to its appeal, even if the gaps between introductory and intermediate documentation could do with some more bridges.
There were a few false starts with getting Qt to co-operate with modifying out-of-the-box behaviour. Thankfully, solutions that were simple rather than complex did the trick. Mapping each problem to the Qt documentation was another story. So many functions and objects with no description, just names that were 60% precise, 40% lingo.
The most striking issue for me was the absence (or camoflauge) of a clear model for the composition of a ComboBox. ItemModel + ListView + LineEdit != ComboBox, at least in their vanilla flavours. The missing pieces are a drop-down button and line edit icon, which seem buried in QStyle rendering code at my best guess (perusing https://code.woboq.org/qt5/qtbase/src/widgets/widgets/qcombobox.cpp.html). Specifically, it's unclear what object to point a qbot fixture (pytest-qt) at to trigger the dropdown menu. Or, how to hide the LineEdit icon without replacing a drawing object. Non-ideal workarounds suffice for now, but any insight or ideas are most welcome.
The interpolation options being only for display does save the need for further detail.

Anywho, stepping out of the technical trenches...
Cheers to all of you for the warm reception, including @sofroniewn who catered to my curiosity at PyCon.
Napari is a nifty workspace for array interaction; it's been neat to see a shout out or two from scientists on twitter. A good sign of a snowball getting rolling.


[question] Request/Reply from many queued subscribers

Hi,
Is there a way to implement a publisher expecting a reply from all queued subscribers?
example:
SUB_A1 = nc.subscribe("hello", "a", cb=message_handler_a)
SUB_A2 = nc.subscribe("hello", "a", cb=message_handler_a)
SUB_B1 = nc.subscribe("hello", "b", cb=message_handler_b)
SUB_B2 = nc.subscribe("hello", "b", cb=message_handler_b)
response = await nc.request("hello", b'hi', 0.2)
-> response contains response one from SUB_A and one from SUB_B
Thanks

You can do it with a normal publish, but not with ns.request. I have a few examples, here is one where we use a hardcoded reply subject and wait for multiple responses, you should probably use a more random response subject.
https://github.com/ConnectEverything/solvingproblems_java/blob/master/src/main/java/solutions/ex18_search/Search.java


[Bug] Task that encounters inconsistent data records number fast fail

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the bug
When using the latest master branch code into production env, I found some tasks failed due to the consistent record number.
After digging this,  I think this is caused by the #1670
Affects Version(s)
master
Uniffle Server Log Output
No response
Uniffle Engine Log Output
No response
Uniffle Server Configurations
No response
Uniffle Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

@zuston Can you please add a new PR to branch-0.9? I think it is a critical fix. But can not cherry-pick to branch-0.9.

#1670 is not merged to branch-0.9, remove this from 0.9.1 milestone.


[FEATURE REQUEST]

Description
Is your feature request related to a problem? Please describe.
Add the new logo on all pages
Describe the solution you'd like
A clear and concise description of the feature or enhancement you would like to see implemented.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional Information
Context
Provide any additional context or information that helps explain the feature request.
Potential Implementation (optional)
If you have any ideas or suggestions on how the feature could be implemented, please share them here.
Screenshots or Mockups (optional)
If applicable, include any screenshots, mockups, or visual representations of the requested feature.
Priority (optional)
Please indicate the priority level of this feature request (e.g., High, Medium, Low).
Additional Notes (optional)
Add any other notes or comments about the feature request here.

I would like to work o. This issue under gssoc'23


Support finetuning of models like google/madlad400-10b-mt and facebook/seamless-m4t-v2-large

Currently I get an error NotImplementedError: Unsloth: facebook/seamless-m4t-v2-large not supported yet!.

Hmmm I'm unsure if we'll support this sorry


Add information about submitter to submission details page

Add a section in the Submission details page to display name of the user who is associated with the sumission.

display display name with( institutional id )
eg: Hanh Vu (hvu13)


pythonPackages.rpy2: 3.2.6 -> 3.3.6, fix test etc.

Motivation for this change
This change updates rpy2 to the latest version and rebases a local patch to that version.
Fixes https://github.com/NixOS/nixpkgs/issues/81345.
Fixes https://github.com/NixOS/nixpkgs/issues/100259.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[x] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/359

Could this be reviewed please :)

This builds for me on NixOS 20.09, and appears to work as expected, based on https://rpy2.github.io/doc/v3.3.x/html/introduction.html#graphics-and-plots

@yurrriq Thank you for testing!
ping? @joelmo

Fails to build for me on linux-x86_64:
unpacking sources
unpacking source archive /nix/store/kckxwy4ngbarm8dnls31h6khbhyj1yc0-lazyeval_0.2.2.tar.gz
source root is lazyeval
setting SOURCE_DATE_EPOCH to timestamp<PHONE_NUMBER> of file lazyeval/MD5
patching sources
configuring
building
running tests
installing
Error: package or namespace load failed for 'utils':
 .onLoad failed in loadNamespace() for 'utils', details:
  call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
  error: cannot popen '/nix/store/nsqal2vl76faah1hs5i8h8glzhlidbp7-which-2.21/bin/which 'uname' 2>/dev/null', probable reason 'Cannot allocate memory'
Error: package or namespace load failed for 'stats':
 .onLoad failed in loadNamespace() for 'utils', details:
  call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
  error: cannot popen '/nix/store/nsqal2vl76faah1hs5i8h8glzhlidbp7-which-2.21/bin/which 'uname' 2>/dev/null', probable reason 'Cannot allocate memory'
During startup - Warning messages:
1: package 'utils' in options("defaultPackages") was not found
2: package 'stats' in options("defaultPackages") was not found
* installing *source* package 'lazyeval' ...
** package 'lazyeval' successfully unpacked and MD5 sums checked
** using staged installation
Error : .onLoad failed in loadNamespace() for 'utils', details:
  call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
  error: cannot popen '/nix/store/nsqal2vl76faah1hs5i8h8glzhlidbp7-which-2.21/bin/which 'uname' 2>/dev/null', probable reason 'Cannot allocate memory'
ERROR: installing package DESCRIPTION failed for package 'lazyeval'
* removing '/nix/store/d18q27vnxzxjki13wynzpf9a249ccpyb-r-lazyeval-0.2.2/library/lazyeval'
Warning in q("no", status = status, runLast = FALSE) :
  system call failed: No such file or directory
builder for '/nix/store/p5r2j6hngwj3bpkjc91x5dsn0cxd7vgc-r-lazyeval-0.2.2.drv' failed with exit code 1
cannot build derivation '/nix/store/lf1q5mhi298g0c9khi72fkfsalm7vmrw-python3.7-rpy2-3.3.6.drv': 1 dependencies couldn't be built
error: build of '/nix/store/lf1q5mhi298g0c9khi72fkfsalm7vmrw-python3.7-rpy2-3.3.6.drv' failed

Thank you for testing, @SuperSandro2000!
It seems that rPackages.lazyeval failed for some reasons. Would you make a new issue for the package, after nix-building it?


登陆失败

thread 'main' panicked at '登陆失败: error sending request for url (https://forums.e-hentai.org/index.php?act=Login&CODE=01): error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
Caused by:
0: error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
1: dns error: failed to lookup address information: Temporary failure in name resolution
2: failed to lookup address information: Temporary failure in name resolution', src/exhentai.rs:33:51

Ubuntu 22.04.1 LTS
外网，没开代理

用其他机器结果是
thread 'main' panicked at '登陆失败: HTTP status client error (403 Forbidden) for url (https://forums.e-hentai.org/index.php?act=Login&CODE=01)', src/exhentai.rs:33:51
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

用第一个机器给第二个机器提供s5代理
外加cookie登陆
第二个机器能用了，不过还是感觉好迷