Fix partition info in segment ZK metadata for changes in stream partition count

Description
This PR handles the bug described in issue #11475 .
Testing Done
Verified the correct behavior by changing LLCRealtimeClusterIntegrationTest:

created a Kafka topic with 2 partitions
created a table with partitioning enabled (partition count was set as 6 to not match the stream partition count)
published 4 events with id's [0, 1, 2, 3] to the topic
verified that partitioning info in segment ZK metadata is correctly set based on 2 partitions, not 6
increased the Kafka topic partition count to 4
published 4 more events with id's [4, 5, 6, 7]
at this point, there's only two consuming segments. Some of the new events went to the existing consuming segments, for some there's no consuming segment available.
kicked off RealtimeSegmentValidationManager periodic job to create new consuming segments
verified that for the new consuming segments, numPartition field in segment ZK metadata is correctly set as 4, while for the existing consuming, the value is 2.
queried the table with different id's in where clause to make sure that partition pruning / routing works as expected
issued a force-commit
verified that partition info for committed segments as well as new consuming segments are correctly set

Note: I think it's best if later we put all the above steps in a separate integration test, so we make sure the expected behavior doesn't change with future refactoring.

Codecov Report

Merging #11476 (4368028) into master (f14700a) will decrease coverage by 48.50%.
Report is 4 commits behind head on master.
The diff coverage is 25.00%.

@@              Coverage Diff              @@
##             master   #11476       +/-   ##
=============================================
- Coverage     62.92%   14.43%   -48.50%
+ Complexity     1108      201      -907
=============================================
  Files          2318     2319        +1
  Lines        124328   124396       +68
  Branches      18980    18993       +13
=============================================
- Hits          78234    17951    -60283
- Misses        40539   104927    +64388
+ Partials       5555     1518     -4037

Flag
Coverage Δ

integration
0.00% <0.00%> (-0.01%)
:arrow_down:

integration1
?

integration2
0.00% <0.00%> (ø)

java-11
14.42% <25.00%> (-48.49%)
:arrow_down:

java-17
14.42% <25.00%> (-48.35%)
:arrow_down:

java-20
0.00% <0.00%> (-62.79%)
:arrow_down:

temurin
14.43% <25.00%> (-48.50%)
:arrow_down:

unittests
14.43% <25.00%> (-48.50%)
:arrow_down:

unittests1
?

unittests2
14.43% <25.00%> (-0.01%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed
Coverage Δ

.../core/realtime/PinotLLCRealtimeSegmentManager.java
60.17% <25.00%> (-0.18%)
:arrow_down:

... and 1496 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

@sajjad-moradi, is this supposed to fix consuming segments? we'd seen this before, but when going from CONSUMING -> COMPLETED, the right partitions were then stored in the ZKMetadata

@sajjad-moradi, is this supposed to fix consuming segments? we'd seen this before, but when going from CONSUMING -> COMPLETED, the right partitions were then stored in the ZKMetadata

Yes, this is for consuming segments. It usually takes hours (depending on stream configs) for consuming segment to start committing. Without the fix, consuming segments for new partitions have wrong ZK metadata => which results in incorrect partition routing => which results in incorrect query responses

This only works when the new segments are created, right? Consuming segments during repartition will still be incorrect until synced?

This is for fixing the metadata of the new consuming segments for the new partitions. The consuming segments for existing partitions keep their existing partitioning metadata. Depending on how you repartition, that might be correct or incorrect. For example, in Linkedin, any partition change is a factor of two. With this repartitioning approach, the existing segment metadata will result in correct partition routing.
Let say there are two partitions and 4 ids in total for the partitioning column. So before repartitioning, here are segment ZK metadata for the two partitions:
p0 -> numPartitions: 2, partitionIds = [0] (corresponding to id's 0, 2)
p1 -> numPartitions: 2, partitionIds = [1] (corresponding to id's 1, 3)
After repartitioning, two new partitions are added, but the existing consuming segment keep their existing metadata:
p0 -> numPartitions: 2, partitionIds = [0] (corresponding to id's 0, 2)
p1 -> numPartitions: 2, partitionIds = [1] (corresponding to id's 1, 3)
p2 -> numPartitions: 4, partitionIds = [2] (corresponding to id 2)
p3 -> numPartitions: 4, partitionIds = [3] (corresponding to id 3)
After the next segment commits, consuming segment zk metadata are like these:
p0 -> numPartitions: 4, partitionIds = [0] (corresponding to id 0)
p1 -> numPartitions: 4, partitionIds = [1] (corresponding to id 1)
p2 -> numPartitions: 4, partitionIds = [2] (corresponding to id 2)
p3 -> numPartitions: 4, partitionIds = [3] (corresponding to id 3)
This shows that with "factor or two" repartitioning approach, the partition routing is correct before, during, and after repartitioning.
If you're using different repartitioning, you need to do a force-commit right after repartition happens. This way you can significantly reduce the duration when partition aware routing is problematic.


directly queue control frames from the stream

Currently, the sending WINDOW_UPDATEs works like this: When data is Read from a stream, we call the onData() callback, which schedules the sending of a new packet, which then will go through all the streams and get the WINDOW_UPDATEs.
This is not optimal for two reasons:

We're calling onData() for every Read, even if no WINDOW_UPDATE needs to be sent. This will result in unnecessary attempts to send an empty packet.
For every time we send a packet, we're going through all open streams and ask them if they have window updates to send.

Note that 1. can trivially be fixed by asking the flow controller if there are window updates before calling onData().
My suggestion for fixing this is the following:
We add a callback to the stream: queueControlFrame(wire.Frame). This callback allows the stream to pass any control frame to the session (which would then pass it on to the packetPacker). The callback would be used to queue WINDOW_UPDATEs, BLOCKED frames as well as RST_STREAM frames.
When packing a packet, there's no need to ask every stream for WINDOW_UPDATEs and BLOCKED_FRAMEs (we only need to ask the connection flow controller for the corresponding connection-level frames). This will solve both issues mentioned above.

SGTM


[SPAKR-35713]Bug fix for thread leak in JobCancellationSuite

What changes were proposed in this pull request?
Bug fix for thread leak in JobCancellationSuite UT
Why are the changes needed?
When we call Thread.interrupt() method, that thread's interrupt status will be set but it may not really interrupt.
So when spark task runs in an infinite loop, spark context may fail to interrupt the task thread, and resulting in thread leak.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Test case "task reaper kills JVM if killed tasks keep running for too long" in JobCancellationSuite

Can one of the admins verify this patch?

@mridulm @Ngone51
Thanks for reviewing.
But I don't think there are some scenes that need this kind busy loop?

Some scenes may not need but this unit test needs it, right? @wankunde

Hi, @Ngone51 @LuciferYang
In our prod environment, some executors failed to kill tasks.  Could you give me some help?
Reaper thread log:
21/09/27 23:44:24,882 WARN [Task reaper-1745] executor.Executor:69 : Thread dump from task 768777879:
21/09/27 23:44:34,882 WARN [Task reaper-1745] executor.Executor:69 : Killed task 768777879 is still running after 2240871 ms
21/09/27 23:44:34,885 WARN [Task reaper-1745] executor.Executor:69 : Thread dump from task 768777879:
21/09/27 23:44:44,885 WARN [Task reaper-1745] executor.Executor:69 : Killed task 768777879 is still running after 2250874 ms
21/09/27 23:44:44,888 WARN [Task reaper-1745] executor.Executor:69 : Thread dump from task 768777879:
21/09/27 23:44:54,888 WARN [Task reaper-1745] executor.Executor:69 : Killed task 768777879 is still running after 2260877 ms
21/09/27 23:44:54,891 WARN [Task reaper-1745] executor.Executor:69 : Thread dump from task 768777879:
21/09/27 23:45:04,891 WARN [Task reaper-1745] executor.Executor:69 : Killed task 768777879 is still running after 2270880 ms
21/09/27 23:45:04,894 WARN [Task reaper-1745] executor.Executor:69 : Thread dump from task 768777879:
21/09/27 23:45:14,894 WARN [Task reaper-1745] executor.Executor:69 : Killed task 768777879 is still running after 2280883 ms
21/09/27 23:45:14,896 WARN [Task reaper-1745] executor.Executor:69 : Thread dump from task 768777879:
21/09/27 23:45:24,897 WARN [Task reaper-1745] executor.Executor:69 : Killed task 768777879 is still running after 2290886 ms
21/09/27 23:45:24,899 WARN [Task reaper-1745] executor.Executor:69 : Thread dump from task 768777879:
21/09/27 23:45:34,899 WARN [Task reaper-1745] executor.Executor:69 : Killed task 768777879 is still running after 2300888 ms
21/09/27 23:45:34,902 WARN [Task reaper-1745] executor.Executor:69 : Thread dump from task 768777879:
21/09/27 23:45:44,902 WARN [Task reaper-1745] executor.Executor:69 : Killed task 768777879 is still running after 2310891 ms
21/09/27 23:45:44,904 WARN [Task reaper-1745] executor.Executor:69 : Thread dump from task 768777879:

Task Thread stack:
"Executor 553 task launch worker for task 768777879, task 26.0 in stage 1285726.0 of app application_1630907351152_13315" #1106477 daemon prio=5 os_prio=0 tid=0x000000002a6b2000 nid=0x20b9f runnable [0x00007f87a9039000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering.compare_0_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering.compare(Unknown Source)
        at org.apache.spark.sql.execution.UnsafeKVExternalSorter$KVComparator.compare(UnsafeKVExternalSorter.java:272)
        at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter$SortComparator.compare(UnsafeInMemorySorter.java:70)
        at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter$SortComparator.compare(UnsafeInMemorySorter.java:44)
        at org.apache.spark.util.collection.TimSort$SortState.gallopRight(TimSort.java:638)
        at org.apache.spark.util.collection.TimSort$SortState.mergeHi(TimSort.java:887)
        at org.apache.spark.util.collection.TimSort$SortState.mergeAt(TimSort.java:536)
        at org.apache.spark.util.collection.TimSort$SortState.mergeCollapse(TimSort.java:462)
        at org.apache.spark.util.collection.TimSort$SortState.access$200(TimSort.java:325)
        at org.apache.spark.util.collection.TimSort.sort(TimSort.java:153)
        at org.apache.spark.util.collection.Sorter.sort(Sorter.scala:37)
        at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.getSortedIterator(UnsafeInMemorySorter.java:364)
        at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spill(UnsafeExternalSorter.java:221)
        at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.createWithExistingInMemorySorter(UnsafeExternalSorter.java:111)
        at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:158)
        at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.destructAndCreateExternalSorter(UnsafeFixedWidthAggregationMap.java:248)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.agg_doConsume_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.agg_doAggregateWithKeys_1$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.agg_doAggregateWithKeys_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:50)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:730)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
        at org.apache.spark.rdd.RDD$$anon$2.hasNext(RDD.scala:332)
        at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:176)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
        at org.apache.spark.scheduler.Task.run(Task.scala:127)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:486)
        at org.apache.spark.executor.Executor$TaskRunner$$Lambda$533/2066049817.apply(Unknown Source)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1379)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:489)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00007f8c72788150> (a java.util.concurrent.ThreadPoolExecutor$Worker)


2.1.2 fails to find some partition types.

2.1.2 fails on ZFS
Traceback (most recent call last):
  File "/usr/local/bin/dataserv-client", line 4, in <module>
    __import__('pkg_resources').run_script('dataserv-client==2.1.2', 'dataserv-client')
  File "/usr/local/lib/python3.4/site-packages/pkg_resources/__init__.py", line 735, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/local/lib/python3.4/site-packages/pkg_resources/__init__.py", line 1652, in run_script
    exec(code, namespace, namespace)
  File "/usr/local/lib/python3.4/site-packages/dataserv_client-2.1.2-py3.4.egg/EGG-INFO/scripts/dataserv-client", line 10, in <module>
    cli.main(sys.argv[1:])
  File "/usr/local/lib/python3.4/site-packages/dataserv_client-2.1.2-py3.4.egg/dataserv_client/cli.py", line 165, in main
    config_path=arguments.pop("config_path"),
  File "/usr/local/lib/python3.4/site-packages/dataserv_client-2.1.2-py3.4.egg/dataserv_client/api.py", line 61, in __init__
    if control.util.get_fs_type(self.store_path) == "vfat":
  File "/usr/local/lib/python3.4/site-packages/dataserv_client-2.1.2-py3.4.egg/dataserv_client/control/util.py", line 51, in get_fs_type
    raise exceptions.PartitionTypeNotFound(path)
dataserv_client.exceptions.PartitionTypeNotFound: Couldn't find partition type for path: '/Z2/storj'!

andrey: hi, got error with latest build
Traceback (most recent call last):
File "dataserv-client", line 10, in
File "E:\dataserv-client\dataserv_client\cli.py", line 165, in main
File "E:\dataserv-client\dataserv_client\api.py", line 61, in init
File "E:\dataserv-client\dataserv_client\control\util.py", line 51, in get_fs_type
dataserv_client.exceptions.PartitionTypeNotFound: Couldn't find partition type for path: 'd:\storj2'!
its win7 soft raid0 dynamic disks

Fixed in https://github.com/Storj/dataserv-client/commit/6aaea61941107ab63c48d1b8c93da4c168ec2ecf


Error when initializing FileWriter (deprecated usage)

Please, update these lines for train and validation FileWriter (or make it compatible with tf 2):
https://github.com/deepmipt/DeepPavlov/blob/b66179e584d3eb6da73c5731ba7b732dab7e94bd/deeppavlov/core/trainers/nn_trainer.py#L150
This gives: AttributeError: module 'tensorboard.summary._tf.summary' has no attribute 'FileWriter'.
Possible fix:
self.tb_train_writer = self._tf.summary.create_file_writer(str(self.tensorboard_log_dir / 'train_log'))
self.tb_valid_writer = self._tf.summary.create_file_writer(str(self.tensorboard_log_dir / 'valid_log'))

And
https://github.com/deepmipt/DeepPavlov/blob/b66179e584d3eb6da73c5731ba7b732dab7e94bd/deeppavlov/core/trainers/nn_trainer.py#L178
produces AttributeError: module 'tensorflow' has no attribute 'Summary'

Replaced with torch.utils.tensorboard.SummaryWriter in #1572


Unrecoverable error when using piano

So, I was using the most resent prerelease. However when I tried playing the piano game the an error occured, after I ignored it and the next error that occured I was left in a gamestate where screenshots don't work as the piano game has disabled them but where there are no piano sprites on screen including the quit sprite.
This could be a result of me not having a midi keyboard on my computer.

Not a result of not having MIDI keyboard as the code handles the case where one doesn't exist.
The pre-release of MAS has made changes to how Piano sprites are loaded and how it works, thus the incompatibility.
For now you can remove MIDIsupport.rpy to remove that submod until new MAS version is released and the submod updated (I'll also separate the Submods so it's not all bundled together).

Closing as fixed


Printer not release after print !

dear sir/madam
i got problem after print, printer doesn't released paper,
is there are any solution ?
thanks.

Are you looking for a paper cut?

On Wed, Jan 8, 2020 at 10:39 AM MTM Suhail<EMAIL_ADDRESS>wrote:

Are you looking for a paper cut?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/mtmsuhail/ESC-POS-USB-NET/issues/2?email_source=notifications&email_token=AG3EI3CWPI3LR6RI3CU6ZH3Q4VDILA5CNFSM4KECKAO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEILCSAI#issuecomment-571877633,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AG3EI3FIHDVMVLY3PGOAGLDQ4VDILANCNFSM4KECKAOQ
.

No I mean function release on printer ,after send print document , user
need to press button release to add new paper for print new document , I
want a function that can auto release paper without pressing button

Do you have any solution sir, i need to implement your framework to my
project !
On Wed, Jan 8, 2020 at 12:16 PM Thunny Nt<EMAIL_ADDRESS>wrote:

On Wed, Jan 8, 2020 at 10:39 AM MTM Suhail<EMAIL_ADDRESS>wrote:

Are you looking for a paper cut?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/mtmsuhail/ESC-POS-USB-NET/issues/2?email_source=notifications&email_token=AG3EI3CWPI3LR6RI3CU6ZH3Q4VDILA5CNFSM4KECKAO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEILCSAI#issuecomment-571877633,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AG3EI3FIHDVMVLY3PGOAGLDQ4VDILANCNFSM4KECKAOQ
.

No I mean function release on printer ,after send print document , user
need to press button release to add new paper for print new document , I
want a function that can auto release paper without pressing button

Still I couldn't get your real issue.
Normally every thermal printer will roll out paper when we do print command. It will cut the paper if the printer supports it.
In this scenario you said that it is not coming the paper until you press the button in your printer. So that really printer dependant problem. May I know what printer you are using?.

I am using EPSON TM-U295 Slip;
On Thu, Jan 9, 2020 at 2:09 PM MTM Suhail<EMAIL_ADDRESS>wrote:

Still I couldn't get your real issue.
Normally every thermal printer will roll out paper when we do print
command. It will cut the paper if the printer supports it.
In this scenario you said that it is not coming the paper until you press
the button in your printer. So that really printer dependant problem. May I
know what printer you are using?.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mtmsuhail/ESC-POS-USB-NET/issues/2?email_source=notifications&email_token=AG3EI3B4TE45CT4WD6SNG2LQ43ESNA5CNFSM4KECKAO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIPHW6A#issuecomment-572423032,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AG3EI3CRXMX24ZPZ4KX2YQDQ43ESNANCNFSM4KECKAOQ
.


The payment page can be opened with the URL access when the HIDE_PAYMENT_INFORMATION is true.

#Date: 2016/06/02
[Test category]: creative Test
[Steps to reproduce]

Set the HIDE_PAYMENT_INFORMATION as true in configsetting.properties file.
Install CTMG.
Login as customer, go to "account" page.
Input the URL as http://g08fnstd131528.g08.fujitsu.local:8080/oscm-portal/marketplace/account/payments.jsf

[Expected result]
After step3, there is no entry for payment.
After step4, the payment page cannot be opened and should be some warning message.
[Observed result]
After step3, there is no entry for payment.
After step4, the payment page can be opened and can do some modification or delete operations.

The page is to be hidden if HIDE_PAYMENT_INFORMATION is set. This is, because it doesn't make sense to offer functionality to the customers for creating payments, if they cannot use such due this configuration. On the other hand, it doesn't cause any harm and there is neither an authority violation nor information leak this the payment page is for customer access.
Please fix. Return a blank body page showing respective message in the error panel.

Fixed in 08f368a


Problem with lorawan.py

I am trying to send data packet from my windows PC through LoStik to My raspberry Pi gateway
The status for transmission of packet to gateway is OK but I am not able to receive anything on gateway. I am using US configuration file and ABP. I ran configure.py and then lorawan.py
One more doubt the default 923.3 Mhz frequency is uplink or downlink frequency. How to configure to a particular uplink frequency? Also what all codes from the repository should be run to be able to send packet to gateway?
This is the output I get:
Connection to LoStik established
mac set devaddr 260211EE
mac set appskey 5C5A6E6DEC9EBF6F01575B7FFC508B07
mac set nwkskey 250AAA4CF48437307EF5ABE39EAB9954
mac join abp
radio get freq
radio set freq 904500000
mac pause
mac resume
radio set pwr 10
radio get freq
STATUS: ok
STATUS: ok
STATUS: ok
STATUS: ok
STATUS: accepted
UPDATING STATE to connected
STATUS: 923300000
STATUS: ok
STATUS:<PHONE_NUMBER>
STATUS: ok
STATUS: ok
STATUS: 904500000
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1<PHONE_NUMBER>
STATUS: ok
STATUS: mac_tx_ok
mac tx uncnf 1 155976883

I increased the frequency deviation it worked. Sorry, for trouble

I tried my gateway with Arduino leonardo with RN2903, it was receiving packets but Lostik is sending packet but not to my gateway but other gateways as I am able to see packets in The things network. What could be the possible problem with my gateway?
I am using Rpi dual channel gateway.


Upside down flag modifier

An upside down flag in many countries is used as a symbol of protest against that country. In the United States, it's used as a symbol of distress.
Petitions

N/A

Proposals

N/A

Specific References

N/A

Standard References

Unicode Technical Standard 51 “Unicode Emoji” (beta)
How to propose emojis to Unicode
How to propose other additions or changes to Unicode (required form, email address: docsubmit＠unicode.org)
Unicode document submission form

The only country Iʼm aware of where this practice is somewhat common is the United States. In many countries it would not even work because of symmetric flags.


yabai-v7.1.0 Unable to move App windows to different spaces and unable to load Scripting addition

Yabai version: 7.1.0
Every time I try to move a window to a different space and focus that space it doesn't work. It seems like the window has moved but instead it stays in the same space but hides itself and the other window takes up the whole screen.
From typing yabai -V I see this output
EVENT_HANDLER_DAEMON_MESSAGE: signal --add event=dock_did_restart action=sudo yabai --load-sa
EVENT_HANDLER_DAEMON_MESSAGE: config focus_follows_mouse autofocus
yabai configuration loaded..

I have to rm /tmp/yabai* and then update the hash in sudoers file and then load scripting addition and that goes successfully but then I can't use yabai -V. I have SIP disabled.
ystem Integrity Protection status: unknown (Custom Configuration).

Configuration:
	Apple Internal: disabled
	Kext Signing: enabled
	Filesystem Protections: disabled
	Debugging Restrictions: disabled
	DTrace Restrictions: enabled
	NVRAM Protections: disabled
	BaseSystem Verification: enabled
	Boot-arg Restrictions: disabled
	Kernel Integrity Protections: disabled
	Authenticated Root Requirement: enabled

This is an unsupported configuration, likely to break in the future and leave your machine in an unknown state.

Every other shortcut set through skhd works fine, I am able to wrap, swap and balance windows.

From my experience, sometimes things break when you use some wonky shortcuts in your skhd config file. Try commenting all your shortcuts and do yabai --restart-service && skhd --restart-service and see if things work fine.

unfortunately, I tried doing that many time it still hasn't worked. My skhd contains this
# moving windows to different spaces
ctrl + alt + shift - d : yabai -m space --destroy
ctrl + alt - 1 : yabai -m window --space 1; yabai -m space --focus 1
ctrl + alt - 2 : yabai -m window --space 2; yabai -m space --focus 2
ctrl + alt - 3 : yabai -m window --space 3; yabai -m space --focus 3
ctrl + alt - 4 : yabai -m window --space 4; yabai -m space --focus 4
ctrl + alt - 5 : yabai -m window --space 5 && yabai -m space --focus 5
ctrl + alt - 6 : yabai -m window --space 6 && yabai -m space --focus 6
ctrl + alt - 7 : yabai -m window --space 7 && yabai -m space --focus 7

And part of the shortcuts work, I focus a window and press the shortcut to move the window to space 1 and it focuses the space and takes me there but the windows stays in the space it is intended to move from.

I have the same problem with the recent update to macos (14.5 Beta) and yabai (7.1.0)

Same issue after updating to MacOS 14.5.

duplicate https://github.com/koekeishiya/yabai/issues/2240


feat(core): add Alluxio e2e test

base on PR: #3548 , we need to add E2E test for Alluxio.

MOST LGTM BTW, we need add health check in docker compose

Excuse me? How can I fix this CI error

Let's address the core's failure first.
thread 'main' panicked at core/src/raw/tests/utils.rs:68:45:
must succeed: Unsupported (permanent) at  => scheme is not enabled or supported

Context:
   scheme: alluxio

It's looks alluxio is not added in via_map at  https://github.com/apache/incubator-opendal/blob/main/core/src/types/operator/builder.rs#L154?

Please add Alluxio to this function.
https://github.com/apache/incubator-opendal/blob/bec1e9db5effb99fe1d369941f6e04036fb2ec99/core/src/types/scheme.rs#L273

It seems that these error contexts don't have the correct scheme set and can be updated together.
https://github.com/apache/incubator-opendal/blob/bec1e9db5effb99fe1d369941f6e04036fb2ec99/core/src/services/alluxio/backend.rs#L151-L166

For health check, you could take the following as references :

https://docs.docker.com/compose/compose-file/compose-file-v3/#healthcheck
https://github.com/apache/incubator-opendal/blob/main/fixtures/s3/docker-compose-minio.yml

The following tests are failure:
failures:
    behavior::test_blocking_write_with_special_chars
    behavior::test_blocking_create_dir_existing
    behavior::test_blocking_stat_not_exist
    behavior::test_blocking_stat_with_special_chars
    behavior::test_blocking_read_not_exist
    behavior::test_check
    behavior::test_blocking_delete_file
    behavior::test_list_dir
    behavior::test_list_dir_with_metakey
    behavior::test_blocking_remove_one_file
    behavior::test_list_non_exist_dir
    behavior::test_list_sub_dir
    behavior::test_list_dir_with_metakey_complete
    behavior::test_scan_root
    behavior::test_list_nested_dir
    behavior::test_scan
    behavior::test_rename_non_existing_source
    behavior::test_rename_file
    behavior::test_remove_all
    behavior::test_rename_nested
    behavior::test_create_dir_existing
    behavior::test_write_with_special_chars
    behavior::test_rename_overwrite
    behavior::test_stat_not_exist
    behavior::test_stat_with_special_chars
    behavior::test_read_not_exist
    behavior::test_read_with_special_chars
    behavior::test_delete_file
    behavior::test_delete_with_special_chars
    behavior::test_delete_not_existing
    behavior::test_writer_abort
    behavior::test_remove_one_file
    behavior::test_list_rich_dir
    behavior::test_delete_stream

You can reproduce those tests locally by OPENDAL_TEST=alluxio cargo test behaivor --features services-alluxio

It seems that these error contexts don't have the correct scheme set and can be updated together.
https://github.com/apache/incubator-opendal/blob/bec1e9db5effb99fe1d369941f6e04036fb2ec99/core/src/services/alluxio/backend.rs#L151-L166

fixed

The following tests are failure:
failures:
    behavior::test_blocking_write_with_special_chars
    behavior::test_blocking_create_dir_existing
    behavior::test_blocking_stat_not_exist
    behavior::test_blocking_stat_with_special_chars
    behavior::test_blocking_read_not_exist
    behavior::test_check
    behavior::test_blocking_delete_file
    behavior::test_list_dir
    behavior::test_list_dir_with_metakey
    behavior::test_blocking_remove_one_file
    behavior::test_list_non_exist_dir
    behavior::test_list_sub_dir
    behavior::test_list_dir_with_metakey_complete
    behavior::test_scan_root
    behavior::test_list_nested_dir
    behavior::test_scan
    behavior::test_rename_non_existing_source
    behavior::test_rename_file
    behavior::test_remove_all
    behavior::test_rename_nested
    behavior::test_create_dir_existing
    behavior::test_write_with_special_chars
    behavior::test_rename_overwrite
    behavior::test_stat_not_exist
    behavior::test_stat_with_special_chars
    behavior::test_read_not_exist
    behavior::test_read_with_special_chars
    behavior::test_delete_file
    behavior::test_delete_with_special_chars
    behavior::test_delete_not_existing
    behavior::test_writer_abort
    behavior::test_remove_one_file
    behavior::test_list_rich_dir
    behavior::test_delete_stream

You can reproduce those tests locally by OPENDAL_TEST=alluxio cargo test behaivor --features services-alluxio

thanks, I'll fix them.


Update: Clavister

https://www.clavister.com/
And yes, they really suck.

Update merged, thanks for adding!


World Machines

So we replace the geck tub with the purifier when a player attempts to use it the first time, if all parts have been found.
That's how I understood it worked, there would be no progress view.

[ ] Item Use Type doesn't work.
[ ] World machines should activate.
[ ] World machines should be functional.

Closes #1

Trashed


Wireguard IPv6 support

Description
Currently wireguard protocol in VPP supports only IPv4.
This issue is related to the addition of IPv6 protocol support.

Patch for vpp:
https://gerrit.fd.io/r/c/vpp/+/32009


Use 16 bits service UUID on esp32 platform

Problem
It seems like some platforms use 16 bits service UUIDs while the esp32 uses 128 bits.
This is a partial fix for #4061 since at the moment, chip-tool does not pass https://github.com/project-chip/connectedhomeip/blob/90df58f7367fbbc3f567c8b503fb67adab562d6d/src/platform/Darwin/BleConnectionDelegateImpl.mm#L175
Summary of Changes

Update esp32 to use 16 bits service UUIDs for both nimble and bluedroid
Update src/platform/Darwin code

@vivien-apple I think this will also fix #3829, right?

@vivien-apple I think this will also fix #3829, right?

Yes and No :)
The above fix will only support 16-bits format. This is what the original code from OpenWeave does.
I initially changed this part while I did the initial work to enable bluetooth with the esp32 on the chip stack as it was not working otherwise, mostly because the esp32 was using 128-bits uuids by default. But looking at the original code, it seems to me that the initial intention was to only support 16-bits services.
I someone strongly believe we should support both I can add that too.


Generated JS code can end up with (reserved) keywords as identifiers

It seems wasm-bindgen does not check that identifiers that are valid in Rust code might not be valid in JavaScript.
Building a project containing a function like this:
use wasm_bindgen::prelude::wasm_bindgen;

#[wasm_bindgen]
pub fn test(class: u8) {}

... generates this piece of JS code:
/**
* @param {number} class
*/
export function test(class) {
    wasm.test(class);
}

which is invalid (class is a keyword and can't be used for identifiers in JS)
Steps to Reproduce

wasm-pack new bug
Paste the code above in src/lib.rs
wasm-pack build
Check pkg/bug_bg.js (filename depends on the project name)

Expected Behavior
Prefix identifier with _ so it can't clash with keywords (or mangle identifiers in some other way)
Actual Behavior
The identifier is not changed and clashes with the JavaScript "class" keyword

I am not sure whether generating idents with prefixes is the "right" approach. It breaks the Rust -> JS interface mapping contract.
I'd suggest issue compiler warnings once user uses idents that are keywords in js.

I guess I will tack on to this issue, but a slightly more insidious version of the same problem is that names in your rust code can also shadow the same names in the generated JS code.
I just ran into this with a function with the signature:
pub fn process_frame(mut wasm: WasmEngine, time: u64)
which then generated this piece of JS shim:
export function process_frame(wasm, time) {
    _assertClass(wasm, WasmEngine);
    if (wasm.ptr === 0) {
        throw new Error('Attempt to use a moved value');
    }
    var ptr0 = wasm.ptr;
    wasm.ptr = 0;
    uint64CvtShim[0] = time;
    const low1 = u32CvtShim[0];
    const high1 = u32CvtShim[1];
    wasm.process_frame(ptr0, low1, high1);
}

which in quite hilarious fashion breaks, because the parameter called 'wasm' shadows the fairly important import:
import * as wasm from './[REDACTED]_wasm_bg.wasm';
at the top of the file. :p
Ideally I guess all generated JS code should prefix their names with something, but some warnings for names like 'wasm' would be a good idea. 😂

Duplicate of #2338.


replacing Arduino methods with IDF's

Hi @rwmingis
I've made the changes required to build the lib without any dependency to Arduino.h. Also it could be build solely as an IDF component via provided CMake file.
Pls, check commits one by one, I've tried to make it as granular as possible. Had to change constructor a little bit.
You may wanna merge this into some separate branch and make you corrections there on top of my changes.
That way it would be easier for me to rebase further changes if any.
Having this as a milestone next thing I'll try to do some more drastic changes and introduce RTOS queue or event to ISR handling. This way I suppose it would be possible to fix the issue with dynamic button objects.
Let me know if you have any questions or concerns.
Cheers!

Hi @vortigont,
Wow, thanks for all that suggested code.  It all looks great, and the one bit I was going to question, you have already fixed in the clean up commit.
I pretty much agree with all your suggestions, so will plan on implementing them in the next few days as free time comes available.
I think this is going to be great.  really hope this gets picked up by the community and finally have a chance to give back. 👍
Cheers,
Rob

Thanks for your feedback, Rob!
This should become a pretty nice button lib for all esp32 family chips. I already use it to control light switches and dimming in my house :) The project is still under development, but works pretty fine so far :+1:


Alanren/notebook password notice

missed these 2 files in my previous pr

Coverage decreased (-0.07%) to 35.235% when pulling 48f74e112ab415fe3503076b2b9449963f2134a6 on alanren/notebook-password-notice into 20c612d9fb7f4b46b901b99cc49612cdf5284fa5 on main.


[BUG] {Unsupported resolution set for detected camera IMX378/214, needs 1080_P / 4_K / 12_MP. Defaulting to 1080_P}

Hi, I am trying to get more fps out of my center rgb camera on my OAK-D by lowering the resolution but keep getting this error... Is there any way to bypass it?

I thought the side depth cameras can reach up to 120fps

These are the only supported resolutions for the center camera @onthegrid007
LEFT & RIGHT ov9282 should work however at higher FPS, likely at 400p resolution CC: @alex-luxonis

Yes, the OV9282 cameras should work at higher FPS, some info here: https://docs.luxonis.com/hardware/sensors/OV9282
but actually slightly more is possible as in the output from device.getConnectedCameraFeatures():
https://github.com/luxonis/depthai-python/blob/main/docs/source/components/nodes/color_camera.rst#full-fov

{width: 1280, height: 720, minFps: 1.687, maxFps: 143.1, type: MONO, fov: {x:0, y: 40, width: 1280, height: 720}}
{width: 1280, height: 800, minFps: 1.687, maxFps: 129.6, type: MONO, fov: {x:0, y: 0, width: 1280, height: 800}}
{width: 640, height: 400, minFps: 1.687, maxFps: 255.7, type: MONO, fov: {x:0, y: 0, width: 1280, height: 800}}

For IMX378 rgb camera, up to 85fps is possible now with THE_2024x1520: https://docs.luxonis.com/hardware/sensors/IMX378
We'll add back binning as an option for IMX378, then up to 118fps will be possible with THE_1080_P https://github.com/luxonis/depthai-python/pull/986 https://github.com/luxonis/depthai-core/pull/972


There was an error running homogenize.py

Hi!
Thank you for the tool and your active support here.
Following another issue (from SqueezeMeta) I run mOTUlizer on bins from numerous independent SquezeMeta projects and got list of mOTUs with MAGs. Now I run SuperPang on each mOTU as it was suggested. No issues with most of the mOTUs, but one specific mOTU contains about 8000 MAGs, so SuperPang is giving me an error:
There was an error running homogenize.py. Please open an issue

Traceback (most recent call last):
  File "/home/miniconda3/envs/SuperPang/lib/python3.8/site-packages/SuperPang/lib/utils.py", line 61, in write_fastq
    outfile.write(f'@{name}\n{seq}\n+\n{qual}\n')
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/miniconda3/envs/SuperPang/bin/homogenize.py", line 332, in <module>
    main(parse_args())
  File "/home/miniconda3/envs/SuperPang/bin/homogenize.py", line 24, in main
    fasta2fastq(args.fasta, current1)
  File "/home/miniconda3/envs/SuperPang/lib/python3.8/site-packages/SuperPang/lib/utils.py", line 65, in fasta2fastq
    write_fastq(read_fasta(fasta), fastq)
  File "/home/miniconda3/envs/SuperPang/lib/python3.8/site-packages/SuperPang/lib/utils.py", line 61, in write_fastq
    outfile.write(f'@{name}\n{seq}\n+\n{qual}\n')
OSError: [Errno 28] No space left on device

I run it on HPC with 120 Gb of RAM (HPC report tells me that only around 30 Gb of RAM were used when error occured) and several Tb of storage, also tried to export temp directory to a larger disk with no success.
My question if it is "legal" to separate 8000 MAGs on subsets, create assemblies with SuperPang and then run SuperPang on those assemblies to get one final assembly.
Or, is there a better approach to handle it?

So I think that the problem is happening because there is no more space left in your /tmp partition, which is where SuperPang writes the temporary files.

Exporting temp directory will not solve it?

How many SqueezeMeta projects are you combining?

There are around 100 projects

Are you combining only the bins from DAStools (in the results/bins dir) or the bins from all the methods?

All cutoffs are default:
mOTUlize.py --fnas $bins/*.fa -o $out --threads 6

SuperPang.py --fasta $mags --checkm Bins/completness.tsv --output-dir $out --threads 4 --force-overwrite

were --fasta $mags leads to the file with mags pathes.

It didn't solve it? What changes have you made to the SuperPang.py script?
No changes to the sropt, just exported my temp directory before runing the script:

export TMPDIR=$TMPDIR
I will try to modify the script itself as you suggested

So I assume $mags contain only the paths for one particular mOTU, right?
Yes, only MAGs, associated with that specific mOTU (or mOTU_0000 is not a mOTU and should be ignored?)

What column of the mOTUlizer output did you use to select the MAGs associated to each mOTU?
MAGs column in the mOTUlizer output

Yeah, that sounds like the right column to use.
I am still surprised that you get 8000 MAGs into a single OTU, for only 100 projects. Are those projects coassemblies, or was each run on a single sample?
I'm asking because we normally work under the assumption that only one MAG per species will be assembled when working with a single sample. This is because we expect assemblers to collapse mostly all intra-species diversity (allele differences, etc). But maybe things are not so simple?

Yeah, that sounds like the right column to use.

Great! Just finished with rest >1300 mOTUs that do not have so high number of MAGs =)

I am still surprised that you get 8000 MAGs into a single OTU, for only 100 projects. Are those projects coassemblies, or was each run on a single sample?

Each project is one sample, no coassemblies. Assembly and bins were created by SqueezeMeta.

I'm asking because we normally work under the assumption that only one MAG per species will be assembled when working with a single sample.

You are right, it is very suspicious. Will double check everything again (though I already checked it).

But maybe things are not so simple?

It is always not so simple for me...

Let me know if you find something weird after double checking. I am also checking with the author of mOTUlizer (Moritz Buck).
Otherwise I am happy that the rest >1300 worked fine!

Did you include a checkm input file when calling mOTUlizer?
It may be that a lot of low quality MAGs are getting into that particular mOTU.

SuperPang will also be much happier if you give it that file (though it can work without it).
You just need to run checkm on all your MAGs, and redirect stdout to a file.

Hi!
I rerun mOTUlizer with completeness info and set cutoff to 70% completeness and 5% contamination. No I have only 200 mOTUs and no more than 50 MAGs in one mOTU.
Thank you for solving it and sorry for your time since it was my error on the step that actually preceded SuperPang.
I will close the issue.

Hi!
I am working with around 450 genomes. I am also getting the same error when I am using the 0.9.4.beta1 version.
But latest version is giving me this error:
ImportError: /usr/lib64/libc.so.6: version `GLIBC_2.25' not found (required by /mmfs1/home/azk0151/miniconda3/envs/SuperPang/lib/python3.8/site-packages/speedict/speedict.cpython-38-x86_64-linux-gnu.so)
Can you please help me how to solve this issue?
Thank you!


StackOverflowException with ApolloClient.clearNormalizedCache()

Using version: 0.4.0-SNAPSHOT
What I'm trying to do is blow away all Apollo related caches in my app when the user logs out. Assuming I'm doing it correctly (not sure), I am invoking the clearNormalizedCache() method on the ApolloClient. This results in a StackOverflowException:
java.lang.StackOverflowError: stack size 8MB
	at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:343)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260)
	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1109)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore.writeTransaction(RealApolloStore.java:111)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore.clearAll(RealApolloStore.java:89)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore$3.execute(RealApolloStore.java:91)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore$3.execute(RealApolloStore.java:89)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore.writeTransaction(RealApolloStore.java:109)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore.clearAll(RealApolloStore.java:89)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore$3.execute(RealApolloStore.java:91)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore$3.execute(RealApolloStore.java:89)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore.writeTransaction(RealApolloStore.java:109)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore.clearAll(RealApolloStore.java:89)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore$3.execute(RealApolloStore.java:91)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore$3.execute(RealApolloStore.java:89)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore.writeTransaction(RealApolloStore.java:109)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore.clearAll(RealApolloStore.java:89)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore$3.execute(RealApolloStore.java:91)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore$3.execute(RealApolloStore.java:89)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore.writeTransaction(RealApolloStore.java:109)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore.clearAll(RealApolloStore.java:89)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore$3.execute(RealApolloStore.java:91)
	at com.apollographql.apollo.internal.cache.normalized.RealApolloStore$3.execute(RealApolloStore.java:89)
...

This leads me to another question. How would I go about blowing away the HTTP cache as well as the "normalized cache"? Looking at the source for ApolloClient, there is a clearHttpCache() but it's not public.

Furthermore, is there a way to invalidate a cache for a given cache key? How does one go about doing cache clearing at a more granular level like that?

Stack overflow looks like a legit issue, thanks! In terms of deleting the http cache, you can call #clearAll  on the http cache instance you provide to apollo client. But I think we can make the clearAll method on the apollo client public for convience.
Can you make a separate issue for deleting single cache entries? We should be able to do something via the imperative store api.

@BenSchwab Sounds good, thank you. Will do.


Private clusters iteration 2: change the server for the cluster kubeconfig

What this PR does / why we need it: Changes the server IP for the kubeconfig in /var/lib/kubelet/ and the one in .kube/ generated by the custom script
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
Release note:

/lgtm


HBASE-27710 ByteBuff ref counting is too expensive for on-heap buffers

This felt like the cleanest way to solve this case, but open to other opinions. Whether we need to checkRefCount is directly tied to whether we use NONE recycler.
A couple other options I considered:

Create a new inheritance hierarchy, i.e. OnHeapSingleByteBuff, etc. This felt like it'd only complicate an already complex system.
Update the SingleByteBuff and MultiByteBuff constructors to take a new boolean onHeap or boolean shouldCheckRefCount. This felt more error prone because it's too easy for someone to forget to pass the correct boolean value for the corresponding recycler.

I added a basic test to validate that we only call checkRefCount for non-NONE recyclers. Beyond that, I think our existing ample coverage should suffice? Let me know if you'd like to see a particular test.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 13s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+0 :ok:
mvndep
0m 11s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m  1s
master passed

+1 :green_heart:
compile
4m  5s
master passed

+1 :green_heart:
checkstyle
1m  9s
master passed

+1 :green_heart:
spotless
1m  8s
branch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
2m 46s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
5m 40s
the patch passed

+1 :green_heart:
compile
4m 58s
the patch passed

+1 :green_heart:
javac
4m 58s
the patch passed

+1 :green_heart:
checkstyle
1m 16s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  3s
The patch has no ill-formed XML file.

+1 :green_heart:
hadoopcheck
20m 32s
Patch does not cause any errors with Hadoop 3.2.4 3.3.4.

+1 :green_heart:
spotless
1m  7s
patch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
3m 27s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 23s
The patch does not generate ASF License warnings.

63m 23s

Subsystem
Report/Notes

Docker
ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5104

Optional Tests
dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile xml

uname
Linux 065127edc507 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / c2b64e7caa

Default Java
Eclipse Adoptium-11.0.17+8

Max. process+thread count
85 (vs. ulimit of 30000)

modules
C: hbase-common hbase-server U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/1/console

versions
git=2.34.1 maven=3.8.6 spotbugs=4.7.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 25s
Docker mode activated.

-0 :warning:
yetus
0m  4s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m 35s
master passed

+1 :green_heart:
compile
0m 57s
master passed

+1 :green_heart:
shadedjars
4m 38s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 37s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 11s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 18s
the patch passed

+1 :green_heart:
compile
0m 57s
the patch passed

+1 :green_heart:
javac
0m 57s
the patch passed

+1 :green_heart:
shadedjars
4m 37s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 34s
the patch passed

_ Other Tests _

-1 :x:
unit
1m 29s
hbase-common in the patch failed.

+1 :green_heart:
unit
207m 13s
hbase-server in the patch passed.

232m 59s

Subsystem
Report/Notes

Docker
ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5104

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux e632eb851284 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / c2b64e7caa

Default Java
Eclipse Adoptium-11.0.17+8

unit
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-common.txt

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/1/testReport/

Max. process+thread count
2418 (vs. ulimit of 30000)

modules
C: hbase-common hbase-server U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 35s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
4m 35s
master passed

+1 :green_heart:
compile
1m 23s
master passed

+1 :green_heart:
shadedjars
6m 34s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 54s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 16s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m 14s
the patch passed

+1 :green_heart:
compile
1m 25s
the patch passed

+1 :green_heart:
javac
1m 25s
the patch passed

+1 :green_heart:
shadedjars
6m 13s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 55s
the patch passed

_ Other Tests _

-1 :x:
unit
1m 54s
hbase-common in the patch failed.

+1 :green_heart:
unit
237m 31s
hbase-server in the patch passed.

272m 39s

Subsystem
Report/Notes

Docker
ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5104

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux e942a0a51051 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / c2b64e7caa

Default Java
Temurin-1.8.0_352-b08

unit
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-common.txt

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/1/testReport/

Max. process+thread count
2491 (vs. ulimit of 30000)

modules
C: hbase-common hbase-server U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 24s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+0 :ok:
mvndep
0m 13s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m 53s
master passed

+1 :green_heart:
compile
3m  6s
master passed

+1 :green_heart:
checkstyle
0m 48s
master passed

+1 :green_heart:
spotless
0m 45s
branch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
2m  2s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 10s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 36s
the patch passed

+1 :green_heart:
compile
3m  8s
the patch passed

+1 :green_heart:
javac
3m  8s
the patch passed

+1 :green_heart:
checkstyle
0m 46s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
hadoopcheck
14m  2s
Patch does not cause any errors with Hadoop 3.2.4 3.3.4.

+1 :green_heart:
spotless
0m 42s
patch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
2m 13s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 16s
The patch does not generate ASF License warnings.

44m 34s

Subsystem
Report/Notes

Docker
ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/2/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5104

Optional Tests
dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile xml

uname
Linux ea0ff56bbf26 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / c2b64e7caa

Default Java
Eclipse Adoptium-11.0.17+8

Max. process+thread count
84 (vs. ulimit of 30000)

modules
C: hbase-common hbase-server U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/2/console

versions
git=2.34.1 maven=3.8.6 spotbugs=4.7.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 25s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 10s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m 22s
master passed

+1 :green_heart:
compile
1m 24s
master passed

+1 :green_heart:
shadedjars
5m  8s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 40s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 12s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 52s
the patch passed

+1 :green_heart:
compile
1m  4s
the patch passed

+1 :green_heart:
javac
1m  4s
the patch passed

+1 :green_heart:
shadedjars
5m  6s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 37s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
2m 28s
hbase-common in the patch passed.

+1 :green_heart:
unit
206m 12s
hbase-server in the patch passed.

234m 54s

Subsystem
Report/Notes

Docker
ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5104

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 178808e9bed2 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / c2b64e7caa

Default Java
Eclipse Adoptium-11.0.17+8

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/2/testReport/

Max. process+thread count
2408 (vs. ulimit of 30000)

modules
C: hbase-common hbase-server U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/2/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m  2s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 11s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
2m 48s
master passed

+1 :green_heart:
compile
0m 58s
master passed

+1 :green_heart:
shadedjars
4m 17s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 38s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 13s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 50s
the patch passed

+1 :green_heart:
compile
0m 58s
the patch passed

+1 :green_heart:
javac
0m 58s
the patch passed

+1 :green_heart:
shadedjars
4m 17s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 38s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
1m 53s
hbase-common in the patch passed.

+1 :green_heart:
unit
224m 18s
hbase-server in the patch passed.

249m 46s

Subsystem
Report/Notes

Docker
ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5104

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 1c8c92624138 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / c2b64e7caa

Default Java
Temurin-1.8.0_352-b08

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/2/testReport/

Max. process+thread count
2405 (vs. ulimit of 30000)

modules
C: hbase-common hbase-server U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/2/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9 thank you very much for the review. I had an idea this morning, and wonder if you have any opinion.
Currently we do this:
protected void checkRefCount() {
  ObjectUtil.checkPositive(refCnt(), REFERENCE_COUNT_NAME);
}

Calling refCnt() goes down the expensive path of getting the real refCnt numeric value.
I think what we really care about is "has this buffer been recycled". In which case, what if we added a volatile boolean to our RefCnt class which gets set to true when the Recycler is called? We don't care about synchronization since it always goes from false to true. The above method could become:
//
// in RefCnt.java
//
private volatile boolean recycled;

public boolean isRecycled() {
    return recycled;
}

@Override
protected final void deallocate() {
    this.recycler.free();
    this.recycled = true; // of note
    if (leak != null) {
      this.leak.close(this);
    }
}

//
// In ByteBuff.java
//
protected void checkRefCount() {
    Preconditions.checkState(!refCnt.isRecycled(), "ByteBuff has been recycled");
}

Of course we'd also rename the method. I plugged this into our test case and it performs similarly to this PR. The benefit of this approach is it might also speed up off-heap usages while still providing protection.

What you proposed is the trick in netty's CompositeByteBuf, where they introduce a freed flag to indicate whether the ByteBuf is still valid.
And for AbstractReferenceCountedByteBuf, the code is like this
    @Override
    boolean isAccessible() {
        // Try to do non-volatile read for performance as the ensureAccessible() is racy anyway and only provide
        // a best-effort guard.
        return updater.isLiveNonVolatile(this);
    }

But  seems we do not have access to the updater field so I think we could go with your current approach. The down side is we will add one more boolean for each ByteBuff but should be OK?

Just commit it as is for now, can open another issue for improvement performance for off heap ByteBuff.

Agreed, thanks for discussing and review.
I had another idea that we could null out the recycler after calling it, so isAccessible would be a null check on that existing reference, rather than a new boolean field check. This would not take any additional memory. But can investigate these options in another issue.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m  0s
Docker mode activated.

-1 :x:
patch
0m  3s
https://github.com/apache/hbase/pull/5104 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hbase/pull/5104

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/3/console

versions
git=2.17.1

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m  0s
Docker mode activated.

-1 :x:
patch
0m  2s
https://github.com/apache/hbase/pull/5104 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hbase/pull/5104

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/3/console

versions
git=2.25.1

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m  0s
Docker mode activated.

-1 :x:
patch
0m  3s
https://github.com/apache/hbase/pull/5104 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hbase/pull/5104

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5104/3/console

versions
git=2.25.1

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.


Remove field macros

This change is

#585  👈
dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @spapinistarkware and the rest of your teammates on  Graphite

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 93.91%. Comparing base (355f2e5) to head (5b0ecc9).

Additional details and impacted files
@@           Coverage Diff           @@
##              dev     #585   +/-   ##
=======================================
  Coverage   93.91%   93.91%
=======================================
  Files          67       67
  Lines        8426     8435    +9
  Branches     8426     8435    +9
=======================================
+ Hits         7913     7922    +9
  Misses        449      449
  Partials       64       64

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


How long is this repo up to date?

I'm writing this project from scratch. Writing is better for me. I would like to help you after writing (I realized I was wrong in the previous issue, ignore it)

Hi @sonerry,
Well, It's a hobby. I'm using this project, because I'm fascinating by compilers. Chibicc is very well-written, easy to understand and great to work on it, to extend it. I'm for now blocked on issue with vector size found during postgres compilation. Any help is always appreciated. Don't hesitate, if you see bugs and have some ideas about how to fix them.  regards


Improve OS X Binaries

Supply extra information into Bundle:

(Hopefully) Apple-compliant version strings
Minimum OS X Version
entitlements file to use with codesign utility
Minor source cleanup

fixes #5


Clean up console errors/warnings when navigating to plans tab

I think we need to add some missing labels to fix this.

This is going to be obsolete as the modal wrapping the wizard is replaced by "in-page" wizard.
BTW the error is because the modal in question needs to have an aria-label property.


Add element hiding readme

Task/Issue URL: https://app.asana.com/0/0/1203533152944692/f
Description
Steps to test this PR
Feature 1

[ ]
[ ]

UI changes

Before
After

!(Upload before screenshot)
(Upload after screenshot)

Current dependencies on/for this PR:

develop

PR #3677   👈

This comment was auto-generated by Graphite.


Fixing 2.6.

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Closes: #8
The only necessary file in recipe folder is the recipe.
As explained here, the conda_build_config.yaml created issue https://github.com/conda-forge/torchsde-feedstock/pull/8#discussion_r1850839688 if we correct the issue, the file useless anyway
The license file is already packaged in the wheel.
More importantly, this is a noarch pure python package. There's no need for compilers or a version for each platform and each python version.

@conda-forge-admin, please rerender

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe/meta.yaml:

ℹ In conda-forge.yml: $.build_with_mambabuild = False.

'Build With Mambabuild' is deprecated.
build_with_mambabuild is deprecated, use conda_build_tool instead.

Schema
{
  "anyOf": [
    {
      "type": "boolean"
    },
    {
      "type": "null"
    }
  ],
  "default": true,
  "deprecated": true,
  "title": "Build With Mambabuild"
}

Uploading to conda-forge with source channel ' conda-forge' is not allowed
https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=878598&view=logs&j=d0d954b5-f111-5dc4-4d76-03b6c9d0cf7e&t=6d4b912b-175d-51da-0fd9-4d30fe1eb4e7&l=994
So, this recipe/conda_build_config.yaml was definetely the main problem

Thanks @HeavyTony2!
I'm not using torchsde these days, so if you want to add yourself as a maintainer, feel free to open an issue:
https://conda-forge.org/docs/maintainer/infrastructure/#conda-forge-admin-please-add-user-username


Is it possible to add pymongo metric after the fact

There are some early runs where I don't have some metrics that I started calculating later. Is it possible to go back and add metrics to the database entry for certain runs? In this question (https://github.com/IDSIA/sacred/issues/875) I was asking about how to retrieve metric values, which exposed how metrics are stored. I could possibly recreate those data structures but

It would be nice if some of that process was in a function somewhere already to save some time (maybe a long shot) and
I don't know much about pymongo, I'm not sure how to push changes to a client.

Thanks for any tips!

After some fiddling I was able to figure out the following process below. In this case I used a metric "macro_avg_f1-score" as an indication that a batch of metrics needed to be calculated and added.
To my surprise (and thankfully) all you need to do is add a metric to client[dbname]["metrics"] using insert_one(), and if the run_id is correct then it will correctly propagate into entry["info"]["metrics"].
client = pymongo.MongoClient()
runs = client[dbname]["runs"]
metrics = client[dbname]["metrics"]

for entry in runs.find():
    if "metrics" not in  entry["info"]:
        print(f"Skipping entry {entry['_id']}, no metrics")
        continue
    available = [x["name"] for x in entry["info"]["metrics"]]
    if "macro_avg_f1-score" in available:
        print(f"Skipping entry {entry['_id']}, report available")
        continue

    print(f"Getting report for {entry['_id']}")
    report = get_report(entry)

    addition = {}
    for stat in ("precision", "recall", "f1-score"):
        for class_name in classes:
            addition[f"{class_name}_{stat}"] = report[class_name][stat]
        for combo in ("macro avg", "weighted avg"):
            addition[f"{combo.replace(' ', '_')}_{stat}"] = report[combo][stat]

    # Get a representative timestamp with a known metric
    index = available.index("Accuracy")
    timestamps = metrics.find_one(
        {"_id": bson.objectid.ObjectId(entry["info"]["metrics"][index]["id"])}
    )["timestamps"]

    for name, value in addition.items():
        new_id = metrics.insert_one({
            "name": name,
            "run_id": entry["_id"],
            "steps": [0],
            "timestamps": timestamps,
            "values": [value],
        })


fixed some typos

@SvenHerrmann thanks for the PR. Can you please fix the failing Testcase sAdminTest::testsValidateStep1WithoutEdit. See: https://travis-ci.org/shopware/shopware/jobs/68279501.
Thanks.

Thank you @SvenHerrmann. Was merged with commit: https://github.com/shopware/shopware/commit/6d78bff0806e7c1ef3ab36d336c324dc74d24f80


Download error

Expected behavior
Actual behavior
Information about the Issue
Steps to reproduce the behavior

...Click download from Kitematic
...It gives "context canceled" message

This problem has been solved by kitematic v0.17.7


Supporting enums

The swagger spec supports enums to specify pre-defined options:
"status": {
          "type": "string",
          "description": "Order Status",
          "enum": [
            "placed",
            " approved",
            " delivered"
          ]
        },

I know that joi doesn't support the enum() function. Any idea how we could add this here?

@jonaswindey Is there a way to extract an enum to ref so multiple types can re-use the same enum?


Reschedule

How to reschedule an appointment? It says there is already an appointment in name of the beneficiary when I try to book again.

Cancel from portal. Details here: https://github.com/pallupz/covid-vaccine-booking/pull/170


Implement Chebyshev class and functions

Adds a Chebyshev convenience class and cheb* functions that act on arrays, as described in #11055.

Hi - thanks for the PR! In the past when we've discussed the polynomial interface, we've made the choice to avoid adding polynomial class types in favor of the more direct functional API (e.g. https://github.com/google/jax/issues/70#issuecomment-907892885). One concern I have here is that when you start to define class-based interfaces, you have to be much more careful to ensure that they're compatible with JAX's transformation model.
Is there any functionality here that cannot be expressed directly via chebyshev functions, as opposed to a full chebyshev class?

I would lean toward keeping only the simplest functional APIs in JAX, partly because it makes long-term maintenance more sustainable (the class definition in this PR is a gigantic API surface to test and maintain). But I'm happy to let other team members weigh-in if they disagree.

Alright, I removed the Chebyshev class and its abstract base class to streamline the API. If it's decided later that they are worth including, I have local copies and can add them back in.

Alright, I've cleaned up a lot of the code following your suggestions and to better match the existing polynomial functions in JAX. I've also been thinking more about JAX compatibility, and I wanted to ask a couple questions. The object arrays and value-based shapes were unnecessary, but loop implementations are necessary (or at least the natural way to implement many functions) because of the relationships between terms in a Chebyshev series.
So, what do you consider "large" for a series or loop parameter? If a typical series is large, would replacing Python control flow  with JAX control flow primitives resolve the incompatibility/inefficiency issues? If so, I will make those changes.

The issue with Python loops is that they are flattened by JAX tracing, and then passed in an unrolled state to the XLA compiler. The mechanics of the compiler are complicated, but a good rule of thumb is that compilation time will be roughly quadratic in the number of operations. So if you write a Python for loop over 100 elements, then compilation will be roughly 10000 times more costly than the equivalent fori_loop implementation. There's not really any hard line regarding what size the loops should be, but if you expect the functions to operate on larger inputs, it's something to keep in mind. Does that make sense?

Ok, I've removed all Python loops from chebyshev.py and replaced them with JAX control flow primitives. I've also cleaned up most of the functions to make them less convoluted where possible. Is there anything else you would like me to do?
Thank you!

Thanks. Another big piece of adding this kind of functionality is testing – before we can merge this, we'll need full test coverage of the functionality. If it's helpful, you can base it off of existing tests in https://github.com/google/jax/blob/main/tests/polynomial_test.py. Let me know if you have questions about that!
Also, just FYI we have some work in progress that will add some constraints to this code (see #10840): essentially, JAX library code will no longer be able to rely on implicit dtype promotion once that is merged. You can run your tests locally with that condition by setting the flag value JAX_NUMPY_DTYPE_PROMOTION=strict

Alright, I have a couple questions. First of all, many NumPy Chebyshev functions trim trailing zeros before returning, a feature I omitted from the JAX versions for jit compatibility. When comparing against NumPy, should I just wrap JAX-based versions of these functions in a trimzeros function, or is there a better way to test that ignores length differences?
Second, what is the desired behavior for functions under the new dtype promotion constraints? Should functions like chebadd raise an error if the input arrays have different dtypes, should they attempt explicit dtype promotion, or should they do something else?
Thank you!

Excellent question - one strategy we typically use in this situation (where the numpy behavior is incompatible with JIT) is to add an additional keyword argument, set so that the default value matches the behavior of numpy, and so that it can be changed to modify the API to be compatible with JIT (you can see an example in the strip_zeros keyword to jnp.roots, which I just finished revamping in #11215)
I think you could use a very similar treatment here.

Yes, that all makes sense. Thank you!

Regarding precision – I suspect this is due to JAX executing in float32, while numpy executes in float64.
Regarding warnings - you can filter them if they're expected. You can see examples of this elsewhere in the tests, e.g. https://github.com/google/jax/blob/4c25ef1d0039ecdd517ce2b09605322797700c2a/tests/lax_numpy_test.py#L579-L584

Ok, all test pass with JAX_ENABLE_x64=true and JAX_NUMPY_DTYPE_PROMOTION=strict. I also updated applicable functions with a "trim" keyword argument so that they match NumPy behavior by default. What's the next step?
Thank you!

Implemented the changes you requested, and hopefully fixed what caused the chebcompanion test to fail last time.

Made the changes you requested. Also, the Chebyshev test suite takes 75-80 seconds to run on my laptop using the command you provided.

After some more testing, I believe the Chebyshev tests actually contribute a much smaller percentage of total testing time: I ran the full test suite on my laptop yesterday, and it took about 2 hours and 18 minutes, or ~8300 seconds. Running the Chebyshev test file individually took 75-80 seconds, which is 0.9-1% of the total. I just ran the test suite remotely on a much more powerful computer, which completed all the tests in 1270 seconds and the Chebyshev tests alone in 11 seconds, which is ~0.866% of the total.
I'm happy to repeat these tests or try other tests you think would be helpful to verify this ratio. 1% of the testing overhead is still significant, so I've also experimented with reducing the Chebyshev tests, as I noticed that polynomial_tests.py only includes tests for jnp.roots.
Many of the cheb* functions call other cheb* functions - for example, chebpow calls chebadd, so the chebadd tests could potentially be removed. Changes that break chebadd would (hopefully) be caught by the chebpow tests. Another category of functions, the 2d and 3d versions of chebval/chebgrid/chebvander, are simply wrappers for nested calls of the respective 1d functions. Tests for these could also be removed, as the 1d function tests should cover everything.
Removing tests for all functions that fall under the two categories described above cuts the time of the Chebyshev tests in half, and a majority of this cost is due to only a few functions: chebfit, chebdiv, and chebfromroots. Reducing the number of test cases for these particular functions or eliminating tests for them entirely could significantly speed up the Chebyshev tests.
Thank you for your time, let me know what you think!

Thanks - that makes sense. Let's go ahead and merge this and we can deal with any test timeout issues separately. Thanks for all the work here!

The test failures here look real: it looks like chebgrid does some implicit type promotion when it is passed integer inputs.

Ok, the chebroots test is now skipped on GPU and TPU. That test also no longer generates inputs with untrimmed trailing zeros, which I expect will fix the other problem.

Hi - there are still a number of failures in the larger tests. I think it's easier for me to try to patch the imported PR to adjust test tolerances; I'll try to look at that today.

Hi, is there anything else you'd like me to change or add before the workflow is approved?

Hi - sorry I lost track of this over the summer. Let me see if I can land it now

For the sake of bazel-based tests, can you add the following block for the new test file in https://github.com/google/jax/blob/main/tests/BUILD, keeping alphabetical order?
jax_test(
    name = "chebyshev_test",
    srcs = ["chebyshev_test.py"],
)

We're still running into a bunch of tolerance-related failures on GPU and TPU.

Any updates on this? I'm also very interested in using orthogonal polynomials in JAX and was planning on building off of this to implement some of the others from numpy

Any updates on this? I'm also very interested in using orthogonal polynomials in JAX and was planning on building off of this to implement some of the others from numpy

I haven't actually had a chance to work on this since August- I started grad school and just haven't had time, but I'm hoping to investigate the tolerance issues this weekend/over the Thanksgiving holiday. Happy to chat or work with you on this if you have any suggestions or ideas.

Is there any plan for this to continue? I'm looking to use Chebyshev polynomials with JAX on a project.

Wanted to bump this for review- I've updated everything to work with the new version of JAX, and tested on CPU, GPU, and TPU. Made some small adjustments, and now all tests are passing on my end. Thank you!

Hi - sorry we lost track of these. I think given the discussions around https://jax.readthedocs.io/en/latest/jep/12049-type-annotations.html, this would now be considered out-of-scope for jax itself. That said, I think there are a lot of people who would find something like this useful, and you might think about releasing this functionality as part of another package.
Sorry we weren't able to get this merged – thanks!


Update code review team name

[sc-6265] - We're using a sub-team of team-engprod to handle our reviews, to avoid pinging everyone in the main group

Wrong repo. Bad automation (bad matt?) 🤦


Schaffhausen, Switzerland

Another Geoshop order (here: http://gis.sh.ch/GIS_SH/BM3.asp -> Ausgabe -> GeoShop). They seem to have WMS servers as well at http://www.sh.ch/Geodienste.3480.0.html, but I could not connect to them via WFS.

Same encoding issue here, I think.

Re-run this, please.

Hm, looks offset just like https://github.com/openaddresses/openaddresses/pull/2312#issuecomment-270543851. Hope this is a source problem, and not a bug introduced in GDAL 2.1.0.


Plugin doesn't work within Angular ng-repeat directive

 //Magnific-popup

$('.single-popup').magnificPopup({
type: 'image',
removalDelay: 300,
mainClass: 'mfp-with-fade',
closeOnContentClick: true
});
An example from the site copied/pasted works fine, however anything with a databinded href value that is pathed via '../' does not work. Instead it routes my site to the home page. There are no additional functions on the images not working.
The code

I've narrowed down the issue on this, if the  resides within an AngularJS ng-repeat it doesn't work. If I move the databinded  code out of the ng-repeat it works.
Below in the code you see the first magPopup declared outside of the repeat and it works. Then the same html is used in the ng-repeat and those don't work. Any reason or work around?

Should I open a new Issue and close this one?

Hi @Bradmartin0905, thank you for posting this issue, as it clarified a few problems I was having with my own AngularJS app. I'm suffering from the same problem with ng-repeat and found a temporary workaround that I thought I would share with you until this is resolved. Many jQuery plugins don't work with dynamically generated DOM elements (like ng-repeat). I suggest using some sort of $watch along with a $timeout to check when the ng-repeat is loaded, then you initialize magnific-popup.
$scope.$watch("images", function (newValue, oldValue) {
      $timeout(function () {
        $('.image-container').each(function() {
          $(this).magnificPopup({
            delegate: '.image',
            type:'image',
            gallery: {
              enabled: true
            }
          });
        });
      });
    });

This isn't the most elegant solution, but I'm sure it can be improved with time. Good luck figuring it out and hope this helps!

I have this issue to can anyone help us about it?

@hurkanakbiyik Did you have any luck using the $watch as described above? You may not even need the $timeout. Watching your items should allow you to call the library after all of the elements in the repeat have loaded. The issue is with catching the dynamically generated elements.

Nothing to add but @aaackerman 's solution works perfectly, much relief after struggling for a while. Thanks

I had the same issue on ng-repeat.   aaackerman's quick solution saved me a lot of time. Thanks to Alexandra.

@digiArtistph  please can you give me your email address for an important issue

@digiArtistph can you give me any video about your hexapod project or your email

@ELsaman sorry that hexapod project wasn't really mine. i just helped wrote some C++ codes to make it work as expected.

@digiArtistph thanks for your responce , but can you explain me why yo  get the lemb lengths
( FEMUR -12.6 ,TIBIA 92.56 ,TIBIAFEMUR 11.3 ) >> why the femuer with a negative reference

Dear friend!
I need  some help, please ask a couple of  simple questions regarding my  new article, here  they are http://personal.thetootclassic.com/e4vxck
chris

`(function () {
angular
.module(APP.NAME)
.directive('magnificPopupGallery', MagnificPopupGallery);
MagnificPopupGallery.$inject = [];

function MagnificPopupGallery() {
    var directive = {
        restrict: 'A',
        link: _linkFunc
    };

    return directive;

    function _linkFunc(scope, element, attrs) {
        $(element).magnificPopup({
            delegate: '.image',
            type: 'image',
            gallery: {
                enabled: true
            }
        });

    };
}

})();`

`
angular
.module(APP.NAME)
.directive('magnificPopupGallery', MagnificPopupGallery);
MagnificPopupGallery.$inject = [];

function MagnificPopupGallery() {
    var directive = {
        restrict: 'A',
        link: _linkFunc
    };

    return directive;

    function _linkFunc(scope, element, attrs) {
        $(element).magnificPopup({
            delegate: '.image',
            type: 'image',
            gallery: {
                enabled: true
            }
        });

    };
}

'
use the directive above on the parent element


Only Admin can add services to this app

Description
I added Status Pages as an admin, I was able to add services and it will be saved automatically. Next time, I open the app, I can see the services. But if I login as a restricted user, even with nerdpack manager role, the services added by me are not saved. When I come back to the app again, it still shows no service.
NOTE: # ( Describe the problem you're encountering. )
TIP:  # ( Do NOT give us access or passwords to your New Relic account or API keys! )
Steps to Reproduce
Expected Behaviour
Relevant Logs / Console output
Your Environment

NR1 CLI version used:
Browser name and version:
Operating System and version:

Additional context

This is a licensing restriction in the New Relic platform - a user must be at core or full license level to access a nerdpack. Restricted or basic users cannot access nerdpacks regardless of role assignment.


SwiftUI - Initial Selection from UIImagePickerController Does Not Appear

Hello,
Only for the initial render, BSImagePicker does not seem to display the images that Core UIKit UIImagePickerController has provided access to. Re-opening the BSImagePicker does however display them.
I've included the SwiftUI code and a .mov file showing this behavior. How can developers make the initial display of the selected images identical to the behavior when BSImagePicker is re-opened?
I am using 750f26abd0fafbcc4124adef436251888a7c962a git ref. Tag 3.3.1.
Thank you for your help, and thank you for developing this piece of software.
https://user-images.githubusercontent.com/31681334/111056110-88a6a100-844a-11eb-9504-e2621aad8e3c.mov
struct ContentView: View {
    @StateObject var imagePickerCoordinatorView = ImagePickerCoordinatorView()
    @State var isPresentedImagePicker = false

    var body: some View {
        Button("Open BSImagePicker") {
            self.isPresentedImagePicker = true
        }
        .sheet(isPresented: self.$isPresentedImagePicker, content: {
            imagePickerCoordinatorView
        })

        if let list = imagePickerCoordinatorView.selectedAssets {
            Text("\(list.count) images selected")
        } else {
            Text("No images selected")
        }
    }
}

//
//  ImagePickerCoordinatorView.swift

import SwiftUI
import Photos

import BSImagePicker

final class ImagePickerCoordinatorView: NSObject, ObservableObject {
    @Environment(\.presentationMode) var presentationMode: Binding<PresentationMode>

    @Published var selectedAssets: [PHAsset]? {
        didSet {
            print("New value of selectedAssets is \(selectedAssets)")
        }
    }

    func dismiss() {
        self.presentationMode.wrappedValue.dismiss()
    }

    func clearSelectedAssets() {
        self.selectedAssets = [PHAsset]()
    }
}

extension ImagePickerCoordinatorView: UIViewControllerRepresentable {

    public typealias UIViewControllerType = ImagePickerController

    public func makeUIViewController(context: Context) -> ImagePickerController {
        let picker = ImagePickerController()
        picker.settings.selection.max = 10
        picker.settings.selection.unselectOnReachingMax = false
        picker.settings.theme.selectionStyle = .numbered
        picker.settings.fetch.assets.supportedMediaTypes = [.image]
        picker.imagePickerDelegate = context.coordinator
        return picker
    }

    public func updateUIViewController(_ uiViewController: ImagePickerController, context: Context) {
    }

    public func makeCoordinator() -> Coordinator {
        return Coordinator(self)
    }
}

extension ImagePickerCoordinatorView {
    public class Coordinator: ImagePickerControllerDelegate {
        private let parent: ImagePickerCoordinatorView

        public init(_ parent: ImagePickerCoordinatorView) {
            self.parent = parent
        }

        public func imagePicker(_ imagePicker: ImagePickerController, didSelectAsset asset: PHAsset) {
            print("Selected: \(asset)")
        }

        public func imagePicker(_ imagePicker: ImagePickerController, didDeselectAsset asset: PHAsset) {
            print("Deselected: \(asset)")
        }

        public func imagePicker(_ imagePicker: ImagePickerController, didFinishWithAssets assets: [PHAsset]) {
            print("Finished with selections: \(assets)")
            parent.selectedAssets = assets
            parent.dismiss()
        }

        public func imagePicker(_ imagePicker: ImagePickerController, didCancelWithAssets assets: [PHAsset]) {
            print("Canceled with selections: \(assets)")
            if (assets.count == 0) {
                print("Canceled with no selection so setting selectedAssets to empty")
                parent.selectedAssets?.removeAll()
            }
            parent.dismiss()
        }

        public func imagePicker(_ imagePicker: ImagePickerController, didReachSelectionLimit count: Int) {
            print("Did Reach Selection Limit: \(count)")
        }
    }
}

If you are using the delegate and presenting the image picker yourself you also need to request permission to the photo library before presenting the picker.
Similar discussion here: https://github.com/mikaoj/BSImagePicker/issues/319

I have the same issue. Any update on this?

@AntonClaesson did you find any solution for this. I am stuck now at this point for more than 3 days.


add test coverage for repo.ts

Coverage: 94.286% (+0.02%) from 94.265% when pulling<PHONE_NUMBER>d10cb0906ca8ed976ce2f038b06225 on tony-add-test-coverage-repo into dcb6d789b504e543a94864afb9c8e20dd3ca2e98 on main.