Increase broadcast and unicast table sizes

The broadcast table size increase should help with NETWORK_BUSY errors. This is the table EmberZNet uses to determine if there are too many broadcasts happening at once.
The unicast table should help enqueue more packets.

For ease of use, here are firmwares for some common sticks:

skyconnect_ncp-uart-hw_<IP_ADDRESS>.gbl.zip
yellow_ncp-uart-hw_<IP_ADDRESS>.gbl.zip
sonoff-e_ncp-uart-hw_<IP_ADDRESS>.gbl.zip

GitHub does not allow .gbl files so I have zipped them. Please unzip them and flash them with an appropriate tool:

SkyConnect: https://skyconnect.home-assistant.io/firmware-update/
Sonoff: https://darkxst.github.io/silabs-firmware-builder/
Yellow: not possible just yet without manually flashing using universal-silabs-flasher.

@jclendineng Let's move firmware discussions over here. If your firmware is showing up as <IP_ADDRESS>, it didn't flash. It'll show up as Firmware: <IP_ADDRESS> build 0.

Feedback has been positive so far. Merging.

Per https://github.com/home-assistant/core/issues/86411 this is a massive improvement but seems to have introduced an issue with group behavior. To summarize from that thread:
Random members of groups don't respond to group commands. Which members of the group fail to respond is random with each command sent. This never happened prior to this build. Groups addressed via a binding on an inovelli blue switch always respond in unison. Only commands to groups from the coordinator since this build exhibit the random failure. I'm not sure how the changes made in this build could have introduced this but I am certain this issue was introduced in this build.

I agree; after installing the latest build it started getting wonky again. Now that could also just be coincidence. On Jul 4, 2024, at 10:04 AM, Andrew M @.***> wrote:
Per home-assistant/core#86411 this is a massive improvement but seems to have introduced an issue with group behavior. To summarize from that thread:
Random members of groups don't respond to group commands. Which members of the group fail to respond is random with each command sent. This never happened prior to this build. Groups addressed via a binding on an inovelli blue switch always respond in unison. Only commands to groups from the coordinator since this build exhibit the random failure. I'm not sure how the changes made in this build could have introduced this but I am certain this issue was introduced in this build.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>


docs: add project guidelines

Summary of Changes
Add guidelines for documentation, tests, and code style.

:tada: This PR is included in version 0.17.0 :tada:
The release is available on:

v0.17.0
GitHub release

Your semantic-release bot :package::rocket:


feat(report): route statistic

Collect statistics of configured routers and runtime usage:

How many routes are configured;
How many paths are configured for routes;
How many headers are configured for routes;
The used router flavor;
How many routes are configured for different kinds of protocols;
How many routes are configured with different path handling;
How many regex paths matched

Configured route statistics are collected every time the router is rebuilt. The statistics are sent along with the ping.
Fix FT-3217

Since router's code has been refactored (PR #9331), we should rebase and change some code.

do we need changelog entry for this? @StarlightIbuki


[Dropdown] Multiple select only returning last selected value

I'm trying to implement a form which allows a user to select multiple values and submit them. However, against what is specified in the docs, it is not creating delimited values. I am only getting the last selected in the results.
I am using php and jquery. Both are the latest versions, as well as the latest version of semantic UI. I have even copied and pasted the examples from the docs with no luck. I'm not sure if this is a bug or the documentation has missed a flag I should include.
According to the docs, I should just be able to do the following and it will work:

<select id="list" class="ui search selection multiple dropdown" name="list" placeholder="Select 1 or more options..." multiple="multiple">
  <?= $listoptions ?> // some list of options, as you'd expect. Nothing special.
</select>

Say I selected 1, 2 & 3, the last one I selected would be the only one available in the $_POST variable. Is this a bug or am I/the docs missing something?
EDIT: pretty much exactly as above but was requested, code on JSFiddle. You can also hit the 'run code' button under the multiple selection heading on the docs and see it isn't creating the input field it's meant to be as specified in the code below it. Multiple Selection Documentation

Hi, please provide a test case as per our contributing guidelines; you can fork this JSFiddle.

JSFiddle link
Couldn't get any more generic than that. When receiving the data at the other end, I only get the last value selected.

@sbaker7 Thank you. The browser submits the data:

Please change name="list" to name="list[]" and let me know if it helped.

@sbaker7
I’ve just read your comment on Gitter and realized that I missed your other question; you’re not getting a hidden input with comma-delimited values when you initialize a dropdown directly on a <select> element, because a dropdown then uses its <select> element to manipulate values.
Another option is to initialize a dropdown on pre-existing HTML, in which case it will use a hidden input to store values with a delimiter. Other than that, the difference between the two is only in performance—it’s faster if it doesn’t need to build a dropdown’s HTML for you.

Hi Banandrew, the square brackets work fine for my case, thank you. I didn't mind how it sent the values, I just wanted to make sure I got all selected values when the form submitted. This however doesn't match up with the idea of using a delimiter as it just posts an array of values and doesn't match the docs.
The only way I got it to work with the delimiter was actually to write my own function on onChange() to set a hidden input. I could have just been doing something wrong (most likely) but trying to initialise dropdown on even the provided code on the website didn't seem to work for me.
I suppose my main issue is that it'd make sense to make sure the docs are updated to demonstrate working cases. They do work to demonstrate selecting multiple values, but it certainly doesn't work from a form submission point of view where you actually collect data. Examples should include the brackets to return an array and/or initialising so the delimited hidden input that supposed to occur according to the docs actually does occur. It's all good to be able to select values but if you can't use those values, then it becomes a little pointless.

@sbaker7
Sorry for keeping you waiting. There’s no need to do anything inside of the onChange() callback, like I said, if you want to use a delimiter, you need a hidden input that’s available only when you use pre-existing HTML. The following initialized pattern will send the data with a delimiter:

  <input type="hidden" name="list">

Default text

Text

$('.ui.dropdown')
  .dropdown({
    delimiter: '|'
  })

------WebKitFormBoundaryHuceWyeb0fGIsBhQ
Content-Disposition: form-data; name="list"

a|b|c

I’ll submit a PR to the documentation repository in the next few days to improve the part that was confusing, but I don’t think we should be putting square brackets everywhere, because it’s not the framework’s concern—it’s a feature of the PHP language (some other technologies allow you to do it as well, but not all); it will be wrong to enforce the PHP view of things, since the HTTP protocol doesn’t know anything about arrays, nor it cares about brackets in general, and SUI can be used with any kind of back-end technology.


bed files are not sorted

Hi,
I noticed that the *unique_2.bed files generated with the reduced_genome.sh script are not sorted correctly (by chromosome and then start position), this breaks the step where you remove the self-ligated and undigested fragments where you find the sequence before and after the bait.  You should include a sorting step in your script.

Hi,
Line 28 of the reduced_genome.sh does sort the file by chr and start position before getting the unique lines. Did you have any other trouble running the script?

 It looks like I was using an older version of your script since I noticed that you updated that particular line on Aug 18, 2016.  I checked and in our script it just says sort not sort -k1,1 -k2,2n.   I think I might have to update your code and rerun the analysis because we were getting some errors as well with nearbait, cis, and trans.  For trans analysis, I believe however I was using a more recent version of the code installed in my laptop and for some of my samples it gave a warning that it couldn't solve the Hessian and that only 1 iteration was done.


refactor for proper defer

in old implementation, defer resource release within for loop will be executed outside for loop, this might starve queries (after first allocated one) in the same request.
this diff makes sure devices are released after processing each query.


[SUPPORT] Delete not functioning with deltastreamer

Tips before filing an issue

Have you gone through our FAQs?

Join the mailing list to engage in conversations and get faster support at<EMAIL_ADDRESS>

If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced
A clear and concise description of the problem.
To Reproduce
Steps to reproduce the behavior:
1.Perform a spark submit job to insert data into Hudi table saved in S3. I am using the Hudi Delta Streamer utility with the "hoodie_is_deleted" column set to "False" for all the records. This works successfully. The data is in parquet file in S3.The parquet file is generated from a panda data frame which is a different process(not part of the spark submit).
2. Perform a spark submit job to delete data from the existing Hudi table saved in S3.I am using the Hudi Delta Streamer utility with the "hoodie_is_deleted" column set to "True"
for all the records that need to be deleted. The delete file is in parquet format in S3 and is generated from a panda data frame which is a different process(not part of the spark submit). This now throws an error and I see roll back files in .hoodie folder.
3. I am running this in Spark on Kubernetes.
https://spark.apache.org/docs/3.1.1/configuration.html#kubernetes
Expected behavior
I expect to see deletion of the row with "hoodie_is_deleted=True"in the existing Hudi table in S3.
A clear and concise description of what you expected to happen.
Environment Description

Hudi version :0.8.0

Spark version :3.1.1
https://spark.apache.org/docs/3.1.1/configuration.html#kubernetes

Hive version :

Hadoop version :3.2

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) : yes

Additional context
SPARK SUMBIT COMMAND FOR INITIAL LOAD TO HUDI TABLE(THIS ONE WORKS SUCCESSFULLY):
./spark-submit --master k8s://https://..sk1.us-west-1.eks.amazonaws.com
--deploy-mode cluster
--name spark-hudi
--jars https://repo1.maven.org/maven2/org/apache/spark/spark-avro_2.12/3.1.1/spark-avro_2.12-3.1.1.jar
--class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer
--conf spark.executor.instances=1
--conf spark.kubernetes.container.image=../spark:spark-hudi-0.2
--conf spark.kubernetes.namespace=spark-k8
--conf spark.kubernetes.container.image.pullSecrets=dockercloud-secret
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
--conf spark.hadoop.fs.s3a.endpoint=s3.us-west-1.amazonaws.com
--conf spark.hadoop.fs.s3a.access.key='A...R'
--conf spark.hadoop.fs.s3a.secret.key='L..gG'
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer s3a://lightbox-sandbox-dev/hudi-root/spark-submit-jars/hudi-utilities-bundle_2.12-0.8.0.jar
--table-type COPY_ON_WRITE --source-ordering-field two
--source-class org.apache.hudi.utilities.sources.ParquetDFSSource
--target-base-path s3a://.../hudi-root/transformed-tables/hudi_writer_mm4/
--target-table test_table --base-file-format PARQUET
--hoodie-conf hoodie.datasource.write.recordkey.field=two
--hoodie-conf hoodie.datasource.write.partitionpath.field=two
--hoodie-conf hoodie.datasource.write.precombine.field=two
--hoodie-conf hoodie.deltastreamer.source.dfs.root=s3a://../jen/example_2.parquet
SPARK SUMBIT COMMAND FOR INITIAL LOAD TO HUDI TABLE(THIS ONE THROWS ERROR):
./spark-submit --master k8s://https://..sk1.us-west-1.eks.amazonaws.com
--deploy-mode cluster
--name spark-hudi
--jars https://repo1.maven.org/maven2/org/apache/spark/spark-avro_2.12/3.1.1/spark-avro_2.12-3.1.1.jar
--class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer
--conf spark.executor.instances=1
--conf spark.kubernetes.container.image=../spark:spark-hudi-0.2
--conf spark.kubernetes.namespace=spark-k8
--conf spark.kubernetes.container.image.pullSecrets=dockercloud-secret
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
--conf spark.hadoop.fs.s3a.endpoint=s3.us-west-1.amazonaws.com
--conf spark.hadoop.fs.s3a.access.key='A...R'
--conf spark.hadoop.fs.s3a.secret.key='L..gG'
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer s3a://lightbox-sandbox-dev/hudi-root/spark-submit-jars/hudi-utilities-bundle_2.12-0.8.0.jar
--table-type COPY_ON_WRITE --source-ordering-field two
--source-class org.apache.hudi.utilities.sources.ParquetDFSSource
--target-base-path s3a://.../hudi-root/transformed-tables/hudi_writer_mm4/
--target-table test_table --base-file-format PARQUET
--hoodie-conf hoodie.datasource.write.recordkey.field=two
--hoodie-conf hoodie.datasource.write.partitionpath.field=two
--hoodie-conf hoodie.datasource.write.precombine.field=two
--hoodie-conf hoodie.deltastreamer.source.dfs.root=s3a://../jen/example_upsert2.parquet
Add any other context about the problem here.
Code to generate the parquet files that I am using:
New insert parquet file(example2.parquet)
import pyarrow.parquet as pq
import numpy as np
import pandas as pd
import pyarrow as pa
df = pd.DataFrame({'one': [-1, 3, 2.5],
'two': [100, 101, 103],
'three': [True, True, True],
'_hoodie_is_deleted': [False, False, False]},
index=list('abc'))
table = pa.Table.from_pandas(df)
print(table)
pq.write_table(table, 'example_2.parquet')
Code to generate the parquet file that I am using:
Upsert/delete parquet file(example_upsert2.parquet)
import pyarrow.parquet as pq
import numpy as np
import pandas as pd
import pyarrow as pa
df = pd.DataFrame({'one': [-1, 3, 2.5],
'two': [100, 101, 103],
'three': [True, True, True],
'_hoodie_is_deleted': [False, False, True]},
index=list('abc'))
table = pa.Table.from_pandas(df)
print(table)
pq.write_table(table, 'example_upsert2.parquet')
Stacktrace
Stack trace from the Kubernetes dashboard is attached.

Add the stacktrace of the error.

have you configured your record keys and partition path correctly ?
I see from your configs, you have given "two" for all 3 configs(record keys, partition path and preCombine).
Ideally all these should be referring to 3 different columns in incoming dataframe.

Also, have you tried doing an update in general. Once that succeeds, then we know delete has some issue. but if update is failing, then could be some config issue or we might have to investigate further.

I changed the input data set a bit so that I can provide 3 different fields for the 3 differnt mentioned configs.
./spark-submit --master k8s://https://..sk1.us-west-1.eks.amazonaws.com
--deploy-mode cluster
--name spark-hudi
--jars https://repo1.maven.org/maven2/org/apache/spark/spark-avro_2.12/3.1.1/spark-avro_2.12-3.1.1.jar
--class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer
--conf spark.executor.instances=1
--conf spark.kubernetes.container.image=../spark:spark-hudi-0.2
--conf spark.kubernetes.namespace=spark-k8
--conf spark.kubernetes.container.image.pullSecrets=dockercloud-secret
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
--conf spark.hadoop.fs.s3a.endpoint=s3.us-west-1.amazonaws.com
--conf spark.hadoop.fs.s3a.access.key='A...R'
--conf spark.hadoop.fs.s3a.secret.key='L..gG'
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer s3a://lightbox-sandbox-dev/hudi-root/spark-submit-jars/hudi-utilities-bundle_2.12-0.8.0.jar
--table-type COPY_ON_WRITE
--source-ordering-field ts
--source-class org.apache.hudi.utilities.sources.ParquetDFSSource
--target-base-path s3a://lightbox-sandbox-dev/hudi-root/transformed-tables/hudi_writer_mm4/
--target-table test_table --base-file-format PARQUET
--hoodie-conf hoodie.datasource.write.recordkey.field=uuid
--hoodie-conf hoodie.datasource.write.partitionpath.field=two
--hoodie-conf hoodie.datasource.write.precombine.field=ts
--hoodie-conf hoodie.deltastreamer.source.dfs.root=s3a://../jen/example_upsert2.parquet
My input parquet file has below columns:
import pyarrow.parquet as pq
import numpy as np
import pandas as pd
import pyarrow as pa
import uuid
df = pd.DataFrame({'uuid': [str(uuid.uuid4()), str(uuid.uuid4()),str(uuid.uuid4())],
'ts': [1, 2, 3],
'two': [100, 101, 103],
'three': [True, True, True],
'_hoodie_is_deleted': [False, False, False]},
index=list('abc'))
table = pa.Table.from_pandas(df)
print(table)
pq.write_table(table, 'example_5.parquet')
The above initial insert into Hudi table works successfully.
I then tried to perform an update this time as you mentioned:
import pyarrow.parquet as pq
import numpy as np
import pandas as pd
import pyarrow as pa
import uuid
df = pd.DataFrame({'uuid': [str(uuid.uuid4()), str(uuid.uuid4()),str(uuid.uuid4())],
'ts': [1, 2, 3],
'two': [100, 101, 103],
'three': [True, True, False],
'_hoodie_is_deleted': [False, False, False]},
index=list('abc'))
table = pa.Table.from_pandas(df)
print(table)
pq.write_table(table, 'example_upsert5.parquet')
SPARK SUBMIT FOR UPSERT:
./spark-submit --master k8s://https://..sk1.us-west-1.eks.amazonaws.com
--deploy-mode cluster
--name spark-hudi
--jars https://repo1.maven.org/maven2/org/apache/spark/spark-avro_2.12/3.1.1/spark-avro_2.12-3.1.1.jar
--class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer
--conf spark.executor.instances=1
--conf spark.kubernetes.container.image=../spark:spark-hudi-0.2
--conf spark.kubernetes.namespace=spark-k8
--conf spark.kubernetes.container.image.pullSecrets=dockercloud-secret
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
--conf spark.hadoop.fs.s3a.endpoint=s3.us-west-1.amazonaws.com
--conf spark.hadoop.fs.s3a.access.key='A...R'
--conf spark.hadoop.fs.s3a.secret.key='L..gG'
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer s3a://lightbox-sandbox-dev/hudi-root/spark-submit-jars/hudi-utilities-bundle_2.12-0.8.0.jar
--table-type COPY_ON_WRITE
--source-ordering-field ts
--source-class org.apache.hudi.utilities.sources.ParquetDFSSource
--target-base-path s3a://lightbox-sandbox-dev/hudi-root/transformed-tables/hudi_writer_mm4/
--target-table test_table --base-file-format PARQUET
--hoodie-conf hoodie.datasource.write.recordkey.field=uuid
--hoodie-conf hoodie.datasource.write.partitionpath.field=two
--hoodie-conf hoodie.datasource.write.precombine.field=ts
--hoodie-conf hoodie.deltastreamer.source.dfs.root=s3a://../jen/example_upsert5.parquet
This one fails with error. The stack trace from Kubernetes dashboard is attached.
logs-from-spark-kubernetes-driver-in-spark-hudi-7c40487ae63865bc-driver.txt

Does the source ordering config and precombine field has to be a timestamp field?

Does the source ordering config and precombine field has to be a timestamp field?

Not necessary, a Comparable is enough.

@mithalee Can you try with the latest master branch? I built the master code and tried to reproduce the scenario in a local docker environment. It runs fine. For example, after first ingest, you can see _hoodie_is_deleted is false for both timestamp and after second ingest (in which I set _hoodie_is_deleted to true for a timestamp), it is present only for one timestamp.
// after first ingest
scala> spark.sql("select symbol, ts, _hoodie_is_deleted from stock_ticks_cow WHERE symbol = 'MSFT'").show(100, false)
+------+-------------------+------------------+
|symbol|ts                 |_hoodie_is_deleted|
+------+-------------------+------------------+
|MSFT  |2018-08-31 09:59:00|false             |
|MSFT  |2018-08-31 10:29:00|false             |
+------+-------------------+------------------+

// after second ingest
scala> spark.sql("select symbol, ts, _hoodie_is_deleted from stock_ticks_cow WHERE symbol = 'MSFT'").show(100, false)
+------+-------------------+------------------+
|symbol|ts                 |_hoodie_is_deleted|
+------+-------------------+------------------+
|MSFT  |2018-08-31 09:59:00|false             |
+------+-------------------+------------------+

My schema is similar to this except that I added _hoodie_is_deleted field with default false.

It would be a regression if its failing in 0.8.0 bcoz, don't think we touched this path in 0.8.0. @codope : Can you please test it out w/ 0.8.0.

@mithalee Can you try with the latest master branch? I built the master code and tried to reproduce the scenario in a local docker environment. It runs fine. For example, after first ingest, you can see _hoodie_is_deleted is false for both timestamp and after second ingest (in which I set _hoodie_is_deleted to true for a timestamp), it is present only for one timestamp.
// after first ingest
scala> spark.sql("select symbol, ts, _hoodie_is_deleted from stock_ticks_cow WHERE symbol = 'MSFT'").show(100, false)
+------+-------------------+------------------+
|symbol|ts                 |_hoodie_is_deleted|
+------+-------------------+------------------+
|MSFT  |2018-08-31 09:59:00|false             |
|MSFT  |2018-08-31 10:29:00|false             |
+------+-------------------+------------------+

// after second ingest
scala> spark.sql("select symbol, ts, _hoodie_is_deleted from stock_ticks_cow WHERE symbol = 'MSFT'").show(100, false)
+------+-------------------+------------------+
|symbol|ts                 |_hoodie_is_deleted|
+------+-------------------+------------------+
|MSFT  |2018-08-31 09:59:00|false             |
+------+-------------------+------------------+

My schema is similar to this except that I added _hoodie_is_deleted field with default false.
FYI, my spark-submit command is same as mentioned here.

Sure.I will try and get back to you.

@codope Hi, I did try the HoodieDeltaStreamer on below version of EMR:
Release label:emr-6.3.0
Hadoop distribution:Amazon 3.2.1
Applications:Tez 0.9.2, Spark 3.1.1, Hive 3.1.2, JupyterHub 1.2.0, Sqoop 1.4.7, Zeppelin 0.9.0, Hue 4.9.0, Presto 0.245.1
HUDI Utility: https://mvnrepository.com/artifact/org.apache.hudi/hudi-utilities-bundle_2.12/0.8.0
The HoodieDeltaStreamer works as expected(upserts/deletes).  But the same data set and same spark submit configuration does not work on the Spark on K8 binaries.
https://spark.apache.org/docs/3.1.1/submitting-applications.html
I can perform the initial insert into Hudi table through the below spark submit but the upserts/deletes are throwing error. Can you confirm if the upserts /deletes works using the DS in Spark on K8s.
SPARK SUBMIT
spark-submit --master k8s://https://...sk1.us-west-1.eks.amazonaws.com
--deploy-mode cluster
--name spark-hudi
--jars https://repo1.maven.org/maven2/org/apache/spark/spark-avro_2.12/3.1.1/spark-avro_2.12-3.1.1.jar
--class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer
--conf spark.executor.instances=1
--conf spark.kubernetes.container.image=d.../spark:spark-hudi-0.2
--conf spark.kubernetes.namespace=spark-k8
--conf spark.kubernetes.container.image.pullSecrets=dockercloud-secret
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
--conf spark.hadoop.fs.s3a.endpoint=s3.us-west-1.amazonaws.com
--conf spark.hadoop.fs.s3a.access.key='A........'
--conf spark.hadoop.fs.s3a.secret.key='L.........'
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer
s3a://l...v/hudi-root/spark-submit-jars/hudi-utilities-bundle_2.12-0.8.0.jar
--table-type COPY_ON_WRITE --source-ordering-field one
--source-class org.apache.hudi.utilities.sources.ParquetDFSSource
--target-base-path s3a://......./hudi-root/transformed-tables/hudi_writer_mm7/
--target-table test_table --base-file-format PARQUET
--hoodie-conf hoodie.datasource.write.recordkey.field=one
--hoodie-conf hoodie.datasource.write.partitionpath.field=two
--hoodie-conf hoodie.deltastreamer.source.dfs.root=s3a://....../jen/example_4.parquet
Parquet file example_4.parquet  :
import pyarrow.parquet as pq
import numpy as np
import pandas as pd
import pyarrow as pa
import uuid
df = pd.DataFrame({'one': [-1, 3, 2.5],
'two': [100, 200, 300],
'three': [True, True, True],
'_hoodie_is_deleted': [False, False, False]},
index=list('abc'))
table = pa.Table.from_pandas(df)
print(table)
pq.write_table(table, 'example_4.parquet')
Delete file:
import pyarrow.parquet as pq
import numpy as np
import pandas as pd
import pyarrow as pa
import uuid
df = pd.DataFrame({'one': [-1, 3, 2.5],
'two': [100, 200, 300],
'three': [True, True, True],
'_hoodie_is_deleted': [True, False, False]},
index=list('abc'))
table = pa.Table.from_pandas(df)
print(table)
pq.write_table(table, 'example_delete4.parquet')

hmmm, could be. I vaguely remember very few people in the community trying to use hudi in k8s. haven't heard any issues from them or I am not sure if there weren't any issues and hence no users complained. sorry couldn't be of much help here.

@nsivabalan @codope Hi, I found the issue. Spark on K8 3.1.1 has some bugs and throws the "Premature Content" error. I tried the Spark on K8 3.1.2 which is the latest release and the Hudi DS upserts work perfectly fine. We can close this ticket. Thank you for your help.
https://spark.apache.org/docs/latest/running-on-kubernetes.html


MP4 "free" chunks can cause very slow parsing

I ran into a couple of small (~80 MB) MP4 files which took about 30 seconds each (in debug builds) for MediaInfoLib to parse, compared to 1-2 seconds for normal files.
I can send them privately if you want. Just let me know where to upload them.
After some investigation, it appears to be because they have "free" chunks/blocks which MediaInfoLib parses in a very slow way. It seems to skip 4 bytes of them at a time, and the blocks are quite large in these files. The files also seem to have a lot of the "free" blocks for some reason, which results in repetitive calls to ZtringListList::Find. Since that Find method does linear lookups, with about 300 strings in the list, it accounted for a lot of the processing time.
I found what seems to be a fix, but am not sure enough that it is correct, so I wanted to raise this as an issue rather than a pull request. (I can make this a pull request if you want, though, of course!)
My fix was to change moov_udta_xxxx() so it returns immediately if it's a "free" block:
void File_Mpeg4::moov_udta_xxxx()
{
    if (Element_Code == Elements::free)
        return;

    //Getting the method
    ...

That makes the files process very quickly and I haven't noticed any side effects, but I don't know the code well enough to be sure it's the right solution.
Before doing that, I also tried making ZtringListList::Find check if it was being asked to find "free" and fail instantly, as a quick test, and that improved speed quite a bit, but not as much as skipping the "free" blocks entirely. So the repeated linear lookups were slowing things down a lot, but other things were involved as well.
(I was thinking about making ZtringListList cache the previous Find to speed up cases where it was called for the same thing again, or adding an std::map for faster lookups, but it doesn't seem needed with the moov_udta_xxxx change skipping "free" blocks entirely. Maybe there are other situations where those would help, but maybe the overheads of maintaining the extra data would cost more in typical usage. I don't know the project well enough to say.)
Many thanks for your time!

I just realised I hadn't tried these file with a standard/official release of MediaInfo.
Just downloaded the Windows build of MediaInfo GUI 20.09 and it opens both files instantly.
The problem might only happen with certain compilation flags defined, in that case. If you're interested, I can try to track down which are doing it.

I just realised I hadn't tried these file with a standard/official release of MediaInfo.
Just downloaded the Windows build of MediaInfo GUI 20.09 and it opens both files instantly.
The problem might only happen with certain compilation flags defined, in that case. If you're interested, I can try to track down which are doing it.

I didn't expect to get free atoms in metadata, but it makes sense (edition etc). Even if not visible in release mode, it makes sense to sip such atom.
You may add the free atom in the list in File_Mpeg4::Data_Parse(), and map it to File_Mpeg4::free().
It is quick to do for who knows the code, don't hesitate to drop me an email with a link to download privately the file or I may send you a private FTP server for upload.

I didn't expect to get free atoms in metadata, but it makes sense (edition etc). Even if not visible in release mode, it makes sense to sip such atom.
You may add the free atom in the list in File_Mpeg4::Data_Parse(), and map it to File_Mpeg4::free().
It is quick to do for who knows the code, don't hesitate to drop me an email with a link to download privately the file or I may send you a private FTP server for upload.

Many thanks! I've uploaded the files to a web server and emailed the link to you. Give me a shout if the mail doesn't arrive.

Many thanks! I've uploaded the files to a web server and emailed the link to you. Give me a shout if the mail doesn't arrive.

I was trying to parse the big free atom as if this is a sequence of tag elements, and as it is a full of zeroes the parser was slow... In debug mode only, because the debug mode is slow with the underlying function calls.
But... This code is actually skipped in the current dev version.
I did a quick and dirty fix https://github.com/MediaArea/MediaInfoLib/pull/1354 for being sure that even if 1 day I want to parse such atoms I don't do the same mistake.
Note that metadata present in the file (the "Xtra" atom from Microsoft) is not parsed, I don't have the time for the moment for implementing that, maybe one day...

I was trying to parse the big free atom as if this is a sequence of tag elements, and as it is a full of zeroes the parser was slow... In debug mode only, because the debug mode is slow with the underlying function calls.
But... This code is actually skipped in the current dev version.
I did a quick and dirty fix https://github.com/MediaArea/MediaInfoLib/pull/1354 for being sure that even if 1 day I want to parse such atoms I don't do the same mistake.
Note that metadata present in the file (the "Xtra" atom from Microsoft) is not parsed, I don't have the time for the moment for implementing that, maybe one day...

I can confirm this works on my test files and my local build/setup, after removing my hacky fix and merging just the #1354 changes into 20.09. Looks good!

I can confirm this works on my test files and my local build/setup, after removing my hacky fix and merging just the #1354 changes into 20.09. Looks good!


Make Device constructor more generic

The previous version of Device.__init__ took three positional arguments:
class name ('tacho-motor'), device name pattern ('motor*' or just '*'),
and port name ('outA').
The new version takes two positional arguments (class name and device
name) and an arbitrary set of keyword arguments. The keyword arguments
are used for matching against device attributes. A match is considered a
match when the provided value is a substring of attribute value. If a
list is provided as keyword argument, then a match against any single
entry of the list is enough.
Compare the previous call:
d = Device('tacho-motor', 'outA', 'motor*')

with the new calls
# Get a motor on output port A:
d = Device('tacho-motor', 'motor*', port_name='outA')

# Get large motor on output port A:
m = Device('tacho-motor', port_name='outA', driver_name='lego-ev3-l-motor')

# Get an ultrasonic sensor (don't care if EV3 one or NXT one):
m = Device('lego-sensor', driver_name=['lego-ev3-us', 'lego-nxt-us'])

This would allow for easy implementation of such classes as LargeMotor
or MediumMotor.  This also makes Device constructor interface compatible
with boostc version of python bindings.

Can the default device name, eg motor* be generated from the systemDeviceNameConvention property in the spec? When I created the binding I was using the generic device name, the port_name parameter was needed for things like the LEDs that don't have arbitrary device names from the spec. How do we plan to instantiate LEDs (for example) using these methods?
Ah, I see it's the optional second parameter - OK, I'll merge this as well - and I like what you've done with the kwargs matching. All specified kwargs have to match, and if a specific arg is a list, then any match in the list counts just for that keyword. Very elegant!

Can the default device name, eg motor* be generated from the systemDeviceNameConvention property in the spec?

I thought about that. The propety has format motor{0}. By coincidence, we directly use it in python as 'motor{0}'.format('*'). But I am not sure we can rely on this.

Ah, I see it's the optional second parameter

Its not optional second parameter, its part of kwargs. So yes, we can still specify port_name. Moreover, now its also possible to implement sensor classes correctly, which require to pass driver_name as well. i2c_sensor even needs an address.

I was talking about the LED devices, for example, that do not have a port_name - they would use the second parameter as the filename, in the same way that motor* as the second parameter means "pick the first motor device you find".
I really do like the improvement in the call for device instantiation!


missing information from etherscan

@dawsbot how would you like me to handle this.
(all images are hyper links to their origin)

merged and closed #79


Pull out dates from GH data

We'll need to do something like this:
unlist(purrr::map(repo_list, ~ .x$timestamp))
and make date ranges for the github metrics collected by get_github_metrics()
Because the data generally looks like this:
{
  "count": 271,
  "uniques": 28,
  "views": [
    {
      "timestamp": "2024-01-05T00:00:00Z",
      "count": 30,
      "uniques": 16
    },
    {
      "timestamp": "2024-01-06T00:00:00Z",
      "count": 1,
      "uniques": 1
    },
    {
      "timestamp": "2024-01-07T00:00:00Z",
      "count": 2,
      "uniques": 2
    },
    {
      "timestamp": "2024-01-08T00:00:00Z",
      "count": 1,
      "uniques": 1
    },
    {
      "timestamp": "2024-01-09T00:00:00Z",
      "count": 5,
      "uniques": 2
    },
...

I came up with something like this:
test <-ari_raw_data %>% map(~ unlist(.))

get_deeper_gh_stats <- function(data, column){
  new_data <- data[column] %>%
  bind_rows() %>% select(-c(count, uniques)) %>%
  matrix( ncol = 3, byrow = TRUE) %>%
  as.data.frame()
  colnames(new_data)<- c("timestamp", "count", "uniques")
  return(new_data)}

clones_data <-get_deeper_gh_stats(test, column = "clones")
views_data <-get_deeper_gh_stats(test, column = "views")`

Which creates:

still figuring out how to add this to your code...

No worries! I can handle that part! #Teamwork


Text composables: Android Basics with Compose

URL of codelab
https://developer.android.com/codelabs/basic-android-kotlin-compose-classes-and-objects
In which task and step of the codelab can this issue be found?
Describe the problem
smartLightDevice.increaseBrightNess()
but it was wrote above as
smartLightDevice.increaseBrightness()

@dendude
The issue that you presented to us is very important, and I thank you for bringing it to our attention. The correct variant smartLightDevice.increaseBrightness() will be published soon.


Update scalafmt-core to 2.4.1

Updates org.scalameta:scalafmt-core from 2.3.2 to 2.4.1.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scalameta", artifactId = "scalafmt-core" } ]

labels: library-update, semver-minor

The reformatted code doesn't compile anyway.  It's turning assignments into named arguments.
We'll try again in 2.4.2.


Document current HTTP 207 handling

Adds @ePaul 's nice assessment of the status quo to the API docs.
Relates to #415

Codecov Report
Patch and project coverage have no change.

Comparison is base (1f982ee) 69.76% compared to head (a5ed6fb) 69.76%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff            @@
##               main     #416   +/-   ##
=========================================
  Coverage     69.76%   69.76%
  Complexity      631      631
=========================================
  Files           114      114
  Lines          2487     2487
  Branches        179      179
=========================================
  Hits           1735     1735
  Misses          622      622
  Partials        130      130

Flag
Coverage Δ

unittests
69.76% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed
Coverage Δ

...main/java/org/zalando/fahrschein/NakadiClient.java
73.61% <ø> (ø)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@otrosien could we also add it to README as a separate paragraph?

@adyach done.

👍

👍


[Docs/Examples] Data Fetching Example has Race Condition

Version
3.0.2
Reproduction link
https://router.vuejs.org/guide/advanced/data-fetching.html#fetching-after-navigation
Steps to reproduce
Load the page with ?id=abc
Have the initial getPost request take 3 seconds to complete
After 1 second, navigate to the page again, but with ?id=def
Have the subsequent getPost request take 1 second to complete
Wait 2 seconds
What is expected?
We show the data for id=def
What is actually happening?
We show the data for id=abc

I'm not sure what the correct fix is

We coulduse a closure to check if we are still on that id's page:
  methods: {
    fetchData () {
      this.error = this.post = null
      this.loading = true
      const id = this.$route.params.id
      // replace `getPost` with your data fetching util / API wrapper
      getPost(id, (err, post) => {
        this.loading = false
        if (err) {
          this.error = err.toString()
        } else if (id === this.$route.params.id) {
          this.post = post
        }
        // if the params id changed, the data we loaded is stale and we don't do anything.
      })
    }
  }


MCrunch cannot run in MATLAB R2022a version

Hi
MCrunch can't run with  CertTest.m

System environment and software version information

Steps to reproduce the behavior:

Windows system
MATLAB R2022a version
Run CertTest. m under the CertTest folder
MATLAB cannot run script file
See the error

Error prompt

CertTest
=======
Test_ 01
=======
Running MCrunch (v1.000.00y-mlb, 04-May-2009)
Reading "DLC2.3_1. out" (1.409390 MB)
Rows=1201, Cols=135
Unable to Resolve Name 'FileInfo. Time'.
Error ReadManyFiles>@ (Chan) eval (CalcChan (Chan). Eqn) (Line 245)
FileInfo. Time (:, NumIC+(CC))=cell2mat (arrayfun (@ (Chan) eval (CalcChan (Chan). Eqn), CC, 'UniformOutput', false));
Error ReadManyFiles (line 245)
FileInfo. Time (:, NumIC+(CC))=cell2mat (arrayfun (@ (Chan) eval (CalcChan (Chan). Eqn), CC, 'UniformOutput', false));
Error MCrunch (line 22)
ReadManyFiles (FileList, TitleLine, NamesLine, UnitsLine, FirstDataLine, CalcChan);
Error CertTest (line 32)
MCrunch'Test_ 01. mcru '

The MATLAB running files have not been modified and are consistent with the official downloaded files.

Can you tell me how to run the CertTest. m script file in the CertTest folder normally in MATLAB?
Thank you very much for your time in advance.
Best regards

Dear @1codecopyer.
Regarding the MCrunch error you are receiving, a similar question was asked and answered in the following topic on our forum: MCrunch - #10 by Mark.Byers (see my post dated Feb 25, '21).
Best regards,

Thank you very much for your reply. Following the comments on the m file, the CertTest. m script program can run normally and generate images.


Failing to build metrics exporter in tests

Hey. Thanks for making this. I'm having an issue and was hoping I could get some help.
I'm having an issue with my tests (using axum-test) where having multiple test functions in the same file causes the tests to fail by being unable to install the metrics recorder. Example integration test file:
mod common;
use common::get_server;

#[tokio::test]
async fn test_404() {
    let server = get_server();

    let response = server.get("not-a-route").await;
    response.assert_status_not_found();
    response.assert_text_contains("Route not found");
}

#[tokio::test]
async fn test_metrics() {
    let server = get_server();

    let response = server.get(METRICS_ROUTE).await;
    response.assert_status_ok();
    assert_eq!(response.header("content-type"), "text/plain; charset=utf-8");
}

So at first I was getting this error for all but one of the tests in that file:
called `Result::unwrap()` on an `Err` value: FailedToSetGlobalRecorder(SetRecorderError { .. })

After some research I instead used the branch you have with the alternative initialise method (#52) by putting this in Cargo.toml:
axum-prometheus = { git = "https://github.com/Ptrskay3/axum-prometheus", branch = "change-default-prometheus-recorder" }

Which now gets me to this error:
Failed to build metrics recorder: FailedToCreateHTTPListener("Address already in use (os error 98)")

Now, I'm a complete noob at all of this so let me know if I'm wrong, but it looks like the tests from the one file are all getting run at the same time, and the recorder needs some specific route for itself and so fails building?
This does not affect other integration tests, as they are all each just one test function per file. Running the above tests in separate files does solve the issue, but is there any way to avoid this while keeping all the tests together? No problem if there isn't, appreciate the help either way.

Overall, I'd recommend getting rid of the metrics layer in your tests (unless that's the part you wish to test), it'll be much less of an issue.
In Rust by default all tests are run in parallel. The root of your problem is that the metrics exporter is initialized globally. Assuming in your example get_server is always trying to initialize it, hence only the fastest test will have a chance to run  successfully, and the rest will fail. If you still prefer to have metrics in the tests, you may use std::sync::Once to run a one-time global initialization, and setup the metrics there similar to this. Probably best way to use the BaseMetricLayer from the in-progress PR you already mentioned.

I see thank you. I will likely either move them all into one test function or separate files if that is the case, as ideally for the integration test I want everything to be initialised the same way as it would be in the main function.
Do you know why it's not an issue if the tests are in separate files? Are the integration test binaries run sequentially?

Well, it depends on details how you run them, but probably the global metrics exporter initialization is isolated.

Ah ok I get it. Thank you for the help, and thanks again for making this.


Is dark theme available for this widget?

Hi ,
Is there a way to enable dark mode for this calender view ?
I was able to change theme for timeLineBuilder and background in Day View. My issue is specifically with dayTitleBuilder. How can I customize dayTitleBuilder.?

Hello @rajendra-pragadareddi . This plugin does not support dark theme by default. As for your issue with dayTitleBuilder its a function that returns the Widget so, you can return any Widget from this function. This can be your customised widget for dark theme.
Hope this solve your query. I'm closing this issue for now.


Add "view site" button to admin

Similar to how WordPress has this "view site" button at the very top of the dashboard. This is useful if you want to quickly get back to your live site from the admin.

:+1:

:+1:


System.BadImageFormatException: Could not load file or assembly 'PhysX.Net'

Hello, I get this error when I try to use PhysX. Is this my error or an error with the library?

Make sure you've set x64 as your platform target

Are the native dll files in your output directory? They're required for PhysX itself to begin

Ah right, probably should have seen that - no I don't, I couldn't find them anywhere when I installed the library. I assumed they were bundled with it but I just reread the install instructions again...
Where could I find the DLLs?

They're in the package, or in Assemblies/x64 directory of this repo.
I just noticed it looks like you've installed a really old version of the package (0.5.1), I'd use the latest version 1.0.0-alpha (pre-release). 0.5.1 is for PhysX 3 which is pretty old now and I haven't touched it in years, v1.0.0-alpha is for v4 which is the latest atm.
Version 1.0.0 of the package will also add a .targets file to your csproj automatically which will copy the dll files out for you.

Sorry for my late response.
It doesn't find the PhysX namespace at all, just gives an error.
Secondly, it tells me I need to remove this:

I also get an error saying "The imported project 'C:\lib\RuntimeFiles.targets' was not found."

I've just pushed up a new package which targets .NET 5: https://www.nuget.org/packages/PhysX.Net/3.0.0-alpha1
I tested creating a new console app, installing the package and setting GeneratePathProperty and this worked, so hopefully more luck with this version.

Looks like that worked. Cheers!


Setup Docker for TPU execution and update infra scripts.

(Still needs some cleanup & doc updates but sending for thoughts.
A few notes:

We could use draccus configs for these infra parameters, but need to adjust it to support remainder args. I think it's okay to do that as a separate cleanup pass.
I moved a few scripts into Python: trying to figure out how to preserve commands through 3 layers of bash is challenging... I think it's worth moving more of these over and cleaning it up a bit over time.
It's still possible to use the git clone with this via a volume mount, but I think this should work as-is for most use cases.
With the current setup each project needs to host the base Levanter image themselves.  We can split the Docker build so that we have a base image and just add the current source on top of it. If this looks reasonable I'll do that.

FROM stanford/levanter # contains all of the setup and package installation
...

)
I tried to optimize the Docker image size a bit using a staged build, as Ray currently requires a source build of Meson, which requires a Clang installation... even with this jax & libtpu are each themselves >250MB installs, so there's no avoiding a large image size at the moment.
Still, with this configuration, a v5-32 (the most I could get given GCPs stingy IP address allocation) takes about 50 seconds to run setup-vm.sh and pull the initial image.  After the initial pull, new deployments take a few seconds to package up the current source directory.

This is amazing and wonderful. Thank you
Mostly just making sure I understand: the new workflow is:

I hack on my own fork of levanter.
Then, when I'm ready, I just run the new launch command
It will automatically spin up a tpu
build a docker image from my current checkout
push the image
deploy and run the image on the tpu machine

One thing that I think might be lost (unless I'm misreading) is that ATM we log the git commit to wandb. Is that right?

Yes mostly: I don't spin up the new TPU yet: you still use spin-up-vm.sh, and that pulls a "default" image, just to warm up the VM. From that point on, when you run launch, it's exactly as you say. It takes negligible time to setup and push any changes, as it's just the last "layer" of the docker image and <1MB of data.
I think having the combined workflow is a really good idea though and I can add that in a follow-up PR.

I'll update the docs and validate testing works and then give you a ping.

Okay, I think I caught all the docs etc. I updated the .github test integration to use launch.py but it won't run directly for me: I don't think Github will inject credentials for external PRs, so it's just failing in spin up.
I've verified it runs correctly (there are a few test failures, but they aren't my fault 😁) with the following command manually:
python infra/launch.py --foreground --tpu=test-spin-up-1 --zone=us-west4-a -- /opt/levanter/.venv/bin/pytest tests

We can of course tweak launch.py to use env/secrets as needed for that.
PTAL.

Running directly off your branch I get
Running  docker tag levanter-dlwh us-west4-docker.pkg.dev/hai-gcp-models/levanter/levanter-dlwh:1716873733
Error response from daemon: No such image: levanter-dlwh:latest
Traceback (most recent call last):
  File "/Users/dlwh/src/levanter/infra/launch.py", line 79, in <module>
    full_image_id = deploy.push_to_gcp(
  File "/Users/dlwh/src/levanter/infra/deploy.py", line 133, in push_to_gcp
    _run(["docker", "tag", image_name, full_image_name])
  File "/Users/dlwh/src/levanter/infra/deploy.py", line 35, in _run
    return subprocess.check_output(*args, **kw)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['docker', 'tag', 'levanter-dlwh', 'us-west4-docker.pkg.dev/hai-gcp-models/levanter/levanter-dlwh:1716873733']' returned non-zero exit status 1.

Running directly off your branch I get
Running  docker tag levanter-dlwh us-west4-docker.pkg.dev/hai-gcp-models/levanter/levanter-dlwh:1716873733
Error response from daemon: No such image: levanter-dlwh:latest
Traceback (most recent call last):
  File "/Users/dlwh/src/levanter/infra/launch.py", line 79, in <module>
    full_image_id = deploy.push_to_gcp(
  File "/Users/dlwh/src/levanter/infra/deploy.py", line 133, in push_to_gcp
    _run(["docker", "tag", image_name, full_image_name])
  File "/Users/dlwh/src/levanter/infra/deploy.py", line 35, in _run
    return subprocess.check_output(*args, **kw)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['docker', 'tag', 'levanter-dlwh', 'us-west4-docker.pkg.dev/hai-gcp-models/levanter/levanter-dlwh:1716873733']' returned non-zero exit status 1.

Doh, that's probably something I just screwed up: I added a timestamp to uniquely identify the tagged image, and neglected to update that command. So you can understand my pain:
Docker has a "latest" tag which works mostly as you expect, except when you're using remote images.
When you're using a remote image, and you use
docker pull xyz.pkg.dev/levanter/levanter:latest
it works correctly the first time, but if you update the package on pkg.dev and call pull again, there's no check for staleness: Docker just assumes if it has that tag locally everything must be okay. So basically we can't use the "latest" tag except for things like a base image that we don't mind going stale.

Okay, it's faster with the incremental build (and a little more predictable).
[+] Building 44.5s (10/10) FINISHED

vs building the base image:
[+] Building 169.1s (16/16) FINISHED                                                                                docker:desktop-linux

In both cases, the build is cached, so you should only pay this once.  Further deployments are almost instantaneous, e.g. to touch a single file I get:
[+] Building 0.1s (10/10) FINISHED

it works correctly the first time, but if you update the package on pkg.dev and call pull again, there's no check for staleness: Docker just assumes if it has that tag locally everything must be okay. So basically we can't use the "latest" tag except for things like a base image that we don't mind going stale.

Ah yes, Docker is very annoying about things like that. :(

I had to run
gcloud auth configure-docker us-central2-docker.pkg.dev
But now I get
name unknown: Repository "levanter" not found
Traceback (most recent call last):
  File "/Users/dlwh/src/levanter-dev/infra/launch.py", line 83, in <module>
    full_image_id = push_docker.push_to_gcp(
  File "/Users/dlwh/src/levanter-dev/infra/push_docker.py", line 153, in push_to_gcp
    _run(["docker", "push", full_image_name])
  File "/Users/dlwh/src/levanter-dev/infra/push_docker.py", line 35, in _run
    return subprocess.check_output(*args, **kw)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['docker', 'push', 'us-central2-docker.pkg.dev/hai-gcp-models/levanter/levanter-dlwh:1716924862']' returned non-zero exit status 1.

I had to run
gcloud auth configure-docker us-central2-docker.pkg.dev
But now I get
name unknown: Repository "levanter" not found

Crap, that was me: I was trying to factor out the push setup to test out pushing to Docker hub and removed the GCP setup from the push_to_gcp flow.  Can you try again now?
I'll also try testing with a different repository name to see if that breaks anything.

I went ahead and added a few cleanups:

The launch script now has retries, TPU starting and setup, so you can use it in place of babysit+spinup+setup+run
I added pushing to ghcr.io: https://ghcr.io/rjpower/levanter. The first push to GCP will take a while with this setup, as you have to push the ~500MB base image, but the build is still quick and after that everything is speedy.
Refactored so the scripts share some CLI helpers etc

I found this setup nice for dependencies, for instance, I had a few changes to Haliax so I just copied haliax into my levanter directory and added cd haliax && pip install -e . to the Dockerfile.
Let me know if you see any other things you'd like addressed!

sorry getting back to this, some thoughts:

it'd be nice if launch would be willing to reuse an existing tpu-vm if it already exists
small bytes-vs-str thing in configure_gcp_docker I'll add a suggestion for (mostly note to self)
after fixing that, i'm getting this:

Running: gcloud alpha compute tpus tpu-vm ssh dlwh-test --worker=all --zone=us-east1-d --command=docker volume create --driver=local levanter
SSH key found in project metadata; not updating instance.
Using ssh batch size of 4. Attempting to SSH into 1 nodes with a total of 4 workers.
SSH: Attempting to connect to worker 0...
SSH: Attempting to connect to worker 1...
SSH: Attempting to connect to worker 2...
SSH: Attempting to connect to worker 3...
levanter
levanter
levanter
levanter
Running  gcloud artifacts repositories describe --location=us-east1 levanter
ERROR: (gcloud.artifacts.repositories.describe) NOT_FOUND: Requested entity was not found.
Error running command.

lol now i'm getting this completely useless error
~/src/levanter-dev (faster-setup)$ PYTHONPATH=. python infra/launch.py --foreground --tpu=test-spin-up-1 --zone=us-central2-b --tpu_type v4-8 -- /opt/levanter/.venv/bin/pytest tests
Listed 0 items.
Creating new TPU test-spin-up-1 in us-central2-b of type v4-8...
Running: gcloud alpha compute tpus tpu-vm create test-spin-up-1 --accelerator-type=v4-8 --version=tpu-ubuntu2204-base --zone=us-central2-b  --quiet
ERROR: (gcloud.alpha.compute.tpus.tpu-vm.create) unrecognized arguments:

To search the help text of gcloud commands, run:
  gcloud help -- SEARCH_TERMS
Error running command.

repr helped me figure it out, lol:
Running: 'gcloud' 'alpha' 'compute' 'tpus' 'tpu-vm' 'create' 'test-spin-up-1' '--accelerator-type=v4-8' '--version=tpu-ubuntu2204-base' '--zone=us-central2-b' '' '--quiet'

Okay, cleaned everything up. I think I can't submit because the TPU tests are failing for an unrelated reason so you may need to merge for me:
gcloud compute tpus tpu-vm delete ci-run-9473410622 --zone us-central2-b --quiet
ERROR: (gcloud.compute.tpus.tpu-vm.delete) Error parsing [tpu].
The [tpu] resource is not properly specified.
Failed to find attribute [project]. The attribute can be set in the following ways:
- provide the argument `tpu` on the command line with a fully specified name
- provide the argument `--project` on the command line
- set the property `core/project`

It appears github secrets aren't available in forked repos: https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions#using-secrets-in-a-workflow (even if approved). I couldn't find a good workaround unfortunately; I'm sure there's some rube-goldberg machine setup where you can have GCP cloud run pick up approved requests and call web services, but I expect it would work 4% of the time.

I couldn't find a good workaround unfortunately; I'm sure there's some rube-goldberg machine setup where you can have GCP cloud run pick up approved requests and call web services, but I expect it would work 4% of the time.

a very simple solution is for me to add you to the repo :-)

(I don't care enough about pre-commit. we can fix it later)

Thanks so much!


Shader cache folder should be versioned to avoid using stale, incompatible cached shaders

Godot version
4.0.alpha11
System information
Fedora 36, GeForce GTX 1080 (NVIDIA 510.68.02)
Issue description
The shader cache folder should be versioned to avoid using stale, incompatible cached shaders. See https://github.com/godotengine/godot/issues/62525 and https://github.com/godotengine/godot/issues/62803 for issues caused by old shader caches lingering around and still being used by recent Godot alphas.
Cache invalidation should ideally happen whenever the commit hash changes, or when any other part of the version identifier (including patch) changes.
This is currently only an issue with the Vulkan rendering backends, not OpenGL (as OpenGL in master doesn't have shader caching to my knowledge).
Steps to reproduce

Open an project made on an older 4.0 alpha in the latest master branch.
Notice that Vulkan shader issues may pop up in the console.

Minimal reproduction project
N/A

I updated my PR to do versioning as suggested here.


Resolve difference between http and https in forc dependency URL declarations

As spotted in https://github.com/FuelLabs/fuels-rs/pull/150, currently forc will consider the same package declared with both http and https at different branches in the package graph as distinct dependencies. This means that even if they ultimately fetch the same git reference and pin to the same commit, forc will track and build them separately in the dependency graph. This doesn't appear to cause any bugs just yet, though results in extra bloat and may potentially cause issues with traits in the future.
It looks like cargo solves this by resolving both to the same URL, preferring the higher-level package's URL in the case that one uses http and the other https. That is, if A depends on B and both A and B depends on C, A's declaration of C is used. This is the case even if A's dependency on C is the weaker security preference. E.g. if A specifies the dependency on C with a http URL and B specifies the dependency on C with a https URL, B's request for C to be fetched over TLS is ignored.
I propose that we take a similar approach to cargo, but rather than selecting the URL based on the higher-level package, we instead select the stronger security approach. That is, if one package specifies http and another specifies https, we always prefer https.

Instead of http/https, could we not enforce git?

Good question! I'm not familiar enough with the differences, or sure what the limitations are w.r.t only supporting the git protocol. We should look into this.
It might catch some git users / Rustaceans off guard if we choose to only support a subset of the URLs that git does. Then again, maybe not a big issue if we have a descriptive error describing how to translate the URL to a git one.
As a side note, I think git itself supports quite a few different protocols, and this issue glosses over that fact. E.g. what if the same lib is specified via http, https, git and ssh under different packages? It's unlikely, but maybe hints at there being some better approach to URL normalisation out there... We should take a closer look at cargo's handling of git dep URLs.


Set up a permanent cert-manager installation to catch issues that only appear in long running deployments

Is your feature request related to a problem? Please describe.

We have had users report issues with their cert-manager installations that only appear over time. Most are related to cert renewal.
We currently test certificate renewal by faking time in our unit and integration tests. When renewal related issues are reported, we usually try to reproduce them manually using a freshly created installation (usually on a kind cluster on developer's laptop) with issuer types that support shorter lived certs or by using test implementations of other issuers (such as pebble), however this is not sufficient:

not all renewal-related issues can be reproduced using a freshly created installation, for example, in cases of issuer types that don't support short lived certs (Let'sEncrypt, which is probably the most widely used cert issuer with cert-manager issues certs with TTL of 3 months, so by default cert-manager would renew the cert after 2 months)

manual renewal testing is usually done only when an issue is already reported, so does not catch bugs before they make it into a release

there are other issues that can appear over time for a long running component and aren't related to renewal, such as resource consumption, upgrade related issues etc

manual renewal testing on people's personal computers means that there is no straightforward way how to give another team member access to the environment in which a bug was reproduced

Some issues that are still not reproducible or would have been caught earlier with automated renewal testing against actual issuers:

https://github.com/cert-manager/cert-manager/issues/4889 (appears to be an issue in long running installations only)
https://github.com/cert-manager/cert-manager/issues/3603 (renewal related)
https://github.com/cert-manager/cert-manager/issues/4345

Describe the solution you'd like

It would be great if we could have a 'permanent', long-running cert-manager installation interacting with actual external services (i.e Let'sEncrypt staging server etc) with a number of certificates for different issuers, so we could both run automated checks to verify renewal and also observe the behaviour of the installation over time.
Describe alternatives you've considered

Additional context

How could we use the long running installation to ensure that (most) renewal-related bug gets caught before they make it into a release? Should the installation always be upgraded to the latest alpha/beta release?

This same installation could also be used to manually reproduce bugs to allow for easier sharing of environments

Any automated checks verifying renewal or other cert-manager functionality in this environment cannot replace need for unit/integration/e2e tests for this functionality as the checks wouldn't not gate the PRs getting merged

The deployment of cert-manager and other required tools could be done using the scripts/tool that will be created as part of https://github.com/cert-manager/cert-manager/issues/4855

/kind feature

This would be a great addition! Just registering my support here as it's goes stale

I think this would be a super great feature because we have a quite static Kubernetes cluster that has cert-manager running and occasionally runs into such problems.


Force downloading of the original image size

The image size is limited in the sync to either 512 pixels tall or 512 pixels wide, this is the default for all google photos photos, you would need to add additional parameters to the URL to support the original size of the image. See the available google documentation on photo urls.

One thing to be aware of is that the API will only returned a JPG fascimile of the original photo. Apparently this is how the API is currently designed.


fix (docs): updated "compiles to" code example for babel-plugin-jsx-dom-expressions

Reasoning

based on this example I was pretty confused when playing around with certain playground implementations as they generally had different output then I'd expect based on the example here. After doing a minimal repo to try and see why the code examples differ I found that the docs didn't seems correct anymore (either due to implementation drift or due to the default configurations not being proper to get the documented output) If the case is the former then I think we should update it. If the case is the latter we should add the configs to the README.md example (and I'd be happy to do that).

To Test

I'd encourage anyone to test a blank project against the example and make sure they get the same output as I did.

Place for feedback

Not sure if the _$delegateEvents(["click"]) piece should be included in the output. Open to input there.

Thanks. I honestly probably haven't touched that part of the Readme in years.


CreateAccount documents

Hi I currently use async-stripe to implement Stripe Connect but if I create the account with a document I encounter this error:
Stripe(RequestError { http_status: 400, error_type: InvalidRequest, message: Some("Received unknown parameter: documents[proof_of_registration]"), code: Some(ParameterUnknown), decline_code: None, charge: None })
I tried with Insomnia and it seems the field of documents must be formatted like that:
documents[company_license][files][] then the file_id
Here a part of my code:
let client = Client::new(secret_key);

Account::create(
    &client,
    CreateAccount {
        business_type: Some(AccountBusinessType::NonProfit),
        email: establishment_informations.email.as_deref(),
        capabilities: Some(CreateAccountCapabilities {
            card_payments: Some(CreateAccountCapabilitiesCardPayments {
                requested: Some(true),
            }),
            transfers: Some(CreateAccountCapabilitiesTransfers {
                requested: Some(true),
            }),
            ..CreateAccountCapabilities::default()
        }),
        country: iso2.as_deref(),
        type_: Some(AccountType::Express),
        documents: Some(CreateAccountDocuments {
            company_license: Some(CreateAccountDocumentsCompanyLicense {
                files: Some(vec![statutes_id]), // TODO : mettre les status
            }),
            proof_of_registration: Some(CreateAccountDocumentsProofOfRegistration {
                files: Some(vec![receipt_id]), // TODO : récipissé de l'inscription en préfecture
            }),
            ..Default::default()
        }),
        company: Some(CompanyParams {
            name: Some(establishment_informations.name),
            phone: establishment_informations.phone_number,
            address: establishment_informations.address.map(|address| Address {
                city: address.city,
                line1: Some(address.address),
                line2: address.additional_address,
                country: address.country_iso2,
                postal_code: address.zip,
                state: None,
            }),
            structure: Some(CompanyParamsStructure::UnincorporatedNonProfit),
            ..CompanyParams::default()
        }),
        ..CreateAccount::default()
    },
)
.await

Also it seems the method create on File struct currently doesn't exist it is planned?

It seem the error is provided by the Stripe API when we use documents proof_of_registration type. No problem with the crate

Glad to hear it, thanks for using async-stripe :)


Fix broken anchors

Theses links wouldn't scroll down.
Firefox 59 on Linux Mint

          1 Warning

:warning:
Too many changes (when adding, please keep it to one project per Pull Request)

  Generated by :no_entry_sign: Danger

@Outpox Thanks for your contribution! Nice catch!


Switch to bigram strategy for text indexing

This change will reduce the size of the result matching set by decreasing the number of possible matches based on the need for similar text to have at least two neighboring matches.
For #1452

More info about https://en.m.wikipedia.org/wiki/Bigram


Radio Button Inline Validation creates an error for each Radio Button

As the title says, example JSFiddle: http://jsfiddle.net/ebcn8xvq/2/
Referencing  this issue as well:  https://github.com/Semantic-Org/Semantic-UI/issues/2501

just ran into this as well. any plans to make a set of radio buttons only show one error message?

rip


noescape by default: deprecate @noescape and @autoclosure(escaping)

What's in this pull request?

Support the @autoclosure @escaping syntax, deprecate @noescape and @autoclosure(escaping). Fixits and tests included.
Resolved bug number: (SR-)

Before merging this pull request to apple/swift repository:

[ ] Test pull request on Swift continuous integration.

Triggering Swift CI
The swift-ci is triggered by writing a comment on this PR addressed to the GitHub user @swift-ci. Different tests will run depending on the specific comment that you use. The currently available comments are:
Smoke Testing
    Platform     | Comment
    ------------ | -------------
    All supported platforms     | @swift-ci Please smoke test
    All supported platforms     | @swift-ci Please smoke test and merge
    OS X platform               | @swift-ci Please smoke test OS X platform
    Linux platform              | @swift-ci Please smoke test Linux platform

A smoke test on macOS does the following:

Builds the compiler incrementally.
Builds the standard library only for macOS. Simulator standard libraries and
device standard libraries are not built.
lldb is not built.
The test and validation-test targets are run only for macOS. The optimized
version of these tests are not run.

A smoke test on Linux does the following:

Builds the compiler incrementally.
Builds the standard library incrementally.
lldb is built incrementally.
The swift test and validation-test targets are run. The optimized version of these
tests are not run.
lldb is tested.

Validation Testing
    Platform     | Comment
    ------------ | -------------
    All supported platforms     | @swift-ci Please test
    All supported platforms     | @swift-ci Please test and merge
    OS X platform               | @swift-ci Please test OS X platform
    OS X platform               | @swift-ci Please benchmark
    Linux platform              | @swift-ci Please test Linux platform

Lint Testing
    Language     | Comment
    ------------ | -------------
    Python       | @swift-ci Please Python lint

Note: Only members of the Apple organization can trigger swift-ci.

@swift-ci please test

@swift-ci please test

@swift-ci test linux

@swift-ci test linux

@swift-ci test linux

@swift-ci test linux

@swift-ci test linux

@swift-ci please test

Failure unrelated:
Serialization/failed-clang-module.swift
/Users/buildnode/jenkins/workspace/swift-PR-osx/buildbot_incremental/swift-macosx-x86_64/test-iphonesimulator-i386/Serialization/Output/failed-clang-module.swift.tmp/MixModA.framework/Modules/module.modulemap:3:10: error: inferred submodules require a module with an umbrella
module * { export * }
^
Which is a new test file that's been problematic:

@milseman Thanks for taking the time to do the changes, the patch looks much cleaner now!


SF Symbols 2 is available

Have you plans for adding support SF Symbols 2?

@delef i think i will do that

we already try, but there are some troubles when using new fonts. i will close this issue and migrate this package as soon as possible.

@virskor Any updates on this?

@virskor Any updates on this?

Flutter has new Cupertino icon assets updated. please use official package. This package is not going to update.

Flutter has new Cupertino icon assets updated.

Ah, great!

This package is not going to update.

Would be great to update the README.md and mark this package as discontinued on pub.dev.


Added host autocompletion for ssh and host commands

It will take host inputs from consul as well as /etc/hosts and use for
bash autocompletion.
This should fix #299
NOTE: this file can be distributed in a package also.

please test this

awesome!!! As soon as those tests pass, I'll merge!

:dancers:


Not Bungee Send connect

[02:37:58] [Server thread/ERROR]: #!#! [02:37:58] [Server thread/ERROR]: #!#! [Skript] Severe Error: [02:37:58] [Server thread/ERROR]: #!#! [02:37:58] [Server thread/ERROR]: #!#! Something went horribly wrong with Skript. [02:37:58] [Server thread/ERROR]: #!#! This issue is NOT your fault! You probably can't fix it yourself, either. [02:37:58] [Server thread/ERROR]: #!#! It looks like you are using some plugin(s) that alter how Skript works (addons). [02:37:58] [Server thread/ERROR]: #!#! Here is full list of them: [02:37:58] [Server thread/ERROR]: #!#! skRayFall (https://sk.rayfall.net/) Skungee TuSKe (github.com/Tuke-Nuke/TuSKe) SK-NBeeT (https://github.com/ShaneBeee/Sk-NBeeT) skUtilities (https://tim740.github.io/) SkQuery SkChoke DiscordSRV (https://github.com/DiscordSRV/DiscordSRV) [02:37:58] [Server thread/ERROR]: #!#! We could not identify which of those are specially related, so this might also be Skript issue. [02:37:58] [Server thread/ERROR]: #!#! You should try disabling those plugins one by one, trying to find which one causes it. [02:37:58] [Server thread/ERROR]: #!#! If the error doesn't disappear even after disabling all listed plugins, it is probably Skript issue. [02:37:58] [Server thread/ERROR]: #!#! In that case, you will be given instruction on how should you report it. [02:37:58] [Server thread/ERROR]: #!#! On the other hand, if the error disappears when disabling some plugin, report it to author of that plugin. [02:37:58] [Server thread/ERROR]: #!#! Only if the author tells you to do so, report it to Skript's issue tracker. [02:37:58] [Server thread/ERROR]: #!#! [02:37:58] [Server thread/ERROR]: #!#! Stack trace: [02:37:58] [Server thread/ERROR]: #!#! java.lang.NullPointerException [02:37:58] [Server thread/ERROR]: #!#!     at me.limeglass.skungee.spigot.elements.effects.EffConnectServer.execute(EffConnectServer.java:31) [02:37:58] [Server thread/ERROR]: #!#!     at ch.njol.skript.lang.Effect.run(Effect.java:52) [02:37:58] [Server thread/ERROR]: #!#!     at ch.njol.skript.lang.TriggerItem.walk(TriggerItem.java:61) [02:37:58] [Server thread/ERROR]: #!#!     at ch.njol.skript.lang.TriggerItem.walk(TriggerItem.java:89) [02:37:58] [Server thread/ERROR]: #!#!     at ch.njol.skript.lang.Trigger.execute(Trigger.java:57) [02:37:58] [Server thread/ERROR]: #!#!     at ch.njol.skript.SkriptEventHandler.check(SkriptEventHandler.java:156) [02:37:58] [Server thread/ERROR]: #!#!     at ch.njol.skript.SkriptEventHandler$1.execute(SkriptEventHandler.java:114) [02:37:58] [Server thread/ERROR]: #!#!     at org.bukkit.plugin.RegisteredListener.callEvent(RegisteredListener.java:70) [02:37:58] [Server thread/ERROR]: #!#!     at org.bukkit.plugin.SimplePluginManager.fireEvent(SimplePluginManager.java:529) [02:37:58] [Server thread/ERROR]: #!#!     at org.bukkit.plugin.SimplePluginManager.callEvent(SimplePluginManager.java:514) [02:37:58] [Server thread/ERROR]: #!#!     at net.minecraft.server.v1_15_R1.PlayerConnection.a(PlayerConnection.java:2147) [02:37:58] [Server thread/ERROR]: #!#!     at net.minecraft.server.v1_15_R1.PacketPlayInWindowClick.a(SourceFile:32) [02:37:58] [Server thread/ERROR]: #!#!     at net.minecraft.server.v1_15_R1.PacketPlayInWindowClick.a(SourceFile:10) [02:37:58] [Server thread/ERROR]: #!#!     at net.minecraft.server.v1_15_R1.PlayerConnectionUtils.lambda$0(PlayerConnectionUtils.java:19) [02:37:58] [Server thread/ERROR]: #!#!     at net.minecraft.server.v1_15_R1.TickTask.run(SourceFile:18) [02:37:58] [Server thread/ERROR]: #!#!     at net.minecraft.server.v1_15_R1.IAsyncTaskHandler.executeTask(SourceFile:144) [02:37:58] [Server thread/ERROR]: #!#!     at net.minecraft.server.v1_15_R1.IAsyncTaskHandlerReentrant.executeTask(SourceFile:23) [02:37:58] [Server thread/ERROR]: #!#!     at net.minecraft.server.v1_15_R1.IAsyncTaskHandler.executeNext(SourceFile:118) [02:37:58] [Server thread/ERROR]: #!#!     at net.minecraft.server.v1_15_R1.MinecraftServer.ba(MinecraftServer.java:918) [02:37:58] [Server thread/ERROR]: #!#!     at net.minecraft.server.v1_15_R1.MinecraftServer.executeNext(MinecraftServer.java:911) [02:37:58] [Server thread/ERROR]: #!#!     at net.minecraft.server.v1_15_R1.IAsyncTaskHandler.awaitTasks(SourceFile:127) [02:37:58] [Server thread/ERROR]: #!#!     at net.minecraft.server.v1_15_R1.MinecraftServer.sleepForTick(MinecraftServer.java:895) [02:37:58] [Server thread/ERROR]: #!#!     at net.minecraft.server.v1_15_R1.MinecraftServer.run(MinecraftServer.java:828) [02:37:58] [Server thread/ERROR]: #!#!     at java.lang.Thread.run(Unknown Source) [02:37:58] [Server thread/ERROR]: #!#! [02:37:58] [Server thread/ERROR]: #!#! Version Information: [02:37:58] [Server thread/ERROR]: #!#!   Skript: 2.4.1 (latest) [02:37:58] [Server thread/ERROR]: #!#!     Flavor: skriptlang-github [02:37:58] [Server thread/ERROR]: #!#!     Date: 2019-12-22 [02:37:58] [Server thread/ERROR]: #!#!   Bukkit: 1.15.2-R0.1-SNAPSHOT [02:37:58] [Server thread/ERROR]: #!#!   Minecraft: 1.15.2 [02:37:58] [Server thread/ERROR]: #!#!   Java: 1.8.0_261 (Java HotSpot(TM) 64-Bit Server VM 25.261-b12) [02:37:58] [Server thread/ERROR]: #!#!   OS: Windows 10 amd64 10.0 [02:37:58] [Server thread/ERROR]: #!#! [02:37:58] [Server thread/ERROR]: #!#! Server platform: Spigot [02:37:58] [Server thread/ERROR]: #!#! [02:37:58] [Server thread/ERROR]: #!#! Current node: null [02:37:58] [Server thread/ERROR]: #!#! Current item: [Skungee] EffConnectServer- Syntax: [[skungee] (send|connect) bungee[[ ]cord] [(player|uuid)[s]] %strings/players% to [bungee[[ ]cord]] [server[s]] %string% [with reason %-connectreason%], [skungee] (send|connect) [(player|uuid)[s]] %strings/players% to [bungee[[ ]cord]] server[s] %string% [with reason %-connectreason%]] [02:37:58] [Server thread/ERROR]: #!#! Current trigger: inventory click (dispense/spawn/drop/craft/pickup/consume/break/despawn/merge) (메뉴.sk, line 11) [02:37:58] [Server thread/ERROR]: #!#! [02:37:58] [Server thread/ERROR]: #!#! Thread: Server thread [02:37:58] [Server thread/ERROR]: #!#! [02:37:58] [Server thread/ERROR]: #!#! Language: english [02:37:58] [Server thread/ERROR]: #!#! Link parse mode: DISABLED [02:37:58] [Server thread/ERROR]: #!#! [02:37:58] [Server thread/ERROR]: #!#! End of Error. [02:37:58] [Server thread/ERROR]: #!#!

Duplicate #149


Move all buttons ignore applied filters

This comes from directory:
I think the default semantics of ">>" and "<<" should be that only filtered items should be moved. My users find it surprising when >> ignores applied filters.
e.g. if I have a set {Apple, Banana, Celery} and filter by "n", not only should only Banana be visible on the left, but only Banana should be moved to the right on ">>".

Here is the PR for this issue: https://github.com/FlowingCode/TwinColGridAddon/pull/77


Store origin data alongside localized data

As discussed here, we should store origin entry data alongside localisation data so it can be queried at the database level.
Let me know if youre happy with this approach as I'll need to add some test coverage on this before it should be merged.

@FrittenKeeZ yep, I was battling the tests a bit so moved this to draft. Should be working now with the latest commit.

@ryanmitchell @jasonvarga we're facing rather big performance issues with blueprint being stored in json instead of its own field, so I just wanted to hear you out if you're okay with pulling blueprint out of json with an index so queries don't take 5+ seconds to complete?

@ryanmitchell do you have any sense of whether this is ready for use? The missing blueprint on translated entries are causing us big issues currently, so I'm torn between using this patch to get the missing data or creating a PR for storing blueprint in a separate field in the database for improving performance as well.

As far as I can see it is, but Jason may feel differently

We just talked it through with the team and I'll be making a PR to fix #126 soon, as it will fix three issues we have regarding performance, code complexity and data duplication for blueprint.

@ryanmitchell I can see I added some additional handling regarding parent, but I can't remember whether it was an issue or I just added it as a precaution.
https://github.com/statamic/eloquent-driver/pull/144/files#diff-31fcf8a95fd0dbc8da9e8dc7ae856aa51bd921c50a8059e23df7d411737ab65fR71

Yeah we're not storing parent going forward - its redundant as we have origin as a column.

Ah right, so german might be the parent of french, but english is the fallback? Ok, storing the ID is fine (as your PR did) - I think until this PR its storing the whole entry which is a lot.

Thanks @FrittenKeeZ


HDFS-16984. Directory timestamp lost during the upgrade process

Description of PR
During the upgrade process, when creating the FSImage the access time of the directory is not serialized to the FSImage. This causes the access timestamp lost.
This PR adds access time field in INodeDirectory just like INodeFile proto definition and persists access time field during the snapshotting process.
How was this patch tested?
(1) Tested with system snapshot + restart.
(2) Tested old version system without the patch => upgrade to system with the patch.
For code changes:

[ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 47s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
buf
0m  1s

buf was not available.

+0 :ok:
buf
0m  1s

buf was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
48m 20s

trunk passed

+1 :green_heart:
compile
1m 25s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
1m 16s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 10s

trunk passed

+1 :green_heart:
mvnsite
1m 25s

trunk passed

+1 :green_heart:
javadoc
1m  8s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 35s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 21s

trunk passed

+1 :green_heart:
shadedclient
40m 13s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 11s

the patch passed

+1 :green_heart:
compile
1m 16s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
cc
1m 16s

the patch passed

+1 :green_heart:
javac
1m 16s

the patch passed

+1 :green_heart:
compile
1m  8s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
cc
1m  8s

the patch passed

+1 :green_heart:
javac
1m  8s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m  1s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15)

+1 :green_heart:
mvnsite
1m 16s

the patch passed

+1 :green_heart:
javadoc
0m 56s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 29s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 20s

the patch passed

+1 :green_heart:
shadedclient
40m 51s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
237m 31s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 46s

The patch does not generate ASF License warnings.

392m  7s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6175/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6175

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat

uname
Linux 3f2641282fce 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 321a01543db9c59c3b8385520c20d886b9b43aa0

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6175/1/testReport/

Max. process+thread count
2386 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6175/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 57s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
buf
0m  1s

buf was not available.

+0 :ok:
buf
0m  1s

buf was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
51m 51s

trunk passed

+1 :green_heart:
compile
1m 26s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m 18s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 17s

trunk passed

+1 :green_heart:
mvnsite
1m 30s

trunk passed

-1 :x:
javadoc
1m 21s
/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.txt
hadoop-hdfs in trunk failed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.

+1 :green_heart:
javadoc
2m 19s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 41s

trunk passed

+1 :green_heart:
shadedclient
43m  9s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 16s

the patch passed

+1 :green_heart:
compile
1m 26s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
cc
1m 26s

the patch passed

+1 :green_heart:
javac
1m 26s

the patch passed

+1 :green_heart:
compile
1m 13s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
cc
1m 13s

the patch passed

+1 :green_heart:
javac
1m 13s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m  4s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15)

+1 :green_heart:
mvnsite
1m 17s

the patch passed

-1 :x:
javadoc
0m 59s
/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.txt
hadoop-hdfs in the patch failed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.

+1 :green_heart:
javadoc
1m 30s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 27s

the patch passed

+1 :green_heart:
shadedclient
42m 38s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
246m 28s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 46s

The patch does not generate ASF License warnings.

411m 37s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestDirectoryScanner

hadoop.hdfs.server.datanode.TestDataNodeMetrics

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6175/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6175

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat

uname
Linux c8ab2dde0233 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 321a01543db9c59c3b8385520c20d886b9b43aa0

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6175/1/testReport/

Max. process+thread count
2841 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6175/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.


Add Google analytics to docs

Closes #847

LGTM. Have we tried if something is actually logged in GA?

Rebased, pls approve it once again. I tried that and it's working.


RedisModule makes Jest testing hang

Description
Nest.js project by default comes with a nice integration with Jest testing framework. Injecting RedisModule into testing module causes tests command to hang after execution and not exit properly.
I already experienced such problems with testing frameworks connecting to databases. It might be resolved by somehow disconnecting from Redis once all tests have passed. The docs do not say how to disconnect.
Reasons to import it into tests:

I may create integration tests that actually use Redis
I may create a client to only mock its methods after that
I have many tests that just import the entire AppModule as is along with RedisModule. Then I test services that may not even actually use the module.

Versions

nestjs: 6.11.7
jest: 24.9.0
nestjs-redis: 1.2.5

Steps to reproduce
Use nest new command from @nestjs/cli to create project. At this point npm run test correctlly exits after execution. Now install nestjs-redis and inject it inside testing module.
That's app.controller.spec.ts file
const app: TestingModule = await Test.createTestingModule({
  imports: [
    RedisModule.register({
      host: 'localhost',
      port: 6379,
    })
  ],
  controllers: [AppController],
  providers: [AppService],
}).compile();

Now run npm run test. The test is green but it does not exit, the command hangs with this output:

Jest did not exit one second after the test run has completed.
This usually means that there are asynchronous operations that weren't stopped in your tests. Consider running Jest with --detectOpenHandles to troubleshoot this issue.
detectOpenHandles flag does not help.

Issue was closed, but #24 has not been released yet, any idea when the new version could be published?


Namespace cops

Is your feature request related to a problem? Please describe.
I have a project which uses only activerecord. For using rubocop-rails, I will have to enable each activerecord-related cop individually and I might miss out on new cops in future upgrades.
Describe the solution you'd like
Can we namespace cops like Rubocop::Rails::ActiveRecord, Rubocop::Rails::ActiveSupport, etc. so we can easily enable/disable them?
Describe alternatives you've considered
Enable individual cops.

I'm not sure about it's best to be subdivided. On the other hand, there are certainly costs of change and migration. Unfortunatary, This ROI doesn't look like it's going to tip too well. An alternative idea, maybe providing a configuration file with only Active Record enabled (and others if needs) as a 3rd party is realistic.


Add boolean comparison to scalar kernels for less then, greater than

Which issue does this PR close?

Closes #959
Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Thank you @xudong963 !

My pleasure! Start my arrow-rs travel from the ticket.

I kicked off the CI tests and when they complete I plan to merge this PR in.


Fix bug on forms

Fix error:
undefined local variable or method `events' for RailsAdmin::Config::Fields::Types::State

when go to new or edit action

@max-konin  thanks, I merged the changes. will update the gem