[SPARK-30865][SQL] Refactor DateTimeUtils

What changes were proposed in this pull request?

Move TimeZoneUTC and TimeZoneGMT to DateTimeTestUtils
Remove TimeZoneGMT
Use ZoneId.systemDefault() instead of defaultTimeZone().toZoneId
Alias SQLDate & SQLTimestamp to internal types of DateType and TimestampType
Avoid one * DateTimeUtils.in fromJulianDay()
Use toTotalMonths in DateTimeUtils.subtractDates()

Why are the changes needed?
This simplifies the common code related to date-time operations, and should improve maintainability. In particular:

TimeZoneUTC and TimeZoneGMT are moved to DateTimeTestUtils because they are used only in tests
TimeZoneGMT can be removed because it is equal to TimeZoneUTC
After the PR #27494, Spark expressions and DateTimeUtils functions switched to ZoneId instead of TimeZone completely. defaultTimeZone() with TimeZone as return type is not needed anymore.
SQLDate and SQLTimestamp types can be explicitly aliased to internal types of DateType and and TimestampType instead of declaring this in a comment.
Avoid one * DateTimeUtils.in fromJulianDay().
Use toTotalMonths in DateTimeUtils.subtractDates().

Does this PR introduce any user-facing change?
No
How was this patch tested?
By existing test suites

Test build #118615 has started for PR 27617 at commit 0b5711e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23368/
Test PASSed.

Test build #118615 has finished for PR 27617 at commit 0b5711e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118615/
Test FAILed.

jenkins, retest this, please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23374/
Test PASSed.

Test build #118623 has started for PR 27617 at commit 0b5711e.

Test build #118623 has finished for PR 27617 at commit 0b5711e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118623/
Test PASSed.

@cloud-fan @HyukjinKwon Please, take a look at this PR.

Test build #118757 has started for PR 27617 at commit 9ba2d3d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23508/
Test PASSed.

Test build #118759 has started for PR 27617 at commit c475f27.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23510/
Test PASSed.

Test build #118757 has finished for PR 27617 at commit 9ba2d3d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #118759 has finished for PR 27617 at commit c475f27.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118757/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118759/
Test FAILed.

jenkins, retest this, please

Test build #118768 has started for PR 27617 at commit c475f27.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23520/
Test PASSed.

Test build #118768 has finished for PR 27617 at commit c475f27.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118768/
Test PASSed.

Test build #118804 has started for PR 27617 at commit cbf69fb.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23554/
Test PASSed.

Test build #118804 has finished for PR 27617 at commit cbf69fb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118804/
Test PASSed.

cc: @viirya @cloud-fan

Move TimeZoneUTC and TimeZoneGMT to DateTimeTestUtils
Remove TimeZoneGMT

A bit confused. TimeZoneGMT was removed why it also moved to DateTimeTestUtils?

A bit confused. TimeZoneGMT was removed why it also moved to DateTimeTestUtils?

@viirya I moved it to DateTimeTestUtils because it is used only in tests, and then I removed it because semantically it is equivalent to TimeZoneUTC.

Test build #118850 has started for PR 27617 at commit dde714b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23599/
Test PASSed.

Test build #118850 has finished for PR 27617 at commit dde714b.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118850/
Test FAILed.

Test build #118853 has started for PR 27617 at commit 6840291.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23602/
Test PASSed.

Test build #118853 has finished for PR 27617 at commit 6840291.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118853/
Test FAILed.

jenkins, retest this, please

Test build #118866 has started for PR 27617 at commit 6840291.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23616/
Test PASSed.

Test build #118866 has finished for PR 27617 at commit 6840291.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118866/
Test PASSed.

Test build #118926 has started for PR 27617 at commit 6a41a3c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23674/
Test PASSed.

Test build #118926 has finished for PR 27617 at commit 6a41a3c.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class MakeDecimal(
case class MultiplyInterval(
case class DivideInterval(

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118926/
Test PASSed.

@cloud-fan Please, have a look at this if you have time.

Test build #118950 has started for PR 27617 at commit 86ccef1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23699/
Test PASSed.

Test build #118950 has finished for PR 27617 at commit 86ccef1.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118950/
Test FAILed.

retest this please

Test build #118959 has started for PR 27617 at commit 86ccef1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23707/
Test PASSed.

Test build #118959 has finished for PR 27617 at commit 86ccef1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118959/
Test PASSed.

@cloud-fan It is ready, please, take a look at it when you have time.

This LGTM but can we hold it a little bit? Recently there is a discussion about date/timestamp parsing backward compatibility and we may touch some datetime code to fix it. I'd like to avoid potential conflicts and merge this refactor at the end.

@cloud-fan @HyukjinKwon Are we waiting for some particular PR merged to the master?

jenkins, retest this, please

Test build #119518 has started for PR 27617 at commit 86ccef1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24247/
Test PASSed.

Test build #119518 has finished for PR 27617 at commit 86ccef1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119518/
Test PASSed.

Test build #119653 has started for PR 27617 at commit 1fd3aa2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24384/
Test PASSed.

Test build #119653 has finished for PR 27617 at commit 1fd3aa2.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119653/
Test FAILed.

jenkins, retest this, please

Test build #119658 has started for PR 27617 at commit 1fd3aa2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24389/
Test PASSed.

Test build #119658 has finished for PR 27617 at commit 1fd3aa2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119658/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24428/
Test PASSed.

Test build #119699 has started for PR 27617 at commit 54c0fe9.

Test build #119699 has finished for PR 27617 at commit 54c0fe9.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119699/
Test FAILed.

jenkins, retest this, please

Test build #119706 has started for PR 27617 at commit 54c0fe9.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24435/
Test PASSed.

Test build #119706 has finished for PR 27617 at commit 54c0fe9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119706/
Test PASSed.

Test build #119947 has started for PR 27617 at commit e12ec28.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24670/
Test PASSed.

Test build #119947 has finished for PR 27617 at commit e12ec28.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119947/
Test PASSed.

Test build #120068 has started for PR 27617 at commit 9d78a7d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24785/
Test PASSed.

Test build #120068 has finished for PR 27617 at commit 9d78a7d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120068/
Test PASSed.

Test build #120108 has started for PR 27617 at commit 8669e29.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24821/
Test PASSed.

Test build #120108 has finished for PR 27617 at commit 8669e29.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120108/
Test FAILed.

I have rebased this on the recent master, and got pretty interesting error:
org.scalatest.exceptions.GeneratorDrivenPropertyCheckFailedException: DateTimeException was thrown during property evaluation.   Message: Invalid date 'February 29' as '1000' is not a leap year       Occurred when passed generated values (    )

in https://github.com/apache/spark/pull/27617#issuecomment-601824955
It seems https://github.com/apache/spark/blob/b27b3c91f113ec49ee87c877dac0a602849d76b1/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala#L369 generates a timestamp which can be represented as local date-time in Julian calendar but not in Proleptic Gregorian calendar, see the error message. /cc @cloud-fan @HyukjinKwon

The issue will be fixed by https://github.com/apache/spark/pull/27980. I am going to re-trigger build, and hope that a leap year in Julian (and not leap year in Gregorian) will be not generated in tests.

jenkins, retest this, please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24892/
Test PASSed.

Test build #120179 has started for PR 27617 at commit 8669e29.

Test build #120179 has finished for PR 27617 at commit 8669e29.

This patch fails due to an unknown error code, -9.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120179/
Test FAILed.

Test build #120186 has started for PR 27617 at commit aa8a62f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24899/
Test PASSed.

Test build #120186 has finished for PR 27617 at commit aa8a62f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
final class VarianceThresholdSelector @Since(\"3.1.0\")(@Since(\"3.1.0\") override val uid: String)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120186/
Test PASSed.

Test build #120280 has started for PR 27617 at commit 9ec98eb.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24990/
Test PASSed.

Test build #120280 has finished for PR 27617 at commit 9ec98eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120280/
Test PASSed.

Test build #120437 has started for PR 27617 at commit d7f343b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25145/
Test PASSed.

Test build #120437 has finished for PR 27617 at commit d7f343b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120437/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25234/
Test PASSed.

Test build #120528 has started for PR 27617 at commit 3b23c52.

Test build #120528 has finished for PR 27617 at commit 3b23c52.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120528/
Test PASSed.

Test build #121098 has started for PR 27617 at commit 3215948.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25788/
Test PASSed.

Test build #121098 has finished for PR 27617 at commit 3215948.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121098/
Test PASSed.

Test build #121336 has started for PR 27617 at commit 77a780f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26019/
Test PASSed.

Test build #121336 has finished for PR 27617 at commit 77a780f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaUserDefinedScalar

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121336/
Test FAILed.

jenkins, retest this, please

Test build #121349 has started for PR 27617 at commit 77a780f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26033/
Test PASSed.

Test build #121349 has finished for PR 27617 at commit 77a780f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaUserDefinedScalar

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121349/
Test FAILed.

jenkins, retest this, please

Test build #121354 has started for PR 27617 at commit 77a780f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26038/
Test PASSed.

Test build #121354 has finished for PR 27617 at commit 77a780f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaUserDefinedScalar

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121354/
Test PASSed.

Test build #121814 has started for PR 27617 at commit 28c8443.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26489/
Test PASSed.

Test build #121814 has finished for PR 27617 at commit 28c8443.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121814/
Test PASSed.

Test build #121878 has started for PR 27617 at commit 70a057f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26549/
Test PASSed.

Test build #121878 has finished for PR 27617 at commit 70a057f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121878/
Test FAILed.

jenkins, retest this, please

Test build #121887 has started for PR 27617 at commit 70a057f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26558/
Test PASSed.

Test build #121887 has finished for PR 27617 at commit 70a057f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121887/
Test PASSed.

Test build #122145 has started for PR 27617 at commit a9e73db.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26816/
Test PASSed.

Test build #122145 has finished for PR 27617 at commit a9e73db.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class AggregateWithHaving(
case class DatetimeSub(
case class Rand(child: Expression, hideSeed: Boolean = false)
case class Randn(child: Expression, hideSeed: Boolean = false)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122145/
Test PASSed.

Test build #122633 has started for PR 27617 at commit 5dfa257.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27295/
Test PASSed.

Test build #122633 has finished for PR 27617 at commit 5dfa257.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122633/
Test PASSed.

Test build #122939 has started for PR 27617 at commit 1eb480a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27581/
Test PASSed.

Test build #122939 has finished for PR 27617 at commit 1eb480a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122939/
Test PASSed.

Test build #123286 has started for PR 27617 at commit 61b85ac.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27908/
Test PASSed.

Test build #123286 has finished for PR 27617 at commit 61b85ac.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123286/
Test PASSed.

Test build #123726 has started for PR 27617 at commit 311a47e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28350/
Test PASSed.

Test build #123726 has finished for PR 27617 at commit 311a47e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123726/
Test FAILed.

jenkins, retest this, please

Test build #123739 has started for PR 27617 at commit 311a47e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28363/
Test PASSed.

jenkins, retest this, please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28524/
Test PASSed.


How to lift asm codes in text files into llvmir

Hi, I am trying to lift a bunch of asm codes which stored in text files into llvmir using capstone2llvmir, but I have no idea how to let capstone2llvmir to do that
I've tried commands like these:

./retdec-capstone2llvmir  -a arm -m arm -t `echo $code`
./retdec-capstone2llvmir  -a arm -m arm -t $(cat ~/Desktop/Project/small_opt/orig_exe/test.ass)
./retdec-capstone2llvmir  -a arm -m arm -t $code
./retdec-capstone2llvmir  -a arm -m arm -t "~/Desktop/Project/small_opt/orig_exe/test.ass"

but none of which works

Have you solved it? I need this function, too. Can you help me? thank you.


ArgumentError: wrong number of argument (given 1, expected 2)

Hello, I tried to follow straight all the documentation for configuring my queue and my application.
However I have an error with the following log:
ArgumentError: wrong number of arguments (given 1, expected 2)
from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/active_elastic_job-2.0.0/lib/active_job/queue_adapters/active_elastic_job_adapter.rb:53:in initialize' 	from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/active_elastic_job-2.0.0/lib/active_job/queue_adapters/active_elastic_job_adapter.rb:145:in exception'
from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/active_elastic_job-2.0.0/lib/active_job/queue_adapters/active_elastic_job_adapter.rb:145:in raise' 	from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/active_elastic_job-2.0.0/lib/active_job/queue_adapters/active_elastic_job_adapter.rb:145:in rescue in queue_url'
from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/active_elastic_job-2.0.0/lib/active_job/queue_adapters/active_elastic_job_adapter.rb:139:in queue_url' 	from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/active_elastic_job-2.0.0/lib/active_job/queue_adapters/active_elastic_job_adapter.rb:122:in build_message'
from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/active_elastic_job-2.0.0/lib/active_job/queue_adapters/active_elastic_job_adapter.rb:96:in enqueue_at' 	from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/active_elastic_job-2.0.0/lib/active_job/queue_adapters/active_elastic_job_adapter.rb:90:in enqueue'
from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/active_elastic_job-2.0.0/lib/active_job/queue_adapters/active_elastic_job_adapter.rb:81:in enqueue' 	from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activejob-<IP_ADDRESS>/lib/active_job/enqueuing.rb:76:in block in enqueue'
Do you know how I did wrong ?
Thanks for helping.

Fixed .
I have put the name of the queue in the wrong place.
I directly put this in the configuration.
config.action_mailer.deliver_later_queue_name = 'awseb-e-cfh****....'

Hi @AhmedLoud
Thanks for your submission. I will look into that.

I'm getting a similar error, probably because I misconfigured the queue.
I think this line should include the aws_region, because NonExistentQueue expects two arguments: https://github.com/tawan/active-elastic-job/blob/master/lib/active_job/queue_adapters/active_elastic_job_adapter.rb#L145


Polysynth Voice Inconsistency

Playing less voices than the max set in PolySynth results in various audio inconsistencies.
Tone@next affected (13.4.10) (Chrome 71.0.35, Firefox 64.02)
https://jsfiddle.net/timboie/7cs3zkbh/
<EMAIL_ADDRESS>OK

using triggerRelease(notes) instead of releaseAll() seems to be OK with tone@next.  need to do more testing

Playing with this now on Chrome 72, i am unable to recreate this issue anymore. i wonder if it was a problem that version of Chrome? tested both with triggerRelease and releaseAll. lmk if you can confirm that.

issue confirmed using  v13.7.7 and Chrome 72.  releaseAll appears to affect the next triggerAttack call when used.

how does it affect the next triggerAttack call?

Same sound "inconsistencies/artifacts" as shown in the demo.
It's difficult to reproduce these occurrences in a controlled manner than live, but here are some differences I noticed while using the updated polysynth:

Some notes sound shallow and missing a full body of sound.
Some chord notes stay stuck on.
Old polysynth handled more voices.  (Compared number of voices each version could handle before not working on my system).
releaseAll seems to be a source of audio problems as demonstrated.  Especially if triggered more than once.

These problems could be on my end.  Maybe missing some details, wrong usage, system problem, but the old version was intuitively more reliable and stable from my experience.
Going to switch back to the older polysynth for now to ensure my observations hold true.
The polysynth is a wicked feature, and just want the best for it :)

Sorry for a newbie question.
I'm trying to use Tone.PolySynth() inside the React component.
I have it set like this:
synthNoEffect = new Tone.PolySynth())
then call it in handler:
`handleSingleNote = () => {
this.synthNoEffect.triggerAttack(this.state.note)
}
handleSingleNoteRelease = () => {
this.synthNoEffect.releaseAll()
}`
App listens to keypress to trigger and keyup to release.
When I try to play continuous note by holding a key, the note stutters instead of making continuous note.
Is it because of React and Tone.js compatibility? It sounds fine when I tested as vanilla js.

Need to see component code @hoshikitsunoda

@tboie
Sorry about the delay.
Not sure what the best way to share is so just pasting a link to my repo:
https://github.com/hoshikitsunoda/react-synth
Let me know if there's better way to share the code.
Thanks!

@hoshikitsunoda Is your keypress event listener being called continuously while pressed down?

@tboie Yes. I tried keydown too but still the same result. DuoSynth, AMSynth and FMSynth works fine but MonoSynth and PolySynth are stuttering.

@hoshikitsunoda both keydown and keypress events are called repeatedly when a key is held down.
See example here:
https://jsfiddle.net/timboie/L54ubz8h/
Is triggerAttack being called repeatedly?  If so, only allow 1 call to triggerAttack

@tboie added event.repeat conditional to eventlistener callback and now it works like a charm:) Thank you so much for help!!

@tboie i think this is fixed now on the typescript branch. I reworked the PolySynth a lot. It does not do any voice stealing which seemed to be a major source of the bugs, but i also optimized voice creation so there shouldn't be a major cost to over-provisioning polysynth voices.
Here's your fiddle with the latest build: https://jsfiddle.net/yotammann/t2b7ksza/
it doesn't cause any of those glitches that you had noticed before.

This is awesome @tambien!  I'll dig into this when I get a chance.  Definitely have to get caught up with the typescript branch.  Getting familiar with typescript myself over the past few weeks.  Thanks!

i just pushed a change to the typescript branch which does not require a polyphony argument, just the synth voice type and the synth's options. The voice allocation and disposing is handled entirely under the hood with the option of setting 'maxPolyphony'.
If you get a chance to try it out on the typescript branch<EMAIL_ADDRESS>it'd be great to know how it performs in the wild.


Terminal suddenly can't find the font Cascadia Mono

Windows Terminal version (or Windows build number)
10.0.19043.1288
Other Software
No response
Steps to reproduce
Just starting the Terminal on Windows 10. Not running as admin.
Expected Behavior
A new terminal window
Actual Behavior
Got this error
`Unable to find the selected font "Cascadia Mono".
"Consolas" has been selected instead.
Please either install the missing font or choose another one.`

Same issue: Windows Terminal automatically closed due to Windows Update (no restarts).
Opening Terminal again resulted in "Could not find Cascadia Code" issue that OP raised.
On 21H1 19043.1288
(The other problem is that Terminal arbitrarily closes even if Windows Update is configured to restart at a later date: All other apps function properly)

this is the thread
We've tracked this in other threads in the past, and we've done work to mitigate this before, but we didn't get all of it. Every time we push an update, more folks start running into this again. We've got an idea of what's wrong here - one of the code paths does the right thing for the SxS font lookup, and the other path doesn't, and that leads to this warning.
In the past, we used #9375 to track this. #9734 attempted to mitigate. #10305 tracked the version of this that caused a crash on startup.
So this was the first thread for the issue during this update, congrats.
IF YOU'RE HITTING THIS
Try "Repairing" the Terminal app, by going to "Settings > Apps > {whatever Terminal Install you have} > Advanced Options > Repair". This shouldn't delete your settings. (But "Reset" will, so be careful!)

If you comment some variation of "me too" in this thread, ~you will be removed from the selfhost program~ your comment will be minimized as spam, thanks.

I worked around by installing Cascadia manually. (Hopefully this message isn't marked as spam :)
I searched for this issue before (opening a new one ) / commenting here btw. So if possible please sticky this.
Also I wasn't sure if it was a Terminal issue or a Cascadia issue, so juggling between those two github repos as well.
This could avoid unintentional 'spam' comments.

I would also like to confirm this issue is now present in the Windows 11 Version after a windows update but I'm not sure if that is related or not I did not have this issue until a couple of days ago after a windows update, I'm currently running version 1.11.2921.0, I'm on the Insider program on the Beta Channel, so whatever this issue is it's now coming through to Windows 11 versions...
Currently on:
Edition	Windows 11 Pro
Version	21H2
Installed on	27/08/2021
OS build	22000.258
Experience	Windows Feature Experience Pack 1000.22000.258.0
I do have a Windows Update pending will update this message if this update resolves the problem for me...

In general: We don't really need to know anything more about OS version and Terminal version for this one. We've got a good idea of what's causing this.
The Windows 11 data point is a little interesting, we thought that the underlying bug in the OS for migrating the font files from one version to the next was fixed, but alas, it seems it also regressed since when it was first fixed in Windows 11. So this will continue to plague users at random till we fix this.

#11032's mitigation will likely mitigate this too.

This is still an issue and will continue to be an issue until we nail nearby font loading.
Notes from discussion:

But the heart of the issue is that our system collection contains a pointer to a font that is locked for deletion, because the app platform locked it for deletion since it's part of our package
We needed to do fallback when we failed to actually load the font, to a collection that doesn't have the issue

Happened to me again on 1.13.10336.0 (current preview version in the store afaict)

This is not fixed on Windows 10 in 1.13.10983.0

Reopening because I found a flaw in my implementation.
In case people wonder how we're failing so hard at fixing this: We're simply unable to repro this issue. All fixes are purely theoretic. Even my new fix is just made on an assumption (a pretty good one this time if I may say so).

I'm not sure if it helps but the issue seems to be system-wide when it happens. This causes crashes in gVim when attempting to load "Cascadia Code" after Windows Terminal has updated on my Windows 11 box. I worked around using @Mindfulplays' suggestion to install manually.
I suspect the issue is less with loading the font incorrectly and something to do with installing the font incorrectly. I tend to be using the font somewhere at all times so perhaps there's some kind of issue with replacing it when it's being used in some way.

@tylerszabo Yes, we know that the actual underlying issue certainly isn't our fault. I'm not sure what the current state of an fix for that is (I can faintly remember hearing it's fixed in a newer Windows 11 build?), but in the meantime we'd of course want to fix it ourselves as quickly as possible.

Happened for me again on 1.13.10983.0 on Windows 10 21H2, making for roughly the fourth time in a year I had this. Some additional data points:

I've rebooted a few hours ago to install Windows updates
This didn't break right on startup. In fact I have two shell tabs open where the font is okay, and only now (a few hours later) opening a new tab of the same type complains about the Cascadia Mono being MIA.

@lhecker You wrote:

because we're simply unable to repro this issue

Perhaps the 3 steps below will help reproduce the issue. (Although @zadjii-msft mentioned you don't really need to know anything more about OS version, I should say I use Windows 11 ARM in Parallels on an M1 Mac).
STEP 1: Download and unzip Cascadia Mono PL:
# Download the archive file
$downloadsPath = "$Env:USERPROFILE\Downloads"
$zipFileName = "CascadiaCode-2111.01.zip"
$zipFilePath = "$downloadsPath\$zipFileName"
$url = "https://github.com/microsoft/cascadia-code/releases/download/v2111.01/$zipFileName"
Invoke-WebRequest -Uri $url -OutFile $zipFilePath

# Expand, then delete, the archive file
$expandedDirectoryPath = "$downloadsPath\CascadiaCode-2111.01"
Expand-Archive -Path $zipFilePath -Destination $expandedDirectoryPath
Remove-Item -Force -Path $zipFilePath

STEP 2: Copy and register the font such that it causes the issue in Windows Terminal:
# Ensure destination directory exists
$fontsDirectoryPath = "$Env:LOCALAPPDATA\Microsoft\Windows\Fonts"
if (-not (Test-Path $fontsDirectoryPath))
{
    $directory = New-Item -Force -ItemType Directory -Path $fontsDirectoryPath
}

# Copy font file
$fontFileName = "CascadiaMonoPL.ttf"
$fontFileSourcePath = "$Env:USERPROFILE\Downloads\CascadiaCode-2111.01\ttf\$fontFileName"
$fontFileDestinationPath = "$fontsDirectoryPath\$fontFileName"
Copy-Item -Path $fontFileSourcePath -Destination $fontFileDestinationPath

# Ensure the registry key exists
$fontsKey = "HKCU:\Software\Microsoft\Windows NT\CurrentVersion\Fonts"
if (-not (Test-Path -Path $fontsKey))
{
    $key = New-Item -Force -ItemType Directory -Path $fontsKey
}

# Add the registry key for the font file
$value = New-ItemProperty -Force -Name "Cascadia Mono PL Regular (TrueType)" -Path $fontsKey -PropertyType $type -Value $fontFileDestinationPath

STEP 3: Verify that Windows Terminal is unable to find the font.
STEP 4: Properly install the font such that it does not cause the issue in Windows Terminal:
(The Windows Shell should ask if you want to replace the 'Cascadia Mono PL Regular' font when you run the script below. You should answer Yes.)
# Create the Windows Shell's automation object and use the fonts namespace
$shell = New-Object -ComObject "Shell.Application"
$fonts = $shell.Namespace(0x14)

# Copy font file
$fontFileName = "CascadiaMonoPL.ttf"
$fontFileSourcePath = "$Env:USERPROFILE\Downloads\CascadiaCode-2111.01\ttf\$fontFileName"
$fonts.CopyHere($fontFileSourcePath, 20)

STEP 5: Verify that Windows Terminal is able to find the font.
Hope this helps!

If any poor soul has found their way here wondering why Windows Terminal can't find their font, let me assure you that this is actually not a Windows Terminal bug, or even a Windows bug. This behavior is caused by not having the font installed for all users. Follow these steps to remedy the problem... https://answers.microsoft.com/en-us/windows/forum/all/solution-installed-fonts-disappearing-after/146f4039-47c3-4017-a9b1-76f72badce39

I just run into this issue. Version 1.18.2822.0. Windows 10. According to the Microsoft Store Terminal was updated today.
The previous comment does not apply in my case, there is no Fonts folder in %appdata%..\Local\Microsoft\Windows.
What's even more confusing is that in "Fonts" they are "there" but are invisible until I hove over them with the mouse in  which case only the names appear:

This was an issue on Windows 10 that Terminal had worked around. We recently (accidentally) broke our workaround. Stay tuned. :smile:

Glad to hear it is a known issue and not just something wrong with my PC.

@DHowett i believe there is a specific term for that it's called a Regression :D

If anyone needs a temporary fix, uninstalling then reinstalling the Terminal app solved it for me.

I fixed this issue by making sure that the font being used was installed for all users instead of locally just for my user.


systemd-resolved takes port 53 on ubuntu

Posting this here with the hopes it will help someone else with the same issue.
On my ubuntu box starting dns-proxy-server fails with an error udp :53: bind: address already in use.  Port 53 has been taken by systemd-resolved, I followed the instructions here to disable it: https://askubuntu.com/questions/907246/how-to-disable-systemd-resolved-in-ubuntu

@russau, thank you for your contribution


Build updates 19.08

Fix pulseaudio settings

Automate TTS, Precise and Intents caching run

Update resolution for mycroft.fb

Looks good, merging


React Native Support

It would be excellent if you added support for React Native, I have a pretty large project and cannot  seem to get the current release to connect to a asp.net core signalr server via websockets with having negotiation errors with the server. If anyone else has managed to get it working I would love to know how but as it currently stands it would be great for a client to be released from Microsoft.

If you want to use raw Websocket, you need to add app.UseWebSockets() and write your own middlewares to route and handle WebSockets connections.

By adding that what URL would we use to connect via web sockets /client?hub= with the hub name  Also  can we not invoke hub methods? I was under the impression that ASP.NET Core SignalR uses websockets by default and downgrades to long polling or server sent events.

Dupe of https://github.com/aspnet/AspNetCore/issues/6565
If you can provide any additional information to what is on that issue that would be helpful 😃

It is really not a Dupe of #6565 because he could not even get a web socket to initialize for some reason .. The issues I have is that if you import the @aspnet/signalr javascript sdk into a react native project when it tries to connect to the hub it will kick back an error negotiating with the server. The same code works fine in a javascript webpage so I am unsure what is missing under the hood or if the headers are even correct coming from react-native. You can easily attempt to duplicate this yourself with just creating a blank react-native app (not with expo!) and npm install @aspnet/signalr into the project. I am also using the aspnet core Signalr ChatHub example for the server side of things which you can use as well.
Sample Code
// Imports import { HubConnectionBuilder, JsonHubProtocol } from '@aspnet/signalr';
`
componentDidMount() {
var connection = new HubConnectionBuilder().withHubProtocol().withUrl('https://localhost:44376/myhub').build();
connection.start().catch((error) => {
console.log(error);
});
connection.on("ReceiveMessage", function (user, message) {
console.log('Message Received: ', user, message);
});
}
`
This will hit the Signalr server but throw an error which I thought was maybe CORS but it seems like the handshake is incorrect and not CORS as the error returned is "Fail to Negotiate with the Server".

Ok sure, "dupe" is the wrong word.
Tracked by https://github.com/aspnet/AspNetCore/issues/6565 is more correct.
We haven't officially supported the client working on React Native yet, but we would like it to work there.


Add the line for asking users for starring the repository.

Description
Add the line for asking users for starring the repository.
This PR fixes #568
Notes for Reviewers
Signed commits

[x] Yes, I signed my commits.

👏


Make generic TypeAdapters possible

Is your feature request related to a problem? Please describe.
Currently, if I have a generic type, I can't register a generic TypeAdapter for that type.
For example, I have the following class (coming from the repository package):
class Id<T> {
  String id;
  ... (custom implementations of ==, hashCode etc.)
  String toString() => '<$T with id $id>';
}

It's then used by my entity classes like this:
@HiveType()
class Article {
  @HiveField(0)
  Id<Article> id;

  @HiveField(1)
  ...
}

Now, I want to save an Article to hive.
I registered the automatically generated ArticleAdapter() and a custom-built Adapter<Id>.
However, hive complains that it doesn't know how to serialize Id<Article>. That's because internally, there's a map of types to their adapters and Id != Id<Article>.
Can this easily be implemented?

Are you using the latest version of Hive? This should actually work...

I'm going to close this since I belive it is already resolved. Please reopen if it doesn't work for you...


--bootstrap-msi-

Versions:

Version of knife-vsphere: knife-vsphere (2.0.0)
Version of chef:
Chef Development Kit Version: 0.10.0
chef-client version: 12.5.1
berks version: 4.0.1
kitchen version: 1.4.2

Platform Details

Version of vSphere: 5.5
Version of ESXi:

Scenario:

I have chef client and other chef rpm located on a nfs mount. I am trying to use knife vsphere to build and then bootstrap new VMs. Is there any option to provide nfs paths with --bootstrap. Since my organization isolates linux/window machines from external network, I need to keep rpm locally on a nfs server or in pulp repo.

Hi, no, file paths won't work, but we do support --bootstrap-msi-url to pass a URI which you could point to your repo.

Hi,
Do I need to both --bootstrap and --bootstrap-msi-url together or only --bootstrap-msi-url.
As I understand --bootstrap downloads and execute install.sh , that will inture download the rpm and will install it. After that the actual "knife bootstrap" step will be executed. When bootstrap gets executed, and it uses user given in --ssh-user, how can I specify sudo ? Normally when we do bootstrap, we use --sudo, to specify that user will be using sudo.

I think --bootstrap-msi-url  doesnt works for Linux VM builds. Is there any other option for Linux machines ?

Hi, the msi options are only for Windows, and yes, you'll need --bootstrap in any case.
Sudo will be automagically enabled if your ssh_user is not root.
Linux bootstrapping without an Internet connection is a bit different. Have a look at custom templates and the --distro option. I've never used this option myself but might be able to help out if you get stuck. This feels like something we should be doing a better job on.
If it is a possibility, there's also --bootstrap-proxy that will point your node at a proxy server to get to the Internet.


Google does not return refresh token without access_type=offline in the auth url.

This can be resolved luckily without code changes by adding it tot the authorization url set in the configuration.

This is did not fix it for me, I've tried using offline instead of offline_access as well.

https://github.com/heptiolabs/gangway/blob/master/docs/google.md#configuring-gangway mentions to leave offline out, but as @Rio mentions, there is no refresh token.

@VihanAgarwal97, make sure you add access_type=offline to your auth url like so: https://accounts.google.com/o/oauth2/v2/auth?access_type=offline

Still missing a refresh token.
apiVersion: v1
data:
  gangway.yaml: >
    apiServerURL: <redacted>

    audience: https://${DNS_NAME}/userinfo

    authorizeURL:
    https://accounts.google.com/o/oauth2/v2/auth?access_type=offline

    clientID: ${GANGWAY_CLIENT_ID}

    clientSecret: ${GANGWAY_CLIENT_SECRET}

    clusterName: <redacted>

    port: 8080

    redirectURL: <redacted>

    scopes:

    - openid

    - profile

    - email

    tokenURL: https://www.googleapis.com/oauth2/v4/token

    usernameClaim: email

I'm using the helm chart to deploy -
      # Values file as block file
      values: |
        gangway:
          scopes: ["openid", "profile", "email"]
          apiServerURL: "<redacted>"
          authorizeURL: "https://accounts.google.com/o/oauth2/v2/auth?access_type=offline"
          tokenURL: "https://www.googleapis.com/oauth2/v4/token"
          clusterName: <redacted>
          # Where to redirect back to. This should be a URL
          redirectURL: "<redacted>/callback"
          usernameClaim: "email"
        env:
        - name: GANGWAY_SESSION_SECURITY_KEY
          valueFrom:
            secretKeyRef:
              name: gangway-test-key
              key: sessionkey
        - name: GANGWAY_AUDIENCE
          valueFrom:
            secretKeyRef:
              name: gangway-secret
              key: client_id # For Google OAuth, clientID is the same as audience
        - name: GANGWAY_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: gangway-secret
              key: client_id
        - name: GANGWAY_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: gangway-secret
              key: client_secret

@Rio the docs specify a different url, I'm not sure what to use.
https://github.com/heptiolabs/gangway/blob/master/docs/google.md#configuring-gangway
This is currently blocking us from using this in prod.

google only issues refresh tokens once per login. To get a refresh token again remove your third party integration from accounts.google.com from your account then login again.


Release v1.0.0

In an effort to avert the absurdity that is ZeroVer, I propose the next pytest-sugar release be anointed v1.0.0, perhaps in conjunction with officially dropping support for Python 3.7, which will very soon reach its end-of-life date.

Thanks @justinmayer -- I do wish that would have happened before the pytest requirements were increased by several major versions in v0.9.7:

we can keep 3.7 in the v1.0.0 release and drop in next release v1.1.0

@vladgurovich: It was not clear that those minimum Pytest version changes were not already in a shipped release — otherwise I would have indeed pushed for this last release to be v1.0.0. In any case, I do not know that there's much we can do about that now, right?
@auvipy: I do not understand. What would that achieve? Python 3.7 is already the minimum supported version.

yes and just keeping that until v1.0.0 unless it is released few months later of EOL of 3.7 will llet users of old systems easier path to upgrade.

I'm not sure what will change in the next two months to make it easier for folks on older systems to upgrade from Python 3.7 to a newer version, so personally I don't think the stated reason is enough to warrant waiting until June to require Python 3.8+.
Other popular packages have already made this change: Flake8, isort, etc.

celery hasn't dropped it yet and will wait until june and next major release v5.3 will include py3.7, although I wanted to drop it but celery is used by lots of old system & big enterprise, so we can't be too aggresive in dropping support.

pytest 8.0.0 was released yesterday:
https://docs.pytest.org/en/latest/changelog.html#pytest-8-0-0-2024-01-27
Please could you make a new release with the https://github.com/Teemu/pytest-sugar/pull/278 fix? It's needed to use pytest-sugar with pytest 8.
Thanks!

Indeed — once #280 has been merged, I will prepare a release. 👍

I merged #280 and submitted PR #281. When the latter PR is merged, pytest-sugar v1.0.0 will be automatically released.

pytest-sugar 1.0.0 has been released, and the package is now available on PyPI 🎉


[HUDI-6990] Configurable clustering task parallelism

Change Logs
Spark executes clustering will generate too many tasks which is equal to the the number of files in clustering plan. Support config it when read files.

Impact
None
Risk level (write none, low medium or high below)
If medium or high, explain what verification was done to mitigate the risks.
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

@hudi-bot run azure re-run the last Azure build

@hudi-bot run azure

@hudi-bot run azure

@hudi-bot run azure

@hudi-bot run azure

Azure looks some problems

You can rebase with the latest master to re-trigger it.

@hudi-bot run azure

Tests have passed: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=20654&view=results


Form validation after making changes in JSON editor

Open app modal, switch to JSON view
Make a 'bad' change like "labels": { "": "12345" }
Switch back to form view
Labels field should be open and the error highlighted

Note: This issue has been migrated to https://jira.mesosphere.com/browse/MARATHON-2550. For more information see https://groups.google.com/forum/#!topic/marathon-framework/khtvf-ifnp8.


Elixir.ArgumentError: assign @user not available in eex template.

https://sentry.io/lifted-studios/atom-tweaks/issues/571304174/
Elixir.ArgumentError: assign @user not available in eex template.

Please make sure all proper assigns have been set. If this
is a child template, ensure assigns are given explicitly by
the parent template as they are not automatically forwarded.

Available assigns: [:view_module, :view_template]

  File "lib/phoenix_html/engine.ex", line 116, in Phoenix.HTML.Engine.fetch_assign/2
  File "lib/atom_tweaks_web/templates/star/blankslate.html.slime", line 1, in AtomTweaksWeb.StarView."blankslate.html"/1
  File "lib/atom_tweaks_web/templates/star/index.html.slime", line 1, in AtomTweaksWeb.StarView."index.html"/1
  File "lib/atom_tweaks_web/templates/layout/app.html.slime", line 1, in AtomTweaksWeb.LayoutView."app.html"/1
  File "lib/phoenix/view.ex", line 332, in Phoenix.View.render_to_iodata/3
...
(3 additional frame(s) were not displayed)

(ArgumentError) assign @user not available in eex template.

Please make sure all proper assigns have been set. If this
is a child template, ensure assigns are given explicitly by
the parent template as they are not automatically forwarded.

Available assigns: [:view_module, :view_template]

Occurs when navigating to /users/:username/stars when the user has not starred any tweaks yet.


trouble with get_historic_data

Hi there
How can I access historical data after I call that function? It just tells us how many bars are there. I tried to read the historic_data_file in the DWX folder but it doesn't;t update when I change the currency. Also, sometimes It doesn't make any file in that directory.

Hi,
If you call the get_historic_data function, the responds should trigger the on_historic_data(self, symbol, time_frame, data) function of the tick_processor class as in the example script.
Per default it just prints the length of the data, but you can also print the whole thing, or do any calculations with it.

Many thanks.
Another question:
Is there any way to get some indicator data? for instance, can I access the RSI of MACD, etc.?
I already know we can recode those indicators but this would be easier :)

Many thanks. Another question: Is there any way to get some indicator data? for instance, can I access the RSI of MACD, etc.? I already know we can recode those indicators but this would be easier :)

Hi @bojnordsky . One option you could employ is the  pandas-ta library module, which offers some common indicators.
If not, I think another alternative would be to include the functions for your indicator into the EA and make calls to it..

Many thanks. Another question: Is there any way to get some indicator data? for instance, can I access the RSI of MACD, etc.? I already know we can recode those indicators but this would be easier :)

Hi @bojnordsky . One option you could employ is the pandas-ta library module, which offers some common indicators. If not, I think another alternative would be to include the functions for your indicator into the EA and make calls to it..

Many thanks. And what about the other question?

You can subscribe to bar data for multiple symbols, but only request historic data for single symbols.
However, you can just send get_historic_data in a loop for multiple symbols. The response to all these requests will be stored in a dictionary self.dwx.historic_data, which you could use.


How to run AndFixDemo?

After I open  the project by Android studio. It looks like a library project.
How can I run "AndFixDemo"?
Thanks

Import samplesI/AndFixDemo to your IDE, append AndFixDemo dependencies with AndFix(library project or aar).
Build project, save the package as 1.apk, and then install on device/emulator.
Modify com.euler.test.A, references com.euler.test.Fix.
Build project, save the package as 2.apk.
Use apkpatch tool to make a patch.
Rename the patch file to out.apatch, and then copy it to sdcard.
Run 1.apk and view log.


Auto DNS on Windows

I'm having issue with Auto DN S(Bind) on Windows Machine
Docker version 17.05.0-ce, build 89658be
docker-compose version 1.13.0, build 1719ceb8
Using Docker Toolbox on Windows
My LOCAL_LISTEN_ADDR=<IP_ADDRESS>:
DNS Server are set to <IP_ADDRESS>; <IP_ADDRESS>; <IP_ADDRESS> In my Network Config
And everything works except for Auto DNS option, which i'd really like, because editing /etc/hosts on Windows is slightly annoying.
When I do edit it, it does propagate properly, so I can confirm everything is working on that side.
But DNS Server is not accessible through <IP_ADDRESS>, I tested it as standalone, and internet becomes inaccessible completely.
Anyways, I saw you are planning on including some Windows related fixes, so you might want to include this as well.
Not sure where exactly does the issue occur, or if I'm not doing something properly(Although I think I covered the most of it), so if anything else is needed, let me know I'll provide further details.

@AgarvaiN this is an issue related to how Docker Toolbox works. It only works via (afaik) a VirtualBox middle man that provides the actual docker functionality.
The httpd server has an internal IP address (inside the toolbox): <IP_ADDRESS> to which the DNS server is resolving the TLD_SUFFIX domains to, however this IP address does not seem to be reachable from your host computer, because you would need it to point to <IP_ADDRESS> (The VirtualBox host). If you changed this value of the DNS server in docker-compose.yml, the the php docker won't be able to issue DNS queries anymore, because it still expects it to go to <IP_ADDRESS>.
Hope this is somehow understandable (a little bit complicated to explain for me).
However, it should work if you use the native Docker for windows: Docker for Windows. Could you give this a try instead of the old Toolbox.
In the mean-time I will think a bit if it can be accomplished with the toolbox via some port-forwarding.

I might have a workaround:
In docker-compose.yml edit WILDCARD_ADDRESS,  DNS_FORWARDER and FORWARD_PORTS_TO_LOCALHOST with the following values:
bind:
  ...
    environment:
      - WILDCARD_ADDRESS=<IP_ADDRESS>
      - DNS_FORWARDER=${BIND_DNS_RESOLVER:-<IP_ADDRESS>,<IP_ADDRESS>}

php:
  ...
    environment:
      - FORWARD_PORTS_TO_LOCALHOST=80:httpd:80,3306:mysql:3306,5432:pgsql:5432,6379:redis:6379,11211:memcd:11211,27017:mongo:27017,5672:rabbit:5672,15672:rabbit:15672

Then the only thing left to do for you is to forward port 80 from <IP_ADDRESS> to your hosts <IP_ADDRESS> address

Yeah I understand it's related to Toolbox, but since I need to be able to use VMWare I can't run native Docker because it needs Hyper-V enabled to run. And I'm guessing there might be others out there like me.
That didn't do the trick for me tho. DNS_FORWARDER was already set to those, I updated WILDCARD_ADDRESS and FORWARD_PORTS_TO_LOCALHOST (I did try with the old IP as well since it points to internal php container's ip, which kinda makes sense to me), and I forwarded port for my docker-machine, but I think that was working already since I was able to access devilbox and my projects through the <IP_ADDRESS>. Unless you meant something else by that.

Docker
Hostname
IP

php
<IP_ADDRESS>

httpd
<IP_ADDRESS>

mysql
<IP_ADDRESS>

redis
<IP_ADDRESS>

memcd
<IP_ADDRESS>

bind
<IP_ADDRESS>

Docker
Host port
Docker port

php
-
9000

httpd
<IP_ADDRESS>:80
80

mysql
<IP_ADDRESS>:3306
3306

redis
<IP_ADDRESS>:6379
6379

memcd
<IP_ADDRESS>:11211
11211

bind
<IP_ADDRESS>:1053/tcp<IP_ADDRESS>:1053/udp
53/tcp53/udp

If it helps

Sorry, one more thing, on my host machine do I leave the DNS set to <IP_ADDRESS>?

The actual issue is to find a common denominator:
The DNS server must resolve TLD_SUFFIX domains always to an IP address where the httpd server is. This must be equal for two places:

From within the php container
From your host computer (not the VirtualBox).

That means there must be one IP address that where you can reach the httpd server and this IP address must work from within the PHP container and from your host computer.
So either <IP_ADDRESS> or <IP_ADDRESS>. In case none of the two work for both cases, we must somehow map it to <IP_ADDRESS>.  And that is what I tried to achive by forwarding the httpd port to the <IP_ADDRESS> of the php container. With that, the DNS server must resolve all domains to <IP_ADDRESS> as well. The last step is to have <IP_ADDRESS>:80 on your host computer point to the webserver.
Additionally, The DNS server's port must be changed in .env to 53 and integrated as your primary DNS server for windows.

Btw, is <IP_ADDRESS> reachable from within the PHP container?

https://github.com/cytopia/devilbox/issues/248
I am already working on this problem. Please find all relevant information in the newly created issue: https://github.com/cytopia/devilbox/issues/248
To reach a broader audience I will close this ticket and link it to the new one (which will be done with multiple related issues as well).


update qps and burst for webhook

Signed-off-by: ldpliu<EMAIL_ADDRESS>

/assign @qiujian16 @elgnay

maybe we should put this as flag and set some default value

it's not easy to add the qps and burst flag for webhook.
As current webhook use the o.RecommendedOptions
https://github.com/open-cluster-management-io/registration/blob/fd33e20727c4d295f49b18c249d279369426c230/pkg/cmd/webhook/webhook.go#L46

I think you can create another AddFlags options for this, right?

I think you can create another AddFlags options for this, right?

Done, please help to  review again.

/hold need test

/unhold


hotfix: treat unknown crowding as :train_level

Summary of changes
See below

This guards against trying to process a multi_carriage_details with < 6 elements, which caused a couple crashes. The underlying data issue is also being addressed, but it may take some time, and even afterward it seems prudent to avoid crashing in case something similar happens again.
Note: The first commit on this branch was deployed to production as a hotfix this morning. I added the second commit as a more targeted fix for the specific case. Technically that makes the original fix redundant, but I'm inclined to leave it as a reasonable fallback for the "emptier location" logic.


enable config driven ack, retry and serializer for kafka

enable config driven ack, retry and serializer for kafka - added on behalf of @Parag.Mangal

enhancement implemented with commit: https://github.com/americanexpress/jakapu/commit/e305136185b1643c2512fed99f98fddbd1d235de


Batch store updates with spread

Since spread uses separate sample statement for each provided target, effector does not batch store updates
spread({
  source: somethingHappened,
  targets: { a: storeA, b: storeB, c: storeC }
})

combine([storeA, storeB, storeC]).watch(console.log)

somethingHappened({ a: 1, b: 2, c: 3 }) // Will re-calculate combine three times

Suggested fix is to check target type and, if it's a store, do store.on(source, ...) instead of sample(...)

@Kelin2025 Can you add test for this behaviour?


Corrections w.r.t. the address of the LCDC register

This PR fixes two mistakes:

the description of the LCDC status interrupt was claiming STAT lives at $FF40
the address of LCDC wasn't mentioned in the section describing it

Looks good to me, thanks @NieDzejkob


[Fix #336] Expand the safety warning on Performance/Detect

In addition to the external library activerecord, the built in Hash class also has compatibility issues.
I'm unsure if this deserves a changelog entry, given it doesn't change the code there shouldn't be an impact (which is why I didn't add tests either).

Before submitting the PR make sure the following are checked:

[x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
[x] Wrote good commit messages.
[x] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[ ] Added tests.
[x] Ran bundle exec rake default. It executes all tests and runs RuboCop on its own code.
[ ] Added an entry (file) to the changelog folder named {change_type}_{change_description}.md if the new code introduces user-observable changes. See changelog entry format for details.

Yeah, this change does not require a changelog entry and tests. Thank you for the improvement!

I understand that this cop's autocorrection can fail depending on the receiver. However, I think that the autocorrection will succeed correctly both when the receiver is a Hash and a ActiveRecord::Relation. Can you give me an example of in which case it would fail?

In the original issue I added an example: https://github.com/rubocop/rubocop-performance/issues/336.

I think you're right that it could be phrased better. Should this be considered sufficient:

This cop is unsafe because is assumes the class implements the Enumerable interface, but can't reliably detect this.

Then drop this part with specific method names?

Hash and ActiveRecord do not implement a detect method and find has its own meaning.

So it becomes:

This cop is unsafe because is assumes the class implements the Enumerable interface, but can't reliably detect this. This creates known compatibility issues with Hash, ActiveRecord and other frameworks. Correcting Hash and ActiveRecord methods with this cop should be considered unsafe.

hmm, since this cop depends on Array#reverse, it seems more appropriate to write the following:

This cop is unsafe because it assumes that the receiver is an Array or equivalent, but can't reliably detect it.

On another point, I feel it would be better to remove the explanation about ActiveRecord. Because in fact #find in ActiveRecord::Relation delegates to Enumerable#find when a block is passed, and ActiveRecord::Relation can also respond to #reverse, so it shouldn't be a problem.
https://github.com/rails/rails/blob/v<IP_ADDRESS>/activerecord/lib/active_record/relation/finder_methods.rb#L68


Update result_5

Removed postcode data as PIP is being rolled out nationally.

Thanks, Liz. Closing in favour of #1948


Typo in comment

Maintenance.py, line 3
# Maintencance stuff, locked to owner only Change to # Maintenance stuff, locked to owner only

fixed with https://github.com/AEnterprise/Outboard/commit/6d028e9f64a5db1df22c88d4a7ee22ebc6a6f142


bug: StoreNotMatch

2017/11/21 12:23:11.374 region_cache.go:447: [info] drop regions of store 1 from cache due to request fail, err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2017/11/21 12:23:11.378 region_cache.go:447: [info] drop regions of store 1 from cache due to request fail, err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2017/11/21 12:23:11.389 region_cache.go:447: [info] drop regions of store 1 from cache due to request fail, err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2017/11/21 12:23:13.545 region_request.go:164: [warning] tikv reports `StoreNotMatch`: , ctx: &{{1009 3 4} id:1009 start_key:"t\200\000\000\000\000\000\000O_i\200\000\000\000\000\000\000\014\0012104\000\000\000\000\373\003\200\000\000\000\013\271\210]" end_key:"t\200\000\000\000\000\000\000O_r\200\000\000\000\013\2662^" region_epoch:<conf_ver:3 version:4 > peers:<id:1010 store_id:1 > peers:<id:1011 store_id:7 > peers:<id:1012 store_id:4 >  id:1011 store_id:7  <IP_ADDRESS>:20160}, retry later

We can ensure that TiDB connects the right PD, TiKV reports correct address and store ID to PD and PD also has correct cluster information, so meeting this error is very strange.
Btw, I find that we will output the error in scheduler but not in coprocessor, and the coprocessor error metric will use "other" for this error, this is not intuitive. /cc @BusyJay

... and the coprocessor error metric will use "other" for this error...

I'm not sure if I understand it correctly. You mean this line will be encountered? But that's impossible. If coprocessor reports other error, All TiDB receives is just an error string, it has no idea what the precise error TiKV returns, hence it can't output a log about StoreNotMatch.

I believe it is caused by data race in region_cache. See https://github.com/pingcap/tidb/pull/5171

@BusyJay
I think we should add a metric for invalid propose like we do for invalid raft message.


Page auto jump to top when ketcher editor inited

Steps to Reproduce
use ketcher editor in a dialog, in a long page. So the ketcher will not render when page loaded, but when user open the dialog
Actual behavior
The page will auto scroll to the page top
Expected behavior
The page keep the old position, no any jump
Screenshots
If applicable, add screenshots to help explain your problem.
code source: https://github.com/epam/ketcher/blob/be6713092078019ed12b7e732bea4da2ac937696/packages/ketcher-react/src/script/ui/App/App.tsx#L67
Desktop (please complete the following information):

OS: iOS
Browser: chrome
Version: 120.0.6099.129

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Ketcher version [e.g. v2.4.2].
"ketcher-core": "2.11.0",
"ketcher-react": "2.8.0",
"ketcher-standalone": "2.11.0",
Test case
Add test case number (in EPMLSOPKET-* format) if bug was found during test case run.
Additional context
Add any other context about the problem here.

I also ran into this and it is really annoying.
The assumption that the editor would always appear at the top of the page (I guess that is the reason) is quite questionable.
In my case the editor should appear inside a large form somewhere in the middle of a page.
This behavior currently makes it impossible to load ketcher lazily which would be quite recommended due to its large bundle size.

Hi @flc and @Valen-C12
Thank you for highlighting the issue.
We will look into it and prioritize.

Dear @flc
We tried to reproduce the issue however we don't see behaviour which you described.
Could you please send us video demonstrating the problem or provide demo page(preferable) where we can reproduce it?

Sure, I'll try to put together a demo soon.
Thanks for the investigation!

I created a repo that demonstrated the problem:
https://github.com/flc/ketcher-scroll-issue
A demo video is also included in the readme.
What's strange is that the problem doesn't occur in Firefox, but it does in every other browser I've tried.

Hi @flc
Thank you, let us check. Will come back soon

Dear @Valen-C12  and @flc
We merged the fix. It will be available in ketcher v2.28.0. First release candidate will be available in npm next week. Public release will be published after stabilisation phase.

Verified.

Ketcher Version 2.28.0-rc.1 Build at 2024-12-03; 17:04:52
Indigo Version 1.27.0-rc.1.0-g8d656a341-wasm32-wasm-clang-19.0.0
Chrome Version 131.0.6778.109 (Official Build) (64-bit)
Win10


Why not return a permissions: array < out string > in the method of onpermissiondenied annotation to indicate the permission set rejected

Why not return a permissions: array < out string > in the method of onpermissiondenied annotation to indicate the permission set rejected

Could you elaborate more why you claimed that and what's the benefit we can obtain then?

let me close the issue due to a lack of supplementary info.


Add model GEITje Ultra / Dutch models

Hi,
Would anyone be able to add GEITje Ultra to ollama? It is a Dutch oriented model based on mistral/mixtral or llama3. The llama3 is called llama3 Dutch. I dont know yet where I would be able to find it, but I suppose you can as it has been opensourced according to this Dutch Tech website; https://tweakers.net/reviews/12036/3/chatgpt-maar-dan-uit-nederland-nederlandse-taalmodellen-geitje-en-geitje-ultra.html
I hope this can happen,
Thanks for your time,
Johan

Just in case anyone wonders why?

Dutch events, persons, politicians, and even the language are not very well integrated into even ChatGPT 4 Turbo
Translations to Dutch are even on ChatGPT 4 Turbo underperforming compared to English, especially in some areas regarding topics related to Dutch persons. It is however getting acceptable now in chatGPT 4 Turbo
the Dutch language is not the same everywhere. You can see that in Belgium (the Dutch part) a more conservative approach exists to language and facts including more Germanic / English approach with all kind of more indirect approach of facts and sentence construction. It makes Belgium Dutch more indirect and abstract, while the Dutch are more direct and progressive, making things less abstract, all while using the same dictionary. A bit as if you would aggregate all English language from different countries in one dictionary, and call it English.

The Model is already available on Ollama: https://ollama.com/bramvanroy/geitje-7b-ultra


[COMMUNITY] An Wang -> Reviewer

Please join me in welcoming An Wang (@anwang2009) as a new reviewer in TVM. An has made significant contributions various TVM frontends and help the community.

Commits History
Code Review

Congrats An! This is now merged.


Validation to enforce buffer size to be an odd number greater than or equal to 3

Two scenarios are covered:
-bufferSize is less or equal to 1, throw an error and notify users that the allowed bufferSize value must be an odd number greater or equal to 3.

-bufferSize is above 1 and is even, throw an error that notifies the users that the allowed bufferSize value must be an odd number greater or equal to 3.

Test cases have been added for the scenarios listed above.

@VMois These are great suggestions and all have been implemented on the latest commit.

I'm hoping I did that correctly. It should be one commit now. Linting has also been fixed. I needed to break several lines up into two lines as they were over the max line length.

Nice work! Thank you for your contribution!


DuckDB query can break the Kestra container

Issue description
This flow will work fine:
id: query
namespace: md

tasks:
  - id: query
    type: io.kestra.plugin.jdbc.duckdb.Query
    sql: |
      SELECT month(order_date) as month, sum(total) as total
      FROM read_csv_auto('s3://kestraio/monthly_orders/*.csv', FILENAME = 1)
      GROUP BY 1
      ORDER BY 2 desc;
    store: true
    timeout: PT20S
    url: "jdbc:duckdb:md:my_db?motherduck_token={{ secret('MOTHERDUCK_TOKEN') }}"

  - id: csv
    type: io.kestra.plugin.serdes.csv.CsvWriter
    from: "{{ outputs.query.uri }}"

Now try to access an S3 directory that doesn't exist:
id: query
namespace: md

tasks:
  - id: query
    type: io.kestra.plugin.jdbc.duckdb.Query
    sql: |
      SELECT month(order_date) as month, sum(total) as total
      FROM read_csv_auto('s3://kestraio/monthly_orders_2023/*.csv', FILENAME = 1)
      GROUP BY 1
      ORDER BY 2 desc;
    store: true
    timeout: PT20S
    url: "jdbc:duckdb:md:my_db?motherduck_token={{ secret('MOTHERDUCK_TOKEN') }}"

  - id: csv
    type: io.kestra.plugin.serdes.csv.CsvWriter
    from: "{{ outputs.query.uri }}"

DuckDB normally would return IOError and exit:

However, this is what happens in Kestra container:
2023-07-24 11:38:09 2023-07-24 09:38:09,826 INFO  default-nioEventLoopGroup-1-5 io.kestra.webserver.access [Date: 2023-07-24T09:38:09.578668125Z] [Duration: 243 ms] [Method: POST] [Url: /api/v1/executions/trigger/md/query] [Status: 200] [Length: 295] [Ip: <IP_ADDRESS>] [Port: 8080]
2023-07-24 11:38:09 2023-07-24 09:38:09,932 INFO  default-nioEventLoopGroup-1-5 io.kestra.webserver.access [Date: 2023-07-24T09:38:09.913291042Z] [Duration: 18 ms] [Method: GET] [Url: /api/v1/flows/md/query?source=true&id=query&namespace=md&deleted=true] [Status: 200] [Length: 477] [Ip: <IP_ADDRESS>] [Port: 8080]
2023-07-24 11:38:10 2023-07-24 09:38:10,368 INFO  jdbc-queue_4 flow.query   [namespace: md] [flow: query] [execution: 3kxBrIkExFs9MZQMXSM2nU] Flow started
2023-07-24 11:38:14 2023-07-24 09:38:14,132 INFO  worker_329   flow.query.query [namespace: md] [flow: query] [task: query] [execution: 3kxBrIkExFs9MZQMXSM2nU] [taskrun: 1qObqqLYXFV5awn614sNcq] [value: null] Type Query started
2023-07-24 11:38:24 /bin/bash: line 1:     6 Aborted                 /app/kestra server standalone --worker-thread=128

can it be that IO errors are not handled properly?

when restarting the container, the execution is stuck in a Running state

Sorry for a bad reproducer, here is one that should work directly, even without MotherDuck, you'd only need to replace AWS credentials and the bucket name variable:
this should work:
id: s3Upload
namespace: md

variables:
  bucket: kestraio

tasks:
  - id: s3upload
    type: io.kestra.plugin.scripts.python.Script
    docker:
      image: ghcr.io/kestra-io/aws:latest
    env:
      AWS_ACCESS_KEY_ID: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      AWS_SECRET_ACCESS_KEY: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
      AWS_DEFAULT_REGION: "{{ secret('AWS_DEFAULT_REGION') }}"
    script: |
      import requests
      import boto3
      from kestra import Kestra

      BUCKET = "{{vars.bucket}}"

      def extract_and_upload(file):
          url = f"https://raw.githubusercontent.com/kestra-io/datasets/main/{file}"

          response = requests.get(url)
          data = response.content.decode("utf-8")
          s3 = boto3.resource("s3")
          s3.Bucket(BUCKET).put_object(Key=file, Body=data)
          print(f"{url} downloaded and saved to {BUCKET}/{file}")

      for month in range(1, 13):
          filename = f"monthly_orders/2023_{str(month).zfill(2)}.csv"
          extract_and_upload(filename)
          Kestra.outputs({f"{filename}": f"s3://{BUCKET}/{filename}"})

  - id: query
    type: io.kestra.plugin.jdbc.duckdb.Query
    sql: |
      INSTALL httpfs;
      LOAD httpfs;
      SET s3_region='{{ secret('AWS_DEFAULT_REGION') }}';
      SET s3_access_key_id='{{ secret('AWS_ACCESS_KEY_ID') }}';
      SET s3_secret_access_key='{{ secret('AWS_SECRET_ACCESS_KEY') }}';
      SELECT month(order_date) as month, sum(total) as total
      FROM read_csv_auto('s3://{{vars.bucket}}/monthly_orders/*.csv', FILENAME = 1)
      GROUP BY 1
      ORDER BY 2 desc;
    store: true
    timeout: PT30S

  - id: csv
    type: io.kestra.plugin.serdes.csv.CsvWriter
    from: "{{ outputs.query.uri }}"

Now changing the path in the DuckDB query from monthly_orders to monthly_orders_2023 should break the container (at least it did for me back then)
but... now on the latest develop full it works as expected 😅

even when using remote DB with MotherDuck
id: s3Upload
namespace: md

variables:
  bucket: kestraio

tasks:
  - id: s3upload
    type: io.kestra.plugin.scripts.python.Script
    docker:
      image: ghcr.io/kestra-io/aws:latest
    env:
      AWS_ACCESS_KEY_ID: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      AWS_SECRET_ACCESS_KEY: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
      AWS_DEFAULT_REGION: "{{ secret('AWS_DEFAULT_REGION') }}"
    script: |
      import requests
      import boto3
      from kestra import Kestra

      BUCKET = "{{vars.bucket}}"

      def extract_and_upload(file):
          url = f"https://raw.githubusercontent.com/kestra-io/datasets/main/{file}"

          response = requests.get(url)
          data = response.content.decode("utf-8")
          s3 = boto3.resource("s3")
          s3.Bucket(BUCKET).put_object(Key=file, Body=data)
          print(f"{url} downloaded and saved to {BUCKET}/{file}")

      for month in range(1, 13):
          filename = f"monthly_orders/2023_{str(month).zfill(2)}.csv"
          extract_and_upload(filename)
          Kestra.outputs({f"{filename}": f"s3://{BUCKET}/{filename}"})

  - id: query
    type: io.kestra.plugin.jdbc.duckdb.Query
    sql: |
      SELECT month(order_date) as month, sum(total) as total
      FROM read_csv_auto('s3://{{vars.bucket}}/monthly_orders_2023/*.csv', FILENAME = 1)
      GROUP BY 1
      ORDER BY 2 desc;
    store: true
    timeout: PT30S
    url: "jdbc:duckdb:md:my_db?motherduck_token={{ secret('MOTHERDUCK_TOKEN') }}"

  - id: csv
    type: io.kestra.plugin.serdes.csv.CsvWriter
    from: "{{ outputs.query.uri }}"

weird.
Closing the issue then for now, thanks for rubberducking @loicmathieu


Potentially superfluous function _addStandardForm?

In the newsletter field's displyPublishPanel function, all the states (cases) use
$this->_addStandardForm()

which creates hidden form inputs.
It is simply unclear if this is (still) needed.

I believe it is. However, I will have to remove the function to see if this is really the case - to see what happens.

I took a look at the HTML source code and did a short test. Without this function, newsletter-specific values get lost when saving the entry. So we should keep it. :-)
I am closing this.


Minor refactor prepare_module

Start to refactor prepare_module to try to decouple functionality.

@lhoestq this is the PR that I mentioned to you, which can be considered as a first step in refactoring prepare_module.


Boosts Dungeoneering

Description:
Requested by:

Changes:

Removed negative boost from soloing;
Adding 10% boost for each player that joins the party;
Changed the quantity of floors done to do the maximum allowed after all boosts/effects is applied, instead of always doing the minimum possible and returning earlier;
Changed Scroll of Mystery to roll for every floor done and not once per trip.

Other checks:

[X] I have tested all my changes thoroughly.

Did you do XP/HR measurements?

Should be 20% faster at solo, due to the removal of the penalty, which was highly requested. Limited trips to 5 players max (out of 12). A 5 man team should around 50% faster than it is now.


Tradfri Test server option support - ikea_ota_use_test_url option missing

zigbee2mqtt add-on version (if edge, please report commit hash):
6126ead
Operating environment (HassOS, Virtual Machine, Device/platform running Home Assistant):

HassOS 5.6
Description of problem:
https://github.com/Koenkk/zigbee2mqtt/issues/2921#issuecomment-741977661 this setting can't be added right now.
Your entire configuration from the frontend (with sensitive fields redacted):
data_path: /share/zigbee2mqtt
external_converters: []
devices: devices.yaml
groups: groups.yaml
homeassistant: true
permit_join: true
mqtt:
  base_topic: zigbee2mqtt
  server: 'mqtt://homeassistant'
  user: CENSOR
  password: CENSOR
  include_device_information: true
  version: 5
serial:
  port: /dev/ttyACM0
advanced:
  log_level: info
  pan_id: CENSOR
  channel: CENSOR
  network_key:
    - CENSOR
  availability_blocklist: []
  availability_passlist: []
  availability_timeout: 60
  report: true
  last_seen: ISO_8601
  rtscts: true
  **ikea_ota_use_test_url: true**
device_options: {}
blocklist: []
passlist: []
queue: {}
frontend:
  port: 8099
experimental:
  new_api: true
  transmit_power: 14
socat:
  enabled: false
  master: 'pty,raw,echo=0,link=/dev/ttyZ2M,mode=777'
  slave: 'tcp-listen:8485,keepalive,nodelay,reuseaddr,keepidle=1,keepintvl=1,keepcnt=5'
  restartdelay: 1
  initialdelay: 1
  options: '-d -d'
  log: false

Your logs from Home Assistant

Just a question because I am interested. Why does the configuration of the addon does not use a scheme in the zigbee2mqtt source to validate the Json or just pass the configuration? It does not make sense to me that for a configuarational change a whole new build and release is needed.

@groenmarsmannetje The config for the add-on is based on the requirements from Home Assistant Supervisor. This allows only for a 2 layer config and no more:
layer1:
  layer2:

In many cases the configuration from Zigbee2mqtt has multiple layers, so this has to be adapter to work with the Supervisor. We do want to have the configuration editable from the Add-on UI as in the future, with the right "schema" we will get better UI for this.
Imagine booleans being toggle switches, ranges to be sliders & such. While this is painful at times to keep in sync, and new features have to be added by hand, it does give a lot of benefit for users that will get to use an easier workflow to maintain their configuration.

TBH I prefer the UI configurable way like we have it right now. HA tries to become more and more user friendly with every new version and switching back to a config file to edit sounds like a big step back. So I fully support the current way to go. And most important... It does not happen very often that a new parameter is added to the config of z2m. So no problem from my side to wait a bit until things sum up until a update is really worth the effort. Just wanted to be sure that it's on the todo list, that's why I made the report ^^

1.17 the same... [supervisor.addons.validate] Unknown options ikea_ota_use_test_url :(

1.17 the same... [supervisor.addons.validate] Unknown options ikea_ota_use_test_url :(

I will try to include this in <IP_ADDRESS> on this repo and in 1.17.0-1 on https://github.com/zigbee2mqtt/hassio-zigbee2mqtt soon.

I will try to include this in <IP_ADDRESS> on this repo and in 1.17.0-1 on https://github.com/zigbee2mqtt/hassio-zigbee2mqtt soon.

still missing in <IP_ADDRESS>
by the way? is there a guide on how to change from this repo to the new one without loosing config/zigbee devices?

still missing in <IP_ADDRESS>
by the way? is there a guide on how to change from this repo to the new one without loosing config/zigbee devices?

https://github.com/zigbee2mqtt/hassio-zigbee2mqtt
This is new repo ?

https://github.com/zigbee2mqtt/hassio-zigbee2mqtt
This is new repo ?

@filewalker1 unfortunately I forgot there was a pending PR on this one, so the release version for this will be <IP_ADDRESS>
The new repo works out of the box with your old config and will reuse your config/ database & all data for devices, so you can swap seamlessly. There are two options removed from socat section that you will see as warning on startup.
@KRISUUUU yes, that will become the new official repo once all the docs changes are complete and merged (https://github.com/Koenkk/zigbee2mqtt.io/pull/544 and https://github.com/Koenkk/zigbee2mqtt/pull/5546)

@filewalker1 unfortunately I forgot there was a pending PR on this one, so the release version for this will be <IP_ADDRESS>
The new repo works out of the box with your old config and will reuse your config/ database & all data for devices, so you can swap seamlessly. There are two options removed from socat section that you will see as warning on startup.
@KRISUUUU yes, that will become the new official repo once all the docs changes are complete and merged (https://github.com/Koenkk/zigbee2mqtt.io/pull/544 and https://github.com/Koenkk/zigbee2mqtt/pull/5546)

@ciotlosm thank you very much! so no need to backup and manually restore anything (except configuration) - that is great. Keep up the good work :)

@ciotlosm thank you very much! so no need to backup and manually restore anything (except configuration) - that is great. Keep up the good work :)

@filewalker1 Thanks. The original repo will also remain for quite a few months in parallel, but will drop all code and adopt the new versioning scheme by just reusing the built artefacts from the new one. In time we will delay updates to the original one (by a week or so) to allow people to rollback in case new one is problematic for them.
I'm not sure how long we will run them in parallel in the long run, but I guess we will see in time how the community adapts to this new flow.

@filewalker1 Thanks. The original repo will also remain for quite a few months in parallel, but will drop all code and adopt the new versioning scheme by just reusing the built artefacts from the new one. In time we will delay updates to the original one (by a week or so) to allow people to rollback in case new one is problematic for them.
I'm not sure how long we will run them in parallel in the long run, but I guess we will see in time how the community adapts to this new flow.

@ciotlosm Thank you for your answer! Great job

@ciotlosm Thank you for your answer! Great job

@KRISUUUU yes, that will become the new official repo once all the docs changes are complete and merged (Koenkk/zigbee2mqtt.io#544 and Koenkk/zigbee2mqtt#5546)

@ciotlosm I've swapped to that repo, but I still get the error:
21-01-23 15:44:03 WARNING (MainThread) [supervisor.addons.validate] Unknown options ikea_ota_use_test_url
Wasn't the new repo supposed to be updated with the OTA option?

@KRISUUUU yes, that will become the new official repo once all the docs changes are complete and merged (Koenkk/zigbee2mqtt.io#544 and Koenkk/zigbee2mqtt#5546)

@ciotlosm I've swapped to that repo, but I still get the error:
21-01-23 15:44:03 WARNING (MainThread) [supervisor.addons.validate] Unknown options ikea_ota_use_test_url
Wasn't the new repo supposed to be updated with the OTA option?

Well, ppl will hate me for making them think the report got an update, buuuut.... I understood that he wanted to make this one the more recent experimental one ad the other the final official one. Correct me if I am wrong here.

Well, ppl will hate me for making them think the report got an update, buuuut.... I understood that he wanted to make this one the more recent experimental one ad the other the final official one. Correct me if I am wrong here.

Working on 1.17.1 ?

Easy to check: https://github.com/danielwelch/hassio-zigbee2mqtt/commit/3793c6cb4c19ae6b5a7d4cb164b8b2cf6507fc93 Nope

OTA Update still not working
Info MQTT publish: topic 'zigbee2mqtt/bridge/log', payload '{"message":"Update of 'Switch.1' failed (Image is truncated: not long enough to contain a valid tag)","meta":{"device":"Switch.1","status":"update_failed"},"type":"ota_update"}'
Info MQTT publish: topic 'zigbee2mqtt/bridge/response/device/ota_update/update', payload '{"data":{"id":"Switch.1"},"error":"Update of 'Switch.1' failed (Image is truncated: not long enough to contain a valid tag)","status":"error","transaction":"8sjj9-1"}'
I have the test url in my configuration.yaml
serial:
adapter: deconz
port: /dev/ttyACM0
advanced:
ikea_ota_use_test_url: true
lpog_level: info
pan_id: 6754
channel: 11
network_key:
Zigbee2mqtt
Current version: 1.17.1

This was now added in 1.17.1-1 and edge for https://github.com/zigbee2mqtt/hassio-zigbee2mqtt. Version 1.17.1-1 is not yet published, but it should be soon.

I just installed 1.17.1-1 and ikea ota is working fine with the test server.
Now updating one of my Ikea lights.
Thanks!

How can i install 1.17.1-1?

From https://github.com/zigbee2mqtt/hassio-zigbee2mqtt

Ah i installed it from : https://github.com/danielwelch/hassio-zigbee2mqtt

Any guide for upgrade to other Repo?

copy config, install new one, paste config

Still no update :(
nfo MQTT publish: topic 'zigbee2mqtt/bridge/log', payload '{"message":"Update of 'Switch.1' failed (Image is truncated: not long enough to contain a valid tag)","meta":{"device":"Switch.1","status":"update_failed"},"type":"ota_update"}'
Info MQTT publish: topic 'zigbee2mqtt/bridge/response/device/ota_update/update', payload '{"data":{"id":"Switch.1"},"error":"Update of 'Switch.1' failed (Image is truncated: not long enough to contain a valid tag)","status":"error","transaction":"4hm5m-1"}'
Error Update of 'Switch.1' failed (Image is truncated: not long enough to contain a valid tag)

I installed it from that other repo and put my old config back but then it looks at the old share again.
When I look in the log it says 1.17.1 is starting.
While I installed 1.17.1-1 from: https://github.com/zigbee2mqtt/hassio-zigbee2mqtt/tree/master/zigbee2mqtt
I'll wait for it to be on my other repo: https://github.com/danielwelch/hassio-zigbee2mqtt

New repo works to me!

copy config, install new one, paste config

Do i need to repair devices after switching?

For me is copy config install new one en past config nog working.
It says installed 1.17.1-1 but in the log he says starting 1.17.1 and by about is also says 1.17.1

@mgiebels it will say 1.17.1 at startup because that's the z2m version. You don't need to repair devices, and now the current repo will show 1.17.1-1 available because we're deprecating the old repo so the images are common, configs are just copied over.

it will say 1.17.1 at startup because that's the z2m version. You don't need to repair devices, and now the current repo will show 1.17.1-1 available because we're deprecating the old repo so the images are common, configs are just copied over.

Yes it just doesn't work.
Ikea update in advanced is not shown and the updates do not work either.
Then what am I doing wrong?

Did you add ikea_ota_use_test_url in advanced and it doesn't let you save?

Did you add ikea_ota_use_test_url in advanced and it doesn't let you save?

I do not know where to enter this, I cannot find an option for this

@mgiebels if you are using HassOS add-on you should find it under Supervisor > Dashboard > Zigbee2mqtt > Configuration. Just add the option there.

this is also still visible in my change log state: Removed transmit_power from experimental
but this is still visible to me.
Looks like it hasn't been updated then.

@mgiebels if you are using HassOS add-on you should find it under Supervisor > Dashboard > Zigbee2mqtt > Configuration. Just add the option there.

I have copied de config file from my old zigbee2mqtt and add it to the new one but it doesnt work

It is working :D

@dreimer1986 is the update working for you? Can we close this issue?

Update is working for me! you me close it

Oh, you are waiting for me... Sorry. ^^ Update installed, test server option working, no updates for my bulbs though, but that was quite expected. I close it then :D Thanks for fixing!