HADOOP-19197. S3A: Support AWS KMS Encryption Context

Add the property fs.s3a.encryption.context that allow users to specify the AWS KMS Encryption Context to be used in S3A.
The value of the encryption context is a key/value string that will be Base64 encoded and set in the parameter ssekmsEncryptionContext from the S3 client.
Contributed by Raphael Azzolini
Description of PR
This code change adds a new property to S3A: fs.s3a.encryption.context\
The property's value accepts a set of key/value attributes to be set on S3's encryption context. The value of the property will be base64 encoded and set in the parameter ssekmsEncryptionContext from the S3 client.
How was this patch tested?
Tested in us-east-1 with mvn -Dparallel-tests -DtestsThreadCount=16 clean verify
I added a new test ITestS3AEncryptionSSEKMSWithEncryptionContext. However, S3's head-object response doesn't contain the object encryption key. Therefore, I enabled CloudTrails data logs in my bucket to verify that the tests were passing the encryption context to the request.
I added this property to auth-keys.xml
<property>
  <name>fs.s3a.encryption.context</name>
  <value>
    project=hadoop,
    jira=HADOOP-19197,
    component=fs/s3
  </value>
</property>

Then I executed the following tests:
mvn clean verify -Dit.test=ITestS3AEncryption* -Dtest=none

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKeyWithEncryptionContext
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.10 s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKeyWithEncryptionContext
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.17 s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKeyWithEncryptionContext
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.575 s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKeyWithEncryptionContext
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.246 s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 2.600 s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.414 s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSES3
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.680 s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSES3
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionDSSEKMSUserDefinedKeyWithEncryptionContext
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.538 s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionDSSEKMSUserDefinedKeyWithEncryptionContext
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionDSSEKMSUserDefinedKey
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.425 s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionDSSEKMSUserDefinedKey
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 53, Failures: 0, Errors: 0, Skipped: 6

mvn clean verify -Dit.test=TestMarshalledCredentials -Dtest=none

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.s3a.auth.TestMarshalledCredentials
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.133 s -- in org.apache.hadoop.fs.s3a.auth.TestMarshalledCredentials
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

mvn clean verify -Dit.test=ITestS3AHugeFilesEncryption -Dtest=none

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesEncryption
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.04 s -- in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesEncryption
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0

Finally, I verified in the CloudTrail logs, that the encryption context was set for both aws:kms and aws:kms:dsse.
(...)
    {
      "eventTime": "2024-06-08T03:49:49Z",
      "eventSource": "s3.amazonaws.com",
      "eventName": "PutObject",
      "awsRegion": "us-west-1",
      "userAgent": "[Hadoop 3.5.0-SNAPSHOT, aws-sdk-java/2.24.6 Linux/5.10.217-183.860.amzn2int.x86_64 OpenJDK_64-Bit_Server_VM/25.412-b08 Java/1.8.0_412 vendor/Private_Build io/sync http/Apache cfg/retry-mode/adaptive hll/cross-region ft/s3-transfer]",
      "requestParameters": {
        "bucketName": "************",
        "x-amz-server-side-encryption-aws-kms-key-id": "arn:aws:kms:us-west-1:************:key/************",
        "Host": "************.s3.us-west-1.amazonaws.com",
        "x-amz-server-side-encryption": "aws:kms:dsse",
        "x-amz-server-side-encryption-context": "eyJjb21wb25lbnQiOiJmcy9zMyIsInByb2plY3QiOiJoYWRvb3AiLCJqaXJhIjoiSEFET09QLTE5MTk3In0=",
        "key": "test/"
      },
(...)

(...)
      "awsRegion": "us-west-1",
      "sourceIPAddress": "<IP_ADDRESS>",
      "userAgent": "[Hadoop 3.5.0-SNAPSHOT, aws-sdk-java/2.24.6 Linux/5.10.217-183.860.amzn2int.x86_64 OpenJDK_64-Bit_Server_VM/25.412-b08 Java/1.8.0_412 vendor/Private_Build io/sync http/Apache cfg/retry-mode/adaptive hll/cross-region ft/s3-transfer]",
      "requestParameters": {
        "bucketName": "************",
        "x-amz-server-side-encryption-aws-kms-key-id": "arn:aws:kms:us-west-1:************:key/************",
        "Host": "************.s3.us-west-1.amazonaws.com",
        "x-amz-server-side-encryption": "aws:kms",
        "x-amz-server-side-encryption-context": "eyJjb21wb25lbnQiOiJmcy9zMyIsInByb2plY3QiOiJoYWRvb3AiLCJqaXJhIjoiSEFET09QLTE5MTk3In0=",
        "key": "test/testEncryptionOverRename-0400"
      },
(...)

echo eyJjb21wb25lbnQiOiJmcy9zMyIsInByb2plY3QiOiJoYWRvb3AiLCJqaXJhIjoiSEFET09QLTE5MTk3In0= | base64 --decode
{"component":"fs/s3","project":"hadoop","jira":"HADOOP-19197"}%

I also executed the test with the following statement in my KMS key:
{
     "Effect": "Deny",
     "Principal": {
         "AWS": "*"
     },
     "Action": "kms:Decrypt",
     "Resource": "*",
     "Condition": {
         "StringNotEquals": {
             "kms:EncryptionContext:project": "hadoop"
         }
     }
}

When using that statement, tests without encryption context fail, and the new test will pass only if the given key-pair is set in fs.s3a.encryption.context.
For code changes:

[X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[X] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[-] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
11m 50s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 9 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
15m 12s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
32m 36s

trunk passed

+1 :green_heart:
compile
17m 22s

trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
compile
16m  6s

trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
checkstyle
4m 27s

trunk passed

+1 :green_heart:
mvnsite
2m 44s

trunk passed

+1 :green_heart:
javadoc
1m 56s

trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javadoc
1m 45s

trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
spotbugs
3m 54s

trunk passed

+1 :green_heart:
shadedclient
35m 57s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 33s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 26s

the patch passed

+1 :green_heart:
compile
16m 44s

the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javac
16m 44s

the patch passed

+1 :green_heart:
compile
15m 50s

the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
javac
15m 50s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
4m 22s

the patch passed

+1 :green_heart:
mvnsite
2m 39s

the patch passed

+1 :green_heart:
javadoc
1m 50s

the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javadoc
1m 44s

the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
spotbugs
4m 16s

the patch passed

+1 :green_heart:
shadedclient
34m 33s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
20m 50s

hadoop-common in the patch passed.

+1 :green_heart:
unit
3m  5s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
1m  4s

The patch does not generate ASF License warnings.

257m  6s

Subsystem
Report/Notes

Docker
ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6874/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6874

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint

uname
Linux 60270e4a493d 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 71ffcd5849f4f4e99f7117b5c84046e86e23aad4

Default Java
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6874/1/testReport/

Max. process+thread count
3151 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6874/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@virajjasani @steveloughran do you need me to provide any more information to review this PR?

@steveloughran, regarding #6884, I didn't find any risks of conflict with this PR, but I think that the new code in S3AFileSystem.java that you added a comment could be moved to the new class S3AEncryption.java you suggested, and that I am adding to the next commit.
I also added a comment to that PR asking if the new properties need to be added to core-default.xml and index.md.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
12m 21s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 11 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 39s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
32m 36s

trunk passed

+1 :green_heart:
compile
17m 50s

trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
compile
16m 18s

trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
checkstyle
4m 52s

trunk passed

+1 :green_heart:
mvnsite
2m 47s

trunk passed

+1 :green_heart:
javadoc
1m 59s

trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javadoc
1m 46s

trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
spotbugs
3m 58s

trunk passed

+1 :green_heart:
shadedclient
34m 33s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 35s

Maven dependency ordering for patch

-1 :x:
mvninstall
0m 28s
/patch-mvninstall-hadoop-tools_hadoop-aws.txt
hadoop-aws in the patch failed.

-1 :x:
compile
16m 17s
/patch-compile-root-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt
root in the patch failed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.

-1 :x:
javac
16m 17s
/patch-compile-root-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt
root in the patch failed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.

-1 :x:
compile
15m 42s
/patch-compile-root-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt
root in the patch failed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08.

-1 :x:
javac
15m 42s
/patch-compile-root-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt
root in the patch failed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08.

-1 :x:
blanks
0m  0s
/blanks-eol.txt
The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
checkstyle
4m 17s

the patch passed

-1 :x:
mvnsite
0m 52s
/patch-mvnsite-hadoop-tools_hadoop-aws.txt
hadoop-aws in the patch failed.

+1 :green_heart:
javadoc
1m 52s

the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

-1 :x:
javadoc
0m 50s
/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08 with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

-1 :x:
spotbugs
0m 50s
/patch-spotbugs-hadoop-tools_hadoop-aws.txt
hadoop-aws in the patch failed.

+1 :green_heart:
shadedclient
35m 55s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
19m 42s

hadoop-common in the patch passed.

-1 :x:
unit
0m 51s
/patch-unit-hadoop-tools_hadoop-aws.txt
hadoop-aws in the patch failed.

-1 :x:
asflicense
1m  4s
/results-asflicense.txt
The patch generated 1 ASF License warnings.

252m  3s

Subsystem
Report/Notes

Docker
ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6874/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6874

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint

uname
Linux a54a2d23b372 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / c2596c7df724a528bb5ce9b5a5c01935dc99556e

Default Java
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6874/2/testReport/

Max. process+thread count
3152 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6874/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 30s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+0 :ok:
markdownlint
0m  1s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 11 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
15m  2s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
32m 45s

trunk passed

+1 :green_heart:
compile
17m 25s

trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
compile
16m  4s

trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
checkstyle
4m 26s

trunk passed

+1 :green_heart:
mvnsite
2m 41s

trunk passed

+1 :green_heart:
javadoc
1m 57s

trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javadoc
1m 43s

trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
spotbugs
4m  1s

trunk passed

+1 :green_heart:
shadedclient
35m 13s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 33s

Maven dependency ordering for patch

-1 :x:
mvninstall
0m 27s
/patch-mvninstall-hadoop-tools_hadoop-aws.txt
hadoop-aws in the patch failed.

-1 :x:
compile
16m  4s
/patch-compile-root-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt
root in the patch failed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.

-1 :x:
javac
16m  4s
/patch-compile-root-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt
root in the patch failed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.

-1 :x:
compile
15m 20s
/patch-compile-root-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt
root in the patch failed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08.

-1 :x:
javac
15m 20s
/patch-compile-root-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt
root in the patch failed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08.

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
4m 13s

the patch passed

-1 :x:
mvnsite
0m 53s
/patch-mvnsite-hadoop-tools_hadoop-aws.txt
hadoop-aws in the patch failed.

+1 :green_heart:
javadoc
1m 51s

the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javadoc
1m 43s

the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

-1 :x:
spotbugs
0m 50s
/patch-spotbugs-hadoop-tools_hadoop-aws.txt
hadoop-aws in the patch failed.

+1 :green_heart:
shadedclient
36m  3s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
19m 44s

hadoop-common in the patch passed.

-1 :x:
unit
0m 48s
/patch-unit-hadoop-tools_hadoop-aws.txt
hadoop-aws in the patch failed.

+1 :green_heart:
asflicense
1m  3s

The patch does not generate ASF License warnings.

239m 22s

Subsystem
Report/Notes

Docker
ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6874/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6874

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint

uname
Linux fda291b182e1 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 4de0f65dc4efa985359155afd8bc4777751a9bf6

Default Java
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6874/3/testReport/

Max. process+thread count
1251 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6874/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

The checks are failing because they are trying to compile three classes that I removed in the second commit. I don't know how to make it not use them.
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-6874/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionDSSEKMSUserDefinedKeyWithEncryptionContext.java:[48,40] cannot find symbol
[ERROR]   symbol:   method getS3EncryptionContext(java.lang.String,org.apache.hadoop.conf.Configuration)
[ERROR]   location: class org.apache.hadoop.fs.s3a.S3AUtils
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-6874/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEKMSUserDefinedKeyWithEncryptionContext.java:[43,40] cannot find symbol
[ERROR]   symbol:   method getS3EncryptionContext(java.lang.String,org.apache.hadoop.conf.Configuration)
[ERROR]   location: class org.apache.hadoop.fs.s3a.S3AUtils
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-6874/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEKMSDefaultKeyWithEncryptionContext.java:[43,40] cannot find symbol
[ERROR]   symbol:   method getS3EncryptionContext(java.lang.String,org.apache.hadoop.conf.Configuration)
[ERROR]   location: class org.apache.hadoop.fs.s3a.S3AUtils

The checks are failing because they are trying to compile three classes that I removed in the second commit. I don't know how to make it not use them.

needs a squashed commit. let me review the changes first

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
11m 56s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 8 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 51s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
32m 18s

trunk passed

+1 :green_heart:
compile
17m 36s

trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
compile
16m 11s

trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
checkstyle
4m 59s

trunk passed

+1 :green_heart:
mvnsite
2m 43s

trunk passed

+1 :green_heart:
javadoc
1m 56s

trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javadoc
1m 45s

trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
spotbugs
3m 51s

trunk passed

+1 :green_heart:
shadedclient
34m 43s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 35s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 29s

the patch passed

+1 :green_heart:
compile
16m 51s

the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javac
16m 51s

the patch passed

+1 :green_heart:
compile
16m  0s

the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
javac
16m  0s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
4m 23s

the patch passed

+1 :green_heart:
mvnsite
2m 40s

the patch passed

+1 :green_heart:
javadoc
1m 53s

the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javadoc
1m 45s

the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
spotbugs
4m 15s

the patch passed

+1 :green_heart:
shadedclient
34m 45s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
19m 49s

hadoop-common in the patch passed.

+1 :green_heart:
unit
3m  1s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
1m  6s

The patch does not generate ASF License warnings.

255m 41s

Subsystem
Report/Notes

Docker
ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6874/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6874

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint

uname
Linux 983bc8e41580 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 898b883bbead9b619014260e9ee4c476bc18e9f2

Default Java
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6874/4/testReport/

Max. process+thread count
3152 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6874/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.


Change user agent from "Safari/vX.X.X" to "Desktop App/vX.X.X"

I confirm (by marking "x" in the [ ] below):

[X] This is not a feature idea. Feature ideas go here: http://www.mattermost.org/feature-ideas/
[X] This is not a troubleshooting question. Troubleshooting questions go here: http://www.mattermost.org/troubleshoot/
[X] This is a product defect issue that can be reproduced using the steps listed below.
[X] I have read CONTRIBUTING.md.

Summary
Change user agent from "Safari/vX.X.X" to "Desktop App/vX.X.X"
Steps to reproduce

Operating System: Windows 10
Mattermost Desktop App v3.4.1
Mattermost Server v3.4

Expected behavior
Account Settings > Security > View and Logout of Active Sessions shows Desktop App/3.4.0 as the user agent for desktop apps
Observed behavior
Account Settings > Security > View and Logout of Active Sessions shows Safari/3.4.0 as the user agent for desktop apps

We should not change the user agent. The whole of it is like Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Mattermost/1.2.1 Chrome/49.0.2623.75 Electron/0.37.8 Safari/537.36 and it's important for webapp.
So webapp/components/activity_log_modal.jsx of /platform repository should be updated like iPhone Native App instead.

I see, got it. Will report to the platform team @yuya-oc

Closing, will be tracked in Jira by the Mattermost core team: https://mattermost.atlassian.net/browse/PLT-4317

Hi, If I want add more string in the user agent, which files should I inspect?
I tried node_modules/ua-parser-js getUA andsetUA, then rebuilt it, but this did not work

@yuya-oc any thoughts? I'm not sure if this would be handled on the server side

You can use session.defaultSession.getUserAgent() and session.defaultSession.setUserAgent() in src/main.js. http://electron.atom.io/docs/api/session/
Existing elements of user agent should be kept to get some features from webapp. See server's webapp/utils/user_agent.jsx

Yeah, It works.
Thank you 👍😄

On 20 Jan 2017, at 7:37 AM, Yuya Ochiai<EMAIL_ADDRESS>wrote:
You can use session.defaultSession.getUserAgent() and session.defaultSession.setUserAgent() in src/main.js. http://electron.atom.io/docs/api/session/ http://electron.atom.io/docs/api/session/
Existing elements of user agent should be kept to get some features from webapp. See server's webapp/utils/user_agent.jsx https://github.com/mattermost/platform/blob/master/webapp/utils/user_agent.jsx
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/mattermost/desktop/issues/352#issuecomment-273933350, or mute the thread https://github.com/notifications/unsubscribe-auth/AC_S-u86gxYDz3cCmi8PXSinhjNFASA8ks5rT_PJgaJpZM4Kj4hM.


Consider increasing contrast on Learn to Validate page

Just a comment on the Learn to Validate page: https://www.missingmaps.org/validate/
I am wondering if you might consider increasing the contrast between the text and background in the tutorial. The W3 Consortium:
https://www.w3.org/WAI/WCAG21/Understanding/contrast-minimum.html
recommends a contrast ratio of at least 4.5:1, by my measurements, using the methods described here:
https://www.w3.org/WAI/WCAG21/Techniques/general/G18
the tutorial text is only 3.39:1, making it quite difficult to read.
Thank you for your consideration.
-Thayer

this should addressed in #444
let us know if still an issue

Thanks that looks good.
-Thayer
On Thursday, April 29, 2021, 02:37:10 PM EDT, Dan Joseph<EMAIL_ADDRESS>wrote:

this should addressed in #444
let us know if still an issue
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.


Link to current nightly

Fix code that was trying reuse an already installed nightly, by making install call out to rustup toolchain link to link the currently installed nightly up to the new "bisector-nightly-xxx" that we expect.
Fix #54

r? @spastorino

This looks slightly wrong to me. The default toolchain could be nightly-2020-01-04, in which case our synthesis of the link path is wrong i believe. I suspect we should instead run rustc --print sysroot or so and get the path from there.
I was also concerned about removing someone's installed nightly, but it looks like remove_dir_all isn't supposed to follow symlinks (though I'm not sure what behavior is like for hardlinks, I believe rustup link doesn't create those). I guess modulo https://github.com/rust-lang/rust/issues/48504 but hopefully we don't hit that.

The default toolchain could be nightly-2020-01-04, in which case our synthesis of the link path is wrong i believe. I suspect we should instead run rustc --print sysroot or so and get the path from there.

Yeah I just realized this while I was experimenting with this and happened to be making local changes to my rustup's default nightly. Let me see if I can fix it.
(It seems like rustup is missing some functionality here, no? I.e. I could imagine we "need" a subcommand like rustup toolchain print-install-dir <toolchain> or something along those lines...)

Yes, rustup isn't really ready for scripting today, semi-intentionally. Or so I recall hearing from @Kinnison... I think.

(just forced-push a new version that uses rustc --print sysroot to figure out the directory to link to, instead of trying to (incorrectly) infer it.)

I was also concerned about removing someone's installed nightly, but it looks like remove_dir_all isn't supposed to follow symlinks

You know, a way to definitely avoid this risk is to "just" fully copy the directory tree locally, rather than use a symbolic link. And then the resulting directory structure for the bisector variants would be totally uniform, and it would also avoid the risk (when using --preserve) of a user independently removing an installed nightly that we linked to...
(it would also be wasteful of disk space, of course... but that might be a micro-optimization for something like this.)
Its at least something to consider, though I would be happy just landing this PR as is, rather than bikeshedding this question.


Duplication bug with capacitor bank

It is possible to duplicate any item that can be charged within a capacitor bank when it is in a multi block form.
How to replicate the bug:
Place 2 or more capacitor banks in the world to create a multi block structure (any tire)
Put an item in the capacitor bank that can be charged
Brake the first capacitor bank that was placed (it needs to be the first one that was placed since it seems like the data for items is stored in the first capacitor bank in the structure)
Place the capacitor bank somewhere in the world and in both capacitor banks there will be a copy of the original item.
Here is a quick video of the bug: https://youtu.be/eBLAgRCGU64

This is already fixed. Also, do not publicize how to perform dupe bugs.


allow cargo vet to be used as a library

This has been requested. Totally in favor of supporting this, though we may want to wait until there's a specific consumer to kick the tires.

I think we'd be interested in using cargo-vet as a library in cargo-deny as it fits in with the general ethos of that tool, which is automatically verifying dependencies meet a project's basic requirements on various axes.

Do you have any idea what you'd want the API surface-area to look like?

We've been slowly reworking the code in the general direction of this. There is still only a main.rs, but most "applicationy" things are now isolated to it. We expect to split part of main.rs into a new lib.rs. What would remain in main.rs is:

fn main
fn real_main
fn cmd_*
the ci module that has all the clap stuff
some odds and ends related to terminal UI

Things like miette::Report and Console would be banished to main.rs, and I've already narrowed their scope a lot to reflect this. Library things are now largely all using type errors, which still uses miette to add extra hierarchy (list of errors nested under high-level errors) but they're normal Errors you can Display if you want, no need to pull in miette "pretty".
Things like minimize_unaudited would be moved to lib.rs as core operations.
The biggest TODO for this split is to cleanup Config which currently contains a copy of the CI flags, which presumably library users don't want to have to fill in/use.
The other big question is "what do library users want from this"?
The current design is that you run cargo-metadata to get the lockfile state and Store::acquire to load+parse the contents of supply-chain. Then you can do whatever analysis/adjustments you want on the parsed structures, and then do store.commit() to write back the results. All cmd_* functions just do this.
The heart of the analysis is resolver::resolve which takes the store and metadata and does vet check, producing either some happy stats or a list of problems. Either way you will get a ton of datastructures describing exactly how you passed/failed. You can then compute_suggest to get the suggest output. This involves a bunch of extra network/disk traffic as we do a bajillion async diffs and Cache those results for subsequent runs.
Commands like certify literally just push a value into the Store before committing. Commands like inspect tell the Network/Cache to ensure we have access to the source of some packages and does some operation on the returned paths.

I can try to block out some time later this month (most everyone is going on vacation here) to try integrating the vet functionality into cargo-deny as a concrete test case to find pain points or issues.

It's worth calling out that the most loosely-coupled approach for embedding cargo-vet would be to just parse the JSON output (like a CI use-case would). The tool is pretty fast to run even with hundreds of dependencies, and so it's worth at least considering the simple approach.


Feature request: auto select default account on startup

I have two accounts, one of which I want to access 99.9% of the time and the other I want to access 0.1% of the time (haven't measured but that's what it feels like).  It would be nice if I didn't have to go through the account picker when starting K-9 Mail.  A "go straight to default account" option would be good.  Then I can go back up through "Folders" to "Accounts" in the rare occasions when I want to check the other account.
Thanks so much.

You can add a widget that does this. If the existing widgets aren't suitable could you clarify why not. I think a widget is the probably right way to address this but I may be missing something.

Thanks.  The "K-9 Unread" widget (I've only found this one) can indeed be set to choose an account by default, and it also obeys that account's "Auto-expand folder" setting (at least it does when multiple accounts are present; for some reason it didn't when I tried it with only one account in the database).
The problem with widgets though is they take more taps to reach.  I use an app called "Simple Home" for big fonts on the home screen (I'm sight impaired), and Simple Home doesn't "do" widgets, so I have to install the widget as a "small app" and reach it via Android 4.4's app list (and then it floats on top of K9 and I have to explicitly close it once the folder is on-screen).  It's not half as convenient as being able to go straight to my chosen folder in my chosen account when K9 is launched from Simple Home.  (And I'd really rather not change back to the phone's default "Xperia™ Home" instead of Simple Home.)


Abort Copy Blob operation.

Currently the library supports the starting of a copy blob operation (CopyBlob function) which starts a single copy and then waits.
I'd like to propose a new function for Aborting such a copy (see https://msdn.microsoft.com/en-us/library/azure/jj159098.aspx) as well as modifying the existing code a tiny bit.
Currently the exported function CopyBlob calls a non-exported (still trying to get my Go jargon right) startBlobCopy, then once the copying has started it loops checking the status.

Add AbortBlobCopy which takes the copyID generated by startBlobCopy and can abort the operation if needed.

No functionality (CopyBlob function) is impacted but gives the users 2 methods (StartBlobCopy and AbortBlobCopy) which are closer (and for me personally more useful) to the REST API.
I've already coded up a PR with the above, but wanted to get opinions before I actually submit it.
Thoughts?

@kpfaulkner sounds good. Folks can help you review if you submit a patch (with tests).

Closing. PR merged


feat: throttle source throughput to limit streaming latency

Is your feature request related to a problem? Please describe.
As described in #6571, the barrier latency i.e. streaming end-to-end latency can increase infinitely when the system is under heavy load.
However, streaming latency is a critical metric for streaming computing, and it must be upper-bounded according to our SLA.
We need to build some negative feedback mechanisms to bound the latency.
Describe the solution you'd like
Currently, we allow users to set the max number of in-flight barriers, but it only controls the injection of barriers. We can enhance it by throttling the source events as well.

If the number of in-flight barriers reaches the upper bound i.e. 40, we may not only stop injecting barriers but also block the source executor to generate any new events. What do you guys think?
Referenced from https://github.com/risingwavelabs/risingwave/issues/6571#issuecomment-1331582075

One possible way to block new DataChunk events from being generated is using Pause/Resume barrier. Any other simpler solutions are welcomed.
Describe alternatives you've considered
None
Additional context
None

@xxhZs Please take a look 🥺  cc. @hzxa21 @BugenZhao

One possible way to block new DataChunk events from being generated is using Pause/Resume barrier. Any other simpler solutions are welcomed.

Do you mean throttle the throughput by the meta service? That sounds like another reasonable idea.
FYI, @yezizp2012 added the mechanism on the compute node to pause the source in case of no barrier due to meta-service failure in https://github.com/risingwavelabs/risingwave/pull/6280. Simply making this more aggressive may also help.

Do you mean throttle the throughput by the meta service?

Yes, when the number of in-flight barriers hits the limit.

Originally I think it's quite dangerous to throttle the throughput using Pause/Resume by the meta service, what if meta exits after sending Pause barrier. But since meta will trigger a recovery process after restart, it's acceptable.

Simply making this more aggressive may also help.
+1 for this.

I think it's quite dangerous to throttle the throughput using Pause/Resume by the meta service

Issuing Pause by the meta service LGTM, while it's indeed dangerous by compute node itself. 🤤

close due to realizing adjusting inflight barrier num is futile to control barrier latency, ref #6943 .
We adjust permits as a temporary fix

Follow up: https://github.com/risingwavelabs/risingwave/pull/6943


Fix US/CO.

Resolves https://github.com/covidatlas/coronadatascraper/issues/928
yarn start --location US/CO works for today.  data.json extract:
{
    "state": "Colorado",
    "country": "United States",
    "aggregate": "county",
    "url": "https://opendata.arcgis.com/datasets/7dd9bfa5607f4c70b2a7c9634ccdca53_0.csv",
    "cases": 15284,
    "deaths": 777,
    "hospitalized": 2697,
    "tested": 72390,
    "rating": 0.49019607843137253,
    "coordinates": [
      -105.551,
      38.9975
    ],
    "tz": [
      "America/Denver"
    ],
    "featureId": "iso2:US-CO",
    "population": 5758736,
    "populationDensity": 21.330560753945722,
    "countryId": "iso1:US",
    "stateId": "iso2:US-CO",
    "name": "Colorado, United States",
    "level": "state"
  }

    "cases": 15284,
    "deaths": 777,

Match the county numbers rolled up, and hosp/tested/pop at the state level are read from the state record.
Added some assertions to ensure that the format is actually valid. :-P

Does this fix #930 too?

#930

I doubt it, I changed the URL.  If we don't have that in the cache, we're out of luck.  I'll try running one of those dates.

Ah hang on, that URL in the cache is still there, at least for 2020-04-25.  I'll see if I can sort that out too.

I added a "fix" that at least parses the cached data for those days ... but the cache is stale (flat numbers).  But it's not zero for Apr 23-27.

I actually left the console log in on purpose, b/c the data was whack.  :-)  But we can remove it, it won't mean anything.  Re the "|| 0", I'm not sure!


Add erc20 balance support

Descriptions should be in the format "erc20:address" to automatically update the balance.

Apologies for the delay getting to this PR.
Overall looks great - thanks for adding the feature! Just a few comments and if you could squash to clean up I'll get this merged asap!

Squash and I'll merge this

@sclem you should have the option to squash when you merge the pull request: https://help.github.com/articles/about-pull-request-merges/
If you don't see the option:
https://help.github.com/articles/configuring-commit-squashing-for-pull-requests/


onSlideChange is fired twice on state change with initial index

I want to call goToSlide using a ref to the AppIntroSlider and I want to save the current index in a state. Consider the following example.
const [index, setIndex] = useState(0)
const ref = useRef(null)

function onSlide(idx) {
    setIndex(idx)
}

function MyCustomButton(props) {
    return <Button title="Next" onPress={() => ref.current.gotToSlide(index + 1, true)} />
}

return (
    <AppIntroSlider<ScreenItem>
           ref={ref}
           onSlideChange={onSlide}
           renderNextButton={() => <MyCustomButton />}
           renderItem={(...) => ...}
)

If I press MyCustomButton, then onSlide is called with the correct index 1. The setIndex causes a rerender and it seems that the onSlideChange function is called again with the old initial index 0.
Is there a way to make this setup work or is this behavior by design?
Version: 4.0.4

I am facing the exact Issue... Anyone able to solve this?

You do not need to create a custom index state.
Follow this example: https://github.com/Jacse/react-native-app-intro-slider/issues/224#issuecomment-1367099899


Wrong result if return type is Task

It appears that if we return false from backend it would appear like we errored and null will be printed instead of false
IChatHub
[Hub]
public interface IChatHub : IBaseHub
{
	public Task<bool> Test();

	public Task<bool> Test2();

	public Task<bool> Test3();

	public Task<bool> Test4();
}

ChatHub
[SignalRHub]
public class ChatHub : Hub<IChatReceiver>, IChatHub
{
	public async Task<bool> Hello()
	{
		throw new NotImplementedException();
	}
	public async Task<bool> Test()
	{
		return true;
	}
	public async Task<bool> Test2()
	{
		return false;
	}
	public Task<bool> Test3()
	{
		return Task.FromResult(true);
	}
	public Task<bool> Test4()
	{
		return Task.FromResult(false);
	}
}

Invoke method to change
https://github.com/nenoNaninu/TypedSignalR.Client.DevTools/blob/main/src/signalr-dev-ui/components/HubMethodInvoker.tsx#L31

Thank you for the report! I fixed this issue and released v1.2.4. Please try it!

Works now, thanks for resolving the issue.


About the evaluation on FedML library

Dear authors.
I am Chaoyang He (https://chaoyanghe.com/), co-founder of FedML Inc. Thanks for proposing such a benchmark to compare with different frameworks. I like your summarization, but It seems many of your comments to FedML are not based on a proper feature or new versions. As your paper reader, I am confused by the version you compared with for different libraries. So it's better to mention the commit/version ID with date for each library. People then know how long the evaluation is outdated.
Please help to upgrade our library to latest version and do a fair comparison.
FedML supports:
FedML Parrot - Simulating federated learning in the real world: (1) simulate FL using a single process (2) MPI-based FL Simulator (3) Sequential Distributed Training (running arbitrary number of clients on arbitrary number of GPUS)
FedML Octopus - Cross-silo Federated Learning for cross-organization/account training, including Python-based edge SDK.
FedML Beehive - Cross-device Federated Learning for Smartphones and IoTs, including edge SDK for Android/iOS and embedded Linux.
FedML MLOps: FedML’s machine learning operation pipeline for AI running anywhere and at any scale.
You can find an overview of new FedML at: https://medium.com/@FedML/fedml-ai-platform-releases-the-worlds-federated-learning-open-platform-on-public-cloud-with-an-8024e68a70b6
FedML Homepage: https://fedml.ai/
FedML Open Source: https://github.com/FedML-AI
FedML Platform: https://open.fedml.ai/ (you mentioned our library does not support deployment... this is not true)
FedML Use Cases: https://open.fedml.ai/platform/appStore
FedML Documentation: https://doc.fedml.ai/ (you mentioned our library does not support doc... this is not true)
FedML Blog: https://medium.com/@FedML
FedML Research: https://fedml.ai/research-papers/
In addition, I have some questions for the selection web.

As some libraries iterate quickly (e.g., we release new versions weekly), how can you upgrade the version and guarantee the selection is in a timely manner?

it would be better to provide multiple answers for each selection since many libraries support the same feature.

how to help users find libraries that support multiple features? Users normally needs not just one feature, especially the case that first doing POC (simulation) and then migrate to deployment without code change for deployment (we support this).

This is important to business development, otherwise it may raise issues for startup like us (e.g., narrowing down our business scope due to not upgrading timely, misguide users to libraries that may not be the most helpful to users since they may need multiple features...)

We are happy to answer any questions you may meet, and good luck to your paper submission.

Hi Chaoyang, thank you for contacting us and for your great work, in bringing the FedML framework to the FL community!
Our evaluation is based on FedML commit 2ee0517a7fa9ec7d6a5521fbae3e17011683eecd (as part of the revision, we’re providing commit version information in our latest version of the paper). While the paper targets to provide a fair snapshot view comparison across different FL frameworks, we are looking forward to collaborating with maintainers of FedML and other frameworks to update the benchmark result continuously. We believe such an approach is similar to other existing ML benchmarks that provide evaluation results based on versioning.
We will update the FedML version as suggested, and continue to periodically check the latest and representative FL frameworks and include them in the UniFed. As relevant evidence, we’ve already been including more results from two new frameworks in our revision. We believe our paper will help standardize the evaluation of FL frameworks, drawing attention from more and more real-world FL practitioners. With foreseeable direct collaborations with maintainers from FedTree, FedML, and other frameworks on building the standard together, we believe it is an important step towards the long-term healthy development of the FL community.
For your questions specifically:

The suggestion is based on a certain version of the FL framework. While we plan to update it periodically on our end, if there is a specific entry (for features/performance numbers) that is severely misaligned with the latest version, we're more than willing to start a discussion with your team to test the latest version and update our benchmark result accordingly.
The purpose of the tool is to find a single final result that matches the need the most, and all selections are backed up by the result in the paper. We do note that (1) the selection is based on the off-the-shelf framework in a specific version, and (2) the result is suggestive and is not suitable for scenarios when the user is considering customizations. Additionally, we refer to https://unifedbenchmark.github.io/leaderboard/index.html#scenario_1_heading for users who want to get a list of candidates that contains certain features, and https://unifedbenchmark.github.io/leaderboard/index.html#scenario_9_heading for more details on our decision procedure.
The seamless migration from development to deployment is certainly a very important usability feature and it is a very interesting problem to have a clear definition of this attribute. Thank you for providing the idea and would you like to start a conversation on your understanding of this matter?

Again, thank you for contacting us and we are looking forward to working together to reflect the latest development of FedML in our benchmark.

@camelop thanks for your reply. Your vision is big and good, but I feel the current writing does not match your vision accurately. For FedML, we do suggest you to upgrade your evaluation ASAP. The commit ID you provided is super outdated. I am open to set more meetings with you guys. You mentioned that one of your goals is to guide practioners in FL, if so, you need to involve more evaluation metrics in industry level, including user experience, API simplification and flexibility, rich of app ecosystem, system performance, user statistics (user number is always the best voting for product strengths), and many more.  Here is a blog for you to have a taste of what I mean: https://medium.com/@FedML/fedml-ai-platform-releases-the-worlds-federated-learning-open-platform-on-public-cloud-with-an-8024e68a70b6.
I wish you continue to polish this work to match your vision and goal. Current version is still too research-oriented with simple metrics that may not be relevant to landing FL into reality.


Dev

7.0.0

Coverage remained the same at 100.0% when pulling 5650c8341052b9de4814749e606c8d464d8fe1da on dev into 8b4a9055e001bfcd39ecad55e95d4c73ac7330b1 on master.


Add support for other OpenAI models like gpt-4o-2024-08-06

Is your feature request related to a problem? Please describe.
Missing some models from OpenAI like gpt-4o-2024-08-06
Describe the solution you'd like
Add this to the scrapegraphai/helpers/models_tokens.py file
Describe alternatives you've considered
N/A
Additional context
I can raise a PR for this if you're okay with it @VinciGit00 (if this isn't already in the works)

hi, we added in the pre/beta. Please update to the new version


Delete class from dataset

Hi,
First of all, thank you for this amazing dataset management framework!
I was wondering how I could delete a class from my dataset. I have two empty classes in my current dataset since it belongs to a CVAT project that represents the final dataset containing all classes of my interest. Keeping these empty classes in my dataset does not work well with training a model.
Kind regards,
Thomas

Great, thank you!


Undefined media query mixins in component SCSS

Describe the bug
The media(">=notebook") and media(">=tablet") mixins are defined in @faststore/ui/src/styles/base/utilities; when we try to use these in the components, they are undefined.
Expected behavior
A component CSS file that uses a media query should  mixin defined in a separate file should include the code for that media query at the top of the component file (potentially with an @import or @use).
Current behavior
Component CSS files are currently missing a method to include/import for these mixins, resulting in them being undefined.
Possible Solutions

Add @import "./../../../styles/base"; at the top of the component file – this works because the mixin is defined in base/utilities, and is then imported into the base file.
Simplify media queries code pattern 🤷♀

Note: This was originally part of https://github.com/vtex/faststore/issues/1904 -- I created this issue because the solution for media query mixins is going to be different than the other mixins.

Thanks for the explanation, Whitney! We're going to propose an easier way to customize the breakpoints values and add it on the documentation soon ☺

Hey Whitney, we have some updates related to this issue ✨
We created a new PR proposing an update on the @faststore/ui to add the possibility of override our breakpoints and other @mixin along with a documentation page with all the steps needed. Feel free to test it and see if it makes sense to your project.
The PR link: #1917


Fix JS parsing perf issue (#7310)

Simple change. Description of issue/cause is in #7310.
Tests are failing on my box for unrelated issues, so I haven't run the full suite yet. Will verify on my other machine later tonight (if any failures on the CI).

Ping @mhegazy , as we were looking at this earlier.

Tests should be fixed too with the commit just pushed ( @RyanCavanaugh is right, those JSDocParser tests are a pain :disappointed: ).

LGTM. thanks!

:+1:


Issue templates

I decided to try my hand at GitHub's issue template functionality. The changes can be previewed at my fork, where I temporarily enabled issues. This is probably a better way to review this PR than by reading the YAML files. The top three categories have forms that you can view by clicking "get started".
I had two main goals:

Steer people away from reporting security vulnerabilities through public issue tickets. Those should be communicated privately to me and Jeremy instead.
Guide inexperienced users to document their issues as clearly as possible.

For expert users, there is still the option to skip the forms and create a blank issue.
I may have gone a bit overboard with the forms. I'm open to all suggestions.

Bump: I would prefer to see opinions from at least two different people (excluding myself), before taking a decision. To anyone considering to do a review: you can skip the code, the preview link in the opening post will show you what it will actually look like in the GitHub web interface.
CC @paulfalgout @RayRaz @GammaGames

I figured out how to preview them, they look good! The bug one might be a little long though, entering all that information would take a while. I left a few comments where I thought they could be combined without missing anything

The bug form is quite long. I suppose things like logs, research, prior discussions, related tickets, etc. could all be wrapped into simply a remarks field?
Steps to reproduce and example code could possibly be merged into one field as well.

I figured out how to preview them, they look good! The bug one might be a little long though, entering all that information would take a while. I left a few comments where I thought they could be combined without missing anything

@GammaGames where can I see your comments?

@GammaGames Ah, it says "pending" in that screenshot, which means they are probably only visible to you. GitHub queues your comments and then publishes them all at once when you submit your "overall" review (i.e., the "verdict"). You can make that final step by going to the "Files changed" tab and pressing the green button in the top right.

@jgonggrijp Thank you, posted! Guess I've accidentally done it correctly the few times I've posted a review

I can only choose between single-line <input> and multi-line <textarea>. In the latter case, it is the same size as all other "large" boxes. Which of these options would you prefer?

[ ] Shorten the placeholder so it fits.
[ ] Turn the behavior fields into <textarea>s.

By they way, do you think that the code size of the debugInfo function is proportional to its usefulness?

@jgonggrijp honestly I'd move that debugInfo into a separate directory/module such that people can
import Backbone from 'backbone';
import { debugInfo } from 'backbone/debug-info';

I like that they're <input>s, I think a shorter example would fit the space and help guide the user. And yes, the added code for debugInfo has a lot of value! Gathering all the versions as well as environment would be tedous without it

Alright, I shortened the placeholder for that <input>, moved debugInfo to a separate module and added documentation for it.
The updated bug report template can be previewed here and the documentation for debugInfo can be previewed here. I would appreciate a final thumbs-up before merging this!
(The linter currently chokes on the ESM syntax in modules/debug-info.js, I'll adress that before merging.)

👍


Add filtering

Add a system to allow the browse recipe list to be filtered. The following filter options should be available:

country
difficulty
total time

More options may be added in the future

This issue has presented major development issues will little user benefit, therefore, it is being scrapped.


I'm getting TypeError: Cannot set property 'next' of undefined

on the following input..
st=>start: Create New Multisite Site
e=>end
op1=>operation: Check if URL has activation ID
sub1=>subroutine: Create Activation ID
sub2=>subroutine: Create User
sub3=>subroutine: Create Domain
sub4=>subroutine: Create Subscription
sub5=>subroutine: Assign Referal
cond=>condition: Yes or No?:
cond2=>condition: User Exists?:
io=>inputoutput: Save activation_id in plugin
io2=inputoutput: Get User
st->op1->cond
cond(yes)->io->e
cond(no)->sub1(right)->cond2
cond2(no)->sub2(right)->io2
cond2(yes)->io2->sub3->sub4->sub5
particularly sticking io2 in the last line breaks things
any idea why?

Same error for this diagram:
st=>start: Start
e=>end: End
recruitment=>operation: Recruitment
eligibility=>operation: Eligibility
consent=>operation: Consent
randomization=>condition: Randomization
intervention=>inputoutput: Intervention
measurement1=>subroutine: Measurement
measurement2=>subroutine: Measurement2

st->recruitment->eligibility->consent-> randomization
randomization(experimental)->intervention->measurement->e
randomization(control)->measurement->e

cond2(no)->sub2(right)->io2 no need for (right), by the way, there's a missing">" afterio2=
st=>start: Create New Multisite Site
e=>end

op1=>operation: Check

sub1=>subroutine: Create Activation ID
sub2=>subroutine: Create User
sub3=>subroutine: Create Domain
sub4=>subroutine: Create Subscription
sub5=>subroutine: Assign Referal

cond=>condition: Yes or No?
cond2=>condition: User Exists?

io=>inputoutput: Save activation_id in plugin
io2=>inputoutput: Get User

st->op1->cond
cond(yes)->io->e
cond(no)->sub1->cond2
cond2(no)->sub2(right)->io2
cond2(yes)->io2->sub3->sub4->sub5


docker-compose support postgresql

@Bram1903
could you please help me improve the docker-compose deployment configuration file for this project? I need it to support both MSSQL and PostgreSQL databases, but I'm not very familiar with Docker. It would be great if you could take a look when you have some time. Thanks!

I'm quite new to Docker Composer as well, but I'll try and figure it out. Shouldn't be that hard. On the other hand I'm uncertain how you would support multiple databases within the same Docker Composer file, as it will probably just create an additional database, but maybe there is like a choose option. I'll see!

you just need to add postgresql database configuration block in docker-compose.yml file. If it's not needed, you can simply comment it out.

Alright, that should be pretty straightforward! I don't have much time this weekend, but I'll be doing it Monday!

Alright, that should be pretty straightforward! I don't have much time this weekend, but I'll be doing it Monday!

Thank you for letting me know. Take your time
Thanks again for your help.

I have been trying to first get the docker composer project to work in the first place, but it's not working in the first place. Besides that, the composer project is using OS-specific directories, which will result in it only working on Windows. When changing everything to make it at least run, I get some error, which I don't understand.


Initial PR

Glues metrics together, so we can have more than one exporter per Pod.

@rebuy-de/exporter-merger (@rebuy-de/it-platform) What do you think?


Region is hard coded in all samples files

The exported value of AWS_REGION is not used in the different samples but hard coded to 'us-east-1' making all the samples fail if models and index are not deployed in this region.

Thanks for bringing it to notice. This is now fixed.


docs: update description

posting a benchmark w/ suggested update to description:
23:17 $ ./perf/ctxswitch 45
performed 45M context switches in 0.924000 seconds
duration of one context switch: 20 ns
context switches per second: 50.000000M

✔  shark@rc /home/shark/libmill  [master|✔ ]
23:29 $ ./perf/chr 20
received 20M messages in 1.004000 seconds
duration of receiving a single message: 50 ns
message received per second: 20.000000M

oh maybe chr benchmark is different, but the go results i'm seeing are similar:
✔  shark@rc /home/shark/libmill  [master|✔ ]
00:19 $ perf/go 20
executed 20M coroutines in 1.018000 seconds
duration of one coroutine creation+termination: 50 ns
coroutine creations+terminatios per second: 20.000000M

Btw, what's the platform you are testing on?

it's a 6-core with like 64 gigs of DDR4 ram:

ubuntu linux kernel: 3.13.0-32-generic
arch: 64
cpu: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz


feature/BRIDGE-941

THIS PR IS DEPENDENT ON https://github.com/Sage-Bionetworks/AppCore/pull/97 and https://github.com/Sage-Bionetworks/ResearchKit/pull/13 THESE MUST BE MERGED FIRST.
Enabled forgot passcode feature in ResearchKit.

Needs to be updated to match new delegate methods. (sorry)

Updated.

Closing. The APHOnboardingManager only adds a passcode view controller with itself as the delegate for the case where it is adding a passcode and not where it is authenticating an existing passcode.

Ah, good point.  I was confused why there were two passcode delegates in the mPower project.

Yeah, it's confusing. (sorry) Reverse-compatibility. There was too much work and I didn't have time to move entirely away from the view controllers in AppCore which are not ORKStepViewControllers, and some of which do not render well when inheriting from that class with an ORKTaskViewController as the parent. My compromise was to have the APCAppDelegate implement both the APC passcode delegate and the ORK passcode delegate for the authentication but to keep the creation/edit implementation in the onboarding manager.


Update ContentRange.cs due to large 2gb+ file transfers causing exception

When using client side encryption for large files transfers you get an exception when downloading a large file from blob storage. Specifically file size greater then max int32 value<PHONE_NUMBER>.
The exception occurs during first stream access. Since long values are expected the string parsing should have been done with long.Parse(). Once changed the issue goes away.
Exception Message : System.OverflowException: Value was either too large or too small for an Int32.
at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info)
at Azure.Storage.Blobs.Models.ContentRange.Parse(String headerValue)
at Azure.Storage.Blobs.BlobClientSideDecryptor.d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
All SDK Contribution checklist:
This checklist is used to make sure that common guidelines for a pull request are followed.

[x] Please open PR in Draft mode if it is:

Work in progress or not intended to be merged.
Encountering multiple pipeline failures and working on fixes.

[x] If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
[x] I have read the contribution guidelines.
[x] The pull request does not introduce breaking changes.

General Guidelines and Best Practices

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[x] Pull request includes test coverage for the included changes.

SDK Generation Guidelines

[x] The generate.cmd file for the SDK has been updated with the version of AutoRest, as well as the commitid of your swagger spec or link to the swagger spec, used to generate the code. (Track 2 only)
[x] The *.csproj and AssemblyInfo.cs files have been updated with the new version of the SDK. Please double check nuget.org current release version.

Additional management plane SDK specific contribution checklist:
Note: Only applies to Microsoft.Azure.Management.[RP] or Azure.ResourceManager.[RP]

[x] Include updated management metadata.
[x] Update AzureRP.props to add/remove version info to maintain up to date API versions.

Management plane SDK Troubleshooting

If this is very first SDK for a services and you are adding new service folders directly under /SDK, please add new service label and/or contact assigned reviewer.

If the check fails at the Verify Code Generation step, please ensure:

Do not modify any code in generated folders.
Do not selectively include/remove generated files in the PR.
Do use generate.ps1/cmd to generate this PR instead of calling autorest directly.
Please pay attention to the @microsoft.csharp version output after running generate.ps1. If it is lower than current released version (2.3.82), please run it again as it should pull down the latest version.

Note: We have recently updated the PSH module called by generate.ps1 to emit additional data. This would help reduce/eliminate the Code Verification check error. Please run following command:
  `dotnet msbuild eng/mgmt.proj /t:Util /p:UtilityName=InstallPsModules`

Old outstanding PR cleanup
Please note:
If PRs (including draft) has been out for more than 60 days and there are no responses from our query or followups, they will be closed to maintain a concise list for our reviewers.

@csolonowicz Thanks for the contribution! Could there be some test coverage for this?

@csolonowicz Thanks for the contribution! Could there be some test coverage for this?

I had a problem getting tests working correctly due to issue with dependencies. I have resolved the issue and validated the test is working properly. Let me know if there is anything else you need.

Let me do one more test and then we can look to merge.

Faster approval and merge of this PR would be appreciated. We have recently upgraded to Storage V12 and our production workloads are impacted. We need to fix this in production asap.


Clarify docs regarding configuration of managed PE from ADF

It seems that the reference below "Data Factory UI" should be "Purview UI"
Such batch managed private endpoint creation is provided on the Data Factory UI only. If you want to create the managed private endpoints programmatically, you need to create those PEs individually. You can find Purview managed resources' information from Azure portal -> your Purview account -> Managed resources.
There is no reference in this doc relative to adding the PE in ADF. https://docs.microsoft.com/en-us/azure/data-factory/connect-data-factory-to-azure-purview#:~:text= Connect Data Factory to Azure Purview (Preview),to a... 4 Next steps.  More

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 27726e55-68e6-ef1d-4cf7-448624687562
Version Independent ID: 0738a523-ba9b-885f-5916-93f178eb7e06
Content: Access a secured Azure Purview account - Azure Data Factory
Content Source: articles/data-factory/how-to-access-secured-purview-account.md
Service: data-factory
GitHub Login: @linda33wj
Microsoft Alias: jingwang

@tifischer Thank you for your feedback, our team will further review and update the document accordingly.

@linda33wj Could you please validate customer feedback and confirm if it needs to be updated? on your confirmation, I can have PR for same. Thank you.


Lazy loaded relations inside whenLoaded()

Laravel Version: 9.21.6
PHP Version: 8.1.7
Database Driver & Version: SQLite 3.36.0 / MariaDB 10.3.12

Description:
Hello. When i use $this->whenLoaded('relation', []) in my resource file, the checked relation is lazy loaded. This is not a expected behavior, i think. When the relation is not present yet, the code inside the brackets should not be visible in the resource output and the relation shouldn't be loaded.
I got this simple relation in App\Models\User
public function relations()
 {
     return $this->hasMany(Relation::class);
 }

Call to the resource
Route::get('/', function () {
    $user = User::first();

    return new App\Http\Resources\User($user);
});

The resource App\Http\Resource\User
public function toArray($request)
{
    return [
        'name' => $this->name,
        'email' => $this->email,
        'is_loaded_first' => $this->relationLoaded('relations'),
        'relations' => $this->whenLoaded('relations', [
            'objects' => Relation::collection($this->relations)
        ]),
        'is_loaded_second' => $this->relationLoaded('relations')
    ];
}

Expected output:
{
    "data": {
        "name": "Test User",
        "email"<EMAIL_ADDRESS>        "is_loaded_first": false,
        "is_loaded_second": false
    }
}

Given output:
{
    "data": {
        "name": "Test User",
        "email"<EMAIL_ADDRESS>        "is_loaded_first": false,
        "relations": {
            "objects": [
            {
                "name": "Darren Collins"
            }
        ]},
        "is_loaded_second": true
    }
}

Thank you, for your time.

You may be able to return the relation as closure that returns plain collection.
$this->whenLoaded('relation', fn () => Relation::collection($this->relation))

Thank you @Rizky92 that helps.
For clarification: When to output a simple collection, its better to use it like that
use App\Http\Resources\Relation;

return [
    'relations' => Relation::collection($this->whenLoaded('relations'))
];

Thats, what the documentation says and it works.
In my case, i tried to sum up numbers from the relation within the whenLoaded() method
return [
    'relationsNumbersSum'  => $this->whenLoaded('relations', $this->relations->sum('numbers'))
];

This does not work, resp. the relationship got lazy loaded. That was the point, where i tested the code in my initial post, not to realize, that i used it not how it was designed.
Anyway, with a closure like you suggested, it works fine.
return [
    'relationsNumbersSum'  => $this->whenLoaded('relations', fn () => $this->relations->sum('numbers'))
];

Thanks again!

@radmen sincerely I don't really understand all this, sorry. Done some changes but this doesn't work anymore when pushing the edited purchase request.


Linter

What changes are proposed in this pull request?
Add prospector for style checking, etc.

Codecov Report

Merging #42 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #42   +/-   ##
=======================================
  Coverage   85.06%   85.06%
=======================================
  Files          19       19
  Lines         991      991
  Branches        5        5
=======================================
  Hits          843      843
  Misses        148      148

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a5a6e07...f1687a0. Read the comment docs.

@phi-dbq thanks! Can you please document in the README (for example in running teh unit tests) that you need prospector installed with a link to the page of that tool?
Also, I am surprised it has not found anything... Are there some good rules we have not turned on?


Error: 502 Bad Gateway

I get error: 502 Bad Gateway when I visite the url of new tenant just after creation.
The webserver command is successfully processed with a queue running as root.
I have to do:
Vagrant reload
For fix the issue

Yes vagrant works differently with the webserver because it configures webserver related stuff during provisioning. This is not something we will be supporting.

What is the proper supported way to setup a development environment with this package and achieve at least two different Hello World homepage?

You can use vagrant, but you'd have to apply changes to your vagrant machines when you add new hosts using vagrant reload. Please understand the webserver changes made by this package are reset if you provision the virtual machine anew.


Improved bulma

This is a pretty trivial improvement of the bulma style.  Basically there is a class that styles a row of buttons -- we should use it.

Gah I just noticed I based this on the wrong master... I'll close this and re-created it on the right one.


The code works without GPU?

Hello @adrianyoung, I'm trying run the code on my laptop, and it have no dedicated GPU, so I want would like know this information. Thank you for your time.

Hi, I have not tried running the code with CPU, it may be pretty slow I think. And I suggest GPU for deep learning environment.


Newsletter 14

Feel free to suggest sections!
If you want to help writing the newsletter:

Grab one of the "🆓 free" sections by leaving a comment like "I'd like to work on {section_name_1} and {section_name_2}".
Mentions in brackets (like "🆓 free (@ozkriff?)") are just an invitation to write the corresponding section if you want, but anyone is free to take it.
Write a short overview:

Use provided links as a starting point.
Try to maintain a consistent style with the rest of the newsletter and run markdownlint, please.
Insert line breaks (softbreak) at 80 chars, please.
Use third-person pronouns even if you write about your project updates.
DM me on Discord or Telegram if there're any questions.

Send a PR to the source branch (example: #75)

Mention this issue in your PR's description to link it all together.

I'm hoping to release this newsletter on 2020.10.06, so the soft deadline for all section PRs is 2020.10.04.
Current structure/status (I'll try to keep this updated):

[ ] Game Updates

[ ] Veloren (1) - 🚧 WIP by @AngelOnFira
[ ] Crate Before Attack (1) - 🚧 WIP by @koalefant
[ ] A/B Street (1, 2) - 🆓 free (@dabreegster?)
[ ] galangua (1, 2) - 🆓 free (@tyfkda?)
[ ] Recall Singularity (1, 2) - 🆓 free (@tom-leys?)
[ ] mimas (1, 2) - 🆓 free (@est31?)
[ ] Zemeroth (1) - 🚧 WIP by @ozkriff
[ ] Akigi (1, 2) - 🆓 free (@chinedufn?)
[ ] BUGOUT (1, 2) - 🚧 WIP by @Terkwood
[ ] Tetris Bane (1, 2) - 🆓 free (@andii1701)
[ ] Bevy Games (1, 2, 3, more?) - 🆓 free (@cart?)

[ ] Learning Material Updates

[ ] Threading WGPU Resource Loading with Rayon (1, 2, 2) - 🆓 free (@sotrh?)
[ ] So you want to live-reload Rust (1) - 🆓 free (@fasterthanlime?)
[ ] Is it easy to draw a line (1) - 🚧 WIP by @pum-purum-pum-pum
[ ] An OpenGL preprocessor for Rust (1) -  🚧 WIP by @krupitskas
[ ] Flappy Bird clone made with BEVY (1) - 🆓 free (@TanTanDev?)
[ ] Rust, gamedev, ECS, and bevy (1, 2) - 🆓 free (@hugopeixoto?)

[ ] Library & Tooling Updates

[ ] legion 0.3 (1) - 🆓 free (@TomGillen?)
[ ] thunderdome (1) - 🚧 WIP by @LPGhatguy
[ ] fontdue (1) - 🆓 free (@mooman219?)
[ ] ultraviolet v0.6 (1) - 🆓 free (@termhn?)
[ ] Mun - 🚧 WIP by @Wodann
[ ] rust-wasm-hotreload (1, 2) - 🆓 free (@shekohex?)
[ ] crevice (1, 2) - 🚧 WIP by @LPGhatguy
[ ] femtovg (1, 2) - 🚧 WIP by @adamnemecek
[ ] gfx/wgpu (1, 2, 3) - 🚧 WIP by @kvark
[ ] Riddle (1, 2) - 🆓 free (@vickles?)
[ ] miniquad/macroquad (1, 2) - 🚧 WIP by @not-fl3
[ ] tetra 0.5 (1, 2) - 🚧 WIP by @17cupsofcoffee
[ ] Bevy (1, 2) - 🚧 WIP by @cart
[ ] rg3d - 🚧 WIP by @mrDIMAS
[ ] godot-rust v0.9 (1) - 🆓 free (@extrawurst?)

[ ] Bonus - 🚧 WIP by @ozkriff

Final steps:

[ ] final review - by everyone
[ ] publish - @ozkriff
[ ] post on /r/rust, /r/rust_gamedv, /r/gamedev, URLO, twitter.com/rust_gamedev - @ozkriff
[ ] add comment links - @ozkriff
[ ] add a draft of the 15 newsletter - @ozkriff

📜 Updated the structure/list of sections!

I'd like to take An OpenGL preprocessor for Rust (1)

I'll take femtovg.

I will submit an update about BUGOUT. ⚫⚪🐛
Thank you.
On Thu, Oct 1, 2020, 13:54 Andrey Lesnikóv<EMAIL_ADDRESS>wrote:

Feel free to suggest sections!
If you want to help writing the newsletter:

Grab one of the "🆓 free" sections by leaving a comment like "I'd
like to work on {section_name_1} and {section_name_2}".
Mentions in brackets (like "🆓 free (@ozkriff
https://github.com/ozkriff?)") are just an invitation to write the
corresponding section if you want, but anyone is free to take it.
Write a short overview:

Use provided links as a starting point.
Try to maintain a consistent style with the rest of the
newsletter and run markdownlint, please.
Insert line breaks (softbreak
https://spec.commonmark.org/0.29/#soft-line-breaks) at 80 chars,
please.
Use third-person pronouns even if you write about your project
updates.
DM me on Discord or Telegram if there're any questions.

Send a PR to the source branch (example: #75
https://github.com/rust-gamedev/rust-gamedev.github.io/pull/75)

Mention this issue in your PR's description to link it all
together.

I'm hoping to release this newsletter on 2020.10.06, so the soft deadline
for all section PRs is 2020.10.04.
Current structure/status (I'll try to keep this updated):

Game Updates

Veloren (1 https://veloren.net/) - 🚧 WIP by @AngelOnFira
https://github.com/AngelOnFira
Crate Before Attack (1 https://twitter.com/CrateAttack) - 🚧
WIP by @koalefant https://github.com/koalefant
A/B Street (1 https://twitter.com/CarlinoDustin, 2
https://reddit.com/r/abstreet/) - 🆓 free (@dabreegster
https://github.com/dabreegster?)
galangua (1 https://github.com/tyfkda/galangua, 2
https://tyfkda.github.io/blog/2020/09/17/galangua.html) - 🆓
free (@tyfkda https://github.com/tyfkda?)
Recall Singularity (1
https://reddit.com/r/rust_gamedev/comments/ilplba/the_recall_singularity_rust_godot_space_factory,
2
https://reddit.com/r/rust_gamedev/comments/imbn5q/recall_singularity_godot_rust_in_sep_2020_space)
🆓 free (@tom-leys https://github.com/tom-leys?)
mimas (1 https://github.com/est31/mimas, 2
https://github.com/est31/mimas) - 🆓 free (@est31
https://github.com/est31?)
Zemeroth (1 https://twitter.com/ozkriff) - 🚧 WIP by @ozkriff
https://github.com/ozkriff
Akigi (1 https://devjournal.akigi.com, 2
https://reddit.com/r/rust_gamedev/comments/iwqb2i/having_fun_building_my_custom_engines)
🆓 free (@todo https://github.com/todo?)
BUGOUT (1 https://github.com/Terkwood/BUGOUT, 2
https://reddit.com/r/rust_gamedev/comments/ixjeoh/bugout_aidriven_multiplayer_goweiqibaduk_for_the/
🆓 free (@Terkwood https://github.com/Terkwood?)
Tetris Bane (1 https://github.com/andii1701/tetris-bane, 2
https://reddit.com/r/rust_gamedev/comments/izhcz7/lockdown_side_project_tetris_bane_using_rust_with/)
🆓 free (@andii1701 https://github.com/andii1701)
Bevy Games (1
https://twitter.com/JpMoresmau/status/1310204411893952512, 2
https://github.com/JPMoresmau/ferris_dodger, 3
https://twitter.com/schneckerstein/status/1309491121555410945,
more?) - 🆓 free (@cart https://github.com/cart?)

Learning Material Updates

Threading WGPU Resource Loading with Rayon (1
https://sotrh.github.io/learn-wgpu/intermediate/tutorial13-threading/,
2
https://reddit.com/r/rust/comments/j16a0w/threading_wgpu_resource_loading_with_rayon,
2
https://reddit.com/r/rust/comments/iw1z6g/updated_learn_wgpu_to_06)
🆓 free (@sotrh https://github.com/sotrh?)
So you want to live-reload Rust (1
https://reddit.com/r/rust/comments/j0ajdy/so_you_want_to_livereload_rust/
🆓 free (@fasterthanlime https://github.com/fasterthanlime?)
Is it easy to draw a line (1
https://vladjuckov.github.io/hqlines) - 🚧 WIP by
@pum-purum-pum-pum https://github.com/pum-purum-pum-pum
An OpenGL preprocessor for Rust (1
https://codecrash.me/an-opengl-preprocessor-for-rust) - 🆓 free
Flappy Bird clone made with BEVY (1
https://youtube.com/watch?v=Qjc0V58lB7A) - 🆓 free (@TanTanDev
https://github.com/TanTanDev?)
Rust, gamedev, ECS, and bevy (1
https://hugopeixoto.net/articles/rust-gamedev-ecs-bevy.html, 2
https://hugopeixoto.net/articles/rust-gamedev-ecs-bevy-p2.html) -
🆓 free (@hugopeixoto https://github.com/hugopeixoto?)

Library & Tooling Updates

legion 0.3 (1
https://reddit.com/r/rust/comments/io6evk/legion_v03_released_and_now_part_of_the_amethyst)
🆓 free (@TomGillen https://github.com/TomGillen?)
thunderdome (1 https://github.com/LPGhatguy/thunderdome) - 🆓
free (@LPGhatguy https://github.com/LPGhatguy?)
fontdue (1
https://reddit.com/r/rust/comments/ilnd41/fontdue_the_fastest_font_renderer_in_the_world/)
🆓 free (@mooman219 https://github.com/mooman219?)
ultraviolet v0.6 (1
https://reddit.com/r/rust/comments/ir24mp/announcing_ultraviolet_06_including_support_for)
🆓 free (@termhn https://github.com/termhn?)
Mun - 🚧 WIP by @Wodann https://github.com/Wodann
rust-wasm-hotreload (1
https://github.com/shekohex/rust-wasm-hotreload, 2
https://twitter.com/ShekoHex/status/1302973994417651714) - 🆓
free (@shekohex https://github.com/shekohex?)
crevice (1 https://github.com/LPGhatguy/crevice, 2
https://twitter.com/LPGhatguy/status/1308499131212599296) - 🆓
free (@LPGhatguy https://github.com/LPGhatguy?)
femtovg (1 https://github.com/femtovg/femtovg, 2
https://reddit.com/r/rust_gamedev/comments/j1fx9u/femtovg_canvas2d_rendering_api_in_rust_based_on)
🆓 free (@adamnemecek https://github.com/adamnemecek?)
gfx/wgpu (1
https://reddit.com/r/rust/comments/ic9e62/release_announcement_for_gfxhal06_and_wgpu06/,
2
https://reddit.com/r/rust_gamedev/comments/i8np5v/simplified_instancing_tutorial_learn_wgpu,
3
https://reddit.com/r/rust_gamedev/comments/ixocl2/real_time_diffuse_global_illumination_for_static)
🚧 WIP by @kvark https://github.com/kvark
Riddle (1 https://github.com/vickles/riddle, 2
https://reddit.com/r/rust_gamedev/comments/j0xa3s/riddle_010_a_rust_native_sdllike_library)
🆓 free (@vickles https://github.com/vickles?)
miniquad/macroquad (1 https://twitter.com/fedor_games/, 2
https://twitter.com/kakoeimon/status/1311171685773701120) - 🚧
WIP by @not-fl3 https://github.com/not-fl3
tetra 0.5 (1
https://twitter.com/17cupsofcoffee/status/1301210538299609088, 2
https://twitter.com/fischspiele/status/1301269497370873867) - 🚧
WIP by @17cupsofcoffee https://github.com/17cupsofcoffee
Bevy (1 https://bevyengine.org/news/bevy-0-2/, 2
https://reddit.com/r/rust/comments/iw1yyp/bevy_02/) - 🚧 WIP by
@cart https://github.com/cart
godot-rust v0.9 (1
https://twitter.com/Extrawurst/status/1307797031210147840) - 🆓
free (@extrawurst https://github.com/extrawurst?)

Bonus - 🚧 WIP by @ozkriff https://github.com/ozkriff

Final steps:

final review - by everyone
publish - @ozkriff https://github.com/ozkriff
post on /r/rust, /r/rust_gamedv, /r/gamedev, URLO,
twitter.com/rust_gamedev - @ozkriff https://github.com/ozkriff
add comment links - @ozkriff https://github.com/ozkriff
add a draft of the 15 newsletter - @ozkriff
https://github.com/ozkriff

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rust-gamedev/rust-gamedev.github.io/issues/278, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AJIPHCCGIWAVHOHJTNFGCTLSIS665ANCNFSM4RJPXZBA
.

I can submit updates about my tagged libraries here!

I will add my progress of my game in the Bevy section :)

Hi, i'm the author of this demo, and I've been proposed to add an entry in the newsletter about it, count me in!

I will write about pGLOWrpg.

I'll write about riddle

@tigleym and I would like to write a game update section on the progress made on space_shooter_rs.

I'll write the update for "Rust, gamedev, ECS, and bevy"

I'd be willing to contribute an update for Nox Futura, a small update for bracket-lib, and ideally sneak in a mention of the book I'm about to release (I should be NDA free by the time this is due, if not it'll have to wait a month).

 :)

Btw, are there any volunteers to take some of the sections that are still free?

@ozkriff I have a decent amount of time today, most of them look like they're free with someone that might do it. Can you assign, say 3, to me?

My plan is to get something put together today. I presented at the
Roguelike Celebration this weekend, so that ate all my time.
On Mon, Oct 5, 2020 at 3:13 AM Andrey Lesnikóv<EMAIL_ADDRESS>wrote:

Ping @AngelOnFira https://github.com/AngelOnFira @not-fl3
https://github.com/not-fl3 @thebracket https://github.com/thebracket
@termhn https://github.com/termhn - how're your sections going? :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rust-gamedev/rust-gamedev.github.io/issues/278#issuecomment-703476386,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADRU432UGRMF5LVQCBMB67TSJF5ZJANCNFSM4RJPXZBA
.

Grabbing "Garden", "galangua", "Threading WGPU Resource Loading with Rayon", and "So you want to live-reload Rust".

Also taking "legion", "fontdue", "ultraviolet", "rust-wasm-hotreload", and "godot-rust".

The final PR: #313 ("Newsletter 14") - everyone is invited to review!

https://rust-gamedev.github.io/posts/newsletter-014 :tada:
As always, huge thanks to all contributors and reviewers! :crab::crab:

Next coordination issue: #320 ("Newsletter 15")


📝 Create Nantes Machine Learning Meetup

I've added a card for Nantes Machine Learning Meetup, hope it will be useful

Thanks Julia 👍


Extra memory usage after first evaluation

System Info
- `Accelerate` version: 0.18.0
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.17
- Python version: 3.8.16
- Numpy version: 1.24.2
- PyTorch version (GPU?): 1.13.0+cu116 (True)
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: NO
        - mixed_precision: no
        - use_cpu: False
        - num_processes: 1
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: 0,1,2
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []

Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
[X] My own task or dataset (give details below)

Reproduction
Under DDP setting
model is t5 language model
for training loop with gradient accumulation:
    (extra memory after first evaluation, not sure where it comes from)
    if eval condition is met:
         model.eval()
         eval loop  (extra memory from eval data embedding)
         model.train()

Expected behavior
It's expected that after first evaluation, there is no extra memory usage. I tried to use accelerator.clear() after each evaluation but it doesn't help. Is there any other suggestion?

This is too vague for us to be able to help. We need a clear reproducer of the issue.


$ref

Hi guys
Do you have a simple example of the use of $ref with Angular-Schema-Form? The example in https://github.com/json-schema-form/angular-schema-form/issues/69 is not clear and I don't understand if  you provide support for it.
Thanks for all

@example2049 it isn't supported yet, it is planned, but no timeline has been determined, I'd follow #69  to get notified of updates. We are closer to being ready to start implementing it, but other than that only using the example provided in that ticket would be my only suggestion prior to me actually trying to build it.

As of 1.0.0-alpha.2 $ref is supported as a preprocessing step.


expose isValidDocumentId

It's useful to haveisValidDocumentId exposed so if we can check validity of an id when it comes from outside.

Yeah, I agree. I'll get it into the next point release.


Refactor use template and single instance to refactor get type info logic

Proposed changes
Issue Number: close #xxx
Problem Summary:
Describe the overview of changes.
Checklist(Required)

Does it affect the original behavior: (Yes/No/I Don't know)
Has unit tests been added: (Yes/No/No Need)
Has document been added or modified: (Yes/No/No Need)
Does it need to update dependencies: (Yes/No)
Are there any changes that cannot be rolled back: (Yes/No)

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

for some certainly types, I think all use unified _typeinfo do not reflect semantics,
such as bf_typeinfo is better than _typeinfo. we may can modify some certainly types variables names.

for some certainly types, I think all use unified _typeinfo do not reflect semantics, such as bf_typeinfo is better than _typeinfo. we may can modify some certainly types variables names.

actually _type_info is a private variable of parent object , so I think it is clear now, we can easy get its usage by code

I think this pr should be discussed sufficiently before merge it in.

I think this pr should be discussed sufficiently before we merge it in.
Unlike the scalar type, the type info of dynamic types (such as Array, Map and etc) can not be generated in compile time. For instance, in get_type_info(const TabletColumn* col), the depth of the col should be calculated first and it is unknown until runtime. The optimization for ArrayType here is powered by small object optimization technique (cache objects with small depth) but it sacrifice the flexibility for objects with bigger depth.
I think there are some possible improvements (may hard to implement):

It is reasonable to restrict the max depth to protect our system from suffering serious performance penalty. I was also planning to do this. However, if we can configure the max depth in runtime, it may be better than the hard coded number.
ArrayTypeInfo can be refactored to store both leaf type info and depth, therefore, we can construct an ArrayTypeInfo with any depth in a quick way. After that, we may cache the objects with small depth statically and use some cache strategies for objects with large depth to speed up.

All in all, the key problem is how to manage the resource of dynamic types if there are generated dynamically. I think we should also consider whether the refactor introduced by this pr is suitable for the type info of MayType (which may be more hard to manage than ArrayType).
For now, ArrayType feature is under development and we have not start the performance stuffs for ArrayType. Thanks for the contribution. What's more, how much the benefit after this modification for ArrayType? Is there any statistics?

In fact, my PR is optimized for scalar type, because get scalar type is used very frequently, so the operation of getting scalar type must be light enough. Here I use templates and static variables to optimize get scalar type, and I I think that for the design of composite types such as arrays and maps, my changes are only expedient. I think composite types may need to be managed with another set of logic in future, because scalar type is the most used type by users, so performance is very important, PR #8500 has already done some performance
optimization work for this operation

Hi @caiconghui , I read the #8500 before and knew the motivation behind this pr, but the challenge is that some functions (such as get_type_info(const TabletColumn* col)) and some members in classes (such as type_info in  ColumnVectorBatch) are used for both scalar types and dynamic types. If we manage the resource created by dynamic types and ignore the static resource for static types, the unified const TypeInfo* returned by some functions is appropriate. I think some functions and some usage for TypeInfo should be SEPARATED first before apply the  optimization for static types.

Hi @caiconghui , I'm doing some work to separate some functions and usage for ArrayType. After that, you can employ the optimization to scalar types.

Hi @caiconghui , I'm doing some work to separate some functions and usage for ArrayType. After that, the optimization of scalar types may be more suitable to be employed.

the work you are doing now aims to merged in dev/1.0.1? if so, we can wait to be merged until your work done.

Due to the timeline of release for 1.0.1, it is ok to merge it first.


Custom modifications

Be able to handle custom modifications. This system should generate a list of Modification::Predefined(.*) (or do I need a new enum option?) that can be loaded into rustyms. One of the main uses is diagnostic ions (see snijderlab/rustyms#6).
Design considerations:

List of modifications, name, chemical composition(, and mass?)

Each has one or more locations
Each (set of) locations can have neutral losses and diagnostic ions, these can be defined as chemical compositions. But be clear that they have to be provided in M (not MH+) mode.

Be able to generate good looking names in the spectrum
Auto save the mods, and document where this file is saved for easy sharing set up of a new pc (serde serialisation, look out for potential version incompatibilities)
Auto detect these mods in the sequence, options:

on name A[C:<name>] (where C stands for custom)
on mass A[42.42] (where this mass matches the predefined one, potentially an issue if you have multiple isobaric mods, or if any previous tool has a different meaning on the mass)
on formula does not look feasible, same problems as mass but less nice to type
with info tags is the 'correct' way to do it in pro forma, but hard to type, this could be a nice thing to use as export

Fully set up on the rustyms side, using [C:name] or [CUSTOM:number]. Now the work that is left is building the UI and thinking on how to store these over multiple uses of the annotator.


Separate badges for package coverage

@meatballhat I was hoping it might resolve after the merge, but the codegangsta/cli badge appears to not work quite right.

Using HTTPS appears to have fixed it: https://github.com/codegangsta/cli/commit/92897b9bf6639511e0830b4fc5e9c1c5e796fc34


Lab Structure

Whilst the LAB works very well, I feel that it almost too short. Based on the fact we will start the instruction from the point of a working Kubectl, Im concerned the lab will get completed in 5-10 mins.
I think we need to spend more time initially explaining/dissecting/showing each of the cert-manager components. i.e. What it is, how it works etc including its use of CRD's etc.
I think we could also do with a section that walks the attendees through some basic troubleshooting steps. e.g. what if the cert-request fails.
Also, we should spend some time explaining more about the K8s architecture in which cert-manager interoperates with. e.g. what an ingress controller is and how it works? What is a work-load ? Why are machine identities important etc.
Carl

Going to close this as a lot more is added now (and we have extra slides)


Parse a whole line of crontab

As far as I know, there's currently no way to parse a full line of crontab using robfig/cron, that is to say, including the script.
This is because the parser will only parse the schedule and return an error if it finds too many fields or a malformed DOW.
I realize this is probably not the most common use case, for those of like myself who are looking parse a whole line from a standard crontab, this is quite the PITA.
It would be great if we would be able to do something like NewParser(Minute| Dom | Month | DowOptional | Script).

Isn't it relatively straightforward to peel the script off prior to handing the crontab to this package?

Well you would have "parse" (however you want to do it) the input to separate the script from the schedule, which would be either pretty adhoc, or, very similar to what the parser in this package already does. Just seems like the parser would be the right place for it.
Alternatively, the parser could only take from the input string that it knows how to deal with, and leave the remainder for the client code. As opposed to failing on extra input.

Changing the parser to have that behavior would mean that it could not return error messages if too many fields are provided by accident. I see that as a useful functionality. Sorry!