[WIP][SPARK-20629] Copy shuffle data when nodes are being shutdown

Why are the changes needed?
Recomputting shuffle blocks can be expensive, we should take advantage of our decommissioning time to migrate these blocks.
Does this PR introduce any user-facing change?
This PR introduces two new configs parameters, spark.storage.decommission.shuffle_blocks & spark.storage.decommission.rdd_blocks that control which blocks should be migrated during storage decommissioning.
How was this patch tested?
New unit test & expansion of the Spark on K8s decom test to assert that decommisioning with shuffle block migration means that the results are not recomputed even when the original executor is terminated.

Test build #121782 has started for PR 28331 at commit 4126c1b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26457/
Test PASSed.

Test build #121782 has finished for PR 28331 at commit 4126c1b.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class RDDBlockSavedOnDecommissionedBlockManagerException(blockId: RDDBlockId)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121782/
Test FAILed.

If you've got some cycles to take a look at the WIP PR @prakharjain09 I'd appreciate it. This builds on top of the cache block migration.
Why this PR is still a work in progress:

Only works with indexed shuffle files
I'm not sure if this is the best way to copy shuffle files between executors

Future work (not planned in this PR but in the future):

Supporting write back to some type of DFS
Updating the listener that tracks location of shuffle blocks for scale down to understand block migrations
Integrate decommissioning into our planned scale down, not just cluster manager/cloud triggered scale downs.

Test build #121784 has started for PR 28331 at commit 8ee8949.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26459/
Test PASSed.

Test build #121784 has finished for PR 28331 at commit 8ee8949.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121784/
Test FAILed.

Test build #121785 has started for PR 28331 at commit 4071ae2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26460/
Test PASSed.

Test build #121785 has finished for PR 28331 at commit 4071ae2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121785/
Test FAILed.

Jenkins retest this please
On Fri, Apr 24, 2020 at 6:11 PM UCB AMPLab<EMAIL_ADDRESS>wrote:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121785/
Test FAILed.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/28331#issuecomment-619296594, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAOT5JBB2W37B3JVTA5KLDROI2DNANCNFSM4MQOCCXA
.

--
Cell :<PHONE_NUMBER>

cc @dbtsai

Test build #122153 has started for PR 28331 at commit ff620ba.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26824/
Test PASSed.

Test build #122153 has finished for PR 28331 at commit ff620ba.

This patch fails from timeout after a configured wait of 400m.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122153/
Test FAILed.

Jenkins retest this please

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26850/
Test PASSed.

The original cache block migration is reverted, isn't it?

Test build #122178 has started for PR 28331 at commit ff620ba.

Oh yeah that's the conflicts, but I'm curious about the 400 minute timeout issue in Jenkins since I'm getting different results with the executor exiting locally.

Test build #122178 has finished for PR 28331 at commit ff620ba.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122178/
Test FAILed.

Test build #122181 has started for PR 28331 at commit adb03db.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26853/
Test PASSed.

Test build #122183 has started for PR 28331 at commit be2a5e7.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26855/
Test PASSed.

Test build #122181 has finished for PR 28331 at commit adb03db.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122181/
Test FAILed.

Test build #122183 has finished for PR 28331 at commit be2a5e7.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122183/
Test PASSed.

Test build #122183 has finished for PR 28331 at commit be2a5e7.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

@holdenk All tests started passing after this commit of yours. This includes all my changes also. Any specific change you know of in your branch which has fixed all the tests?

Jenkins retest this please.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26947/
Test PASSed.

Test build #122279 has started for PR 28331 at commit be2a5e7.

Test build #122279 has finished for PR 28331 at commit be2a5e7.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122279/
Test FAILed.

Test build #123152 has started for PR 28331 at commit dbe2418.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27783/
Test PASSed.

Test build #123152 has finished for PR 28331 at commit dbe2418.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123152/
Test FAILed.

Test build #123154 has started for PR 28331 at commit a240f98.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27784/
Test PASSed.

Test build #123154 has finished for PR 28331 at commit a240f98.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123154/
Test FAILed.

There are two issues I can see.

Locally (at my side) the verify that an already running task which is going to cache data succeeds on a decommissioned executor always fails.

It fails with sbt and with maven too. Last time I tried with:
$ ./build/mvn clean install -DskipTests
$ ./build/mvn -pl :spark-core_2.12 test -Dtest=none '-Dsuites=*.BlockManagerDecommissionSuite'

The failure is:
Run starting. Expected test count is: 2
BlockManagerDecommissionSuite:
- verify that an already running task which is going to cache data succeeds on a decommissioned executor *** FAILED ***
  Vector(rdd_1_0) did not equal List() (BlockManagerDecommissionSuite.scala:141)
- verify that shuffle blocks are migrated.
Run completed in 26 seconds, 757 milliseconds.
Total number of tests run: 2
Suites: completed 1, aborted 0
Tests: succeeded 1, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***

The relevant log from core/target/unit-tests.log:
20/05/27 00:06:11.432 ScalaTest-main-running-BlockManagerDecommissionSuite INFO StandaloneSchedulerBackend: Propagating executor decommission to driver.
20/05/27 00:06:11.433 dispatcher-CoarseGrainedScheduler ERROR CoarseGrainedSchedulerBackend$DriverEndpoint: Received decommission executor message 1.
20/05/27 00:06:11.433 dispatcher-CoarseGrainedScheduler INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Starting decommissioning executor 1.
20/05/27 00:06:11.434 dispatcher-CoarseGrainedScheduler INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Finished decommissioning executor 1.
20/05/27 00:06:11.434 dispatcher-CoarseGrainedScheduler INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Starting decommissioning block manager corresponding to executor 1.
20/05/27 00:06:11.435 dispatcher-CoarseGrainedScheduler INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Acknowledged decommissioning block manager corresponding to 1.
20/05/27 00:06:11.922 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_0 in memory on <IP_ADDRESS>:57044 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:11.927 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_1 in memory on <IP_ADDRESS>:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:11.933 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 10) in 610 ms on <IP_ADDRESS> (executor 1) (1/10)
20/05/27 00:06:11.940 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 12, <IP_ADDRESS>, executor 0, partition 2, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:11.940 task-result-getter-3 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 11) in 616 ms on <IP_ADDRESS> (executor 0) (2/10)
20/05/27 00:06:12.454 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_2 in memory on <IP_ADDRESS>:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:12.459 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 13, <IP_ADDRESS>, executor 0, partition 3, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:12.460 task-result-getter-0 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 12) in 521 ms on <IP_ADDRESS> (executor 0) (3/10)
20/05/27 00:06:12.971 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_3 in memory on <IP_ADDRESS>:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:12.977 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 4.0 in stage 1.0 (TID 14, <IP_ADDRESS>, executor 0, partition 4, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:12.977 task-result-getter-1 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 13) in 518 ms on <IP_ADDRESS> (executor 0) (4/10)
20/05/27 00:06:13.489 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_4 in memory on <IP_ADDRESS>:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:13.494 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 5.0 in stage 1.0 (TID 15, <IP_ADDRESS>, executor 0, partition 5, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:13.495 task-result-getter-2 INFO TaskSetManager: Finished task 4.0 in stage 1.0 (TID 14) in 518 ms on <IP_ADDRESS> (executor 0) (5/10)
20/05/27 00:06:14.007 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_5 in memory on <IP_ADDRESS>:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:14.012 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 6.0 in stage 1.0 (TID 16, <IP_ADDRESS>, executor 0, partition 6, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:14.012 task-result-getter-3 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID 15) in 519 ms on <IP_ADDRESS> (executor 0) (6/10)
20/05/27 00:06:14.523 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_6 in memory on <IP_ADDRESS>:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:14.527 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 7.0 in stage 1.0 (TID 17, <IP_ADDRESS>, executor 0, partition 7, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:14.528 task-result-getter-0 INFO TaskSetManager: Finished task 6.0 in stage 1.0 (TID 16) in 517 ms on <IP_ADDRESS> (executor 0) (7/10)
20/05/27 00:06:15.040 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_7 in memory on <IP_ADDRESS>:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:15.044 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 8.0 in stage 1.0 (TID 18, <IP_ADDRESS>, executor 0, partition 8, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:15.045 task-result-getter-1 INFO TaskSetManager: Finished task 7.0 in stage 1.0 (TID 17) in 518 ms on <IP_ADDRESS> (executor 0) (8/10)
20/05/27 00:06:15.559 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_8 in memory on <IP_ADDRESS>:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:15.564 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 9.0 in stage 1.0 (TID 19, <IP_ADDRESS>, executor 0, partition 9, PROCESS_LOCAL, 7454 bytes) taskResourceAssignments Map()
20/05/27 00:06:15.564 task-result-getter-2 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID 18) in 520 ms on <IP_ADDRESS> (executor 0) (9/10)
20/05/27 00:06:16.079 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_9 in memory on <IP_ADDRESS>:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:16.084 task-result-getter-3 INFO TaskSetManager: Finished task 9.0 in stage 1.0 (TID 19) in 520 ms on <IP_ADDRESS> (executor 0) (10/10)
20/05/27 00:06:16.084 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
20/05/27 00:06:16.085 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 1 (countAsync at BlockManagerDecommissionSuite.scala:92) finished in 4.771 s
20/05/27 00:06:16.085 dag-scheduler-event-loop INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job
20/05/27 00:06:16.085 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 1: Stage finished
20/05/27 00:06:17.105 ScalaTest-main-running-BlockManagerDecommissionSuite INFO SparkContext: Starting job: count at BlockManagerDecommissionSuite.scala:134
20/05/27 00:06:17.106 dag-scheduler-event-loop INFO DAGScheduler: Got job 2 (count at BlockManagerDecommissionSuite.scala:134) with 10 output partitions
20/05/27 00:06:17.106 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 2 (count at BlockManagerDecommissionSuite.scala:134)
20/05/27 00:06:17.106 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List()
20/05/27 00:06:17.107 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List()
20/05/27 00:06:17.107 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[1] at mapPartitions at BlockManagerDecommissionSuite.scala:63), which has no missing parents
20/05/27 00:06:17.110 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.2 KiB, free 2.1 GiB)
20/05/27 00:06:17.111 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 2.1 GiB)
20/05/27 00:06:17.112 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on apiros-mbp16.lan:57037 (size: 2.4 KiB, free: 2.1 GiB)
20/05/27 00:06:17.112 dag-scheduler-event-loop INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1293
20/05/27 00:06:17.113 dag-scheduler-event-loop INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 2 (MapPartitionsRDD[1] at mapPartitions at BlockManagerDecommissionSuite.scala:63) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
20/05/27 00:06:17.113 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 2.0 with 10 tasks resource profile 0
20/05/27 00:06:17.117 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 20, <IP_ADDRESS>, executor 0, partition 1, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.129 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on <IP_ADDRESS>:57048 (size: 2.4 KiB, free: 546.3 MiB)
20/05/27 00:06:17.135 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID 21, <IP_ADDRESS>, executor 0, partition 2, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.136 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID 20) in 20 ms on <IP_ADDRESS> (executor 0) (1/10)
20/05/27 00:06:17.141 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 3.0 in stage 2.0 (TID 22, <IP_ADDRESS>, executor 0, partition 3, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.142 task-result-getter-1 INFO TaskSetManager: Finished task 2.0 in stage 2.0 (TID 21) in 7 ms on <IP_ADDRESS> (executor 0) (2/10)
20/05/27 00:06:17.147 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 4.0 in stage 2.0 (TID 23, <IP_ADDRESS>, executor 0, partition 4, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.147 task-result-getter-2 INFO TaskSetManager: Finished task 3.0 in stage 2.0 (TID 22) in 6 ms on <IP_ADDRESS> (executor 0) (3/10)
20/05/27 00:06:17.153 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 5.0 in stage 2.0 (TID 24, <IP_ADDRESS>, executor 0, partition 5, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.153 task-result-getter-3 INFO TaskSetManager: Finished task 4.0 in stage 2.0 (TID 23) in 6 ms on <IP_ADDRESS> (executor 0) (4/10)
20/05/27 00:06:17.159 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 6.0 in stage 2.0 (TID 25, <IP_ADDRESS>, executor 0, partition 6, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.159 task-result-getter-0 INFO TaskSetManager: Finished task 5.0 in stage 2.0 (TID 24) in 6 ms on <IP_ADDRESS> (executor 0) (5/10)
20/05/27 00:06:17.166 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 7.0 in stage 2.0 (TID 26, <IP_ADDRESS>, executor 0, partition 7, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.166 task-result-getter-1 INFO TaskSetManager: Finished task 6.0 in stage 2.0 (TID 25) in 8 ms on <IP_ADDRESS> (executor 0) (6/10)
20/05/27 00:06:17.171 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 8.0 in stage 2.0 (TID 27, <IP_ADDRESS>, executor 0, partition 8, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.171 task-result-getter-2 INFO TaskSetManager: Finished task 7.0 in stage 2.0 (TID 26) in 6 ms on <IP_ADDRESS> (executor 0) (7/10)
20/05/27 00:06:17.177 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 9.0 in stage 2.0 (TID 28, <IP_ADDRESS>, executor 0, partition 9, PROCESS_LOCAL, 7454 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.177 task-result-getter-3 INFO TaskSetManager: Finished task 8.0 in stage 2.0 (TID 27) in 6 ms on <IP_ADDRESS> (executor 0) (8/10)
20/05/27 00:06:17.183 task-result-getter-0 INFO TaskSetManager: Finished task 9.0 in stage 2.0 (TID 28) in 7 ms on <IP_ADDRESS> (executor 0) (9/10)
20/05/27 00:06:20.661 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 29, <IP_ADDRESS>, executor 0, partition 0, NODE_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:20.679 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 29) in 18 ms on <IP_ADDRESS> (executor 0) (10/10)
20/05/27 00:06:20.679 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
20/05/27 00:06:20.680 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 2 (count at BlockManagerDecommissionSuite.scala:134) finished in 3.572 s
20/05/27 00:06:20.680 dag-scheduler-event-loop INFO DAGScheduler: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job
20/05/27 00:06:20.680 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 2: Stage finished
20/05/27 00:06:20.680 ScalaTest-main-running-BlockManagerDecommissionSuite INFO DAGScheduler: Job 2 finished: count at BlockManagerDecommissionSuite.scala:134, took 3.575365 s
20/05/27 00:06:20.699 ScalaTest-main-running-BlockManagerDecommissionSuite INFO BlockManagerDecommissionSuite:

===== FINISHED o.a.s.storage.BlockManagerDecommissionSuite: 'verify that an already running task which is going to cache data succeeds on a decommissioned executor' =====

As I cannot see  a "Starting block manager decommissioning process..."  somewhere after the "Starting decommissioning block manager corresponding to executor 1" line I suspect it must be the asynchronous execution.
Here https://github.com/apache/spark/blob/a240f9857dcf53631b269422b4652a05304691be/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L320 we return with a Future  which might not be completed, see:
https://github.com/apache/spark/blob/a240f9857dcf53631b269422b4652a05304691be/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L160-L163

The 2nd problem is the test "verify that shuffle blocks are migrated" runs successfully even as ShuffleBlockBatchId  blocks are targeted for migration and not correct ShuffleDataBlockId blocks.
One reason could be the recompute kicks in. Although the completed task are counted and checked whether we have the expected number of successfully finished tasks. I need some time to find out what's going on here.

So it is not recalculation but the timing: the decommissioned executor is still giving back those shuffle data files:
You can check it by adding  a temporary log:
diff --git a/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala b/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
index b959e83599..ca45dc0905 100644
--- a/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
+++ b/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
@@ -85,9 +85,11 @@ private[spark] class IndexShuffleBlockResolver(
    */
    def getDataFile(shuffleId: Int, mapId: Long, dirs: Option[Array[String]]): File = {
     val blockId = ShuffleDataBlockId(shuffleId, mapId, NOOP_REDUCE_ID)
-    dirs
+    val file = dirs
       .map(ExecutorDiskUtils.getFile(_, blockManager.subDirsPerLocalDir, blockId.name))
       .getOrElse(blockManager.diskBlockManager.getFile(blockId))
+    logInfo(s"Attila: shuffle file: ${file.getAbsolutePath()}", new Exception())
+    file
   }

   /**

Then please remove the work directory content and execute this test only i.e. by sbt:
> testOnly *.BlockManagerDecommissionSuite -- -z "verify that shuffle blocks are migrated"
[warn] Multiple main classes detected.  Run 'show discoveredMainClasses' to see the list
[info] Packaging /Users/attilazsoltpiros/git/attilapiros/reviewing/core/target/scala-2.12/spark-core_2.12-3.1.0-SNAPSHOT.jar ...
[info] Done packaging.
[info] BlockManagerDecommissionSuite:
[info] - verify that shuffle blocks are migrated. (12 seconds, 968 milliseconds)
[info] ScalaTest
[info] Run completed in 15 seconds, 145 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 19 s, completed May 28, 2020 7:17:21 PM

You can check the logs in the work directory:
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/reviewing/work ‹pr/28331*›
╰─$ ag Attila
app-20200528101709-0000/0/target/unit-tests.log
69:20/05/28 19:17:14.044 Executor task launch worker for task 10 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/33/shuffle_0_10_0.data
86:20/05/28 19:17:14.363 Executor task launch worker for task 10 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/33/shuffle_0_10_0.data
106:20/05/28 19:17:14.886 Executor task launch worker for task 12 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/31/shuffle_0_12_0.data
123:20/05/28 19:17:14.894 Executor task launch worker for task 12 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/31/shuffle_0_12_0.data
143:20/05/28 19:17:15.409 Executor task launch worker for task 13 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/30/shuffle_0_13_0.data
160:20/05/28 19:17:15.416 Executor task launch worker for task 13 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/30/shuffle_0_13_0.data
180:20/05/28 19:17:15.933 Executor task launch worker for task 14 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/2f/shuffle_0_14_0.data
197:20/05/28 19:17:15.940 Executor task launch worker for task 14 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/2f/shuffle_0_14_0.data
217:20/05/28 19:17:16.457 Executor task launch worker for task 15 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0e/shuffle_0_15_0.data
234:20/05/28 19:17:16.463 Executor task launch worker for task 15 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0e/shuffle_0_15_0.data
254:20/05/28 19:17:16.975 Executor task launch worker for task 16 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/13/shuffle_0_16_0.data
271:20/05/28 19:17:16.981 Executor task launch worker for task 16 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/13/shuffle_0_16_0.data
291:20/05/28 19:17:17.497 Executor task launch worker for task 17 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0c/shuffle_0_17_0.data
308:20/05/28 19:17:17.502 Executor task launch worker for task 17 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0c/shuffle_0_17_0.data
328:20/05/28 19:17:18.017 Executor task launch worker for task 18 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/15/shuffle_0_18_0.data
345:20/05/28 19:17:18.022 Executor task launch worker for task 18 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/15/shuffle_0_18_0.data
365:20/05/28 19:17:18.538 Executor task launch worker for task 19 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0a/shuffle_0_19_0.data
382:20/05/28 19:17:18.543 Executor task launch worker for task 19 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0a/shuffle_0_19_0.data
412:20/05/28 19:17:19.177 Executor task launch worker for task 20 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0a/shuffle_0_19_0.data
437:20/05/28 19:17:19.204 Executor task launch worker for task 21 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/33/shuffle_0_10_0.data
467:20/05/28 19:17:19.248 Executor task launch worker for task 23 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/31/shuffle_0_12_0.data
492:20/05/28 19:17:19.259 Executor task launch worker for task 24 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/30/shuffle_0_13_0.data
517:20/05/28 19:17:19.271 Executor task launch worker for task 25 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/2f/shuffle_0_14_0.data
542:20/05/28 19:17:19.282 Executor task launch worker for task 26 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0e/shuffle_0_15_0.data
567:20/05/28 19:17:19.295 Executor task launch worker for task 27 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/13/shuffle_0_16_0.data
592:20/05/28 19:17:19.305 Executor task launch worker for task 28 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0c/shuffle_0_17_0.data
617:20/05/28 19:17:19.316 Executor task launch worker for task 29 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/15/shuffle_0_18_0.data
646:20/05/28 19:17:20.375 Executor task launch worker for task 30 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0a/shuffle_0_19_0.data
671:20/05/28 19:17:20.386 Executor task launch worker for task 31 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/33/shuffle_0_10_0.data
701:20/05/28 19:17:20.411 Executor task launch worker for task 33 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/31/shuffle_0_12_0.data
726:20/05/28 19:17:20.422 Executor task launch worker for task 34 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/30/shuffle_0_13_0.data
751:20/05/28 19:17:20.433 Executor task launch worker for task 35 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/2f/shuffle_0_14_0.data
776:20/05/28 19:17:20.443 Executor task launch worker for task 36 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0e/shuffle_0_15_0.data
801:20/05/28 19:17:20.454 Executor task launch worker for task 37 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/13/shuffle_0_16_0.data
826:20/05/28 19:17:20.466 Executor task launch worker for task 38 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0c/shuffle_0_17_0.data
851:20/05/28 19:17:20.476 Executor task launch worker for task 39 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/15/shuffle_0_18_0.data

app-20200528101709-0000/1/target/unit-tests.log
97:20/05/28 19:17:14.044 Executor task launch worker for task 11 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
114:20/05/28 19:17:14.363 Executor task launch worker for task 11 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
132:20/05/28 19:17:19.231 shuffle-server-6-1 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
189:20/05/28 19:17:20.397 shuffle-server-6-1 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data

So from this you can see shuffle_0_11_0.data is only accessed in one of the executor. You can see this more easily by this:
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/reviewing/work ‹pr/28331*›
╰─$ ag shuffle_0_11_0.data
app-20200528101709-0000/1/target/unit-tests.log
97:20/05/28 19:17:14.044 Executor task launch worker for task 11 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
114:20/05/28 19:17:14.363 Executor task launch worker for task 11 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
132:20/05/28 19:17:19.231 shuffle-server-6-1 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
189:20/05/28 19:17:20.397 shuffle-server-6-1 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/reviewing/work ‹pr/28331*›

And you can check the call stack too for these accesses.
We should harden this test to produce failure when this happens. The test should fail even when the decommission is not triggered.
And I believe we need some tests at deeper level (this one might be too high: way above unit testing).
And It came to my mind how good it would be to have some metrics for the number of migrated blocks too (but is not a priority but something to think about).

Probably the easist thing to do is delete the executor in the K8s test after it's had a chance to migrate the blocks.

Test build #123244 has started for PR 28331 at commit 838a346.

Oh wait we already do that in the K8s integration test. So I think that's tested but I'll see what I can do to improve the unit tests as well.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27867/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27870/
Test PASSed.

Test build #123248 has started for PR 28331 at commit 2da0f2d.

Test build #123248 has finished for PR 28331 at commit 2da0f2d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123248/
Test FAILed.

Test build #123252 has started for PR 28331 at commit a92025c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27874/
Test PASSed.

I've verified the updated unit test fails when I manually set migrate shuffle blocks to false, thanks @attilapiros :)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27876/
Test PASSed.

Test build #123254 has started for PR 28331 at commit fe265d7.

Test build #123244 has finished for PR 28331 at commit 838a346.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123244/
Test PASSed.

Test build #123252 has finished for PR 28331 at commit a92025c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123252/
Test FAILed.

Test build #123254 has finished for PR 28331 at commit fe265d7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123254/
Test FAILed.

Test build #123259 has started for PR 28331 at commit 70c3871.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27881/
Test PASSed.

Test build #123259 has finished for PR 28331 at commit 70c3871.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123259/
Test PASSed.

Test build #123297 has started for PR 28331 at commit 069dd3b.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27919/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123297/
Test PASSed.

Test build #123301 has started for PR 28331 at commit 6340f9b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27923/
Test PASSed.

Test build #123301 has finished for PR 28331 at commit 6340f9b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123301/
Test FAILed.

Test build #123394 has started for PR 28331 at commit 4cfeb8e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28013/
Test PASSed.

Test build #123396 has started for PR 28331 at commit e81aa5a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28014/
Test PASSed.

Test build #123394 has finished for PR 28331 at commit 4cfeb8e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123394/
Test FAILed.

Test build #123400 has started for PR 28331 at commit 155aeb2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28018/
Test PASSed.

Test build #123396 has finished for PR 28331 at commit e81aa5a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123396/
Test FAILed.

Test build #123407 has started for PR 28331 at commit 7f93df6.

If this passes the new tests I'm going to make a new non-WIP PR. Thank you for all your help reviewing @attilapiros the suggestions have really sped up my work here :)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28024/
Test PASSed.

Merged build finished. Test PASSed.

Test build #123409 has started for PR 28331 at commit 7e32341.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28025/
Test PASSed.

Test build #123400 has finished for PR 28331 at commit 155aeb2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123400/
Test PASSed.

Test build #123409 has finished for PR 28331 at commit 7e32341.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123409/
Test FAILed.

Test build #123407 has finished for PR 28331 at commit 7f93df6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123407/
Test PASSed.

Test build #123446 has started for PR 28331 at commit a3aa8eb.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28064/
Test PASSed.

I've made a non-WIP PR based on this, so closing this on for now.

Test build #123446 has finished for PR 28331 at commit a3aa8eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123446/
Test PASSed.


Root Motion offsetting mesh position

Tested versions
Godot 4.3, 4.2.2, 4.2.1
System information
Godot v4.3.stable - Windows 10.0.19045 - Vulkan (Forward+) - dedicated GeForce GTX 1060 6GB - Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz (12 Threads)
Issue description
I have a 3D character model that uses Root Motion in its animation. At random, the model will be offset by the Root Motion and end up floating in the air or standing several dozen feet away from its actual origin point.
To fix it, I have to:
-Go to the Animation Player
-Unassign the Root Motion Track
-Play animations where the model is at the correct point of origin- sometimes going to a different animation and then back again
-Then setting the Root Motion Track again
Information:
-The character model is using a saved Animation Library, derived from its own Animations
-The character uses several Animation Trees that are linked to its Animation Player. Only one is Active at a time.
-Each Animation Tree has the same Root Motion Track selected, matching the Animation Player
-Toggling "Show Rest Only" on the Skeleton3D works as it should, and turning it off returns the character to the incorrect position
-Neither playing any given animation nor any amount of scripted movement has an effect on the offset
-Animations played through the Animation Player have no effect on the offset
-The issue never triggers in-game, only in the editor. The issue remains in effect when going in-game
Steps to reproduce
I can't determine what the circumstances under which it keeps reoccurring, but it can be achieved manually:
-Import an animated model with Root Motion in the animations
-Leave the Root Motion Track empty
-Play an animation that includes Root Motion
-Assign the Root Motion Track when the model is moved away from its origin point in the animation
But the issue crops up randomly. I haven't been able to catch when it happens, so I can't figure out why.
Minimal reproduction project (MRP)
"If the reproduction steps are not project dependent (e.g. the bug is visible in a brand new project), you can write "N/A" in the field."
N/A

MRP is required

Honestly. Does every post here have to be a fight? Mind your manners.
I have no idea whether the actual issue is project dependent or not, but the way to recreate it is not. If that can be fixed, then we can tell if the two issues were the same thing. Since both trigger only in the editor, the easy assumption is that they are.
Keep in mind that the recreation method doesn't involve any scripts, and only relies on Root Motion's movement insofar as it moves the model away from its origin point. It's not that it's wrong, just that it's not resetting as it should.

RootMotionMRP.zip
Here's an MRP- it's a cube with Root Motion that should be inappropriately offset already. If not, just follow the reproduction steps.

The mesh is moved in the MRP because the bone pose is actually set on the Skeleton. If the problem continues to occur after Reset Bone Poses in the Skeleton Editor, then it is a bug, but if not, then it is not a bug.

If you want to set a RootMotion and not leave the Pose that was playing before, that is a feature proposal, so it is more appropriate thet you send the issue to the https://github.com/godotengine/godot-proposals repository with the title "Call Reset All Bone Poses when setting RootMotionTrack to AnimationMixer".

Reset All Bone Poses fixes the position issue on the recreation method, so I assume it will fix the position being changed in the issue, itself, so that's good- an easier way to undo when it changes.
Unfortunately, that still leaves the 'why' of it- I have no idea why this randomly occurs. An important note is that I never actually see it happening, I just notice it when it has happened, so if trying to puzzle it out doesn't bear any fruit, I can simply wait until I actually catch it happening and update the thread.
Some musing on the issue:
When the Root Motion Track is set the mesh shouldn't move at all, but setting the Root Motion Track when the mesh is offset will trigger the issue. Does this mean that the Root Motion Track is being cleared and reset by some process in the engine?
Reset On Save sounds like it could possibly be a culprit, as it resets animation settings in the Editor, but I don't actually have any Reset Tracks, so perhaps not.
The Editor is completely unaffected by what occurs in the Remote in-game instances, right? Would that mean that it can't be related to any scripting going on, since they're all limited to in-game?

If the reproduction steps are not actually reproduction steps, then we can't figure it out more and more.
It would certainly be weird if you set up a RootMotionTrack with no RESET animation, then run Reset Bone Pose, and the offset still occurs after that.
It would help if you could send a video or something of when it occurs. There must be some kind of trigger there, so please update if you find out what it is.

I think it triggering on reimport seems like a likely possibility.
I'll try and catch it when it occurs and report back.

Oh-ho!
I rearranged the Node Tree where the character model is, and Godot threw this error for every Animation Tree I have in the scene that the character model uses:

Found invalid node path '_Armature/Skeleton3D:_Root_Motion' on node 'Root/PlayerBase/LocomotionControl/AnimationTree_WalkRunFall'

I don't know if they're relevant, but it also throws these errors:

core/object/undo_redo.cpp:147 - Condition "!p_callable.is_valid()" is true.

Node not found: "../../VISUAL/GeorunnerPlaceholder_SeparateCoat" (relative to "/root/@EditorNode@16886/@Panel@13/@VBoxContainer@14/DockHSplitLeftL/DockHSplitLeftR/DockHSplitMain/@VBoxContainer@25/DockVSplitCenter/@VSplitContainer@52/@VBoxContainer@53/@PanelContainer@98/MainScreen/@CanvasItemEditor@9272/@VSplitContainer@9094/@HSplitContainer@9096/@HSplitContainer@9098/@Control@9099/@SubViewportContainer@9100/@SubViewport@9101/Root/ActionSystem/R2/Action_DodgeDash/@AnimationPlayer@237519").

(Since 4.2.2, it's been throwing a lot of strange Editor errors that occur in the Editor's source code, but that's tangential.)
The Animation Trees still have the correct bone and path set in the Root Motion Track when I go to check, but this seems to be 'fake', as it were, because if I click on any of the Animation Trees, the issue occurs and the model is moved- everything I'd previously described happens.
And this happens until I go to the Root Motion Track section of each Animation Tree and re-select the Root Motion Track, though I still have to make the character model go back to its origin point manually (The Reset All Bone Poses thing was very helpful for that, thank you)
I'd imagine the same thing can happen when I re-import the model: it seems like the Animation Trees are losing track of the Root Motion Track when it's moved or refreshed in some way.


Autocompletion is not smart enough to handle step's "parameters" behavior

When '$username' clicks on '$button_id' button
should be proposed on:

When 'Bob
When 'Bob' clicks
When 'Bob' clicks on 'Send
When 'Bob' clicks on 'Send' butt

I'll use an algorithm that will sort the suggestion by the weight of the solution. Algorithm is available but not yet commited (i need time for all those pending tasks...)
The weight of the solution will be the sum of the fragment that the input fullfills within an entry:
Fragments are either literal or regex. The fragments for the sample are then:

When ': note the quote at the end, that is not part of the next fragment which is regex)

regex: bound to the parameter username

`' clicks on ': note again the quote on start and end

regex bound to the parameter button_id

' button

Wh weight: 1 the input fullfills the start of the first fragment

When ' weight: 2, the input fullfills completely the first fragment

When 'Bob weight: 3, the input fullfills completely the first fragment, and matches regex

When 'Bob' weight: 4, the input fullfills completely the first fragment, matches regex and fullfills the start of the 3rd fragment

When 'Bob' clicks o weight 4

When 'Bob' clicks on ' weight 5, the 3rd fragment is completely fullfills

When 'Bob' clicks on 'enter weight 6

When 'Bob' clicks on 'enter' bu weight 7

When 'Bob' clicks on 'enter' button weight 8

Fixed in master branch
Available on next christmas 2011 release: 1.0.4


Support escapes and proper numbers

The current parser doesn't properly support all valid escapes and also doesn't support all valid floating point numbers:

officially only ", /, \ and the letters b, f, n, r, t, u are allowed for escapes. But perhaps you don't mind being more flexible than the standard, but
the parser needs to support \uxxxx to be able to accept all valid JSON documents, and
numbers without a . can also be valid numbers, eg. 5e10 is a valid number (and would have to be parsed as a Double).

It's probably a good idea to take a look at syntax diagram here: https://www.json.org/json-en.html
There is also a test suite that's very useful: https://www.json.org/JSON_checker (at the bottom where it says "test suite")

Hey there,
First of all: thanks for taking an interest in JsonWheel! First issue not raised by me. ;)

Hehe, I'm one of the jbang devs, that's how I found about it.

officially only ", /, \ and the letters b, f, n, r, t, u are allowed for escapes. But perhaps you don't mind being more flexible than the standard, but

Yes, to put it shortly: I have no problem being more lenient than the spec if it saves some lines. That's why my impl is very bare ones right now. But I'll look into it!

Yeah, that's what I thought, perfectly reasonable.

the parser needs to support \uxxxx to be able to accept all valid JSON documents, and

Same: Let me look into that and see if I can fit it in. Seems like it's a common case (I suppose this is about non-ASCII chars).

Well this would mean that there's a certain class of valid documents that you wouldn't be able to parse. But that might be okay for your use-case of course. :slightly_smiling_face:

numbers without a . can also be valid numbers, eg. 5e10 is a valid number (and would have to be parsed as a Double).

There's already #2 for that. I'm not happy with the number parsing either atm. slightly_smiling_face

Ah, I hadn't seen that :+1:

There is also a test suite that's very useful: https://www.json.org/JSON_checker (at the bottom where it says "test suite")

Thank you, I was actually looking for something exactly like that. I referenced json.org many times, but didn't come across this test suite! Perfect!

Yeah, it's somewhat hidden. I only found it the first time because the simple-json code mentions it.

All that being said: Please understand that I am absolutely not aiming for 100% spec compliance. I want to cover the 80% mainly. If I can fit in more while staying below, let's say, 250 lines, I'll consider it.

Understood.

Best, Roman

Good luck!

FYI: I created #6 and #7 (in addition to #2), so one can track the progress of each of the items individually. I'm going to close here as I think all concerns will sufficiently be covered through those issues (at least those I want to address in the first place ;)).
Best,
Roman


SendPaymentV2 pays only from v0.11 to v0.11 but not pays from v0.11 to v0.12

I use SendPaymentV2 gRPC API call for rebalancing (i fill there payment_addr fields from invoice) but I found a strange thing. I cannot pay invoices from lnd-0.11.* created from LND 0.12 (0.12.0-beta.rc1 commit=v0.12.0-beta.rc1-18-gcc0321d1881ed23c9608cf898af2e2a7b347304a)
Here some examples from my logs with arguments data:
Invoice (lnd-21 & lnd-06 are lnd 0.11.* versions):
2021-01-18T19:13:53.829Z lnbig:paidRebalance invoice (lnd-21->lnd-06): { r_hash: <Buffer 2d 3a ab 8f 18 15 e7 10 0b f4 4e 39 d4 09 00 8f 2d 7a 74 5c cf 8e d2 4a 27 0b db 49 fc 36 6f 63>, payment_request: 'lnbc900220n1psqth0epp595a2hrcczhn3qzl5fcuagzgq3ukh5azue78dyj38p0d5nlpkda3sdru2fjkyctvv9hxxefqveex7mfqd3hxgtfjxys8gmeqd3hxgtfsxcsrjvpsxgezqumpw3ejqargwfhh2emgyqmrydpn8ymnyd3sxgurwwfkxuerxveqvd5xzmnwv4kqcqp7sp5gduntsntlfckghq48ws2sxrgf7mt5wv4y3a9qmx7mly2utrmpcgs9qy9qsqj7ctc2rje2kgzrrvhkmxewsxumte5wclr97z4s5ylsj5899zw8m8saepfl9nfcjtltkxv07mc0pudn707vl9f05nng2yrvnp49ywf8gqs3202w', add_index: '19260' } (команда { invoiceFrom: { key: 'lnd-06', pubKey: '03bc9337c7a28bb784d67742ebedd30a93bacdf7e4ca16436ef3798000242b2251' }, payWho: { key: 'lnd-21', pubKey: '02c91d6aa51aa940608b497b6beebcb1aec05be3c47704b682b3889424679ca490' }, amount: 90022, chanId: '624397260287967233', capacity: 16777215, blockHeight: 666645, decodedPayReq: { route_hints: [], features: { '9': [Object], '15': [Object], '17': [Object] }, destination: '03bc9337c7a28bb784d67742ebedd30a93bacdf7e4ca16436ef3798000242b2251', payment_hash: '2d3aab8f1815e7100bf44e39d409008f2d7a745ccf8ed24a270bdb49fc366f63', num_satoshis: '90022', timestamp: '1610997241', expiry: '3600', description: 'Rebalance from lnd-21 to lnd-06 90022 sats through<PHONE_NUMBER>87967233 channel', description_hash: '', fallback_addr: '', cltv_expiry: '30', payment_addr: <Buffer 43 79 35 c2 6b fa 71 64 5c 15 3b a0 a8 18 68 4f b6 ba 39 95 24 7a 50 6c de df c8 ae 2c 7b 0e 11>, num_msat: '90022000' } })

Result from SendPaymentV2:
2021-01-18T19:13:55.013Z lnbig:paidRebalance payment update (lnd-21->lnd-06) of channel, status=SUCCEEDED, data: { htlcs: [ { status: 'SUCCEEDED', route: [Object], attempt_time_ns: '1610997237438284135', resolve_time_ns: '1610997238517741562', failure: null, preimage: <Buffer c1 63 04 65 5f 8e 33 f7 af 1b fb 6b 44 0c c3 8c 7f cd 8b 93 fd 45 e2 ed 73 8d 4e cc 17 85 d6 a2> } ], payment_hash: '2d3aab8f1815e7100bf44e39d409008f2d7a745ccf8ed24a270bdb49fc366f63', value: '90022', creation_date: '1610997237', fee: '0', payment_preimage: 'c16304655f8e33f7af1bfb6b440cc38c7fcd8b93fd45e2ed738d4ecc1785d6a2', value_sat: '90022', value_msat: '90022000', payment_request: '', status: 'SUCCEEDED', fee_sat: '0', fee_msat: '0', creation_time_ns: '1610997237348176180', payment_index: '93601', failure_reason: 'FAILURE_REASON_NONE' }

And lnd 0.12.* exampes:
Invoice from lnd-10 (0.12.0-beta.rc1 commit=v0.12.0-beta.rc1-18-gcc0321d1881ed23c9608cf898af2e2a7b347304a):
2021-01-18T19:13:55.365Z lnbig:paidRebalance invoice (lnd-03->lnd-10): { r_hash: <Buffer 44 f0 05 fa 7e 2c 3e 9b 1c 7a 2f 12 db 67 5b 08 e2 f7 43 50 e6 f9 6a bd ab 57 75 99 66 a5 ff 5d>, payment_request: 'lnbc42949670n1psqth05pp5gncqt7n79slfk8r69ufdke6mpr30ws6sumuk40dt2a6eje49lawsdrl2fjkyctvv9hxxefqveex7mfqd3hxgtfsxvs8gmeqd3hxgtf3xqsrgv3exsunvdeqwdshgueqw358ymm4va5zqd3jxvmrwwfjxuunyvecx56r2dps8ysxx6rpdehx2mqcqp7sp5hzztmrxhkk52wjq3ll6vvqvjwwd5j349h756dv028vdxwtf83s3q9qyyssquyqgakm5lzv7syycwcnhhhrdyfm4sp8ds8zpg29dwqy6zy0km5zpnz9hpjd8mrwclpfal2v78zpjckjdvxde70j7wvaj8jzfh3g25pcq2qc9hv', add_index: '299537' } (команда { invoiceFrom: { key: 'lnd-10', pubKey: '02bb24da3d0fb0793f4918c7599f973cc402f0912ec3fb530470f1fc08bdd6ecb5' }, payWho: { key: 'lnd-03', pubKey: '032679fec1213e5b0a23e066c019d7b991b95c6e4d28806b9ebd1362f9e32775cf' }, amount: 4294967, chanId: '623679279238545409', capacity: 16777215, blockHeight: 666645, decodedPayReq: { route_hints: [], features: { '9': [Object], '14': [Object], '17': [Object] }, destination: '02bb24da3d0fb0793f4918c7599f973cc402f0912ec3fb530470f1fc08bdd6ecb5', payment_hash: '44f005fa7e2c3e9b1c7a2f12db675b08e2f74350e6f96abdab57759966a5ff5d', num_satoshis: '4294967', timestamp: '1610997236', expiry: '3600', description: 'Rebalance from lnd-03 to lnd-10 4294967 sats through<PHONE_NUMBER>38545409 channel', description_hash: '', fallback_addr: '', cltv_expiry: '30', payment_addr: <Buffer b8 84 bd 8c d7 b5 a8 a7 48 11 ff f4 c6 01 92 73 9b 49 46 a5 bf a9 a6 b1 ea 3b 1a 67 2d 27 8c 22>, num_msat: '4294967000' } })

And response from lnd-03 (v0.11.*):
2021-01-18T19:13:55.637Z lnbig:paidRebalance payment update (lnd-03->lnd-10) of channel, status=FAILED, data: { htlcs: [ { status: 'FAILED', route: [Object], attempt_time_ns: '1610997235414548090', resolve_time_ns: '1610997235614145813', failure: [Object], preimage: <Buffer > } ], payment_hash: '44f005fa7e2c3e9b1c7a2f12db675b08e2f74350e6f96abdab57759966a5ff5d', value: '4294967', creation_date: '1610997235', fee: '0', payment_preimage: '0000000000000000000000000000000000000000000000000000000000000000', value_sat: '4294967', value_msat: '4294967000', payment_request: '', status: 'FAILED', fee_sat: '0', fee_msat: '0', creation_time_ns: '1610997235369048375', payment_index: '69225', failure_reason: 'FAILURE_REASON_INCORRECT_PAYMENT_DETAILS' }

For paying i use a following code (nodejs):
                let req = {
                    dest: Buffer.from(command.decodedPayReq.destination, 'hex'),
                    amt: command.decodedPayReq.num_satoshis,
                    payment_hash: Buffer.from(command.decodedPayReq.payment_hash, 'hex'),
                    final_cltv_delta: command.decodedPayReq.cltv_expiry,
                    payment_addr: command.decodedPayReq.payment_addr,
                    fee_limit_sat: 0,
                    outgoing_chan_ids: [command.chanId],
                    timeout_seconds: 9,
                }

                debug("sendPaymentV2 (%s->%s) req: %o", command.payWho.key, command.invoiceFrom.key, req)

                let stream = nodeStorage.nodes[command.payWho.key].client.Router.sendPaymentV2(req)

Is it some bug of lnd 0.12.* or I do something wrong?

Can you re-phrase your scenario?
Is it that: you can't pay 0.12 from 0.11 (using your script)?

Can you re-phrase your scenario?
Is it that: you can't pay 0.12 from 0.11 (using your script)?

If you instead use the entire payment request (as we recommend to do so, and in your script it appears you have it available), does your scenario work as intended?

If you instead use the entire payment request (as we recommend to do so, and in your script it appears you have it available), does your scenario work as intended?

@LNBIG-COM perhaps try setting the dest_features in the request?
Specifically with FEATURE_BIT_MPP_OPT and FEATURE_BIT_TLV_ONION_OPT (from https://github.com/lightningnetwork/lnd/blob/master/lnrpc/rpc.proto#L3266)

@LNBIG-COM perhaps try setting the dest_features in the request?
Specifically with FEATURE_BIT_MPP_OPT and FEATURE_BIT_TLV_ONION_OPT (from https://github.com/lightningnetwork/lnd/blob/master/lnrpc/rpc.proto#L3266)

If you're paying from lnd 0.11 -> 0.12, then you need to use the full payment request as 0.11 doesn't have that field (payment_addr).

If you're paying from lnd 0.11 -> 0.12, then you need to use the full payment request as 0.11 doesn't have that field (payment_addr).

@LNBIG-COM perhaps try setting the dest_features in the request?
Specifically with FEATURE_BIT_MPP_OPT and FEATURE_BIT_TLV_ONION_OPT (from https://github.com/lightningnetwork/lnd/blob/master/lnrpc/rpc.proto#L3266)

I didn't it.
I did the same as was recommended by @Roasbeef in https://github.com/lightningnetwork/lnd/issues/4900#issuecomment-758376751:

if you were makign payments by manually specifying the details (destination, pay hash, etc), then you should switch to either generating and paying full invoices, or specifying the new payment_addr field in the existing APIs

But now i see in the sources of LND v0.11.1-beta.rc5 and i see there the method SendPaymentV2 but in the SendPaymentRequest message I don't see payment_addr in proto file. I saw in API doc at the site and I thought that if lnd has the SendPaymentV2 that it has payment_addr...
In this case, I still don't understand even more - how can I pay invoices for lnd 0.12 from lnd 0.11 without using a full request (I want to thus control the sending process more, because I need to send the payment only through certain channels by outgoing_chan_ids).

If you instead use the entire payment request (as we recommend to do so, and in your script it appears you have it available), does your scenario work as intended?

I didn't use the entire payment request. I wanted to try your hint #3 to use payment_addr from there: https://github.com/lightningnetwork/lnd/issues/4900#issuecomment-758376751

@LNBIG-COM perhaps try setting the dest_features in the request?
Specifically with FEATURE_BIT_MPP_OPT and FEATURE_BIT_TLV_ONION_OPT (from https://github.com/lightningnetwork/lnd/blob/master/lnrpc/rpc.proto#L3266)

I didn't it.
I did the same as was recommended by @Roasbeef in https://github.com/lightningnetwork/lnd/issues/4900#issuecomment-758376751:

if you were makign payments by manually specifying the details (destination, pay hash, etc), then you should switch to either generating and paying full invoices, or specifying the new payment_addr field in the existing APIs

But now i see in the sources of LND v0.11.1-beta.rc5 and i see there the method SendPaymentV2 but in the SendPaymentRequest message I don't see payment_addr in proto file. I saw in API doc at the site and I thought that if lnd has the SendPaymentV2 that it has payment_addr...
In this case, I still don't understand even more - how can I pay invoices for lnd 0.12 from lnd 0.11 without using a full request (I want to thus control the sending process more, because I need to send the payment only through certain channels by outgoing_chan_ids).

If you instead use the entire payment request (as we recommend to do so, and in your script it appears you have it available), does your scenario work as intended?

I didn't use the entire payment request. I wanted to try your hint #3 to use payment_addr from there: https://github.com/lightningnetwork/lnd/issues/4900#issuecomment-758376751

If you're paying from lnd 0.11 -> 0.12, then you need to use the full payment request as 0.11 doesn't have that field (payment_addr). If you switch to using the invoice, then the same script should work for all versions of lnd released in the past year or so.

If I use a full payment request, but at the same time indicate certain channels (outgoing_chan_ids) for outgoing payment - will this be taken into account internally?

If you're paying from lnd 0.11 -> 0.12, then you need to use the full payment request as 0.11 doesn't have that field (payment_addr). If you switch to using the invoice, then the same script should work for all versions of lnd released in the past year or so.

If I use a full payment request, but at the same time indicate certain channels (outgoing_chan_ids) for outgoing payment - will this be taken into account internally?

Now I started using the full payment request. So far, invoices have been paid without problems. I hope there are no errors and everything works as it should. I propose to correct the documentation - indicate that the payment_addr field does not work in versions before v0.12
                let req = {
                    payment_request: res.payment_request,
                    fee_limit_sat: 0,
                    outgoing_chan_ids: [command.chanId],
                    timeout_seconds: 9,
                }

Now I started using the full payment request. So far, invoices have been paid without problems. I hope there are no errors and everything works as it should. I propose to correct the documentation - indicate that the payment_addr field does not work in versions before v0.12
                let req = {
                    payment_request: res.payment_request,
                    fee_limit_sat: 0,
                    outgoing_chan_ids: [command.chanId],
                    timeout_seconds: 9,
                }


operations changes for v0.7 release

every commit here has been previously reviewed as part of a PR that merged it to this staging branch. it just needs a rubber stamp.


Auto-attach (or exclude from auto-attach) based on pattern

I have autocmds for SOPS files that automatically decrypt them and update the buffer with decrypted contents, then re-encrypt before writing
example (implementation of Sops.decrypt is elided for brevity):
vim.api.nvim_create_autocmd({'BufReadPost', 'BufWritePost'}, {
    pattern = {'*/secrets/secrets.yaml', '*/secrets/secrets.json', '*/secrets/secrets.env', '*/secrets/secrets.ini'},
    callback = function() Sops.decrypt() end,
})

This is great for working with SOPS files, but gitsigns is showing signs in the gutter based on the diff between the buffer (cleartext) and the index (encrypted), making the signs rather useless and potentially misleading. Worse, the commands to stage chunks are a vector for leaking cleartext versions of the file.
It would be nice if I could exclude certain files from the auto_attach feature based on pattern in a similar way.
As a workaround, I imagine it would be fairly straightforward to create an autocmd calling Gitsigns detach for files matching the pattern. I haven't tested every event, but it seems like attaching happens after the BufReadPost event, so I wasn't sure what event to tie this autcmd to.
(Tangentially, I think it would be possible to actually make the signs be correct, using an approach like the one outlined here in the SOPS repo that allows git diffs to show the diffs in cleartext. But that seemed like it was both more complex and less potentially useful as a general feature.)

There's already two methods:
require('gitsigns').setup { auto_attach = false }

vim.api.nvim_create_autocmd({ 'BufFilePost', 'BufRead', 'BufNewFile' }, {
  callback = function()
    if vim.bo[bufnr].filetype == 'foo' then
      return -- do not attach
    end
    require('gitsigns').attach()
  end
})

or
require('gitsigns').setup {
  on_attach = function(bufnr)
    if vim.bo[bufnr].filetype == 'foo' then
      return false -- do not attach
    end
  end
}

(Tangentially, I think it would be possible to actually make the signs be correct, using an approach like the one outlined here in the SOPS repo that allows git diffs to show the diffs in cleartext. But that seemed like it was both more complex and less potentially useful as a general feature.)

This relates to #480 where git allows you to define smudge/clean filters in .gitattributes which people user to encode/decode files stored in git.


create-platformatic does not create the Platformatic Cloud GH actions

...even if we ask to do so:

This is fixed in #1669


TF1 support broken

Hi
URL Header of TF1 replay video has been changed recently, could you help to reactivate ? Thanks
youtube-dl http://lci.tf1.fr/sept-a-huit/videos/sept-a-huit-du-24-mai-2015-8611550.html -v
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'http://lci.tf1.fr/sept-a-huit/videos/sept-a-huit-du-24-mai-2015-8611550.html', u'-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.05.20
[debug] Python version 2.7.6 - Darwin-14.3.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 2.2.2, rtmpdump 2.4
[debug] Proxy map: {}
[generic] sept-a-huit-du-24-mai-2015-8611550: Requesting header
WARNING: Falling back on generic information extractor.
[generic] sept-a-huit-du-24-mai-2015-8611550: Downloading webpage
[generic] sept-a-huit-du-24-mai-2015-8611550: Extracting information
ERROR: Unsupported URL: http://lci.tf1.fr/sept-a-huit/videos/sept-a-huit-du-24-mai-2015-8611550.html
Traceback (most recent call last):
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 969, in _real_extract
doc = parse_xml(webpage)
File "/usr/local/bin/youtube-dl/youtube_dl/utils.py", line 1558, in parse_xml
tree = xml.etree.ElementTree.XML(s.encode('utf-8'), **kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1300, in XML
parser.feed(text)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1642, in feed
self._raiseerror(v)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
raise err
ParseError: undefined entity: line 9, column 138
Traceback (most recent call last):
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 649, in extract_info
ie_result = ie.extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 273, in extract
return self._real_extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 1504, in _real_extract
raise UnsupportedError(url)
UnsupportedError: Unsupported URL: http://lci.tf1.fr/sept-a-huit/videos/sept-a-huit-du-24-mai-2015-8611550.html

Wil be fixed in the next version. Thanks for the report.

thx vm

On May 30, 2015, at 7:19 PM, Sergey M<EMAIL_ADDRESS>wrote:
Wil be fixed in the next version. Thanks for the report.
—
Reply to this email directly or view it on GitHub https://github.com/rg3/youtube-dl/issues/5848#issuecomment-107018970.


Documentation: Automatize screenshot generation

We could use the screenshot functionality of playwright to automatically generate screenshots. Currently they get outdated rather fast and need to be fixed manually.

C'est la vie. I don't think Playwright is the right tool to ensure that our documentation stays up-to-date.


Make sure links in source do not hardcode a locale

No links should have a hard-coded language, i.e.:
        /// For more information, see <see href="https://docs.microsoft.com/rest/api/storageservices/creating-a-snapshot-of-a-blob">

Please remove "en-us/" from this and other links within this PR so that users are taken to the preferred language of their browser/OS.
@scbedd should DocWarden be catching these?
Originally posted by @heaths in https://github.com/Azure/azure-sdk-for-net/pull/17825#discussion_r575557453

These are caught during CI as part of analysis now.   Closing this out.   @heaths - if I'm overlooking some nuance, please feel free to reopen.

It's been a while since I opened this, but I think it was meant (and I should've been more clear) to check all files. That said, I had also more recently opened Azure/azure-sdk-tools#6107 that should cover that.


dnscrypt-wrapper: log level of "Short query received" changed to debug

Hi Frank,
today I've noticed that there are a lot of dnscrypt-wrapper messages

[warning] [udp_request.c:302] Short query received

in the Docker log.
As you use dnscrypt-wrapper in your Docker image, can you check if you can integrate the approved and merged https://github.com/cofyc/dnscrypt-wrapper/pull/165 in your own jedisct1/dnscrypt-wrapper (branch xchacha-stamps)?

Sure!

Done!


fix(): serveStaticOptions is not correctly passed to fastify-static

PR Checklist
Please check if your PR fulfills the following requirements:

[x] The commit message follows our guidelines: https://github.com/nestjs/nest/blob/master/CONTRIBUTING.md
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Other... Please describe:

What is the current behavior?

Issue Number: https://github.com/nestjs/serve-static/issues/187
What is the new behavior?
Does this PR introduce a breaking change?
[ ] Yes
[x] No

Other information

ping @kamilmysliwiec

Hi @kamilmysliwiec , does anyone care to have a look on this, thanks!

Ping @kamilmysliwiec

Thanks!


fix(docs): Move reporting telemetry section out of metadata-ingestion/source_docs.

Reporting telemetry is not a source-specific feature, and is configurable for any pipeline. Refactor the documentation out.
Checklist

[x] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[ ] Links to related issues (if applicable)
[ ] Tests for the changes have been added/updated (if applicable)
[ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Since docs have been overhauled using a different PR, closing this one as it is no longer applicable.


Cannot import a project that uses android-sdk-plugin

Even though I have set in my build.sbt the sdkPath variable:
sdkPath in Global := "/home/user/bin/android-sdk"

and set the variable ANDROID_HOME in .bashrc, I cannot import the project in IDEA.
[error] set the env variable ANDROID_HOME pointing to your Android SDK

Related bug that I have opened in IDEA:
https://youtrack.jetbrains.com/issue/SCL-9836
But I can launch sbt from the command line and it works.

remove in Global
On Tue, Feb 2, 2016, 8:58 AM David Pérez Carmona<EMAIL_ADDRESS>wrote:

Even though I have set in my buildsbt the sdkPath variable:
sdkPath in Global := "/home/user/bin/android-sdk"
and set the variable ANDROID_HOME in bashrc, I cannot import the project
in IDEA
[error] set the env variable ANDROID_HOME pointing to your Android SDK
Related bug that I have opened in IDEA:
https://youtrackjetbrainscom/issue/SCL-9836
But I can launch sbt from the command line and it works
—
Reply to this email directly or view it on GitHub
https://github.com/pfn/android-sdk-plugin/issues/248.

I've tried also:
sdkPath in android := "....."

You must also launch the ide after setting the variable. If you launch from
a desktop environment, then you must log in again, and as far as I know, a
desktop environment does not read bashrc.
On Tue, Feb 2, 2016, 8:59 AM Perry Nguyen<EMAIL_ADDRESS>wrote:

remove in Global
On Tue, Feb 2, 2016, 8:58 AM David Pérez Carmona<EMAIL_ADDRESS>wrote:

Even though I have set in my buildsbt the sdkPath variable:
sdkPath in Global := "/home/user/bin/android-sdk"
and set the variable ANDROID_HOME in bashrc, I cannot import the project
in IDEA
[error] set the env variable ANDROID_HOME pointing to your Android SDK
Related bug that I have opened in IDEA:
https://youtrackjetbrainscom/issue/SCL-9836
But I can launch sbt from the command line and it works
—
Reply to this email directly or view it on GitHub
https://github.com/pfn/android-sdk-plugin/issues/248.

I've tried also to launch from the command line, where I know the value of each environment variable.

Not an issue, use gitter for chatty support

OK

Cause of the problem was I mispelled the value of ANDROID_HOME in the idea.sh script, and I didn't remember about it.


Docs/admin customization 'extracss' block doesn't exist

The admin customization section of the docs refers to a block extracss that doesn't seem to exist. It looks like it should refer to the extrastyle block instead.
http://docs.viewflow.io/admin_customization.html#block-extracss

Thank you for pointing that.
Should be fixed right now.


v2, component updates not propagated in some cases

Ok so I've been pulling my hair out trying to track this error down in my app, and I finally decided to create an isolated repository that reproduces the issue(s).
I'm not sure if this is a baobab issue, or a baobab-react issue.
Please pull this repository down (it's a very very simple app that uses baobab). Instructions to run it are in the readme.
https://github.com/marbemac/baobab-playground
Steps to reproduce:

Type in a username
Click submit, error should show up
Try and change the username. The baobab tree is correctly updated (type state.get() in the web developer console), but the components (ui) are not updated.

This seems to happen when a parent component listens to the same cursor as a child component? If you remove the "session" cursor from components/App.jsx, everything will behave as expected. I'm not sure what the underlying issue(s) is with baobab/baobab-react.

Ok @marbemac. Will check this tomorrow morning.

Awesome, thank you :).

I think this could be the issue I've been experiencing as well.

Ok, I got it. The fault lies with baobab-react componentWillReceiveProps. Fixing this asap.

Ok, the issue seems to be fixed now. I released<EMAIL_ADDRESS>and<EMAIL_ADDRESS>to tackle this. Beware however for the latest baobab-react dropped cursor access for components as it is enforcing bad practices.

Awesome, thank you, a whole host of weird behavior in my app is now fixed :)!
Yes, cursors in components = bad - I used them in that playground example simply because it was faster to implement. Thanks again!

I will probably add a lightweight action layer in baobab-react soon so we can experiment things (in a hot reloadable way of course).

Thanks @Yomguithereal and @marbemac! I believe this fixed some of the strange issues I've been seeing.


test: refactor mempool_node test_valid_config towards refactor

This change is

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#116
#115  👈
#114
#113
#111
#110
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @ArniStarkware and the rest of your teammates on  Graphite

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 44.68%.  Consider uploading reports for the commit 67f5208 to get more accurate results

Additional details and impacted files
@@                      Coverage Diff                       @@
##           arni/config/test_utils_folder     #115   +/-   ##
==============================================================
  Coverage                          44.68%   44.68%
==============================================================
  Files                                 17       17
  Lines                                611      611
  Branches                             611      611
==============================================================
  Hits                                 273      273
  Misses                               328      328
  Partials                              10       10

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Remove need for KEYCHAIN_PASSWORD secret in deploying-xcode-applications

Why:
Generating the ephemeral keychain's password in the script removes the need for the user to generate and store one manually.

What's being changed (if available, include any code snippets, screenshots, or gifs):
Step describing KEYCHAIN_PASSWORD secret is removed. Script modified to generate random bytes to use as the keychain's password automatically (openssl rand – but it could be $RANDOM).

Check off the following:

[x] I have reviewed my changes in staging, available via the View deployment link in this PR's timeline (this link will be available after opening the PR).

For content changes, you will also see an automatically generated comment with links directly to pages you've modified. The comment won't appear if your PR only edits files in the data directory.

[x] For content changes, I have completed the self-review checklist.

Hey @cweider, thanks so much for opening this PR to update the docs! ✨
The reason we recommend users create a secret and pass the secret to the workflow is because it is more secure than generating the password via the command line. You can see this documented in Using secrets in a workflow with a little more information on why we make that recommendation.
While your solution is simpler to implement 👍, it is less secure. For that reason, I am going to close out this PR. Thanks again for your interest in improving GitHub's docs!


es5 code depends on dependency with arrow function (clone-regexp)

Hi, I am probably doing something wrong, but I tried to use this (great library) in a fresh Angular 7 project. TS compiler target is es5.
I tried
import { asyncFilter } from "iter-tools/es5"; or just
import { asyncFilter } from "iter-tools";
IE11 shows a syntax error in clone-regexp. Maybe this is a problem with Angular's Typescript setup?

Seems like a pretty straightforward issue. I opened https://github.com/sindresorhus/clone-regexp/pull/4 for you.

As a workaround you can import asyncFilter's module directly:
import asyncFilter from "iter-tools/es5/async-filter";

clone-regexp has been designed for working with node (es2015). I don't see the point in setting the transpilation of external deps for a 15 lines module. I got rid of it.
Version 6.1.5 should be ok @sladiri