[RFC] harden/improve rabit framework

Over the course of last half, I have been working on some of largest xgboost jobs.
Often we face some of weird issues while running thousands of workers over billion rows of training data-set. Minor set of PRs submitted https://github.com/dmlc/dmlc-core/pull/512 https://github.com/dmlc/rabit/pull/81 https://github.com/dmlc/rabit/pull/73
As distributed training moves into production with hundreds of production models running on top of XGB, this lead to the need of harden rabit framework with better code layout/documentation as well as improved function test coverage. Here is list of work I am planning to execute and request for feedbacks from community.

increase test coverages with split of _robust implementations, add test coverage and harden on ring based restore (haven't able to find where that code path were used). There should be a waterfall flow where filesystem based checkpoint and job restart will be executed only when ring based recovery can't  restore.

switch distributed xgb checkpoint restore from external checkpoint(HDFS/S3) to peer restore. Redistribute training dataset can take 30 - 40 mins pulling from HDFS, restore single worker failure (increase with cluster getting larger) without restart entire training stage can save significant amount of time. This is particularly interesting while workers running on top of preemptive instances. If small percentage of tasks might get rescheduled due to executor host get preempted, whether job can restore fast. AFAIK, not of other MPI frameworks has build such recovery mechanism to run on top of preemptive instances.

more MPI features, primitive metrics and profiling capability. Investigate job failure can be burdensome, primitive metrics report aggregation could improve user experience a lot.

Thanks,
Chen

@chenqin Hi, first of all, I'd like to say thank you. Your work in distributed training has been instrumental in brining XGBoost to industrial-grade standard.
I'd like to receives your suggestions about improving the CI infrastructure, i.e. how we should test distributed training. See discussion at #4234.
For example, we should maybe test dmlc/rabit together in dmlc/xgboost, so that no future change of dmlc/rabit would break distributed XGBoost.

Thanks for all the work @chenqin ! Regarding rabbit's future, I was also wondering if it would be possible to introduce a scatter-reduce algorithm like LightGBM uses.
The benefits when user 2^x workers seem to be significant.
For job restores, what is the state that we need to be restoring? I'm not familiar with the term peer restore, does that mean communicating data and models from other nodes instead of a single node reading from HDFS?
My assumption is that if a worker fails and we launch a new one, the new one would need to first read its data partition into memory from a distributed FS, then read the current model version as well.
Am I missing something here?
Do you think there are ideas we can use from Flink/Spark job recovery?

I'd like to receives your suggestions about improving the CI infrastructure, i.e. how we should test distributed training. See discussion at #4234.

Sounds good, split build/test , adding OSX, LINUX, WINDOWS should be straight forward.

For example, we should maybe test dmlc/rabit together in dmlc/xgboost, so that no future change of dmlc/rabit would break distributed XGBoost.

In term of distributed XGBoost, we can start with including XGBoost4j-Spark tests as start point. Possibly next adding simulate rabit failure tests using mock engine to verify item 2)

Thanks for all the work @chenqin ! Regarding rabbit's future, I was also wondering if it would be possible to introduce a scatter-reduce algorithm like LightGBM uses.

yes, that should be item 3) once we can verify framework failure recovery works as expected. We can add more functions with good failure resilient recovery support.

The benefits when user 2^x workers seem to be significant.
For job restores, what is the state that we need to be restoring? I'm not familiar with the term peer restore, does that mean communicating data and models from other nodes instead of a single node reading from HDFS?
My assumption is that if a worker fails and we launch a new one, the new one would need to first read its data partition into memory from a distributed FS, then read the current model version as well.
Am I missing something here?
That's mostly correct to start a job. item 2) is mostly around the case when you have single worker failure. How distributed XGBoost deal with entire worker fleet. What we have is to drop all works since last checkpoint and restart job.(entire load data -> restore checkpoint -> continue train)  What item 2) trying to solve is only restore worker that lost (load only partial data assigned to lost worker(s)-> restore last checkpoint from other peers ->catch up on next allreduce) The catch is there will be situation where XGBoost lost too many workers in tree-reduce (most likely) topology, so we need to basically fallback to what we have right now.

Do you think there are ideas we can use from Flink/Spark job recovery?
AFAIK underlaying Flink/Spark is continuous processing and map-reduce paradigms. In map-reduce, we can retry mapper in case they lost, sync (barrier) happens after a map stage complete. In continuous processing, the sync might never happens hence flink introduced control messages to coordinate workers with different roles.

In distributed XGBoost, global sync(ALLREDUCE/BRAODCAST) happens frequently to update histogram on fixed topology (network topology is fixed after all workers registered in tracker). AllReduce -> BroadCast->AllReduce ... Unless we could prove in math we can upper bound error ratio based on partial histogram, rabit still have to block till MPI func complete each operation.

What item 2) trying to solve is only restore worker that lost (load only partial data assigned to lost worker(s)-> restore last checkpoint from other peers ->catch up on next allreduce)

OK if I understand correctly then when a worker fails in this scenario, the others continue with the iteration while we launch a new one and it reads its data partition. We complete the current iteration using the limited information from the live workers, and in the next iteration, the new worker gets the model state and we continue as before.

Unless we could prove in math we can upper bound error ratio based on partial histogram, rabit still have to block till MPI func complete each operation.

OK I see what you are saying. We have discussed early termination of histogram creation/communication with @hcho3 in the past, where concentration bounds like the Hoeffding bound can be used to prove convergence, similar to the VFDT tree or this recent work  by @arapat and Freund (code here).

I'll try to go through this and find correspondences in the code, and hopefully annotate the code with comments to make it more readable.

What item 2) trying to solve is only restore worker that lost (load only partial data assigned to lost worker(s)-> restore last checkpoint from other peers ->catch up on next allreduce)

OK if I understand correctly then when a worker fails in this scenario, the others continue with the iteration while we launch a new one and it reads its data partition. We complete the current iteration using the limited information from the live workers, and in the next iteration, the new worker gets the model state and we continue as before.

I agree we should trace back to rabit paper and see how they designed failure recovery initially. At same time, binary tree wise tree reduce might not be most performant network topology considering time cost of {allreduce-> broadcast} e.g when we try to grow tree depth on large worker cluster total number of {allreduce-> broadcast} per iteration goes up exponentially (2^n) so as number of network hops (log (n) ). In theory, if we get infinite workers (ignorable build histogram time) and end of end overhead is small, total time to train a deeper tree increase stay linear with depth of tree. Throw extra hardware (never fail, never congest) in best case will not reduce time of getting a deeper tree. In reality, when spread cluster thin, failure and congestion probability just increase.

Unless we could prove in math we can upper bound error ratio based on partial histogram, rabit still have to block till MPI func complete each operation.

OK I see what you are saying. We have discussed early termination of histogram creation/communication with @hcho3 in the past, where concentration bounds like the Hoeffding bound can be used to prove convergence, similar to the VFDT tree or this recent work  by @arapat and Freund (code here).

Thanks! Let me dig in a bit and will come back to you latter.

Rabit2 design proposal
keyword FSM, EPOLL, C10K, data linerage, map-reduce, allreduce, fast recovery, large scale, MPI
https://github.com/dmlc/rabit/issues/89

@chenqin Is this task complete?

I think we can say hardening part is complete, improving is still undergoing.

Improvement
I have been working on enable native rabit checkpoint in YARN/SPARK deployment.
here is test I run to instrument failure and test recovery
https://github.com/chenqin/xgboost/blob/master/tests/cli/runsingle.sh
https://github.com/chenqin/xgboost/blob/master/tests/cli/runtests.sh
so far I identified two issues with current rabit based recovery in xgb.

saved checkpoints missed some configurations (e.g max_depth, dsplit) Although, it can be solved relative straightforward by adding those into payload.

DMatrix::Load calls allreduce before loadcheckpoint which failed recovery payload size check. It turns out to be a bit tricky to fix.

In order to construct a Partitioned DMatrix and build learner (which will get injected with loadcheckpoint) we need to sync number of columns in all workers before loadcheckpoint.
https://github.com/chenqin/xgboost/blob/master/src/cli_main.cc#L160

if we allow allreduce runs before loadcheckpoint happens, we will see https://github.com/dmlc/rabit/issues/63 in recovered worker. As we are looking for a u64 byte and get checkpoint payload much bigger  from other peers.
if we force user add config num_col_, user experience would be not optimal and tedious.

AFAIK, a simple way is to keep this configuration in tracker (each worker just call once when it runs or reruns) and expose via rabit.h with following changes https://github.com/chenqin/rabit/commit/11adddc83fa8ec0041b547fb2fe77721cc072eae
https://github.com/chenqin/dmlc-core/blob/master/tracker/dmlc_tracker/tracker.py#L271-L279
here is code change in XGB
https://github.com/chenqin/xgboost/blob/master/src/data/data.cc#L227-L248
then it pass failure recovery tests with rabit.
cc @hcho3 @trivialfis @CodingCat

@chenqin Thanks.  Honestly I don't quite understand what's happening ...  So I will stay out of this.

AFAIK, a simple way is to keep this configuration in tracker (each worker just call once when it runs or reruns) and expose via rabit.h with following changes chenqin/rabit@11adddc
https://github.com/chenqin/dmlc-core/blob/master/tracker/dmlc_tracker/tracker.py#L271-L279
here is code change in XGB
https://github.com/chenqin/xgboost/blob/master/src/data/data.cc#L227-L248
then it pass failure recovery tests with rabit.
cc @hcho3 @trivialfis @CodingCat

follow your github repo @chenqin ,   dmlc/rabit#63  is solved. but there is another issue when one worker stopped and restart. I add some logging, the error occured when gmat_.Init()
[09:56:15] INFO: /opt/xgboost/src/learner.cc:215: Tree method is selected to be 'hist', which uses a single updater grow_quantile_histmaker.
[09:56:15] INFO: /opt/xgboost/src/gbm/gbtree.cc:267: gbm:BoostNewTrees() start
[09:56:15] INFO: /opt/xgboost/src/gbm/gbtree.cc:255: GBM::InitUpdater(), pstr: 'grow_quantile_histmaker'
[09:56:15] INFO: /opt/xgboost/src/gbm/gbtree.cc:290: gbm:BoostNewTrees() call up->Update()
[09:56:15] INFO: /opt/xgboost/src/tree/updater_quantile_hist.cc:55: QuantileHistMaker::Update() start
[09:56:15] INFO: /opt/xgboost/src/tree/updater_quantile_hist.cc:58: QuantileHistMaker::Update() call gmat_.Init()
[09:56:16] INFO: /opt/xgboost/src/common/hist_util.cc:177: HIstCutMatrix::Init() call sreducer.Allreduce() start
[debug] allreduce_robust::TryGetResult(), data_size: 6678336, size: 342115888
Allreduce Recovered data size do not match the specification of function call.
Please check if calling sequence of recovered program is the same the original one in current VersionNumber

can you share tests case you were running?

follow your github repo @chenqin , dmlc/rabit#63 is solved. but there is another issue when one worker stopped and restart. I add some logging, the error occured when gmat_.Init()
[09:56:15] INFO: /opt/xgboost/src/learner.cc:215: Tree method is selected to be 'hist', which uses a single updater grow_quantile_histmaker.
[09:56:15] INFO: /opt/xgboost/src/gbm/gbtree.cc:267: gbm:BoostNewTrees() start
[09:56:15] INFO: /opt/xgboost/src/gbm/gbtree.cc:255: GBM::InitUpdater(), pstr: 'grow_quantile_histmaker'
[09:56:15] INFO: /opt/xgboost/src/gbm/gbtree.cc:290: gbm:BoostNewTrees() call up->Update()
[09:56:15] INFO: /opt/xgboost/src/tree/updater_quantile_hist.cc:55: QuantileHistMaker::Update() start
[09:56:15] INFO: /opt/xgboost/src/tree/updater_quantile_hist.cc:58: QuantileHistMaker::Update() call gmat_.Init()
[09:56:16] INFO: /opt/xgboost/src/common/hist_util.cc:177: HIstCutMatrix::Init() call sreducer.Allreduce() start
[debug] allreduce_robust::TryGetResult(), data_size: 6678336, size: 342115888
Allreduce Recovered data size do not match the specification of function call.
Please check if calling sequence of recovered program is the same the original one in current VersionNumber

my test as follow:

run a tracker, using https://github.com/chenqin/dmlc-core/blob/master/tracker/dmlc_tracker/tracker.py, as python tracker.py --num-workers 2

start 2 workers,  using https://github.com/dmlc/xgboost/blob/master/tests/distributed/test_basic.py

after training started for a few round, one worker killed. then restart the worker.
the conf:
{
  "silent": false,
  "n_estimators": 500,
  "max_depth": 5,
  "nthread": 10,
  "learning_rate": 0.07,
  "early_stopping_rounds": 50,
  "subsample": 0.8,
  "tree_method": "hist",
  "verbosity": 3,
  "eval_metric": "auc"
}

the above error was tree_method=hist.
but, with tree_method = approx,  it worked without any error.
so it seems there is still some problem with tree_method=hist

Thanks for sharing @wstian , will take a look!

the downside of third approach is we need to get this side car agent runs in same phase, which can be a bit tricky to get it right.

Thank you, @chenqin for all the work. We are experimenting with XGBoost on a large scale and I am very interested in working on making xgboost-spark training reliable.
I read this thread and I am a bit confused. As far as I understand xgboost-spark is not reliable. At least it has TaskFailedListener which kills the job if one of tasks fail. On another hand you have a test showing that distributed xgboost without Spark can recover from a node failure (great achievement!)
Do I understand correctly that distributed training in xgboost-spark is still not reliable (in 0.9)?
What should we do to make it reliable? It seems that the underneath tech support fault tolerance.
I would gladly hear your opinion.

@trams xgb-spark is reliable and able to do checkpoint restore, we daily runs large number of jobs with hundreds of containers and thousands of cores deployment. This doc is tracking works around rabit (worker synchronization layer). As we speak right now, XGBSpark works fine with remote checkpoint and restart / resume job.
The goal of improvement is to allow distributed xgb tolerate a few failures without restart/resume entire training job. It can also help moving larger xgb workers on cheaper preemptive instances which can only last a few iterations with much cheaper cost.

Thank you, @chenqin for all the work. We are experimenting with XGBoost on a large scale and I am very interested in working on making xgboost-spark training reliable.
I read this thread and I am a bit confused. As far as I understand xgboost-spark is not reliable. At least it has TaskFailedListener which kills the job if one of tasks fail. On another hand you have a test showing that distributed xgboost without Spark can recover from a node failure (great achievement!)
Do I understand correctly that distributed training in xgboost-spark is still not reliable (in 0.9)?
What should we do to make it reliable? It seems that the underneath tech support fault tolerance.
I would gladly hear your opinion.

@trams fail-all-and-make-a-checkpoint on any error is a very typical strategy for machine learning systems.
The very original reason to fail everything on any error in spark layer is that the rabit fault tolerance is a bit off here and there leading to the stuck training process. Fixing those "offs" is the topic in this thread. In future, we should provide two types of fault tolerance strategy, auto task recovery functionality from this thread, and checkpoint in spark (to handle situation that the whole job is down for some reason)

I am glad to hear about your plans on making distributed xgboost more reliable. And thank you for a quick response
As for checkpointing it works but https://github.com/dmlc/xgboost/issues/3946 prevents us to use to full extend

@wstian can you try my latest master and see if it can pass your previous tests? I can see it pass https://github.com/chenqin/xgboost/blob/master/tests/cli/runhist.sh

rabit one off recovery cache
https://github.com/dmlc/rabit/pull/98

@wstian can you try my latest master and see if it can pass your previous tests next week?
I can see it pass https://github.com/chenqin/xgboost/blob/master/tests/cli/runhist.sh

sorry for late response.
try with your latest master:

https://github.com/chenqin/xgboost.git @807ff491d666a93586bcb2aa1e4d6f1769fc402d

https://github.com/chenqin/rabit.git @ 87c90639f3594046697179fd8dd56ee8fbaba954

and repeat the previous tests, which is

start a distributed training, with worker-0 and worker-1

after data loaded and training started, shutdown worker-0

after tracker receive recover signal from worker-1, restart worker-0

after the shutdown worker-0 restarted,  worker-0 hit an error and exit.
[16:22:39] XGBoost distributed mode detected, will split data among workers
[16:22:39] Load part of data 0 of 2 parts
[16:22:44] 100017x10438 matrix with 330975794 entries loaded from train.200k.libsvm
Too many nodes went down and we cannot recover.., shutting down process

worker-1 also come across this error and exit.

@wstian I added more logging https://github.com/chenqin/rabit/commit/2bd993a8d0483ddb8a4661561cf2bfd16bcd8a91
please paste log from your side
The error you saw was might be a rabit existing rabit bug. in my test, it shows checkpoint collective calls sometimes unexpectedly interact with allreduce/broadcast calls in healthy nodes and cause peer to peer getresult lack of necessary roles to operate (KRequestData, KHaveData)

@wstian I added more logging chenqin/rabit@2bd993a
please paste log from your side
The error you saw might be a rabit long existing rabit bug. in my test, it means checkpoint/loadcheckpoint collective calls sometimes unexpectedly interact with allreduce/broadcast calls from healthy nodes. Which causes trygetresult fail due to lack of necessary roles to operate (KRequestData, KHaveData)

try with [chenqin/rabit@2bd993a]
worker-0 shut down when restart, log as below:
2019-07-09 09:54:45,298 - start-worker.py[line:92] - INFO: load train: train.200k.libsvm
[09:54:45] XGBoost distributed mode detected, will split data among workers
[09:54:45] Load part of data 0 of 2 parts
[09:54:50] 100017x10438 matrix with 330975794 entries loaded from train.200k.libsvm
[0] RecoverExec caller GetCache seq 0
[0] caller GetCache requester 1
[0] checkpoint req - |0|0|0|1|0| - |0|1|0|
[0] checkpoint ack - |0|1|0|1|1| - |33554432|0|1|
[0] role is 1 in trygetresult seqno 0 seq_counter 0
Allreduce Recovered data size do not match the specification of function call.
Please check if calling sequence of recovered program is the same the original one in current VersionNumber, shutting down process

worker-1  is still running (not shut down this time), log as below:
[1] RecoverExec caller CheckPoint_ seq 9
[1] caller CheckPoint_ requester 0
[1] checkpoint req - |33554432|1|0|0|0| - |33554432|0|0|
[1] checkpoint ack - |0|1|0|1|1| - |33554432|0|1|
[1] role is 0 in trygetresult seqno 0 seq_counter 9

@wstian quick look into log you posted, I think it was same as I saw before.
wha happens is worker-1 calls into checkpoint procedure while worker-0 is still in bootstrap/allreduce.
https://github.com/chenqin/rabit/blob/test/src/allreduce_robust.cc#L362
it force worker1 and worker0 into collective checkpoint state
https://github.com/chenqin/rabit/blob/test/src/allreduce_robust.cc#L1018
I guess the initial intention of this code is to see if any nodes are missing latest checkpoint payload before collective checkpointing. The problem you saw was that worker 0 caller give a buffer pointer and size that suppose to do allreduce/broadcast. When actual checkpoint payload recover starts. payload size mismatch.
working on fixing.

try chenqin/rabit@1e82c33
worker-0 shutdown with following log:
[0] RecoverExec caller LoadCheckPoint seq 0
[0] RecoverExec caller LoadCheckPoint seq 0
[0] RecoverExec caller LoadCheckPoint seq 0
[0] load checkpoint global 17351 version 13
[0] RecoverExec caller LoadCheckPoint seq 0
[0] RecoverExec caller Allreduce seq 0
[0] RecoverExec caller Allreduce seq 1
[0] RecoverExec caller CheckPoint_ seq 2
[0] RecoverExec caller CheckPoint_ seq 0
[0] RecoverExec caller GetCache seq 0
[0] RecoverExec caller GetCache seq 0
[0] RecoverExec caller Allreduce seq 0
[0] RecoverExec caller Allreduce seq 1
[0] send req - |1|0|0|0|0| - |4|0|0|
[0] recv ack - |-55050240|0|0|0|1| - |33716452|0|1|
[0] pre trygetresult seqno -55050240 requester 0
AssertError:likely minimal seqno overflow, shutting down process

try chenqin/rabit@1e82c33
worker-0 shutdown with following log:
[0] RecoverExec caller LoadCheckPoint seq 0
[0] RecoverExec caller LoadCheckPoint seq 0
[0] RecoverExec caller LoadCheckPoint seq 0
[0] load checkpoint global 17351 version 13
[0] RecoverExec caller LoadCheckPoint seq 0
[0] RecoverExec caller Allreduce seq 0
[0] RecoverExec caller Allreduce seq 1
[0] RecoverExec caller CheckPoint_ seq 2
[0] RecoverExec caller CheckPoint_ seq 0
[0] RecoverExec caller GetCache seq 0
[0] RecoverExec caller GetCache seq 0
[0] RecoverExec caller Allreduce seq 0
[0] RecoverExec caller Allreduce seq 1
[0] send req - |1|0|0|0|0| - |4|0|0|
[0] recv ack - |-55050240|0|0|0|1| - |33716452|0|1|
[0] pre trygetresult seqno -55050240 requester 0
AssertError:likely minimal seqno overflow, shutting down process

Thanks you quick response! This is also a known issue where I put an assert to avoid unexpected state
https://github.com/chenqin/rabit/blob/test/src/allreduce_robust.cc#L932
Could you also post log from worker-1 as well, -55050240 is really weird seqno

try chenqin/rabit@aa2153f
worker-0 shutdown, log as below:
[0] load checkpoint global 17351 version 13
[0] send req - |1|0|0|0|0| - |4|0|0|
[0] recv ack - |-55050240|0|0|0|1| - |33716452|0|1|
AssertError:likely minimal seqno overflow, shutting down process

worker-1 alive, waiting, but no other log.

humm, last try. adding assertion of negative seq no as  "recv ack - |-55050240|" as well as flag logging.
passing multiple run of https://github.com/chenqin/xgboost/blob/master/tests/cli/runxgbtests.sh
https://github.com/chenqin/rabit/commit/0a522a77829162f2dbea8c99487819b8b50b47c2

Let's document what is "expected" fault recovery behavior in current rabit design so we can have all on same page with reasonable expectation.

allreduce/broadcast has to happen between loadcheckpoint and checkpoint calls.
no bootstrap allreduce/broadcast is allowed.(only rerun iteration n allreduce/broadcast from last iteration checkpoint (iteration n-1) is supported)
in case too many nodes are down, it will recover from very beginning.

example of supported use case:

loadcheckpoint-0, allreduce-0 checkpoint-0 -> allreduce1,... ,broadcast1, checkpoint 1
if any node fail, it loadcheckpoint and repeat execute allreduce since that checkpoint. Notice the there is always one checkpoint exists and it's done at end of last iteration. It means the operations within each iteration can be different. (e.g if user want to do a compare results every n iterations)

example of not supported use case:

allreduce loadcheckpoint-0 ... checkpoint 0
if node fail and user code tried to do allreduce before loadcheckpoint, other nodes will not be able to recover results as it was discard in checkpoint 0 ack phase.

try chenqin/rabit@aa2153f
worker-0 shutdown, log as below:
[0] load checkpoint global 17351 version 13
[0] send req - |1|0|0|0|0| - |4|0|0|
[0] recv ack - |-55050240|0|0|0|1| - |33716452|0|1|
AssertError:likely minimal seqno overflow, shutting down process

worker-1 alive, waiting, but no other log.

So I found root cause of negative overflow, it was caused by introduce additional bit and update left shift 5 bits. It cause kSpecialOp << 5 + flags greater than signed int32 range. I submitted a fix which will use unsigned_int32 instead of signed one. https://github.com/chenqin/rabit/pull/4/files#diff-5404c32553915533fabb6d48e626e98fR274
The issue should be fix.

@chenqin Can we close this one?

Yes please


make prep_lite -j with Docker build

I followed the instructions to build BP with Docker, and once I got to the step where you're supposed to run make prep_lite, in the comments it says you can run make prep_lite -j for faster builds. However, I received the error below when I tried this. When I ran make prep_lite without the -j, it worked just fine. Not sure if it is a problem on my end or on BP end.
CXX      utils/sc_mempool.lo
CXX      utils/sc_pq.lo
CXX      utils/sc_report.lo
CXX      utils/sc_report_handler.lo
CXX      utils/sc_stop_here.lo
CXX      utils/sc_string.lo
CXX      utils/sc_utils_ids.lo
CXX      utils/sc_vector.lo
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'datatypes/int/sc_signed.lo' failed
make[7]: *** [datatypes/int/sc_signed.lo] Error 1
make[7]: *** Waiting for unfinished jobs....
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'datatypes/bit/sc_bv_base.lo' failed
make[7]: *** [datatypes/bit/sc_bv_base.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'tracing/sc_wif_trace.lo' failed
make[7]: *** [tracing/sc_wif_trace.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'datatypes/int/sc_unsigned.lo' failed
make[7]: *** [datatypes/int/sc_unsigned.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'datatypes/int/sc_int_base.lo' failed
make[7]: *** [datatypes/int/sc_int_base.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'datatypes/int/sc_uint_base.lo' failed
make[7]: *** [datatypes/int/sc_uint_base.lo] Error 1
Makefile:1072: recipe for target 'communication/sc_signal_ports.lo' failed
make[7]: *** [communication/sc_signal_ports.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'kernel/sc_phase_callback_registry.lo' failed
make[7]: *** [kernel/sc_phase_callback_registry.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'datatypes/bit/sc_lv_base.lo' failed
make[7]: *** [datatypes/bit/sc_lv_base.lo] Error 1
Makefile:1072: recipe for target 'datatypes/fx/sc_fxnum.lo' failed
make[7]: *** [datatypes/fx/sc_fxnum.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'kernel/sc_module.lo' failed
make[7]: *** [kernel/sc_module.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'datatypes/misc/sc_concatref.lo' failed
make[7]: *** [datatypes/misc/sc_concatref.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'datatypes/fx/sc_fxval.lo' failed
make[7]: *** [datatypes/fx/sc_fxval.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'tracing/sc_vcd_trace.lo' failed
make[7]: *** [tracing/sc_vcd_trace.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'communication/sc_prim_channel.lo' failed
make[7]: *** [communication/sc_prim_channel.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'utils/sc_vector.lo' failed
make[7]: *** [utils/sc_vector.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'kernel/sc_process.lo' failed
make[7]: *** [kernel/sc_process.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'communication/sc_event_queue.lo' failed
make[7]: *** [communication/sc_event_queue.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Makefile:1072: recipe for target 'communication/sc_clock.lo' failed
make[7]: *** [communication/sc_clock.lo] Error 1
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
 Most likely a Docker / environment issue. If you’re on Ubuntu, I would recommend just building natively unless there’s some strong reason to use Docker.

No problem. There is actually a strong reason to use Docker. I'm running on a Mac, and I've heard that there are problems with setting it up on a Mac. Anyways got it up and running, thanks!

Oh, I misunderstood you! I thought you were running Docker on Ubuntu, not an Ubuntu Docker image.  Which version of MacOS/Docker are you running?

MacOS Catalina version 10.15.4
Docker version 19.03.8

@dpetrisko
This error also occasionally occurs for me on Ubuntu without docker for any of the make commands.
I think it's probably due to high resource usage conflicting with the operating system.
Initially I thought I could specify the number of cores like for verilator. However it seems like it's not possible or I'm missing something.
Currently I either have 10 minutes of build time or 1 minute with the risk crashing.

There's nothing too fancy going on here, it's just multithreaded GCC commands for the most part. Are you on a VM or native install?

Just a regular native install.
I mean I don't think there is anything wrong with using all threads if you have a vm or dedicated server.
It would be nice to set it if you run it on a native install.

all threads if you have a vm

In an ideal world, but I've seen buggy VMs.

It would be nice to set it if you run it on a native install.

I'm a bit confused. Is the issue only coming up when you run with all threads? Of course you can use make -j  to taste.

Ok that's what I thought initially but looking at my resource monitor if I just run -j without specifying the number of threads my 16 threads will all be utilized at 100%. If I specify -j 16 only a few threads will be pushed to 100% at a time.
I'll read up some more how -j works as this is really confusing me.

Optimal makefile threads depend on a lot of factors: core count, SMT, available memory... Generally accepted wisdom is to use [number of logical cores]+1 as a default (to prevent blocking on I/O threads).
Fun exercise, but we should be running prep pretty rarely, so I wouldn't worry about it too much.

Actually this occured for me both on prep and just regular build.sc.
Anyways apparently j is not the number of threads/cores but actually just "jobs". So if I don't specify the number make will just spam the CPU with jobs (risking a crash) and it's just how make is designed. So I'll just go with your suggestion to get a modest speedup. Thanks!

Closing as I think the root cause here was the usage of unrestricted -j


Created storefront API example using React and Redux for JavaScript Buy SDK

I wanted to create a react example that would be easily scalable.
My solution was to create a working example by refactoring the React-JS-Buy example that rewrites the components to be functional components in stead of class based components.
I added Redux to create a store that is accessible throughout the app.
I added the folder reactHooks-redux-js-buy with my files as to not replace the originals in case anyone needs to reference those.
Let me know if there is anything that I need to change.
Thanks.

Thanks @mikeq1225
I got some warnings while running it. Does these matter?
Compiled with warnings.

./src/components/Products.js
  Line 10:5:  React Hook useEffect has a missing dependency: 'fetchProducts'. Either include it or remove the dependency array  react-hooks/exhaustive-deps

./src/components/App.js
  Line 22:5:  React Hook useEffect has missing dependencies: 'createCheckout', 'createShop', and 'fetchProducts'. Either include them or remove the dependency array  react-hooks/exhaustive-deps

./src/components/ProductView.js
  Line 40:5:  React Hook useEffect has a missing dependency: 'fetchProduct'. Either include it or remove the dependency array  react-hooks/exhaustive-deps

./src/components/Cart.js
  Line 53:5:  React Hook useEffect has a missing dependency: 'getCount'. Either include it or remove the dependency array  react-hooks/exhaustive-deps

Otherwise, it appears to be working well 👍

@swalkinshaw
I don't think the warnings matter.
I got these from the React Docs:

"It is only safe to omit a function from the dependency list if nothing in it (or the functions called by it) references props, state, or values derived from them."

"You can try moving that function outside of your component. In that case, the function is guaranteed to not reference any props or state, and also doesn’t need to be in the list of dependencies."

All of the functions that are triggering this warning are coming from functions that are outside of the component.  They are all in redux/ducks/shopify index.js.  Therefore they don't reference props, component state or values, and the function shouldn't change.
Thanks for the question.

Sounds good, thanks 👍


Swish Payment

Does your api supports Swish payment or invoice. Couldn't find any info or example ?

According to this article https://www.payson.se/sv/vi-lanserar-swish-i-payson-checkout/ dated February 21, 2022 they should accept swish.
Using their demo site https://demoshop.payson.se/Checkout/Index/0 it's possible to select swish.


Movies in multiple job categories

De-duplicate items in multiple job categories when pertinent.

resolved in development branch v0.3, see 181c2d680d8f9256fd350a160f10d0516e5e8cfb


Feel free to add our team...

https://github.com/tiakansara/kubera

https://TiaKansara.com
https://MichalStefanow.com
http://www.boymaas.nl/
https://www.linkedin.com/in/kahuna

Added in https://github.com/friedger/eosprojectslondon/commit/69b055b0f548a232384bbfe115b1c72c17df1b7d
Thank you!


Consequences of Save not being enabled if file does not parse correctly

My nightmare scenario would be a Molesworth putting a lot of code in (without saving), then getting a hard to detect/correct parse error 2 minutes before the end of the lesson and not being to save it.
I'm not sure what, if any, is the answer to this.

No way around this. It has to parse to be saved, otherwise you will not be able to load the program again.
N.B. I did consider adding the rule that you must make a field parse before you leave the field. Decided (in part on the advice of Michael Kölling and team, who had considered the same thing at one point in their design for Stride) that this would be too brutal.
Teachers do need to reinforce the message: don't leave fields incomplete (amber), or not parsing (red) for long - and certainly don't  let them build up. It is OK to leave compile errors (e.g. calling a procedure that does not yet exist), but not parse errors. There really isn't any excuse for leaving parse errors and writing more code.
#getting a hard to detect/correct parse error 2 minutes before the end of the lesson
This isn't a very realistic scenario. Parse errors are always localised to the field (c.f. compile errors). Worst case is you delete the contents of just that field, and put in the minimal code  to parse - then save.

Absorbed into #686


[CARBONDATA-3334] fixed multiple segment file issue for partition

Problem:
During partition load, while writing merge index files the FactTimestamp in load model is being changed to current timestamp due to which a new file with mergeindex entry is written.
Solution:
Set new timestamp if FactTimestamp in load model is 0L(meaning nothing is set).
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[ ] Any interfaces changed?

[ ] Any backward compatibility impacted?

[ ] Document update required?

[ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Build Failed  with Spark 2.3.2, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/11083/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/2823/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/2824/

Build Success with Spark 2.3.2, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/11084/

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/3056/

retest this please

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/3057/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/2825/

Build Success with Spark 2.3.2, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/11085/

feel like this modification is not elegant...
It is the MergeIndexWriter for partition loading that cause the problem, but you modify the loadModel instead. It seems that the writer and loadModel have some negotiations which makes the code complicated especially when the loadModel is at the beginning of loading while the MergeIndexWriter is at almost the end of loading.

Hi xuchuanyin,
The problem was that before launching the job to merge index files we are changing the fact timestamp sue to which a new segments file is written with mergeindex details.
Example:-
load writes 0_t1.segment
while the mergeindex will write 0_t2.segment.
My fix is that if the load has already identifier a timestamp for the segments file then mergeindex should use the same

retest this please

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/2873/

Build Failed  with Spark 2.3.2, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/11133/

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/3104/

retest this please

LGTM

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/2876/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/3107/

Build Success with Spark 2.3.2, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/11136/

@ravipesala build passed..please merge


Storywright silently fails with Storybook 8

Storywright relies on the deprecated __STORYBOOK_CLIENT_API__?.raw() to get a list of stories. In Storybook 8 this already deprecated API was removed. Rather than throwing, Storywright just silently falls back to an empty array and makes the error hard to notice.

Hi @NotWoods have you found a way to run storywright with storybook 8?

I ended up getting rid of Storywright and just using Storybook Test Runner to generate screenshots in the same folders as Storywright.

Since I got asked by a couple folks last week, I wrote up our Storybook Test Runner configuration today: How to fix Storybook screenshot testing | Tiger Oakes


Add TRANSFER NFT

Coverage decreased (-0.1%) to 43.771% when pulling d23bd0d08637e0dda7289b1622020873d145f391 on jplomas:master into ebdd1774698540025302c9c0342cd20cb1cce90d on theQRL:master.


422 Unprocessable Entity

Similar to the 404, I'm getting an inscrutable error when creating a machine (this is a different machine):
Create request failed: 422 Unprocessable Entity, &{ID: Name: State: Region: InstanceID: PrivateIP: Config:{Env:map[] Init:{Exec:[] Entrypoint:[] Cmd:[]} Image: Metadata:<nil> Restart:{Policy:}
Services:[] Mounts:[] Guest:{CPUKind: Cpus:0 MemoryMb:0}} ImageRef:{Registry: Repository: Tag: Digest: Labels:{}} CreatedAt:0001-01-01 00:00:00 +0000 UTC}

I have no idea what I did wrong, at the moment it looks like I'll need to resolve this via trial and error.
Improving the error message to indicate

what was sent
what was expected
why this was a problem
potential remedies

would be wonderful.

I've entered the incorrect region, it works after updating the region with sg.
https://fly.io/docs/reference/regions/


pip install uniswap-python not getting latest files??

This could just be me, however I ran into an issue with excessive approvals that I think is related to an outdated package on PyPi.
I found the excessive approvals was fixed in issue #136 around mid July, however when I compared the decorator code on github with what was installed via the 'pip install' I noticed they were different (note - I ran the pip install around 1-oct).
Digging a little deeper it appears the pypi repository is using v0.5.2 from Jun 23, 2021, which I suspect doesn't include any code changes after that date??
I'm just speculating here as I don't really know how the 'pip install' works, but google tells my it uses the pypi repository. :)

I just released a new version and updated the package on PyPI, should work fine now.


angular material color pallette

Implement $primary and $accent pallettes
$warm ? if not implemented defaults to red => $warn:    mat-palette($mat-red, 600);.

Default $warn value is ok.

Closed by #45


How to use only Gaussian curvature for iteration?

I wanted to know if we can only use absolute Gaussian curvature to remove the vertex.

You could. That would be closer (but still different) from the memoryless version of our method. More details can be found here https://www.cs.wm.edu/~ppeers/publications/Shoemaker2023IMS/Shoemaker_ARXIV2023.pdf

Thank you for sharing the IMS paper, actually I was looking into that. Do you know what changes can be made in the transportation cast for this?

This should be easy, you just need to compute cost using the element wise summation of Gaussian curvature difference dK in here.
This will ignore the transport and solely focus on the difference in Gaussian curvatures.
You would need to do the same change for the following functions

pre_flatten_interior_vertex_and_cost
flatten_interior_vertex_and_cost
pre_flatten_boundary_vertex_and_cost
flatten_boundary_vertex_and_cost
and commenting out the parts about parallel transport. Hope I don't miss anything.

Thank you for the explanation. If I comment out the parts about parallel cost from here, two variable fs_twin and Rij are used in the code after that also, so shall I make Rij as unity matrix or anything else is required?

Oh, I think you can comment out the part from l168 to l216 in this script. This part keep track of the curvature transport history (stored as T) and use the historical information to compute cost (l204-l213). R is the rotation needed to compute parallel transport. Since you are not using the transport cost, all the code from l168-l216 won't be needed (this also applies to other scripts such as faltten_bopundary_vertex_and_cost)

Thank you for the explanation. So, the cost will be dk.sum(), is it right?


Cannot find 'kGREYConfigKeyAnalyticsEnabled' in scope

I'm trying to set the path of where screenshots should be saved with this code snippet, but it's throwing this error as noted on the topic. Where does the class it's calling in the EarlGrey directory?
override func setUp() {
        let directory = FileManager.SearchPathDirectory.desktopDirectory
        let domainMask = FileManager.SearchPathDomainMask.allDomainsMask
        let paths = NSSearchPathForDirectoriesInDomains(directory, domainMask, true)
        let path = "\(paths[0])/FailedTests"
        // Disable analytics.
        GREYConfiguration.sharedInstance().setValue(path, forConfigKey: kGREYConfigKeyAnalyticsEnabled)
    }

kGREYConfigKeyAnalyticsEnabled no longer exists in EarlGrey as we do not collect any usage analytics. I think you have that key wrong.


Sprites and Image Optimization based on Reduce Request

Hi Andrew,
You might already be working on Issue #73 or have some ideas how you want to do this with cassette but I decided to take a stab at implemented some sprite and image optimization magic into cassette using Reduce Request as an example.
I liked how things just magically worked with ReduceRequest I tried to integrate it with Cassette. However I hit a few roadblocks. Some parameters in the BackgroundImageClass are marked Internal and ReduceRequest is Targeting .NET 4 Full Version (Not .NET 4 Client framework) and System.Web. To get the magic bits I moved 11 classes to Cassette from Reduce Request. Not sure how I feel about that one.
I had a number of issues and the code can do with some re-factoring and tests.
Any ways this is what I come up with let me know what you think the Example site should be working.

I've not started any work on spriting yet. So I appreciate you looking into this. I don't mind borrowing code from RequestReduce, as long as we include proper attribution.
I may speak to Matt about making the image spriting code more reusable in general. Then Cassette could just have it has a nuget dependency. For now though, let's experiment with whatever works for Cassette :)
I've not had a chance to review your code yet (too much client work on right now). I will take a look on my next open-source day. Thanks again for all the time you're putting into Cassette. I really appreciate the support!


Support for 1.16.4?

When I updated my server to 1.16.4 the plugin stopped working, I noticed on the plugin page that it goes to 1.16.3 so I figured that would be the problem.
Any thoughts as to when you’ll get it up to 1.16.4, if ever?

There's a beta for it available on my Discord. There are just a lot of bigger code base changes wrapped into this that it's just taking a while to get it all 100% and tested enough where I'm happy. It's close though

Duplicate of #33


Add verifiability for Mailchimp plugin

As @dgzlopes helpfully linked to, https://github.com/streaak/keyhacks#MailChimp-API-Key shows how we can add this.
See #194 for an example of how we do this in detect-secrets.
The parent issue (#153) is still opened but figured I'd make this as a good first issue.

Adding in https://github.com/Yelp/detect-secrets/pull/241/commits/977c4fb5606b42a9c73dfb598fa0a6cd0ab77c90

PR was merged, going to close this issue, will release a new version soon.


Config OCamlFormat but don't enable it ;)

Can this be added a standalone PR? It should be at the top level to cover all files.

https://github.com/ocurrent/ocaml-ci/pull/501#discussion_r927233433
There it is to help @novemberkilo a bit. As I'm not sure whether we want to enable it globally, I've disabled ocamlformat for now. It could be used just for the "standalone" dream-web module, if needed.

I think this is subsumed by #514


Compiler leaking globals when (define var) not direct member of a lambda

The following code leaks a as a global:
(define (func)
   (if #t
             (define a 5)
   ))

I could fix this by duplicating get-defined-vars from compiler.lsp (was used by expander + compiler) and changing get-defined-vars for the compiler like this:
(define (get-defined-vars expr)
  (letrec ((get-defined-vars-
            (lambda (expr)
              (cond
               ((atom? expr) ())
               ((and
                 (eq? (car expr) 'define)
                 (pair? (cdr expr))
                 )
                (or
                 (and
                  (symbol? (cadr expr))
                  (list* (cadr expr) (get-defined-vars- (caddr expr )))
                  )
                 (and (pair? (cadr expr))
                      (symbol? (caadr expr))
                      (list (caadr expr)))
                 ()
                 )
                )
               ((not (member (car expr) (list 'lambda 'trycatch )))
                (apply nconc (map get-defined-vars- (cdr expr))))
               (else ())))))
    (delete-duplicates (get-defined-vars- expr))))

Before that change the scanner did only descent into "begin" blocks but not deeper.

Thanks for finding this. Is there a reason not to use the corrected code in both the expander and compiler?


Update Character “gerald-stumpton”

Automatically generated by Netlify CMS

👷 Deploy Preview for dynamic-cheesecake-5fbd8b processing.

Name
Link

🔨 Latest commit
b61edbf8d7af2a056512633cef77876498695c76

🔍 Latest deploy log
https://app.netlify.com/sites/dynamic-cheesecake-5fbd8b/deploys/633727c8509c7900097437d3


Include more reservedPaths. Fixes #1

I ran npm test and found these messages which I did not tackle in this PR:
index.js
  ‼   66:1   Function shortenURL has a complexity of 22.  complexity
  ‼  146:29  Do not nest ternary expressions              no-nested-ternary

Oh LOL

Looking good! Do you mind requiring the JSON directly? It saves a require trip

const reservedPaths = require("github-reserved-names/reserved-names.json");

LOL cool... so many ways to do the same thing ;)

LOL just as you merge, I found another reserved name... I'll publish it in a minute.

If it matches ^1.0.3 it'll be in automatically :)


bug header embed links break file.outlinks

What happened?
Header embed links like ![[link#header]] break the link in file.outlinks.
the console still says it is a link with a subpath.
the thing that confuses me is that it is the first time that i see the actual header link inside the file.outlinks. as far as i knew it never did that in the past.

it seems that the faulty indentation comes from the minimal theme. if i use another theme the link still doesn't work but it shows it as normal bullet point list without additional space.
DQL
Table without id file.inlinks as Inlinks, file.outlinks as Outlinks
where file.path = this.file.path

JS
No response
Dataview Version
0.5.53
Obsidian Version
1.1.12
OS
Windows

Oh i found where the issue comes from. it tries to embed the link but because it is an md file it doesn't work.
do i only notice it now because i never really used outlinks tables before and it was always like that or is that a new bug?
the image shows how an image is embedded by the same table as shown above. it would be preferable if inside the file.outlinks the links are saved with "embed: false" even if it is an embedded link (not sure how the data is saved and thus how doable that is).

This bug also happen for Block embed links like ![[filename^block-id]]


Put a working example or some pictures to the install instructions

Literally reported by a user:
"Put a working example or at least some pictures to the install instructions, that one can have an idea what it will look like in the end."

we need to add this to the readthedocs content to be developed


ListRepositoryWorkflowRuns doesn't return queued runners

After investigating why autoscaling is not increasing the amount of runners ("desired": "1") although having a queue of over 20 runs with at least 3 queued jobs inside each run, it seems the request to GitHub is returning { "total_count": 0 } runs.
opts := github.ListWorkflowRunsOptions{
    Status: status,
}

for {
    list, res, err := c.Client.Actions.ListRepositoryWorkflowRuns(ctx, user, repoName, &opts)
}

(pagination was removed as a test, without success)
However, when executing the request without opts, the runs are returned. Printing them shows runs with field "status": "queued".
runs, res, err := c.Client.Actions.ListRepositoryWorkflowRuns(ctx, user, repoName, nil)

Raw output after adding some logs and replacing the controller in our cluster with the custom built one:

Side question: why would you execute the request without capturing the response? (disclaimer: no experience with Go)
func (c *Client) listRepositoryWorkflowRuns(ctx context.Context, user string, repoName, status string) ([]*github.WorkflowRun, error) {
    c.Client.Actions.ListRepositoryWorkflowRuns(ctx, user, repoName, nil)

Seems like the missing workflow runs in the API response is due to today's incident on GitHub: https://github.community/t/list-workflow-runs-api-not-detecting-in-progress-builds/165029

🤦🏼♂
Thanks for the quick fix! 🚀


Wriggly - Submitting a Coding Challenge

Hello,
We would like to add our application to be a part of your web3 coding challenges. Wriggly is a web3 blockchain learning platform to practice Solidity based coding challenges to prepare for interviews.
Please find attached our .md file.
Thanks,
Sidd
wriggly.md

Hi, please submit a PR with your md. Thank you!

Hey Wesley,
I'm getting this error when I try to push to my branch. I also used ssh to clone the repo. If you could help me, I'd really appreciate it.

Thanks,
Sidd

It seems like you haven't correctly configured your credentials.
What happens if you ssh -T<EMAIL_ADDRESS>?
Make sure to follow the steps at https://docs.github.com/en/authentication/connecting-to-github-with-ssh

Followed the steps and still getting the same error. This is what I get when I try
ssh -T<EMAIL_ADDRESS>

Got it to work, thanks!


ConcurrentModificationException will be reported when concurrent

java.util.ConcurrentModificationException
        java.base/java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:756)
        java.base/java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:778)
        io.github.reactivecircus.cache4k.RealCache.expireEntries(RealCache.kt:174)
        io.github.reactivecircus.cache4k.RealCache.put(RealCache.kt:120)

+1:
java.util.ConcurrentModificationException at
java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:760) at
java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:782) at
kotlin.collections.CollectionsKt___CollectionsKt.firstOrNull(_Collections.kt:272) at
io.github.reactivecircus.cache4k.RealCache.evictEntries(RealCache.kt:206) at
io.github.reactivecircus.cache4k.RealCache.put(RealCache.kt:140)

let cacheBuilder optionally accept a parameter to be used as a map underhood

I think we need to revert back to using stately which is thread-safe on JVM.

@ychescale9 will you be making a release anytime soon ?

I'll try to find some time in the coming weeks.

So I'm having issues with cache4k and ConcurrentModificationExceptions caused by stately on jvm.
Ktor's io.ktor.util.collections.ConcurrentMap seems to be stable, maybe their implementation could be used instead?

Thanks, will look into it.

+1 for Ktor Utils Collections 😊

Just released 0.13.0 with ConcurrentHashMap on JVM. Please let me know if this fixes the issue.


[Suggestion] On x100 levels, group players in non-boss Lanes so LN spam doesn't help trolls reset nuke cooldowns

WH doesn't require that you be in a boss lane to benefit from the level skip.
LN however does require that you be in the lane with other LN users, so that you rest their WH cooldowns.
The downside of being in a boss lane is that using LN to reset WH cooldowns also resets the cooldowns on people's nukes, so (whether trolling or not) they can keep nuking the boss to death.
So if we have script users move to lane 1 whenever the x100 boss isn't in lane 1, and move to lane 2 whenever the x100 boss is in lane 1, then we could spam WH and LN forever in those lanes, without helping the nukers nuke the boss down prematurely.
This is nice because there's nothing the nukers can do about it - it depends on on the scripter's behaviour. There is a way trolls could still get their cooldowns reset, but it would slow them down a lot, and would be automatic if they're using a script.

The idea is interesting. However, this would require some coordination between different scripts, such as steamdb's; and the spammers just have to go to the right lane one second to have their abilities reset (granted, if they are manual spammers, they probably won't think about it).

True, but we are already pushing pretty hard to only have updated wchills users in the record groups.
This would get rid of manual/oblivious nukers, and make things annoying for intentional trolls, unless they write/mod their own scripts to automate swapping and nuking

Trolls can just buy their own LNs to reset their nukes, and since they aren't buying any WHs, they can easily have >50k LNs. Then it is just a matter of pushing "{"requested_abilities":[{"ability":27},{"ability":10}],"gameid":"xxx"}" to the server every second.
Lane switching is not effective at dealing with nukers.

This isn't a silver bullet but I think its a great idea and should help somewhat.

At this point I doubt it's going to be implemented, but for the record, most of the reported trolls I've looked at today have tens of thousands of WHs, and very few if any LNs - the trolling is even parts using WH on the 99 levels as it is nuking on the 100s. So it would have worked fine on them, and forced them to leave game (and hence drop to a lower one) to change their purchases.
It looks like today's most promising game is miraculously free of trolls so we're good for today. Just gotta hope for the same luck tomorrow.

Looks merged. Would it allow us to upgrade elemental/turret, as we are no longer in the same lane as the boss ?

This is hilariously useless.  People do not NEED the script to play the game, and in fact, adding this to the code only gives players more incentive to skip the level beforehand since the game will throw those players in a non-boss room and avoid most of the gold gains while idling and being useless.
Anyone who thinks anti-trolling should be included in the code is no different from parents who try to ban violent video games;  adding anti-troll in the code does not do a goddamn thing because:

The script isn't required to play the game.  At all.
It takes minutes to remove it from the code.
You can literally refresh your page right before you want to troll.
It removes rewards from legitimate players since wormhole bosses give the most gold (since they are alive the longest)

Having anti-troll in the code only gives me more of a reason to wormhole PAST the boss, because I am forced to idle during wormhole bosses and do nothing.  It might be a script but I miss out on a ridiculous amount of gold so it's still an annoyance.

@Metalsand if you are using this script you should have enough gold anyway that it doesn't matter.
Regardless, it's already in the code. Closing.


Issue with Hero transition in homepage under JWOC

Yellow lines underneath the text if someone comes back from thriller section from homescreen. Need to tweak some easy stuffs there

i can work on this issue , would be a quick fix

yesss you can work on this, sorry got the issue closed by mistake

Ok no problem, it will be a quick fix

You can now close this issue


Make it optional to start VM in the host netns

Now we always create a netns for shim and hypervisor processes. As discussion in https://github.com/kata-containers/runtime/issues/113#issuecomment-403684825, we should provide a option to let VM run in the host netns.

I plan to work on this. But CNM model may not work in this scenario.


Doesnt work

I enabled the mod, yet it doesnt work with my bone sword

Did you use other mods? Are you sure OmniSwing is listed as enabled?


训练voc数据集  ifa != nullptr. Unable to find address for: ib0

RuntimeError: [enforce fail at /pytorch/third_party/gloo/gloo/transport/tcp/device.cc:83] ifa != nullptr. Unable to find address for: ib0
看之前的问题说是把all_reduce_norm(self.model)注释了，我注释了还是不行

This is a bug of NCCL confugure. Plz pull the latest updates.


fix(41078): Convert to ES6 Class Refactoring removes methods named with quoted properties

Fixes #41078

Sorry, I don't think this should go into 4.1 -- I'll merge this on Monday after master switches to 4.2.


Refactor test_get_execution_info

This change is

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (6f44608) 70.93% compared to head (cb95205) 70.99%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1284      +/-   ##
==========================================
+ Coverage   70.93%   70.99%   +0.05%
==========================================
  Files          57       57
  Lines        7154     7157       +3
  Branches     7154     7157       +3
==========================================
+ Hits         5075     5081       +6
+ Misses       1666     1663       -3
  Partials      413      413

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Added missed args to MediaElementAudioSourceNode at w3c_audio.js

According to
https://developer.mozilla.org/en-US/docs/Web/API/MediaElementAudioSourceNode/MediaElementAudioSourceNode
and
https://webaudio.github.io/web-audio-api/#MediaElementAudioSourceNode-constructors
there are two optional missing args in MediaElementAudioSourceNode constructors: AudioContext and MediaElementAudioSourceOptions

https://github.com/google/closure-compiler/issues/3894

Note, from https://webaudio.github.io/web-audio-api/#MediaElementAudioSourceNode-constructors, the two arguments are non-optional.

@rishipal thanks for catching that, fixed

Fixed this internally in 2 CLs. First added the 2 args but kept them optional (ca1b48174f9a9bda1a3719cd9ec7a3fdf350b62e). Second made the args non-optional (ca1b48174f9a9bda1a3719cd9ec7a3fdf350b62e).


SAM deploy issue with Node.js version

Since the runtime parameter of nodejs12.x is no longer supported for creating or updating AWS Lambda functions, changing those version to recent one is needed.

Well obviously this is orphaned.
What's strange is sam keeps throwing this error even when I update template.yaml to
Runtime: nodejs18.x

Hi, I've updated the template to Node. js v18. Please ensure you are running the latest version of SAM CLI before deploying.

Confirmed I was able to build the project after re-cloning post-fix and updating SAM from 1.104.0 to 1.105.0.


Enabling Vertical body scroll undesirably enables infinite Virtual paging along with it.

I'm submitting a ...  (check one with "x")
[x] bug report => search github for a similar issue or PR before submitting
[ ] feature request
[ ] support request => Please do not submit support request here, post on Stackoverflow or Gitter

Requesting here because nobody answered my stackoverflow question - https://stackoverflow.com/questions/44523109/how-to-disable-page-output-event-being-called-on-scroll-in-ngx-datatable
Current behavior
Enabling ScrollbarV by default fires (page) Pagination Output on body scroll irrespective of whether actual pagination button is clicked. This has become a bottle in my project as I want body scroll ( Otherwise my page becomes too long for large tables!) with fixed headers but not infinite Virtual paging.
Expected behavior
Infinite scrolling should be enabled by another Input and not by ScrollbarV. I am in the early stage of adopting ngx-datatable for my highly data-driven project. But, if I can't work around this I'll have to move to other tools
Reproduction of the problem
This is my HTML :
<ngx-datatable [rows]="rows" [columns]="columnMeta" class="dark" columnMode="force" [loadingIndicator]="loadingIndicator" [scrollbarV]="true" [scrollbarH]="true" [footerHeight]="100" [externalPaging]="true" [count]="page.totalElements" [offset]="page.pageNumber"
[limit]="page.size" (page)='setPage($event)' (scroll)="scrolled($event)">

This is SCSS :
.ngx-datagrid {
.datatable-body {
min-height: 700px;
}
}
This is my js for fetching the server side data :
setPage($event) {
console.log("page no ", $event.offset);
this.pageChange.emit($event.offset);
this.page.pageNumber = $event.offset;
}
What is the motivation / use case for changing the behavior?
We are using a really old back end java server that doesn't bear too many load data requests. Hence I wan't to disable infinite loading while retaining body scroll.
Please tell us about your environment:
Ubuntu, Visual Studio code , angular cli dev server.

Table version: 0.8.x
latest

Angular version: 2.0.x
4.x

Browser: [all]

Language: [ TypeScript ]

The purpose of the page event is just to tell you that the content of the viewport has shifted by a multiple of whatever page size is. I suspect your problem is that you're expecting the page event to do more than that one job. It is up to you to determine when to fetch more data, not the table. If you want to reduce the number of requests for data you need to either make a single request on load, or if too large for that either cache the request in the browser using request headers or in your JS.

https://github.com/swimlane/ngx-datatable/blob/master/demo/paging/paging-virtual.component.ts
Includes an example of how to cache results.

I agree with OP - it looks like the page() event is raised on both page change and scroll, even if the actual page # didn't change. I feel like its one job is to notify that page changed.

I agree too - simply adding an indication of where this page event originates (bodyScroll || footerScroll) would help. At the moment he scroll kicks in as soon as I click in the table, reverting back to page 1.

Does the new [virtualization]="false" parameter help in this case?


feat(agents): allow trimming of intermediate steps to last N

Added an option to trim intermediate steps to last N steps. This is especially useful for long-running agents. Users can explicitly specify N or provide a function that does custom trimming/manipulation on intermediate steps. I've mimicked the API of the handle_parsing_errors parameter.

Fixes #6370
Before submitting

Who can review?
Tag maintainers/contributors who might be interested:
@hwchase17

we maybe want to think more carefully about how to handle intermediate steps and allow people to pass in a function saying how to handle
@eyurtsev thoughts?

@hwchase17 another proposal can be to allow trim_intermediate_steps to be an int for the number of intermediate steps to consider. Then if anyone wants to do more complex logic, they can explicitly override _prepare_intermediate_steps.  WDYT?

lgtm, thanks @nirga!


Remove more_itertools dependency

Checklist

[X] I added a descriptive title
[X] I searched open requests and couldn't find a duplicate

What is the idea?
conda-index only uses one more_itertools "ichunked" feature. Remove dependency by bundling our own iterate-over-chunks implementation?
Why is this needed?
No response
What should happen?
No response
Additional Context
No response

It's a Python 3.12 feature! https://docs.python.org/3/library/itertools.html#itertools.batched


Bug: DraggableBlockPlugin uses unexposed apis

Getting Failed to resolve import "lexical/LexicalUtils" when trying to use DraggableBlockPlugin from the playground. getElementByKeyOrThrow is exported by lexical/LexicalUtils but the npm package only contains Lexical.{dev,prod}.js and getElementByKeyOrThrow is not exported anywhere else.
Lexical version: 0.5

I get same problem, and using https://lexical.dev/docs/api/classes/lexical.LexicalEditor#getelementbykey instead of getElementByKeyOrThrow


Old link to SD article 404ing

https://developers.google.com/web/fundamentals/primers/shadowdom/
needs a redirect to:
https://developers.google.com/web/fundamentals/getting-started/primers/shadowdom

Fixed and deployed


Improvements from TheForge

The work was performed by collaboration of TheForge and Google. I am merely splitting it up into smaller PRs and cleaning it up.
This is the most "risky" PR so far because the previous ones have been miscellaneous stuff aimed at either improve debugging (e.g. device lost), improve Android experience (add Swappy for better Frame Pacing + Pre-Transformed Swapchains for slightly better performance), or harmless ASTC improvements (better performance by simply toggling a feature when available).
However this PR contains larger modifications aimed at improving performance or reducing memory fragmentation. With greater modifications, come greater risks of bugs or breakage.
Changes introduced by this PR:
Transient memory
TBDR GPUs (e.g. most of Android + iOS + M1 Apple) support rendering to Render Targets that are not backed by actual GPU memory (everything stays in cache). This works as long as load action isn't LOAD, and store action must be DONT_CARE. This saves VRAM (it also makes painfully obvious when a mistake introduces a performance regression). Of particular usefulness is when doing MSAA and keeping the raw MSAA content is not necessary.
Immutable samplers
Some GPUs get faster when the sampler settings are hard-coded into the GLSL shaders (instead of being dynamically bound at runtime). This required changes to the GLSL shaders, PSO creation routines, Descriptor creation routines, and Descriptor binding routines.
Toggle

bool immutable_samplers_enabled = true

Setting it to false enforces the old behavior. Useful for debugging bugs and regressions.
Immutable samplers requires that the samplers stay... immutable, hence this boolean is useful if the promise gets broken. We might want to turn this into a GLOBAL_DEF setting.
Linear Descriptor Pools
Instead of creating dozen/hundreds/thousands of VkDescriptorSet every frame that need to be freed individually when they are no longer needed, they all get freed at once by resetting the whole pool. Once the whole pool is no longer in use by the GPU, it gets reset and its memory recycled. Descriptor sets that are created to be kept around for longer or forever (i.e. not created and freed within the same frame) must not use linear pools. There may be more than one pool per frame. How many pools per frame Godot ends up with depends on its capacity, and that is controlled by rendering/rendering_device/vulkan/max_descriptors_per_pool.

Possible improvement for later: It should be possible for Godot to adapt to how many descriptors per pool are needed on a per-key basis (i.e. grow their capacity like std::vector does) after rendering a few frames; which would be better than the current solution of having a single global value for all pools (max_descriptors_per_pool) that the user needs to tweak.

Toggle

bool linear_descriptor_pools_enabled = true

Setting it to false enforces the old behavior. Useful for debugging bugs and regressions. Setting it to false is required when workarounding driver bugs (e.g. Adreno 730).
 Ridiculous because the original code should've done this in the first place. Previously Godot was doing the following:

Create a command buffer pool. One per frame.
Create multiple command buffers from the pool in point 1.
Call vkBeginCommandBuffer on the cmd buffer in point 2. This resets the cmd buffer because Godot requests the VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT flag.
Add commands to the cmd buffers from point 2.
Submit those commands.
On frame N + 2, recycle the buffer pool and cmd buffers from pt 1 & 2, and repeat from step 3.

The problem here is that step 3 resets each command buffer individually. Initially Godot used to have 1 cmd buffer per pool, thus the impact is very low.
But not anymore (specially with Adreno workarounds to force splitting compute dispatches into a new cmd buffer, more on this later). However Godot keeps around a very low amount of command buffers per frame.
The recommended method is to reset the whole pool, to reset all cmd buffers at once. Hence the new steps would be:

Create a command buffer pool. One per frame.
Create multiple command buffers from the pool in point 1.
Call vkBeginCommandBuffer on the cmd buffer in point 2, which is already reset/empty (see step 6).
Add commands to the cmd buffers from point 2.
Submit those commands.
On frame N + 2, recycle the buffer pool and cmd buffers from pt 1 & 2, call vkResetCommandPool and repeat from step 3.

Possible issues: @dariosamo added transfer_worker which creates a command buffer pool:
transfer_worker->command_pool = driver->command_pool_create(transfer_queue_family, RDD::COMMAND_BUFFER_TYPE_PRIMARY);

As expected, validation was complaining that command buffers were being reused without being reset (that's good, we now know Validation Layers will warn us of wrong use). I fixed it by adding:
void RenderingDevice::_wait_for_transfer_worker(TransferWorker *p_transfer_worker) {
	driver->fence_wait(p_transfer_worker->command_fence);
	driver->command_pool_reset(p_transfer_worker->command_pool); // ! New line !

Secondary cmd buffers are subject to the same issue but I didn't alter them. I talked this with Dario and he is aware of this. Secondary cmd buffers are currently disabled due to other issues (it's disabled on master).
Toggle

bool RenderingDeviceCommons::command_pool_reset_enabled

Setting it to false enforces the old behavior. Useful for debugging bugs and regressions.
There's no other reason for this boolean. Possibly once it becomes well tested, the boolean could be removed entirely.
Descriptor set batched binding
Adds command_bind_render_uniform_sets and add_draw_list_bind_uniform_sets (+ compute variants).
It performs the same as add_draw_list_bind_uniform_set (notice singular vs plural), but on multiple consecutive uniform sets, thus reducing graph and draw call overhead.
Toggle

bool descriptor_set_batching = true;

Setting it to false enforces the old behavior. Useful for debugging bugs and regressions.
There's no other reason for this boolean. Possibly once it becomes well tested, the boolean could be removed entirely.
Do not wait so long for swapchain
Godot currently does the following:

Fill the entire cmd buffer with commands.
submit()

Wait with a semaphore for the swapchain.
Trigger a semaphore to indicate when we're done (so the swapchain can submit).

present()

The optimization opportunity here is that 95% of Godot's rendering is done offscreen. Then a fullscreen pass copies everything to the swapchain. Godot doesn't practically render directly to the swapchain.
The problem with this is that the GPU has to wait for the swapchain to be released to start anything, when we could start much earlier. Only the final blit pass must wait for the swapchain.
TheForge changed it to the following (more complicated, I'm simplifying the idea):

Fill the entire cmd buffer with commands.
In screen_prepare_for_drawing do submit()

There are no semaphore waits for the swapchain.
Trigger a semaphore to indicate when we're done.

Fill a new cmd buffer that only does the final blit to the swapchain.
submit()

Wait with a semaphore for the submit() from step 2.
Wait with a semaphore for the swapchain (so the swapchain can submit).
Trigger a semaphore to indicate when we're done (so the swapchain can submit).

present()

Dario discovered this problem independently while working on a different platform.
However TheForge's solution had to be rewritten from scratch: The complexity to achieve the solution was high and quite difficult to maintain with the way Godot works now (after Übershaders PR). But on the other hand, re-implementing the solution became much simpler because Dario already had to do something similar: To fix an Adreno 730 driver bug, he had to implement splitting command buffers. This is exactly what we need!. Thus it was re-written using this existing functionality for a new purpose.
To achieve this, I added a new argument, bool p_split_cmd_buffer, to RenderingDeviceGraph::add_draw_list_begin, which is only set to true by RenderingDevice::draw_list_begin_for_screen.
The graph will split the draw list into its own command buffer.
Toggle

bool split_swapchain_into_its_own_cmd_buffer = true;

Setting it to false enforces the old behavior. This might be necessary for consoles which follow an alternate solution to the same problem. If not, then we should consider removing it.
Free Shader memory
PR #90993 added shader_destroy_modules() but it was not actually in use.
This PR adds several places where shader_destroy_modules() is called after initialization to free up memory of SPIR-V structures that are no longer needed.
What's missing?
The following improvements from TheForge were left out from this PR:

Render Pass Optimizations.

They were mostly good but I need more time to analyze what they do and that they actually help.

Replace Push Constants by UBOs.

They were not included as they were a bit controversial since point 3 was left out too (continue reading)

UMA buffers. They were the most anticipated feature, but unfortunately full of race conditions. It's the feature that needs the most work. Without this feature, it's unclear if replacing push constants by UBOs are a net win or a net loss.

CI failures

[x] Deal with DISABLE_DEPRECATED removing INITIAL_ACTION_CLEAR_REGION & co.
[x] Fix D3D12.
[x] Fix Metal.

I added TRANSIENT support for Metal too (it's called "Memoryless" there) but I did not test it on a real device.

I'm unsure. Removing what's no longer necessary is often a good practice to save RAM. But we could ask TF if they found a problem in particular that motivated them to do it.

Should be using linear uniform sets as much as possible?

"Yesn't".
Linear Uniform Sets (let's call it LUS and US for Uniform Set) are useful when we must recreate a US often.
Ideally we should try to create a US (or a collection of them) once and bind them when needed. Godot already tries to do this.
But there are times where, whether by the nature of the problem or a design issue, the US must be modified often which means re-creation. That's where LUS are useful.
There's still the case of "what happens when the LUS changes sometimes".
For example such is the case of the code being modified by this PR: _update_render_base_uniform_set was doing the following (simplified):
void RenderForwardMobile::_update_render_base_uniform_set() {
    if( !render_base_uniform_set || is_dirty( render_base_uniform_set ) {
       recreate();
    }
}

That is, it only recreates if the uniforms haven't been created yet, or it is dirty and must be updated.
And now it has been changed to:
void RenderForwardMobile::_update_render_base_uniform_set() {
    if( driver->use_linear_pools() || !render_base_uniform_set || is_dirty( render_base_uniform_set )  {
       recreate();
    }
}

That is, if the Godot driver (i.e. Vulkan backend) supports LUS, always recreate them.
The analysis here is deeper:

If the US is always dirty, it's a win.
If the US is never dirty, it's likely a loss. BUT, this is likely a simple scene so it won't matter.
If the US is somewhat dirty, then branch prediction goes into play. Unfortunately it would maybe be easier to if all driver backends supported LUS (if it even makes sense for the backend) so that there are no if statements: It always recreates. Sometimes the cost of the misprediction is equal or higher than the cost of just doing the operation.

Perhaps it's beneficial to always recreate (even in D3D12 & Metal) due to branch prediction woes.

So the short version is:

If the US requires recreating somewhat often, then prefer just recreating it every frame and use LUS.
If the US can be created once and almost never changes, then prefer not using LUS.

Note that this is why Vulkan has sets (at least 4 guaranteed by spec) that puts the "Set" in "Uniform Set": There are sets that will rarely change and will take 1 or 2 slots. And there are sets that will change every frame and should use the remaining slots.
Ideally one would use slot 0 for data that will never change and slot 1 for data that will always change (fun fact: D3D12 prefers the inverse: slot 0 for data that always changes). But reusability between PSOs creates reasons that some data (e.g. Material bindings) may have their own slot (despite that they could be put together with other static data).

Should we use as many immutable samplers as possible?

On Desktop, it doesn't matter (unless we're using Border Color addressing mode on AMD, which is a rare addressing mode).
On Mobile (both iOS and Android) there are benefits.
Yes, we should try to Immutable samplers as much as possible. But also try to have as fewer samplers per PSO as possible.
This is hard to balance because defining more samplers (even if unused) allows for sharing the same Uniform Set with other PSOs as they become compatible.
Swapping US too much is costly on the CPU and trashes GPU caches. But having too many unused resources in the PSO increases the GPU cost.

For D3D12 and Metal, do we have a fallback for immutable samplers? I didn't notice any code that obviously included the immutable sampler

We have a fallback and will use normal samplers. All 3 APIs support immutable samplers, but unfortunately don't agree how to define them:

Vulkan just wants us to have the VkSampler being passed alongside PSO creation.
D3D12 unfortunately doesn't want the equivalent of VkSampler, but rather the equivalent of VkSamplerCreateInfo. This needs a little more thinking to see if it's easy to adapt from what's been done, or requires further changes.
Metal wants the sampler defined in the shader code. Probably the best approach if everyone did this too. Since our native shader code is GLSL, this would require more analysis to see how to approach it.

@darksylinc With respect to immutable samplers, I ask because we have these which are included in many shaders
https://github.com/godotengine/godot/blob/f952bfe9985ad8f507cc29b2c7601bbba18b8039/servers/rendering/renderer_rd/shaders/samplers_inc.glsl#L1-L12
I'm trying to figure out if it would make sense to unconditionally expose these through the PSO instead.
From your description though, it sounds like it might be best to use immutable samplers for samplers that will change between uniform sets so we can share uniform sets more

Just to summarize the state of this PR so things don't get lost:

The transient flag needs to be removed from the depth resolve texture
This is not quite working on my M2 mac, so we need to debug why
The other comments are for follow up stuff, so they don't need any changes

OK I've removed the flag, and as you found out; turns out that also fixed the M1 problem. Two birds one stone.
So I guess all that matters is approval & merge!
Btw the addition of TRANSIENT flag means this PR might cause merge conflicts with #98670

Do NOT merge.
I spotted the shadow maps are inverted in certain situations. It's probably something wrong with the reverse Z patches I applied (TheForge made the switch to immutable samplers before Rev Z got merged).

OK the bug was fixed.
As I suspected, I simply failed to apply the Reverse Depth correction to the immutable samplers. It's ready for review/merge again.

Am I correct in assuming that this would fix the additional frame of lag caused by waiting on the swapchain before submitting commands? Even with a frame queue size of 1, it looks like the existing behavior essentially ends up waiting on two V-Syncs before the results end up on-screen:

By both creating and executing the commands before the upcoming V-blank, we'd end up saving a frame.

So I tested this PR. The new swapchain wait behavior almost works. It looks like it's still waiting on the next V-Blank before it submits the first command list (though the second one goes off without a hitch):

Am I correct in assuming that this would fix the additional frame of lag caused by waiting on the swapchain before submitting commands?

No. This PR would eliminate pipeline bubbles between drawing and presenting, which results in lower performance.
Those measurements you're posting looks like something to send to NVIDIA. The capture you're posting shows the GPU literally doing nothing and waiting on nothing only to present two VBlanks later.

No. This PR would eliminate pipeline bubbles between drawing and presenting, which results in improved performance thanks to the bubble removal.

Wouldn't this just move the bubble? The drawing and presenting were previously being done in one command list. Now it's being done in two, creating a bubble there, while removing the one between creating and submitting the command list. From your description:

Do not wait so long for swapchain
Godot currently does the following:

Fill the entire cmd buffer with commands.

submit()

Wait with a semaphore for the swapchain.
Trigger a semaphore to indicate when we're done (so the swapchain can submit).

present()

The optimization opportunity here is that 95% of Godot's rendering is done offscreen. Then a fullscreen pass copies everything to the swapchain. Godot doesn't practically render directly to the swapchain.
The problem with this is that the GPU has to wait for the swapchain to be released to start anything, when we could start much earlier. Only the final blit pass must wait for the swapchain.

Which mirrors what I'm seeing in these PIX traces. It's waiting until the swapchain is available, which doesn't happen until V-Sync. Only after the V-Sync marker you'll see the command list being executed. The proposed fix changes it to the following:

Fill the entire cmd buffer with commands.

In screen_prepare_for_drawing do submit()

There are no semaphore waits for the swapchain.
Trigger a semaphore to indicate when we're done.

Fill a new cmd buffer that only does the final blit to the swapchain.

submit()

Wait with a semaphore for the submit() from step 2.
Wait with a semaphore for the swapchain (so the swapchain can submit).
Trigger a semaphore to indicate when we're done (so the swapchain can submit).

present()

In particular, the lack of wait on the swapchain in step 2.

Those measurements you're posting looks like something to send to NVIDIA. The capture you're posting shows the GPU literally doing nothing and waiting on nothing only to present two VBlanks later.

But it's waiting on the swapchain, no? Aren't we controlling the synchronization primitives? Isn't that the purpose of APIs like Vulkan and D3D12?

@darksylinc Just to confirm, are we drawing to the same framebuffer every frame, and using that same framebuffer as the source when we copy it to the swapchain image? If so, it's likely that the framebuffer is locked waiting on the swapchain, meaning we still have to wait for an available swapchain image before we can run any commands. We might be able to fix this by having one framebuffer for each swapchain image, though if consistency of framebuffer contents between frames is important we'd have to add an extra copy, which might add some overhead.

Just to confirm, are we drawing to the same framebuffer every frame, and using that same framebuffer as the source when we copy it to the swapchain image?

Yes.

If so, it's likely that the framebuffer is locked waiting on the swapchain, meaning we still have to wait for an available swapchain image before we can run any commands.

No, because the swapchain was not acquired by the present engine during that time.
Rendering is roughly the following:

Draw into RTT0.
Wait until Swapchain Semaphore is released by the Present engine.
Copy RTT0 into Swapchain.
Present's engine must wait for the copy to release the execution semaphore so it can reacquire the Swapchain.
We render the next frame, to RTT0 (i.e. back to step 1 again). We only must wait on the semaphore released in step 3, not the one that was acquired in step 4.

The copy from step 3 can roughly take 0.5ms (give or take depending how fast the GPU is and screen resolution).
The presentation itself can take anywhere from 0 to 33.33ms to be released.
While present engine is still holding on to the Swapchain, we can start rendering to RTT0. Hopefully by the time we're done rendering to RTT0, the Swapchain was already released, causing no unnecessary bubbles.
If the swapchain hasn't yet been released, there will be a bubble. But it will be smaller than if we would have to wait just to even start rendering to RTT0.

No, because the swapchain was not acquired by the present engine during that time.

But if all the images in the swapchain are currently waiting on the next V-Sync interval, how do we hand off the framebuffer? We can't reuse it until we perform the copy, and we need something to copy to.

The copy from step 3 can roughly take 0.5ms (give or take depending how fast the GPU is and screen resolution). The presentation itself can take anywhere from 0 to 33.33ms to be released.

This is my main concern. In cases where the swapchain is full (which is going to be all the time if the GPU load is light), we cannot perform that copy until we acquire a new image. Because of this wait, we can't execute the command list for nearly a full V-Sync period. And because the command execution is delayed, we have to wait another V-Sync to get the final results on-screen instead of doing everything within the same V-Sync period.

If the swapchain hasn't yet been released, there will be a bubble. But it will be smaller than if we would have to wait just to even start rendering to RTT0.

I'd think the bubble would be between executing the command list and presenting. The fact that it isn't, and command execution is still waiting on V-Sync (at least on my system) seems to imply that it's not quite working as expected.

Thinking about this some more, it's quite possible that this may just be solved with waitable swapchain, since we don't even bother to check if a swapchain image is available before we poll input and run game logic for the next frame. But we might get more throughput if we can ensure there's always an available framebuffer.

Please don't tag me randomly though, it just creates a lot of unnecessary noise (also I had already made these corrections lol)

Thanks!


[Program]: Write a R program to find lcm of two numbers by recursion

Description
Write a R program to find lcm of two numbers by recursion
LCM (Least Common Multiple) of two numbers is the smallest number that can be divided by both numbers. For example, the LCM of 15 and 20 is 60 and LCM of 10 and 15 is 30.
Input  : 12 18
Output : 36

How to contribute

Save the solution in program/program/find-lcm-of-two-numbers-by-recursion/find_lcm_of_two_numbers_by_recursion.r file

Kindly assign me to this

!asssign


[Doc] Add KoP authentication

Doc architecture

@BewareMyPower @Jennifer88huang @Huanli-Meng could you please help review? Thanks


Replace node-find-files with glob

glob is much more widely used

node-find-files seemed to have a race that caused the node test to
fail consistently

Also:

removed the skip on the node files

Change-Id: I081177f792ab858bb46349997fa95e1da427a66f

Current coverage is 91.10%

Merging #13 into master will increase coverage by +4.90% as of a3f7f46

@@            master    #13   diff @@
=====================================
  Files            3      3
  Stmts          500    472    -28
  Branches        74     71     -3
  Methods          0      0
=====================================
- Hit            431    430     -1
  Partial          0      0
+ Missed          69     42    -27

Review entire Coverage Diff as of a3f7f46

Powered by Codecov. Updated on successful CI builds.


Deprecate Speak (except on AdaptiveCard)

Obsolete attribute added to Speak property on ActionBase, CardElement, ChoiceSet and FactSet.
HtmlRenderer and WPFVisualizer still support .Speak on these types for backward compatibility (should maybe show/speak something about them being deprecated?).

@EricDahlvang,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

sdgf


User didn’t know national insurance number or wouldn’t have their nation insurance numbers to hand

It's a bug in the prototype. Closing this as duplicate of #544


Invalid memory accesses after returning raxIterator from function

If a raxIterator is built and returned by a function, Valgrind reports invalid frees on values below stack pointer on a subsequent call to raxStop.
This can be reproduced with the sample:
#include "rax/rax.h"

static raxIterator _prepareIterator(rax *map) {
	raxIterator iter;
	raxStart(&iter, map);
	raxSeek(&iter, "^", NULL, 0);
	return iter;
}

int main() {
	rax *empty = raxNew();
	raxIterator it = _prepareIterator(empty);

	raxStop(&it);
	raxFree(empty);

	return 0;
}

Which causes the Valgrind output:
==6980== Invalid free() / delete / delete[] / realloc()
==6980==    at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6980==    by 0x10C595: raxStop (in /home/tower-linux/dev/scratch/rax_test/empty_test)
==6980==    by 0x108C14: main (empty_test.c:14)
==6980==  Address 0x1ffefff040 is on thread 1's stack
==6980==  368 bytes below stack pointer
==6980==
==6980== Invalid free() / delete / delete[] / realloc()
==6980==    at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6980==    by 0x108C14: main (empty_test.c:14)
==6980==  Address 0x1ffefff0e0 is on thread 1's stack
==6980==  224 bytes below stack pointer

Is this an incorrect usage of raxIterators?

There is a array in raxIterator, raxIterator it = _prepareIterator(empty) wouldn't copy this array, that makes it->key != it->key_static_string equal true in raxStop, and then attempt to free a stack variable.

Understood. Thanks for the clarification, @zhuer91 !


http://<IP_ADDRESS>:8000/chart.html

Edit 16/11/17: Issue not resolved. I was able to run the code till the end (verified by "Done. Goto <IP_ADDRESS>:8000/chart.html"), and went into http://MYPCNAME:8000 but I could not find any "Chart" file in the server... Help from anyone??

Same issue here

I got a feeling the .json file created can be or was meant to feed into a html interactive chart, but i don't think  that process is automated... gotta find a way to make it work

yeah ok so a silly error really, chart.html file is in the ml folder but my dumbass didn't see it. However I'm trying to load the file but no chart appears so I'm trying to work that out.

Did either of you figure out how to get the chart to display?  I am just getting a white page when it loads.


Fix version data

I have made better the way to loading current version of project. It shares version info to setup.cfg and budoux/main.py.
When you would like to update projects version, please modify the version in budoux/__init__.py.

That's a good point. Thank you!