pe: cache DataAfterSection for performance

cache the results of DataAfterSection since they're expensive to compute (lots of data copied around). instead, trade this for higher memory usage (expected to be O(number of (sections * file size) / 2)).
before:
❯ time ./GoReSym 11d7cb5750c44c40e767c7c4fa0f388ed64f636cf6b97ceee7bf9ae77683a3c0 > /dev/null
2023/07/21 13:18:58 profile: cpu profiling enabled, cpu.pprof
GoReSym: profile: cpu profiling disabled, cpu.pprof

________________________________________________________
Executed in   15.20 secs    fish           external
   usr time   15.01 secs  151.00 micros   15.01 secs
   sys time    0.87 secs  136.00 micros    0.87 secs

❯ pprof -cum -top cpu.pprof
File: GoReSym
Type: cpu
Time: Jul 21, 2023 at 1:18pm (CEST)
Duration: 15.19s, Total samples = 15.42s (101.50%)
Showing nodes accounting for 14.99s, 97.21% of 15.42s total
Dropped 88 nodes (cum <= 0.08s)
      flat  flat%   sum%        cum   cum%
         0     0%     0%     14.67s 95.14%  main.main
         0     0%     0%     14.67s 95.14%  runtime.main
     0.01s 0.065% 0.065%     14.40s 93.39%  main.main_impl
         0     0% 0.065%     12.78s 82.88%  github.com/mandiant/GoReSym/objfile.(*Entry).PCLineTable
         0     0% 0.065%     12.78s 82.88%  github.com/mandiant/GoReSym/objfile.(*File).PCLineTable (inline)
         0     0% 0.065%     12.65s 82.04%  github.com/mandiant/GoReSym/objfile.(*peFile).pcln
     0.29s  1.88%  1.95%     12.65s 82.04%  github.com/mandiant/GoReSym/objfile.(*peFile).pcln_scan
     4.57s 29.64% 31.58%     12.14s 78.73%  github.com/mandiant/GoReSym/objfile.findAllOccurrences (inline)
     1.03s  6.68% 38.26%      7.57s 49.09%  bytes.Equal (inline)
     5.48s 35.54% 73.80%      5.48s 35.54%  memeqbody
         0     0% 73.80%      1.47s  9.53%  github.com/mandiant/GoReSym/objfile.(*Entry).ModuleDataTable
         0     0% 73.80%      1.47s  9.53%  github.com/mandiant/GoReSym/objfile.(*File).ModuleDataTable
         0     0% 73.80%      1.47s  9.53%  github.com/mandiant/GoReSym/objfile.(*peFile).moduledata_scan
         0     0% 73.80%      1.39s  9.01%  github.com/mandiant/GoReSym/debug/pe.(*File).DataAfterSection
     1.06s  6.87% 80.67%      1.06s  6.87%  runtime.memequal
     0.79s  5.12% 85.80%      0.79s  5.12%  runtime.memmove
         0     0% 85.80%      0.79s  5.12%  runtime.systemstack
         0     0% 85.80%      0.72s  4.67%  runtime.gcBgMarkWorker
         0     0% 85.80%      0.72s  4.67%  runtime.gcBgMarkWorker.func2
         0     0% 85.80%      0.72s  4.67%  runtime.gcDrain
         0     0% 85.80%      0.56s  3.63%  github.com/mandiant/GoReSym/debug/pe.(*Section).Data
         0     0% 85.80%      0.53s  3.44%  runtime.growslice

after:
❯ time ./GoReSym 11d7cb5750c44c40e767c7c4fa0f388ed64f636cf6b97ceee7bf9ae77683a3c0 > /dev/null
2023/07/21 14:56:40 profile: cpu profiling enabled, cpu.pprof
GoReSym: profile: cpu profiling disabled, cpu.pprof

________________________________________________________
Executed in    7.34 secs    fish           external
   usr time    7.15 secs  151.00 micros    7.15 secs
   sys time    0.10 secs  141.00 micros    0.10 secs

❯ pprof -cum -top cpu.pprof
File: GoReSym
Type: cpu
Time: Jul 21, 2023 at 2:56pm (CEST)
Duration: 7.33s, Total samples = 7.15s (97.58%)
Showing nodes accounting for 6.99s, 97.76% of 7.15s total
Dropped 66 nodes (cum <= 0.04s)
      flat  flat%   sum%        cum   cum%
         0     0%     0%      7.11s 99.44%  main.main
         0     0%     0%      7.11s 99.44%  runtime.main
     0.01s  0.14%  0.14%      6.88s 96.22%  main.main_impl
         0     0%  0.14%      6.72s 93.99%  github.com/mandiant/GoReSym/objfile.(*Entry).PCLineTable
         0     0%  0.14%      6.72s 93.99%  github.com/mandiant/GoReSym/objfile.(*File).PCLineTable (inline)
         0     0%  0.14%      6.62s 92.59%  github.com/mandiant/GoReSym/objfile.(*peFile).pcln
     0.04s  0.56%   0.7%      6.62s 92.59%  github.com/mandiant/GoReSym/objfile.(*peFile).pcln_scan
     2.52s 35.24% 35.94%      6.55s 91.61%  github.com/mandiant/GoReSym/objfile.findAllOccurrences (inline)
     0.47s  6.57% 42.52%      4.03s 56.36%  bytes.Equal (inline)
     3.11s 43.50% 86.01%      3.11s 43.50%  memeqbody

Thanks for the performance improvements Willi! This has been a long time goal, I'm impressed by the results so far. I will review and merge these as soon as I can

I also had an implementation that copied all the section data into a single contiguous buffer and passed this around. It was approximately as fast without the additional copies. However, I was working against v2.3 due to not having yara and v2.4 working, and there were merge conflicts when updating to v2.4, so all I have is this PR.
sidebar: I would recommend removing the yara dependency, but we can discuss this elsewhere.

I noticed some issues with this PR. The new dataAfterSection routine doesn't actually get all the data after a given section, it only returns the data of the particular found section. We also need to do this optimization for the other OS's


Error saving very large LightGBM models

How you are using LightGBM?
Python package
LightGBM component:
Environment info

Operating System: Windows 10
CPU/GPU model: GPU
C++ compiler version: NA
CMake version: NA
Java version: NA
Python version: 3.6.6
R version: NA
Other: NA

LightGBM version or commit hash: 3.1.0
Error message and / or logs
I'm observing errors when trying to train models with sufficiently large tree models (on either CPU or GPU). Namely, when the max_leaves and num_boosting_rounds are sufficiently high, the boosting rounds all finish, but when trying to serialise and deserialise the model back, an error occurs.
To avoid the automatic to and from_string calls after the final boosting round, I've tried setting keep_training_booster=True and then saving the model out to disk, then reloading it. Saving the model as text or as pickle both succeed on save but then fail on model load.
I've investigated this issue and found that when writing out to a text file the last tree written is "Tree=4348" even though I've requested more boosting rounds than this. When loading the model there's obviously a mismatch between the the number of elements in the "tree_sizes" attribute of the file (5000) and the actual number of trees in the file (4348) which causes an error
I believe the underlying issue is the same as here: https://github.com/microsoft/LightGBM/issues/2828
I also found this comment alluding to a 2GB limit of string stream and my text file is almost exactly 2GB: https://github.com/microsoft/LightGBM/issues/372#issuecomment-325695953
I added some of my own logging inside the lightgbm python layer and have the following logs

....
[LightGBM] [Debug] Trained a tree with leaves = 5000 and max_depth = 38
[LightGBM] [Debug] Trained a tree with leaves = 5000 and max_depth = 43
[LightGBM] [Debug] Trained a tree with leaves = 5000 and max_depth = 46
CASEY: Finished final boosting iteration
Training complete: 21154.90s
Attempting to save model as pickle
Attempting to convert model with default buffer len 1048576
Creating string buffer for actual len<PHONE_NUMBER>)
Converting model with actual buffer len<PHONE_NUMBER>)
Converted model to string
Decode model string to utf-8
Successfully saved model as pickle
Attempting to load model from pickle
<program hangs and pulls up dialog box indicating python has stopped working>

Reproducible example(s)
Note the model must be really large to observe this error. This took almost 6 hours on a V100 GPU. If model size is not dependent on number of rows or columns, you might be able to use smaller numbers than I did and speed things up a little.
Before getting to enough boosting rounds for the model to crash, performance of the model continues to increase so there's reason to believe a model this big is really necessary.
n = int(2e7)
m = 250
max_leaves = 5000
max_bin = 255
x_train = np.random.randn(n, m).astype(np.float32)
A = np.random.randint(-5, 5, size=(m, 1))
y_train = (x_train @ A).astype(np.float32)

print(f"x_train.shape = {x_train.shape}, y_train.shape = {y_train.shape}")
n_boosting_rounds = 5000
params = {
    'task': 'train',
    'boosting_type': 'gbdt',
    'objective': 'regression',
     'device': 'gpu',
    'metric': {'rmse'},
    'device': 'gpu',
    'num_leaves': max_leaves,
    'bagging_fraction': 0.5,
    'feature_fraction': 0.5,
    'learning_rate': 0.01,
    'verbose': 2,
    'max_bin': max_bin,
}
ds_train = lgb.Dataset(x_train, y_train.ravel(), free_raw_data=False).construct()
start = time.perf_counter()
gbm = lgb.train(
    params,
    ds_train,
    num_boost_round=n_boosting_rounds,
    keep_training_booster=True, # set this to False and the code will crash here
)

print(f"Training complete: {elapsed_train_time:.2f}")

model_file = f"{train_start}_model.txt"
print(f"Attempting to save model as {model_format}", flush=True)
gbm.save_model(model_file)
print(f"Successfully saved model as {model_format}!", flush=True)

print(f"Attempting to load model from {model_format}", flush=True)
gbm = lgb.Booster(model_file=model_file) # program dies here!!
print(f"Successfully loaded model from {model_format}", flush=True)

Steps to reproduce

Generate some fake linear data
Train a gbdt with sufficient boosting_rounds and max_leaves to cause an error (note the boosting is fine, it's the clean up bit after boosting that's problematic)

Hi @CHDev93 !
We already have a feature request to support extremely large models: #2265.
Are errors you observe the same as there?

Hi @CHDev93 !
We already have a feature request to support extremely large models: #2265.
Are errors you observe the same as there?

Thanks for the quick follow @StrikerRUS . Don't know how I missed this issue when looking through tickets the past couple days. Yes this problem looks very similar, though I don't get an error as helpful as the one shown in that ticket (or at all, actually).
When I've set keep_training_booster=True as you mention in that issue, that does allow the lgb.train call to succeed but then I still have no way to use persist the model. Basically, if I set this flag, how can I put the model on disk such that I can load it back and make predictions? This comment seems to imply saving and loading should work but in practice I'm finding the file is corrupt https://github.com/microsoft/LightGBM/issues/2265#issuecomment-514597938

Thanks for the quick follow @StrikerRUS . Don't know how I missed this issue when looking through tickets the past couple days. Yes this problem looks very similar, though I don't get an error as helpful as the one shown in that ticket (or at all, actually).
When I've set keep_training_booster=True as you mention in that issue, that does allow the lgb.train call to succeed but then I still have no way to use persist the model. Basically, if I set this flag, how can I put the model on disk such that I can load it back and make predictions? This comment seems to imply saving and loading should work but in practice I'm finding the file is corrupt https://github.com/microsoft/LightGBM/issues/2265#issuecomment-514597938

From your logs:
...
Attempting to convert model with default buffer len 1048576
Creating string buffer for actual len<PHONE_NUMBER>)
Converting model with actual buffer len<PHONE_NUMBER>)
...

2147483648 it is exactly INT_MAX + 1. I don't think this is a coincidence...

This comment seems to imply saving and loading should work but in practice I'm finding the file is corrupt #2265 (comment)

Unfortunately, it is only a potential fix. It is not implemented yet. This feature request is still open.
#2302

From your logs:
...
Attempting to convert model with default buffer len 1048576
Creating string buffer for actual len<PHONE_NUMBER>)
Converting model with actual buffer len<PHONE_NUMBER>)
...

2147483648 it is exactly INT_MAX + 1. I don't think this is a coincidence...

This comment seems to imply saving and loading should work but in practice I'm finding the file is corrupt #2265 (comment)

Unfortunately, it is only a potential fix. It is not implemented yet. This feature request is still open.
#2302

Okay was just checking there wasn't a work-around involving this keep_training_booster combined with some method of getting it to disk that would work around this issue. If I understand correctly, there's no way around it at the current time.
Thanks for your help on this. Feel free to close this issue

Okay was just checking there wasn't a work-around involving this keep_training_booster combined with some method of getting it to disk that would work around this issue. If I understand correctly, there's no way around it at the current time.
Thanks for your help on this. Feel free to close this issue

Setting keep_training_booster=True is a workaround to complete training successfully.
I'm quite surprised that

Saving the model as text or as pickle both succeed on save but then fail on model load.

for part about pickle. Maybe you hit the following pickle issue or something similar? Could you please try joblib or some other alternative to pickle?

Setting keep_training_booster=True is a workaround to complete training successfully.
I'm quite surprised that

Saving the model as text or as pickle both succeed on save but then fail on model load.

for part about pickle. Maybe you hit the following pickle issue or something similar? Could you please try joblib or some other alternative to pickle?

Thanks @StrikerRUS ! I did know pickle has some issues at the 4GB limit but thought I might be safe at 2GB. I will kick off a run now with joblib to see if that helps.
I'm not certain exactly how these serialisation libraries work so hopefully they're not calling some of the objects methods during serialisation which could lead to the string conversion issue again. Will comment here when I have some results though

Thanks @StrikerRUS ! I did know pickle has some issues at the 4GB limit but thought I might be safe at 2GB. I will kick off a run now with joblib to see if that helps.
I'm not certain exactly how these serialisation libraries work so hopefully they're not calling some of the objects methods during serialisation which could lead to the string conversion issue again. Will comment here when I have some results though

Reran my minimum working example above with the below replacement code in the save portion (also reduced the data size to 3e6 and learning rate to 0.001 which just speeds up the cycle time but should keep the model size the same)
...
model_file = f"{train_start}_model.pkl"
print(f"Attempting to save model as {model_format}", flush=True)
with open(model_file, 'wb') as fout:
    joblib.dump(gbm, fout)

print(f"Attempting to load model from {model_format}", flush=True)
with open(model_file, 'rb') as fin:
    gbm = joblib.load(fin)
print(f"Successfully loaded model from {model_format}!", flush=True)

Boosting does again finish, saving works, and then loading the model causes the python process to crash
[LightGBM] [Debug] Trained a tree with leaves = 5000 and max_depth = 25
[LightGBM] [Debug] Trained a tree with leaves = 5000 and max_depth = 25
[LightGBM] [Debug] Trained a tree with leaves = 5000 and max_depth = 23
[LightGBM] [Debug] Trained a tree with leaves = 5000 and max_depth = 25
Finished final boosting iteration
Training complete: 5838.44
Attempting to save model as joblib
Attempting to convert model with default buffer len 1048576
Creating string buffer for actual len<PHONE_NUMBER>)
Converting model with actual buffer len<PHONE_NUMBER>)
Converted model to string
Decode model string to utf-8
Attempting to load model from joblib

Reran my minimum working example above with the below replacement code in the save portion (also reduced the data size to 3e6 and learning rate to 0.001 which just speeds up the cycle time but should keep the model size the same)
...
model_file = f"{train_start}_model.pkl"
print(f"Attempting to save model as {model_format}", flush=True)
with open(model_file, 'wb') as fout:
    joblib.dump(gbm, fout)

print(f"Attempting to load model from {model_format}", flush=True)
with open(model_file, 'rb') as fin:
    gbm = joblib.load(fin)
print(f"Successfully loaded model from {model_format}!", flush=True)

Boosting does again finish, saving works, and then loading the model causes the python process to crash
[LightGBM] [Debug] Trained a tree with leaves = 5000 and max_depth = 25
[LightGBM] [Debug] Trained a tree with leaves = 5000 and max_depth = 25
[LightGBM] [Debug] Trained a tree with leaves = 5000 and max_depth = 23
[LightGBM] [Debug] Trained a tree with leaves = 5000 and max_depth = 25
Finished final boosting iteration
Training complete: 5838.44
Attempting to save model as joblib
Attempting to convert model with default buffer len 1048576
Creating string buffer for actual len<PHONE_NUMBER>)
Converting model with actual buffer len<PHONE_NUMBER>)
Converted model to string
Decode model string to utf-8
Attempting to load model from joblib

Ah OK, I see now.
Trained Booster calls self conversion to string internally during serialization.
https://github.com/microsoft/LightGBM/blob/d4658fbb6fe943e9afee8c639934339cff38fd90/python-package/lightgbm/basic.py#L2288-L2295
So I'm afraid that without implemented workaround for #2265 it is not actually possible to save trained model in binary format too.

Ah OK, I see now.
Trained Booster calls self conversion to string internally during serialization.
https://github.com/microsoft/LightGBM/blob/d4658fbb6fe943e9afee8c639934339cff38fd90/python-package/lightgbm/basic.py#L2288-L2295
So I'm afraid that without implemented workaround for #2265 it is not actually possible to save trained model in binary format too.

Hmmm, however, this issue is marked as closed.
https://bugs.python.org/issue16865

Python version: 3.6.6
Maybe you could try newer Python version?

Hmmm, however, this issue is marked as closed.
https://bugs.python.org/issue16865

Python version: 3.6.6
Maybe you could try newer Python version?

Updated to python 3.8.7 and ran one job with keep_training_booster=True and one with the default. First one fails after trying to load the model after dumping it to disk with joblib. Second one fails after finishing boosting but before the code I have to save the model.
Forgot to add back all the print statements I had put into the lightgbm python so don't have any more details but I'm fairly confident this is the same issue as with 3.6.6. The pickle file is also 2,097,153KB (so very close to 2GB as before).
Also of note is this job took 5x longer than my 3.6.6 job so not sure if something changed. Looked through the params and they all look the same

Updated to python 3.8.7 and ran one job with keep_training_booster=True and one with the default. First one fails after trying to load the model after dumping it to disk with joblib. Second one fails after finishing boosting but before the code I have to save the model.
Forgot to add back all the print statements I had put into the lightgbm python so don't have any more details but I'm fairly confident this is the same issue as with 3.6.6. The pickle file is also 2,097,153KB (so very close to 2GB as before).
Also of note is this job took 5x longer than my 3.6.6 job so not sure if something changed. Looked through the params and they all look the same

OK, got it! Thanks a lot for a lot of details! I'm going to link this issue to the feature request of supporting huge models so that they will be available there.

OK, got it! Thanks a lot for a lot of details! I'm going to link this issue to the feature request of supporting huge models so that they will be available there.

An additional data point: I had a similar issue that was fixed by setting keep_training_booster=True, except python would crash with no error (whether at the terminal or jupyter kernel).
I could train in R and command line, but loading the model out put by lightgbm.exe crashed python too, which led me to finding this solution in the repo.
R could train the model, but if I tried to save the model for input into python (or load the model trained externally by lightgbm.exe) R crashed.

@StrikerRUS  Although this issue is closed, I'll leave this here for reference in case there are plans for a fix
The original issue was seen on Windows with 5000 leaves and 5000 boosting rounds being sufficient to observe the problem consistently on data of shape (3e6, 250). I reran the same experiment on a Linux machine with the CPU using both 5000 boosting rounds and 8000 boosting rounds. Both models produced an output text file over 2GB (which I never observed on Windows) and didn't produce any python crashes.
The larger of the two files was 3.7GB and I manually checked the tail of the file and found "Tree=7999" indicating the full model is contained without the truncation I found previously.
All of this strongly suggests this is the issue as another user referenced in a previous comment

One more thing to add to this is that I can train a huge model on Linux (larger than 2GB), then load the model in Windows and do inference. I cross referenced the predictions with the Linux ones on a few thousand random data points and the L1 norm of the error is 0 so I'm fairly confident the model loaded in Windows is not corrupt (I was worried it was silently only loading 2GB of trees). The model load function appears to use string streams as well so I'm less sure about my previous hypothesis about the cause.

Forgot to add back all the print statements I had put into the lightgbm python code so I don't have any more details but I'm fairly confident this is the same issue as with 3.6.6. The pickle file is also 2,097,153KB (so very close to 2GB as before).

I can attest to the same issue. I first used pickle to save my models on disk, then reverted to "save_model" into a text file.

Both have 2097153 KB size on the windows machine I'm using to train. This means that my model can never leave RAM without being corrupt past the 2GB file size mark, which is frustrating. I might try running that on linux/docker at some point just to be able to end the training but this makes LGBM a poor choice for very large models.

Sorry, this was locked accidentally. Just unlocked it. We'd still love help with this feature!


[Suggestion] Api Docs / Github Pages

Hi,
first of all, great work with that package as you already know 👍
I just realized that we don't have any Api Docs or any nice GitHub Page which contains an overview of the project.
So maybe it might be a nice addition to the repository.
Feel free to hit me up at work so we can talk about it in more details if you want to 🙂
Regards,
Patrick

Hey,
I think we actually have an API doc and an overview. 😄
I am not saying they are perfect though 🙂
Improvements are of course always welcome. Sure we can talk about that later.
Best,
Christian

@ChristianNavolskyi This is how it could look like using just the README content and one of the default themes:
Hosted on my fork
We could also go for something more custom that looks nicer, what do you think?

Hey Patrick,
I think that looks actually very nice.
The missing GIF are of course a bummer, do you think that is easily fixable? Maybe just the link is broken or something.
Best,
Christian


Improving shots estimate to give errors within user-specified precision.

Summary
Fixes #issue-12414 (https://github.com/Qiskit/qiskit/issues/12414). When the user specifies the precision in EstimatorV2.run, the actual standard deviation from results is not bounded by or close to the user-specified precision for an observable that is a general linear combination of Pauli strings. This pull request gives a more accurate approximation to the number of shots needed for the desired precision for such observables. The tests to verify this are based on https://docs.quantum.ibm.com/api/qiskit-ibm-runtime/qiskit_ibm_runtime.EstimatorV2 and https://github.com/Qiskit/qiskit-ibm-runtime/issues/1684.
Details and comments
The shots estimation accounts for the sum of the absolute value squared of the coefficients of the observable as proposed in https://github.com/Qiskit/qiskit/issues/12414 to give a standard deviation that is bounded by and closer to the user-specified precision for any observable. In backend_estimator_v2.py, the _run function is modified to include the sum of the absolute value squared of the coefficients of the observable in the shots estimation.
The following plot shows that the ratio of precision from results to the user-specified precision changes with varying observable term coefficients using the current implementation. In the proposed implementation, this ratio is fixed and close to 1.

The plot below shows that in the current implementation, having more than one occurrence of a term in an observable leads to precision results that are different from the user-specified precision due to a fixed number of shots for observables with different numbers of terms and error propagation. In the proposed implementation, the precision from the results remains close to the user-specified precision.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Closed in favor of #12711


How to prefetch data

I would love to improve navigation UX, with prefetching: http://dev.apollodata.com/react/prefetching.html
I have tried to load data in advance calling withDocument on an empty component so I could execute the query - I know it is the best approach, I just wanted to do a quick try-. The data gets downloaded and added to the store. However, when I am trying to access the document I have already downloaded, the page is not rendered straight away and the loading property is still set to true. After this, if I leave the document and come back the document is rendererd straight away.

Have you tried doing what the documentation says?

in GitHunt, we use the withApollo higher-order component to directly call a query as soon as the user hovers over a link to the comments page.

I don't really see a good way to do it with withDocument to be honest.


[Enhancement] log of removing expired job

What type of PR is this：

[ ] BugFix
[ ] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Which issues of this PR fixes ：

Fixes #16289
Problem Summary(Required) ：

This PR adds label of the broker load job to the log of removing expired/redundant job.
Checklist:

[x] I have added test cases for my bug fix or my new feature
[ ] This pr will affect users' behaviors
[ ] This pr needs user documentation (for new or modified features or behaviors)
[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto backported to target branch
[ ] 2.5
[ ] 2.4
[ ] 2.3
[ ] 2.2

[FE PR Coverage Check]
:heart_eyes: pass : 2 / 2 (100.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/load/loadv2/LoadManager.java
2
2
100.00%
[]

run starrocks_admit_test


Update log4j version to 2.17.1

Fix securty issues https://nvd.nist.gov/vuln/detail/CVE-2021-44832

@technoLord Any ETA on this release?

I opened an issue https://github.com/lightbody/browsermob-proxy/issues/893 to request a new release with this fix included.

Is there any update on this? when this shall be released.

looks like this project is abandoned

Yeah, I figured as much.  If anyone ever picks it up this needs to be fixed.  Until then if anyone is using this dependency, you will either need to fork and patch it locally or force maven to use only newer versions of log4j


How can I add the seal library into Clion project after using brew to install seal?

I use the command "brew install seal" to install SEAL, but when I tried to use the SEAL library in the Clion project, it went wrong.
The following is some document contents.
terminal output
"
atal error: 'seal/seal.h' file not found
#include <seal/seal.h>
^~~~~~~~~~~~~
1 error generated.
ninja: build stopped: subcommand failed.
"
main.cpp
"
#include
#include <seal/seal.h>
int main() {
std::cout << "Hello, World!" << std::endl;
return 0;
}
"
CMakeLists.txt
"
cmake_minimum_required(VERSION 3.23)
project(untitled)
set(CMAKE_CXX_STANDARD 17)
add_executable(untitled main.cpp)
set(INC_DIR /opt/homebrew/Cellar/seal/4.1.2/include)
set(LINK_DIR /opt/homebrew/Cellar/seal/4.1.2/lib)
link_libraries(seal)
target_link_libraries(untitled seal)
"

We don't maintain the Homebrew package for this. I would recommend following the (pretty detailed) README.md we wrote for installation and use instructions.


Feature request: add a flag or command to force initial source screenshots regeneration

Hi,
It would be great to have something that will regenerate source screenshots from command line out of the box, like casperjs reference test.js.
It's possible to force source screenshots regeneration only by deleting them manually or using Node.js in test.
Sergey

Thanks for the suggestion. I've added an option, you can use that with casper.cli.get("reference")  for casperjs test testfile.js --reference.  Take a look at the demo test suite for a clearer idea.


RSR-232 COMPAS : attributes to consider that Data Type Templates are different or identical one

Context : this US is to define the key attributes which defines that Data Type Template are different or identical. This US is an evolution of RSR-6 import DTT function of ICD in SCD.
Here under for each Data Type Templates, all the following attributes values have to be equal for attribute value and equal for node value which are tagged as "value of the node" ELSE the Data Type Template is consider as different one and an id attribute is renamed as defined in RSR-6.
In LNodeType, the order of DO.name node are not considered in the comparison ; as a consequence, same DO node but with different order will be considered as the same LNodeType.
In DOType, the order of SDO.name and DA.name are defined by the norm and has to be respected.
In DAType, the order of BDA.name are defined by the norm and has to be respected.
In EnumType, the order of EnumVal are defined by the norm and has to be respected.
In all kind of objects, the private sub nodes are not part of the comparison. This happen if xml fragments are in the Privates.
For LNodeType :

LNodeType.id
LNodeType.lnClass
LNodeType.iedType
DO.name
DO.type
DO.accessControl
DO.transient

For DOType :

DOType.id
DOType.cdc
DOType.iedType
SDO.name
SDO.type
SDO.count
DA.name
DA.fc
DA.dchg
DA.qchg
DA.dupd
DA.sAddr
DA.bType
DA.valKind
DA.type
DA.count
DA.valImport
DA.Val (value of the node)
DA.Val.sGroup
DA.ProtNs (value of the node)
DA.ProtNs.type

For DAType :

DAType.iedType
DAType.id
BDA.name
BDA.sAddr
BDA.valKind
BDA.valImport
BDA.type
BDA.bType
BDA.count
BDA.Val (value of the node)
BDA.Val.sGroup
ProtNs (value of the node)
ProtNs.type

For EnumType,

EnumType.id
EnumVal (value of the node)
EnumVal.ord

Private are not included into the comparison of DTT because it can include a complex object like xml

This issue can be closed because merge with https://github.com/com-pas/compas-sct/issues/5


UI, data cleanup and removing unusable functionality from the UI

Things like code annotations are meant to be working in the next milestone.
Until then we should simplify the UI and remove any data or functionality which is not working and might cause confusion.

[x] Remove "annotate code" buttons / links / mentions.
[x] Look for any other functionality or links which can be cut down/trimmed down for now.
[x] Remove broken or incomplete entries on the index page.

Deleted all test entries and broken pending review entries.

Deleted all test entries and broken pending review entries.


Added a function to get resource by business group with example

Added a function to get resource by business group with example - having a function like this would allow people to inject tags in thier vm description and organise the vm based on that tag group similar to AWS dynamic inventory,

Nice job!


Defaut ordering of modular should be folder, not date

Hi !
Currently it seems default ordering for modular pages is by date. I think it doesn't make much sense to sort content by date for modulars, and it's also quite disorientating for new users, as page order will seem to change randomly upon saving files.
I'd suggest to change default ordering to folder name.
Thanks for the great work !

ordering in the admin is based on if you have a content.order.by in your page header, it will try to order by that type.  However this doesn't work well for modular page with custom ordering.  The best solution I've found is to simply use folder prefix ordering, with no order.by:
01._showcase
02._text
...


Feature Request: Notify with Image or Camera

Right now I can trigger a notify event with a specific icon.  I'd like to be able to use an image and ideally a camera feed.
Use case: When I arrive home, send a "Jesse is Home" notification with my picture
Use case: When I open the garage, send a notification with a camera feed of the garage

Notification should not include an image. What you are looking for is probably to open door entry with camera feed for 120 seconds when certain event occurs.

for "icon", is there a way to use images instead of mdi icons?  That would solve this feature for me as well as allow me to use icons outside to materialdesignicons.  Thanks

Sure, just create your own css rule for each of your own icon. mid-xxx is simply a css class


In terminal Vim, execute() does not capture the output of :write_c

Steps to reproduce
vim --clean -c 'call setline(1, "asdf")' -c 'echom execute("w !cat") =~ "asdf"'

It prints 0 to :mess.
Expected behaviour
1 should be printed. That is, execute("w !cat") should capture the output "asdf". GVim and Neovim work as expected.
Version of Vim
9.0.0150
Environment

Ubuntu 22.04
GNOME Terminal 3.44.0 using VTE 0.68.0 +BIDI +GNUTLS +ICU +SYSTEMD
$TERM: tmux-256color
shell: bash

Logs and stack traces
No response

This is to be expected.  execute() returns the output of the Vim command, but the "!cat" will invoke a shell to execute "cat" and the output goes to stdout, not to Vim.  Thus it goes to the terminal.
To capture the output of a shell command use system().
In gvim it works, because the shell output is captured to be able to display it in the Vim window.

Understood. Thanks!


Can't install following the web instructions

Following this part of the docs, it states that to install pinnwand we should type: python3 -m venv virtual-environment virtual-environment/bin/pip install pinnwand.
But whenever I try that, I got: Error: Unable to create directory '/home/pastebins/pinnwand-1.5.0/virtual-environment/bin/pip' and therefore I can't use virtual-environment/bin/pinnwand http to start pinnwand.
I've also tried pip3 install --user pinnwand but then I can't find the executable anywhere.
I'm a bit lost here.
EDIT: I can't use docker.
Cheers

I think that's an error with how the docs are being displayed,
python3 -m venv virtual-environment virtual-environment/bin/pip install pinnwand is displayed on one line but it's actually two commands.
python3 -m venv virtual-environment

and
virtual-environment/bin/pip install pinnwand

Oh, that makes much more sense.
I've managed to install it from source and I must say that I'm quite impressed with this piece of software.
Great job!

That's a nice bug, thanks for PR #223 @wookie184 which fixes this.


remove +x permissions on some files

FYI, Debian static analyzer (lintian) is complaining about that

W: librust-futures-core-dev: executable-not-elf-or-script usr/share/cargo/registry/futures-core-0.2.1/src/task/atomic_waker.rs

Weird-- good catch!


pkg/buildqueue: remove unused package

It seems this package is not being used anywhere and so we can remove
it.
Signed-off-by: André Martins<EMAIL_ADDRESS>

This change is

test-me-please

test-me-please

(Marked as backported needed because #7896 will depend on it)

Coverage decreased (-0.2%) to 41.333% when pulling c4222bf4be3aef5d89870c3fab554842750165c8 on pr/remove-pkg/buildqueue into 7aed47b0c72da4b6033702b59d197dbe72782076 on master.


Renamed objects get moved to root

When renaming objects (files or folders) they get renamed and move to the root of the SD

#7


[Machine] Concurrent requests to same resource does not update

I am not 100% sure what the root cause of this and will update this if I can figure it out.  The sequence of events that caused this for me was:

in go-machine-server, there is an API update to machine resource to add the "machineDir" parameter within the data field.
On a separate go routine, there is a ticker that is periodically sending transitioning messages for the same machine resource.  If this updates too close to the previous machine update, "machineDir" never gets added or gets clobbered.

If you submit multiple requests to create events, the same thing happens.

Reply event 1
Reply event 2

Only event 1 gets updated while event 2 seems to be ignored/dropped.  If I put a sleep in between the two events will event 2 get updated.

@will-chan any chance that this and #203 are relate?

@will-chan @cjellick Was this resolved by doing +data?

Ping @will-chan we can close this?


Update SVG icons

Summary
Fixes two SVGs that contained encoded PNGs and thus were very large.
Also runs all other SVGs through ImageOptim (yes, we already have SVGO during build, but just in case)

See #868

The failing E2E test seems unrelated / intermittent.


Add unit tests for cursor

@kiplingw Looks like this build doesn't include my fix:
...
[ RUN      ] stream.test_cursor_offset_for_array
/<<PKGBUILDDIR>>/tests/stream_test.cc:149: Failure
Value of: strcmp(cursor.offset(), "efgh")
  Actual: false
Expected: true
[  FAILED  ] stream.test_cursor_offset_for_array (0 ms)
...

There is no strcmp call in /tests/stream_test.cc:149 after my fix:
...
   147	  size_t shift = 4u;
   148	  cursor.advance(shift);
   149
   150	  std::string result{cursor.offset(), strlen(data) - shift};
   151	  ASSERT_EQ(result, "efgh");
   152	}
...

Hey @hyperxor. I suggest you consider monitoring the builds here, and maybe consider joining the Launchpad team to get the automated failed build notifications.

@kiplingw Hello, I have found all pistache builds on Launchpad and I see that ppc64el is failed but in another test:
23/24 Test #12: http_server_test .................   Passed   30.04 sec
24/24 Test #24: https_server_test ................***Timeout 1500.11 sec
Running main() from /build/googletest-DTPjhg/googletest-1.8.1/googletest/src/gtest_main.cc
[==========] Running 6 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 6 tests from http_client_test
[ RUN      ] http_client_test.basic_tls_request
[       OK ] http_client_test.basic_tls_request (46 ms)
[ RUN      ] http_client_test.basic_tls_request_with_auth
[       OK ] http_client_test.basic_tls_request_with_auth (48 ms)
[ RUN      ] http_client_test.basic_tls_request_with_auth_no_client_cert
70367077527296:error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate:../ssl/statem/statem_srvr.c:3678:
[       OK ] http_client_test.basic_tls_request_with_auth_no_client_cert (5 ms)
[ RUN      ] http_client_test.basic_tls_request_with_auth_client_cert_not_signed

96% tests passed, 1 tests failed out of 24

Total Test time (real) = 1501.61 sec

The following tests FAILED:
	 24 - https_server_test (Timeout)

it's not related to my changes at all. It's a problem with cerificate, may @dennisjenkins75 knows about it?

That's a test that has been failing on and off for a long time. It's likely a race condition that pops up under unknown circumstance. Differences in thread scheduler implementations across different platforms, in this case maybe a kernel for ppc64el hardware, sometimes make it apparent on occasion.


test 16 blocks pillar interval

Purpose

How does the solution address the problem

Changes made

/deploy

PR NET pr-2795 Successfully deployed!explorerhttps://explorer-pr-2795.prnet.taraxa.io  RPChttps://rpc-pr-2795.prnet.taraxa.io Indexerhttps://indexer-pr-2795.prnet.taraxa.io Boot NodesBoot Node <IP_ADDRESS>Boot Node 2<IP_ADDRESS>Boot Node 3<IP_ADDRESS> RPC NodesRPC Node <IP_ADDRESS>RPC Node <IP_ADDRESS>RPC Node 3<IP_ADDRESS>

/cleanup

prnet cleaned-up successfully

/deploy

PR NET pr-2795 Successfully deployed!explorerhttps://explorer-pr-2795.prnet.taraxa.io  RPChttps://rpc-pr-2795.prnet.taraxa.io Indexerhttps://indexer-pr-2795.prnet.taraxa.io Boot NodesBoot Node <IP_ADDRESS>Boot Node <IP_ADDRESS>Boot Node 3<IP_ADDRESS> RPC NodesRPC Node 1<IP_ADDRESS>RPC Node <IP_ADDRESS>RPC Node 3<IP_ADDRESS>

/clean

/cleanup

prnet cleaned-up successfully

/deploy

/cleanup

prnet cleaned-up successfully


Getting typing error with ClsStore interface

I am getting the following typing error when trying to use the package.
node_modules/nestjs-cls/dist/src/lib/cls.interfaces.d.ts:42:6 - error TS1023: An index signature parameter type must be either 'string' or 'number'.

42     [key: symbol]: any;
        ~~~

Reproduction repository: https://github.com/ruslanguns/cqrs-with-cls
System information
[System Information]
OS Version     : Linux 5.10
NodeJS Version : v16.13.1
NPM Version    : 8.3.0

[Nest CLI]
Nest CLI Version : 8.0.2

[Nest Platform Information]
platform-express version : 8.0.0
common version           : 8.0.0
core version             : 8.0.0
cqrs version             : 8.0.0

Morning FYI @joebowbeer I have created two different tests:

Using version rolling back cls to v1.6.2 and works smoothly: https://github.com/ruslanguns/cqrs-with-cls/tree/Using-CLS-1.6.2

Rolling back typescript to v4.4 and works smootly: https://github.com/ruslanguns/cqrs-with-cls/tree/RollbackTypeScript-4.4

@ruslanguns regarding 2)
Didn't you mean upgrading to typescript 4.4? I remember you saying it didn't work on 4.3. That would mean this is actually a non-issue and it's only needed to add the minimum version requirement to the readme.
However, if your PR fixes it for lower versions as well, I'll be more than happy to merge it

Thanks for clarifying and yes, sorry for my mistake I was meaning rolling up... definitely the problem happens in versions before v4.4 like v4.3.5 which I was... the change with Record allows me to work with v4.3.5 smoothly

Works as intended with typescript >= 4.4. See the linked PR for more info.


Add playbook and task for k8s UT job

This Playbook and related role will be used in upcoming k8s UT job that will create a node on PowerVS using kubetest2 tf and run the make test command for running k8s UT.
Playbook added will be used in https://github.com/ppc64le-cloud/test-infra/pull/505/files#diff-8fa974d1d2e179f5ad0a4cc4882a41fcd96356b008bc93583847b58da5b02b9aR63

how the test logs are pulled into the pod where the tests are run.

scp from node to the pod after fetching the EXTERNAL_IP of node from hosts file.
We have been doing similar thing for job periodic-kubernetes-containerd-e2e-node-tests-ppc64le here
For the UT job it would be like https://github.com/ppc64le-cloud/test-infra/pull/505/files#diff-8fa974d1d2e179f5ad0a4cc4882a41fcd96356b008bc93583847b58da5b02b9aR67


I18n::ArgumentError with i18n v0.8.2+

Steps to reproduce
https://gist.github.com/cbeer/140cb717506e97181644c59faaef92b2
Using i18n v0.8.2+, calling unauthorized_message causes an exception.
Upstream, they introduced a new check in 0.8.2: https://github.com/svenfuchs/i18n/pull/236.
Cancancan is explicitly passing a nil key: https://github.com/CanCanCommunity/cancancan/blob/2d21842d4415913cb9d15d71bfa3165087be29c4/lib/cancan/ability.rb#L225.
Expected behavior
The unauthorized message should be provided.
Actual behavior
An exception is thrown:
I18n::ArgumentError: I18n::ArgumentError
    /Users/cabeer/.gem/ruby/2.4.1/gems/i18n-0.8.3/lib/i18n.rb:152:in `translate'
    /Users/cabeer/.gem/ruby/2.4.1/gems/cancancan-2.0.0/lib/cancan/ability.rb:226:in `unauthorized_message'

System configuration
Rails version: 5.0.2
Ruby version: 2.3.4
CanCanCan version 1.17.0

I have just released I18n 0.8.4 which should fix this issue. Please try it out :)

This might be just my opinion but I think it's more CanCanCan's fault than i18n's: A call to I18n.translate with nil doesn't semantically make sense. It literally says "Give the translation of nothingness". I think what CanCanCan means to do here is use a fallback chain, but it takes a shortcut by piggybacking on the default option. What it should instead do is build up an array of potential keys and then shift off the first element as the starting key – similar to what simple_form does here: https://github.com/plataformatec/simple_form/blob/master/lib/simple_form/inputs/base.rb#L173-L189. Then i18n could be fixed accordingly as well.

I agree with @clemens , PR is welcome!

is the problem with i18n solved now @cbeer ?

I guess it is solved by the i18n update but (IMO) it's not fixed in the right way ...


issues with theme after ~5 after reloading window

I'm experiencing an issue with my custom theme after updating VSCode
System Info:
Version: 1.43.0
Commit: 78a4c91400152c0f27ba4d363eb56d2835f9903a
Date: 2020-03-09T19:34:44.548Z
Electron: 7.1.11
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Darwin x64 19.3.0
Steps to Reproduce:

install extension josef.rouge-theme
change theme to Rouge 2

Does this issue occur when all extensions are disabled?: Yes
Details

Link to video https://twitter.com/josefaidt/status/1237905816096845835?s=20

There is also some odd behavior from the inspector. It looked like some of the tokens had updated so I started updating my theme, and was able to get some of it changed but others show it's supposed to be one color (and is that color <5s of reloading the window) then flips to another.
The color flipping after window reload can be found in the video linked above

What you see is the new semantic highlighting feature updating the highlighting with resolved information from the TypeScript server. The server takes a while to load, depending on the size of the project, that's why the highlighting come in delayed.
As a result, each identifier gets colored & styled with the color of the symbol it resolves to. You can see this the easiest in the imports where you can now see what is a function, variable or type. Another example are that readonly variables ('consts') and members can get a different style that modifiable variables, if the theme has defined different colors & styles.
There are still bugs with the way symbols are classified ((help welcome)[https://github.com/microsoft/vscode/wiki/Semantic-Highlighting-Overview#the-semantic-highlighting-for-typescript--javascript-files-looks-wrong-how-can-i-debug-this]), and probably also other bugs but the color changes as such are intended.
You can turn the semantic highlighting feature off with
"editor.semanticHighlighting.enabled": false

Thank you @aeschli that setting fixed the issue. I don't mind updating my theme to reflect the changes (it really changed things to a point where the theme is mostly one color), but then I started running into issues like shown above in the screenshot where it shows two selectors, the textmate scope, and the semantic type with the updated color. Though it's not actually that color, and instead is taking the foreground of some other selector with my default theme color.
It looks to be classified properly but I'm not sure how to begin troubleshooting why the color isn't receiving the right one. Here we can see it's not being overwritten by another selector (assumably):

duplicate of #92308

@josefaidt  Somethings fishy, I will investigate

Hi, i have the same problem, i use 1mm themes and methods colors switches to the same color as variables after reloading

A fix is to put the following in your settings:
"editor.semanticHighlighting.enabled": false

Taken from the Twitter link.

The semantic highlighting code finds the wrong rule for functions (not the most specific one).
That's why the function are colored with the wrong color and so many identifiers end up looking the same. I push a fix to insiders.

Verify:

install Rouge 2 theme
set in user settings

	"editor.tokenColorCustomizations": {
		"[Rouge 2]": {
			"semanticHighlighting": true
		}
	}

verify that in function foo() { const x = 1; }, foo and x have different colors.

@aeschli should I check this in Insiders?

Verified, Insiders shows the correct token colors


[PR] Load image options

New thumb image load options for iOS
Related  pull
#318

I tried to run the project, and when I used the DefaultOption, it seemed to break the previous default behavior.
For a 3000x2002 picture:

In the 0.6.x branch, before you commit the code, entity.thumbDataWithSize (500, 500) the width and height of the image will get 748x500 thumb.
Then, I checkout to the PR branch, and use the DefaultLoadOption(500,500), then the entity.thumbDataWithSize(option) will get 3000x2002 thumb.
In your branch, I use the iOSLoadOption(500,500), the method will get 748x500.

I think this is a problem.
Expect: DefaultOption (500,500) should return no less than 500x500 images (such as 748x500).

By the way: The test picture I used is one of the pictures that comes with the iOS simulator.

I changed default values in plugin side. So now it will use default parameters as before when deliveryMode, contentMode, resizeMode was not set @CaiJingLong

Good job, thanks for your work.


Visual bugs with EditorHelpBit

Godot version:
b2f79cac9a4b69a24ccf5f2bfeaeb5b38b7db369
Issue description:

Font size is incorrect.
Text cuts off (likely due to the previous point).
Bold and underline text is not set properly.

Duplicate of #39235.


shell, web and return could be lists

It would be nice to allow shell, web and return as lists if requested (keeping the old syntax as well)

closing this due to plans to remove all non-explicit interfaces (that is: they can be both strings and list) since v3.


Update shared static analysis pipeline

What does this PR do?
A brief description of the change being made with this pull request.
Motivation
What inspired you to submit this pull request?
Additional Notes
Anything else we should know when reviewing?
Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] Make sure you discussed the feature or bugfix with the maintaining team in an Issue
[ ] Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)

Codecov Report

Merging #1622 (3dd49a3) into develop (206dcd2) will increase coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop    #1622      +/-   ##
===========================================
+ Coverage    83.57%   83.60%   +0.02%
===========================================
  Files          452      452
  Lines        15652    15649       -3
  Branches      2324     2323       -1
===========================================
+ Hits         13081    13082       +1
+ Misses        1950     1945       -5
- Partials       621      622       +1

see 24 files with indirect coverage changes

This fixes #1389


Implement DescribeCollection() interface

/assign

Already implemented by #109 since the GetCompactState() depends on this interface.


Add button for reconnecting

What's the most-correct way to do this? A switch?
Or is the expectation that users create a script which calls the service?

I think it's ok to only have the service. That even seems to be "the Home Assistant way".


screenshot with alternative image formats (PNG?)

When taking screenshots of our osciliscope pictures using robotframework we end up using jpeg files which is not optimal in our use case.

our images have 5-10 pixel values, not a single smooth transition and large areas with just one pixel value. PNG compression is efficient
jpeg artifacts are especially painful with our thin single lines

Is there a specific reason why only jpeg is supported for screenshots? Or would a PR enabling additional formats be welcome?
Thanks in advance,
Franz

use screencap library: https://pypi.org/project/robotframework-screencaplibrary/#usage

It doesn't look like there's any specific reason it "must" be jpeg.  It does get a bit more complicated to support multiple formats when working with all the filesystem checks, but all the tools used to capture the screenshots appear to support those formats.
I think an enhancement request would be the way to go about this so as to define what formats should be supported (there are a lot of them).

@chriscallan that is a good idea. how can I do that?

Docs on contributing are here

I just added the text Enhancement to the title... I hope that's all whats needed...


🐞 Cannot use dagger -s call ... terminal

What is the issue?
I was running some functions repeatedly with dagger -s to get less verbose output to my terminal. I wanted concise output to share with colleagues as a repro. My dagger -s call commands were working fine until I added terminal to the end of a call that returned a Container type. I expected it to work as normal, but dagger threw an error.
An example:
Error: response from query: input: container.from.terminal resolve: failed to forward exit code: failed to close terminal: EOF

Dagger version
dagger v0.13.3
Steps to reproduce
Repros
dagger-cache-volume-issue ➤ dagger -s call populate-cache --input "foo" terminal
Error: response from query: input: container.from.withMountedCache.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.terminal resolve: failed to forward exit code: failed to close terminal: EOF

Run 'dagger call populate-cache terminal --help' for usage.

dagger -s core container from --address alpine terminal
vs

dagger core container from --address alpine terminal

### Log output

see above

I have a similar error. I'm following the steps on this page: https://docs.dagger.io/quickstart/env/
When I use the command "dagger call build-env --source=.", Dagger works correctly. But when I add "terminal --cmd=bash" to the command, I get this error:
Error: response from query: input: container.from.withDirectory.withMountedCache.withWorkdir.withExec.terminal resolve: failed to forward exit code: failed to close terminal: EOF
This is the full console output:


add code to BspHeader to get GOAL types for shrubs

This adds support to BspHeader to unpack and print shrubbery related types. I was getting confused by some stuff in the EE asm (I had assumed this is similar to tie), but annoyingly there are a number of small differences between the BVH for tie/tfrag and shrub. I don't think it will cause any issues with later stuff, but it means we have to find the actual instances a little bit differently.  To check this output, you can run something like:
./tools/level_tools/level_dump ../iso_data/DGO/MIS.DGO

and it will print out a giant dump of all the instance/prototype info.
I moved the shrub-related game types into BspHeader.h so it is a little easier to do the unpacking. We should still keep our custom shrub related types in extract_shrub.h.
This also puts the static DMA data in Shrubbery::obj/vtx/col/stq - we'll probably need to look at this more to fully understand what the EE asm is actually sending/doing - the destination addresses/format may be in this static data for example.  I only did it for the normal shrubbery type - we should probably get this one working first and then go back and add the others if needed.

Most of the lines changed are just moving around types in BspHeader.h - I didn't put them in a very good order at first so I rearranged them in a better way.


Added Ability to expand node on hover

This functionality has been added by @pkatta and is available as of v2.18.0.


Wrapping rules for let-chains?

Feature tracking issue: rust-lang/rust#53667
Should we introduce a new guideline for wrapping let-chains?
The rationale is that let-chains allow let bindings to be anywhere in a line of code. let is no longer consitently at the left side. It is good for let bindings to stand out and be easy to discover since they introduce new variables for the following scope. Oftentimes when reading code, I look backwards with the question "where did this variable come from?"
Possible answers

No change. Wrap let expressions the same as any other expression.

Always wrap let expressions. let is only ever preceded by one token in the same line: if, while, &&, ||, etc.

Wrap let expressions if they begin N characters or more past the current indentation.
For example, a let-expression may be on the same line if it is 15 characters from the left margin. If it is 16 characters from the left margin, it must break to the next line.

Wrap let expressions if there are 2 (?) or more let expressions in the condition

Allow single_identifier && let on one line (if it fits line length limits), but require a line break before the && for any other expression. Always wrap after a let expression.

Potential concerns for each answer

The introduction of x may be hard to find in this case:if my_long_condition(foo, bar, baz) && some_more && let Some(x) = foo() {
    // lots of code...
    use_x(x);

Wrapping for this case seems a little unnecessary since it only shifts let leftward by one character:if a
    && let Some(b) = foo()
{

Added complexity to formatting logic. Difficult to follow/verify the guideline without automated tooling. The subtlety of the logic may be surprising to users.
This case is suboptimal (similar to 1):if my_function(Some(get_thing()), variable, and_another) && let Some(x) = foo() {

Other considerations
The rules for if and while will be the same.
If a let expression wraps, it will cause all of the && or ||-separated expressions to be on separate lines.
if something
    && let Some(x) = foo() && something_else { // bad - `&& something_else` should be on its own line

If there is just one let binding at the beginning, the condition may be on one line:
if let Some(foo) = do_something() && another_condition {

What if the wrapping rules had more to do with the number of items in the chain? For example, something along the lines of "if there are two or more Let expressions in the chain than always wrap". That way each Let expression gets its own line and will be more readable. If there's just a single Let it's probably easy enough to figure out what's going on, and if we need to wrap the existing wrapping rules should cover us.
So this:
if let Some(ref first) = opt && let second = first {
}

Would always become this:
if let Some(ref first) = opt
    && let second = first
{
}

But your earlier example, would remain unbroken:
if a && let Some(b) = foo() {
}

A similar rule could be introduced that states if there are 3 or more items in the chain and the Let expression isn't the first chain item then always break. So we'd break in both of the following cases:
// Let at the end
if my_long_condition(foo, bar, baz)
    && some_more
    && let Some(x) = foo()
{
    // lots of code...
    use_x(x);
}

// Let in the middle
if my_long_condition(foo, bar, baz)
  && let Some(x) = foo()
  && some_more
{
    // lots of code...
    use_x(x);
}

But we wouldn't break here:
if let Some(x) = foo() && my_long_condition(foo, bar, baz) && some_more {
}

One might argue that it's still hard to understand where x is coming from in the above so if the rule were amended to be "If there are 3 or more items in the chain and there is at least 1 Let expression than always break", then we would break:
if let Some(x) = foo()
  && my_long_condition(foo, bar, baz)
  && some_more
{
}

I agree that these rules might be confusing to some users, but I think the improved readability is also important to consider.
Let me know your thoughts!

What if the wrapping rules had more to do with the number of items in the chain?

I think the horizontal position of let is the important thing, not the number of lets. I wouldn't gamble that the following case is rare:
if my_function(Some(get_thing()), variable, and_another) && let Some(x) = foo() {

What about breaking after the && so that the lets line up nicely?

This would be a diversion from the current style guide on Binary Operations, and I don't think changing that is an option at this point. Also that would not resolve the main concern here - the case where all the conditions fit on one line.

Thanks for sharing your perspective @terhechte! @camsteffen is correct about the binops though, so that wouldn't be an available option anyway

I'd like to propose two additional variations on option (2):
2a) Option 2, with an additional requirement: always break before and after every let expression, such that let pat = expr is only ever preceded by one token in the same line (if, while, &&) and a boolean expression never appears after it:
if condition()
    && let Some(binding) = expr
    && condition2(binding) && condition3()
    && let Some(binding2) = expr2
    && condition4(binding, binding2)
{
    body();
}

2b) When a let expression is present anywhere in a condition, always break each top-level expression (whether let or boolean) onto its own line:
if condition()
    && let Some(binding) = expr
    && condition2(binding)
    && condition3()
    && let Some(binding2) = expr2
    && condition4(binding, binding2)
{
    body();
}

Of these two, (2a) would be more consistent with how we currently format boolean expressions, while (2b) seems preferable in general (and I'd propose doing that for all boolean expressions that require breaking if we ever have a rustfmt change over an edition).

I thought we already break all binary operations if there are any line breaks?

Ah, it would appear that we do, thank you.
In that case, I think this reduces to option 2.

While I understand the angle around visual scanning, I'm concerned that option 2 (always wrap) is a bit too blunt an instrument where something more surgical would be helpful.
A few questions/things to note

The formatting of the potential concern for option 2 isn't correct because it would fail to comply with the If the control line is broken for any reason, then the opening brace should be on its own line and not indented. clause for control flow expressions. It would instead have to be formatted as below. Obviously subjective, but this makes me grimace:

if a
    && let Some(b) = foo()
{
    // stuff
}

I understand the scenario that option 2 is trying to guard against (vertical scanning up through the block and then horizontal scanning to the end of the control line), however, do we really think that type of scenario will be more common than the simpler type shown above?
The existing rules in relevant contexts where these will typically appear (e.g. guards, control flows) include their own wrapping rules. So while the scenario option 2 protects against can, and certainly will, occur, it does require somewhat of a perfect storm wherein it's long enough to start being a scanning challenge, but too short to trigger those other wrapping rules (and obviously every expression in the control line would need to be single-line formattable too)

The formatting of the potential concern for option 2 isn't correct

Good point! Fixed.

it does require somewhat of a perfect storm

Yeah that may be true (it's unusual for let to actually end up very far to the right). But in a way that may be a bad thing. Your brain might get used to seeing let on the left side cause that's how it usually works out, but in the unusual case that it isn't, you might miss it. It would be interesting see statistics on that. In real code using let-chains, what is the distribution of the horizontal position of let?

In real code using let-chains, what is the distribution of the horizontal position of let?

Agreed! Not sure there's a good way to systematically inspect, but even getting feedback from individuals/teams would be helpful.
I should also note that I'm absolutely open to incorporating a config option in rustfmt to give users some flexibility over this aspect of formatting, though obviously the default behavior remains an important question.
I suppose there's also some variants on option 3, but which could incorporate factors like the total length of the control line and/or the number of elements, when let exprs are present (e.g. wrap the control line if the control line contains one or more let expressions unless the control line is not "small"). These types of rules do indeed have a higher cognitive load, but we do also have some fairly longstanding precedents e.g. various "small" measures, and chains to a degree as well

If you guys want to customize the formatting of let chains, then option 3 is the most flexible and satisfies most scenarios/objectives.

I used this feature in the compiler, and I'm in the camp 1 I think, i.e. if a && b && c { ... } should be formatted identically regardless of whether a/b/c have any lets in them or not.
It seems like the simplest variant, let in the middle of a line looks fine in practice, and I'm more interested in stabilizing this sooner than in continuing the bikeshedding.
Feature stabilization in the language being blocked on formatting decisions seems like an unfortunate situation to me.

Feature stabilization in the language being blocked on formatting decisions seems like an unfortunate situation to me.

To be clear, feature stabilization is not blocked on formatting and I'm also adamantly opposed to doing that. It was briefly suggested for let-else statements, but was decided against after subsequent discussion and context. I've not seeing anything about let chain stabilization being blocked on formatting rules, but please feel free to loop me in on any conversations if that's being discussed.

I'm more interested in stabilizing this sooner than in continuing the bikeshedding.

We need greater alignment between the formatting and language processes to provide more expeditious timing and a better user experience, but this is also why I feel so strongly that feature stabilization needs to remain orthogonal and not be gated on formatting. Formatting deserves thoughtful conversation and consideration, especially because whatever is decided will essentially be permanent.
I absolutely do not want to be in a position where formatting considerations are rushed and forced through just because people want a language feature stabilized. That route will provide some short term gains in exchange for significant long term pain. It's wrong and we're not going to do that.
I'd also push back on the characterization of these conversations as bikeshedding. The entire scope of this repository and process is explicitly to define specific formatting rules, so discussion of options of formatting rules is very much the focus here; not some unimportant detail of a larger topic.

The more I think about this the more I feel that option 1 (do nothing) is the best approach for default behavior. I think it will be the most consistent with existing prescriptions, has the least cognitive burden and lower risk of formatting that's bad (or at least a poor use of space) in what I imagine will be the more common cases.
I believe sticking with the status quo will work best for the majority of cases and that the scenarios discussed on this thread which appear to be the impetus for always wrapping seem to have viable workarounds.
Some of those workarounds would be readily available to developers already (reordering the expressions, creating a local before the control flow expression to reduce the size of the binary expression that ends up in the control flow), etc. I'd also be open to some additional config options that could be used for non-default behavior to give additional control for the cases where they are needed (which could come in the form a single-line limit variants for binary expressions, as width based, expr count/nesting based, and/or the presence of certain expression kind variants).

A concrete example where I think always breaking && let to the start of a new line helps make bindings easier to notice: https://github.com/rust-lang/rust/pull/95631/

Yeah that may be true (it's unusual for let to actually end up very far to the right). But in a way that may be a bad thing. Your brain might get used to seeing let on the left side cause that's how it usually works out, but in the unusual case that it isn't, you might miss it.

I agree with this point entirely. Because of the common length of a let binding, let expressions will commonly wrap to the start of a line anyway, so when they don't, they'll be easy to overlook.
I also think a line break helps maintain a clear distinction between separate expressions and the RHS of a let. With let pat = expr && anotherexpr, even if you know that expr doesn't have type bool, it's easy for your mental parser to group expr && anotherexpr together. Breaking to a new line makes it easier to do quick mental parsing and skimming.

@camsteffen Per your comment https://github.com/rust-dev-tools/fmt-rfcs/issues/169#issuecomment-1072370574
We already require wrapping the entire condition if it doesn't fit on one line. So the only time this would come up is with a short condition containing a single let followed by one or more boolean conditions that all fit on one line. I personally think in that case we should still force wrapping.

Correct me if I'm wrong, but strictly from an AST perspective let chains are simply a Binary expression that has at least one Let expression as an operand somewhere within that bin expression tree yes?
Assuming that's accurate, I want to draw a distinction between the fact that rustfmt currently does nothing for Let expressions, regardless of whether or not they appear in a chain. This currently cascades back up and results in rustfmt not changing formatting for nodes higher in the tree, for example, rustfmt is currently unable to apply any formatting to the entire match expression in the below snippet.
fn main() {
match    foo {
 a    if let Some(b) = c => {}
    }
}

As such, I'm curious if folks would be willing to codify the rules for let expression variant in the Style Guide here so that we can at least handle cases like the one above while the conversation continues for chain handling? (yes, rustfmt can maintain current do-nothing behavior for chains while also handling formatting of a let expression in a guard)

As for chains/let expr in a binary expr, do we perhaps need to revisit the rules for binary expressions more broadly? Do we need more aggressive and/or complex wrapping rules for those in general?
I ask because I believe these expressions are currently only wrapped when approaching the max_width which can lead to some long running lines, and also because I wonder if some the theme behind the reasonings shared here for wrapping with a let expr could also reasonably/desirably be applicable for other expr variants too.
I should note that question/hypothetical about broader binary expr rules would have to be associated with an edition change and is thus likely a separate/broader topic for discussion. However, we could certainly add an additional clause to those rules that require wrapping in the presence of let expr(s) at any point.
Finally, I feel like we're a bit stuck on this. We've not had a wide enough volume of input nor even anything approaching a clear plurality IMO. Given that I wouldn't feel comfortable codifying any rules in my current role trying to maintain/triage the Style Guide. There's also not currently an authoritative body to make a call either.
I'm not sure what would be our best path forward. @joshtriplett would you be comfortable making the call (to always wrap) as a former+future member of the Style Team, or if not too trivial, is it a call the Lang team (as the likely parent team of the Style team) would want to make?

if let 1 = 1
  && true
{
}

If you guys are going to force wrap, please provide a stable Rustfmt option to disable it. Not everyone wants this formatting behaviour.

FWIW @c410-f3r I agree with the perspective and it's part of why I've argued against the always wrap option in my preceding comments.
However, it's important to keep in mind that these discussions and the goals of the style guide aren't to identify a formatting behavior everyone wants. I'd argue that doesn't actually exist, and it's why I'm not surprised that I've yet to find a single person who agrees with/prefers 100% of the Style Guide.
I also previously mentioned an openness to config options on the rustfmt side. Understand your desire to have an escape hatch if the proposed option were selected (and to have that available on stable), but would suggest we keep discussion rustfmt specifics and stabilization processes separate

I'm not sure what would be our best path forward. @joshtriplett would you be comfortable making the call (to always wrap) as a former+future member of the Style Team, or if not too trivial, is it a call the Lang team (as the likely parent team of the Style team) would want to make?

I'm not comfortable making that call unilaterally.
I would be happy to poll the Lang team for an interim decision to unblock things, while we're working on getting the style team started up.

FWIW I think let chains and let-else are going to be contentious formatting-wise no matter what choice is taken. This is basically guaranteed to be a "nobody's favorite" kind of choice.
overly cute variant
In the code I've written with let chains, I've always written it such that the lets all appear visually aligned — but very interestingly, without any actual visual alignment indentation, or relying on tab width! Specifically, I've writ
let Some(x) = x &&
let Some(y) = y
else {
    // diverge
}

if let Some(x) = x
&& let Some(y) = y {
    // use
}

However, I freely admit that this is too cute, doesn't generalize well to e.g. else if or match guards, and has some objective problems beyond that[^1].
[^1]: the big one is that the let-else case requires seeing the && at the end of the line to know the two lets are both attached to the else, as opposed to only the second; indentation is used to indicate expression continuation everywhere elsewhere in the rustfmt style

Ultimately I think the most reasonable default is "do nothing special" (option 1). This falls squarely into being predictable and nonoffensive.
This is definitely a place where rustfmt would benefit from having an "intent preserving" mode/ability, though, as the presence of a line break is perhaps the best indicator that a line break would be useful.

This is definitely a place where rustfmt would benefit from having an "intent preserving" mode/ability, though, as the presence of a line break is perhaps the best indicator that a line break would be useful

Not that this will help solve the more immediate decision at hand, but, for broader awareness this something we've been working on in different contexts with some measure of success, albeit not-yet-released (chains and function calls specifically). No promises, but I will keep this theme in mind for binary expressions as we look towards future work planning for rustfmt.

This is basically guaranteed to be a "nobody's favorite" kind of choice.

This made me laugh out loud, both because I think it's 100% correct but also because I'm not sure if/how often we actually see a different kind of choice in these contexts :grin:

This is basically guaranteed to be a "nobody's favorite" kind of choice.

This made me laugh out loud, both because I think it's 100% correct but also because I'm not sure if/how often we actually see a different kind of choice in these contexts :grin:

At the very least, I 101% agree with a good number of the choices made by rustfmt that were contentious (e.g. block indentation only, no visually aligned indentation) so at least some choices are "someone's favorite" 🙂

I keep coming back to solution number 3 (wrap let if it is N+ characters from the margin). I just think that will produce the most consistently agreeable results. The other solutions have warts as outlined in OP. I think the "complexity" downside of 3 is negligible - I generally don't give any thought to what is the logic behind what the formatter does. I would like to have the assurance that let doesn't drift too far from the margin.

A possible fifth possibility that came up in some discussions, that I'd like to capture here:

Allow single_identifier && let on one line (if it fits line length limits), but require a line break before the && for any other expression. Always wrap after a let expression.

This would avoid all four of the concerns listed in the top post of this: it avoids making the let bindings hard to find, it avoids absurd wrapping in the if a && let, it doesn't add complexity (people can easily and intuitively do it by hand), and it avoids having complex expressions to the left of a let.
We could, similarly, use some other metric of "simple expression" in place of single_identifier. I like "no more than a single identifier as a compromise though, to still make bindings easy to find when scanning code.

Semantic note: I use the phrase "wrap the expression" to mean "insert a line break before the expression" - or more specifically -  "...before the binary operator preceding the expression".
Another assumption to note: "always wrap let expressions" probably assumes "...unless it is the start of a condition expression". That is, I assume we never want to have ...something = if\n  let....

Correct me if I'm wrong, but I believe option 5 could be re-worded as: "Always wrap let expressions. But they can be on the same line if it is only preceded by condition consisting of a single identifier." So it is adding an exception and then adding an exception to the exception, and is actually more complex than option 3 IMO.

Always wrap after a let expression.

I think this could be added onto any of the other options. We can consider adding a "wrap after" rule orthogonally to our consideration of adding a "wrap before" rule. (Note the rationale for "always wrap after" was given in https://github.com/rust-lang/fmt-rfcs/issues/169#issuecomment-1087054004)

Minor point of pedantry, but would encourage striving for clarity when detailing suggested formatting rules to ensure sufficient delineation between the let expression itself vs. a binary expression that contains a let expression (a chain).
My interpretation of prior comments is that most/all are likely referring to the latter, but also want to note that the style guide will need prescriptions for both.

I just wanted to relay my experience:
I've been using let chains (and let else) a lot in rustc, and I've found that

I lean towards the same formatting rules as regular if conditions (I believe this is 1?)
let in the middle of short conditions have been easy to find for me, due to syntax highlighting and repeating text in the path of the types involved (hir::ExprKind::* might as well be a bullseye emoji for me now, I can identify it written on the side of a car driving down a highway while I'm skydiving)
let chains tend to always be long enough any time there are more than 3 elements that the condition needs to be broken up so each line ends up starting with a let, making them obvious as desired by some here
again, let chains with two conditions don't present trouble for me to notice it
I personally prefer to have the { on its own line, at the same indentation level as the if to be able to identify the body of the expression at a glance (aligning it to the condition or leaving it in the last let is quite hard for me to parse)
I keep having trouble formatting long conditions, where I find myself having to be... "creative"
currently as soon as rustfmt encounters a let chain it completely gives up on the expression, meaning that I keep finding improperly formatted if bodies just because the condition had let chains, I'd like to know if there's anything I can help with to change this

I'm also intrigued if we there's any mechanism for rustfmt to format nightly features up and only up the moment they are stabilized. If at the time stabilization happens a style decision hasn't been arrived at, then stop formatting in some way. This is purely for the benefit of rustc and similar codebases, so it is purely a self-serving request.

@estebank We talked about this in today's @rust-lang/style meeting.
We agreed that with the help of syntax highlighting or IDE support it'd be easier to find a let in the middle of the line, but it's a general guiding principle of the Rust style that Rust should be readable even in non-syntax-aware contexts such as diffs. Given that, I think it's worth having a line break to aid with finding the let.

I personally prefer to have the { on its own line, at the same indentation level as the if to be able to identify the body of the expression at a glance (aligning it to the condition or leaving it in the last let is quite hard for me to parse)

The Rust style already has the rule that once you've broken the condition across lines, the { goes on a line by itself. I'm hoping that'll capture the property you're going for here.
If an if let ends up all on one line, { will end up at the end of that line if it fits, and I don't personally think we should change that unless we were to change it systematically for all if conditions, which I don't think we should do.

I think the only qualms I have with the potentially settled decision is the need for splitting let chains always, but in practice chains in the compiler are long enough that there will be little difference between my preferred solution (follow the current if condition formatting) and the agreed upon one (more than 1 let expression makes the conditions go on their own line). I don't think that my ability to parse let embedded in conditions is purely due to syntax highlighting, and I haven't encountered a case where missing them would have caused problems, but I understand it is not everyone's position. I'm really glad to find out there's forward movement on this.

It seems that https://github.com/rust-lang/rust/pull/110568 and https://github.com/rust-lang/rustfmt/pull/5910 are trying to push this feature forward with a "one-ident-one-pattern-match" single line format, aka rule number 5, for whatever the reason (I couldn't find a discussion about it in a Zulip channel).
If that is really the case and as previously commented, please consider providing a stable configuration to disable such force-wrap behaviour in order to allow wrapping according to the specified max_width.
#![feature(let_chains)]

fn _let_chains() {
    let _foo = true;
    let _bar: Option<i32> = Some(1);
    let _baz = true;

    if _foo && let Some(elem) = _bar && _baz {
        dbg!(elem);
    }
}

fn _regular() {
    let _foo = true;
    let _bar: Option<i32> = Some(1);
    let _baz = true;

    if _foo && _bar.is_some() && _baz {
        dbg!(_bar);
    }
}

fn main() {
    _let_chains();
    _regular();
}

@c410-f3r - please note this is not the correct place to continue repeating requests for rustfmt features, nor to discuss the stabilization of rustfmt features which do not even exist yet.
I've already noted the possibility of rustfmt adding a configuration a few times, most recently in https://github.com/rust-lang/style-team/issues/169#issuecomment-1145320589.
However, neither the Style Team nor the Style Guide's default prescriptions cover rustfmt features, nor rustfmt's stabilization process.
Finally, neither of the PRs you've mentioned are setting anything in stone. Let chains are still nightly only syntax, and the purpose of moving ahead is based on our procedure for nightly only syntax so that we can get something in place experimentally while still retaining the ability to evolve the formatting for nightly only syntax.
This thread would be a good place for users to weigh in (particularly after the proposed formatting is provided) and share specific, concrete feedback and input on the formatting and/or why/how they think an alternative would be better.

@RalfJung suggested this as an example of where the currently-proposed rules may not be optimal:
if let Ok(name) = str::from_utf8(name) && is_dyn_sym(name) {
    let ptr = this.fn_ptr(FnVal::Other(DynSym::from_str(name)));
    this.write_pointer(ptr, dest)?;
} else {
    this.write_null(dest)?;
}

Under the current proposed rules, this would be formatted as:
if let Ok(name) = str::from_utf8(name)
    && is_dyn_sym(name)
{
    let ptr = this.fn_ptr(FnVal::Other(DynSym::from_str(name)));
    this.write_pointer(ptr, dest)?;
} else {
    this.write_null(dest)?;
}

He (and others) have suggested that the first formatting may be better.
One problem with adding so many line breaks as the second version does is that it makes a match shorter (and arguably prettier), so such formatting may discourage the use of let-chains.  E.g.:
match str::from_utf8(name) {
    Ok(name) if is_dyn_sym(name) => {
        let ptr = this.fn_ptr(FnVal::Other(DynSym::from_str(name)));
        this.write_pointer(ptr, dest)?;
    }
    _ => this.write_null(dest)?,
}

From discussion, it seems that the issues that led to the tentative decision to format it in the second way are:

Multiple let PAT = EXPR && let PAT2 = EXPR2 being confusing to read because the second let is not left-aligned.
Trailing expressions being confused because of the precedence of the &&

(Thanks to @compiler-errors for these.)
Given that, here are two concrete proposals for how we could make this better:

We could allow trailing parenthesized expressions to fall on one line since that would negate concerns over precedence.
We could allow function or method calls (and unary operations on those) in the trailing expressions to fall on one line as those too do not raise concerns over precedence.

There is of course also this option:

Allow all trailing expressions to fall on one line (but not multiple let PAT = EXPR fragments).

We could consider option 3 if it turns out that the costs of breaking these lines (e.g. in discouraging use of the feature) turn out to be higher than the potential for confusion over the precedence of &&.  It's perhaps worth discussing at least.
Setting aside option 3 for the moment, the question for discussion on options 1 and 2 is whether they have any notable drawbacks and whether there are other similar carve-out rules we should consider adopting.

The potential for precedence confusion remains in that example: between if (let pat = expr) && test versus if let pat = (expr && test).

FWIW while I did assume that "let should be at the left" was the reason for the ewlines, the precedence concern did not occur to me. When would one want to write if let pat = (expr && test)? The chances that that is what I meant seem basically 0, so I don't think we should add 2 newlines to defend against this extremely unlikely interpretation.

In the spirit of rust-lang/rust#117161, we could also choose to handle potentially confusing uses by linting rather than by our choice of formatting.


Fable.Haunted.Sutil

Given that you can bring your own renderer in haunted, we should be able to expose a custom function that takes the Sutil.DOM.mountOn function to make the web component stuff available rather than craft something specific for Sutil like I wanted in https://github.com/davedawkins/Sutil/issues/32

Didn't work so we went away with custom elements ourselves 😆


Open API

Other Mixin Bots can create articles in PRSDigg, and share the 10% revenue with PRSDigg.
Authenticate Mixin Bot

User can apply a Mixin Bot account in PRSDigg
Generate a access_token for Bot Account

Create article via API
Third-Party Mixin Bot with access_token can create articles via API, providing

Author (Mixin Messenger user or just a Mixin Network wallet address?)
Author's access_token can be authenticated by Mixin API
Article's Arguments (title, content, price etc.)

Revenue Distribution

Author revenue will transfer to author's wallet directly
For the 10% platform revenue, 5% will transferred to the Third-Party Mixin Bot, 5% will stay for PRSDigg

Feature request:

query articles belonging to a Mixin Messenger user

What we want is that we pass in an user_id, and the API returns a list of articles he/she published on PrsDigg. Details about the articles are not needed, only titles and uuids.
We are building a bot allowing users to advertise their articles. To give users better experience, we list all their articles so that they can select which article to advertise by one click, instead of copying and pasting the long URL.

Feature request:

query articles belonging to a Mixin Messenger user

What we want is that we pass in an user_id, and the API returns a list of articles he/she published on PrsDigg. Details about the articles are not needed, only titles and uuids.

Try : https://prsdigg.com/api/articles?author_id=7ed9292d-7c95-4333-aa48-a8c640064186
other available params:
https://prsdigg.com/api/articles?limit=1&offset=1&author_id=7ed9292d-7c95-4333-aa48-a8c640064186
https://prsdigg.com/api/articles?limit=5&order=asc&offset=2021-01-18T07:41:36.624Z&query=BTC,Xin
or check the detail here


Black publisher screen with PiP sample

Context

Running the Picture-in-Picture sample (https://github.com/opentok/opentok-android-sdk-samples/tree/master/PictureInPicture)
Using the TokBox playground to create session/token(s), etc.
Nexus 5X / Android 8.1.0
When Prompted, the Gradle/Gradle Plug-in was not updated to avoid differences with the sample.
Android Studio:

Android Studio 3.3.2
Build #AI-182.5<IP_ADDRESS>14842, built on February 16, 2019
JRE: 1.8.0_152-release-1248-b01 amd64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Linux 5.0.0-13-generic

Server: The server is a trial instance of ToxBox; I tried two projects, one with VP8 Codec and one with H.264. Same results.

Steps to Reproduce
Using the PiP Tokbox Sample App:

Update ApiKey, SessionID, and TokenID as usual
Launch app, give permissions, wait for your feed (and if there are subscribers, you should also see them).
With the app perfectly working (both publisher/subscriber are visible and transmitting both-ways), tap the PiP button.
App enters PiP ok. Observe you can still see/hear the subscriber's feed.
Tap the PiP window to go back to "full-screen".
PiP closes and the activity opens in full screen again.
Observe: Remote Feed still works, Publisher's screen is now "black"

What is expected?

My own feed (publisher) OpenGL view is properly restored and works after returning from PiP.

What actually happens?

My own feed (publisher) is "black", no feed and no way to bring it back.

How about the samples?

My initial reaction was to follow the OpenTok official tutorials, more specifically both iOS and Android to build a sample from scratch to learn about the key pieces myself. I created a blank Android project (w/Kotlin) and didn't update Gradle to the new beta version (3.5.x), just left the version Android Studio set up for me. (note: for iOS, I created an Xcode project, used Cocoapods to get the framework and ran the basic-chat sample, I did not try PiP on iOS, but both apps -Android and iOS- were able to see/hear each other and so did the Web playground I used to test too).
After I got those two 1:1 basic video/chat working (always using OpenTok's playground to create tokens/sessions), I took at look at the official samples and found/ran the basic Video-Chat  sample.
Then I decided to try PiP and didn't initially see the PiP sample from ToK, so I took a look at what was needed in the official Google sample for PiP repository.
With some degree of success/failure, I searched and found the picture-in-picture TokBox sample; this gave me some tips of things I wasn't doing the same way, so I updated my sample project and got it working but with the bug I am describing here.
Then I decided to run the PiP sample "as-is" and here I am, reporting that it exhibits the same issue.

Screenshot(s)

Low quality GIF

Additional Information: I ran this on a borrowed Samsung S10 and it appears that it works fine there. I have not been able to fix the issue on a Nexus 5X. I'll ask our Q.A. department to run the demo on a bunch of Android devices and see which ones work and which fail, maybe it's a Nexus 5X issue (which wouldn't surprise me)

Going to close this due to inactivity.
Please reopen if the sample is not working well.


[ENH] Add DL anomaly detection algorithm: LSTM-AD

Reference Issues/PRs
Issue: #1637

What does this implement/fix? Explain your changes.
Implemented LSTM-AD algorithm for time series anomaly detection.

Does your contribution introduce a new dependency? If yes, which one?
No, it uses tensorflow and scipy.

I agree with @hadifawaz1999 and I do not see any reason to rush this in. So, if @itsdivya1309 agrees, we can first draft the base deep learning interfaces for anomaly detection before we implement the individual algorithms.
The anomaly_detection module is experimental, so there is no issue with deprecations right now.

I agree with @hadifawaz1999 and I do not see any reason to rush this in. So, if @itsdivya1309 agrees, we can first draft the base deep learning interfaces for anomaly detection before we implement the individual algorithms.
The anomaly_detection module is experimental, so there is no issue with deprecations right now.

I am in for the deep learning sub-module.

@itsdivya1309 Sure, happy to have somebody taking care of this! Do you want to create a draft first, and then meet with @hadifawaz1999 to discuss this draft in a dev meeting?

@itsdivya1309 Sure, happy to have somebody taking care of this! Do you want to create a draft first, and then meet with @hadifawaz1999 to discuss this draft in a dev meeting?

Yeah, sure, I am making the changes suggested by @hadifawaz1999 in this branch itself.

Hi @CodeLionX @hadifawaz1999, I’ve noticed a couple of tests are failing, and I’m unable to fix them. Could you please provide insights on these test failures and how to resolve them?

So basically i think the best thing to do is to keep in mind that if a deep anomaly detection model is forecasting based, like the one this pr implements (if i understand correctly) then at some point it would be better to implement it at as deep learning in forecasting module (once the new forecasting module is ready) and wrap it in anomaly_deteciton/deep_learning as anomaly detector with the whole config needed.
A lot of deep anomaly detection are self-supervised models, mostly representation learning, so given we are planning to start a self supervised module (soon) then these models will implemented in self supervised module and then wrapped into anomaly detection deep learning submodule with the whole config needed
Will take a look soon on the PR to see the network and all and get back to you

So basically i think the best thing to do is to keep in mind that if a deep anomaly detection model is forecasting based, like the one this pr implements (if i understand correctly) then at some point it would be better to implement it at as deep learning in forecasting module (once the new forecasting module is ready) and wrap it in anomaly_deteciton/deep_learning as anomaly detector with the whole config needed.
A lot of deep anomaly detection are self-supervised models, mostly representation learning, so given we are planning to start a self supervised module (soon) then these models will implemented in self supervised module and then wrapped into anomaly detection deep learning submodule with the whole config needed
Will take a look soon on the PR to see the network and all and get back to you

Thanks for the feedback! That sounds like a solid approach. I’ll also keep the self-supervised module in mind for models focused on representation learning.
Looking forward to your review and any specific feedback on the network design. Thanks again!


highlightColor defined as a function on CartoLayer doesnt seem to work

This doesnt seem to work. But CartoLayer inherit from MVTLayer which inherit from Layer in the end (I think)
https://deck.gl/docs/api-reference/core/layer#highlightcolor
return new CartoLayer({
  ...cartoLayerProps,
  id: id,
  pickable: true,
  autoHighlight: true,
  highlightColor: (info) => {
    console.log('test')
  },
});

Indeed, CartoLayer inherits from MVTLayer.
Did you try to specify [uniqueIdProperty](https://deck.gl/docs/api-reference/geo-layers/mvt-layer#uniqueidproperty)? Needed for highlighting a feature split across two or more tiles if no feature id is provided.
You will need to set it in your useCartoLayerProps hook like:
const uniqueIdProperty = 'your_unique_id_prop';
const cartoLayerProps = useCartoLayerProps({ source, uniqueIdProperty });

Check CARTO docs for more info [here](https://docs.carto.com/react/library-reference/api/#usecartolayerprops).


Update - revision 11

Hi @gcarreno ,
I replaced LGenerics with Generics.Collections for the time being.

1.11

Revision release - Sequential approach. 3-5 mins on my Inspiron 15 7510 laptop, around 2m55s (no improvement on speed).
Replaced LGenerics with Generics.Collections for the time being.

Thank you.
-Iwan

Hey Iwan(@ikelaiah),
There were, past tense, some merge conflicts.
Can you check if I made the right decisions in solving them?
Cheers,
Gus

Looks good to me. Thank you.


South African English Survey

Trello card link TBA.

South African English is no longer a targeted language. Closing.


Support GHC 7.10.*, fix sdist

Data.List.sortOn is in base now, so hide it until we drop 7.8 support
Rename cabal file to fix cabal sdist (and hopefully stack), i.e. should this sequence should succeed

$ cabal sdist
$ cabal install dist/*.tar.gz

cc ambiata/yak#63

That commit is poorly named, I should probably add a stack.yaml or reword it

Also missed some AMP stuff, one sec...

does this currently build on 7.10? You could add a travis 7.10 build if you want, as this is public

Builds with -Werror in 7.8.4 and 7.10.2 for me, I'll figure out some Travis now.

here's an existing one https://github.com/ambiata/icicle/blob/master/.travis.yml

Build should pass now. Mafia hash needs to be bumped before this is mergeable, so blocked by ambiata/mafia#23

I think the cabal file should still be called p.cabal (even though the package name is ambiata-p)
Other than that :+1:

:+1:

As IRL'd, renaming cabal files to fix sdist + stack. Travis job tests for this