Running rebuild-index on s3 with new default layout corrupts repo.

Output of restic version
restic 0.6.0 (v0.6.0-0-g820c88ea)
compiled with go1.8.3 on darwin/amd64
How did you start restic exactly? (Include the complete command line)
Init repo with new -o s3.layout=default option
Script used for below in this gist:
https://gist.github.com/vsivsi/b45b2064c75a38fac80ac6a964b1ce0d
Full reproduction:
export RESTIC_REPOSITORY=s3:http://localhost:9000/restictest/restic
export RESTIC_PASSWORD=Password
export MINIO_ACCESS_KEY=AccessKey
export MINIO_SECRET_KEY=SecretKey
export AWS_ACCESS_KEY_ID=$MINIO_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=$MINIO_SECRET_KEY

minio server ./test_server &

# Endpoint:  http://<IP_ADDRESS>:9000  http://<IP_ADDRESS>:9000
# AccessKey: AccessKey
# SecretKey: SecretKey
# Region:    us-east-1
#
# Browser Access:
#   http://<IP_ADDRESS>:9000  http://<IP_ADDRESS>:9000
#
# Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
#    $ mc config host add myminio http://<IP_ADDRESS>:9000 AccessKey SecretKey
#
# Object API (Amazon S3 compatible):
#   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
#   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
#   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
#   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
#
# Drive Capacity: 928 GiB Free, 1.0 TiB Total

sleep 5

./restic init -o s3.layout=default

# created restic backend bfd8e53e9c at s3:http://localhost:9000/restictest/restic
# Please note that knowledge of your password is required to access
# the repository. Losing your password means that your data is
# irrecoverably lost.

# Create 10 16MB test files with long random names, containing random data:
mkdir test_data
for ((x=0;x<10;x++))
do
   FN=$(xxd -l 32 -p -c 32 /dev/urandom)
   head -c 16000000 /dev/urandom > ./test_data/$FN
done

./restic backup ./test_data

# scan [/Users/admin/restic6/test_data]
# scanned 1 directories, 10 files in 0:00
# [0:01] 100.00%  103.550 MiB/s  152.588 MiB / 152.588 MiB  11 / 11 items  0 errors  ETA 0:00
# duration: 0:01, 89.86MiB/s
# snapshot 85f2d1a0 saved

./restic snapshots

# ID        Date                 Host        Tags        Directory
# ----------------------------------------------------------------------
# 85f2d1a0  2017-05-31 06:11:40  dev.local               /Users/admin/restic6/test_data

./restic check

# Create exclusive lock for repository
# Load indexes
# Check all packs
# Check snapshots, trees and blobs

./restic check --read-data

# Create exclusive lock for repository
# Load indexes
# Check all packs
# Check snapshots, trees and blobs
# Read all data
# [0:00] 0.00%  0 / 31 items
# duration: 0:00

./restic rebuild-index

# counting files in repo
# [0:00]   0 / 0 packs
# finding old index files
# saved new index as 39a40733
# remove 1 old index files

./restic check

# Create exclusive lock for repository
# Load indexes
# Check all packs
# Check snapshots, trees and blobs
# error for tree fb9ed061:
#  id fb9ed0615766b7b057cd950a9160732978a3daedbc5483f02b5f25a22832acc2 not found in any index
# Fatal: repository contains errors

./restic check --read-data

# Create exclusive lock for repository
# Load indexes
# Check all packs
# Check snapshots, trees and blobs
# error for tree fb9ed061:
#   id fb9ed0615766b7b057cd950a9160732978a3daedbc5483f02b5f25a22832acc2 not found in any index
# Read all data
# [0:00]   0 / 0 items
# duration: 0:00
# Fatal: repository contains errors

What backend/server/service did you use?
S3 with latest Minio version:
Version: 2017-05-05T01:14:51Z
Release-Tag: RELEASE.2017-05-05T01-14-51Z
Commit-ID: 40985cc4e3eec06b7ea82dc34c8d907fd2e7aa12
Expected behavior
rebuild-index works
Actual behavior
rebuild-index corrupts repo
Steps to reproduce the behavior
See above...
Do you have any idea what may have caused this?
Maybe the same issue I fixed in my fork with this commit:
https://github.com/vsivsi/restic/commit/654c69fb3b6e2fe291c797193e5723ecd8964abc

BTW,  I'd try to merge the fix in the commit above into a PR to fix this but I'm literally on a ship in the middle of the Pacific Ocean (at 31N 185W to be exact) with the worlds crappiest satellite internet connection (and a million other things to do), so this time it's all I can to do just report this and hope someone else will take it from there. It took me 90 minutes to download the 0.6.0 darwin binary today to try it out, but I will happily try it again when 0.6.1 is released with a fix for this bug!
Thanks, -Vaughn

I'm not 100% sure this is just for S3, for laughs I ran a restic check myself and got a nasty surprise.
$ /usr/local/sbin/restic -r rest:http://qnap.ch.polyware.nl:8000 check
Create exclusive lock for repository
Load indexes
pack 1b182587 contained in several indexes: {c18786e0 e818c425}

run `restic rebuild-index' to correct this
Check all packs
pack eca9eacf586b67d0e9c0cfc8b70539bfc5eff517d0b5d1e93e2713b315d50322: not referenced in any index
pack ecaa076063c957af1b7381bb90c837a764622d0f0614038beaff2d9b9b494948: not referenced in any index
pack ecac32f93108ff4b688323c6b35fd953404b7ce9ea832d0c48c9fb05ef720df6: not referenced in any index
pack ecd6eb8e403a6756d512e190e59138e134247ad4df76c8b1c88c101ec0ca2236: not referenced in any index
pack ecdd4693007a3fa98f7b6467ea129bca81640b51a24d7f38b52d52f4fd6fecdb: not referenced in any index
pack ecdd8e1cf7f5bf27c5a006ce6d41c60be72d458dc1d63dd0e0d07e8e9de218be: not referenced in any index
pack ece206120bc8603c46103a41f960e7e810c7be4ed15fe0437b973b9767ed2bab: not referenced in any index
pack ecf1a7c0403576a6ac46e8da66a18d9620c5686e1f81ecfc8ca404f30dbef85a: not referenced in any index
pack ecf1f95aef4409dc1e4d0089a800d804afb67b777e6db01032e069342b682577: not referenced in any index
pack ecf7b976f02d5827257d50ace21ba369c2c1553a9ccfc21afab91ba75a39278d: not referenced in any index
pack ecfae977fad87bec45694026442aaf6c7679043a441acfe83911dc15b33db173: not referenced in any index
pack ecfaf0b77434bb27d5e434bc69c9160e447751ade8d6650b4d44fbc923c4aed8: not referenced in any index
pack ecfea1c91711141efca2980c242b9363889e38b62798caa2fa536eadc835b3e0: not referenced in any index
pack ed0452179342b340420a1e6332536f8458427be34d2e680798241e65e869553d: not referenced in any index
pack ed06a0cfc8063742da2f092764594fa4a228a3ed430f200c863420a0fb1d636b: not referenced in any index
pack ed1733f00a2b4de339fb1d82da220ab86b21e4cb2da4c0aecd7719a4720cb802: not referenced in any index
pack ed1aa22918c3d6e5d0418043aea11695a570585684990ea103d9a8b4e1d2e36d: not referenced in any index
pack ed1c246bb460453de8f5c87ed74f449afed5ceba3d01692644b6ffaf3831f330: not referenced in any index
pack ed1d00d249750e740d7810ee7abdeba2844647e11d39856aff5ea6238d84b955: not referenced in any index
pack ed25a4abf9850447168cba2315561518741ca3deb8838e8738e8c08d25ee5f1d: not referenced in any index
pack ed2b7519c1cdb52bbc2addab6c7ea2da395f92b06d26ad4ecaae964b816b6d41: not referenced in any index
pack ed3a57ec997af9007c5f3a9c52bddcb17b622fc2aee4cef4b88778acf2728bb6: not referenced in any index
pack ed3d5efbd2ffc074e344009eae00390929cc988b1a7d0914f3d930ed50321cdd: not referenced in any index
pack ed3db8d06f272151af2dbd11bb27f1f95b5ed9646519e851daae8e51dc78dec1: not referenced in any index
pack ed41fa097946138941bfc6496ca4af36b7f68e26579959276a1888e2882dbd4e: not referenced in any index
pack ed43c070458b5af48bc9ae066e64743d11c376ba4ec4731bac4d7b4e66945ae5: not referenced in any index
pack ed46b464403bc6890fcbea99399367bb80f2d2cfc49e7e7ca8a7d36421c973bb: not referenced in any index
pack ed4cdeefb501bc9f038ac58dcc596395a1c5084ae31cd2088e9d60aac6e9c1fc: not referenced in any index
pack ed4e898f4c00fee7ac6c97cc4afe636f672f2f005b04c89699aca617145d7725: not referenced in any index
pack ed510e257fe7f804c809d4886fa02ab3faae27ea4dbf82d380b406beceaa65b9: not referenced in any index
pack ed5c036e3adb29cadf3851a4cb49ea870fefb3c4a45bdbd1147fe6d79bc58120: not referenced in any index
pack ed5c9640f92173871c729549da177bf6a7b959b83a1119cbfbb845675e6ce69c: not referenced in any index
pack ed5f2e755ebbfac8835c94caac3fa513bb86eab9938cede0346693e66c2b642f: not referenced in any index
pack ed674f53e7c51241e07e6097cb968034bd19786252258e3edce1661adc73a502: not referenced in any index
pack ed6acd8e1baef4158ad0791271b45a03d1ea1a541a94f6ddda53ad54180564d1: not referenced in any index
pack ed6c2db6c71e8258a6f6a3b80b1993aa7cd74805321f55571421cac40b2713f8: not referenced in any index
pack ed6cff0f875a7dbe56cdd489df36739559091d0a95dcf7cf5a766b092d0d2730: not referenced in any index
pack ed6d59c3b6ca87dad3061ff5a529113720816a937fc516d93fa134153309d177: not referenced in any index
pack ed6fc9a5b4e6d0c9acb414d62413a06963c3a0a24bfcc36ca180cc4da4ef0761: not referenced in any index
pack ed7065fe1c19d8e1d8c88ddc7894b5e28d460980b3208b57d5d07aba63bf203e: not referenced in any index
pack ed71589cf3dff22b76700e97475d3822053ae0c28088bdb62a4552a3ded0d968: not referenced in any index
pack ed7796e9f46703c8e797bf6540dd953b075acbb373c19c1c31beaa3b52412679: not referenced in any index
pack ed7a34e4bdeceb0e89cc763fde75750c156e4a7a18841d55c9375350c3a942b4: not referenced in any index
pack ed7eae7382404187f3d01edf0914f7176aea07237d505dba743e3cf52938964b: not referenced in any index
pack ed846dab5ca5a4d2fa5f4f58f5959fec99273a642c8c04f79a613adf0b59984e: not referenced in any index
pack ed857768ceb0da527bf5aad91e4410655577d265224f6a17476303ae5dac96d5: not referenced in any index
pack ed85c37025dfade297e22f5517416cd2935043b14dbece93b1b3cd36495068cb: not referenced in any index
pack ed884d1673574a894ea3eb24243c6dffee6a91699a1e5e6776e484b8572feda8: not referenced in any index
pack ed94673420ccec78c909d312ef98e308222930adf715f33fa1ff98e15ca0d82a: not referenced in any index
pack ed9532d0be70531b565a0db666b9e6d9df65f5bcca61bb013eba8ca181b785dd: not referenced in any index
pack ed953330d15d3603ef23fbc33c444e82bc574b42fd2de98c313d4e0670abfec6: not referenced in any index
pack ed95cebdabaccab1e8eb7bf0680f8b39a104d143ad1ab6d1596bee0f88ff1179: not referenced in any index
pack ed980b65604eb60cde0ced4fb332464ec3b506ca4d99a27f5053044c228ba07b: not referenced in any index
pack ed9935ea1c33485cfa1bdaadb51d2f3d2d2615dbde946e8d365995d03c5ec048: not referenced in any index
pack ed9e558c649d687975ad27b16083d2e8f9a763cf8640e71a6b2da427170e6f0d: not referenced in any index
pack ed9e5f4f0622a6a48ac741b2aa97b30f5dedccedb972bf38ab2b6647e66b0def: not referenced in any index
pack eda376dab4630e0d4d43c981cc0c6a97fab7805fa6641f437fe2526beedd0118: not referenced in any index
pack eda3e41090074bd8a53693aeb665d785a1a9703686e92ce10ce0e0d4f8cbc9c2: not referenced in any index
pack eda7018cce5a8a73fc2d93f9e88a9839dd01b6bdd588f8b6a0317f2ffcccdbbe: not referenced in any index
pack eda7037d4e7059e9ef5b890e9532decf230c60f4eb07114c238c928362af469e: not referenced in any index
pack eda711e5551ed9add092d6bb2ec4d3023e22750ca9b7f39b2420f7de2d66ace0: not referenced in any index
pack edb398295e295711e26462599864eed006530d88e05d3b50cb08ad04eb35b57e: not referenced in any index
pack edb6d021ee31a756293541e814c23c1f9920ae8fdd7665a63df0b74e52147d59: not referenced in any index
pack edb860750f2dd3b50853ce4801ee967f54269ea09d6a92ce1fe3dc068de7d5e0: not referenced in any index
pack edbe8143fdd0b3b86f10e3d4305b282e50401c1cab864bfc098cd48e49253a9b: not referenced in any index
pack edbf2c5e5193a63cb1b367d81ccc21b6f3c2100cfcc06021c6e9342182030c16: not referenced in any index
pack edc51b604ac1d58c90f419b9ed9545c4a11ea67730c77da46142f7eec8461518: not referenced in any index
pack edd672371a606815cd0f9e6c1517fa5dcf5d1f054473f8ad8432c09c09976fe4: not referenced in any index
pack edd67638f6c25266de43bde9cf001c6bd64fa245137e5805f9afa33e01b20ca7: not referenced in any index
pack edd71d60779ef9d26d4ecc068b3ce3cfe4f3a5317dcae7460e15ff3830bb2094: not referenced in any index
pack edde2286202e3a4a160118ad652d2a3a13447c00db81121780469b582c71bcd1: not referenced in any index
pack ede8eda38dd9b0759b228c22e9d9b3b3d415f2feff2f4d599834f124be249860: not referenced in any index
pack ede95658852d5791257f68c692b16df86698c33f5da1ac6500ba74d65f683ca8: not referenced in any index
pack ede9e2fe1fdf1272e4049f81cd0d602a9a615d498ec29ecf750c4c58ef0aae5f: not referenced in any index
pack edeb23e4eacd4116f22bfc0921b2558440c635ac491ecc2ddfe98f410830b920: not referenced in any index
pack edf3e53163c36d8ee137d9827b2bc23e476ba7118c6f1020ff67ae62a9fcc2b1: not referenced in any index
pack edf83015fa1b0432ca694457b1529a5773d385bcd09756d02b94bef8f15f2dfe: not referenced in any index
pack edf9cb73fdda2695ec070f7b4ec298cfc9530733111be535155d0842164b32f8: not referenced in any index
pack edfa73d20809a3cb481614a50d4a84b28478bed2035dc3e17cee6eccc16b2a95: not referenced in any index
pack edfde86af2ae600847bf7b838facda6dd5bc9d2ea415a2c1740a03e2bbf41df6: not referenced in any index
pack edfe95a5e135ffa4aac4c04d2a099e6978f90a39cdb585970772a5c0672e7527: not referenced in any index
pack ee014e5a7c6d1b6de8c7c620983bd366c2bcfe996b380ac20df1162b92f1d5ad: not referenced in any index
pack ee020635a043211d0a4f310139f9c7ced4fb0f1a0059b97849eff558b3275822: not referenced in any index
pack ee033f767f4e7e6c7ecc6a9402c021bbbd7b4413e14f34e5b5faa6cff6dc8158: not referenced in any index
pack ee06cb6779ce9de7a46ae0467fa7f04371d989d178f23deb8190f5f1fcf119cd: not referenced in any index
pack ee0d679211843d6da82d7246d0d9d50d26e49ef0b9a6b050edf742a5565f7ab8: not referenced in any index
pack ee0ed6884b48a114f26590a01df7b18a5858d7dd549fc89f418af42dd66f16a9: not referenced in any index
pack ee12cb7b5eba0a9d6a5813adf1599dd739692b7672c39f12d95f57da38edb082: not referenced in any index
pack ee16930ae5705e292afc1e04808914efb84b4902b9eacfda4a241b1a0ee3ae01: not referenced in any index
pack ee177e09dc5275f193d2eb5cf6f3bdc6c7c025b32550296f0e74d034a5064df4: not referenced in any index
pack ee22678d4f410048e87b6906f5e95e69de4c86b7154ec2d5522fa602019d0171: not referenced in any index
pack ee263d5539dbc100d52d30ae225bf6d5ccb8d852984dbd670c44957c35d67704: not referenced in any index
pack ee29e354f062b265a4b8e85baadb4bb2421973f93d127283c0a32c81ab13d197: not referenced in any index
pack ee2ac5cefb17e60ab0c746006b401c3a4e47dcc927854caab118f06ef49db7b3: not referenced in any index
pack ee3144f5853aeb2a5d7e2e3bdd5a8279d92570215e894a95afc2cb4d698b44ca: not referenced in any index
pack ee388622e89a57abec903f8d77841062e0f6bde50d87fc9e9e6fe8df30c131b4: not referenced in any index
pack ee39068cd7216bd6665089decc7165983b89eeaf8f152c691dd2b482f62f448f: not referenced in any index
pack ee3b64c04d056458824ee7b973518d207338d13fd28e86081f2c1a693dc8f6e0: not referenced in any index
pack ee3eca22704db6d778fae29edd22da7dfe0b837d2aec88983922568fe9888acf: not referenced in any index
pack ee408dfb4823c8738c8020e516d0ba896c0336dba207d2d6217723c89c931adc: not referenced in any index
pack ee5e071df702dc2066c165560950ee519c8e2541135a1e0c85cab81dc4c2af76: not referenced in any index
pack ee65149637628bc3139594398b7f5d1effcf5437295c855b718cb86018b69c9b: not referenced in any index
pack ee81c2f0fd19a6e4f341c9a2b0a5f9651f05030a9a08b0a1e41869a61c8306a9: not referenced in any index
pack ee8405fe56bcff502def3f221ed11a6aba2cb2b6fda82b699b2eb0781d3b9049: not referenced in any index
pack ee8473b5f865fdfaeed331ac5ddc00e0aed79d77687dd236abd4987a1a992f61: not referenced in any index
pack ee9275c21543d515653f945f0be8f9a6c3e1597f2a190c1dd5dd3d9aac126aa4: not referenced in any index
pack f8abd1fe67afb4fb611f26c02cf6fce2d065beb84a65945ba7b190dabd403667: not referenced in any index
Check snapshots, trees and blobs
error for tree dfd61fab:
  id dfd61fab1a4ee3aa4f14dd7e0c2938a024344c8d5372beaf9909fb7e34651faf not found in any index
error for tree dbc579ff:
  id dbc579ffd37f91df2fee86385b19c7f7e46f720ab8be3badc56408d63509382d not found in any index
error for tree b3c9e379:
  id b3c9e37930e3d171ac254c7135dced644515b80a5c2ffa0b8cb254b47791eed0 not found in any index
error for tree 59f815e2:
  id 59f815e29478c04a69b3f77c24d966833f8849d6cf856da9c692bae223581d5e not found in any index
error for tree a4ac832a:
  id a4ac832a9e076a584cebf1201456ec869efeeace30de127cfa08cc2d9d0ed1c1 not found in any index
error for tree a986655c:
  id a986655cceac838573d85de3f7e85ae8e686730e4e4d95929f086b72a48e0a35 not found in any index
error for tree a4a5e036:
  id a4a5e0362bb601fcf4a5cfe600b76ca86d1e31c4d649256e6d304b0ce928d1bf not found in any index
error for tree 98ac2e26:
  id 98ac2e264222c3377e893f5f539cfd316bc1b05d55b095347753455b64892380 not found in any index
error for tree 129f69ae:
  id 129f69ae7dc18a161e42a04e85427bb2c77b2d92e0b069f5c7588ff2067a5296 not found in any index
error for tree 34e1c4a1:
  id 34e1c4a11ff0797d37581d7f8c76fada90bcb07d41ca3e8a6d49888cdd3a3208 not found in any index
error for tree 77963d25:
  id 77963d2510ab3482a2aa4d03cd0e6573501d705482664d3ec566a1a3772a2905 not found in any index
error for tree 71173899:
  id 71173899bea06aee0330554528995f593f9d31990273e431078d0a0a733a8b49 not found in any index
error for tree 7398f0ee:
  id 7398f0eec764cd26699d690d994aaa695602996edfae15a2c0963af91c2d454d not found in any index
error for tree 7abe7d96:
  id 7abe7d966d68d556aab73087f53207213de704a8cca05ee8a70da45647095785 not found in any index
error for tree 3bfa1ee9:
  id 3bfa1ee9ccda57ff226ec943e0416a25b8441ad85a814b558f29deb040dd42f7 not found in any index
error for tree a3eed9aa:
  id a3eed9aae6e8054d05bbc970181a4384e13ab203636825b2585968b62c8fe484 not found in any index
error for tree 543775b1:
  id 543775b1fe8ebb80d513a50b86ec3472102b63a8f64ed9a471ce54ae07042063 not found in any index
error for tree e96e33d4:
  id e96e33d4cbfd8f79322a2b1d2c3cf69a52c6905d2293fa2245cb5f571a93fe4c not found in any index
error for tree d484ca3e:
  id d484ca3e5f4f0372dd1ebf589606306ab70ae7ee428cb521e3d5274921736259 not found in any index
error for tree da511ada:
  id da511ada7fd398eb6ef0ab272ee559c16d73acfa27454417e63fec23f1889719 not found in any index
error for tree fef3e679:
  id fef3e67999a37fdf0ee1394b66f6ed796904f21a0a97b30975bfaaad3d4bdf7c not found in any index
error for tree 7ae2f675:
  id 7ae2f675570524d7a5089a312940370b73562f70689dcb7e6a53a6e40720024c not found in any index
error for tree 80d0b856:
  id 80d0b856955027dcf2e234f3325070fc0f24157d59dad59fa58094ff64619e4b not found in any index
error for tree 35a2c7ab:
  id 35a2c7ab442270240bfa33a7797c00b407ff9c00207347e79f432e7506dab209 not found in any index
error for tree ccc3e20e:
  id ccc3e20e1d881dcee2ae861714766207356f15aea0d255491f578f117a81971e not found in any index
error for tree 62c4992b:
  id 62c4992bb1443a22d6499bd4cc3372f2250c22818d27cacff533b2f4f7af1c02 not found in any index
error for tree fffe4ac8:
  id fffe4ac841980afe351bd52fba57ce758f123f86fc3fbc9da130feaed2c9d1c3 not found in any index
error for tree 66811069:
  id<PHONE_NUMBER>277462b217e504cb889e6b51dcf0c90ff400cfca3dfe446f222deb not found in any index
error for tree 68038c12:
  id 68038c12e15e1c7abcf4c6f2412e40e1bce01da08d4d51edd1fd1f6c15e56df0 not found in any index
error for tree 4b0c7bd8:
  id 4b0c7bd8953912bb0e2c3003a215fb7b469031926ebe3e64d01014f52367d83d not found in any index
error for tree 179a71c2:
  id 179a71c2695b3562c947fa6dedb02b681506bad777f033734d951f2ebaf257b8 not found in any index
error for tree 8f854700:
  id 8f854700fb9f90ea37eb36eadad846c2b9a8815712a5c326ed3c54e3af782b05 not found in any index
error for tree db3c3f4c:
  id db3c3f4c410398648790efa537c940c073f40fa263e050b765f17670ef34e00f not found in any index
error for tree da3a7c5d:
  id da3a7c5d474f49d36f542032f6f3e1c568938f5d17397790a0a19d37930a5b7b not found in any index
error for tree ae75b074:
  id ae75b074a3a9a5fdc17f067670244f8474db120277dfbab5052fe6aa80f4ddb5 not found in any index
error for tree 1ff37782:
  id 1ff37782f92ae4254d37947f2a87426c9ecd5d3ab113bb26861e2cb32c9b3e6b not found in any index
error for tree 32a88971:
  id 32a889716b6c30ce28b1a889ade713a305f891e376a03f04e37757303245edd7 not found in any index
error for tree d2cf2790:
  id d2cf27901bbd20e80a2260ee8c88f8a24049419b3461384353729011364df16f not found in any index
error for tree 93f4538b:
  tree 93f4538b, blob 018dcdf7: not found in index
error for tree 6d056908:
  tree 6d056908, blob fb69df7d: not found in index
error for tree 946c0f3f:
  tree 946c0f3f, blob ad68d5c0: not found in index
error for tree 02ba04ae:
  tree 02ba04ae, blob ad68d5c0: not found in index
error for tree 662fc57e:
  tree 662fc57e, blob fb69df7d: not found in index
error for tree c601567a:
  tree c601567a, blob 7e784209: not found in index
  tree c601567a, blob d583bb34: not found in index
  tree c601567a, blob 252086d0: not found in index
error for tree b10815a7:
  tree b10815a7, blob f8185565: not found in index
  tree b10815a7, blob afd007cf: not found in index
  tree b10815a7, blob c07eef18: not found in index
  tree b10815a7, blob dc55f2d8: not found in index
  tree b10815a7, blob 08b5a91c: not found in index
  tree b10815a7, blob 491f8642: not found in index
  tree b10815a7, blob e8d5cf51: not found in index
  tree b10815a7, blob be612d39: not found in index
error for tree 6bafa778:
  id 6bafa7784097de4de149e6afc388ce74ada8415bc112737b7af46c30827f4a0e not found in any index
error for tree 33083318:
  tree 33083318, blob f8185565: not found in index
  tree 33083318, blob 610189d8: not found in index
  tree 33083318, blob 1de2b25f: not found in index
  tree 33083318, blob afd007cf: not found in index
  tree 33083318, blob 287dbc96: not found in index
  tree 33083318, blob c07eef18: not found in index
  tree 33083318, blob 5eecc50e: not found in index
  tree 33083318, blob dc55f2d8: not found in index
  tree 33083318, blob 08b5a91c: not found in index
  tree 33083318, blob 491f8642: not found in index
  tree 33083318, blob e8d5cf51: not found in index
  tree 33083318, blob be612d39: not found in index
error for tree 695311e1:
  id 695311e15a8c44f39379797a5e5517c036d48da2ca8b0cf1b725450fa839acba not found in any index
error for tree d998c45f:
  id d998c45f64e86d88a3344fafebea5e11ed2f72100d7633e2e14922361e0ae785 not found in any index
error for tree daec9cf6:
  tree daec9cf6, blob fb69df7d: not found in index
error for tree de5eccc9:
  tree de5eccc9, blob ad68d5c0: not found in index
error for tree 0aececca:
  tree 0aececca, blob ad68d5c0: not found in index
error for tree 06407a04:
  tree 06407a04, blob fb69df7d: not found in index
error for tree 2a4bc5da:
  tree 2a4bc5da, blob 018dcdf7: not found in index
error for tree e8133e82:
  tree e8133e82, blob b8e75fca: not found in index
error for tree 233b5ee9:
  id 233b5ee90a04ccdfb9c1ae109db8a59c09a3a92c2fa860541291a446b9107687 not found in any index
error for tree 8bae0453:
  id 8bae0453178200d48e2ecb03cc48079a9093996207ceafc947581a8caa46065f not found in any index
error for tree 2fc05701:
  id 2fc0570126da6aad1f98ba1bbf3d80441f1972a9924ff0c7cb4bb0663c367382 not found in any index
error for tree 9ac17b23:
  tree 9ac17b23, blob ad68d5c0: not found in index
error for tree d26fbce2:
  tree d26fbce2, blob fb69df7d: not found in index
error for tree 739e2c17:
  tree 739e2c17, blob 7e784209: not found in index
  tree 739e2c17, blob d583bb34: not found in index
  tree 739e2c17, blob 252086d0: not found in index
error for tree 995b5741:
  tree 995b5741, blob 7e784209: not found in index
  tree 995b5741, blob d583bb34: not found in index
  tree 995b5741, blob 252086d0: not found in index
error for tree 84453170:
  tree 84453170, blob fb69df7d: not found in index
error for tree c2c0a3f5:
  tree c2c0a3f5, blob ad68d5c0: not found in index
error for tree fabca633:
  tree fabca633, blob ad68d5c0: not found in index
error for tree 07260e1b:
  tree 07260e1b, blob fb69df7d: not found in index
error for tree be5ca81e:
  tree be5ca81e, blob fb69df7d: not found in index
error for tree fb241cd5:
  tree fb241cd5, blob ad68d5c0: not found in index
error for tree 9a283d5b:
  tree 9a283d5b, blob 7e784209: not found in index
  tree 9a283d5b, blob d583bb34: not found in index
  tree 9a283d5b, blob 252086d0: not found in index
error for tree e312027f:
  tree e312027f, blob ad68d5c0: not found in index
error for tree 31cd4651:
  tree 31cd4651, blob fb69df7d: not found in index
error for tree b51e6a16:
  tree b51e6a16, blob ad68d5c0: not found in index
error for tree 80f519ac:
  tree 80f519ac, blob fb69df7d: not found in index
error for tree d8c6349b:
  tree d8c6349b, blob ad68d5c0: not found in index
error for tree 8d73974a:
  tree 8d73974a, blob fb69df7d: not found in index
error for tree 9e3521be:
  id 9e3521be4665a1ace03e4ba0d51e5f11f24589d42af866d7e0089d171e650c8b not found in any index
error for tree 92668021:
  id 926680211b708232e22455a8db55d927d46b610cb55bb94e5e32a60db340a22d not found in any index
error for tree 3317042f:
  id 3317042fce19fa93350a1759c3cbe351983c6e237751f95a84cba325f344de5c not found in any index
error for tree d5e93206:
  tree d5e93206, blob fb69df7d: not found in index
error for tree 363f2cd8:
  tree 363f2cd8, blob ad68d5c0: not found in index
error for tree bfc7d003:
  id bfc7d00315cf655739865c4acc5e7228816ffb0fd4532a94680254ef86a8a0e6 not found in any index
error for tree 50aa00dc:
  id 50aa00dc98c1ad16b4460142eac6f7c5955ad4a04e336158d60ec950dce241f0 not found in any index
error for tree 701d766d:
  id 701d766d3ffd98918e8a04ab2e5b204508d43d18cc005c11a7076b128bcaf1e9 not found in any index
error for tree ea106896:
  tree ea106896, blob fb69df7d: not found in index
error for tree 87fee5da:
  tree 87fee5da, blob ad68d5c0: not found in index
error for tree 66333a30:
  tree 66333a30, blob 018dcdf7: not found in index
error for tree 8b5b57b2:
  tree 8b5b57b2, blob b8e75fca: not found in index
error for tree db4120b8:
  tree db4120b8, blob 018dcdf7: not found in index
error for tree 898b866d:
  tree 898b866d, blob ad68d5c0: not found in index
error for tree efa32682:
  tree efa32682, blob fb69df7d: not found in index
error for tree 13912cfb:
  id 13912cfbb3ffccc8b87adedb547a9a266c9af794e758dbbe0a7fee6871c4b27c not found in any index
error for tree d7bfdfaa:
  id d7bfdfaa62f2fdd4d817f9fbf09659287b27cb500e23b37714e3bac00efcf8c6 not found in any index
error for tree 1ed757df:
  id 1ed757dfbcb876766da81670b01a8f6b72e3f9e33aec6daa71896d9b5d885946 not found in any index
error for tree cd95e9a1:
  id cd95e9a10369bf33eeb8ec0fc127027ce7eec6df2198a22e3e646b5df4a2269d not found in any index
error for tree 3092b2f7:
  id 3092b2f734c3c603e26cfb5853a350b910abc9f074e40b2dfecac708f7d1feda not found in any index
error for tree 543cee9b:
  id 543cee9b309650e6785bcfeafff7a09fd61459339d9e5ad55692c373ce1e27d1 not found in any index
error for tree ab49f956:
  id ab49f956c098bd08b1be70bda780b7bd6cb3dcac259a1df8d71ac8480c049c00 not found in any index
error for tree 42a7541e:
  id 42a7541e8d10e5b04374055d8bad41a2ea876286bb3fa92500e6468ccd6c39bb not found in any index
error for tree 14f7bf4b:
  id 14f7bf4b793d2a86c022572efd0ee0de053a86cb51896092a9af6667ba71bac3 not found in any index
error for tree 37002681:
  id<PHONE_NUMBER>100594b893cc949c197ad43c5e3ed5e97e674c8b273057a6748779 not found in any index
error for tree 49da46b7:
  id 49da46b7a90851bd82eced32d9d6f56784f2c3f16eb5fc4593ed1d6e49cfe94c not found in any index
error for tree 0ae6192f:
  id 0ae6192fccb2b9f17297aea7a2c6aebcf0afb9942dee4f7284bfa3d7cb3790fc not found in any index
error for tree 359ca8f2:
  id 359ca8f2dd0db784abddd1d3cadc50d640099590e83ee917740fbd6ce214a9f7 not found in any index
error for tree cd4089a4:
  id cd4089a4afc1000f862745f32b0223683fb41be876634a99ca00e1e6b82d3275 not found in any index
error for tree 949bbe08:
  id 949bbe08b6a2e016bbd7ecceec6f0add572e49cbc4a20341a07b5f08c3a37a35 not found in any index
error for tree fe9c7184:
  id fe9c7184966f0704684a42c450b98d2d8d7b569f19223c34258db4dc7eef4438 not found in any index
error for tree 7debca39:
  id 7debca39e1a26977114e19faf3e8639b526b2f655150fa1cfa3f92d77b80ac51 not found in any index
error for tree b70d5ac9:
  id b70d5ac9f1a02186848f48581f7509dc3e29a781aaa991af93a80ffe878dccf4 not found in any index
error for tree 50921d00:
  id 50921d0031aab9fa163d5186a674b04c14cf87ae58c6a7aeae0c8807e7bf0009 not found in any index
error for tree 3b8ae290:
  id 3b8ae290aacfa4b2e42ef92da0d53e79384da4709553b0f75fff331c10918ca7 not found in any index
error for tree de3ead62:
  id de3ead62973169185ddc82ce6db8c656c0fef0bd46bcfd8df5961c5237ffa4d8 not found in any index
error for tree e2782924:
  id e2782924001a8473d1f5a1ee74a72afa2361e80af1a7da83b054d308b9f77260 not found in any index
error for tree a87c2626:
  id a87c2626b4318bafec7440c8610721e0ceab71c5bb23b68033af9ba79b31816f not found in any index
error for tree efd019f5:
  id efd019f5156038f42e605e87478e7f1c7002bca2ad2fad7bb8094893cbbd5c1a not found in any index
error for tree 410aa445:
  id 410aa44594ff97981075135c64e80de17c8d4cdceefc771cf5264e867897e4ea not found in any index
error for tree afc71ebe:
  id afc71ebe4eb4d5508b188dc8a29a15a919e7355e1a471971c28e63b2d36c67d1 not found in any index
error for tree f670f866:
  id f670f86689407148cf9a4ec1003a485f24933a3657d7d21632ea02d43097dced not found in any index
error for tree 07041343:
  id 07041343e91bc7858aa50481bc8596052513726b7df5318fb2bb54ee140e1236 not found in any index
error for tree 7c9d15db:
  tree 7c9d15db, blob cee337c2: not found in index
error for tree 81d1f923:
  tree 81d1f923, blob 7f210da4: not found in index
error for tree b5df24f8:
  tree b5df24f8, blob 3f41e3d4: not found in index
  tree b5df24f8, blob 4c80aac8: not found in index
error for tree ea012a94:
  tree ea012a94, blob 7e784209: not found in index
  tree ea012a94, blob d583bb34: not found in index
  tree ea012a94, blob 252086d0: not found in index
error for tree 935b04d8:
  id 935b04d8b174280857d7d0b35f463d1d4020e05c1ab6285c0e5db6a0a538a604 not found in any index
error for tree 80db2ce8:
  id 80db2ce8238df5e10aeb4352fc1bcabb04493c85ef6c3d44e5e4f6ba6359499a not found in any index
error for tree d1b485ed:
  id d1b485ed6b3497b10969a961d17a573b2b222fdef50f57735cbb4738cf1a7afb not found in any index
error for tree 822d37ee:
  id 822d37ee6cd4dd3702b12dbc247c3547eec9daa60e86d8532f6c067daf3aab71 not found in any index
error for tree f6c55535:
  tree f6c55535, blob ad68d5c0: not found in index
error for tree f3e5a2e9:
  tree f3e5a2e9, blob ad68d5c0: not found in index
error for tree 522f9adb:
  tree 522f9adb, blob fb69df7d: not found in index
error for tree fc349fe1:
  id fc349fe1fcbada58a03ca23d4630072e57b3b4146455d3854d3b970b7669ca30 not found in any index
error for tree 0fbc7814:
  id 0fbc78144e88c90cfc2ef445fcd65aa4569e5e630e6ae9d571056c8f281e153c not found in any index
error for tree e16ab722:
  id e16ab72283e17b51ab77323d526c367094d05e3a749a647da5ce953ff78df79b not found in any index
error for tree c7069cbf:
  id c7069cbfe849dd006f8fe29ba45b04022d5927e38321bda00362d71757ac49a4 not found in any index
error for tree 313e95a1:
  id 313e95a1e6e2a56f47b6d8e5216cf61c792c4d77c66376930f163da2c638e08b not found in any index
error for tree 49dc4ba4:
  id 49dc4ba4e1a1b9010495051bd7c3a3b961f0c0706e7aa7421d9ddf0563c0bb08 not found in any index
error for tree 31b8eb93:
  id 31b8eb93dbcae85e1fd4b738236a0f964409b378cff8453001d2f3ba14d66afc not found in any index
error for tree 9a8a9618:
  tree 9a8a9618, blob ad68d5c0: not found in index
error for tree 990c7954:
  tree 990c7954, blob ad68d5c0: not found in index
error for tree f5e0295d:
  id f5e0295d3f838e364749563fe5f10b6f7c2c8f90f1588cc703381c3c5b5eeff1 not found in any index
error for tree 2dd8a023:
  tree 2dd8a023, blob ad68d5c0: not found in index
error for tree 2b548096:
  tree 2b548096, blob fb69df7d: not found in index
error for tree bb576ee1:
  tree bb576ee1, blob f8185565: not found in index
  tree bb576ee1, blob afd007cf: not found in index
  tree bb576ee1, blob c07eef18: not found in index
  tree bb576ee1, blob dc55f2d8: not found in index
  tree bb576ee1, blob 08b5a91c: not found in index
  tree bb576ee1, blob 491f8642: not found in index
  tree bb576ee1, blob e8d5cf51: not found in index
  tree bb576ee1, blob be612d39: not found in index
error for tree 83cbdefc:
  tree 83cbdefc, blob 7e784209: not found in index
  tree 83cbdefc, blob d583bb34: not found in index
  tree 83cbdefc, blob 252086d0: not found in index
error for tree 953f43e9:
  tree 953f43e9, blob ad68d5c0: not found in index
error for tree ad0ed77d:
  tree ad0ed77d, blob fb69df7d: not found in index
error for tree 038901e7:
  tree 038901e7, blob ad68d5c0: not found in index
error for tree 772258d1:
  tree 772258d1, blob fb69df7d: not found in index
error for tree 80c6e69e:
  tree 80c6e69e, blob ad68d5c0: not found in index
error for tree a5096423:
  tree a5096423, blob ad68d5c0: not found in index
error for tree 8a6436d7:
  tree 8a6436d7, blob fb69df7d: not found in index
<stopped looking...>

I'm going to run restic rebuild-indexes to see if it fixes things up. (As one of my VMs OOMs every night due to restic ballooning, it is likely I have half baked packs in the repo, but all the missing tree's are unexpected. ... Are we sure restic check is even ok?

Aside, restic check is using too many file descriptors on rest backend. I'm seeing these:
  client.Do: Get http://qnap.ch.polyware.nl:8000/data/0f05fa962f1afb155afece9fe16f46661f3286d1b93edf6ee511fd10c1621ddc: dial tcp [2a02:168:5204:0:265e:beff:fe0f:5327]:8000: connect: cannot assign requested address
error for tree 97c78ddd:
  client.Do: Get http://qnap.ch.polyware.nl:8000/data/26173a38e51153726034605a690f0d8aab2892600ca478f92c3240d9e5f6df86: dial tcp [2a02:168:5204:0:265e:beff:fe0f:5327]:8000: connect: cannot assign requested address

And checking ls /proc/$(pgrep restic)/fd | wc -l I see spikes in the 1000's of fd's... (coming and going btw)

Oh dear, thanks for reporting! Good thing we haven't enabled the default layout by default yet... I'll have a look ASAP.

@fd0  Yes, that's exactly what I ended up fixing in the commit I referenced at the end of the issue above.

btw, you can force the layout with -o s3.layout=default for the rebuild-index operation.

BTW, In my case I was not using S3, so it could not have been the layout issue. My problem was fixed though after running restic rebuild-index, so  I can only presume  restic check has some problem with old, superseded indexes.

Yes, I think your issue is different. Maybe that the there are unfinished pack files (I know that we need to work on that).

Ok, rebuild-index for s3 with the default layout creates an empty index

Listing pack files on s3 with the default layout is broken, patch is here:https://gist.github.com/fd0/15039537035ce2d6f8f203bd247c1724
I'll commit this fix later today.

Fixed in https://github.com/restic/restic/commit/5c59484d2bb6b66e2cca5962125df6848aa3e0c0.

@fd0
Is this change in the earlier patch not necessary for the fix?  Just noticed that it didn't make it into the commit you reference above...
https://gist.github.com/fd0/15039537035ce2d6f8f203bd247c1724#file-s3defaultfix-patch-L9-L10

The first hunk isn't needed (it is even wrong), the second hunk is the one that's important.

I'll release 0.6.1 shortly. Together with #986 this warrants a new release (btw, the s3 backend used with Minio should also be also affected by #985).


Compiling with the simple command fails

Here's the output
https://pastebin.com/3CUhc0yH

https://github.com/doitsujin/dxvk#troubleshooting
Probably this.

Outdated MinGW too.


Bandage BD just won't install Could not find DiscordCanary.exe!

You have weird scaling on your PC which is hiding the checkboxes.

I don't really, it's just THIS window, and I highly doubt that effects installed programs

Had the same problem, Try clicking on the words "able"  or the cut off word at the stop. Or you could download exactly what it's asking for the links are here - https://discordia.me/canary then https://discordia.me/ptb . After that once they finish downloading it just close both of them and run Better Discord again and it should show that both of those are downloaded.

Had the same problem, Try clicking on the words "able" or the cut off word at the top. Or you could download exactly what it's asking for the links are here - https://discordia.me/canary then https://discordia.me/ptb . After that once they finish downloading it just close both of them and run Better Discord again and it should show that both of those are downloaded.

Yep tried that too, even when I got them both to light up, the install button is still greyed out

Try opening Display settings and changing "Scale and layout" to 100%

I had same problem and  the install button simply isn't clickable just as the OP  said.I found old  downloaded  Bandagedbd and installed,worked for me .It's the 0.3.0

also I had that scaling thing that is shown in the screenie

https://drive.google.com/open?id=19XC2wg3SjQUzHe3o4QKCieoeYPFiAhh3
if you wanna try

Same problem :(

tick the ptb and canary boxes next to them and then install

okay just so everyone knows, these solutions no longer work as BD has had many updates, and when I went to try to install it again everything seemed to work fine, except that discord seemed to run as normal instead of being BD, I've more or less given up

okay just so everyone knows, these solutions no longer work as BD has had many updates, and when I went to try to install it again everything seemed to work fine, except that discord seemed to run as normal instead of being BD, I've more or less given up

same thing happend here

BandageBD can't find Discord.Exe alone what I can do ?

Help I need BetterDiscord

I am Also having that same problem Can anyone help me
Plz

Got the problem Fixed  Just enable Install to Stable Option  The IMG is here:-https://imgur.com/a/eLJ8Wvl
Then the option for installing will come

I am on a mac, and I don't know anything about this, and I really, really need help.  Whenever I reinstall it, and click Instal Bandaged BD, there is nothing in the stable or anything, and I don't know what to type in, since whenever I press "Browse", it won't work. It is really stressing me out, and I really need help T_T

Same with me. I'm on a Mac and BetterDiscord is just doing nothing and not installing onto my discord. I had it already but I realized that it was gone from discord so I tried to reinstall it but it just didn't work. Even uninstalling doesnt work so I'm pretty confused cause I also cant click browse and everytime i click stable ptn or canary it says, Failed to install to : ReferenceError: Cannot access 'fs' before initialization

I am on a mac, and I don't know anything about this, and I really, really need help. Whenever I reinstall it, and click Install Bandaged BD, there is nothing in the stable or anything, and I don't know what to type in, since whenever I press "Browse", it won't work. It is really stressing me out, and I really need help T_T

heree just saw this on their discord server

Don't Know anything about Mac Sory
But I know that betterDiscord is against DiscordTos So I am not using it anymore

Yup, I'm on MAC & I'm having the exact same issue. It used to work for me.. I uninstalled it a couple weeks ago and decided to install it again but now it's not working. Also, can someone create a discord invite for the BetterDiscord server with unlimited uses and post it here for us who are trying to get support please. I can't seem to find their discord server either.

I'm having the same problem too


Fix dependabot alert

https://github.com/Textualize/rich/security/dependabot/2

It seems that the fact that Rich supports Python 3.6 prevents us from being able to upgrade the package:

The current project's Python requirement (>=3.6.3,<4.0.0) is not compatible with some of the required packages Python requirement:

notebook requires Python >=3.7, so it will not be satisfied for Python >=3.6.3,<3.7

Because notebook (6.4.12) requires Python >=3.7

The security fix looks to not have been backported to older versions of the notebook package, unfortunately :pensive:

https://github.com/jupyter/notebook/security/advisories/GHSA-v7vq-3x77-87vg

As Python 3.6 stopped receiving security fixes more than 6 months ago, maybe we could stop supporting it in an upcoming version of Rich? :thinking:
(which as a side-effect would also bring the joys of v3.7 into the Rich codebase of course, like modern typing annotations :star_struck: )


Umlaute not displayed correctly

Atom is currently having issues with displaying umlaute (äüö).
When an umlaut character is typed or pasted it appears until the end of the line.

When it is removed, everything looks just fine:

It also seem to be related to syntax highlighting, when selecting "Plain Text", everything is fine:

Issue found on Arch Linux, build 0.171.0-296409d
Working without issues on Windows.
Document is UTF-8.
Tested with Syntax Highlighting for C, C++, Javascript, Latex and PHP. (Might affect other languages too, but I did not test it. I believe every language is affected.)

Can't reproduce in Atom 0.170.0, OS X Yosemite, with UTF-8 char encoding.
Steps taken:

Create new file (e.g. File > New File)
Change syntax to C, C++, JS, and PHP
Type umlauts
Try copying and pasting umlauted U's

(Btw, did you build from master? 0.171.0 hasn't been tagged yet)
I'll check on my Ubuntu later today to see if it's a general Linux issue. (Maybe it's a 0.171.0-specific issue though?)

Yes, I built from master. Give me some time to test other releases too.

Hmmm, can't reproduce on Ubuntu 14.04 LTS either. What's your keyboard layout?

@Prior99 can you upload a file demonstrating the issue somewhere?

@postcasio https://cronosx.de/share/download.php?id=8d23cf6c86e834a7aa6eded54c26ce2bb2e74903538c61bdd5d2197997ab2f72
The file looks like this:

@mnquintana
[fg716463@rz-e4310 cluster]$ locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
[fg716463@rz-e4310 cluster]$ localectl
   System Locale: LANG=en_US.UTF-8
       VC Keymap: de-latin1-nodeadkeys
      X11 Layout: n/a

Maybe the issue is related to me not having set LC_ALL?
But this should not affect Atom in such way.

Update: I tested this on a second machine, also running arch linux and can not reproduce it there.
This seems to be related to a misconfiguration on the first machine or version mismatches.
The machine it works on:
[prior@cronosl ~]$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE=en_US.UTF-8
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
[prior@cronosl ~]$ locale
locale      localectl   localedef   locale-gen
[prior@cronosl ~]$ localectl
   System Locale: LANG=en_US.UTF-8
       VC Keymap: de-latin1
      X11 Layout: n/a
[prior@cronosl ~]$ uname -a
Linux cronosl 3.17.3-1-ARCH #1 SMP PREEMPT Fri Nov 14 23:13:48 CET 2014 x86_64 GNU/Linux
[prior@cronosl ~]$ atom -v
0.124.0-024d7f8

The machine this bug occurs on:
[fg716463@rz-e4310 cluster]$ locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
[fg716463@rz-e4310 cluster]$ localectl
System Locale: LANG=en_US.UTF-8
VC Keymap: de-latin1-nodeadkeys
X11 Layout: n/a
[fg716463@rz-e4310 ~]$ uname -a
Linux rz-e4310 3.17.4-1-ARCH #1 SMP PREEMPT Fri Nov 21 21:14:42 CET 2014 x86_64 GNU/Linux
[fg716463@rz-e4310 ~]$ atom -v
0.170.0-b39cee3

Let me update the second machine to the latest versions of atom and compare the issue again.

Update:
Both machines updated to atom 0.172.0-366ded2 without effect.

Update:
Fixing the corrupted locales on the machine the issue occured on fixed the issue. The file is displayed correctly now.
I would however not to close this issue but suggest to display a warning message when corrupted locales are detected.

@Prior99 Nice work! It definitely seems to me like Atom should display a warning notification when corrupted locales are detected, but it might be better to close this one and open a new issue just for that (to increase visibility / relevance for the core team).

Opened new issue #5073 and closing this issue accordingly.


docker-compose failure

Docker compose sometimes spits back this message:
rabbitmq_1 | /usr/lib/rabbitmq/bin/rabbitmq-server: 42: /usr/lib/rabbitmq/bin/rabbitmq-server: cannot create<EMAIL_ADDRESS>Permission denied
redis_1    | [1] 03 Apr 17:04:22.583 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1    |                 _._
redis_1    |            _.-``__ ''-._
redis_1    |       _.-``    `.  `_.  ''-._           Redis 2.8.19 (00000000/0) 64 bit
redis_1    |   .-`` .-```
.
```\/    _.,_ ''-._
redis_1    |  (    '      ,       .-`  | `,    )     Running in stand alone mode
redis_1    |  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
redis_1    |  |    `-._   `._    /     _.-'    |     PID: 1
redis_1    |   `-._    `-._  `-./  _.-'    _.-'
redis_1    |  |`-._`-._    `-.__.-'    _.-'_.-'|
redis_1    |  |    `-._`-._        _.-'_.-'    |           http://redis.io
redis_1    |   `-._    `-._`-.__.-'_.-'    _.-'
redis_1    |  |`-._`-._    `-.__.-'    _.-'_.-'|
redis_1    |  |    `-._`-._        _.-'_.-'    |
redis_1    |   `-._    `-._`-.__.-'_.-'    _.-'
redis_1    |       `-._    `-.__.-'    _.-'
redis_1    |           `-._        _.-'
redis_1    |               `-.__.-'
redis_1    |
redis_1    | [1] 03 Apr 17:04:22.584 # Server started, Redis version 2.8.19
redis_1    | [1] 03 Apr 17:04:22.584 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1    | [1] 03 Apr 17:04:22.584 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1    | [1] 03 Apr 17:04:22.584 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1    | [1] 03 Apr 17:04:22.590 * DB loaded from disk: 0.006 seconds
redis_1    | [1] 03 Apr 17:04:22.590 * The server is now ready to accept connections on port 6379
scripts_rabbitmq_1 exited with code 2
Gracefully stopping... (press Ctrl+C again to force)
Stopping scripts_redis_1...
Stopping scripts_mongodb_1...

When that happens, you must remove data_rabbit and data_mongo directories.
We should update the readme..

I fixed the docker-compose.yml not to use a local directory for data volumes (not sure what I was thinking the first time). Should be much smoother now.


invariant violation : view config not found for name RNSVGPath

my code --
 import { G, Path } from 'react-native-svg';
 import SvgIcon from 'react-native-svg-icon';

  const SVG = {
      SortArrows: <G><Path d="M0 45h90L45 0 0 45z"/><Path d="M0 55h90l-45 45L0 55z"/></G>,
        Tick: {
                svg: <Path d="M38.729 75.688a4.48 4.48 0 0 1-3.21-1.356L16.558 55.004c-1.774-1.807-
                1.774-4.736-.001-6.543a4.48 4.48 0 0 1 6.42 0l15.753 16.056 37.749-38.474a4.478 4.478
                 0 0 1 6.419 0c1.773 1.806 1.773 4.736 0 6.543L41.939 74.332a4.48 4.48 0 0 1-3.21
                 1.356z"/>,
                  viewBox: '0 0 50 50',
  },
}

    const Icon = (props) => <SvgIcon {...props} svgs={SVG} />;

    export default class App extends React.Component {
     render() {
         return(

         <React.Fragment>
            <Icon name="SortArrows" height="20" width="20" />
            <Icon name="Tick" fill="#0f0" viewBox="0 0 200 200" />
            <Icon name="Star" fill="transparent" stroke="#fff" strokeWidth="5" />
         </React.Fragment>

    );
}
 }

This sounds like react-native-svg isn't linked properly. Can you see if that module by itself works?

sorry im new in RN , how can i see if that module by itself works?
and thanks for answer

again im change my code to
import {Svg, G, Path, Circle } from 'react-native-svg';
export default class App extends React.Component {
 render() {
    return(

          <React.Fragment>
           <Svg
             height="100"
             width="100"
         >
            <Circle
                  cx="50"
                  cy="50"
                  r="45"
                  stroke="blue"
                  strokeWidth="2.5"
                  fill="green"
             />
          </Svg>
          </React.Fragment>

       );
   }
   }

and give this Erro now

Looks like you may need to relink react-native-svg. It can be a bit finicky at times. They have some instructions for manual linking if automatic doesn't work.

I had the exact same issue and I can confirm, relinking with rnpm link react-native-link works like a charm.


bug

Environment (please complete the following information):

OS (e.g., Windows 10):
Extension version (e.g., 1.2.3):

Describe the bug:

Repro steps:

Expected behavior:

Additional context:

Closing as incomplete. Please provide more information about your issue if you have any.


Add attempt limiter to Retry Prune Networks

In the CI, we are constantly seeing errors like this.
This is occurring after the test has run, when trying to clean up docker.
    setup.go:226: Failed to prune networks during docker cleanup: All attempts fail:
        #1: Error response from daemon: a prune operation is already running
        #2: Error response from daemon: a prune operation is already running
        #3: Error response from daemon: a prune operation is already running
        #4: Error response from daemon: a prune operation is already running
        #5: Error response from daemon: a prune operation is already running
        #6: Error response from daemon: a prune operation is already running
        #7: Error response from daemon: a prune operation is already running
        #8: Error response from daemon: a prune operation is already running
        #9: Error response from daemon: a prune operation is already running
        #10: Error response from daemon: a prune operation is already running
panic: test timed out after 30m0s

This retry command will now give up after 5 retries if there is already a network prune in progress. It will not return an error if it was unable to prune networks.
I think this is OK, since I've only ever seen this error in the Github CI.

closes: https://github.com/strangelove-ventures/interchaintest/issues/532

Closing. Have not seen this issue since bumping the docker version.


Some imports vanish during organize imports

e.g.
injectable Logger from the parser source

Hey @n00dl3
Alright I found the bug:
All Identifiers were parsed, but not those on class definitions. So if you have an identifier inside a class (i.e. PropertyAssignment or whatnot), the identifier was not parsed.
I tried it with multiple files. It should work as expected right not.


[WIP] plugins: add RoonBridge plugin

This is a first attempt to add RoonBridge as a plugin (and familiarise myself with balenablocks at the same time!).
Currently, the roon support when it is pushed to balenaCloud as a single container works perfectly. However when running with all of the other services (audio, supervisor etc etc) it fails with a Device or resource busy error:

To me it smells like me missing some pulseaudio / alsa config so the roon container can talk to the audio container but I need to spend a bit more time debugging.
Also - running the install script from Roon in the build process currently fails due to this issue - https://github.com/balena-io/balena-builder/issues/826 - so I have currently got the script directly in the repo.

Some progress...
root@2ec9b5a:/opt/RoonBridge# speaker-test -c2 -t wav

speaker-test 1.1.8

Playback device is default
Stream parameters are 48000Hz, S16_LE, 2 channels
WAV file(s)
Rate set to 48000Hz (requested 48000Hz)
Buffer size range from 96 to 1048576
Period size range from 32 to 349526
Using max buffer size 1048576
Periods = 4
was set period_size = 262144
was set buffer_size = 1048576
 0 - Front Left
 1 - Front Right

But RoonBridge still seems to fail with Failed to open the audio device - so must be something in the alsa / pulse config of roonbridge I guess...

Weirdly, it works with the onboard Pi audio / headphone jack - but it doesn't like the soundcard. Will try with a few other soundcards and see if that makes a difference...

Beginning to think maybe it is some permissions issue with accessing an external soundcard or something?

Weirdly, it works with the onboard Pi audio / headphone jack - but it doesn't like the soundcard. Will try with a few other soundcards and see if that makes a difference...

Beginning to think maybe it is some permissions issue with accessing an external soundcard or something?


Reference the Asset microservices documentation

Asset microservices custom workers are headless Project Firefly apps.
We should add a reference to the documentation for developers looking for specific details and related walkthrough.
Link: https://docs.adobe.com/content/help/en/experience-manager-cloud-service/assets/manage/asset-microservices-configure-and-use.html

Good point! I've mentioned the same in my docs :)


Expose url configuration for mongoDB

merging with https://github.com/run-llama/helm-charts/pull/28


Flubu.Runner: Some iis tasks don't work because of wrong assembly manifest definition

Could not load file or assembly 'System.Reflection.TypeExtensions, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
https://github.com/dotnet/standard/issues/282

Fixed with .net core sdk 2.0


feat: add quantityLabel prop to ProductDetail

Builds of #338
Adds quantityLabel prop to ProductDetail that is then forwarded to form and set as an aria-label to the quantity field since we don't have a visible label.

Closing #338 in favor of this PR.


Add hourly CEMS (for real this time)

Sorry for all the mess of changes.  Reverting a reverted commit isn't great, but it's where we are. The tests pass (but take forever because loading the CEMS is slow.)
Notable changes
Postgres

Switch to an in-memory CSV for all postgres copy_from operations

By default, CSVs are not written to disk

Batch the CEMS imports, accumulating 300 MB of DataFrames before writing and committing to postgres

I picked 300 MB because the code holds all that data and its CSV representation in memory, but increasing it should lead to fewer postgres commits and faster CEMS load time. What seems reasonable to you?

Total load time is now around 5 hours, which is better than the >20 I had without batching the imports, but still grim.

Downloads

All failing downloads get retried at least once
FTP downloads use a single connection for all files in a source-by-year (currently only affects CEMS)
Downloads are done in parallel (at the source-by-year level) with concurrent.futures.ThreadPoolExecutor.

Someone on Windows should test that this works for them.

CEMS specific

Switch to new EPA FTP site
Only load data through 2016 because the current implementation deals with a whole year at a time, and 2017q4 isn't ready.
Make the CEMS table unlogged for performance
fac_id and unit_id are currently being dropped

Remaining to-do

[ ] Create date and time columns as interval
[ ] Calculate UTC time (cc @gschivley)
[ ] Think about making the CEMS import asynchronous—read next file while postgres is ingesting
[ ] Postgres optimization: add indexes and run analyze

Look at further import optimizations here: 1 and 2
Maybe make postgres_copy stop committing after each operation
I set no indexes in the SQLAlchemy model because it's faster to write indexes after all the bulk copy froms are done, but we will want to add indexes to a few of the columns.

But this PR doesn't need to wait on those.

This week and next are both busy for me but I want to put in a couple thoughts on calculating offset from UTC:

With lat/lon from either 860 or the EIA API it's possible to find the timezone of a power plant.
All times in CEMS are local, standard (no daylight savings). Using standard timezone/tzwhere tools will screw up DST. Find an offset from the location in January rather than using something like datetime.now.


Support reading Iceberg equality delete files (Design)

Description
In https://github.com/facebookincubator/velox/pull/7847we introduced IcebergSplitReader and the support of reading positional delete files. In this doc we will discuss the implementation of reading equality delete files.
Iceberg Equality Deletes Overview
A general introduction of equality delete files can be found at https://iceberg.apache.org/spec/#equality-delete-files. Some key takeaways:

An equality delete file can contain multiple fields(could be sub-fields), and the values for the fields in the same row are in AND relationship. E.g. The following equality delete file

equality_ids=[1, 3]
1: id  | 2: category | 3: name
-------|-------------|---------
 3     | NULL        | Grizzly

means:
    - A row is deleted if (id = 3 AND name = 'Grizzly') is true. Or
    - A row is selected if (id <> 3 OR name <> 'Grizzly') is true

The equality delete field value could be NULL, which means a row is deleted if that field is NULL.

equality_ids=[2]
1: id  | 2: category | 3: name
-------|-------------|---------
 3     | NULL        | Grizzly

The expression specifies:
    - A row is deleted if category IS NULL. Or
    - A row is selected if category IS NOT NULL

An equality delete file could contain multiple rows.

equality_ids=[1, 3]
1: id | 2: category | 3: name
-------|-------------|---------
3      |    NULL     | Grizzly
5      |    Bear     | Polar

means:
    - A row is deleted if (id = 3 AND name = 'Grizzly') OR (id = 5 AND name = 'Polar')  is true. Or
    - A row is selected if (id <> 3 OR name <> 'Grizzly') AND (id <> 5 OR name <> 'Polar')  is true

A split can contain multiple equality or positional delete files, and a row is deleted if any row expression in these delete files is true. E.g. a split may come with 3 delete files:

Equality delete file 1
equality_ids=[1, 3]
 1: id | 2: category | 3: name
-------|-------------|---------
 3     | NULL        | Grizzly

Equality delete file 2
equality_ids=[3]
`1: id | 2: category | 3: name
-------|-------------|---------
 1     | NULL        | Polar

Positional delete file 1
	100
	101

means
    - a row is deleted iff (id = 3 AND name = 'Grizzly') OR (name = 'Polar') OR (row_in_file = 100)  OR (row_in_file = 101)
    - a row is selected iff (id <> 3 OR name <> 'Grizzly') AND (name <> 'Polar')  AND(row_in_file <>100)  AND (row_in_file <> 100)

A split can contain many equality and positional delete files.

Design considerations
Build Hash Tables or Filters/FilterFunctions?
The equality delete files can be interpreted as logical expressions and become the remaining filters that can be evaluated after all rows in a batch is read out into the result vectors. Or alternatively, they can be used to construct a number of hash tables that will be probed against after all rows are read  into the result vectors. Suppose the equality delete file contains the following information:
equality_ids=[2, 3]
 1: id | 2: category | 3: name
-------|-------------|---------
 1      |   Bear       | Grizzly
 3      |   Bear      | Brown

It means
- a row is deleted iff (category = 'Bear' AND name = 'Grizzly') OR (category = 'Bear' AND name = 'Polar')
- a row is selected iff (category <> 'Bear' OR name <> 'Grizzly') AND (category <> 'Bear' OR name <> 'Polar')

To build the hash tables, we will need to concatenate the hash values of column 2 and 3 together, and the hash table will contain two hash values for 'Bear##Grizzly' and 'Bear##Brown'. Then in the matching phase, the hash values of column 2 and 3 for all the rows in the output RowVectors would be calculated and concatenated before probing the hash table. If it's not a match, it means this row was definitely not deleted; if it is a match, then the row needs to be compared with the original delete values to confirm if it's really deleted. Only when all the values are the same the row shall be confirmed to have been removed. Note that the final comparison is necessary, because there is still a very small possibility that hash probe collision could happen, especially when there are many columns involved.
Note that creating hash tables on single columns is not correct without additional processing. For example, suppose the base file is as follows:
1: id | 2: category | 3: name
-------|-------------|---------
 1       |   Bear       | Grizzly
 2      |   Bear       | Brown
 3      |   Bear       | Polar
 4      |   Dog        | Brown

If we build one hash table on the second column "category" that contains {'Bear'}, and another hash table on name that contains {'Grizzly', 'Brown'}, then probing the category hash table to exclude rows with category = 'Bear' would incorrectly remove row 3, probing the name hash table to exclude 'Grizzly' and 'Brown' would incorrectly remove row 4. Taking logical AND or OR on the two probe results is also incorrect.
Now let's take one step back, and build the hashtables on single values. So we have hashtable A on "category" with one value 'Bear', and another hash table B on "name" with value "Grizzly" and another hash table C on "name" with value "Brown", then a row would pass if (category <> 'Bear' OR name <> 'Grizzly') AND (category <> 'Bear' OR name <> 'Polar')  by probing hash table A twice, and hash table B and C once, then compute the logical ORs and ANDs. However, this is no difference than just comparing the values and no hash tables are actually needed.
The other way is to compile these delete values into logical expressions that can be executed as the remaining filter functions, or even domain filters that can be pushed down to the base file reader. This can be more efficient than taking the hash table approach. Firstly, filter pushdown can eliminate a lot of decoding costs; Secondly, computing and concatenating the hash values for all rows are very expensive. In fact, it is much slower than just performing simple comparisons on single column values. The latter could be efficiently done by SIMD operations, while the hash value computation cannot be efficiently implemented using SIMD. And lastly, the values need to be compared anyways even for the hash table approach.
We should also notice that if we convert them into logical expressions as remaining filters, the existing Velox expression evaluation implementation can automatically choose the best evaulation strategy, e.g. whether to build hash tables, or do efficient SIMD logical comparisons when it sees fit. This is much more flexible than building fixed hash tables in the connector DataSources. In many cases, the ExpressionEvaluator can choose more efficient way to evaluate the equivalent expressions. Today, it's already very easy to construct the logical expressions, and for the equality delete to work, there is no additional code needed beyond the expression constructions what's so ever. The existing Velox data source and readers can already handle them, so the implementation would be fairly simple. Plus, we can additionally improve existing filter function / expression evaluation implementations that can potentially benefit other components of the driver pipeline in the future. So we propose to choose the remaining filter path and just convert the equality delete files into filters and filter functions.
Where and How to open the equality delete files
Query engines like Presto or Spark usually have some central coordinators, where the distributed plan and splits are created. The splits would then be sent to the workers and executed there using the Velox stack. A query may issue many splits for each table, and each of them may include many (may be up to hundreds) delete files. We have the choice to open the delete files in the coordinator, and/or in the workers. There are some basic design considerations here:
Native workers running Velox need to have the ability to read both equality delete and positional delete files.

Although it's possible for Prestissimo to open the (equality) delete files on the coordinator(s), we cannot assume other engines can process the equality delete files internally by themselves. The Iceberg splits come with a list of delete file paths and they could be positional or equality or both. With the normal scheduling implementations in most engines, the splits would be directly sent to the workers for local executions. An engine would need a fairly large amount of special handling to break this procedure and open the delete files, update the distributed query plan, creating filters that can be pushed down, or even change the scan into a join, etc. before sending out the splits. This makes the engines logic much more complex and the integration with Velox much harder.

Opening files is a very expensive operation. Opening all delete files on the coordinator may become the bottleneck of the system. Even though we could cache the parsed expressions from each distinct delete file, opening hundreds of them on a single or a small number of coordinator nodes is still not practical.

Based on these considerations, I think we need to implement reading equality deletes in Velox. However it doesn't mean we cannot open some of the equality delete files on the coordinator for optimization purpose. But that optimization should not be mandatory for the engines built on top of Velox.
Performance Considerations
We want to push down the filters as much, and as deep as possible.
By pushing down the filters to the readers or even decoder's level, we can efficiently avoid the costs of decoding skipped rows, or even save some decompression costs. This savings could be huge if the selectivity rate is very small. We shall notice that some of the equality delete files and all positional delete files could be converted to TupleDomain filters or initial row numbers that can be pushed to the readers. In order to achieve this, we will need to extract the parts that can be pushed down, and guarantee the rest parts are evaluated or tested correctly.
We want to avoid opening the delete files as much as possible
A split may include hundreds of delete files, and a worker could receive many splits with the same set of delete files. Ideally, each delete file should be opened only once on one worker. This is because 1) opening files is expensive 2) expression compilation, or building hashtables that can be later probed are also not cheap. There're a couple of ways to achieve this

Building a hash table for each HiveDataSource, or a long living cache on the compiled expressions on each node

Convert the scan with equality delete files to a broadcast join, with the delete files becoming one data source, and the base file becoming another data source. This shows good improvements in Presto, but it also misses the opportunities of

filter pushdown for reading the base file,
more efficient evaluation comparing to hash joins
Cross file filters and filter functions merge
ExpressionEvaluator's ability to extract common sub-expressions
logical expression simplifications
Simpler plan shape, less data movement
No additional data size limit for broadcast joins

We want to reduce the amount of expression evaluation as much as possible.
We have shown that the equality delete files can be interpreted as some conjunctive logical expressions. However, logical expression evaluations are also expensive when the expression contains many terms. We notice that Velox can already extract common sub expressions and flatten adjacent logical expressions, but the more general logical expression simplifications is still not implemented. Nonetheless, there are some ways to simplify the expressions for some simple cases for Iceberg. We will discuss them later.
Design
EqualityDeleteFileReader
We will be introducing the EqualityDeleteFileReader class, and each reader is responsible for opening one equality delete file. The content will be read in batches, and for each batch, the logical expressions will be built and merged with existing remainingFilter in the HiveDataSource.
The equality delete file schema is not fixed and can only be known at query run time. The equality Ids and the base file schema are used together to get the output row type and build the ScanSpec for the equality delete file. The equality Ids are the same as the id in Velox TypeWithId, and therefore we can directly use dwio::common::typeutils::buildSelectedType() to get the delete file schema. Note that this Id is not necessarily from a primitive type column, but could also be a sub-field from a complex type column. For example, deleting from an ARRAY[INTEGER] column c where c[i]=5 can also be expressed as an equality delete file. The field Ids for this column is
0: root
1: ARRAY
2: INTEGER

Therefore the equality id for this predicate is 2, and the content of the equality delete file is value 5.
Once we read the delete values, we can build the ExprSet and add it to the existing remainingFilterExprSet_ in HiveDataSource. Then the expresionEvaluator_ in HiveDataSource will evaluate them after all relevant vectors are loaded. There are two ways to add the newly created ExprSet to the existing remainingFilterExprSet:

By conjuncting with the Expr in remainingFilterExprSet_
By adding the Expr to the array in remainingFilterExprSet_

Note that the current HiveDataSource assumes remainingFilterExprSet_ has only one Expr, and the owned SimpleExpressionEvaluator only evaluates the first Expr in an ExprSet. There're a couple of facts that we discover:

SimpleExpressionEvaluator is only used in TableScan and HiveConnector
The remainingFilterExprSet_ would always be a special kind of ExprSet that it only contains logical expressions.

While I think SimpleExpressionEvaluator should indeed evaluate all Exprs in the passed in ExprSet, I think we can alternative create a new LogicalExpressionEvaluator, in which we can have special logical expression evaluation improvements in the future. Then it seems that adding the new Expr to remainingFilterExprSet_ as an array element is the most clean and simple way.
Extraction of domain filters
When the equality delete file only has one field, we can extract it as a domain filter. Such filter can be pushed down to the readers and decoders, where performance savings could happen. In this case we will create a NOT IN filter for it. This is done in connector::hive::iceberg::FilterUtil::createNotInFilter(), which in turn would call into the utility functions in common:Filter.h/cpp. The values will be de-duplicated and nulls will be treated separately. Velox can optimize it into different kinds of filter, e.g. a range filter when there is only one value.
Note that we need to verify the field is not a sub-field, since Velox currently doesn't support pushing down filters to sub-fields. This restriction will be removed once Velox supports sub-field filter pushdowns.
An equality delete file with multiple fields cannot be pushed down as domain filters at this moment, no matter if there's a single row or multiple rows. E.g. this delete file can be interpreted as id <> 3 || name <> 'Grizzly'. Currently Velox does not support pushing down disjunctives but we may do it in the future.
equality_ids=[1, 3]
1: id | 2: category | 3: name
-------|-------------|---------
 3     | NULL        | Grizzly

Domain Filter Merge
A split may come with multiple equality delete files. Some of them may have the same schema. If they all have the same single field, the extracted domain filters will be deduped and merged with the existing one.  E.g.
Equality delete file 1
equality_ids=[2]
2: category
---------------
    mouse

Equality delete file 2
equality_ids=[2]
2: category
---------------
    bear
   mouse

The domain filter built from these 2 files will be category NOT IN {'bear', 'mouse'} This is using  the mergeWith api in the Filter class.
Remaining Filter Function Merge
If the equality delete files have the same schema but not the single field, For example
Equality delete file 1
equality_ids=[1, 3]
1: id | 2: category | 3: name
-------|-------------|---------
 3     | NULL        | Winnie

Equality delete file 2
equality_ids=[1, 3]
1: id | 2: category | 3: name
-------|-------------|---------
 4     | NULL        | Micky
3     | NULL        | Winnie

This will create 2 Expr in the final ExprSet:
 - (`id <> 3 || name <> 'Winnie')
 - (`id <> 3 || name <> 'Winnie') && (`id <> 4 || name <> 'Micky')

Today Velox supports common sub-expressions recognition in the ExpressionEvaluator, and such expression would be evaluated only once. In this example  (`id <> 3 || name <> 'Winnie') evaluation result would be cached internally and does not need to be evaluated twice.
Logical Expression Simplification
As far as I understand, Velox can do logical expression flattening, but still can't automatically simplify the logical expression. For example, the expression a AND (b AND (c AND d)) would be flattened as AND(a,b,c,d), but a AND (a OR b) cannot be automatically simplified to a, therefore to evaluate a AND (a OR b), a and b will both be evaluated, and one AND and one OR operation need to be performed. While we hope to improve logical expression simplification in the future, we can still do some simple improvements for Iceberg now.
An Iceberg split can come with multiple equality delete files and their schemas could have overlaps. For example
Equality delete file 1
equality_ids=[1, 2, 3]
1: id | 2: category | 3: name
-------|-------------|---------
 1      |   mouse   | Micky
 2      |   mouse   | Minnie
 3      |     bear     | Winnie
 4      |     bear     | Betty

Equality delete file 2
equality_ids=[2]
2: category
---------------
   mouse

Equality delete file 3
equality_ids=[2, 3]
2: category  | 3: name
----------------|-------------
   bear           | Winnie

We see that equality delete file 2 is on the category column and would remove all tuples with value mouse. This means that the first two rows in equality delete file 1 are already contained and doesn’t need to be read or compiled. Similarly, the single row in file 3 contains row 3 in file 1, therefore row 3 in file 1 doesn’t need to be read or compiled. The simplified delete files are like the follows:
equality_ids=[1, 2, 3]
1: id | 2: category | 3: name
-------|-------------|---------
 4      |     bear     | Betty

and
equality_ids=[2]
2: category
---------------
   mouse

and
equality_ids=[2, 3]
2: category  | 3: name
----------------|-------------
   bear           | Winnie

With this simplification, the resulted expression would be simpler and the evaluation cost will be reduced.
When the delete file only has one field, the domain filter built from it can be used as a filter when reading other equality delete files whose fields include this one. In the above example, category <> 'mouse' can be pushed to file 1, whose row 1 and 2 would be filtered out. This not only helps final expression evaluation, but also improve the read performance for reading file 1.
If the delete file has more than 1 field, the situation is more complex. In the above example, file 3 would be compiled to category <> 'bear' OR name <> 'Winnie, but it cannot be pushed to file 1 nor the base file directly because it's a disjunctive expression. So far Velox only supports domain filters in conjunctive expressions. So for now we will only use single field equality delete files to do the simplifications. For this, we will go over the equality ids from all equality delete files and pick all single field ones to read first. Then the filters will be pushed to the other equality file readers.
In the future, we can even implement disjunctive expression push downs. For example category <> 'bear' OR name <> 'Winnie can be pushed to the SelectiveColumnReaders, with the category and name columns as a ColumnGroup. This will save the cost of having to read all values out before applying the filter function as a remaining filter, and the selectivity vector can be reused among them. Moreover, the reduction of rows from applying this filter directly on this ColumnGroup would benefit the reading of other columns later.
Expression Caching
We know that a unique HiveDataSource object is created for a unique TableScan operator, and the splits received by a HiveDataSource instance belong to the same query and same table. Additionally for Iceberg splits, they must be reading the same snapshot of an Iceberg table. When the HiveDataSource  receives a new Iceberg split with some equality delete files, it would create a new IcebergSplitReader, which would open the delete files. If the equality delete file can be interpreted into some domain filters or filter functions,  the scanSpec_ and remainingFilterExprSet_ in HIveDataSource may need to be updated.
Currently, the Iceberg library selects the qualified data and delete files based on partitions and snapshot Ids or transaction sequence numbers. For a single transaction, the snapshot is fixed, and all delete files from the same partition would go with the base data files when the splits are enumerated. So we can assume for now that all splits received from the same partition are the same for a single HiveDataSource. However, the delete files for different partitions could be different, and the splits from multiple partitions could arrive out of order. If we updated the scanSpec_ and remainingFilterExprSet_ for previous partition, we will need to restore them back to the original before applying the current set of delete files. As the first implementation, we will make a copy of these objects in the IcebergSplitReader and restore them back when the IcebergSplitReader is destructed.
In some user's workloads, the deletions are quite frequent, and the number of delete files coming with a split for a subsequent SELECT query can be many. For all splits in a partition, the delete files may be the same. We don't want to repeatedly read such equality delete files for every split a HiveDataSource needs to handle. One way of overcoming this is to build an expression cache. There are 2 levels of the caching ideas:

A hash table in HiveDataSource
A process wide cache for all Iceberg scans.

In 1, the key of the hash table is <partition, snapshotId> and the values are the compiled filters and expressions.
In 2,  the key of the cache is <table, partition, snapshotId>  and the values are the compiled filters and expressions. To avoid excessive contentions, we can divide the cache into multiple levels. The implementation will be adjusted with more experiments and observations of the customer workloads in the future.
If the Iceberg library changes its TableScan or FileScan in the future and can additionally prune the delete files based on each individual base data files, we will need to change the cache keys and add the information for the base file.
We will work on caching in the future when we understands the workloads better.
LogicalExpressionEvaluator Improvements
Currently the remaining filter is evaluated in HiveDataSource::evaluateRemainingFilter()
vector_size_t HiveDataSource::evaluateRemainingFilter(RowVectorPtr& rowVector) {
  …
  expressionEvaluator_->evaluate(
      remainingFilterExprSet_.get(), filterRows_, *rowVector, filterResult_);
  auto res = exec::processFilterResults(
      filterResult_, filterRows_, filterEvalCtx_, pool_);
  return res;
}

This code evaluates the remainingFilterExprSet_  as a general expression instead of a special logical expression, and would put the result of the Expr's in remainingFilterExprSet_ in a FlatVector as bool type, then processFilterResults() would perform logical AND/OR on these vectors. This incurs additional memory copies. Moreover, it contains special handling of nulls, while the logical expressions would NOT produce NULLs at all, so this part of cost can be saved as well. Our newly introduced LogicalExpressionEvaluator will have its own evaluate() implementation that is more performant for logical expressions.
Testing
Prestissimo End To End Tests
In addition to unit tests we will have

TPCH, TPCDS built-in tests in Presto
IcebergExternalWorkerQueryRunner

Microbenchmarks
We will build microbenchmarks in Velox and Presto and cover both delete files.

cc @tdcmeehan @nmahadevuni

Agree that this must be done in worker/Velox not in coordinator
We need to support multi-column ID pushdown, so build a hash table with multi-key support is needed.  The remaining filter method only works for single column key.

@tdcmeehan Thanks for the explanation, so from the point of view of execution engine that can be used in both batch and streaming cases, equality deletion is used more frequently in row oriented formats with keys, e.g. Avro?  And Avro only has single column key, so the priority is not high to support multi column key deletion.
However I think as a general execution engine, we need to get it right (at least at design level) in the first place, so that if someone comes with a row-oriented format with multi-column keys, we can cope with it without major change.  In this sense I still think investment in proper support of multi-column keys is worth doing.
Also another point is in big company sometimes the producer of the data is not aware of the consumers.  It's not uncommon that streaming data is stored in data warehouse and later read by a different tool.  So on read side we need to be as flexible as possible.

@Yuhta I don't think it's necessarily correlated to the underlying file format, it's really whether or not the system generating the deletes wants or is capable of performing a table scan.  This is because positional deletes require the position, which requires a table scan to determine the row that is to be deleted.  I think this applies equally to any file format.
To be clear, I believe it is beneficial and useful for Velox to support reading equality deletes as it is consistent with Velox's mission to be a pervasive execution layer in query engines.  I am simply adding some context on when one typically sees equality delete files in real world settings.

We need to support multi-column ID pushdown, so build a hash table with multi-key support is needed. The remaining filter method only works for single column key. I can work on adding a common interface for multi-key hash table in mutation pushdown.

@Yuhta The remaining filter approach already supports multi-column expressions, e.g. a <> 1 OR b <> 1. My draft PR https://github.com/facebookincubator/velox/pull/8728 already works for that case and there is a test for multiple column delete values.
And as I explained in "Build Hash Tables or Filters/FilterFunctions?" section, I believe the remaining filter approach is more advantageous than building additional mutation hash tables in all dimensions including performance, implementation easiness, and code cleanliness. I don't think you need to add any new interface for multi-key hash table if you just go with the remaining filter way.
Also, correct me if I understood this wrong: I think the "Disjunctive(OR) predicate pushdown" I mentioned is different than your "multi-column ID pushdown". The essential point is to push the OR predicates to the ColumnReaders and decoders as domain filters, so the benefit of filter pushdown can be honored. E.g. In a predicate like (a = 1 OR b = 1) AND (c = 1), today we can only push down c = 1, but my idea is that we can also push (a = 1 OR b = 1) down to the ColumnReader level in the future. Of course this needs fundamental changes in the current reader implementations. It was an idea emerged while I was working on Iceberg, and I don't think any other engines have it. I want to try it in the future but not the near future. Whereas the "multi-column ID pushdown" you mentioned seems to be AFTER all data is read since you mentioned you wanted to build hash tables in mutation object, thus won't have the benefit of filter pushdown.

Thanks @tdcmeehan for the background introduction. While not being able to do a scan WAS the reason why some engines produce equality delete files, the performance benefit will be another major reason why users want to use equality deletes in the future. I believe equality deletes WILL out-perform the positional deletes after we implement it in Prestissimo, because 1) we will save a big scan and semi join in the delete query. 2) we can push down some of the equality delete as domain filters to Velox Column readers, plus some other optimizations that are specific to equality delete files only. Given the fact that many engines scan is not as efficient, making the "merge" happen in Velox/Presissimo will have better performance. Also the dynamic filter support is limited nowadays, but for equality delete, we can definitely pushdown domain filters while not worrying about the data size etc. So we will use equality delete in TPCDS publication and implement equality delete in Prestissimo in the next step.

@yingsu00 For single column key we can push it down to decoders, but I don't think you can do the same for multi-column keys due to the correlation between columns.  Putting huge list of ORs in remaining filter would just destroy the performance.  So

For single key, we merge a filter to the corresponding column, which will be pushdown to decoder level
For multi-key, we need to build the hash table and filter after we read all the key columns

@yingsu00 For single column key we can push it down to decoders, but I don't think you can do the same for multi-column keys due to the correlation between columns. Putting huge list of ORs in remaining filter would just destroy the performance. So

For single-column key, we merge a filter to the corresponding column, which will be pushdown to decoder level
For multi-column key, we need to build the hash table and filter after we read all the key columns (or can you show me how do you push down OR filter on multiple columns?)

@Yuhta Thanks for your questions. This is a very preliminary idea now. It's essentially to pushdown the expression evaluation into ColumnReaders for some special expressions like logical expressions with non-overlapping columns.  e.g. for expression (b=1 OR c=1) AND a=1, right now we will have a domain filter a=1 which will be pushed down to the ColumnReader, and all rows passing a=1 would be decoded and extracted into a b vector and a c vector, then either build a hash table directly or utilize existing expression evaluation framework to evaluate (b=1 OR c=1). Note that this part is done relative less efficiently now, since the expression evaluation is aimed for all general cases, and calculating hash values for multiple columns is expensive. Also all rows, even those don't satisfy (b=1 OR c=1), would have to be decompressed, decoded and copied out in order to evaluate (b=1 OR c=1).
Now, if we can create a ColumnReader to read a first(as is done today), and a GroupColumnReader that contains ColumnReader for b and c, then after a was read, the GroupColumnReader's read() function would be

The inner ColumnReader for b would produce a bitmap for b=1, and the inner ColumnReader for c would produce a bitmap for c=1, note that it doesn't extract or copy the data out at this moment. If we push down the filter to the encoded data, we don't even need to decode the data now.
The GroupColumnReader can directly OR the two bitmaps and produce the rowset passing (b=1 OR c=1).  Unlike the hash table or general expression evaluation, this doesn't need to allocate new memory and can be done really fast.
If b or c are not required to be extracted, we are done. Otherwise extract the data with the ORed bitmap. This may need to decode more data than step 1, but now the data is all in memory and extracting them now may still be a lot faster than reading them all and filter later.

This has multiple benefits:

Filter b=1 and c=1 can be pushed down on column b and c respectively, thus avoiding the cost to decode unnecessary data
It may benefit reading the rest columns other than a,b,c, if any, if b=1 OR c=1 can remove a lot of rows.

So I think it would benefit the performance generally, especially after we push the filters to ENCODED data. Even if we don't push the filters to ENCODED data, it may still be faster in a lot of cases.
You said " Putting huge list of ORs in remaining filter would just destroy the performance", I think you meant the case that the number of values in the expression is much smaller than the number of row to be read, such that the bitmap may be larger than the hash table. But remember, you have to extract and copy all rows first, which is mostly larger than the bitmap itself. For each batch, we read at most 10,000 rows, and the bitmap for it would just be around 1KB for each column. Actually, the RowSet itself for each ColumnReader nowadays may be larger than that. Even your hash table may be larger than several KBs itself. And most importantly, calculating hash values on all relevant columns for all rows may be much more expensive.
But I agree that in some cases building hash table or evaluating the remaining filter afterwards may be faster, e.g. when the equality delete file contains many many columns. So I think the execution strategy should be self-adapted at run time. And the criteria and policy shall be determined on extensive performance testing.
Anyways, this idea is preliminary and needs a lot of refinement and generalization. I may try prototyping something next year, but not in this Iceberg work. But your feedback and discussion are very welcome!

The inner ColumnReader for b would produce a bitmap for b=1, and the inner ColumnReader for c would produce a bitmap for c=1, note that it doesn't extract or copy the data out at this moment. If we push down the filter to the encoded data, we don't even need to decode the data now.

At this point you already go through all the key data and pay almost the price of reading them.  The saving from selective reading is mainly by skipping, in this case you cannot skip reading b according to the values you read in a, so it will not be much different from using a hash table, and the framework would be much more complex.  The saving we are aiming for is on payload data, so pushing down to key column readers does not seem worth doing it except for single column.
For the first implementation I would suggest we do the single column case only, since that is the thing everyone is agreed on, and covers most of the real world use cases.

At this point you already go through all the key data and pay almost the full price of reading them (decompressing and decoding,

Not quite. The GroupColumnReader does need to decompress the whole ColumnChunk, but it does NOT necessarily need to decode all data, if we can evaluate the filter on ENCODED data.

you cannot skip reading c according to the values you read in b, so it will not be much different from using a hash table

I think there will be a difference, but I don't have time to try it now so we can forget it. Whether to try it or not also depends on the portion of the remaining filter or hash table mutation cost in the whole query. If it's already fast then no need to do it. We will send a PR for IcebergReadBenchmark and it will cover the equality delete case. Then we will have more insights on where the time is spent.

Also I don't see we can use this to speed up mutability. The method does not work well with tree of logic expression. How do you push down (b = 1 OR c = 1) AND (b = 2 OR c = 2)?

The naive way is to have two GroupColumnReaders, one for (b = 1 OR c = 1) and the other for (b = 2 OR c = 2). Thus we'll have to read b and c twice, but we can skip some rows for (b = 2 OR c = 2), and also use filter on ENCODED data to avoid decoding all data.
Then the improved version is to

Get all distinct single columns and the predicates on them. In this example, b=1, b=2, c=1, c=2. Note that we don't need to care how they are joined together
While reading a column, apply all relevant filters on the encoded data. In this example, we can apply b=1 and b=2 on b. The comparisons can be done with b loaded in some registers, so we don't need to load b twice. Then each of them will produce a SelectivityVector.
Then perform the logical computations to get the final SelectivityVector, and extract the values if necessary.

In this approach, each row uses 4 comparisons and 3 logical ops.
This approach can also be applied to the LogicalExpressionEvaluator. The current general ExprSet evaluation can recogonize common expressions. If the expression is (b = 1 OR c = 1) AND (b = 1 OR c = 2), then b=1 is common and will only be executed once. But for (b = 1 OR c = 1) AND (b = 2 OR c = 2) there is no common expression, and both b and c would be read twice.
I agree this approach has big limitations, it only works for logical expressions, and doesn't work for complex functions and those involve multi-columns like x+y > 1. But it for Iceberg it's good.
Now let's consider the hash table approach for (b = 1 OR c = 1) AND (b = 2 OR c = 2), for this we'll need to build 4 hash tables b = 1, c =1, b = 2, c = 2, and the probe results will need to be ANDed and ORed to get the final result. To build the hash tables, you'll need to apply hash function on these 4 values(actually 2 distinct, 1 and 2, but you don't know if they're distinct). And to verify if a row satisfies, you will need to hash 2 values and use 4 probes. You can say if b=1 then don't need to probe b=2, but this if check is a perf killer. Alternatively we can convert it to disjunctive of conjunctives: (b=1 AND c=2) OR(c=1 AND b=2), then you need one hash table on b##c that contains two values 1##2 and 2##1. This looks better, and building the hash table requires 4 hash and some bit shift, and probing requires 2 hash and some bit shift for every row. If it's a match, you will need to compare again if the values really match. Let's be generous and don't cost the hash table build. Then for each row, it needs 2 hashes, 1 probe, and 2 comparisons. It may look faster than 4 comparisons and 3 logical ops, but hash is many times slower than comparison, and probe is also costly. While using the above approach, or the improved expression evaluation, the 4 comparisons and 3 logical ops can be done in a very simple loop and all in simple arrays. So for this simple case the hash table will be slower.
But when there are many many values, e.g. (b = 1 OR c = 1) AND (b = 2 OR c = 2) AND (b = 3 OR c = 3) ... AND (b = 1,000,000 OR c = 2,000,000), I would expect the hash table approach be faster.

For the first implementation I would suggest we do the single column case only, since that is the thing everyone is agreed on, and covers most of the real world use cases.

Yes this is exactly what was done in the PR https://github.com/facebookincubator/velox/pull/8728. It only pushes single column filters and the rest are evaluated as remaining filters. All rest optimizations are not included in this PR. Your review is much appreciated. I'll ping you when the tests pass.
Actually this particular optimization(push down disjunctives) is the last thing I may want to try, since it requires lots of code change and thus bigger risk. The other mentioned optimizations, e.g. logical expression simplifications and caching, will be tried first, if necessary. So we agree on this.

if we can evaluate the filter on ENCODED data.
That's the only way to achieve it.  However I am not sure if the gain worth it, it might be still slower than just copying the key columns out and probing a hash table.

(b = 1 OR c = 1) AND (b = 1 OR c = 2)
Sorry I made a mistake, the relevant expression should be (b = 1 AND c = 1) OR (b = 2 AND c = 2).  In this case we can use a single hash table supporting 2 column keys, but with the remaining filter approach we can only get a compromised filter on each column, and the filtering expression evaluation is still not avoidable.

Yes this is exactly what was done in the PR https://github.com/facebookincubator/velox/pull/8728. It only pushes single column filters and the rest are evaluated as remaining filters.
I would suggest we passing all these key information separately from presto_cpp into Velox.  Then we can decide in Velox, whether we want to build a hash table using these keys, or we just convert them into remaining filter expression.


Changed skipToNext and skipToPrevious from PUT to POST

Next and Previous api uses POST requests, and returns a 405 on PUT.
https://developer.spotify.com/web-api/skip-users-playback-to-next-track/

Coverage remained the same at 98.324% when pulling 3fdc91261a3f879c89a86d5152447506389eef1a on ikantspelgud:master into ac0fbe3dc4b902b9e3e610a712124feaa1e14f7f on JMPerez:master.

@ikantspelgud thanks for your PR!


Made QueryPart public

QueryPart is used in ParsedExpr, which is exported. QueryPart is the only field in the struct and will be used in the subsequent stages so should be public too.

LGTM