[bug]: commitment transaction dips peer below chan reserve: internal error causing eclair to FC

Background
For unknown reason (possible racecondition) my LND instance identified that a downstream HTLC commitment would have caused a channel peer balance going negative, and below our chan reserve for that channel.
That caused my LND to send my peer an internal error. @DerEwige running Eclair, which immediately going on-chain when receiving an internal error.
Two things to validate:

the negative balance might happen quite often, for instance when > 1 HTLCs opened at the same time. Why would this cause an internal error and not just a failing HTLC?
could we identify a different error message, being more verbose, to avoid other implementations interpreting this as a major issue and going on-chain?

Your environment

version of lnd: lnd version 0.16.2-beta commit=v0.16.2-beta
which operating system (uname -a on *Nix): Linux debian-nuc 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux
version of btcd, bitcoind, or other backend: Bitcoin Core version v24.0.1
any other relevant environment details: LND latest release AMD64 binary

Steps to reproduce
Log excerpt (full log grep of channel-id and pubkey attached)
2023-05-01 01:48:09.421 [WRN] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): insufficient bandwidth to route htlc: 239283580 mSAT is larger than 234681168 mSAT
2023-05-01 01:48:17.098 [WRN] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): insufficient bandwidth to route htlc: 766964171 mSAT is larger than 746420614 mSAT
2023-05-01 01:49:35.142 [WRN] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): insufficient bandwidth to route htlc: 74920943 mSAT is larger than 11413159 mSAT
2023-05-01 01:53:07.050 [WRN] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): insufficient bandwidth to route htlc: 63151008 mSAT is larger than 11413159 mSAT
2023-05-01 01:59:08.683 [WRN] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): insufficient bandwidth to route htlc: 21053936 mSAT is larger than 11413159 mSAT
2023-05-01 01:59:50.614 [WRN] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): insufficient bandwidth to route htlc: 207614045 mSAT is larger than 11413159 mSAT
2023-05-01 02:02:31.406 [WRN] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): insufficient bandwidth to route htlc: 50011450 mSAT is larger than 11413159 mSAT
2023-05-01 02:02:47.147 [WRN] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): insufficient bandwidth to route htlc: 50011450 mSAT is larger than 11413159 mSAT
2023-05-01 02:03:09.108 [WRN] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): insufficient bandwidth to route htlc: 20556850 mSAT is larger than 11413159 mSAT
2023-05-01 02:04:24.172 [WRN] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): insufficient bandwidth to route htlc: 11863529 mSAT is larger than 11413159 mSAT
2023-05-01 02:06:55.592 [WRN] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): insufficient bandwidth to route htlc: 73006424 mSAT is larger than 11413159 mSAT
2023-05-01 02:08:36.525 [ERR] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): failing link: unable to update commitment: commitment transaction dips peer below chan reserve: our balance below chan reserve with error: internal error
2023-05-01 02:08:36.525 [ERR] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): link failed, exiting htlcManager
2023-05-01 02:08:36.525 [INF] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): exited
2023-05-01 02:08:36.525 [INF] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): stopping
2023-05-01 02:08:55.257 [INF] PEER: Peer(023631624e30ef7bcb2887e600da8e59608a093718bc40d35b7a57145a0f3db9af): Loading ChannelPoint(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0), isPending=false
2023-05-01 02:08:55.257 [INF] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): starting
2023-05-01 02:08:55.257 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0)
2023-05-01 02:08:55.257 [INF] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): HTLC manager started, bandwidth=0 mSAT
2023-05-01 02:08:55.257 [INF] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): attempting to re-synchronize
2023-05-01 02:11:54.862 [INF] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): stopping
2023-05-01 02:11:54.862 [WRN] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): error when syncing channel states: link shutting down
2023-05-01 02:11:54.862 [INF] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): exited
2023-05-01 02:12:04.983 [INF] PEER: Peer(023631624e30ef7bcb2887e600da8e59608a093718bc40d35b7a57145a0f3db9af): Loading ChannelPoint(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0), isPending=false
2023-05-01 02:12:04.983 [INF] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): starting
2023-05-01 02:12:04.983 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0)
2023-05-01 02:12:04.983 [INF] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): HTLC manager started, bandwidth=0 mSAT
2023-05-01 02:12:04.984 [INF] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): attempting to re-synchronize
2023-05-01 02:15:03.445 [INF] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): stopping
2023-05-01 02:15:03.445 [WRN] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): error when syncing channel states: link shutting down
[....]
2023-05-01 02:40:20.520 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0)
2023-05-01 02:40:20.520 [INF] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): attempting to re-synchronize
2023-05-01 02:41:14.987 [INF] NTFN: Dispatching confirmed spend notification for outpoint=b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0, script=0 b0b3c81deb80ca8d88999cee148c5b2b5f1abef226d4d434139bdde0679f903e at current height=787714: 3daf8760142f17a9a5156be37a6da706deafe088c61e1b0275d313be666cb67b[0] spending b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0 at height=787714
2023-05-01 02:41:14.996 [INF] CNCT: Unilateral close of ChannelPoint(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0) detected
2023-05-01 02:41:14.997 [INF] CNCT: ChannelArbitrator(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): remote party has closed channel out on-chain
2023-05-01 02:41:28.618 [INF] CNCT: ChannelArbitrator(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): still awaiting contract resolution
2023-05-01 02:41:28.737 [INF] CNCT: *contractcourt.commitSweepResolver(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): Sweeping with witness type: CommitmentToRemoteConfirmed
2023-05-01 02:41:28.737 [INF] CNCT: *contractcourt.commitSweepResolver(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): sweeping commit output
2023-05-01 02:43:14.539 [INF] CNCT: *contractcourt.commitSweepResolver(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): local commitment output fully resolved by sweep tx: 6cf35d29f4170599173ed44cbd0bb5e8280699049447528c51405c906f416192
2023-05-01 02:43:14.578 [INF] CNCT: ChannelArbitrator(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): a contract has been fully resolved!
2023-05-01 02:43:14.578 [INF] CNCT: ChannelArbitrator(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): still awaiting contract resolution
2023-05-01 02:59:20.520 [ERR] PEER: Peer(023631624e30ef7bcb2887e600da8e59608a093718bc40d35b7a57145a0f3db9af): Channel(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0) request enabling failed due to inactive link

Expected behaviour
Proposal above, details better evaluated by folks who're more acustomed to cross-implementation error relays.

let the HTLC fail with usual process
don't send internal error

Actual behaviour
Sending an internal-error causing eclair to force-close
speedupln-fc.log

Here is the log from my side:
eclair.2023-05-01_02.log:2023-05-01 02:07:43,762 INFO  f.a.e.p.r.ChannelRelay PAY h:170265fb62c658c174b0dd7688e2d8d669c0271507991d608897b051af2148e0 p:9d5d0308 - relaying htlc #230485 from channelId=ebe6bd43f9ee41617fcc455c869eaba86a5edc98a0e4bd25ac698b2054bc508e to requestedShortChannelId=768220x2458x0 nextNode=037f66e84e38fc2787d578599dfe1fcb7b71f9de4fb1e453c5ab85c05f5ce8c2e3
eclair.2023-05-01_02.log:2023-05-01 02:07:49,525 INFO  f.a.e.p.r.ChannelRelay PAY h:170265fb62c658c174b0dd7688e2d8d669c0271507991d608897b051af2148e0 p:89da5cb3 - relaying htlc #230486 from channelId=ebe6bd43f9ee41617fcc455c869eaba86a5edc98a0e4bd25ac698b2054bc508e to requestedShortChannelId=768220x2458x0 nextNode=037f66e84e38fc2787d578599dfe1fcb7b71f9de4fb1e453c5ab85c05f5ce8c2e3
eclair.2023-05-01_02.log:2023-05-01 02:08:31,599 INFO  f.a.e.p.r.ChannelRelay PAY h:170265fb62c658c174b0dd7688e2d8d669c0271507991d608897b051af2148e0 p:d1610534 - relaying htlc #230491 from channelId=ebe6bd43f9ee41617fcc455c869eaba86a5edc98a0e4bd25ac698b2054bc508e to requestedShortChannelId=768220x2458x0 nextNode=037f66e84e38fc2787d578599dfe1fcb7b71f9de4fb1e453c5ab85c05f5ce8c2e3
eclair.2023-05-01_02.log:2023-05-01 02:08:36,629 ERROR f.a.e.c.fsm.Channel n:037f66e84e38fc2787d578599dfe1fcb7b71f9de4fb1e453c5ab85c05f5ce8c2e3 c:efc328abfaace666d49abf6d4eaea0589c0552feef9f34a5c81e74252b62e5b7 - peer sent error: ascii='internal error' bin=696e7465726e616c206572726f72
eclair.2023-05-01_02.log:2023-05-01 02:08:36,654 INFO  f.a.e.c.p.TxPublisher n:037f66e84e38fc2787d578599dfe1fcb7b71f9de4fb1e453c5ab85c05f5ce8c2e3 c:efc328abfaace666d49abf6d4eaea0589c0552feef9f34a5c81e74252b62e5b7 - publishing commit-tx txid=3daf8760142f17a9a5156be37a6da706deafe088c61e1b0275d313be666cb67b spending b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0 with id=39906b51-14ec-4cbf-a4e5-4b631ec3a1a1 (0 other attempts)
eclair.2023-05-01_02.log:2023-05-01 02:08:36,654 INFO  f.a.e.c.p.TxPublisher n:037f66e84e38fc2787d578599dfe1fcb7b71f9de4fb1e453c5ab85c05f5ce8c2e3 c:efc328abfaace666d49abf6d4eaea0589c0552feef9f34a5c81e74252b62e5b7 - publishing replaceable local-anchor spending 3daf8760142f17a9a5156be37a6da706deafe088c61e1b0275d313be666cb67b:0 with id=d09acaa5-d561-47dc-9a62-38bf41aec4b1 (0 other attempts)
eclair.2023-05-01_02.log:2023-05-01 02:08:36,661 INFO  f.a.e.db.DbEventHandler n:037f66e84e38fc2787d578599dfe1fcb7b71f9de4fb1e453c5ab85c05f5ce8c2e3 c:efc328abfaace666d49abf6d4eaea0589c0552feef9f34a5c81e74252b62e5b7 - paying mining fee=0 sat for txid=3daf8760142f17a9a5156be37a6da706deafe088c61e1b0275d313be666cb67b desc=commit-tx
eclair.2023-05-01_02.log:2023-05-01 02:08:36,661 INFO  f.a.e.c.p.TxPublisher n:037f66e84e38fc2787d578599dfe1fcb7b71f9de4fb1e453c5ab85c05f5ce8c2e3 c:efc328abfaace666d49abf6d4eaea0589c0552feef9f34a5c81e74252b62e5b7 - publishing local-main-delayed txid=b1a8e6c334074579d7cc5ad4726dc213f45ac49297b033585fa1dab4a60d3dc1 spending 3daf8760142f17a9a5156be37a6da706deafe088c61e1b0275d313be666cb67b:5 with id=a8602e0e-210c-40a2-a91e-d854330f4434 (0 other attempts)
eclair.2023-05-01_02.log:2023-05-01 02:08:36,662 INFO  f.a.e.c.p.TxPublisher n:037f66e84e38fc2787d578599dfe1fcb7b71f9de4fb1e453c5ab85c05f5ce8c2e3 c:efc328abfaace666d49abf6d4eaea0589c0552feef9f34a5c81e74252b62e5b7 - publishing replaceable htlc-timeout spending 3daf8760142f17a9a5156be37a6da706deafe088c61e1b0275d313be666cb67b:4 with id=0cd27a09-d278-4d6b-b266-8ba550bf9f00 (0 other attempts)
eclair.2023-05-01_02.log:2023-05-01 02:08:36,662 INFO  f.a.e.c.p.TxTimeLocksMonitor n:037f66e84e38fc2787d578599dfe1fcb7b71f9de4fb1e453c5ab85c05f5ce8c2e3 c:efc328abfaace666d49abf6d4eaea0589c0552feef9f34a5c81e74252b62e5b7 t:a8602e0e - local-main-delayed has a relative timeout of 360 blocks, watching parentTxId=3daf8760142f17a9a5156be37a6da706deafe088c61e1b0275d313be666cb67b
eclair.2023-05-01_02.log:2023-05-01 02:08:36,688 INFO  f.a.e.c.p.TxTimeLocksMonitor n:037f66e84e38fc2787d578599dfe1fcb7b71f9de4fb1e453c5ab85c05f5ce8c2e3 c:efc328abfaace666d49abf6d4eaea0589c0552feef9f34a5c81e74252b62e5b7 t:0cd27a09 - delaying publication of htlc-timeout until block=788378 (current block=787704)
eclair.2023-05-01_02.log:2023-05-01 02:08:36,704 INFO  f.a.e.c.p.ReplaceableTxFunder n:037f66e84e38fc2787d578599dfe1fcb7b71f9de4fb1e453c5ab85c05f5ce8c2e3 c:efc328abfaace666d49abf6d4eaea0589c0552feef9f34a5c81e74252b62e5b7 t:d09acaa5 - skipping local-anchor: commit feerate is high enough (feerate=2500 sat/kw)

I just see that my node received an "internal error" from my peer and then force closed.
@Crypt-iQ Looks like another case of a link sending out "internal error" causing a force close

@DerEwige could you check in your logs which was the latest UpdateAddHTLC msg which lead to this behaviour, especially the amt you tried to put on the remotes commitment transaction. What essentially happened is, your node tried to add an HTLC to the remotes commitment but at the current feerate your peer would have been forced to go below reserve.
According to the spec your SENDING node should:
if it is not responsible for paying the Bitcoin fee:
SHOULD NOT offer amount_msat if, once the remote node adds that HTLC to its commitment transaction, it cannot pay the fee for the updated local or remote transaction at the current feerate_per_kw while maintaining its channel reserve.

The RECEIVING node followed the spec in a way:
receiving an amount_msat that the sending node cannot afford at the current feerate_per_kw (while maintaining its channel reserve and any to_local_anchor and to_remote_anchor costs):
SHOULD send a warning and close the connection, or send an error and fail the channel.

Definitely worth checking why eclair added the htlc although it would force the peer to go below reserve.

ould you check in your logs which was the latest UpdateAddHTLC msg which lead to this behaviour, especially the amt you tried to put on the remotes commitment transaction. What essentially happened is, your node tried to add an HTLC to the remotes commitment but at the current feerate your peer would have been forced to go below reserve.

Unfortunately, I have most of my eclair running on log level “WARN”. This is due to the excessive amount of logs my rebalancing produces. So I don’t have any additional logs from my side.
I can only see from the logs of my plugin that I tried to send out payment through this channel:
eclair.2023-05-01_02.log:2023-05-01 02:08:36,329 INFO  f.a.c.ChannelDB_ActiveBalancer_Multibalance - 768220x2458x0	Send through route [768220x2458x0, 734274x2495x1, 562892x872x0, 778983x2811x0] 0 0 0.0

The RECEIVING node followed the spec in a way:

receiving an amount_msat that the sending node cannot afford at the current feerate_per_kw (while maintaining its channel reserve and any to_local_anchor and to_remote_anchor costs):
SHOULD send a warning and close the connection, or send an error and fail the channel.

This case applies if the sender falls below the reserve.
But in this case, the receiving node could not afford the fees.

According to the spec your SENDING node should:

if it is not responsible for paying the Bitcoin fee:
SHOULD NOT offer amount_msat if, once the remote node adds that HTLC to its commitment transaction, it cannot pay the fee for the updated local or remote transaction at the current feerate_per_kw while maintaining its channel reserve.

This one applies. My node should not have sent the HTLC.
But there might bee a fee estimate discrepancy at this point?
My node might want 10 sat/vb and think it is good.
And when I try to add the HTLC the remote peer might disagree and want 15 sat/vb which would dip him below the reserve?
Somehow the specs does not say anything about how the receiver should act if it would dip him below the reserve? Did I miss that part?

This case applies if the sender falls below the reserve.
But in this case, the receiving node could not afford the fees.

From my perspective this is the same, when you add an HTLC to the remote commit (sending an Update_AD_HTLC), you as the sender need to make sure that you evaluate the transaction limits beforehand meaning that adding this new output in the current fee-environment should not let your peer fall below reserve. Your peer will just receive the amt and try to add it to the transaction at current feelimits, if that forces the peer to fall below its reserve it should fail this attempt (this is also what happened)

Somehow the specs does not say anything about how the receiver should act if it would dip him below the reserve? Did I miss that part?

Every channel has a synchronized feerate in sat/kweight which only the initiator can change via the update_fee msg. So the feerate was definitely the same, but why eclair and lnd diverged in the final calculation of the transaction is strange, maybe some rounding issue. Thats why I asked you supply the amount so that one could backtrace whether the boundaries were breached or not, but without logs very hard to do so.
But I must say this new check for an HTLC which is added to our commitment is very new (only 16.2) so maybe something which will happen more often now

could we identify a different error message, being more verbose, to avoid other implementations interpreting this as a major issue and going on-chain?

From the spec PoV, the channel basically can't continue at this point. Similarly, if someone else us an invalid pre-image, that's also an invalid action and the channels can't continue.
This might ultimately be a interoperability issue: does the HTLC eclair send actually dip below the reserve? If so, can we repro that exact scenario to dig into the accounting between the implementations? Is it an mSAT rounding thing?

This might ultimately be a interoperability issue: does the HTLC eclair send actually dip below the reserve? If so, can we repro that exact scenario to dig into the accounting between the implementations? Is it an mSAT rounding thing?

This is the important question.
Eclair does check reserve requirements (for sender and receiver) before sending an HTLC: https://github.com/ACINQ/eclair/blob/77b333731f618395f33d69d1fe0aba2c86cdba58/eclair-core/src/main/scala/fr/acinq/eclair/channel/Commitments.scala#L440
@DerEwige your node has all the information we need to debug this. Can you:

Run the channel command for the force-closed channel and share its output: this will give us the state of the channel (including the feerate for the commitment transaction) before the force-close
Search your logs for the outgoing HTLC that triggered the force-close, and share its amount
If there are multiple HTLCs that were added at the same time, make sure to include all of them

We should be able to create a unit test based on that data on both eclair and lnd to see why reserve calculation doesn't match.

@t-bast

Run the channel command for the force-closed channel and share its output: this will give us the state of the channel (including the feerate for the commitment transaction) before the force-close

channel_fc1.log
channel_fc2.log
Channel data for both force closes

Search your logs for the outgoing HTLC that triggered the force-close, and share its amount
If there are multiple HTLCs that were added at the same time, make sure to include all of them

Unfortunately I run this logback config:
	<logger name="fr.acinq.eclair.router" level="WARN"/>
	<logger name="fr.acinq.eclair.Diagnostics" level="WARN"/>
	<logger name="fr.acinq.eclair.channel.Channel" level="WARN"/>
	<logger name="fr.acinq.eclair.channel.fsm.Channel" level="WARN"/>
	<logger name="fr.acinq.eclair.io.Peer" level="WARN"/>
	<logger name="fr.acinq.eclair.payment.send.PaymentLifecycle" level="WARN"/>

So the HTLC is not logged.

@TrezorHannes Could you provide more logs so we could understand what had happened during those 90s?
2023-05-01 02:06:55.592 [WRN] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): insufficient bandwidth to route htlc: 73006424 mSAT is larger than 11413159 mSAT

...

2023-05-01 02:08:36.525 [ERR] HSWC: ChannelLink(b7e5622b25741ec8a5349feffe52059c58a0ae4e6dbf9ad466e6acfaab28c3ef:0): failing link: unable to update commitment: commitment transaction dips peer below chan reserve: our balance below chan reserve with error: internal error

@TrezorHannes yep that's where I got those two lines. Need more info to debug the issue.

@yyforyongyu I'm afraid I shared all I had. But perhaps @t-bast can help, who was able to reproduce the issue in a test-environment.

I've already shared everything in my previous comment. This confirmed that this issue seems to be on the lnd side, I don't have much else I can add...

@t-bast thanks for that throughly analysis! We'll def prio this and look into what's going on.

I have an interesting case possibly related to this issue.
Me: LND 0.16.4-beta
My peer: LND 0.16.4-beta (Umbrel)
Both nodes are online and ok. the connection is established without any problems.
My log:
[ERR] HSWC: ChannelLink(xxx:1): failing link: ChannelPoint(xxx:1): received error from peer: chan_id=xxx, err=internal error with error: remote error
[ERR] HSWC: ChannelLink(xxx:1): link failed, exiting htlcManager

My peer log:
[ERR] HSWC: ChannelLink(xxx:1): failing link: unable to update commitment: commitment transaction dips peer below chan reserve: our balance below chan reserve with error: internal error
[ERR] HSWC: ChannelLink(xxx:1): link failed, exiting htlcManager

All of the liquidity in the channel is on my side, remote balance = 60,218 sats.
I'm ready to provide any necessary information.
Can anyone of us do something to resolve this problem?
Should my peer use max-channel-fee-allocation=1 for temporary resolving it?

I have the same issue between 2 LND nodes (mine and Boltz), for the second time now.
This resulted in an FC before, due to a pending HTLC which can't be resolved, and my node can't re-establish a connection with Boltz, while all Liq is on my side. The logs point to a HTLC from my channel partner dipping my funds below chan reserve.
Here is my output, lncli debuglevel --level HSWC=debug,PEER=debug,
2023-08-21 17:28:12.805 [INF] PEER: Peer(026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2): Loading ChannelPoint(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0), isPending=false
2023-08-21 17:28:12.805 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): starting
2023-08-21 17:28:12.805 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0)
2023-08-21 17:28:12.805 [INF] LNWL: ChannelPoint(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): Reducing local balance (from<PHONE_NUMBER>7 mSAT to 353999 mSAT): remote side does not have enough funds (3454930 mSAT < 4174000 mSAT) to pay for non-dust HTLC in case of unilateral close.
2023-08-21 17:28:12.806 [INF] LNWL: ChannelPoint(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): Reducing local balance (from<PHONE_NUMBER>7 mSAT to 353999 mSAT): remote side does not have enough funds (3454930 mSAT < 4174000 mSAT) to pay for non-dust HTLC in case of unilateral close.
2023-08-21 17:28:12.806 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): HTLC manager started, bandwidth=353999 mSAT
2023-08-21 17:28:12.806 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): attempting to re-synchronize
2023-08-21 17:28:12.806 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): received re-establishment message from remote side
2023-08-21 17:28:12.825 [DBG] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): loaded 0 fwd pks
2023-08-21 17:28:13.943 [ERR] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): failing link: ChannelPoint(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): received error from peer: chan_id=aa6b1c0c1cad2f2cac9cdf4b1211bff9e8c60a0b87e0c9a6eb9457de7d1b5c35, err=internal error with error: remote error
2023-08-21 17:28:13.943 [ERR] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): link failed, exiting htlcManager
2023-08-21 17:28:13.943 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): exited
2023-08-21 17:28:13.944 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): stopping
2023-08-21 17:30:48.096 [INF] PEER: Peer(026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2): Loading ChannelPoint(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0), isPending=false
2023-08-21 17:30:48.096 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): starting
2023-08-21 17:30:48.096 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0)
2023-08-21 17:30:48.096 [INF] LNWL: ChannelPoint(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): Reducing local balance (from<PHONE_NUMBER>7 mSAT to 353999 mSAT): remote side does not have enough funds (3454930 mSAT < 4174000 mSAT) to pay for non-dust HTLC in case of unilateral close.
2023-08-21 17:30:48.096 [INF] LNWL: ChannelPoint(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): Reducing local balance (from<PHONE_NUMBER>7 mSAT to 353999 mSAT): remote side does not have enough funds (3454930 mSAT < 4174000 mSAT) to pay for non-dust HTLC in case of unilateral close.
2023-08-21 17:30:48.096 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): HTLC manager started, bandwidth=353999 mSAT
2023-08-21 17:30:48.096 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): attempting to re-synchronize
2023-08-21 17:30:48.097 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): received re-establishment message from remote side
2023-08-21 17:30:48.100 [DBG] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): loaded 0 fwd pks
2023-08-21 17:30:48.741 [ERR] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): failing link: ChannelPoint(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): received error from peer: chan_id=aa6b1c0c1cad2f2cac9cdf4b1211bff9e8c60a0b87e0c9a6eb9457de7d1b5c35, err=internal error with error: remote error
2023-08-21 17:30:48.741 [ERR] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): link failed, exiting htlcManager
2023-08-21 17:30:48.741 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): exited

My listchannel details
lncli listchannels --peer<PHONE_NUMBER>92521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2
{
    "channels": [
        {
            "active": false,
            "remote_pubkey": "026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2",
            "channel_point": "355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0",
            "chan_id": "883905094207733760",
            "capacity": "15000000",
            "local_balance": "14756557",
            "remote_balance": "150822",
            "commit_fee": "2634",
            "commit_weight": "1460",
            "fee_per_kw": "1793",
            "unsettled_balance": "89327",
            "total_satoshis_sent": "876618",
            "total_satoshis_received": "8200514",
            "num_updates": "26932",
            "pending_htlcs": [
                {
                    "incoming": false,
                    "amount": "67338",
                    "hash_lock": "12ad998e3ea534d68bf0b3c4a3610757a03231049bb92b96fd8d12738449ecc8",
                    "expiration_height": 804685,
                    "htlc_index": "18132",
                    "forwarding_channel": "0",
                    "forwarding_htlc_index": "0"
                },
                {
                    "incoming": true,
                    "amount": "21989",
                    "hash_lock": "b03330854ae00b661c923456af74c2626a3907e35734fb84fb2d2c1e72d255ae",
                    "expiration_height": 804683,

Here is the log excerpt from Boltz, disconnecting my reconnection attempts immediately
lncli debuglevel --level HSWC=debug,PEER=debug,LNWL=debug
24380:2023-08-21 15:30:48.059 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): HTLC manager started, bandwidth=0 mSAT
24381:2023-08-21 15:30:48.059 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): attempting to re-synchronize
24410:2023-08-21 15:30:48.122 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): received re-establishment message from remote side
24945:2023-08-21 15:30:48.305 [DBG] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): loaded 1 fwd pks
24966:2023-08-21 15:30:48.346 [DBG] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): forwarding 1 packets to switch: replay=true
25090:2023-08-21 15:30:48.706 [DBG] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): queueing removal of FAIL closed circuit: (Chan ID=803907:873:0, HTLC ID=18132)->(Chan ID=789708:2796:1, HTLC ID=208504)
25091:2023-08-21 15:30:48.706 [DBG] LNWL: ChannelPoint(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): Funds below chan reserve: ourBalance=149897930 mSAT, ourReserve=150000000 mSAT
25092:2023-08-21 15:30:48.706 [ERR] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): failing link: unable to update commitment: commitment transaction dips peer below chan reserve: our balance below chan reserve with error: internal error
25093:2023-08-21 15:30:48.706 [ERR] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): link failed, exiting htlcManager
25096:2023-08-21 15:30:48.706 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): stopping
25098:2023-08-21 15:30:48.706 [INF] HSWC: ChannelLink(355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0): exited

Boltz' listchannel details with me
{
  "active": false,
  "remote_pubkey": "037f66e84e38fc2787d578599dfe1fcb7b71f9de4fb1e453c5ab85c05f5ce8c2e3",
  "channel_point": "355c1b7dde5794eba6c9e0870b0ac6e8f9bf11124bdf9cac2c2fad1c0c1c6baa:0",
  "chan_id": "883905094207733760",
  "capacity": "15000000",
  "local_balance": "171887",
  "remote_balance": "14610049",
  "commit_fee": "3561",
  "commit_weight": "1976",
  "fee_per_kw": "1793",
  "unsettled_balance": "213843",
  "total_satoshis_sent": "8200514",
  "total_satoshis_received": "876618",
  "num_updates": "27817",
  "pending_htlcs": [
    {
      "incoming": true,
      "amount": "67338",
      "hash_lock": "12ad998e3ea534d68bf0b3c4a3610757a03231049bb92b96fd8d12738449ecc8",
      "expiration_height": 804685,
      "htlc_index": "18132",
      "forwarding_channel": "868293128730968065",
      "forwarding_htlc_index": "208504"
    },
    {
      "incoming": true,
      "amount": "23457",
      "hash_lock": "707349e7fdcc0cd2547883c072be73a46db01a69826cfcd384096875ab7a7664",
      "expiration_height": 804709,
      "htlc_index": "18133",
      "forwarding_channel": "0",
      "forwarding_htlc_index": "0"
    },
    {
      "incoming": true,
      "amount": "29969",
      "hash_lock": "2b36ef68cfc6a810a571f511ef52d15a8c0c4fa2dca6556c50d13faeb3d87105",
      "expiration_height": 804685,
      "htlc_index": "18134",
      "forwarding_channel": "0",
      "forwarding_htlc_index": "0"
    },
    {
      "incoming": true,
      "amount": "63109",
      "hash_lock": "ff64853d3f7a987c8b8a5cf4a8a8e701afd8551f38dd1dd890471faeaf06fe95",
      "expiration_height": 804621,
      "htlc_index": "18135",
      "forwarding_channel": "0",
      "forwarding_htlc_index": "0"
    },
    {
      "incoming": true,
      "amount": "29970",
      "hash_lock": "2b36ef68cfc6a810a571f511ef52d15a8c0c4fa2dca6556c50d13faeb3d87105",
      "expiration_height": 804685,
      "htlc_index": "18136",
      "forwarding_channel": "0",
      "forwarding_htlc_index": "0"
    }
  ],
  "csv_delay": 1802,
  "private": false,
  "initiator": true,
  "chan_status_flags": "ChanStatusDefault",
  "local_chan_reserve_sat": "150000",
  "remote_chan_reserve_sat": "150000",
  "static_remote_key": false,
  "commitment_type": "ANCHORS",
  "lifetime": "9573",
  "uptime": "4846",
  "close_address": "",
  "push_amount_sat": "7500000",
  "thaw_height": 0,
  "local_constraints": {
    "csv_delay": 1802,
    "chan_reserve_sat": "150000",
    "dust_limit_sat": "354",
    "max_pending_amt_msat": "14850000000",
    "min_htlc_msat": "1",
    "max_accepted_htlcs": 483
  },
  "remote_constraints": {
    "csv_delay": 1802,
    "chan_reserve_sat": "150000",
    "dust_limit_sat": "354",
    "max_pending_amt_msat": "14850000000",
    "min_htlc_msat": "1",
    "max_accepted_htlcs": 483
  },
  "alias_scids": [],
  "zero_conf": false,
  "zero_conf_confirmed_scid": "0",
  "peer_alias": "HODLmeTight",
  "peer_scid_alias": "0"
}

Questions

is there anything we can do to rescue the channel? Last time, the expired HTLC caused an FC
do you need anything else to further dig into this specific scenario, before we're losing the channel and can't provide further logs
@Roasbeef and @saubyk why did we move this down the priority list, looks like an LND <-> LND issue, since my new channel with the eclair node (my original issue on top here) seems to be working fine now

Thanks for looking into this 🙇

@t-bast I was verifying your number and figured out that you kinda neglected the commitment fees, the lnd node has to pay:

lnd's next commitment (signed by eclair, previous not revoked yet):
commitment number: 3_195_512
to_local: 34_367_703 msat
to_remote: 2_756_274_986 msat
HTLC IN: id=1_477_859 amount=117_941_049 msat
HTLC IN: id=1_477_861 amount=80_040_806 msat
HTLC OUT: id=120_896 amount=11_375_456 msat

cap is 3 mio, 34_367_703 + 2_756_274_986 + 117_941_049 + 80_040_806 + 11_375_456 = 3_000_000_000 msats
additional commitment fees: (3*173 + 1124) * 2500 sats/kweight = 4107,5 sats (in fees) + 660 sats (2 anchor values) = 4767,5 sats
Taking these costs into account the local balance falls below the reserve
to_local: 34_367_703 msat - 4767500 = 29600203 (which is below the reserve)

Actually I think one cannot guarantee these situations will not happen. The problem is that the Lnd node does yet not know before adding its own HTLC that the remote as adding one (additional 173 weight * 2500 sats/kweight = 432500) which is exactly the difference which dips us below the reserve, same happens for eclair when adding their input, they don't know yet that we are adding an incoming HTLC to them?

As a potential fix I would add a safety net to the peer who opened the channel, lets say we always keep a buffer of 10*IncomingHTLC_Weight (173) * CFee (2500 sat/kweight) = 4325 sats (in this example with a CFee of 2500) additional buffer so that we would not fall below reserve if the peer added 10 incoming HTLCs in one revocation window.

Nice, very good catch @ziggie1984 that explains it and the numbers match.

Actually I think one cannot guarantee these situations will not happen.

Right, in that case we cannot completely prevent this with the current commitment update protocol, since eclair and lnd concurrently add an HTLC, and each have to take into account the remote peer's HTLC (there's no way to "unadd") which forces dipping into the reserve.
If we actually allowed dipping into the reserve here, that could be abused: instead of 1 concurrent HTLC in each direction, one of the peers could send many of them to dip arbitrarily low into the reserve, which creates bad incentives.

As a potential fix I would add a safety net to the peer who opened the channel, lets say we always keep a buffer

I'm surprised you don't already do that as part of https://github.com/lightning/bolts/pull/919? I thought every implementation had put this mitigation in place 2 years ago (otherwise lnd is potentially at risk of the kind of dust attack described in that PR). Eclair has that kind of mitigation, which should have avoided that situation if eclair had been the channel funder.
I guess that issue is another good argument for https://github.com/lightning/bolts/pull/867, which will remove all of those edge cases :sweat_smile:

I'm surprised you don't already do that as part of https://github.com/lightning/bolts/pull/919? I thought every implementation had put this mitigation in place 2 years ago (otherwise lnd is potentially at risk of the kind of dust attack described in that PR). Eclair has that kind of mitigation, which should have avoided that situation if eclair had been the channel funder.

what I am wondering tho, why is eclair signing this breach of the contract, also for eclair it must be known that the remote_commitment dips below reserve, do you actually log this even as a warning ?

I'm surprised you don't already do that as part of https://github.com/lightning/bolts/pull/919?

Hmm we have a dust exposure check in place but this does only count for dust htlcs in the code base.

I guess that issue is another good argument for https://github.com/lightning/bolts/pull/867,

Thanks for the hint, need to look at this proposal. Sounds promising hopefully not decreasing the speed to much :)

what I am wondering tho, why is eclair signing this breach of the contract, also for eclair it must be known that the remote_commitment dips below reserve, do you actually log this even as a warning ?

Eclair would indeed have complained at a later step and sent an error.

Hmm we have a dust exposure check in place but this does only count for dust htlcs in the code base.

Sorry that was the wrong "additional reserve", I meant this one: https://github.com/lightning/bolts/pull/740

could you maybe point me to the code part where eclair defines the additional buffer, just to get a feeling about the size of it :)

Sure, you can find that code here: https://github.com/ACINQ/eclair/blob/3547f87f664c5e956a6d13af530a3d1cb6fc1052/eclair-core/src/main/scala/fr/acinq/eclair/channel/Commitments.scala#L435
This function is called when trying to send an outgoing HTLC: if that HTLC violates some limit (such as this additional buffer), we simply don't send it (and try another channel with the same peer, or if none work, propagate the failure to the corresponding upstream channel).

Sure, you can find that code here: https://github.com/ACINQ/eclair/blob/3547f87f664c5e956a6d13af530a3d1cb6fc1052/eclair-core/src/main/scala/fr/acinq/eclair/channel/Commitments.scala#L435

Basically Eclair (I think also CLN) has a fee buffer of 2x the current commitment fee (the additional HTLC consideration will be consumed by the htlc which will be send out by the current payment request so I do not count it here). Every new Incoming HTLC adds approx. 20% fees, so the buffer of 200% (2x current commitment fee) buys us 10 htlc slots (for concurrent htlcs).This increases when htlcs are on the channel (commitment tx bigger) which acts as a nice HTLC limiter when our balance is very low). I like this approach as well and would implement the additional feebuffer, let me know what you guys think 👻

Indeed LND is missing the fee spike buffer (see https://github.com/lightningnetwork/lnd/issues/7721).  We should implement that.
The fee spike buffer helps, but IIUC it doesn't completely prevent concurrent HTLC problems.  But even if an HTLC race occurs, why do we need to error and force close?  If the next commitment would violate the channel reserve, can't we just disconnect to drop the proposed updates?  Then on reconnect we can do exponential backoff to prevent the race from reoccurring indefinitely.

@morehouse hehe I came here to link that very issue. AFAICT, it's an edge case where we send an HTLC (as the initiator), can pay for that HTLC, but then the remote party sends another HTLC concurrently (they haven't seen our yet), once that HTLC is ACK'd and a sig ready to go out, we fail as adding that would dip us below the reserve.

@t-bast we had an initial work around which helped to spur that spec PR here. In practice, it looks like it wasn't enough to fully address this edge case.

@morehouse hehe I came here to link that very issue. AFAICT, it's an edge case where we send an HTLC (as the initiator), can pay for that HTLC, but then the remote party sends another HTLC concurrently (they haven't seen our yet), once that HTLC is ACK'd and a sig ready to go out, we fail as adding that would dip us below the reserve.

Right.  So we could just disconnect to get out of that bad state (should also send a warning).  No need to send an error and cause a force close.

Edit: Never mind, I think I see the problem now. The 2 commitments contain different HTLCs that are OK on their own, but combined they cause the channel reserve to be violated. And we can't go back to the previous state because it's already been revoked.

I think this race could still be detected and avoided without Rusty's simplified commitment update.
When we receive a commitment_signed to update our commitment from X to X+1, we do the usual checks.  Then we also create a "speculative commitment" X+2 with any HTLCs we've previously offered that haven't been ACKed yet.  If channel reserve or other checks fail on the speculative commitment, we disconnect to reset state to X and backoff on reconnect.  Otherwise, (the speculative commitment checks out), we can safely advance our commitment to X+1 and revoke X.

Then we also create a "speculative commitment" X+2 with any HTLCs we've previously offered that haven't been ACKed yet.

I think the issue is we don't know what to include in X+2 because the incoming HTLC has not yet arrived or processed. We can easily handle parallel adding update_add_htlc, but not a combination of update_add_htlc followed closely with commitment_signed.

The case we've covered in lnd,

Alice sends update_add_htlc
Bob receives Alice's HTLC
Bob sends update_add_htlc, which would fail because of the validateAddHtlc check.
Alice would fail too due to the same check.

The case we should cover in lnd,

Alice sends update_add_htlc
Bob sends update_add_htlc
Alice receives Bob's HTLC, and should fail here
Bob receives Alice's HTLC, and should fail here

We should add a method validateReceiveHtlc similar to validateAddHtlc and use it in ReceiveHTLC.
The case cannot be fixed atm,

Alice sends update_add_htlc
Alice sends commitment_signed <- commits this HTLC on Alice's remote chain
Bob sends update_add_htlc
Bob sends commitment_signed <- commits this HTLC to Bob's remote chain
Alice receives Bob's HTLC and commit sig, but fails it due to dipping below channel reserve
Bob receives Alice's HTLC and commit sig, but fails it due to dipping below channel reserve

There are a few mitigations for this case,

implement #7721 using @ziggie's suggestion
change IdealCommitFeeRate used here to return min relay fee for anchor channels and let the fee bumping be handled by CPFP.

When we receive a commitment_signed to update our commitment from X to X+1, we do the usual checks. Then we also create a "speculative commitment" X+2 with any HTLCs we've previously offered that haven't been ACKed yet. If channel reserve or other checks fail on the speculative commitment, we disconnect to reset state to X and backoff on reconnect. Otherwise, (the speculative commitment checks out), we can safely advance our commitment to X+1 and revoke X.

Hmm I am not sure how this is possible. When we send out HTLC to the peer which they haven't acked, does not mean we can forget about it (signature is out and we need a revocation for this), they will eventually revoke_ack their state including the HTLCs we offered them, forcing us to include them as well in our Commitment when verifying their signature (because they anticipate that we have it on our local commitment as well when they acked our offered HTLCs?)

The case we should cover in lnd,
Alice sends update_add_htlc
Bob sends update_add_htlc
Alice receives Bob's HTLC, and should fail here
Bob receives Alice's HTLC, and should fail here

With the current protocol where each node immediately sends out a CommitSig (50ms) after sending out the UpdateAddHTLC we would also make the channel unusable because we need a revocation from the peer. So I think the feebuffer is our best shot for this problem? We just keep slots for the peer to add some htlcs, in case it happens concurrently?

With the current protocol where each node immediately sends out a CommitSig (50ms) after sending out the UpdateAddHTLC we would also make the channel unusable because we need a revocation from the peer. So I think the feebuffer is our best shot for this problem?

Yes, but then this would be the third case.

Just to add a little bit more color:

We send an HTLC to the peer, this calls AddHTLC. This will call validateCommitmentSanity for the remote commitment with their updates on our commitment transaction (lc.localCommitChain.tail().theirMessageIndex) and all of our updates (lc.localUpdateLog.logIndex). It'll also call validateCommitmentSanity for the local commit with all of their updates (lc.remoteUpdateLog.logIndex) and all of our updates (lc.localUpdateLog.logIndex)
We sign the commit and send it over
We receive an HTLC from the peer, calling ReceiveHTLC.  This doesn't trigger the reserve check because both HTLC's aren't included as they haven't yet revoked the tail commit
They send a commit sig
We send a revoke
They send a revoke and a sig
When we receive the revoke, we'll attempt to sign a new commitment. The issue though is that we'll now include both HTLC's and fail the reserve check

There are probably a bunch of things we could do here -- we could be more strict in ReceiveHTLC and include more HTLC's in the call to validateCommitmentSanity (e.g. check all of our updates, all of their updates), but that could be too strict. I think the best thing here is to use the fee-buffer and not try to worry about involved update accounting. It won't work for every case -- maybe there's a large fee spike and a concurrent update or maybe somebody sends a lot of HTLC's in a single update round. But I think it should be good enough. I don't know that it makes sense to change the error message because with LND-LND people might have to force close anyways since the channel gets stuck in a reestablish loop.


Use files in package.json

I noticed that idea files got included in the package.

This avoids the problem :smile:

Thanks


How to include images that work in the notebook and the export

I saw in Pluto that it is best to load images with the load function from Images.jl.
This works fine in the notebook
https://github.com/JuliaManifolds/Manopt.jl/blob/update-pluto-export/pluto/bezier.jl
when I run it locally – note especially lines like
https://github.com/JuliaManifolds/Manopt.jl/blob/02895b47e24be38a0ebd47001f129620380ee1b8/pluto/Bezier.jl#L124
The image is included in the notebook when running; when one activates rendering (not on CI) then Asymptote is used to render it; here the images are checked in and are found (without rendering) at least locally.
In PlutoStaticHTML, I get the error
https://github.com/JuliaManifolds/Manopt.jl/runs/4398948388?check_suite_focus=true#step:4:806
would it be possible to display loaded images?

The error from the log is
ERROR: LoadError: Dict{Symbol, Any}(:msg => "syntax: extra token after end of expression\n\nBoundaries: [45, 58]", :stacktrace => Dict{Symbol, Any}[Dict(:call => "top-level scope", :inlined => false, :file => "none", :line => 1, :path => "none")])

This is a message passed inside Pluto meaning that the Pluto notebook execution failed. I'm closing this issue for now since it doesn't seem to be related to PlutoStaticHTML. Please reopen if you disagree.

Well, let‘s say: In the last commit the CI formatter recognises something and wanted to format the code which introduced a second line/command which made Pluto error on that one.
So the error you got was just the cell being destroyed by the formatter, the most recent one should error again on the image task. Sorry for that – I was already 2 hours into trying to get it work.

Could you give a bit more information on what is the image problem because I don’t understand it yet
Most image mine types (like plots) should be converted to an base64 encoded image and work thanks to that. This way it hacks around having to deal with file paths

Sure, sorry, I should have given more detail here.
I am using Asymptote (included in usual tex distributions) to render the images, this command
https://github.com/JuliaManifolds/Manopt.jl/blob/02895b47e24be38a0ebd47001f129620380ee1b8/pluto/Bezier.jl#L96
generates the asy file, and this
https://github.com/JuliaManifolds/Manopt.jl/blob/02895b47e24be38a0ebd47001f129620380ee1b8/pluto/Bezier.jl#L117
renders. On the CI I do not have a LaTeX distribution (or I am to lazy to get one), so I have a switch that does not run the rendering.
Anyways in both cases a png is created, which I load here
https://github.com/JuliaManifolds/Manopt.jl/blob/02895b47e24be38a0ebd47001f129620380ee1b8/pluto/Bezier.jl#L124
into base64, since that was what I saw worked in notebooks.
So the problem is indeed a problem with file paths, yes.

On the CI I do not have a LaTeX distribution (or I am to lazy to get one), so I have a switch that does not run the rendering.

This sounds very complex to manage because it means writing code locally and debugging in CI. How about installing LaTeX in CI? For example, I've been using something like the code below for a while:
    - name: Install LaTeX
      run: |
        sudo apt-get -qq update
        # Don't make this a fancy multiline thing which is hard to debug on the runner.
        sudo apt-get install -y pdf2svg
        sudo apt-get install -y texlive-latex-base
        sudo apt-get install -y texlive-binaries
        # Includes Tikz.
        sudo apt-get install -y texlive-pictures
        sudo apt-get install -y texlive-latex-extra
        # In response to error luatex85.sty not found.
        sudo apt-get install -y texlive-luatex

But even using LaTeX in the CI the asymptote would write a PNG to the file. Would that help?

But even using LaTeX in the CI the asymptote would write a PNG to the file. Would that help?

Ah. So it also doesn't work locally? It is possible to write an image to a file and render that but it's usually tricky because there is a difference between the file path en the online URL path. Also, it requires ensuring that the file ends up in a public directory on the server.
Would it be possible to solve it via
# hideall
Images.load(path_to_png)

Yes, running documenter also does not work locally, since it can not load the saved png.
I do not understand your solution there, the goal is to have the image in the final html page.

I do not understand your solution there, the goal is to have the image in the final html page.

If the image is shown in Pluto, then it will be shown in the final HTML page too.

It is shown in pluto (but loaded from same file not from URL), but not in local or CI HTML conversion. So the problem is loading the file and Asymptote only writes to disc (since it is a command line tool).

Asymptote only writes to disc

And load can read a file from disk. The idea is that you know the path to the image inside Pluto and use load to show it. Then PlutoStaticHTML will handle the rest.

I know the (relative) path from the original pluto notebook to the file in the repo ( it works in Pluto), just not from your temp folder, when you run the notebook.

just not from your temp folder, when you run the notebook.

My temp folder?

I have to check the error message again (in one go the logs) it stated that the HTML war running in some tmp/subfolder? I will check the error messages again, or it is that it is run from the make file (and that is in a different folder than the Pluto notebook).

Here this
https://github.com/JuliaManifolds/Manopt.jl/runs/4399900505?check_suite_focus=true#step:4:805
is nearly the right path up to the tmp/ upfront the bezier/... in the error message. I locally do not have tmp and I do not know how that ends in up there – unless the Rendering is run in a tmp folder?

Try to get an absolute path. Usually more reliable anyway

How would I do that? currently the image is in the bezier subfolder of the pluto folder of the repo. But an absolut path would be /Users/ronny/Repositories/Julia/Manopt.jl/pluto/bezier/ and that would definitively not exist on CI ;)

to be precise my current approach uses @__DIR__ and hence is basically an absolute path. Just the wrong one when running PlutoStaticHTML somehow

Manopt.jl does exist in CI because it clones the repo. Would
joinpath(pkgdir(Manopt), "pluto", "bezier" work?
On Fri, 3 Dec 2021 at 10:24, Ronny Bergmann @.***>
wrote:

How would I do that? currently the image is in the bezier subfolder of
the pluto folder of the repo. But an absolut path would be
/Users/ronny/Repositories/Julia/Manopt.jl/pluto/bezier/ and that would
definitively not exist on CI ;)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/rikhuijzer/PlutoStaticHTML.jl/issues/22#issuecomment-985358784,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AE6DZMSXU2KLFKMM3RFNF5DUPCEDVANCNFSM5JHY52TQ
.

I will check, first for completeness,
on my machine: I set the path using image_prefix = String(@__DIR__) * "/bezier" (i.e. in a subfolder of the place where the notebook itself is)

(works) Running Pluto itself the path gets set to ~/Repositories/Julia/Manopt.jl/pluto/bezier
(works) Running the notebook using include sets the path to ~/Repositories/Julia/Manopt.jl/pluto/bezier
(does not work) Running docs/make.jl and hence PlutoStaticHTML locally sets the path to /var/folders/_v/wg192lpd3mb1lp55zz7drpcw0000gn/T/bezier
(does not work) Running on make.jl on CI sets it to /tmp/bezier (that's what I meant above with temp dir)

Using your proposed approach yields

(does not work) As a Pluto Notebook run in Pluto ~/.julia/packages/Manopt/rcbsQ/pluto/bezier (since it uses the Package Manager and not the local version) and hence fails (since I only have the images on the new branch)
(works) Running the notebook using include sets the path to ~/Repositories/Julia/Manopt.jl/pluto/bezier (since that is in global environment)
(does not work) Running docs/make.jl yields the folder ``~/.julia/packages/Manopt/rcbsQ/pluto/bezier` and hence also does not work (on Manopt.jl#master see two points above)
(does not work) Running docs/make.jlin CI behaves as the first and third case here and uses /home/runner/.julia/packages/Manopt/rcbsQ/pluto/bezier so it might work on current release commits.

Another variant that works with the first two (in Pluto and REPL) but not in make.jl (or CI) is
    localpath = join(splitpath(@__FILE__)[1:(end - 1)], "/") # remove file to get files folder
    image_prefix = localpath * "/bezier"

This was my last idea on how to maybe access these.

image_prefix = String(@__DIR__) * "/bezier"

In general, use joinpath(@__DIR__, "bezier"). Manual string concatenations are not platform independent and require manual ensuring that the forward slash is appropriately added in between.
So, I see that the path is indeed absolute via image_prefix so that isn't the problem.
 Otherwise, it will locally work due to the images being available from the last time, but not online because the images need to be rendered still.

If I set render_asy to true in the CI I would also have to install TeX there and stuff, but it would also not help for the last two images - since they are results of a large experiment that would run longer than GitHub allows me to run the CI.

You can also use PlutoUI.LocalResource("../images/hello.png"), it does the base64 trick automatically. Images.jl is a very large package, so it might be nicer to avoid.

I will try, but the main problem seems to be that the rendering here takes place in a different folder such that relative paths won‘t work (and absolute ones neither).

Sadly the idea of PlutoUI does not even work locally, I get
      From worker 3:    [ Info: //var/folders/_v/wg192lpd3mb1lp55zz7drpcw0000gn/T/bezier
      From worker 3:    ┌ Warning: `LocalResource` **will not work** when you share the script/notebook with someone else, _unless they have those resources at exactly the same location on their file system_.
      From worker 3:    │
      From worker 3:    │ ## Recommended alternatives (images)
      From worker 3:    │ 1. Go to [imgur.com](https://imgur.com) and drag&drop the image to the page. Right click on the image, and select "Copy image location". You can now use the image like so: `PlutoUI.Resource("https://i.imgur.com/SAzsMMA.jpg")`.
      From worker 3:    │ 2. If your notebook is part of a git repository, place the image in the repository and use a relative path: `PlutoUI.LocalResource("../images/cat.jpg")`.
      From worker 3:    └ @ PlutoUI ~/.julia/packages/PlutoUI/iqFqj/src/Resource.jl:80
      From worker 3:    ┌ Warning: `LocalResource` **will not work** when you share the script/notebook with someone else, _unless they have those resources at exactly the same location on their file system_.

and then  an error
ERROR: LoadError: Dict{Symbol, Any}(:msg => "SystemError: opening file \"//var/folders/_v/wg192lpd3mb1lp55zz7drpcw0000gn/T/bezier/Casteljau-illustr.png\": No such file or directory", :stacktrace => Dict{Symbol, Any}[Dict(:call => "var\"#systemerror#69\"(::Nothing, ::typeof(systemerror), ::String, ::Int32)", :inlined => false, :file => "error.jl", :line => 174, :path => "./error.jl"), Dict(:call => "#systemerror#68", :inlined => true, :file => "error.jl", :line => 173, :path => "./error.jl"), Dict(:call => "systemerror", :inlined => true, :file => "error.jl", :line => 173, :path => "./error.jl"), Dict(:call => "var\"#open#636\"(::Bool, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::typeof(open), ::String)", :inlined => false, :file => "iostream.jl", :line => 293, :path => "./iostream.jl"), Dict(:call => "open", :inlined => true, :file => "iostream.jl", :line => 282, :path => "./iostream.jl"), Dict(:call => "var\"#open#355\"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Base.var\"#362#363\"{Tuple{}}, ::String)", :inlined => false, :file => "io.jl", :line => 328, :path => "./io.jl"), Dict(:call => "open", :inlined => true, :file => "io.jl", :line => 328, :path => "./io.jl"), Dict(:call => "read", :inlined => true, :file => "io.jl", :line => 434, :path => "./io.jl"), Dict(:call => "LocalResource(::String)", :inlined => false, :file => "Resource.jl", :line => 86, :path => "/Users/ronnber/.julia/packages/PlutoUI/iqFqj/src/Resource.jl"), Dict(:call => "top-level scope", :inlined => true, :file => "jl_p5g7GI#==#38b8f8af-ba81-4d3b-83b7-56b0d41ea708", :line => 1, :path => "/var/folders/_v/wg192lpd3mb1lp55zz7drpcw0000gn/T/jl_p5g7GI#==#38b8f8af-ba81-4d3b-83b7-56b0d41ea708")])

so compared to load (which at least works locally)  LocalResource does not even work locally, but yields warnings that seem to indicate that the error from above will persist on CI.


feat: add update_in_normal

Updates the preview when text changes while in normal mode.
probably should have a better name, but I couldn't think of a good one :sweat_smile:

Hi, can you provide me with some reproducible scenario in which setting that autocmd would be necessary? Things like paste and substitute seem to be covered by currently used autocmds. Unless of course you set update_in_insert = false but that kind of implies that you want to update on buffer write only. Don't want to needlessly set too much autocmds or introduce too much options. I probably am missing something so clarification is required :)

Hi, can you provide me with some reproducible scenario in which setting that autocmd would be necessary?

just from some basic testing

deleting text with d
surrounding text with nvim-surround
undo and redo

Okay, I removed this plugin and have the same problem now, sorry for the fuss.
Do you think that maybe we could simply set this autocmd inside update_in_insert check, without introducing new option? Cause as i said earlier update_in_insert = false basically means 'update on write only'. I don't think anyone expects updates on changes in normal mode when they set update_in_insert = false.

Do you think that maybe we could simply set this autocmd inside update_in_insert check

that's fine.. but it's a little odd naming wise. I'd expect that update_in_insert to select if it will update during insert, not normal.
I initially was going to replace update_in_insert with update_on_change, but that would be a breaking change...
If you do want to do that I feel like it's common practice on a breaking change to keep update_in_insert as a undocumented alias with a warning on startup that the user should change it to update_on_change.

If you do want to do that I feel like it's common practice on a breaking change to keep update_in_insert as a undocumented alias with a warning on startup that the user should change it to update_on_change.

I'm okay with that.

There you go. If you give setup update_in_insert it'll emit a warning while still working as intended. This also ends up making the code a bit cleaner since CursorMoved and TextChanged now get grouped with their insert counter parts.

Thanks!


fix OR operator

Since nil is falsey, I used ||= to make the OR operator more meaningful. If @result_ class is already initialized it will keep the same value, else if it’s nil it will be initialized with the value of CivilService::Result.

Hi @lhmzhou, thanks for the PR!  I'm a little confused about the intended effect of doing this.  As far as I can tell, it doesn't actually alter the externally-visible behavior of either of these attributes.  From a user POV, I think this is how it would play out on, for example, @result_class:

Action
Current behavior (without patch)
New behavior (with patch)

Don't set MyService.result_class =, but read from MyService.result_class
Returns CivilService::Result.  Internally, @result_class remains nil.
Returns CivilService::Result.  Internally, @result_class changes to CivilService::Result as a result of having read the attribute.

Set MyService.result_class = CustomResult.  Read from MyService.result_class
Returns CustomResult
Returns CustomResult

So, the only change here is that @result_class and @logger can sometimes be set automatically to a default value.  Externally, there is no change in behavior (because externally, you are always calling the reader method anyway).  Internally to the class, the value is accessible but you still can't rely on it because it will only be set if the reader method has been called already.
It could be there's something I'm missing about this.  Could you write more about how you're intending to make use of this change?

Hi @nbudin. Thanks for the comments.
From this line of code, @result_class || CivilService::Result, I wondered what if @result_class was nil, then it would would return nil NOT the initialized value.
The table really clears the fog of confusion --- thanks.

Hi @lhmzhou, thanks for the PR! I'm a little confused about the intended effect of doing this. As far as I can tell, it doesn't actually alter the externally-visible behavior of either of these attributes. From a user POV, I think this is how it would play out on, for example, @result_class:
Action	Current behavior (without patch)	New behavior (with patch)
Don't set MyService.result_class =, but read from MyService.result_class	Returns CivilService::Result. Internally, @result_class remains nil.	Returns CivilService::Result. Internally, @result_class changes to CivilService::Result as a result of having read the attribute.
Set MyService.result_class = CustomResult. Read from MyService.result_class	Returns CustomResult	Returns CustomResult
So, the only change here is that @result_class and @logger can sometimes be set automatically to a default value. Externally, there is no change in behavior (because externally, you are always calling the reader method anyway). Internally to the class, the value is accessible but you still can't rely on it because it will only be set if the reader method has been called already.
It could be there's something I'm missing about this. Could you write more about how you're intending to make use of this change?


FR: Crashlytics - Add tags to non fatal events

Feature proposal

Firebase Component: Crashlytics (Auth, Core, Database, Firestore, Messaging, Storage, etc)

Describe your use case and/or feature request here.
It would be really nice if we could add "tags" to events on the Firebase Crashlytics portal that are logged through the Crashlytics.crashlytics().record(error: error) method. Sentry currently offers this and I think it would be a great addition that would make filtering non fatals a lot easier

Hi @ethan021021 ,
Crashlytics now supports filtering on custom keys - does this fit the use case you had in mind for tags?

Yes! This is perfect. Thank you!


Fix Documentation: Correct File Path and Script Command for Serving Docs

Updated the documentation to correct the file path and the command for running the serve_docs.sh script.

@Swader @BrianSeong99 can you please this small PR.


Upgrade headers break HTTP to HTTPS redirects

use isahc::prelude::*;
use isahc::config::RedirectPolicy;

fn main() {
    //let req = Request::get("https://yahoo.com")
    let req = Request::get("http://yahoo.com")
        .redirect_policy(RedirectPolicy::Follow)
        .body(())
        .unwrap();

    let resp = req
        .send()
        .unwrap();

    eprintln!("{:?}", resp.effective_uri());
    eprintln!("{:?}", resp);
}

In a case I can't share, a 302 caused NoResponse errors which was confusing.
Issue does not exist in the curl crate (Easy and Easy2 tested).

This change is now available in the 0.9 release!


Shade Check Fatal Errors

Currently shade checking will not generate any blocking errors. In some cases, a policy with a shaded term could generate an ACL that is very different from the intended output.
This issue represents a call for discussion around cases where a shading error might be fatal, meaning that aclgen should not generate output for that policy.

It could be better to have validation/check as separate step for identifying specific issues and react on it if feasible or ignore them for another task/review. Like validating a new build configuration prior to restarting the service.
What I expect aclgen to do is to generate the ACLs as described. If there are shaded terms or the mixed ports scenario that is currently spit out as warning during an aclgen run it can be confusing.
To give an example (these commands do not exist):

acl check-> running all checks and present in a human readable or machine parsable format

based on the output action can be taken to address individual/all issues or ignore them for now

acl build -> generate the filter output as defined w/o warnings

would bail if definitions/references are not matching -> build error

I actually like this idea, wdyt @jtwb

„Check“ is already burned by aclcheck but maybe a different keyword to tell the same.

On our side we leveraged CI/CD pipelines with multiple stages:

aclgen for rendering only (no shade check)
testing stage with --shade_check flag set
It works well. The only drawback is that we render the filters twice...


Add remote tag for tutorial

Types of changes
Thanks for sending a pull request! Please fill in the following content to let us know better about this change.
Please put an x in the box that applies

[ ] Bugfix
[X] New feature
[ ] Refactoring
[ ] Breaking change (any change that would cause existing functionality to not work as expected)
[ ] Documentation Update
[ ] Other (please describe)

Description
Add remote tag to tutorial event.
Steps to Test This Pull Request
Steps to reproduce the behavior:

Go to admin -> events -> proposedtutorialevent
Edit any item
Cannot select is_remote field

Expected behavior
Can select is_remote toggle to proposedtutorialevent

Codecov Report

Merging #941 into master will increase coverage by 0.04%.
The diff coverage is 80.00%.

@@            Coverage Diff             @@
##           master     #941      +/-   ##
==========================================
+ Coverage   75.01%   75.05%   +0.04%
==========================================
  Files          80       80
  Lines        3082     3083       +1
==========================================
+ Hits         2312     2314       +2
+ Misses        770      769       -1

Impacted Files
Coverage Δ

src/events/templatetags/events.py
37.62% <0.00%> (ø)

src/events/admin.py
95.23% <100.00%> (ø)

src/events/models.py
87.85% <100.00%> (+0.08%)
:arrow_up:

.../proposals/management/commands/recent_proposals.py
98.76% <0.00%> (+1.23%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 016db2f...c7eb08c. Read the comment docs.


Bootstrap map missing

Hi,
i get an error on site and in admin panel, tinymce sometimes is freezing.

Could you post the main.css (or main.scss) file from your theme? Often, this is a result of not correctly using the asset pipeline in a production environment.
One potential workaround, and the one that I use, is to use the bootstrap-sass gem. In your gemfile:
gem 'bootstrap-sass', '~> 3.3.7'

Then, in main.scss
@import "font-awesome";
@import "bootstrap-sprockets";
@import "bootstrap";
@import "theme";

This strategy could also be used for Bootstrap 4, if you have the need for it.

Hi @WMPayne
This is the full code of the theme:
https://github.com/owen2345/camaleon-cms/tree/master/app/apps/themes/camaleon_first

Thanks for that. Totally forgot to check that he was using one of the included themes... I assumed he was using a custom one and didn't check the URL. My bad.


there is always an indention when using continued option

there is always an indention when using continued option

+1


plot ValueError replaced with PloomberValueError

Changes

Catching ValueError and raising  PloomberValueError for plot functions.
Existing plot tests changed from ValueError to PloomberValueError
New tests added

Issue: #153

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

242 of 254 (95.28%)  changed or added relevant lines in 16 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.6%) to 94.711%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/sklearn_evaluation/plot/classification.py
26
27
96.3%

src/sklearn_evaluation/plot/cumulative_gain_lift_curve.py
3
4
75.0%

src/sklearn_evaluation/plot/ks_statistics.py
2
3
66.67%

src/sklearn_evaluation/plot/calibration.py
13
17
76.47%

src/sklearn_evaluation/plot/metrics.py
6
11
54.55%

Files with Coverage Reduction
New Missed Lines
%

src/sklearn_evaluation/telemetry.py
2
96.92%

Totals

Change from base Build<PHONE_NUMBER>:
0.6%

Covered Lines:
2489

Relevant Lines:
2628

💛  - Coveralls

I think some of the tests are broken

Yes, in python 3.7 this line:
plot.silhouette_analysis(X, clf, range_n_clusters=[-1])

raises AssertionError instead of ValueError in one of the tests.
I removed it.

Also make sure you're linting with Black.

a few things:
when raising an error in an except statement, it should be as follows:
try:
    # stuff
except ValueError as e:
    raise AnotherError("stuff") from e

this is called a chained exception. without the from e, Python will print some verbose non-informative error like "an error happened while  handling error X"
The problem with these approaches (both with and without the from e) is that the traceback becomes too verbose. One hacky solution is this:
try:
    # stuff
except ValueError as e:
    e.args = (e.args + "custom message", )
    raise

I never thought about catching generic exceptions (I just considered replacing our explicit raise ValueError with raise PloomberValueError, but now that I think about it, I think it's worth it, to see if it helps us get more user feedback.
But this try catch approach means we have to wrap all our code into it, and customize the message everywhere. Let me add this to ploomber-core so we grab it from there.

Ok, I pushed ploomber-core 0.1.1, ensure you pin it in setup.py ploomber-core>=0.1.1,<0.1.* (i'm actually not sure if this is the right format so, please check) - it might take a few mins for this to reach PyPI
docs on the decorator added: https://ploomber-core.readthedocs.io/en/latest/exceptions.html#catching-generic-exceptions

And I just realized we don't need PloomberValueError if the function is decorated, since the decorator will add the slack link a second time

Yes, I agree. This wrapper does the job and provides a more robust solution. I'll keep the tests, remove the PloomberValueError and use this decorator.

Shouldn't this be enough?

we're using semver for all our projects. at some point, we might break the API in ploomber-core (although we hope not too soon or too often), meaning upgrading to 0.2 might start breaking things. That's why I suggested the second part. so once release 0.2, new installation will not use it (and will use the latest 0.1.* version), this will give us change to update each library and migrate to 0.2
Now that I think about, I think what we want is: ploomber-core>=0.1.1,<0.2

there is a file conflict, please address


Explode button label for translation

Since that user can change language of the calendar, I think adding “cancel-label” and “confirm-label” for the button to let user translate would be great

@hktuto Thanks for the issue. Someone thought about this before and it has been added in as an enhancement. You can sure now change the labels but only applies to app-datepicker-dialog if you're aware of that.
You can change the labels to any text you want via confirmLabel and dismissLabel, like so:
<!-- Changing labels of the datepicker dialog to 'zh-CN' -->
<app-datepicker-dialog confirm-label="确定" dismiss-label="取消"></app-datepicker-dialog>

Hope this helps.

@motss great !! Thanks


Make sys.path hack in _get_credentials.py script slightly less horrible

Rather than adding the path to the auto_nag/bugzilla package to
sys.path, use the actual project root path, and then import stuff from
auto_nag.bugzilla.

Coverage remained the same at 48.333% when pulling bf62d670cfa87a707e205c6559b13b417f358379 on jcristau:sys-path-hack into a5f1ca8ef5c270095d9727b2a832c0b43ea87107 on mozilla:master.


Pre Trained Model

Does anyone have a pre trained model they could share? Specifically one trained on music (piano) data?

This is a model trained with 72000+ steps with original configuration using VCTK corpus. Still far from fully trained, but could save sometime.
only changed SILENCE_TRESHOLD = 0.1 other than original 0.3
2017-12-04T13-48-11.zip
copy the model to the path "logdir/train/" and run it by type
python3 train.py --data_dir=corpus --gc_channels=32 --restore_from logdir/train/2017-12-04T13-48-11

@DiyuanLu what was the loss you were getting for this model? Just curious - I'm around 40k steps and its between 1.7 and 2.5. I think I need to adjust some parameters.

@DiyuanLu How is your result of the final trained model? Does SILENCE_TRESHOLD = 0.1 work better than the original 0.3?


Fix API review for Python due to missing code file

Verify creating API review for Python. It's currently failing due to missing code file issue which is caused by an incorrect case in path name.

Pipeline run against azure-core to ensure api review is generated.
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=3659926&view=logs&j=b70e5e73-bbb6-5567-0939-8415943fadb9&t=8dd92614-7454-569d-3061-0c20dc3c788d

/check-enforcer override


Added the Record.#recall method to fetch previous state of key/value …

…pair N revisions ago.
@jtnelson my man - This is still an imperfect diff because, I'm getting back into this project. If you could help me understand how Revisions, and Compact Revisions work together, and how they fit in concourse's design (because I think those are new implementations) that would help me close this [issue] (https://cinchapi.atlassian.net/browse/CON-231) out a lot faster. You can hit me up on google hangouts or email. Thanks!

Yo @dearyhud
For this, you don't need to make any changes to the storage engine or the Record class directly. Here is how you should approach this change.

Add method stubs to thrift-api/concourse.thrift
recallKeyRecord
recallRecord

Run utils/compile-thrift-all.sh
This will generate the RPC code for the new methods in ConcourseServer and add method stubs in all the drivers.

Open ConcourseServer.java and implement the new methods
Eclipse will complain about you needing to implement the methods and will add the stubs for you. Then, you'll need to add the logic to implement the recall routine.

Once you get to step 3, lets chat about the specific implementation. Recall would need to be implemented as an AtomicOperation. There are a lot of examples that I can show you for how to do that

yo @jtnelson - perfect. I went through said steps with thrift, and the stubs are in the drivers. Let me know when's a good time for you to chat.


feat: add #rwhnf command for reducible whnf

Better naming suggestions are welcome.

The MetaM function is called whnfR, so I'd suggest #whnfR or #whnf_r as a name.

bors merge


Error invalid_keys when sending message with secondary attachments API

Reproducible in:
mvn dependency:tree | grep com.slack.api
gradle dependencies | grep com.slack.api
java -version
sw_vers && uname -v # or `ver`

The Slack SDK version
+--- com.slack.api:bolt-socket-mode:1.14.0
|    +--- com.slack.api:slack-api-model:1.14.0
|    +--- com.slack.api:slack-api-client:1.14.0
|    |    +--- com.slack.api:slack-api-model:1.14.0 (*)
|    +--- com.slack.api:slack-app-backend:1.14.0
|    |    +--- com.slack.api:slack-api-model:1.14.0 (*)
|    |    +--- com.slack.api:slack-api-client:1.14.0 (*)
|    \--- com.slack.api:bolt:1.14.0
|         +--- com.slack.api:slack-api-model:1.14.0 (*)
|         +--- com.slack.api:slack-api-client:1.14.0 (*)
|         \--- com.slack.api:slack-app-backend:1.14.0 (*)
+--- com.slack.api:bolt-socket-mode:1.14.0 (n)

Java Runtime version
openjdk version "11.0.12" 2021-07-20 LTS
OpenJDK Runtime Environment Corretto-<IP_ADDRESS>.2 (build 11.0.12+7-LTS)
OpenJDK 64-Bit Server VM Corretto-<IP_ADDRESS>.2 (build 11.0.12+7-LTS, mixed mode)

OS info
ProductName:    macOS
ProductVersion: 11.6.1
BuildVersion:   20G224
Darwin Kernel Version 20.6.0: Tue Oct 12 18:33:42 PDT 2021; root:xnu-7195.141.8~1/RELEASE_X86_64

Steps to reproduce:
The issue occurs when I use secondary message attachments with the blocks field:
var slack = new Slack();
var client = slack.methods();
var req = ChatPostMessageRequest.builder()
    .channel("<user_id>")
    .attachments(asAttachments(
        attachment(a -> a.mrkdwnIn(List.of("pretext"))
            .fallback("Lorem ipsum")
            .pretext("Hello, this is a *test* message")
            .blocks(asBlocks(
                section(s -> s.text(plainText("Test message inside block")))
            ))
        )
    ))
    .unfurlLinks(false)
    .unfurlMedia(false)
    .token(BOT_TOKEN)
    .build();
var res = client.chatPostMessage(req);
if (!res.isOk()) {
  logger.error("error={}, errors={}, errorMetadata={}", res.getError(), res.getErrors(), res.getResponseMetadata());
}

Expected result:
The message is sent successfully.
Actual result:
The message was failed to sent with the error:
error=invalid_attachments, errors=[invalid_keys], errorMetadata=ErrorResponseMetadata(messages=[[ERROR] invalid_keys])

I found a similar ticket https://github.com/slackapi/java-slack-sdk/issues/415 and the cause seems to be because of the attachment has the blocks field. I can confirm that removing the blocks field fixed the issue, but our use case requires using Block Kit inside attachments.

Hi @erathorus, thanks for writing in!
The issue #415 was about the proper support of blocks in color bar attachment. And that's the only available use case that developers can have blocks inside an attachment. Refer to the code in my pull request to learn how it looks like: https://github.com/slackapi/java-slack-sdk/pull/416/files
As far as I know, the attachment you mentioned here does not work. Please consider using straight forward blocks (=not inside an attachment).
I hope this was helpful to you.

Thank you for your supersonic speed support @seratch ❤
It's working now when I remove the mrkdwnIn and fallback fields and set the color field. As such, I'll close this ticket now. Here is an example of a working example for feature readers:
var req = ChatPostMessageRequest.builder()
    .channel("<slack_user_id>")
    .text("This is the text in flyout")
    .blocks(asBlocks(
        section(s -> s.text(plainText("This is the text outside block")))
    ))
    .attachments(asAttachments(
        attachment(a -> a
            .color("#FFFFFF")
            .fallback("This is the fallback text")
            .blocks(asBlocks(
                section(s -> s.text(plainText("Test message inside block")))
            ))
        )
    ))
    .unfurlLinks(false)
    .unfurlMedia(false)
    .token(BOT_TOKEN)
    .build();


Long term module plans

Hi @mholt,
First of all I want to thank you for your work on caddy-server and also for this module!
Despite the experimental nature, we are using the module successfully in our projects and would appreciate if you can share your thoughts on its long-term maintenance prospects.
Are there  plans to integrate this as standard module in caddy-server and to address some of the issues listed in https://github.com/mholt/caddy-grpc-web/blob/2deb9fd4ddd703e0037e63edb5bcd7a4df85f66f/grpcweb.go#L37 ?

Hey @vlahan -- good question.
These days, for a module to be included in our standard distribution, it has to be generally applicable and relatively popular. Right now I'm not sure how many users are bridging gRPC-Web (thanks to lack of telemetry, which the community voted against years ago, despite my better judgment, sigh). So all I have to go on is feedback like yours, which I appreciate.
I'm really glad to hear you're using it successfully!
As for long-term plans, I don't personally use this module myself, so it basically comes down to low maintenance tasks to keep it functional, things like updating dependencies or reviewing pull requests; but at the current usage level I don't foresee myself investing much time into it, since it works as-is.
That said, I'm always able to help our sponsors with whatever they need. In this case, this module simply wraps another library which I don't have control over. If the sponsorship tier is sufficient, we can fork and maintain dependencies if needed, implement optimizations, etc. I also maintain dev branches for some sponsors, like Stripe, who have specific dependency and patch requirements.
So yeah, I'd be happy to address those points with this library. Can we get you set up on a sponsorship? Higher tiers offer significant discounts for dev work (and for some it's free -- for example, for the work I do for Stripe is included in their sponsorship).

Hi @mholt, thanks a lot for taking the time for a comprehensive feedback!
I am glad to hear that the basic maintenance to keep the module in step with the caddy-server releases is not endangered despite its experimental nature.  I don't have a good grasp on the size of the overall grpc-web community. I assume many in the cloud / API servers domain gravitate to envoy for grpc-web out of inertia. Our use is a bit specific as we deploy it on an embedded device as part of the configuration and telemetry system. In this context caddy has clear benefits, both for efficiency and portability reasons, but at the same time the size of the market is definitely smaller.
Thanks for pointing the benefits of the different sponsorship tiers, we will take a look and would be glad to support within our means as a small company.

Yeah this module is sticking around, I just consider it "done" for now -- (it's not "done" as we discussed but until it is funded I'm basically done with it).


[Feature][connector-jdbc] Support PostgreSQL based databases to use Copy statements to accelerate data writing.

Purpose of this pull request

Support the COPY table1 from STDIN method for fast data writing,
[Feature] #6372
Does this PR introduce any user-facing change?

Add Connector-Jdbc Options [use_copy_statement] For PostgreSQL Driver.
How was this patch tested?

Add e2e test.
/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-3/src/test/resources/jdbc_postgres_source_and_sink_copy_stmt.conf
Check list

[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update change log that in connector document. For more details you can refer to connector-v2
Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist

[ ] Update the release-note.

cc @hailin0 @Carl-Zhou-CN

Please fix CI. Thanks

@Hisoka-X Ci. success.
I often encounter Ci stopping in Kafka or RocketMQ e2e testing.

@Hisoka-X
I added it to knowledge dependencies.txt and checked the debug log. The dependency: copy-dependencies command skipped connector jdbc, which is a dependency in the connector jdbc component.
Logs:
2024-03-07T06:54:28.7284111Z ##[debug]Code: 0
2024-03-07T06:54:28.7285606Z ##[debug]Signal: null
2024-03-07T06:54:29.6886880Z Command completed after 1 attempt(s).
2024-03-07T06:54:29.6922824Z ##[debug]Node Action run completed with exit code 0
2024-03-07T06:54:29.6927130Z ##[debug]Set output total_attempts = 1
2024-03-07T06:54:29.6927664Z ##[debug]Set output exit_code = 0
2024-03-07T06:54:29.6928412Z ##[debug]Finishing: Install
2024-03-07T06:54:29.6937509Z ##[debug]Evaluating condition for step: 'Check Dependencies Licenses'
2024-03-07T06:54:29.6939150Z ##[debug]Evaluating: success()
2024-03-07T06:54:29.6939655Z ##[debug]Evaluating success:
2024-03-07T06:54:29.6940152Z ##[debug]=> true
2024-03-07T06:54:29.6940621Z ##[debug]Result: true
2024-03-07T06:54:29.6941269Z ##[debug]Starting: Check Dependencies Licenses
2024-03-07T06:54:29.6951031Z ##[debug]Loading inputs
2024-03-07T06:54:29.6951867Z ##[debug]Loading env
2024-03-07T06:54:29.6994139Z ##[group]Run tools/dependencies/checkLicense.sh
2024-03-07T06:54:29.6994643Z tools/dependencies/checkLicense.sh
2024-03-07T06:54:29.7059615Z shell: /usr/bin/bash -e {0}
2024-03-07T06:54:29.7059956Z env:
2024-03-07T06:54:29.7060366Z   JAVA_HOME: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.402-6/x64
2024-03-07T06:54:29.7061028Z   JAVA_HOME_8_X64: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.402-6/x64
2024-03-07T06:54:29.7061550Z ##[endgroup]
2024-03-07T06:54:29.7193306Z ##[debug]/usr/bin/bash -e /home/runner/work/_temp/b429e46b-370e-4878-a7cd-f3659645b8ec.sh
...
...
2024-03-07T06:54:35.4433398Z [INFO] ----------------< org.apache.seatunnel:connector-jdbc >-----------------
2024-03-07T06:54:35.4434552Z [INFO] Building SeaTunnel : Connectors V2 : Jdbc 2.3.5-SNAPSHOT        [65/194]
2024-03-07T06:54:35.4435672Z [INFO] --------------------------------[ jar ]---------------------------------
2024-03-07T06:54:35.8255340Z [INFO]
2024-03-07T06:54:35.8256438Z [INFO] --- maven-clean-plugin:3.3.2:clean (default-clean) @ connector-jdbc ---
2024-03-07T06:54:35.8267264Z [INFO] Deleting /home/runner/work/incubator-seatunnel/incubator-seatunnel/seatunnel-connectors-v2/connector-jdbc/target
2024-03-07T06:54:35.8472265Z [INFO]
2024-03-07T06:54:35.8477947Z [INFO] --- flatten-maven-plugin:1.3.0:clean (flatten.clean) @ connector-jdbc ---
2024-03-07T06:54:35.8480451Z [INFO] Deleting /home/runner/work/incubator-seatunnel/incubator-seatunnel/seatunnel-connectors-v2/connector-jdbc/.flattened-pom.xml
2024-03-07T06:54:35.8481960Z [INFO]
2024-03-07T06:54:35.8483001Z [INFO] --- maven-dependency-plugin:3.1.1:copy-dependencies (default-cli) @ connector-jdbc ---
2024-03-07T06:54:35.8484125Z [INFO] Skipping plugin execution
...
...

Please fix CI

waiting for ci passed

@Hisoka-X
@hailin0


warning The getter 'key' is not defined for the class 'KeyboardEvent'.

I am getting an error in Dartium and Chrome when trying to access the key property on a KeyboardEvent object. I tested this on DartPad and I see a warning. Oddly enough the app works. Although maybe it is because DartPad is on Dart V1.16.0 and I'm running Dart V1.17.1.
https://dartpad.dartlang.org/1ba18057a006a6654086783dc43ea5c2
https://gist.github.com/1ba18057a006a6654086783dc43ea5c2

Looks like a dup of #26699

Thanks @zoechi - does indeed look like a dup of #26699.


Question about the max density?

Hi! I am confused with the metric "Max Density". Does the metric "Max Density" refer to the maximum density of fluid particles, or is it the relative error between the maximum density of the fluid and the maximum density of the ground truth?
In the paper, Formula (15) for "Max Density" likely denotes the latter, i.e., the relative error of maximum density, where a value closer to 0 is preferable.

However, Table 2 in the appendix suggests that "Max Density" is the former, meaning the maximum density of fluid particles, where values closer to 1 are considered better.

Looking forward to your early reply! Thanks!

Hi! Yes I can see why this can be confusing.
The latter is true. The values in Table 2 are not the raw error values but relative accuracy values as described in Figure 6. I.e. a value of 1 corresponds to the error of our final method, while small values represent a lower relative accuracy and thus larger error. A value of 0.5, for example, would mean half the accuracy and double the error. We chose this format to relate the error to the final method, which we felt was important in an ablation study, and to normalise the error evaluation for better visualisation in the graph. I hope that answers your question?


cloud-provider broke the ability to drain the node

What kind of request is this (question/bug/enhancement/feature request):
BUG
Steps to reproduce (least amount of steps as possible):

Create a create with cloud provider enabled(aws in this case)
Try to drain a node or enable node drain during cluster upgrade

Result:
Node failed to drain, the node status stuck in draining
Other details that may be helpful:
It looks like the cloud provider is replacing the node name to ip-172-31-39-209.us-east-2.compute.internal while the Rancher is trying to drain the node with ip-172-31-39-209
kubelet logs:
W0827 01:35:26.912391    9849 setters.go:183] replacing cloudprovider-reported hostname of ip-172-31-39-209.us-east-2.compute.internal with overridden hostname of ip-172-31-39-209
W0827 01:35:36.943886    9849 setters.go:183] replacing cloudprovider-reported hostname of ip-172-31-39-209.us-east-2.compute.internal with overridden hostname of ip-172-31-39-209

Rancher logs:
2020/08/27 02:08:04 [ERROR] nodeDrain: kubectl error draining node [ip-172-31-39-209] in cluster [c-l856v]:
2020/08/27 02:08:04 [INFO] Draining node ip-172-31-39-209 in c-l856v with flags --delete-local-data=false --force=false --grace-period=-1 --ignore-daemonsets=true --timeout=120s
2020/08/27 02:08:04 [ERROR] nodeDrain: kubectl error draining node [ip-172-31-39-209] in cluster [c-l856v]:
2020/08/27 02:08:04 [INFO] Draining node ip-172-31-39-209 in c-l856v with flags --delete-local-data=false --force=false --grace-period=-1 --ignore-daemonsets=true --timeout=120s
2020/08/27 02:08:04 [ERROR] nodeDrain: kubectl error draining node [ip-172-31-39-209] in cluster [c-l856v]:
2020/08/27 02:08:04 [INFO] Draining node ip-172-31-39-209 in c-l856v with flags --delete-local-data=false --force=false --grace-period=-1 --ignore-daemonsets=true --timeout=120s
2020/08/27 02:08:04 [ERROR] nodeDrain: kubectl error draining node [ip-172-31-39-209] in cluster [c-l856v]:

k get node
NAME                                          STATUS   ROLES                      AGE   VERSION
ip-172-31-39-209.us-east-2.compute.internal   Ready    worker                     18m   v1.15.12
ip-172-31-45-250.us-east-2.compute.internal   Ready    controlplane,etcd,worker   21m   v1.16.13

Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): v2.4.5
Installation option (single install/HA): HA

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): custom
Machine type (cloud/VM/metal) and specifications (CPU/memory): 2/2
Kubernetes version (use kubectl version):

all verions

Docker version (use docker version):

Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:45:36 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:44:07 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

gz#11932

This is most likely resolved with https://github.com/rancher/rancher/issues/23333

@superseb Thanks for the information, the issue has fixed in Rancher v2.4.6. Closing this ticket.


normalize doesn't sort Base Localizable.strings

Hello there, thanks for this amazing project!
I noticed that Base Localizable.strings doesn't seem to be supported, is that by design? For example sortByKeys doesn't apply to Localizable.strings in Base.lproj.
[update.normalize]
path = "MyApp/Resources"
sourceLocale = "Base"
harmonizeWithSource = true
sortByKeys = true
Thanks!

Yes, that's by design. Actually there shouldn't be a Localizable.strings file within the Base.lproj folder in your project. It's only there for Storyboard and XIB files, at least that's how I always understood it. Correct me if I'm wrong.
I have seen in some projects, that a Localizable.strings file is accidentally placed there. But I usually just delete it without any disadvantages. What should it be for? I haven't seen any iPhone user, running his device in the language "Base", so it wouldn't ever be used AFAIK.
I hope this clarifies the issue. I'll close this as a non-issue until you prove me otherwise.

Base is meaningful for people that not using English as the default language. If a user preferred language is not in the list of supported language of the app, then it will fallback to Base ahead of English.

Can you point me to the documentation of this behavior?

Actually you are right, I can't find a way to let the Base stings show up. So yeah it was false statement.

From this Apple Documentation:

... Xcode modifies your project folder according to the selections you make in this dialog. Xcode creates a Base.lproj folder in your project folder and adds to it the resource files you select. Xcode creates a language folder for the development language but only adds resources that need translation to the folder. For example, if you select English as the development language, Xcode inserts the resource file in the Base.lproj project folder but not the en.lproj folder because the resource is already in English.

I'm new to localization and just throwing this out there, but wouldn't this suggest that you have the translation for the development language directly in the .xib/.storyboard files inside the Base.lproj folder?
I feel like that would make so much more sense from a workflow perspective as well.
You build the interface files in a real language - the development language. And then any further languages you add via .strings files.
Instead of building the UI in some pseudo 'base' language, and then having the extra effort of translating from the pseudo language to the development language via a strings file.


What software would you like us to add to our image catalog? (one per issue please)
https://github.com/grafana/grafana
which versions of the containerized software do you need?
latest
Add some justification for why this specific image and versions are important.
Grafana allows you to query, visualize, alert on and understand your metrics no matter where they are stored. Create, explore, and share dashboards with your team and foster a data-driven culture:
Visualizations: Fast and flexible client side graphs with a multitude of options. Panel plugins offer many different ways to visualize metrics and logs.
Dynamic Dashboards: Create dynamic & reusable dashboards with template variables that appear as dropdowns at the top of the dashboard.
Explore Metrics: Explore your data through ad-hoc queries and dynamic drilldown. Split view and compare different time ranges, queries and data sources side by side.
Explore Logs: Experience the magic of switching from metrics to logs with preserved label filters. Quickly search through all your logs or streaming them live.
Alerting: Visually define alert rules for your most important metrics. Grafana will continuously evaluate and send notifications to systems like Slack, PagerDuty, VictorOps, OpsGenie.
Mixed Data Sources: Mix different data sources in the same graph! You can specify a data source on a per-query basis. This works for even custom datasources.

It looks like that the grafana is already present in Chainguard catalogue

@gerbil could you kindly confirm it or otherwise?

As I can see - latest and latest:dev are available now in catalogue