text:
Wallet.dat corrupted
I ran Bitcoin during 2009 and 2010.
I found an old file I had renamed to xxxx.dat. I recently downloaded the Bitcoin.org wallet and let it run.
I get the "wallet.dat corrupted, salvage failed" message after I put in the file I believe to be my old wallet.dat. I have deleted the installation and restarted my computer. I hope to be able to start over. Any suggestions?

Can you be more precise about what it is you are trying to do? Do you have bitcoins you want to recover?

I'm trying to see how many bitcoins are in it, and to recover them to cash in.

I fear all of my bitcoins and altcoins have been stolen by mg

I ran the blockchain during 2009 and 2010

By "ran the blockchain" I guess you mean you ran Bitcoin-core wallet software sometimes known as Bitcoin-qt.

I downloaded the Bitcoin.org wallet

Bitcoin.org lists many wallets, I guess you mean Bitcoin-core wallet.

I found an old file I had renamed to xxxx.dat.

There are many different data formats and software applications that use ".dat" as a filename extension. So your xxxx.dat might not be a wallet.

Any suggestions?

Check the format of your xxxx.dat file. See if it matches the expected format.
There are tools that might assist with this - such as pywallet.
Otherwise maybe you have to download an ancient version of Bitcoin-core

It is from 2011. I thought it was a wallet from bitcoin core I purchased in 2011 to 2012.

Down load an ancient? I thought it would handle any bitcoin core wallet. Back then it was the only wallet. I've forgotten who or where I purchased them.

Bitcoin wallets are free, you would not have needed to purchase a wallet in 2011 or 2012.


text:
Bitcoin transaction inquiry
if suppose a someone offers a prize for anyone that can solve a specific challenge, how will the solver make sure that he/she collects the reward? Does the solver need to send the solution directly to this anonymous person?

One example of this are the hash collision bounties. There are bitcoins out there that can be unlocked by submitting an example of a hash collision for SHA1 (claimed in 2017), SHA256, RIPEMD160, RIPEMD160(SHA256()) and SHA256(SHA256()).
As the bitcointalk post above mentions, since the scriptSigs unlocking these bounties don't cryptographically sign the transaction outputs, anyone can copy them into a transaction that would send the coins to them. This applies to all bitcoin "bounties" or "puzzles".
The posts suggests you mine the block containing this transaction yourself, but unless you have a lot of mining hardware you won't be able to do that. Even then, you wouldn't be safe against a chain reorganization by a large miner or mining pool if the bounty were big enough. Simply broadcasting the transaction carries a high risk of a miner replacing your transaction with theirs or someone intercepting your transaction and broadcasting their version to miners faster and/or with a higher fee.

So if the solver does not have a large computing power, he/she always carries the risk of not receiving the bounty? Is there a way to remedy this problem other than by upgrading mining hardware?

Yes, receiving a reward for publicly submitting information that can be copied by anyone else to also try to claim the reward can never be safe. The Bitcoin network will reach consensus on which of those transactions was first, but it cannot know which of them was actually first.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Bitcoin node gets stuck on a stale tip
o7
I'm having a issue with my node. I'm running it on a 2017 mba, and randomly it'll get stuck on a stale tip and apparently can't find the active branch, as it'll try for hours until manually restarted. The most i can figure out is it fails to connect to a peer and then just stops trying? Relevant logs are attached below system info.
Bitcoin core version: Bitcoin Core version v23.0
Config:
## General
server=1
txindex=1
blockfilterindex=1
daemonwait=1
mempoolexpiry=24
disablewallet=1
sysperms=1
datadir=/run/media/<user>/Chain/bitcoin
shrinkdebugfile=1

## Services
peerbloomfilters=1
peerblockfilters=1

zmqpubrawblock=tcp://127.0.0.1:29000
zmqpubrawtx=tcp://127.0.0.1:29000
zmqpubhashtx=tcp://127.0.0.1:29000
zmqpubhashblock=tcp://127.0.0.1:29000

## Networking
port=8333
rpcuser=mempool
rpcpassword=mempool
rpcbind=0.0.0.0
rest=1
listen=1
discover=1
maxconnections=25
whitelist=download@127.0.0.1
rpcallowip=192.168.1.0/24
### Tor
listenonion=1
onion=127.0.0.1:9050

## Tor Seeds
addnode=kpgvmscirrdqpekbqjsvw5teanhatztpp2gl6eee4zkowvwfxwenqaid.onion
addnode=bnx4povtqynvwnui5oqm5xcxqvat3j7yzgn6mqciuyszqawhuayvyaid.onion:8333
addnode=qxkgr5u4rmwme5jticjuuqrecw5vfc5thcqdxvbffwbqgqkdin7s7vid.onion

System info:
OS: EndeavourOS Linux x86_64
Host: MacBookAir7,2 1.0
Kernel: 5.18.12-zen1-1-zen
Shell: bash 5.1.16
CPU: Intel i5-5350U (4) @ 2.900GHz
Relevant logs:
2022-07-27T12:16:11Z UpdateTip: new best=00000000000000000004d531a2a6bda3200d27f01e1788fa8542eecd337c9759 height=746776 version=0x2222a000 log2_work=93.648170 tx=752013680 date='2022-07-27T12:15:46Z' progress=1.000000 cache=44.8MiB(342049txo)
2022-07-27T12:17:59Z Socks5() connect to z35xbo2q2mclub2pfljc6q4czxuvbivohmuoi3jrw5yukzqb635li2id.onion:8333 failed: host unreachable // first connection failure
2022-07-27T12:18:14Z New outbound peer connected: version: 70016, blocks=746776, peer=281 (block-relay-only)
// ..normal logs...
2022-07-27T12:59:53Z New outbound peer connected: version: 70015, blocks=746783, peer=307 (block-relay-only)
2022-07-27T13:02:14Z UpdateTip: new best=000000000000000000089e9978c6b5e69d0d02109ad3e418ec096ce03d3292b0 height=746784 version=0x20800004 log2_work=93.648258 tx=752022658 date='2022-07-27T13:00:22Z' progress=1.000000 cache=52.2MiB(380114txo) // Seems to die here, as this was the height i found it at when i restarted at block 746826
2022-07-27T13:24:16Z Disconnecting outbound peer 9 for old chain, best known block = 000000000000000000098d0c86665071e838f3483d582c0af7a16ef07745020e
2022-07-27T13:24:16Z Disconnecting outbound peer 1 for old chain, best known block = 000000000000000000098d0c86665071e838f3483d582c0af7a16ef07745020e
2022-07-27T13:24:16Z Disconnecting outbound peer 17 for old chain, best known block = 000000000000000000098d0c86665071e838f3483d582c0af7a16ef07745020e
2022-07-27T13:24:16Z Disconnecting outbound peer 16 for old chain, best known block = 000000000000000000098d0c86665071e838f3483d582c0af7a16ef07745020e
2022-07-27T13:36:10Z Potential stale tip detected, will try using extra outbound peer (last tip update: 2036 seconds ago)
2022-07-27T13:46:40Z Potential stale tip detected, will try using extra outbound peer (last tip update: 2666 seconds ago)
2022-07-27T13:57:10Z Potential stale tip detected, will try using extra outbound peer (last tip update: 3296 seconds ago)
2022-07-27T14:07:40Z Potential stale tip detected, will try using extra outbound peer (last tip update: 3926 seconds ago)
2022-07-27T14:18:10Z Potential stale tip detected, will try using extra outbound peer (last tip update: 4556 seconds ago)
// ...continues...
2022-07-27T17:27:10Z Potential stale tip detected, will try using extra outbound peer (last tip update: 15896 seconds ago)

Edit: Also seems to hang on exit, and it even logs stale tip messages. pkill -9 doesnt kill it either, only a reboot will.

The cause of the issue seems to have been my vpn, after disabling it my node stopped getting stuck.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Estimating Cryptocurrency Gap
Estimating Cryptocurrency Gap
I'm trying to do a rough estimate for the tax gap for cryptocurrency in Canada and I'm having a hard time producing it due to a lack of data on the topic. The tax gap is the amount of tax not paid that should have been paid, so it's a type of measure of non-compliance with tax laws. In Canada, crypto assets are taxed when they are disposed of. They are taxed under business income or capital gains and could also be taxed under GST/HST. Business income tax applies to the full amount and capital gains tax applies to only 50% of the amount.
I am new to cryptocurrency and not quite familiar with the best data sources to look at. So far, here are my ideas to give a rough estimate of a possible upper bound for the tax gap:
Given that x% of the global cryptocurrency market is based in Canada, approximately $y in crypto transactions can be sourced to Canada. Under a very extreme assumption that all Canada attributed volume reflects unreported income taxable at the top rate of z%, cryptocurrency would contribute roughly $(zy%) to the tax gap, with the actual tax gap value being much lower. This $ (zy%) would serve as an upper bound. There are obvious issues with the estimate, hence why it is a very rough estimate.
To do this estimate, I need to figure out what the values of x% and $y are. Any suggested data sources to look at for this?
If anyone has suggestions of how I can formulate a better upper estimate or approximate a lower bound estimate, or if there are any good data sources to look at that could help me formulate a better estimate, it would be appreciated.
Thank you!

Since Bitcoin is a global network without user registry or other user telemetry that is designed to provide cash-like financial privacy, it is inherently difficult to:

measure user counts
assign transaction activity to specific users
identify users
allocate user activity to specific jurisdictions
classify transaction activity by purpose
even notice transaction activity at all when e.g. performed on second layers such as the Lightning Network or between accounts of financial services

Your best bet might be to look for survey results, business data provided by Canadian services, or figures published by academic or industry researchers.
Progressing beyond something that would classify as “very rough estimates” should take considerable research effort, maybe on par of a thesis rather than a few hours here or there.

I'm trying to start with a rough estimate but I do intend to put in a lot of effort into trying to develop a closer estimate. I have a background in math/stats so I have many ways to try and develop an estimate; however, I am struggling to find data and solid statistics. I'm new to cryptocurrency, so I am wondering if I'm approaching my research incorrectly and if there is other available data out there that I can work with

No, your approach sounds reasonable, it's simply that this sort of data is inherently unavailable with the exception of prior work by other groups that have published their own research. There seems to be some data on Statista for example, and there might be more in the context of e.g. parliament hearings about cryptocurrencies. Sorry, I'm not based in Canada and have not looked into data specifically for Canada before.


text:
what IS Bitcoin's pourpse and what do they need me to do and why do they need me to do it and what is said thing for?
this program is advertiseing btcoins but not eplaining what they want to pay me to do and why dos it need to be done? its running algorithms but what is it doing that for? is it trying to protect something and if so what is it they are trying to protect and why?

it is now corrected.

possible duplicate of What is a good way to concisely explain Bitcoin?.  See also What exactly is Mining?

Still terrible question: You don't say what "this program" is. You ask multiple questions in one. You have done 0 research yourself, as all of those questions have been answered many times already. And it's full of confusing typos, indicating you don't even spend a few minutes to ask a concise question.

I have not very clear what exactly you are asking, but I will try to answer according to what I think you are asking.
The purpose of Bitcoin is creating a new currency where neither governments nor private institutions will have the power to manipulate. As you may already know, when a country needs more money or has financial problems, they just push a button and make new money out of the blue. Thus, the value of that money compared to the rest of the world is devaluated, which has two important implications:
That countrie's exports are cheaper, so other countries will be willing to buy their goods.
That countrie's imports are more expensive, so inside the country people will start to buy goods made inside the country because the imported ones are too expensive.
This will also lead to a raise of prices of the products that are done with raw materials from other countries which in the end will lead to a devaluation of the money people has been keeping in the banks. They can now buy less things with the same money they had.
Bitcoin, being decentraliced and independent can not be devaluated, transactions are quite fast (an international money transer takes about 2 days if you do it with a bank and few minutes if you do it with Bitcoins -the truth is it doesn't take time, time is taken waiting for confirmations of the transaction-) and the number of coins that will be mined is limited (about 20million).
Some stores start to see the potencial, people can pay and no commisions are applied by banks so they have that as an addition for starting to use that coin.
Now we get to the point... what do they want you to do to earn money? Nothing, nobody wants you to do anything, if you think someone is interested in buying Bitcoins then you can try to mine some and sell them to those that want them, you are creating coins and selling them, so it is up to you to decide if you want to do that.
Who gives the value to the Bitcoin? As always, people and the trust of the people on that coin. Who gives the value to the dollar, the central bank? no, it is the people that trust that if they go to a store and give a painted sheet of paper called money to the shopkeeper he will give them goods for it.
So people willing to buy Bitcoin and people willing to sell Bitcoin give the price to Bitcoin itself. If demand is higer price will rise, if demand is lower price will go down.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
How to check Bitcoin wallet balance without any API
I am building an application and that will involve large set of bitcoin wallets and need to keep on checking for balance. I don't want to use any third party APIs as they all have hourly/secondly limits.
Is there any way I can check bitcoin wallets balance without any API? I am not sure is Bitcoin Demon and Bitcoin Core is for this purpose. Can anyone please suggest me the best way to verify bitcoin balances and transactions without any limits.

If you are not going to use a 3rd party API, you'll need to run a node and add each address you want to query as a watch only address. If you are looking to create a database with all addresses, similar to a block explorer website, you'll need to parse the blockchain using something like abe, and then continually update your database for each new transaction.  There's also the painful approach of turning on txindex=1 in your bitcoin.conf, and then going through every single block since genesis, that checks all txs outputs and then keeps a record of all addresses and transactions.

This actually might be harder than it seems.
The blockchain does not store addresses and their corresponding balances. The blockchain stores transactions. So if someone were to look for the final balance of an address, he/she would have to iterate through every transaction(inputs/outputs) that a particular address was involved in starting from the beggining of the blockchain and calculate the final balance for that address.

As suggested you can run your own node , and run a service like abe to index transactions. You could also run copay wallet along with the node. Though this is not recommended as it would take lot of time and would consume lot of resources(150 GB HDD/ 4 GB RAM).
You could also try blockonomics - wallet watcher . They are optimized for checking balance of large number of addresses at once. The free plan is restricted to 50 address, you can easily upgrade to upto 1000 addresses. Their API can be used to add/delete addressees.

You could craft the query as an HTML POST or GET message sent to a blockchain-explorer's website and parse the output received.

Doesn't it again calling a third-party website blockchain.info??

It would be calling a third party website but avoid using APIs

yes, like this: https://blockchain.info/address/12sENwECeRSmTeDwyLNqwh47JistZqFmW8?format=json

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Can you attack the network in a UASF-way?
Let's say that an attacker wants to deploy a softfork, that isn't desired by the majority of the Bitcoin users. Can he enforce that softfork in a UASF-way, essentially flooding the network with nodes that support it?
It is said that miners follow the demand. How can they know if the nodes correspond to real users and not to an attacker?

I may flirt with personal opinions here but as always alternative answers are welcome if people disagree with how I classify things.
Anytime there is an attempt at a consensus rule change and there is resistance or opposition to that consensus rule change Bitcoin is then in chain split territory or as close as Bitcoin gets to a "war" over the consensus rules like we saw with the block size wars. Hence I and others have argued that consensus rule changes should be made (to the extent that they are made at all) infrequently and with a very high bar of testing, review and community consensus.

Let's say that an attacker wants to deploy a softfork, that isn't desired by the majority of the Bitcoin users. Can he enforce that softfork in a UASF-way, essentially flooding the network with nodes that support it?

The scenario you describe of a lone attacker with lots of sybil nodes isn't particularly concerning. At best it could cause disruption on the P2P network but it isn't going to succeed in changing the consensus rules on the blockchain and on the Bitcoin network without community adoption. Andreas Antonopoulos has described Bitcoin as having "constituents of consensus". We could argue over how to break these constituents down but they clearly include (in no particular order) economic full nodes (nodes verifying their economic activity), protocol developers, miners, exchanges, wallets and protocols built on top of the base layer. If you are attempting to change the consensus rules and you don't have the majority of these constituents onboard then Antonopoulos has described it as attempting an economic blockade in the middle of the ocean. Or in other words you may shout but no one listens to you and just sails around you. You may fork yourself off the Bitcoin blockchain and the Bitcoin network but you won't succeed in changing the consensus rules for everyone else.
Hopefully we won't have another consensus rule "war" anytime soon (ideally ever). During "war" like scenarios there are "fog of war" like dynamics and misinformation deliberately spread to cause confusion. What constitutes a "war" is also unclear. A single individual in the middle of the ocean declaring "war" is ignored. If it starts to look like the constituents of consensus are split and one side is going to attempt to force through their consensus rule change against the other constituents' wishes then we are definitely in consensus rule "war" territory again.
Personally (opinion alert) I think there will be future soft fork(s) attempted with the vast majority (ideally all) of the "constituents of consensus" onboard and comfortable that the change being made is optimal and worth taking the chain split risk for. If we ever do attempt it it will be all our responsibilities to ensure accurate information is circulated and any troublemakers from say altcoin communities don't succeed in turning that consensus rule change into a "fog of war" like scenario. But as I said I think there is general acceptance at this point that the technical and community consensus bar should be very high before embarking on an attempted consensus rule change and that, perhaps more than anything else, protects against chain splits occurring out of nowhere.


text:
How to deal with this situation in Lightning Network?
What happens if Alice sends excess amount by mistake in Lightning Channel, and Bob denies to send back excess and broadcast the last transaction to Blockchain?

It's not clear to me how that would ever happen. Do you have a concrete scenario in mind?

@Murch yeah that's a catch, Alice needs to pay bob through Invoice right, Thank you got it :) I guess the above might not be possible

If the payment completes, each party along the route receives the hash preimage and updates their channel balances, then the latest state of the channel between Alice and Bob includes the excess amount. At this point Alice is relying on Bob (or the destination of the payment) to either send the excess amount back or cooperatively settle the channel excluding the excess amount sent. If Bob refuses to do this then Alice cannot reclaim the excess amount. She can try to close the channel with a prior state but if Bob is alert to this he will broadcast a justice transaction. Lightning provides effective instant finality unlike delayed probabilistic finality onchain.

So, it means there is no way to get Alice Excess amount back right?

Alice could ask Bob to send the excess amount back but he is unlikely to agree as he moved money to Carol in a channel between him and Carol assuming a route of Alice -> Bob -> Carol -> Destination. Or Alice could ask the destination to send the excess amount back. But yeah it is relying on others to send it back rather than anything built into the protocol.

If the payment hasn't completed Alice can ask the destination to not accept the incoming payment https://bitcoin.stackexchange.com/questions/112342/is-it-possible-to-cancel-a-lightning-invoice


text:
Number of transactions per blk.dat files
I am working on a parser for the blk.dat files being returned by the Bitcoin Core node. It seems to slow down considerably after a point and I am pretty sure it's because the number of transactions is increasing in the files. My question is, how many transactions do these files contain and is there any way to figure out how many there are without parsing?

No. blk*.dat files are limited by actual size on disk, not number of transactions. Blocks are added to them until they reach a maximum size, not whether they exceed a block or transaction count. As transaction sizes are variable, it is not possible to know how many transactions are in a blk*.dat file without having parsed it first.
However, as Bitcoin Core has already done that parsing for you, you can use the LevelDB database that it creates which contains that information. This answer provides an explanation of the format of the LevelDB records in the block index (the database stored in blocks/index).

Oh so two differently speced systems would have different blk files ?

The size of BLK files is 128MB max on any system regardless of specification. However even two identically specified systems would have  BLK files whose contents differ due to receiving data in different order and due to differing stale blocks. Early BLK files contain many blocks, each with few transactions. Recent: the reverse


text:
Would a consensus algorithm like the one I describe work?
Given n nodes(miners/validators),

Every unspent transaction output is broadcasted to all n nodes.
Each node will select a batch of transactions and add it to their
block. The specific mechanism of how they select transactions will
be dependent on each node’s source code.
A node gets randomly selected to add a block to the blockchain with a chance of 1/n.
The node broadcasts its block to all other nodes before adding the
block to the chain. If the block contains illegitimate transactions
or tampered data, other nodes will reject the block. A block gets
added only if majority of the nodes in the blockchain agree that the
block is valid.
Before the block gets added, it is hashed along with the hash of the previous block.
In order to incentivize people to set up their own nodes, a small
portion of the actual transaction gets rewarded to the node that
added a block. This is different from Ethereum’s additional GAS
fees. It is more analogous to taxes, where 100% of the burden is on
the seller(recipient of the cryptocurrency).

This algorithm also guards against many common security issues:

Corrupting data of previously added nodes: Resolved, since even if a malicious actor tampers with data of previously added blocks, the protocol can always compare the hashes of the tampered chain with that of every other node(nodes constantly communicate), and update its chain to the ‘correct’ chain(once again accepting majority’s chain as correct data, assuming that the 51% rule is always observed).
Adding a faulty block in the first place: Resolved, since it is
impossible to add a block without getting approval of majority of
nodes. Even if a malicious actor adds the block to its own chain,
other nodes wouldn’t add it to theirs, causing the malicious node to
eventually update an obsolete chain.
Data retrieval is also based on proof of majority, so every time a transaction occurs, it checks with all n nodes to find a valid historical precedence of the transaction(e.g. A can only send 100 crypto to B if he or she owned it in the first place) and accepts the majority’s historical record(once again proving why it is meaningless for malicious nodes to update their own chain)

This algorithm doesn’t cost huge amounts of computing power and money to purchase equipment(unlike PoW) and doesn’t cost anything to set up nodes aside from basic equipment cost(unlike PoS).
It should work as a potential consensus algorithm, but I want critiques on potential flaws of the consensus algorithm. Furthermore, blockchain technology is becoming more and more mainstream, but it is rather difficult to research about different consensus algorithms without taking graduate level math. What's the easiest way for newly acquainted scholars of blockchain technology to study it and stay up to date with latest developments?

How does the node in step 3 get selected? How is majority decided in step 4? I have the feeling you want to initiate a discussion rather than get an answer to a specific problem you face. This Q&A website is probably the wrong place for a discussion - it just wasn't designed for that.

In step 3 I can just spin up a million nodes and I get to pick all the blocks?

Also in step 4, how do nodes know that a majority agrees?

A node gets randomly selected to add a block to the blockchain with a chance of 1/n.

This step is the key problem. How is the node randomly selected? What source of randomness would you use? If a node is chosen to add a block to the blockchain but goes offline what happens then? Perhaps a replacement node is "randomly" chosen? But then half the network finds out that previous node did actually provide a block and half the network continues to add blocks on top of the block the replacement node provided. What then? With (perhaps rare) edge cases it needs an ultimate authority to clean up uncertainty and confusion. Bitcoin's proof of work is decentralized, incredibly robust to edge cases, does not need a network agreement on a source of randomness and does not need network agreement on time measurement (e.g. we'll wait 30 seconds for a node to provide a block). It can only do this because it only provides eventual finality and eventual network consistency. Although increasingly unlikely any length re-org (2 block, 3 block, 4 block etc) is theoretically possible.


text:
How does Zerocoin work?
How does Zerocoin work?
I got the idea that it is used to add anonymity in bitcoin.
It converts bitcoin to Zerocoin and back to bitcoin. But how does this process work? I tried reading the research paper on Zerocoin, but the example provided is difficult to understand (for me).

Imagine that all users share access to a physical bulletin board. To
  mint a zerocoin of fixed denomination $1, a user Alice first generates
  a random coin serial number S, then commits to S using a secure
  digital commitment scheme. The resulting commitment is a coin, denoted
  C, which can only be opened by a random number r to reveal the serial
  number S. Alice pins C to the public bulletin board, along with $1 of
  physical currency. All users will accept C provided it is correctly
  structured and carries the correct sum of currency. To redeem her coin
  C, Alice first scans the bulletin board to obtain the set of valid
  commitments (C 1 , . . . , C N ) that have thus far been posted by all
  users in the system. She next produces a non-interactive
  zero-knowledge proof π for the following two statements: (1) she knows
  a C ∈ (C 1 , . . . , C N ) and (2) she knows a hidden value r such
  that the commitment C opens to S. In full view of the others, Alice,
  using a disguise to hide her identity, 1 posts a “spend” transaction
  containing (S, π). The remaining users verify the proof π and check
  that S has not previously appeared in any other spend transaction. If
  these conditions are met, the users allow Alice to collect $1 from any
  location on the bulletin board; otherwise they reject her transaction
  and prevent her from collecting the currency.

One of the problems (or advantages) of Bitcoin is that funds can be traced. Even if each address is used only once, it is possible to link individual transactions to each other similarly to how it is possible to track bank notes by their serial numbers (with the addition that everyone records serial numbers of all the notes they handle).
Zerocoin solves this problem by providing a new kind of transaction outputs that are not linked to inputs that spend them. Actually, outputs are linked to inputs, but the link is cryptographically obscured, so that an observer can't find which input an output is linked to. However, it is still possible to check that there is only at most one input linked to any particular output, so that double-spends are not possible.
To obscure the link, a cryptographic construction is used which is called zero-knowledge proof. Zero-knowledge proof is a way to show that something is true without revealing any useful information. For example, a digital signature is a zero-knowledge proof that shows that the signer knows the private key. There is no way, however, to learn the private key from the signature.
Proofs used in Zerocoin are more complex. Basically, when generating a Zerocoin, a user picks a random serial number and includes a special irreversible function of that number in the output. When spending, he reveals the number and provides a proof that the number corresponds to one of the outputs. Only one serial number corresponds to each of the outputs, so double-spends are prevented by checking that each serial number is used only once.

Considering that privacy is specifically called out in the Bitcoin whitepaper as a feature and "linkability" is not, I think it's erroneous to describe it as "advantages". :)

The process is basically as follows.
1. You generate a random serial number S and a secret r. S will be used to denote a zerocoin. Then you create a commitment C = f(S,r) s.t. without knowing both S and r it is very hard to get those back from C. Then you disclose C into the public channel.
2.To redeem the coin S you disclose S and give a proof that S corresponds to some valid commitment in a set of large number of commitments (provided by you), say {C1, C2, ..., C, ..., Cn}. As you don't explicitly mention which commitment does S correspond to, you retain your anonymity.
3. When other users verify that your proof is valid and the zerocoin S is not already spent, then only you can spend the coin. You probably understand that S needs to be revealed when spending money to prevent double spending.

Zerocoin is a proposed extension on Bitcoin to make it more confidential using zk-SNARKs (more detailed information on zk-SNARKs is founf here: https://blog.ethereum.org/2016/12/05/zksnarks-in-a-nutshell/). Zerocoin only hides the origin of a payment, the destination and amounts are still public.


text:
bitcoin-cli getbalance different from listunspent
I have probably misunderstood the meaning of bitcoind 'getbalance' command.
I am using bitcoind in regtest mode.
I am using imported multisig bitcoin watch-only addresses, which are the following:
mirko_180603243: 2MvaEkzFyw6EcQGR2vrFWQd4rDNSiP4Qn7p
mirko_1169810029: 2N1DwxUDghdk2DRWee8PVLtcY9xxTM374Nj
mirko_1977534765: 2NANSy3VqQZmhQH36tNADVxEahLHJzQovdz

In the initial state I have no such transaction, as expected:
    listtransactions mirko_180603243 10 0 true: []
    listtransactions mirko_1169810029 10 0 true: []
    listtransactions mirko_1977534765 10 0 true: []
Then I send some money to the first account (mirko_180603243)
sendtoaddress 2MvaEkzFyw6EcQGR2vrFWQd4rDNSiP4Qn7p 10 "Initial load"
setgenerate true 6

From the point of view of the balance, everything is right:
getbalance mirko_180603243 0 true: 10
listaccounts 1 true: "mirko_180603243" : 10.00000000

listunspent:
    {
        "txid" : "3c9970cda24b87ebf7080ff40d3be7cdded1afb6eafe3d9b7ade31f9c7fadc13",
        "vout" : 0,
        "address" : "2MvaEkzFyw6EcQGR2vrFWQd4rDNSiP4Qn7p",
        "account" : "mirko_180603243",
        "scriptPubKey" : "a9142480ec8ccaea0328c6392c2ba4a4cd6fb0f6638b87",
        "amount" : 10.00000000,
        "confirmations" : 6,
        "spendable" : false
    }

From the point of view of the transactions, everything is consistent:
listtransactions "mirko_180603243" 10 0 true:
    {
        "involvesWatchonly" : true,
        "account" : "mirko_180603243",
        "address" : "2MvaEkzFyw6EcQGR2vrFWQd4rDNSiP4Qn7p",
        "category" : "receive",
        "amount" : 10.00000000,
        "vout" : 0,
        "confirmations" : 12,
        "blockhash" : "0000a083a8b0543d68059d9c3421b4ff681630601e80c3f4d4e58a05e228385d",
        "blockindex" : 1,
        "blocktime" : 1414401010,
        "txid" : "3c9970cda24b87ebf7080ff40d3be7cdded1afb6eafe3d9b7ade31f9c7fadc13",
        "walletconflicts" : [
    ],
        "time" : 1414400968,
        "timereceived" : 1414400968,
        "comment" : "Initial load"
    }

Then I send 3 BTC from the first account (mirko_180603243: 2MvaEkzFyw6EcQGR2vrFWQd4rDNSiP4Qn7p) to the second one (mirko_1169810029: 2N1DwxUDghdk2DRWee8PVLtcY9xxTM374Nj). Since I only have 10 BTC in one block, I send back 7 BTC to myself.
This is the final transaction:
gettransaction 153c45f199eba376d43cad614c2cd410140d2c9597b0de8392fa71209f6e43a5
{
    "amount" : 10.00000000,
    "confirmations" : 6,
    "blockhash" : "0000da6254c04d050c93bb9949a5fdaec9256038edce3daa9798d2e0feca8199",
    "blockindex" : 1,
    "blocktime" : 1414401566,
    "txid" : "153c45f199eba376d43cad614c2cd410140d2c9597b0de8392fa71209f6e43a5",
    "walletconflicts" : [
],
    "time" : 1414401480,
    "timereceived" : 1414401480,
    "details" : [
],
    "hex" : "010000000113dcfac7f931de7a9b3dfeeab6afd1decde73b0df40f08f7eb874ba2cd70993c00000000fa0047304402205bb11653b7668488cf60a61c458bc729619b395d7b5d82c597d223956c0dc62d0220396fef209b3077b129648797ba72b9da0be94334dcebf067d1d5e31bb26ecf4a014730440220203dcdd4169db7c1259c14c7fe8ed0eb449b3f26a2732cbba3713779034afd9d022020c85a7ec67a8aaad642ff96b63b0c98ff9dedb35a68c3dfddd296ee7db08ac1014c67522102a880b200040eb0f4e3984263ce3fb86b39cb5ba4c9ea247bb2c4de4d36eec4ff4104abc0c6d83b138ff68b60ee11b5deb98da1bd4fe9d4894fe1041e436f5cebfb7be7cf886f15ea34a746302700008c88c1257b2f37128d764bc922940b2e01038752aeffffffff0200a3e1110000000017a914578335446ee73c6379c7cef95c4fc8c3c576e3a0870027b9290000000017a9142480ec8ccaea0328c6392c2ba4a4cd6fb0f6638b8700000000"
}

decoderawtransaction 010000000113dcfac...cd6fb0f6638b8700000000
{
    "txid" : "153c45f199eba376d43cad614c2cd410140d2c9597b0de8392fa71209f6e43a5",
    "version" : 1,
    "locktime" : 0,
    "vin" : [
        {
            "txid" : "3c9970cda24b87ebf7080ff40d3be7cdded1afb6eafe3d9b7ade31f9c7fadc13",
            "vout" : 0,
            "scriptSig" : {
                "asm" : "0 304402205bb11653b7668488cf60a61c458bc729619b395d7b5d82c597d223956c0dc62d0220396fef209b3077b129648797ba72b9da0be94334dcebf067d1d5e31bb26ecf4a01 30440220203dcdd4169db7c1259c14c7fe8ed0eb449b3f26a2732cbba3713779034afd9d022020c85a7ec67a8aaad642ff96b63b0c98ff9dedb35a68c3dfddd296ee7db08ac101 522102a880b200040eb0f4e3984263ce3fb86b39cb5ba4c9ea247bb2c4de4d36eec4ff4104abc0c6d83b138ff68b60ee11b5deb98da1bd4fe9d4894fe1041e436f5cebfb7be7cf886f15ea34a746302700008c88c1257b2f37128d764bc922940b2e01038752ae",
                "hex" : "0047304402205bb11653b7668488cf60a61c458bc729619b395d7b5d82c597d223956c0dc62d0220396fef209b3077b129648797ba72b9da0be94334dcebf067d1d5e31bb26ecf4a014730440220203dcdd4169db7c1259c14c7fe8ed0eb449b3f26a2732cbba3713779034afd9d022020c85a7ec67a8aaad642ff96b63b0c98ff9dedb35a68c3dfddd296ee7db08ac1014c67522102a880b200040eb0f4e3984263ce3fb86b39cb5ba4c9ea247bb2c4de4d36eec4ff4104abc0c6d83b138ff68b60ee11b5deb98da1bd4fe9d4894fe1041e436f5cebfb7be7cf886f15ea34a746302700008c88c1257b2f37128d764bc922940b2e01038752ae"
            },
            "sequence" : 4294967295
        }
    ],
    "vout" : [
        {
            "value" : 3.00000000,
            "n" : 0,
            "scriptPubKey" : {
                "asm" : "OP_HASH160 578335446ee73c6379c7cef95c4fc8c3c576e3a0 OP_EQUAL",
                "hex" : "a914578335446ee73c6379c7cef95c4fc8c3c576e3a087",
                "reqSigs" : 1,
                "type" : "scripthash",
                "addresses" : [
                    "2N1DwxUDghdk2DRWee8PVLtcY9xxTM374Nj"
                ]
            }
        },
        {
            "value" : 7.00000000,
            "n" : 1,
            "scriptPubKey" : {
                "asm" : "OP_HASH160 2480ec8ccaea0328c6392c2ba4a4cd6fb0f6638b OP_EQUAL",
                "hex" : "a9142480ec8ccaea0328c6392c2ba4a4cd6fb0f6638b87",
                "reqSigs" : 1,
                "type" : "scripthash",
                "addresses" : [
                    "2MvaEkzFyw6EcQGR2vrFWQd4rDNSiP4Qn7p"
                ]
            }
        }
    ]
}

Everything is consistent with the unspent blocks:
{
    "txid" : "153c45f199eba376d43cad614c2cd410140d2c9597b0de8392fa71209f6e43a5",
    "vout" : 1,
    "address" : "2MvaEkzFyw6EcQGR2vrFWQd4rDNSiP4Qn7p",
    "account" : "mirko_180603243",
    "scriptPubKey" : "a9142480ec8ccaea0328c6392c2ba4a4cd6fb0f6638b87",
    "amount" : 7.00000000,
    "confirmations" : 6,
    "spendable" : false
}
{
    "txid" : "153c45f199eba376d43cad614c2cd410140d2c9597b0de8392fa71209f6e43a5",
    "vout" : 0,
    "address" : "2N1DwxUDghdk2DRWee8PVLtcY9xxTM374Nj",
    "account" : "mirko_1169810029",
    "scriptPubKey" : "a914578335446ee73c6379c7cef95c4fc8c3c576e3a087",
    "amount" : 3.00000000,
    "confirmations" : 6,
    "spendable" : false
}

What I can not understand is the final balance:
getbalance mirko_180603243 1 true:
    17 instead of 7

getbalance mirko_1169810029 1 true:
    3

Basically, it seems to be that 'getbalance' sums up all the incoming transactions, ignoring the ones that go out.
Currently, I am computing the available amount summing up all the unspent blocks for each account. Unfortunately, though, that is not very user-friendly, as when you pay for 3 BTC and you have a single 10 BTC block, before the confirmations you see 0. I might include the 0 confirmations transactions as well, but the user may think that he could spend that money, which is false.
Sorry for the long message.
P.S.
I was forgetting to mention that I have discovered that accounts are NOT used for debit: if you want to pay, you have to use 'move' or 'sendfrom', but I do want just to have watch-only addresses, as I do not want to keep the private keys.
For this reason, I can not rely on 'move' or 'sendfrom' commands but I have to manually create and multi-sign my transaction.

This is a very good explaination. https://github.com/bitcoin/bitcoin/issues/4572
However, I do not know how to compute the balance...

From the bitcoin "Accounts Explained" wiki page (https://en.bitcoin.it/wiki/Accounts_explained):

When you receive bitcoins, they are always assigned to one of your accounts, and you can change which account is credited based on which bitcoin address receives the coins, just like you tell a bank teller which account to credit when you deposit cash in your bank. However, sending bitcoins is like withdrawing cash from the bank; the coins that are sent out and debited from an account are almost always not the same coins that were deposited into that account.

So you can't really do what you said (below), sending from an account to another account. You can, however, move balances from an account to another account, with the move command, but this doesn't require a bitcoin transaction.

Then I send 3 BTC from the first account (mirko_180603243: 2MvaEkzFyw6EcQGR2vrFWQd4rDNSiP4Qn7p) to the second one (mirko_1169810029: 2N1DwxUDghdk2DRWee8PVLtcY9xxTM374Nj).

I think if you do:
./bitcoin-cli listaccounts

You will see some accounts with negative balances to account for the extra high balance you are seeing.
Note: The "fromaccount" in the sendfrom command just tells the client which account to debit when it sends to the address specified.

Hi Stephen, thank you for your answer.
You are right, I have not been very clear: when I say I send 3 BTC from the first account to the second one, I mean that I create a raw transaction, multisign it and finally submit it. That tx has a p2ph script that corresponds to the multisig address of the second account.
All in all, what I would like to do would be to find the balance without using the 'getbalance' rpc method.
Thanks again for your help.


text:
How to install bitcoin-cli on MacOS?
I have installed bitcoin-core with Homebrew on my MacOS, but the only package installed is Bitcoin-Qt. How do I also install bitcoin-cli?
(Just want to play with the commands for learning.)

I didn't know you could brew install Bitcoin Core until your question. If you download Bitcoin Core from bitcoincore.org bitcoin-cli is included. I'd be surprised if that wasn't the case for the brew install but this is the first time I've seen it

@MichaelFolkson Homebrew installs the same .dmg published on the official website, take a look: https://formulae.brew.sh/api/cask/bitcoin-core.json

Indeed, so it includes the bitcoin-cli. Don't know why the OP doesn't have it. Maybe not following the Core docs on how to access it?

Are you following these instructions for accessing the bitcoin-cli? You should have it installed https://github.com/bitcoin/bitcoin/blob/master/doc/build-osx.md#other-commands

@1440000bytes thanks, yes, it actually reveals that "there's no bitcoin-cli built in the dmg file for mac os". So you need to compile from source.

@MichaelFolkson thanks for you help! I ended up playing with Bitcoin using built-in graphical console.

This will install bitcoind and bitcoin-cli:
$ brew install bitcoin

This is how you install Bitcoin Core (MacOS app with UI):
$ brew install --cask bitcoin-core


text:
Minimum range when importing descriptors?
When importing descriptors with a range less than 1000, it seems to be ignored. Is this deliberate? I could‘t find anything about this in the docs.
To reproduce, first, create an empty wallet:
bitcoin-cli -regtest -named createwallet wallet_name=test disable_private_keys=true blank=true descriptors=true
Import a descriptor with range 10:
bitcoin-cli -regtest -named importdescriptors requests='[{ "desc":  "'"wpkh([9a6a2580/84'/1'/179']tpubDCMRAYcH71Gib1rASZpnMPpJbj7S1Nzmao1nPv5Jrb8pwSUimsH5TmK842UhnGPJbiNThzxhK4BpRUCsjSckpNYkH9kjKGWLd94yhZFN94J/0/*)#rljv8npg"'", "timestamp":"now", "active": true, "range":10}]'
Check the range with listdescriptors:
{
  "wallet_name": "test",
  "descriptors": [
    {
      "desc": "wpkh([9a6a2580/84'/1'/179']tpubDCMRAYcH71Gib1rASZpnMPpJbj7S1Nzmao1nPv5Jrb8pwSUimsH5TmK842UhnGPJbiNThzxhK4BpRUCsjSckpNYkH9kjKGWLd94yhZFN94J/0/*)#rljv8npg",
      "timestamp": 1666876980,
      "active": true,
      "internal": false,
      "range": [
        0,
        999
      ],
      "next": 0
    }
  ]
}

In contrast, if I import a descriptor with a range larger than 1000, it seems to be respected:
bitcoin-cli -regtest -named importdescriptors requests='[{ "desc":  "'"wpkh([9a6a2580/84'/1'/180']tpubDCMRAYcH71GibuLuWcDkwmmY1gXkXhf162QuEHxkMpZPSi7xck2eGQ6MRGKxNTeY8P1FiFTPCLA5x7qZpFx84fnnrNQFpSnUCwd1nPG8Mk9/0/*)#u49n3vuy"'", "timestamp":"now", "active": true, "range":2000}]'
Check the range again with listdescriptors:
(…)
{
      "desc": "wpkh([9a6a2580/84'/1'/180']tpubDCMRAYcH71GibuLuWcDkwmmY1gXkXhf162QuEHxkMpZPSi7xck2eGQ6MRGKxNTeY8P1FiFTPCLA5x7qZpFx84fnnrNQFpSnUCwd1nPG8Mk9/0/*)#u49n3vuy",
      "timestamp": 1666876980,
      "active": true,
      "internal": false,
      "range": [
        0,
        2000
      ],
      "next": 0
    }

Every time we call the DescriptorScriptPubKeyMan::TopUp method (which happens quite a lot internally, including whenever we add a new descriptor) it automatically adjusts the range-end to be at least as big as (next index + keypool size).
If you launch bitcoind with specifying a keypool smaller than the range you want to use (e.g. bitcoind -regtest -keypool=5), you will observe the behaviour you expected.
(Thank you for providing the steps to reproduce.)

Thanks!

Do you think the importdescriptors documentation should be improved? Your answer was not obvious to me. I wouldn't mind creating a PR clarifying this.

Is the key pool size related to address lookahead? I.e. with a range of [0, 999], will Bitcoin Core detect incoming transactions on the 1000 first addresses of that output descriptor? If so, does importing a lot of brand new descriptors warrant a smaller range to save on memory/processing, in cases where we’re in full control of the addresses shared from the various descriptors?

You're definitely welcome to open a PR or issue! I think clarifying the connection between keypool is appropriate. It's probably not super crucial because a larger range should never lead to unexpected/dangerous behaviour, but yeah I suppose (but am not sure) that when importing large amounts of descriptors this can lead to performance impact. Even if it would have just saved you a bunch of time figuring this out, it's probably worth adding to the docs imo!


text:
Connection to my full node via Python script suddenly fails
I run a mynode full node on a raspi for quite some time now and was playing around with Python scripts to connect and send txs. This worked fine until lately. Turns out the connection establish process fails because my full node refuses the connection request I send from my PC.
Apparently, he's not fine with me connecting via port 8333 which I always did in the past. Why would the request be rejected? Both, full node and PC are in the same VLAN.
Below a photo link from wireshark to a connection request with port 8333 (fails) and a request with port 8332 (no issues).
https://postimg.cc/gallery/7ZwCxDH (server responds to TCP handshake with a TCP reset (RST))
The relevant code block comes down to:
    import socket
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    HOST = "192.168.30.87"
    PORT = 8333
    s.connect((HOST, PORT))

The full node is fully synced with plenty of connections (Tor and Non-Tor).
Below the result from netstat -tulp (updated with admin rights to show the program using the socket):
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:amanda          0.0.0.0:*               LISTEN      25329/lnd
tcp        0      0 localhost:xtell         0.0.0.0:*               LISTEN      5398/electrs
tcp        0      0 0.0.0.0:20000           0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:9824            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:3011            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:5031            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 localhost:7656          0.0.0.0:*               LISTEN      785/i2pd
tcp        0      0 0.0.0.0:5001            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:62602           0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 localhost:6379          0.0.0.0:*               LISTEN      3559/redis-server 1
tcp        0      0 0.0.0.0:8011            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:28332           0.0.0.0:*               LISTEN      3846/bitcoind
tcp        0      0 0.0.0.0:8332            0.0.0.0:*               LISTEN      3846/bitcoind
tcp        0      0 localhost:6668          0.0.0.0:*               LISTEN      785/i2pd
tcp        0      0 172.17.0.1:8333         0.0.0.0:*               LISTEN      3846/bitcoind
tcp        0      0 localhost:8333          0.0.0.0:*               LISTEN      3846/bitcoind
tcp        0      0 0.0.0.0:28333           0.0.0.0:*               LISTEN      3846/bitcoind
tcp        0      0 0.0.0.0:8013            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 localhost:8334          0.0.0.0:*               LISTEN      3846/bitcoind
tcp        0      0 0.0.0.0:2222            0.0.0.0:*               LISTEN      5037/docker-proxy
tcp        0      0 0.0.0.0:28334           0.0.0.0:*               LISTEN      3846/bitcoind
tcp        0      0 0.0.0.0:8335            0.0.0.0:*               LISTEN      7928/node
tcp        0      0 0.0.0.0:2223            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:8015            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:4080            0.0.0.0:*               LISTEN      7295/docker-proxy
tcp        0      0 0.0.0.0:http            0.0.0.0:*               LISTEN      729/python3
tcp        0      0 0.0.0.0:9136            0.0.0.0:*               LISTEN      785/i2pd
tcp        0      0 0.0.0.0:50001           0.0.0.0:*               LISTEN      5398/electrs
tcp        0      0 0.0.0.0:2225            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:8017            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:4081            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:49393           0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:50002           0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:5011            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:8019            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN      759/sshd
tcp        0      0 0.0.0.0:3031            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:61208           0.0.0.0:*               LISTEN      733/python3.8
tcp        0      0 0.0.0.0:10009           0.0.0.0:*               LISTEN      25329/lnd
tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:3001            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:61209           0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:3002            0.0.0.0:*               LISTEN      8432/node
tcp        0      0 0.0.0.0:8890            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 localhost:9050          0.0.0.0:*               LISTEN      3178/tor
tcp        0      0 0.0.0.0:https           0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 0.0.0.0:3003            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 localhost:9051          0.0.0.0:*               LISTEN      3178/tor
tcp        0      0 localhost:4444          0.0.0.0:*               LISTEN      785/i2pd
tcp        0      0 0.0.0.0:5021            0.0.0.0:*               LISTEN      3341/nginx: master
tcp        0      0 localhost:7070          0.0.0.0:*               LISTEN      785/i2pd
tcp        0      0 localhost:4447          0.0.0.0:*               LISTEN      785/i2pd
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN      759/sshd
udp        0      0 0.0.0.0:51194           0.0.0.0:*                           1920/openvpn
udp        0      0 0.0.0.0:bootpc          0.0.0.0:*                           1516/dhclient
udp        0      0 0.0.0.0:9136            0.0.0.0:*                           785/i2pd
udp        0      0 0.0.0.0:mdns            0.0.0.0:*                           595/avahi-daemon: r
udp        0      0 localhost:7655          0.0.0.0:*                           785/i2pd
udp        0      0 0.0.0.0:42496           0.0.0.0:*                           595/avahi-daemon: r
udp6       0      0 [::]:44900              [::]:*                              595/avahi-daemon: r
udp6       0      0 [::]:mdns               [::]:*                              595/avahi-daemon: r

I only see localhost and 172.17.0.1 with port 8333, so thats odd.
In the bitcoin.conf I explicitly added the IP of my client PC to the whitelist (192.168.30.156)
# bitcoin configuration
# /home/bitcoin/.bitcoin/bitcoin.conf

# Bitcoin options
server=1
daemon=1

# Bind to localhost and docker
bind=127.0.0.1
bind=172.17.0.1

# Needed for BTC-RPC-Explorer
txindex=1

# CPU count
par=-1

# Debug log settings
shrinkdebugfile=1

# Connection settings
rpcauth=mynode:ac633d1117f36eb0d44651f76e2f4b13$ccb595afbb45b19718ae3870715f39f8dc7dbcaa137c5ffd969f47e47fbade13

rpcport=8332
rpcbind=0.0.0.0
rpcallowip=127.0.0.1
rpcallowip=10.0.0.0/8
rpcallowip=172.16.0.0/12
rpcallowip=192.168.0.0/16

# Set default wallet so some wallet APIs work
main.wallet=wallet.dat
main.wallet=joinmarket_wallet.dat

test.wallet=wallet.dat
test.wallet=joinmarket_wallet.dat

# Enable Deprecated RPCs
deprecatedrpc=addresses

# Whitelist local connections (and docker)
whitelist=127.0.0.1
whitelist=192.168.30.156
whitelist=172.16.0.0/12

# Dont limit data to electrum server or local network
whitelist=download@10.0.0.0/8
whitelist=download@192.168.0.0/16

# Enable Bloom filters (local network)
whitelist=bloomfilter@10.0.0.0/8
whitelist=bloomfilter@192.168.0.0/16

zmqpubrawblock=tcp://0.0.0.0:28332
zmqpubrawtx=tcp://0.0.0.0:28333
zmqpubhashblock=tcp://0.0.0.0:28334

# myNode Optimizations
dbcache=1000
maxorphantx=10
maxmempool=250
maxconnections=40
maxuploadtarget=1000
rpcworkqueue=32

# IPv4 Disabled
dnsseed=0
dns=0

# Tor config
proxy=127.0.0.1:9050
listen=1
#onlynet=onion

Update:
So I checked with nmap and the port is closed:
admin@myNode:~ $ nmap 192.168.30.87 -p 8333
Starting Nmap 7.70 ( https://nmap.org ) at 2022-11-24 01:42 CST
Nmap scan report for 192.168.30.87
Host is up (0.0090s latency).

PORT     STATE  SERVICE
8333/tcp closed bitcoin

However using the ufw command it returns the port is already open. Any hints?
Update: below the ifconfig result:
admin@myNode:~ $ ifconfig -a
br-0b53e988a5ad: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.28.0.1  netmask 255.255.0.0  broadcast 172.28.255.255
        ether 02:42:a4:07:0d:c1  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

br-64d06c0adaf9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.26.0.1  netmask 255.255.255.0  broadcast 172.26.0.255
        ether 02:42:f3:41:f8:72  txqueuelen 0  (Ethernet)
        RX packets 8998434  bytes 819555438 (781.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9009649  bytes 36218286954 (33.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

br-68444510781f: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.30.0.1  netmask 255.255.0.0  broadcast 172.30.255.255
        ether 02:42:6e:7b:52:df  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

br-85aa3cb4cfcd: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.29.0.1  netmask 255.255.0.0  broadcast 172.29.255.255
        ether 02:42:22:1b:53:1a  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

br-f1547965b042: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.18.0.1  netmask 255.255.0.0  broadcast 172.18.255.255
        ether 02:42:77:f6:03:a1  txqueuelen 0  (Ethernet)
        RX packets 1334303  bytes 101434833 (96.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2113575  bytes 1022268903 (974.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:f9:d8:80:7b  txqueuelen 0  (Ethernet)
        RX packets 5154  bytes 732024 (714.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6071  bytes 2700322 (2.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.30.87  netmask 255.255.255.0  broadcast 192.168.30.255
        inet6 fe80::4690:abb5:3d80:5b6b  prefixlen 64  scopeid 0x20<link>
        ether dc:a6:32:77:11:60  txqueuelen 1000  (Ethernet)
        RX packets 7749728  bytes 3536205167 (3.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9660173  bytes 8853888597 (8.2 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 68625988  bytes 17225478612 (16.0 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 68625988  bytes 17225478612 (16.0 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet 10.8.0.1  netmask 255.255.255.0  destination 10.8.0.1
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 100  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth1693a12: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 76:3b:3f:ab:62:62  txqueuelen 0  (Ethernet)
        RX packets 1024704  bytes 109773777 (104.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1103554  bytes 1164128434 (1.0 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth2066f12: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 66:21:35:37:84:83  txqueuelen 0  (Ethernet)
        RX packets 13803  bytes 11002093 (10.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14065  bytes 5219579 (4.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth7961b83: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 8a:c8:c4:b0:d3:85  txqueuelen 0  (Ethernet)
        RX packets 2437851  bytes 1284242989 (1.1 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3138277  bytes 1132042455 (1.0 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth930bbd0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 72:86:21:2b:cb:75  txqueuelen 0  (Ethernet)
        RX packets 9004689  bytes 950162045 (906.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9015339  bytes 36228912911 (33.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethc1bb991: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 5e:a3:21:cc:f6:a8  txqueuelen 0  (Ethernet)
        RX packets 36284  bytes 3813244 (3.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 36602  bytes 3599357 (3.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethdff0d05: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 4e:1e:fe:e0:93:e2  txqueuelen 0  (Ethernet)
        RX packets 5154  bytes 804180 (785.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6071  bytes 2700322 (2.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlan0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether dc:a6:32:77:11:63  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Update: I checked the firewall rules:
admin@myNode:~ $ sudo ufw status numbered
Status: active

     To                         Action      From
     --                         ------      ----
[ 1] 8889                       ALLOW IN    Anywhere                   # allow lndg HTTP
[ 2] 25441                      ALLOW IN    Anywhere                   # allow Specter Desktop
[ 3] 8890                       ALLOW IN    Anywhere                   # allow lndg HTTPS
[ 4] 49392                      ALLOW IN    Anywhere                   # allow BTCPay Server-direct
[ 5] 2224                       ALLOW IN    Anywhere                   # allow wetty HTTP
[ 6] 49393                      ALLOW IN    Anywhere                   # allow BTCPay Server-direct HTTPS
[ 7] 2225                       ALLOW IN    Anywhere                   # allow wetty HTTPS
[ 8] 51194                      ALLOW IN    Anywhere                   # allow VPN
[ 9] 61208                      ALLOW IN    Anywhere                   # allow Glances
[10] 61209                      ALLOW IN    Anywhere                   # allow Glances HTTPS
[11] Anywhere                   ALLOW IN    127.0.0.1                  # allow from localhost
[12] 19999                      ALLOW IN    Anywhere                   # allow netdata HTTP
[13] 20000                      ALLOW IN    Anywhere                   # allow netdata HTTPS
[14] 22                         ALLOW IN    Anywhere                   # allow SSH
[15] 80                         ALLOW IN    Anywhere                   # allow WWW
[16] 443                        ALLOW IN    Anywhere                   # allow Secure WWW
[17] 1900                       ALLOW IN    Anywhere                   # allow SSDP for UPnP discovery
[18] Anywhere                   ALLOW IN    10.0.0.0/8 1900            # allow UPnP from router
[19] Anywhere                   ALLOW IN    192.168.0.0/16 1900        # allow UPnP from router
[20] Anywhere                   ALLOW IN    172.16.0.0/12 1900         # allow UPnP from router
[21] 9911                       ALLOW IN    Anywhere                   # allow Lightning Watchtower
[22] 10009                      ALLOW IN    Anywhere                   # allow Lightning gRPC
[23] 10080                      ALLOW IN    Anywhere                   # allow Lightning REST RPC
[24] 9735                       ALLOW IN    Anywhere                   # allow Lightning
[25] 8332                       ALLOW IN    Anywhere                   # allow Bitcoin RPC - filtered by rpcallowip
[26] 8333                       ALLOW IN    Anywhere
[27] 18333                      ALLOW IN    Anywhere                   # allow Bitcoin testnet
[28] 28332                      ALLOW IN    172.17.0.0/16              # allow Dojo zmqrawblock
[29] 28332                      ALLOW IN    172.28.0.0/16              # allow Dojo zmqrawblock
[30] 28333                      ALLOW IN    172.17.0.0/16              # allow Dojo zmqrawtx
[31] 28333                      ALLOW IN    172.28.0.0/16              # allow Dojo zmqrawtx
[32] 28334                      ALLOW IN    172.17.0.0/16              # allow Dojo zmqhashblock
[33] 28334                      ALLOW IN    172.28.0.0/16              # allow Dojo zmqhashblock
[34] 8335                       ALLOW IN    Anywhere                   # allow corsproxy for btc rpc
[35] 8443                       ALLOW IN    Anywhere                   # allow Lightning Terminal
[36] 2222                       ALLOW IN    Anywhere                   # allow WebSSH2
[37] 2223                       ALLOW IN    Anywhere                   # allow WebSSH2 HTTPS
[38] 3000                       ALLOW IN    Anywhere                   # allow LndHub
[39] 3001                       ALLOW IN    Anywhere                   # allow LndHub HTTPS
[40] 3002                       ALLOW IN    Anywhere                   # allow BTC RPC Explorer
[41] 3003                       ALLOW IN    Anywhere                   # allow BTC RPC Explorer HTTPS
[42] 3010                       ALLOW IN    Anywhere                   # allow RTL
[43] 3011                       ALLOW IN    Anywhere                   # allow RTL HTTPS
[44] 3020                       ALLOW IN    Anywhere                   # allow Caravan
[45] 3030                       ALLOW IN    Anywhere                   # allow Thunderhub
[46] 3031                       ALLOW IN    Anywhere                   # allow Thunderhub HTTPS
[47] 3493                       ALLOW IN    Anywhere                   # allow Network UPS Tools
[48] 4080                       ALLOW IN    Anywhere                   # allow Mempool
[49] 4081                       ALLOW IN    Anywhere                   # allow Mempool HTTPS
[50] 5000                       ALLOW IN    Anywhere                   # allow LNBits
[51] 5001                       ALLOW IN    Anywhere                   # allow LNBits HTTPS
[52] 5010                       ALLOW IN    Anywhere                   # allow Warden Terminal
[53] 5011                       ALLOW IN    Anywhere                   # allow Warden Terminal HTTPS
[54] 5353                       ALLOW IN    Anywhere                   # allow Avahi
[55] 8010:8019/tcp              ALLOW IN    Anywhere                   # allow USB Extras HTTP/HTTPS
[56] 8899                       ALLOW IN    Anywhere                   # allow Whirlpool
[57] 9823                       ALLOW IN    Anywhere                   # allow CKBunker
[58] 9824                       ALLOW IN    Anywhere                   # allow CKBunker HTTPS
[59] 50001                      ALLOW IN    Anywhere                   # allow Electrum Server
[60] 50002                      ALLOW IN    Anywhere                   # allow Electrum Server
[61] 53001                      ALLOW IN    Anywhere                   # allow Sphinx Relay
[62] 56881                      ALLOW IN    Anywhere                   # allow myNode QuickSync
[63] 51413                      ALLOW IN    Anywhere                   # allow myNode QuickSync
[64] 6771                       ALLOW IN    Anywhere                   # allow myNode QuickSync (LPD)
[65] 2189                       ALLOW IN    Anywhere                   # allow UPnP
[66] 5351                       ALLOW IN    Anywhere                   # allow NAT-PMP
[67] 5020                       ALLOW IN    Anywhere                   # allow jam HTTP
[68] 5021                       ALLOW IN    Anywhere                   # allow jam HTTPS
[69] 5030                       ALLOW IN    Anywhere                   # allow lilywallet HTTP
[70] 5031                       ALLOW IN    Anywhere                   # allow lilywallet HTTPS
[71] 8088                       ALLOW IN    Anywhere                   # allow tailscale HTTP
[72] 8089                       ALLOW IN    Anywhere                   # allow tailscale HTTPS
[73] 62601                      ALLOW IN    Anywhere                   # allow JoinMarket Orderbook
[74] 62602                      ALLOW IN    Anywhere                   # allow JoinMarket Orderbook HTTPS
[75] 27183                      ALLOW IN    Anywhere                   # allow JoinMarket API
[76] 28183                      ALLOW IN    Anywhere                   # allow JoinMarket API

There's the port 8333 but it doesn't seem to be associated with Bitcoin. I assume it's listed because I opend it manually. So is the full node not using port 8333 at all?

Are you sure the Raspberry Pi  node is configured to listen on 8333? What is in it's bitcoin.conf or equivalent? What shows up in netstat -tulp on the Pi?

Thanks, I added the results to the post. I guess I miss a 0.0.0.0:8333 ?

So I checked with nmap and the port is closed:
`admin@myNode:~ $ nmap 192.168.30.87 -p 8333
Starting Nmap 7.70 ( https://nmap.org ) at 2022-11-24 01:42 CST
Nmap scan report for 192.168.30.87
Host is up (0.0090s latency).
PORT     STATE  SERVICE
8333/tcp closed bitcoinUsing theufw` command it returns the port is already open. Any hints?

It looks like the pi either has multiple Ethernet interfaces or, more likely, IP aliases - it has two external IP addresses and the bitcoin service is only bound to the 172 address not the 192 address.
The command `ifconfig -a might clarify this.
I also think the netmask in your rpcallowip=172.16.0.0/12 might not allow for 172.17 as that differs in bit 16:which is outside the 12 bit range specified?
The p option on the netstat command should show the process name but maybe that depends on being run with root permissions.
I believe restarting Bitcoind after sorting out the IP addressing in the pi, or at least  configuring Bitcoind to bind to the 192 address, should resolve the problem.
Try adding
bind=192.168.30.87

To your bitcoin.conf and restart Bitcoind.

See attached the ifconfig results. Not sure if that supports your assumption?

Regarding the rpcallowip=172.16.0.0/12 you're probably right, however shouldn't affect the node access via port 8333? (rpc is port 8332)
The updated netstat -tulp shows that bitcoind does not use a socket with the 192 address.
I clearly lack understanding, but for me that would have been the socket to communicate with all the other ip4 nodes I'm connected to....apparently the full node does that differently.

Try adding the bind directive I added to my answer. I assume you know how to stop and restart bitcoind

Thanks, adding the bind directive seems to be sufficent, I can now connect via 192.168.30.87:8333
You see any advantage to restart bitcoind? Does it affect the snycd blockchain?


text:
Analyze Failed Routing in Core-Lightning
In my node I see many failed Routing Transactions. I did not really find information on how to analyze what went wrong.
First I see

Failed Transactions
Local Failed Transactions - reason always "Temporary Channel Failure"

What is the difference between them? I guess that "local failed" means that my node was not able to forward them? What are the reasons? Referring to lnbook:

Meanwhile if we get a "temporary channel failure" error, the most likely reason is a lack of liquidity for the specific amount.

But there are also many which failed where I had enough liquidity.
For "Failed transactions" I can find no reason. I guess this are transactions nodes behind my node can not route forward?

If a transaction is routed from A to C over B where I am B, my liquidity on my side is relevant in the B-C channel. If A to B liquidity is not enough I would conclude that the transaction is not even reaching my node. So I thought it should fail because of my liquidity in B-C.

You are searching for "Relay"? Bank is not really on point. A Bank guarantees you a balance of money they do not have. A Lightning relay can forward liquidity which is in a channel.

this is difficult currently but not impossible, simple core lightning should return more information when it fails a payment

Okay, I will try to expand on my comment in an answer. Currently, it's difficult to determine why a payment is failing with our core lightning node because it doesn't provide much information.
However, in the last year, I worked on a lightning network metrics specification, which is available here or on Github. Using this specification, you can try to find the channel through which the payment is being sent and potentially gather useful metrics or ideas to improve the current system.
Regarding your question, there are several well-known problems that we're trying to work around or resolve.
You were correct in assuming that "local failed" means that your node could not forward the payment.
Regarding the general failure, this is not your fault, but one of the nodes along the path that fails;
This can be caused by several factors, such as:

the sender selected a channel in the route path that isn't suitable for sharing the payment (note that the sender chooses the route and not the current node), and due to the unknown real capacity of the channel, the sender path is just a guess. Currently, this unknown real capacity of the channel is considered a privacy feature;
the node that you have a channel with being offline (which can be common if the node uses a hidden network, aka tor),
the sender sending a payment that has already dead from the source (also known as a probing attack, which aims to steal information from the network graph regarding which channel can forward the payment), or
a bug in the software.

In some cases, like with a probing attack, you can try to mitigate the issue by using lnmetrics to look at the channel that has only been sending failed payments in the last day or 10 days. With lnmetrics, you can analyze the local failure and try to understand what went wrong with your node. Additionally, you can check the node uptime of the node that you have a channel with to get a better understanding of what the reason is.
However, this info does not provide any final answer, but sometimes removing a bad node can help. Ideally, this analysis with lnmetrics should be automated, but you can do it manually for now.

For "Failed transactions" I can find no reason. I guess these are transactions nodes behind my node can not route forward?

Mh, currently, I cannot find where we return this error message, you can add the correct error here that you find in your core lightning?
P.S: I noted that you pointed out several times the word transaction, as the HTLC is a pre-signed transaction, in lightning this is called payment, but maybe I am missing something here, so I hope you can clarify

Do you know if other Lightning implementations currently provide more info than CLN on failed payments?

No I am not in sync with others, sorry


text:
Lightning node with two channels to peer not able to select funded channel
Description
I have two peers: p1 and p2 running lightning-core
p1 opened a channel to p2 and p2 opened another channel to p1.
Now p2 has two channels. One with outbound funding and one with 0. Output of listfunds:
"channels": [
      {
         "peer_id": "02e7b2196a87513ef322fc2c23d5739ed4ac696a9010a855dbfe7185fe8e133c93",
         "connected": true,
         "state": "CHANNELD_NORMAL",
         "short_channel_id": "2410508x70x0",
         "channel_sat": 0,
         "our_amount_msat": "0msat",
         "channel_total_sat": 113495,
         "amount_msat": "113495000msat",
         "funding_txid": "1f5284dbfcaf76e8a333c1c607465a9474294f43cfbcbbb34504d97aac2189a4",
         "funding_output": 0
      },
      {
         "peer_id": "02e7b2196a87513ef322fc2c23d5739ed4ac696a9010a855dbfe7185fe8e133c93",
         "connected": true,
         "state": "CHANNELD_NORMAL",
         "short_channel_id": "2410509x222x0",
         "channel_sat": 90433,
         "our_amount_msat": "90433000msat",
         "channel_total_sat": 90433,
         "amount_msat": "90433000msat",
         "funding_txid": "befde7cbd4965f21fe9df3a1f06eb1968109a7246457aabed58bfe39f6694b08",
         "funding_output": 0
      }
   ]

Problem
Now I tried to pay a 1000msat p1 invoice from p2 and get the error:

failed: WIRE_TEMPORARY_CHANNEL_FAILURE (WIRE_TEMPORARY_CHANNEL_FAILURE: Capacity exceeded - HTLC fee: 183sat)

I think this is because p2 tries to send using the not funded channel 2410508x70x0. Because if I delete this channel, all works as expected.
Additional information:
Also getroute shows that channel 2410508x70x0 is used:
$ lightning-cli --testnet getroute 02e7b2196a87513ef322fc2c23d5739ed4ac696a9010a855dbfe7185fe8e133c93 1000 0
{
   "route": [
      {
         "id": "02e7b2196a87513ef322fc2c23d5739ed4ac696a9010a855dbfe7185fe8e133c93",
         "channel": "2410508x70x0",
         "direction": 1,
         "msatoshi": 1000,
         "amount_msat": "1000msat",
         "delay": 9,
         "style": "tlv"
      }
   ]
}

Questions

Why does p2 does not auto select the funded channel?
Is there a way to select the channel? I did not find an option for pay.

Is it possible that https://blockstream.info/testnet/tx/befde7cbd4965f21fe9df3a1f06eb1968109a7246457aabed58bfe39f6694b08 didn't have three confirmations yet, when you tried to make that payment?—Does it work meanwhile?

Tried once again with the same channel, still the same error. (the one I deleted was from a previous test).

Please look at additional information. Also, getroute shows that the wrong channel is used.

Why does p2 not auto-select the funded channel?

In this case, I guess that the pay plugin is smart enough to select the peer with the correct channel, or better is not able to choose the right channel of the peer. I guess this is a limitation of the pay plugin, also if I do not remember the code, maybe if you open an issue we can take a look at what is going on in your case!

Is there a way to select the channel? I did not find an option for payment.

Yes you can use the sendpay command that allows you to specify the route, this is a low-level command but it is a good way to have the power to choose the route


text:
Lightning routing fees received between 2 channels with 0 feerate?
Something strange just occurred on my LND node, making me think I don't understand something about lightning fees.
I have channels to A & B, who only ever asked me to route transactions going to C.
Based on the assumption that the fee is calculated from the fee rates of the outbound channel of a route, I figured I could set the fees of the channels to A & B to 0 (base + feerate) since they are never the outbound channel, so I never earn fees through them, so it incentivizes routes in the other way. Not earning me fees, but rebalancing the channels for free.
Minutes later, I got 4 routes between A to B. But to my surprise, those routes supposedly earnt me fees, according to RTL, corresponding to the feerate before i set it to 0.
How can this happen if I have set 0 fees for both channels? Could it be some kind of update issue, where the rest of the network thinks the channels are now free, but the updated fee is not yet applied? Or something else I don't understand?

There is a feerate relative to the amount sent and there is a flat base fee. Could you clarify whether you have set both to zero, or just one?

Changing the fee on existing channels may take time to propagate. As the payments you mentioned happened soon ('minutes later'), is it possible that the sending A node acted still with the old fee setting? Afterall it's the sender who specifies all the fees for all the routing nodes upfront.

Most probable explanation for now : the channels who never routed betwen them before happened to route just after the change, but the sender did not know about the fee change yet


text:
ThreadOpenConnections and break versus continue: why the one versus the other?
In the inner loop of ThreadOpenConnections() in net.cpp sometimes there is a break to the outer loop and sometimes a continue. This is my understanding of the decision between break and continue in this part of the code:

A break fully 'restarts' the search for an outward connection, the type of which will be determined again. Also new incoming potential peers will be included in the addrman tables.
A continue effectivily just asks addrman for a next candidate peer from the existing tables. The type of outward connection that we wish for remains the same.

My question is about the choice of break versus continue in two cases. Here the code does a break:
// if we selected an invalid or local address, restart
if (!addr.IsValid() || IsLocal(addr)) {
    break;
}

While a few lines below there is a continue:
if (!IsReachable(addr))
    continue;

My question is: why 'restart' in the first code snippet and not continue? In other words: why not ask addrman a few times more, as it might supply a valid address soon? Or, for the second snippet, if we got an unreachable address: why not restart, so addrman might get refreshed with more reachable addresses?
I tried to find the answer by tracking back the history of this part of the code. The break in the first code snippet above appears to stem from the introduction of the address manager (pr/787). It did not seem to answer my question.

// if we selected an invalid or local address, restart
if (!addr.IsValid() || IsLocal(addr)) {
    break;
}

This looks like some kind of a belt-and-suspenders check which shouldn't happen under normal circumstances, because we shouldn't have added invalid or local addresses to AddrMan in the first place. I'm not sure why a break was chosen here - a continue here would probably be fine as well, it probably doesn't matter much.
if (!IsReachable(addr))
    continue;

This, on the other hand, is something that could happen frequently, because IsReachable is dynamic, it depends on user configuration - we may have run on Tor in the past and deactivated with the next restart, so that all Tor addresses that were previously reachable are now unreachable.
Using continue instead of break in this case seems important to me: Encountering an unreachable address is no reason to skip the inner loop and have a 500ms sleep in the outer loop - we'd like to continue immediately without a sleep.

why not restart, so addrman might get refreshed with more reachable
addresses?

The process of AddrMan getting more addresses is independent of the ThreadOpenConnection logic - this happens in another area of the code (net_processing) and another Thread - whenever we receive addr messages from our peers, they are added to addrman.


text:
What does it mean for an integer to be a member of secp256k1 group?
I'm reading the spending section of BIP47:

Alice calculates a scalar shared secret using the x value of S:
s = SHA256(Sx).
If the value of s is not in the secp256k1 group, Alice MUST increment the index used to derive Bob's public key and try again.

Where S is a point on the elliptic curve and s is an integer to be used as a private key in subsequent sections.
My question is

What does it mean for the integer s to be in a group of points?

I believe it simply means: if s, when interpreted as an integer, is larger or equal to the curve order, one needs to restart and increment. The probability for this happening is negligible, as the curve order for secp256k1 is very close to 2256.

Thanks, it makes sense now. Is there any particular reason that we don't do a mod n in case the integer s happens to be greater than n? (n being the order of the group)

I suspect the rationale is avoiding a bias towards the numbers in range 0...(2^256-N); the authors may have taken this from BIP32 which has a similar rule (which I'm the author of). In retrospect, this is dumb, because the number is so close to 2^256 that hitting a number in that range at all (much less resulting in an observable bias) only has a negligible probability.

clearly explained and sorry, my reputation is too low to cast a +1 vote to the answer.

One last question. I was viewing the source code for the two famous implementations of this standard, and I came across something I cannot understand; they are both comparing the bitlength of s to the bitlength of n (instead of comparing the numerical value). Do you have any comments on that?
Impl 1
and,
Impl 2

That makes no sense to me; if the integer is the result of SHA256 output, it'll never exceed 256 bits. But again, all of this is unobservable anyway.

For any curve over any field, algebraic geometers are interested in an associated group called the Picard group. It is a certain quotient of the free abelian group on points of the curve. It consists of formal sums of points on the curve modulo those formal sums that come from looking at the zeroes and poles of rational functions. It is a very important tool in the study of algebraic curves.
Why is an elliptic curve a group?

The smaller p2pkh addresses in Bitcoin are derived from a larger public key. This key is made from a scalar private key, the public key is basically an x and y coordinate on the secp256k1 elliptic curve derived from the scalar private key. If the private key is not within the curve group you cannot derive a valid x and y coordinate aka public key from it.
In the case you presented s, the scalar shared secret, needs to be within the elliptic curve group because they are deriving a public key that someone could claim funds with from s.
Deeper explanation: What is the math behind Bitcoin's elliptic curve?
Edit: The scalar and the curve order are integers, their elements are points.

Thank you for the answer.
"If the private key is not within the curve group you cannot derive a valid x and y coordinate aka public key from it."
I understand that secp256k1 forms a group. But the elements of this group are a set of planar points: (x,y). The private key s is a scalar/integer. What I do not understand is that how could an integer be part of a set whose elements are not integers but rather a bunch of ordered pairs (x,y).

This basically explains it, the ordered pairs are derived from the scalar through this complex elliptic curve math. I can't really go much deeper into it as I am not an expert.

to find the public key Q corresponding to a private key k, you have to "walk" k generator steps from the base point:
Q = G+G+…+G [k-times] = kG
Now secp256k1 is defined as
y² = x³+7 mod P
where P is
P = 2256 - 232 - 29 - 28 - 27 - 26 - 24 - 1
In secp256k1, the base point is G(Gx,Gy), where
Gx = 550662...
Gy = 326705...
The addition of two points is defined as
P + Q = R
(xp, yp) + (xq, yq) = (xr, yr)
Which works out to be
xr = (yq - yp) / (xq - xp) - xp - xq
yr = ((yq - yp) / (xq - xp)) × (xp - xr) - yp

Taken from https://bitcoin.stackexchange.com/questions/102940/what-is-the-math-behind-bitcoins-elliptic-curve

Derivation of a public key from a private key is rather clear to me. The unclear part is what I quoted from BIP47 document: "If the value of s is not in the secp256k1 group". It's like saying if the number 2 is not in the set {(1,2),(4,5),...}. How do I even check that? How do you compare an integer to a pair?

So as the expert has pointed out above if s, when interpreted as an integer, is larger or equal to the curve order, one needs to restart and increment. So effectively the curve order is representable as an integer and s is interpreted as an integer, that means it can also be represented as a curve point as well.

The curve order (= the number of points on the curve) is an integer. The elements of the curve are not integers.

Right so the curve order and the scalar secret are integers and elements are points, which means that the points are derived from the integers. I'm sorry if I made it confusing I'm obviously still learning myself.

Every integer can be converted to a group element by multiplication with the generator. The language used is confusing, because there is no "in" the group or not, integers are integers and not group elements, and every integer can be converted to one (it loops around every (group order) steps, because cyclic group). The rule isn't there to make sure you end up in the group (that's always the case); it's there to prevent some group elements from being reached more frequently than others (but as I explain in my answer, this effect is so negligible it's also not actually worth caring about).

Thanks for the clarification!


text:
What happens if two miners have a few identical transactions in their block when one of them validates his block and appends it to the blockchain?
Let's assume 2 miners, A and B, are adding / validating transactions and build a block. Both blocks contain 50 identical transactions. All other transactions differ. Now one miner, B, solves the mathematical puzzle and appends his block to the chain. What would then happen to the remaining transactions within the other block of miner A? Will these be automatically removed from the block that didn't make it and the miner needs to add 50 different ones? Or does miner A needs to start all over from zero? How's that solved and does this actually happen in reality?

Let's assume 2 miners, A and B, are adding / validating transactions and build a block. Both blocks contain 50 identical transactions. All other transactions differ. Now one miner, B, solves the mathematical puzzle and appends his block to the chain. What would then happen to the remaining transactions within the other block of miner A?

When miner A receives the block constructed by B, A will remove the transactions in its mempool that conflict with that block, which in your case includes at least the 50 common transactions.
Subsequent block candidates A tries to construct will be drawn from that mempool, which has been updated to not include the conflicts anymore.

Will these be automatically removed from the block that didn't make it and the miner needs to add 50 different ones?

Blocks once produced cannot be changed, but A in your scenario never constructed a block. A has been constructing block candidates (which don't satisfy proof of work) based on its mempool, and the mempool changed, so future candidates will reflect that change.

Or does miner A needs to start all over from zero?

Miner A is already, continuously, "starting over" all the time. It is entirely irrelevant for the miners in A whether the mempool the transactions are drawn from, or the block it is building on, changed in between two attempts. It's just a new attempt like any other.
It's important to see that mining is progress free. There is not a single special block that all miners are competing to find first. There are an infinite number of potentially valid solutions, and miners are all looking for different ones. Mining proceeds by constructing candidate blocks, computing their hash, and if the hash does not have enough zeroes, continue with another candidate block. Each attempt has the same probability of success, independent of how many candidates the miner has already tried, and independent of how many candidates anyone else has already tried.

How's that solved and does this actually happen in reality?

Yes, it happens all the time. Every time someone finds a block, all other miners will update their mempools so their future block candidates reflect the state after the found block.

What would then happen to the remaining transactions within the other block of miner A?

Miner A would need to construct a new block which doesn't include the transactions that have already been mined and "solve the mathematical puzzle" for the new block.

Will these be automatically removed from the block that didn't make it and the miner needs to add 50 different ones?

Their mining software should construct a new block that doesn't include transactions from the now mined block. (If it doesn't that could be a costly mistake as it could successfully mine a block that the rest of the network rejects.) It doesn't need to add exactly 50 new ones and in some cases won't be able to if some of the new transactions are particularly big.

Or does miner A needs to start all over from zero? How's that solved and does this actually happen in reality?

Effectively miner A needs to start from zero, yes. It doesn't need to re-verify transactions it has already verified but in terms of the mining aspect it starts the block construction and mining process from scratch again. It happens regularly in reality as there will be a large overlap between the transactions in the various blocks that miners are competing to mine.


text:
What are prerequisites and best online resources to learn everything about Lightning Network
I want to learn Lightning network from basic to Advance from scratch, So what are:
1. Prerequisites to learn lightning network
2. What are best online resources to learn Lightning network from scratch

When I started to learn about the Lightning Network I started pretty much from 0. In particular I had pretty much no idea how Bitcoin worked. So I will argue it is possible to learn about bitcoin on the way and you can do it given a computer science or math or physics or engineering background.
As for the resources:

Mastering the Lightning Network is available to read online
There is a 300 pages slide deck on which Mastering the Lightning Network heavily builds upon. A presentation of the slide deck has been recorded in 3 videos with a total runtime of 4.5 hours. Video 1, Video 2, and video 3
Chaincode labs made a seminar series together with recorded videos. Though I'd argue that the discussed protocoal stack has been superceeded by the one presented in mastering the Lightning network
On my youtube channel you find a lot of technical videos about the Lightning Network
If you want to go really deep it makes sense to look at the bolts repository where the Lightning Network Protocol is being defined.
Similarly you can read through the lightning-dev mailinglist.
The Bitcoin Optech Newsletter also has a regular section about the recent progress of the Lightning Network.
On my way I learned everything that I needed to know about bitcoin from the Bitcoin wiki
Nowadays there are many podcasts that are mainly focused around the Lightning Network. eg: connect the world, lightning junkies or kevin rooke.

Last but not least my - not so secret - pro trick: Once I decided to regularly answer questions about the Lightning Network here on stack exchange I was forced to look up stuff in the bolts. So you could certainly subscribe to the lightning-network tag here on this site or go through the questions that have been asked and even challenge or improve the answers.


text:
Why do PinSketch sketches consist of the sum of odd powers?
The following is taken from BIP-330 (aka Erlay):

A short ID sketch with capacity c consists of a sequence of c field elements. The first is the sum of all short IDs in the set, the second is the sum of the 3rd powers of all short IDs, the third is the sum of the 5th powers etc., up to the last element with is the sum of the (2c-1)th powers.

Why is it that the first c odd powers are picked instead of the first c? How is it that we can reconstruct those c elements from their first c odd power sums? It is more or less straightforward using Newton's identities if we had their first c, but it is not obvious to me how to do so with just the odd ones.

You can reconstruct the elements whenever you have 2c consecutive power sums for k=a until k=a+2c-1 (power sum Sk is x1k + x2k + ... xnk) of the elements, regardless of which a you start at.
However, since PinSketch works in a characterstic-2 field, it holds that (a + b)2 = a2 + b2 (the Freshman's dream). Because of that, it holds that S2k = Sk2, or in other words, we can compute the even power sums by squaring the half power sum. Thus, having the odd power sums between k=1 and k=2c-1 (i.e., setting a=1) suffices to reconstruct all power sums (including even ones) up to k=2c too.
There is a more elaborate description in the Miniscript code documentation.

Thanks! That's a very clever trick :-)

Just one more extra question: doesn't it suffice with the first c power sums? Given c elements x_1, ..., x_c, one can reconstruct the c elementary symmetric polynomials evaluated in x_1, ..., x_c with the first c power sums. The polynomial prod (x-xi) has those elementary symmetric polynomials as coefficients, and since it has integral simple roots those can be found by bisection.
Am I missing some detail here?

@CosmikDebris If that were true, you'd be able to communicate c elements by only sending c/2 powersums - that's information theoretically not possible, so there must be something wrong in your reasoning.

Yup, you're absolutely right. What I meant by a bisection argument above referred to checking signs and using the mean value theorem but that obviously works for polynomials over Z, not for polynomials over a finite field. Thanks again.