text:
Sign Bitcoin transaction with several inputs using bitcoinj. Broadcasting of transaction failed
Good day!
I'm working with low level API to make tx and sign it offline. But after all preparations I try to broadcast it and than i get an error from network:
16: mandatory-script-verify-flag-failed (signature must be zero for failed check(multi)sig operation)
First of all I create tx without signing inputs, and after that i'm trying to sign every input with my priv keys.
Here is raw tx before signing :
01000000028bcf82619e93eb92cc78d5bb0b7b22d6ee419820428402e721fbac1386c9673f0000000000ffffffffd50ea1cd97af5295cddcd4f0c7c0d531fed5683bde80af45ede25a0571c31add0100000000ffffffff02087f0300000000001976a9146fb643fe63bcc6c352899298d6072cc5d4178c0f88acfa6b0100000000001976a914196e59ee4c151e742af69bca308fc1b4445c577288ac00000000
Parsing tx from the https://live.blockcypher.com/btc/decodetx/  :
{
    "addresses": [
        "16Yb7QP5jVmjXvHawH2ZA66k1b46FmtmE3",
        "13KU69194fJj13G1SajUWokudsuekhT1kw",
        "1BBgML68FuyRwU8RzwYn1FvqAXDDSgjVKC"
    ],
    "block_height": -1,
    "block_index": -1,
    "confirmations": 0,
    "double_spend": false,
    "fees": 55194,
    "hash": "2010928be1525a89aae1e0ece367757f6deb3754bfc4c891ffddbad403ecc3c9",
    "inputs": [
        {
            "addresses": [
                "16Yb7QP5jVmjXvHawH2ZA66k1b46FmtmE3"
            ],
            "age": 534819,
            "output_index": 0,
            "output_value": 251628,
            "prev_hash": "3f67c98613acfb21e7028442209841eed6227b0bbbd578cc92eb939e6182cf8b",
            "script_type": "pay-to-pubkey-hash",
            "sequence": 4294967295
        },
        {
            "addresses": [
                "13KU69194fJj13G1SajUWokudsuekhT1kw"
            ],
            "age": 534818,
            "output_index": 1,
            "output_value": 125872,
            "prev_hash": "dd1ac371055ae2ed45af80de3b68d5fe31d5c0c7f0d4dccd9552af97cda10ed5",
            "script_type": "pay-to-pubkey-hash",
            "sequence": 4294967295
        }
    ],
    "outputs": [
        {
            "addresses": [
                "1BBgML68FuyRwU8RzwYn1FvqAXDDSgjVKC"
            ],
            "script": "76a9146fb643fe63bcc6c352899298d6072cc5d4178c0f88ac",
            "script_type": "pay-to-pubkey-hash",
            "value": 229128
        },
        {
            "addresses": [
                "13KU69194fJj13G1SajUWokudsuekhT1kw"
            ],
            "script": "76a914196e59ee4c151e742af69bca308fc1b4445c577288ac",
            "script_type": "pay-to-pubkey-hash",
            "value": 93178
        }
    ],
    "preference": "high",
    "received": "2018-08-05T18:04:52.877018121Z",
    "relayed_by": "54.162.141.93",
    "size": 160,
    "total": 322306,
    "ver": 1,
    "vin_sz": 2,
    "vout_sz": 2
}

Here is my code of signing every input:
for (int i = 0; i < transaction.getInputs().size(); i++) {

    TransactionInput transactionInput = transaction.getInput(i);
    String addressFromUtxo = mUTXOs.get(i).getAddress();
    byte[] privKeyBytes = getPrivKeyBitesForAddress(addressFromUtxo);
    ECKey ecKey = ECKey.fromPrivate(privKeyBytes);

    Script scriptPubKey = ScriptBuilder.createOutputScript(Address.fromBase58(params, mUTXOs.get(i).getAddress()));

    Sha256Hash hash = transaction.hashForSignature(i, scriptPubKey, Transaction.SigHash.ALL, false);
    ECKey.ECDSASignature ecSig = ecKey.sign(hash);
    TransactionSignature txSig = new TransactionSignature(ecSig, Transaction.SigHash.ALL, false);
    transactionInput.setScriptSig(ScriptBuilder.createInputScript(txSig, ecKey));
}

//serialization and broadcasting
byte[] bytesRawTransaction = transaction.bitcoinSerialize();
String rawTransaction = HEX.encode(bytesRawTransaction);
broadcastTx(rawTransaction);

And here how my raw tx looks after the signing:
01000000028bcf82619e93eb92cc78d5bb0b7b22d6ee419820428402e721fbac1386c9673f000000006a473044022054720cc14562000ed1a6fa7952d2343bef48e2fa4fd0f1c1894579026ae83692022042735beb03c4d53c8d72126e1e4e8bee7acd5530284b05d6707a4e71cc891e9a01210312e1997fc0cb4c037c94885e7e4a8d0eca9d5cff8628cc3d870e5c158c9b0901ffffffffd50ea1cd97af5295cddcd4f0c7c0d531fed5683bde80af45ede25a0571c31add010000006a47304402206a70b6dacea9e9435c21fe920471daccf2edb636592b9d9163b1f5d740e582e102203416f80e9859fcfacc9f41ea3e04774ef47c9b0b9fa5a61f2b77dffee4c2710e012103522a472dd00a929e0f0980b98eb6a0d10a6e5a6ff7d45709cdb9f022b881ca66ffffffff02087f0300000000001976a9146fb643fe63bcc6c352899298d6072cc5d4178c0f88acfa6b0100000000001976a914196e59ee4c151e742af69bca308fc1b4445c577288ac00000000
Parsing tx from the https://live.blockcypher.com/btc/decodetx/  :
{
    "addresses": [
        "16Yb7QP5jVmjXvHawH2ZA66k1b46FmtmE3",
        "13KU69194fJj13G1SajUWokudsuekhT1kw",
        "1BBgML68FuyRwU8RzwYn1FvqAXDDSgjVKC"
    ],
    "block_height": -1,
    "block_index": -1,
    "confirmations": 0,
    "double_spend": false,
    "fees": 55194,
    "hash": "26859671de7bd5f0ab758517d90dcb2ef7c8a8a331a2d4fc59c7d98e07857d09",
    "inputs": [
        {
            "addresses": [
                "16Yb7QP5jVmjXvHawH2ZA66k1b46FmtmE3"
            ],
            "age": 534819,
            "output_index": 0,
            "output_value": 251628,
            "prev_hash": "3f67c98613acfb21e7028442209841eed6227b0bbbd578cc92eb939e6182cf8b",
            "script": "473044022054720cc14562000ed1a6fa7952d2343bef48e2fa4fd0f1c1894579026ae83692022042735beb03c4d53c8d72126e1e4e8bee7acd5530284b05d6707a4e71cc891e9a01210312e1997fc0cb4c037c94885e7e4a8d0eca9d5cff8628cc3d870e5c158c9b0901",
            "script_type": "pay-to-pubkey-hash",
            "sequence": 4294967295
        },
        {
            "addresses": [
                "13KU69194fJj13G1SajUWokudsuekhT1kw"
            ],
            "age": 534818,
            "output_index": 1,
            "output_value": 125872,
            "prev_hash": "dd1ac371055ae2ed45af80de3b68d5fe31d5c0c7f0d4dccd9552af97cda10ed5",
            "script": "47304402206a70b6dacea9e9435c21fe920471daccf2edb636592b9d9163b1f5d740e582e102203416f80e9859fcfacc9f41ea3e04774ef47c9b0b9fa5a61f2b77dffee4c2710e012103522a472dd00a929e0f0980b98eb6a0d10a6e5a6ff7d45709cdb9f022b881ca66",
            "script_type": "pay-to-pubkey-hash",
            "sequence": 4294967295
        }
    ],
    "outputs": [
        {
            "addresses": [
                "1BBgML68FuyRwU8RzwYn1FvqAXDDSgjVKC"
            ],
            "script": "76a9146fb643fe63bcc6c352899298d6072cc5d4178c0f88ac",
            "script_type": "pay-to-pubkey-hash",
            "value": 229128
        },
        {
            "addresses": [
                "13KU69194fJj13G1SajUWokudsuekhT1kw"
            ],
            "script": "76a914196e59ee4c151e742af69bca308fc1b4445c577288ac",
            "script_type": "pay-to-pubkey-hash",
            "value": 93178
        }
    ],
    "preference": "high",
    "received": "2018-08-05T10:44:42.584042561Z",
    "relayed_by": "54.162.141.93",
    "size": 372,
    "total": 322306,
    "ver": 1,
    "vin_sz": 2,
    "vout_sz": 2
}

And after all above I got:
16: mandatory-script-verify-flag-failed (signature must be zero for failed check(multi)sig operation)
Please, help me, what I'm doing wrong???

That error means that the signature is invalid. Make sure that you are using the correct private keys and signing the correct thing.

I`m understand that signatures are invalid, but in what way I should to sign tx with several inputs?

Finally I found solution and what i was doing wrong. The way of signing tx is totally correct, the problem was occurred in the other thing. Before I want to broadcast tx I calculate fee per kb (not fixed one). After signing tx - size of raw tx increses and fee amount increases accordingly. After that I should provide new change for address for change (value of the second output of tx). And after that I should sign inputs one more time, because the value of the output for change was changed. Unfortunately, I didn`t do this before. Maybe it'll be useful for someone.


text:
Do Bitcoins still need users to trust each other to feel confident?
So we know that bitcoin is secure up to the point of basically proving P=NP. (Its an open question)
So bitcoin is secure. But I feel it still suffers one problem, as with all transactions(not just bitcoin). One person must either send their stake in the barter first or receive the others and be trusted to return the payment if no escrow is involved.
I am paraphrasing and hopefully not destroying the idea, but when I read the bitcoin whitepaper it seemed like it was aimed so that no escrow needs to be involved to transfer funds online.
So no escrow means we all must trust one another if we want to do transactions with one another? Because one person must receive or send the goods first, and must do so on good faith that they will receive the compensation.

Bitcoin solves many problems, but this is not one of them.

This definitely a fault of bitcoin. I made a post here and on reddit bitcoins, before. There are escrow services for bitcoins..

I will add Lohoris's comment as an answer, because it really is the answer to the question. Bitcoin does not directly try to address the issue of trust in a transaction.
Therefore, yes, some trust is needed because one party still has to be the first to hand over goods and/or cash. I just read a post on the bitcoin subreddit that relayed a story of getting ripped off after handing over BTC.
Now, Bitcoin does make some aspects that make fraud harder, most specifically a transaction log. If the recipient address is either well known, displayed on an invoice, or recorded in an email, it is trivial to prove that you sent BTC to that address. (Of course if you obfuscate the transaction through a mixer, that removes this benefit.) But, even then, that is just providing tools to law enforcement/court or some other sort of after-transaction dispute process. And, given law enforcement's lack of experience with Bitcoin, and the irreversibility of Bitcoin transactions, if you don't trust the party you are trading with then you may have to engage some sort of escrow service.
Update: Per request, this is the reddit post I mentioned.

Can you edit with a link to the post on /r/bitcoin?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
What was the first altcoin?
I'm aware that LiteCoin is (was?) the "silver" to bitcoin's "gold".
The question is:

What was the first altcoin? Was it actually LiteCoin or was there something else?

For the purposes of "alt coin"

What was the first fork of bitcoin
What was the first altcoin that more than, say, 10 people mined?

Litecoin was not the first altcoin. It actually uses ideas for altcoins that precede it but had died before it was created. This is mentioned in its announcement thread on bitcointalk.. The very first altcoin was actually Namecoin, announced in April 2011. Namecoin was actively used and mined, and can still be used today.

As described by Ava Chow, Namecoin was the first altcoin created as such. It was announced on 2011-04-18.
However, according to the topic "Testnet version history" the Testnet1 was reset after its coins started to be traded for value. Testnet2 was kicked off on 2011-02-03, which would allow a case to be made for Testnet1 to have been the first altcoin.

Namecoin, for sure, can read more about it here https://en.wikipedia.org/wiki/Namecoin

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Can transaction data be altered before creating a block?
I'm a novice and had this thought. I understand that data in an existing block can't be altered, as it changes the hash. But what if, before creating the block, I take transactions from the pool, alter one transaction, and then proceed to create the block? Let's say I have the most powerful hardware and finish it before other miners. Will other nodes be able to verify the transaction data inside the block? (example: say A sends 10 tokens to B, and I change that to 100. A has 200 tokens in his wallet)

You can modify it, but in doing so, you will invalidate the signature on the transaction, since the data is no longer the same as what was signed.
You could do so on transactions that don't involve signatures, but they are generally non-standard on the mainnet.
Note that even after modification, you can only update it within the bounds of the input(s) sum. You can never modify a transaction to have an output greater than the sum of the inputs.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
do isolated blocks create forks in block chain? (with forks I mean branches that are no longer valid) If yes then how does the node redeem itself?
I'm a newbie to cryto and wanted understand this.
what happens to the  chain when the isolated block is formed (in case of a double spend) ? Since chains only add blocks that come from the  longest running chain. it could be easily possible that initially 2 nodes proposed 2 blocks at the same time of the double spend attack. so
Node A -Block A(Double Spend)
Node B- Block B (Genuine)
Both have same length at this time but Node A decides to go rouge.
Lets consider Node C is close to Node A and hears block A. it adds it to it's block Chain. now another Node D hears about both Block B and A and decides to add A since it's coming from the longest blockchain. Now, It's a successful double spend if the customer didn't wait for any validations.
(But the money is only spend once as we can see Block B is not accepted by any other blocks)
In this case Block B is orphaned.. How does not Node B redeem itself?? How does the Node B add new chain to it's orphaned Block. I guess the orphaned one is just left there and the new Node is added from the previous block.
For easier understanding:I was considering one transaction per block. ;)

Edit: Anyone Coming to this page may find this Flow helpful:
1) New transactions are broadcast to all nodes.
2) Each node collects new transactions to form block.
3) Each node competes with one another to  solve the  mining puzzle
  (Proof of Work)for its block.
4) When a node cracks the puzzle, it broadcasts the block to all
  nodes.
5) Nodes accept the block only if all transactions in it are valid and
  not already spent.
6) Nodes express their acceptance of the block by working on creating
  the next block in the chain, using the hash of the accepted block as
  the previous hash.
Nodes always consider the longest chain to be the correct one and will
  keep working on extending it. If two nodes broadcast different
  versions of the next block simultaneously, some nodes may receive one
  or the other first. In that case, they work on the first one they
  received, but save the other branch in case it becomes longer. The tie
  will be broken when the next block is found and one branch becomes
  longer; the nodes that were working on the other branch will then
  switch to the longer one branch.
  Also to prevent double spend it creates a DB of Unspent Bitcoin and validates every transaction based on that.How is a transaction verified

If a block is mined on top of block A (let's call it block C), when Node B hears about block C, it will change its blockchain state to have block C as its tip and block B will become orphaned. This is because block C's chain has more work than block B's chain.
Seeing a double spend in blocks as you describe it does not make a block invalid. If I see a conflicting transaction in block A and block B, that conflict does not make one of those blocks invalid to me.

Thanks for the response Andrew. so the whole block will not be declared invalid. only that  transaction will be made invalid. depending on which block  gets accepted. am I correct?

Yes, that is correct.

another quick question andrew if you can spare a few minutes. Lets bring into picture orphaned blocks. what if the produced block is orphaned.. How does a Node merge with the longest chain in this case when it has a orphaned block at it's tip?

Nothing is merged. It just switches to using the chain with the most work. It discovers this chain because it will hear about blocks from its peers. Its peers will announce a block which it does not have so it will fetch that block. That block will have a parent it does not have (or it ignored because it conflicted with its current blockchain because its best block and that block have the same parent) so it fetches that block. Since the chain containing this new block has more work than the chain containing the node's best block, it will simply switch to using that other chain.

got it. Thanks Andrew. I'll keep learning Blockchain mean while.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
If I received two blocks at the same time, which one should I choose to mine the next block?
Is the situation could happend?
If true, What's the role? Is the Bitcoin pick one block randomly?

https://bitcointalk.org/index.php?topic=108864.msg1184007#msg1184007 I think this answers your question

If I received two blocks at the same time, which one should I choose
  to mine the next block?

Up to you. You can take the first one. Or the smallest one. Or known by the majority of your peers. Nobody cares.
In worst case you will mine the orphan chain. But nobody cares about you in decentralized system.
In fact you should compare two chains by total work. But this algorithm gives you poor results on comparing two latest blocks with the same height.

Is the situation could happend?

Depends on what you mean by "the same time". Sometimes the blocks with the same height happen. They came in few seconds ( up to 10-20 seconds ) one after another.

Is the Bitcoin pick one block randomly?

You can look to the source code on github. But this does not guarantee that the mining pools use it unmodified.

You can choose the first one. Or the smaller one. Or choose the familiar one by the majority of your contemplate. No one cares. In lowest case you will mine the waif chain. But nobody cares about you in decentralized system. Literally, you should analyze two chains by total work. But this algorithm gives you bad results on comparing two latest blocks with the having same height.


text:
mining revenue equation difference to calculators
I am interested in the revenue calculation of mining process.
From a few references like this I found a equation that people say it's correct.
Unfortunatelly it differs to the results which are calculated by public available calculators like this.
with a hashing power of 13.5 TH/s the referenced calculator and others resulting in 0.001304 BTC per day. The equation in the other thread results in 0.002134 BTC per day.
H = 13500 GH/s * 10^9 = 13,500,000,000,000
B = 12.5
D = 1,590,896,927,258

(86400 * 13,500,000,000,000 * 12.5) / (2^32 * 1,590,896,927,258) =
  0.002134 BTC/d

Could anybody please help me to understand whats the right way to calculate an approx. revenue?
Thanks in advance

Your equation matches my understanding of what is correct.  At the bottom of the Cryptocompare page is a note that "Mining metrics are calculated based on a network hash rate of 18,633,837,452 GH/s".  Multiplying by 600 and dividing by 2^32, this would correspond to a difficulty of 2.6e12 rather than your 1.6e12 which is in fact the current figure.  Perhaps this represents some estimate of future difficulty?

yes. looks like thats the point. thank you very much. that means I have to calculate the network hash rate all the time to use the same equation.


text:
Adversarial Question: Why not fork LND Loop and provide the service freely/cheaply?
Adversarial Question: Now that LND Loop is paid product, what stops someone from forking the software and selling the product themselves, or integrating it freely?
The only reason I can think of is that Loop is still maturing as a product, and that it is LND who will bring  it to maturity, so it makes sense to stay with the proprietors. But that's not even relevant when the service becomes mature.

LND Loop is an open-source solution with MIT Licence. So, you are free to fork the code and create your own solution without any restrictions. In fact, there are a number of submarine swap companies out there that offer you a solution to import/export liquidity from your channel. LND Loop will just be one amongst a sea of such solutions. Market share winners and losers would be differentiated from their customer service and how they are going to tailor the solution to the customer needs.
A classic example to consider is Docker. Its code is completely open source (Apache 2.0 license) for Linux distributions and you can fork it if you want. However, Docker has raised $307.9M in funding to date and has a valuation in billions. They make money through their enterprise editions, docker hub and other professional services.
In my opinion, a similar analogy can be drawn for lightning. In the future, the company that will be able to handle huge requests of loop-ins and loop-outs without sacrificing time will grab a majority market share. They will have to tailor their solution to their customers and provide the best service possible. This will require considering multiple factors including good backend technology to handle the load, good professional services and if the company is able to handle enterprise implementations for their Lightning payment needs. They will also need to figure out liquidity and fluctuations in market price of Bitcoin to handle such operations. Offering all the services at a low cost is what is going to differentiate the competition. Forking off the code is just the first small step.

This answer is incorrect. I have no idea why it's marked as correct answer. Loop server is not open source. 2nd paragraph doesn't apply because, again, loop server is not open source.

LND loop server is not open source. You can interact with Loop from LND which is open source. On the other hand there are already other commercial products that offer submarine swaps as a service.
You also can not offer a loop service for free because someone still needs to bear the costs of moving funds from off-chain to on-chain or vice versa. You could compete with LND with lower fees.

Loop is on github and it seems there are non-lnd developers making commits. I don't doubt you, but could you explain this?

If you read the repo the key sentence there is "This repository is home to the Loop client..". So this is the loop client included in LND. The loop server, which actually does the submarine swap operation is not open source.

The problem isn't software related. All of the off-chain options invariably require a custodian to be managing the funds. Whether that is you on your own node, or Bluewallet, or other 3rd party app/option. The stated purpose of 2nd layer/offchain "solutions" is (ostensibly) to reduce the cost of casual transactions, and hasten an acceptable standard of "confirmed" status for txs for use-cases such as billing for streaming, and retail commerce.
The problem is this. Although the use of off-chain txing will temporarily reduce the cost of tx fees, invariably the cost per tx will continue to increase. Without increasing block-space to reduce the cost of txs, you're still competing for space in those blocks. In addition, the subsidy is continuing to reduce with each halving. This means miners will need to seek their due in the form of tx fees. I believe this means that eventually all private ln nodes (like your's) will be dusted as the tx fees eclipse their off-chain balances, and you will not be able to afford to establish new payment channels because you won't be able to afford the tx fees to have your initializing tx included in a block.
What this means for competitors for Loop is that unless you can cover the tx fees, and put up enough Bitcoin to make opening the payment channel worth doing, you'll be forced to rely on the services of 3rd party payment service providers. I'm assuming that because they just received $10 million in series A funding that they are going to be able to afford to put up more Bitcoin than you, or I ever could.
If you can see a flaw in this thesis, please feel free to let me know. They just block me on twitter, and reddit when I say stuff like this.

RE: paragraph #2 -
I believe your points are sensationalist but plausible.
I  think it's too early to say, especially with entire technologies that have yet to come to fruition, but are
expected to [channel factories for instance]
RE: being banned -
Could you point me to some discussions on twitter or reddit where you've been told you'll be blocked/banned?
In twitter blocks are swift, so it's plausible you've been blocks here and there. On reddit I sincerely doubt that you've blocked without warning if your behavior/prose is similar on those forums as it is here.

Can you tell me what is going to prevent the miner fees from increasing beyond what you can afford to spend on them? The Bitcoin reddit my posts have been removed from/hidden/not visible anymore were on like r./Bitcoin or r/btc. It's been so long since I'd bothered trying that I couldn't tell you which one it was.


text:
How to generate a large number of accounts via hw wallet and save addresses to file?
I work in a company which uses a ledger blue hw wallet.  They need to generate bitcoin and eth accounts to send to users to deposit to.  Currently they have people generating one account at a time in ledger, copy the address, paste the address into our system.  We need thousands of addresses (whilst keeping the private keys safe on hw wallets) so this takes hundreds of hours.
I was surprised that neither trezor nor ledger blue have an app or utility which will spit out a CSV of accounts on the attached PC, keeping the private keys on the device.
Has anyone found such an app?
Ledger blue seems to have no api, but trezor does. Has anyone found a way to do something like this through the trezor api?
We could easily write a program to generate thousands of wallets, and run it on a new pc which was never connected to the net, but this won’t allow us to use the hw wallet to keep the private keys unfortunately.

Your best bet would be to export the xpub key for each account from your ledger. Once you have that, generating new addresses can be automated via any number of bitcoin libraries, such as bitcoinjs-lib. Similar options exist for ethereum.
I'm not entirely sure if they have a straighforward way to export the xpub, but I recall seeing it in the ledger live logs at some point. You can either use that, or connect to a wallet such as Electrum, which would export it for you.


text:
Did Bitcoin Core relay blocks sequentially or in parallel to peers before Compact Blocks?
As I can see in the Bitcoin code, currently a node sends a block in parallel to all its peers. This makes sense as the block sending time is relatively small compared to the P2P latency. However, my question is how this mechanism worked before BIP152 when a peer had to send 1MB block? In that case, it makes more sense to send the block first to 1 peer before start sending to the others. This is because if we finish sending the block to the first peer, we will utilize his upload bandwidth faster.

is it so significant?

Hey, I did a small touch-up on your question. Please feel free to roll back or further edit to your liking. I couldn't figure out what you mean with "block ng time", I think it may be a typo.

It isn't as simple as "sending sequentially" or "sending in parallel".  Each connection is its own socket and the kernel performs packet scheduling. The Bitcoin protocol doesn't have any acknowledgement. When a node sends a message it hands it to the TCP stack which often will just immediately accept the whole message. It's then up to the kernel to send it out.
So in one sense it's not even really possible to send it out 'sequentially' without some kind of massive redesign,  but it also wouldn't be generally desirable to do so: TCP won't send more than the current receive window without getting an acknowledgement. So if you were to wait until TCP has sent all the data to one peer before starting to send the next, you would find yourself waiting around sending nothing part of the time.  You would also increase exposure to tarpit attacks, where a malicious peer intentionally slows down reception in order to delay propagation to everyone else.
So Bitcoin has always gone through the peer list and handed whatever it had to send to each peer, leaving it up to the kernel to schedule the packets.   It has been proposed in the past to hand it to just a few and then wait for them to get it before hitting the rest, but it's not obvious how to robustly prevent tarpits,  and BIP152 largely moots the question.


text:
Is there any python library could be used to decodescript in output
For example decode the ScriptPubKey like
4104678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac
to some data like this
{
"result": {
    "asm": "04678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5f OP_CHECKSIG",
    "reqSigs": 1,
    "type": "pubkey",
    "addresses": [
        "1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa"
    ],
    "p2sh": "3DjjKyU38gSfuVxajV43MUy4vHkg1JVL7T"
},
"error": null,
"id": null
}

In general, something like the function in decodescript rpc.
http://chainquery.com/bitcoin-api/decodescript

I just added that to my toy library.
>>> from btctools import decode_scriptpubkey
>>> decode_scriptpubkey('76a914b3c2dc3a7f68a024a5d2ee9d4012dca1134dea8588ac')
{'address': '1HPVPU3xHMPcxmgdP9puPbnDcHDV5JhURh',
 'asm': 'OP_DUP OP_HASH160 b3c2dc3a7f68a024a5d2ee9d4012dca1134dea85 OP_EQUALVERIFY OP_CHECKSIG',
 'hex': '76a914b3c2dc3a7f68a024a5d2ee9d4012dca1134dea8588ac',
 'type': 'P2PKH'}

Resolved and save the code in here.
https://github.com/keviny/bitcoin-block-parser
Many details are found in https://en.bitcoin.it/wiki/Script.
The code to decode the ScriptPubKey in transaction output:
class TxOutput:
  def __init__(self, blockchain, idx):
    self.idx = idx
    self.value = uint8(blockchain)
    self.script_len = varint(blockchain)
    self.pubkey = blockchain.read(self.script_len)
    self.addr = "UNKNOWN"
    self.decode_scriptpubkey(self.pubkey)

  def to_string(self):
    sb = []
    sb.append("Value: %d" % self.value + " Satoshi")
    sb.append("Script Len: %d" % self.script_len)
    sb.append("ScriptPubkey: %s" % self.addr)
    sb.append("Addr: %s" % self.addr)
    return sb

  def decode_scriptpubkey(self, data):
    ''' https://en.bitcoin.it/wiki/Script '''
    hexstr = hashStr(data)
    # Get the first two bytes.
    # which might some problem.
    # https://www.blockchain.com/btc/tx/7bd54def72825008b4ca0f4aeff13e6be2c5fe0f23430629a9d484a1ac2a29b8
    try:
      op_idx = int(hexstr[0:2], 16)
    except:
      self.type = "EXCEPTION"
      self.addr = "UNKNOWN"
      return
    try:
      op_code = OPCODE_NAMES[op_idx]
    except KeyError:
      if op_idx==65:
        self.type = "P2PK"
        # Obsoleted pay to pubkey directly
        # For detail see: https://en.bitcoin.it/wiki/Script#Obsolete_pay-to-pubkey_transaction
        pub_key_len = op_idx
        op_code_tail = OPCODE_NAMES[int(hexstr[2 + pub_key_len * 2:2 + pub_key_len * 2 + 2], 16)]
        self.pubkey_human = "Pubkey OP_CODE: None Bytes:%s tail_op_code:%s %d" % (pub_key_len, op_code_tail, op_idx)
        self.addr = pubkey_to_address(hexstr[2:2 + pub_key_len * 2])[0]
      else:
        # Some times people will push data directly
        # e.g: https://www.blockchain.com/btc/tx/d65bb24f6289dad27f0f7e75e80e187d9b189a82dcf5a86fb1c6f8ff2b2c190f
        self.type = "UN"
        pub_key_len = op_idx
        self.pubkey_human = "PUSH_DATA:%s" % hexstr[2:2 + pub_key_len * 2]
        self.addr = "UNKNOWN"
      return
    try:
      if op_code == "OP_DUP":
        self.type = "P2PKHA"
        # P2PKHA pay to pubkey hash mode
        # For detail see: https://en.bitcoin.it/wiki/Script#Standard_Transaction_to_Bitcoin_address_.28pay-to-pubkey-hash.29
        op_code2 = OPCODE_NAMES[int(hexstr[2:4], 16)]
        pub_key_len = int(hexstr[4:6], 16)
        op_code_tail2 = OPCODE_NAMES[int(hexstr[6 + pub_key_len * 2:6 + pub_key_len * 2 + 2], 16)]
        op_code_tail_last = OPCODE_NAMES[int(hexstr[6 + pub_key_len * 2 + 2:6 + pub_key_len * 2 + 4], 16)]
        self.pubkey_human = "%s %s %s %s %s" % (op_code, op_code2,  hexstr[6:6 + pub_key_len * 2], op_code_tail2, op_code_tail_last)
        self.addr = gen_addr(hexstr[6:6 + pub_key_len * 2])[0]
      elif op_code == "OP_HASH160":
        self.type = "P2SH"
        # P2SHA pay to script hash
        # https://en.bitcoin.it/wiki/Transaction#Pay-to-Script-Hash
        pub_key_len = int(hexstr[2:4], 16)
        op_code_tail = OPCODE_NAMES[int(hexstr[4 + pub_key_len * 2:4 + pub_key_len * 2 + 2], 16)]
        hash_code = hexstr[4:4 + pub_key_len * 2]
        self.pubkey_human = "%s %s %s" % (op_code, hash_code, op_code_tail)
        self.addr = hash_code
      elif op_code == "OP_RETURN":
        self.type = "OP_RETURN"
        pub_key_len = int(hexstr[2:4], 16)
        hash_code = hexstr[4:4 + pub_key_len * 2]
        self.pubkey_human = "OP_RETURN %s" % (hash_code)
        self.addr = hash_code
      else:  # TODO extend for multi-signature parsing
        self.type = "UN"
        self.pubkey_human = "Need to extend multi-signaturer parsing %x" % int(hexstr[0:2], 16) + op_code
        self.addr = "UNKNOWN"
    except:
      self.type = "ERROR"
      self.addr = "UNKNOWN"

Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference.


text:
How do you make a coin resistant to soft forking?
Soft forks are probably worse than hard forks, since it is hard for a client to reject a soft fork (indeed, that is the point of soft forks). The only way to do so is either to gain 51% of the hash power, or to do a hard fork rejecting it.
How can you make an altcoin resilient to soft forking (or only allow certain types of soft forks)? I think doing this is in general might be impossible, but making it harder would be good. Having no soft fork policy built into the coins philosophy would help, but probably isn't good enough.

You can't prevent miners from enforcing more rules than the rest of the network is. However, to actually turn that into a soft fork, a significant portion of the network also needs to start enforcing it.
However, the only way a softfork can affect users of the system in a negative way is by disallowing certain transactions that used to be useful. That problem is nothing more than the inherent censorship rights a majority of miners have over the system. The solution to that problem is not trying to prevent enforcement of rules (which is not possible; miners can do it even without announcing it), but by improving the privacy of the system. If miners cannot see what transactions are doing, they cannot selectively block.

Although privacy prevents a black list, it doesn't prevent a white list.

Also, there remember that it is miners, not coin holders, the enforce rules on the network. If under the old rules I have some coins, but under the new rules I'm not allowed to spend them, and the miners are using the new rules, I'm not going to spend them. The only way around this is to hard fork away from the main chain (which will have won before I noticed).

Are we less strict these days about discussing altcoins? Or this is OK as a purely hypothetical?

@HansBKK Since 2019, altcoin questions are off-topic here: https://bitcoin.meta.stackexchange.com/q/1011/208. Before that time, the topic was broader. Questions/discussion about what is on-topic belongs on the meta site.


text:
How does the bech32 length-extension mutation weakness work?
A bech32 address ending with p can be modified be inserting or removing q characters immediately before the final p character to make a new valid bech32 address. Why does this work?
Are there any other similar mutation weaknesses in bech32?
References:

issue raised against bech32 repo
mailing list post

I'm going to provide a more algebraic explanation of the issue.
Every valid (and invalid) Bech32 string can be seen as a polynomial in the variable x, where the coefficients are elements of GF(32). I'm going to ignore the GF(32) part in what follows, so it suffices to say that they're simply numbers from 0 to 31 inclusive, with weird addition and multiplication rules that guarantee every operation always stays within the range 0-31.
The data part (which means the string excluding the human-readable part (HRP) and the "1" that follows it, but including the 6 checksum characters at the end) uses the character set that's included in BIP173 in table form. For example, a "q" means 0, "p" means 1, "3" means 17, and "l" means 31. Each of these characters forms the coefficient of the term of our polynomial. I'm going to ignore the human-readable part as well in what follows as it doesn't matter for the issue, but for completeness the HRP also encodes coefficients of the polynomial, but using a different (and slightly more complicated) translation.
For example, the string "3qlp" corresponds to the polynomial 17x3 + 31x + 1. As "q" represents 0, there is no x2 term.
Now, every valid Bech32 string corresponds to a polynomial p(x) that can be written as p(x) = f(x)⋅g(x) + 1 for some f(x), where g(x) = x6 + 29x5 + 22x4 + 20x3 + 21x2 + 29x + 18.
Why is that + 1 there? If it wasn't, then for every valid p(x) = f(x)⋅g(x), x⋅f(x)⋅g(x) = x⋅p(x) would also be a valid polynomial. As multiplying by x is simply adding a 0 (or "q") at the end, this would let anyone accidentally mutate valid Bech32 strings into distinct valid Bech32 strings by simply appending a "q".
However, it turns out that adding the + 1 does not fully solve this sort of issue. It isn't possible anymore to simply multiply by x, but it's still the case that for every valid p(x), xi⋅(p(x) - 1) + 1 for any i > 0 is also valid. If p(x) happens to end in a "p", it can be written as x⋅e(x) + 1 for some e(x). In this case, xi⋅e(x) + 1 is also a valid string, for every value of i. In string form, this corresponds to inserting "q" characters in the penultimate position (just before the final "p").
Similarly, if p(x) happens to end in a "qp", it can be written as xi⋅e(x) + 1 for some e(x) and i > 1. In this case, xj⋅e(x) + 1 where j < i is also a valid string. In string form, this corresponds to removing "q" character in the penultimate position (just before the final "p").

It might be worth mentioning that addresses that end with q('s) and then p can equally have q's removed.  You also might want to point out the invalid padding constraint. Also that v0 is immune due length restrictions.

Pieter Wuille's comment gives a nice summary:

basically: take a bech32 string, xor a 1 into the last character, then push or pop as many 'q's as you like, and then xor a 1 into the last character again... should always give you a valid new bech32 string

Checksum code taken from Bitcoin Core is:
uint32_t PolyMod(const data& v)
{
    uint32_t c = 1;
    for (const auto v_i : v) {
        uint8_t c0 = c >> 25;

        c = ((c & 0x1ffffff) << 5) ^ v_i;

        if (c0 & 1)  c ^= 0x3b6a57b2;
        if (c0 & 2)  c ^= 0x26508e6d;
        if (c0 & 4)  c ^= 0x1ea119fa;
        if (c0 & 8)  c ^= 0x3d4233dd;
        if (c0 & 16) c ^= 0x2a1462b3;
    }
    return c;
}

Basically, the checksum function has an internal variable that is modified with every 5-bit character, similar to SHA without padding and different sizes. What you should notice is that it both this and SHA are prone to length extension attacks, which means someone who doesn't know x but knows H(x) can calculate H(x || A), where A is any data sequence and || is the concat operator.
Called from:
data CreateChecksum(const std::string& hrp, const data& values)
{
    data enc = Cat(ExpandHRP(hrp), values);
    enc.resize(enc.size() + 6); // Append 6 zeroes
    uint32_t mod = PolyMod(enc) ^ 1; // Determine what to XOR into those 6 zeroes.
    data ret(6);
    for (size_t i = 0; i < 6; ++i) {
        // Convert the 5-bit groups in mod to checksum values.
        ret[i] = (mod >> (5 * (5 - i))) & 31;
    }
    return ret;
}

In this function, simply the result of PolyMod is serialized, but the checksum's least significant bit is XOR'ed with 1. If you undo that, you can extend the checksum by feeding the PolyMod function (or the Initialize-Update-Finalize version of it) a number of five-bit zeros more, and if you XOR the least significant bit with 1 again, you'll get a valid checksum.
Why? Because a zero byte does not cause any if branch to run in the PolyMod function loop. c was zero, and it stays zero.
You can now skip to the bottom to my more clear explanation
The reason XOR-1 was a part of the specification is (from my understanding) was to prevent adding an additional character at the end in a way that the checksum is still correct. Otherwise,

ii2134hk2xmat79tqq
ii2134hk2xmat79tqqq
ii2134hk2xmat79tqqqq

would all be correct, which is worse (even though there are length checks...)
In the CreateChecksum function, you'll see that to create a signature, 6 empty five-bits are appended (because it's how BCH works, checksum is set to empty before being calculated). Because PolyMod has no knowledge if the zeros are part of the input or the empty checksum, they're treated the same. Therefore you can:
1) Take any Bech32 address as a five-bit byte sequence
2) Xor the last byte with 1
3) Append arbitrary numbers
4) Add 6 empty bytes
5) Calculate PolyMod result
6) Xor the last byte with 1
7) Encode
The quirk described in the link given is when those "arbitrary numbers" in step 3 are zeros. The PolyMod requests a payload with 6 zeros appended to be checksummed (and trailing zeros make a difference in this case as c is nonzero), and the checksum should come just after the payload, if we want the checksum to be correct. You can add zeros to the end, and it'll stay correct. Only the XOR-1 thing has to be done.

Alternative explanation without distracting details
Here's how the BCH checksum works and why it's unique:
If you give the PolyMod function a payload and 6 empty bytes (which is the length of the signature) it'll give you a value.
If you serialize this value in the place of the last 6 empty bytes and call the PolyMod function, it'll return zero. Whenever the checksum is correct, it returns zero. You can then discard the last 6 bytes and use the rest while decoding
Consider the PolyMod function's state after the payload and the checksum is processed. It returns zero, which is the value of the c variable (you can look at the code at the top). Same for c0. Since then, as long as v_i equals zero, c will stay zero. Therefore you can append zeros at the end without breaking the checksum. Or the converse, if you have a valid address, you can remove zeros at the end.
Plus some Xor-ing with 1, as I explained in the middle of the post. In Bech32 alphabet, p ^ 1 = q and q ^ 1 = p.


text:
CHECKMULTISIG a worked out example
I was wondering if somebody could point me to or give me a worked out example of how OP_CHECKMULTISIG works behind the curtains?
I expect that CHECKMULTISIG will call the CHECKSIG function and manipulate the stack to run CHECKSIG in a loop until it terminates and either has its M-out-of-N valid sigs or not?
But how exactly does it work? E.g. for 2-of-3 multisig. If we have OP_0 (sig1) (sig2) in the scriptSig, then the stack will look like:

(sig2)
(sig1)
0

Now from the general way of writing CHECKMULTISIG in the scriptPubKey: 2 (pubKey1) (pubKey2) (pubKey3) 3 OP_CHECKMULTISIG, when pushing all the data onto the stack, the stack for the scriptPubKey will look like:

3
(pubKey3)
(pubKey2)
(pubKey1)
2

What exactly will the first step be of CHECKMULTISIG?
Will it pop the "3" from the stack and then maybe check if there are three pubKeys on the stack?
Or will it just pop the "3" and then immediately run CHECKSIG on (sig2) and (pubKey3), which will not be valid, hence CHECKSIG will push a "0" on the stack?
Will then CHECKMULTISIG remove the "0" that CHECKSIG introduced on the stack and try again with (sig2) and (pubKey2)? Since this is valid CHECKSIG will push a "1" on the stack, but where does CHECKMULTISIG store this "1"?
And will CHECKMULTISIG at the end count how many "1"'s CHECKSIG made and this number must correspond to the last value on the stack, in this case "2", but in general "M"?
I would really like to find a low-level description of CHECKMULTISIG, similarly to the one I could find for CHECKSIG:
https://en.bitcoin.it/wiki/OP_CHECKSIG
Any help would be very appreciated!

https://en.bitcoin.it/wiki/BIP_0011

I expect that CHECKMULTISIG will call the CHECKSIG function and manipulate the stack to run CHECKSIG in a loop until it terminates and either has its M-out-of-N valid sigs or not?

That's not really how it's implemented internally, so there are a couple of pitfalls if you imagine it operating like that.
OP_CHECKMULTISIG
Here's the high-level description of how it works:

Pop n off of the stack (number of public keys)
Pop n public keys off of the stack.
Pop m off of the stack (number of required signatures)
Pop m signatures off of the stack.
Pop one more element off of the stack, and ignore it. (This is a bug, but it can't be fixed because this is consensus-critical code.)
Loop through all of the public keys, starting with the keys at the top of the stack.

For each public key, check a single signature.
For the first public key checked, start with the signature closest to the top of the stack.
If it fails to verify, go to the next public key and check the same signature.
If it succeeds, go to the next public key with the next signature.
Note that the signatures need to be in the same order as the key's that they're signing for.

If all of the signatures succeeded with one of the keys, CHECKMULTISIG returns 1, otherwise 0.

Your example
You've got this scriptSig:
    (sig2)
    (sig1)
    0

This scriptPubKey:
    3
    (pubKey3)
    (pubKey2)
    (pubKey1)
    2

When executing, the stack looks like this just before OP_CHECKMULTISIG executes:
    3
    (pubKey3)
    (pubKey2)
    (pubKey1)
    2
    (sig2)
    (sig1)
    0

This is what it looks like after initializing:
n->    3
ikey-> (pubKey3)
       (pubKey2)
       (pubKey1)
m->    2
isig-> (sig2)
       (sig1)
       0

It tries to verify pubKey3 against sig2, but that fails. Advance ikey, but not isig:
n->    3
       (pubKey3)
ikey-> (pubKey2)
       (pubKey1)
m->    2
isig-> (sig2)
       (sig1)
       0

Verifying pubKey2 against sig2 suceeds, so you advance both ikey and isig:
n->    3
       (pubKey3)
       (pubKey2)
ikey-> (pubKey1)
m->    2
       (sig2)
isig-> (sig1)
       0

...and so on.

Thank you so much! This example has helped me a lot to understand what is really going on in the C++ Code of Bitcoin!

I can't add a comment to Nick ODell so I do it here, a tiny nano pocket error : "It tries to verify pubKey3 against sig2, but that fails. Advance ikey, but not isig:" Instead of : "It tries to verify pubKey3 against sig1, but that fails. Advance ikey, but not isig:"

It's far simpler than relying on values pushed by OP_CHECKSIG internally! It calls the same function that OP_CHECKSIG uses - in a loop - after parsing the public key and required signature count.
The procedure starts by checking the stack's size. Next it takes the keyCount (at the top of the stack as you observed in your question), and limits the number to 20 or less. It stores the position of the first public key in the stack.
Taking the value at position (top - keyCount) should give you the number of signatures. Some sanity checks are now performed (the required number of signatures should not exceed the number of keys). It then stores the position of the first signature.
Next the software enters a loop: So long as none of the signatures fail, it takes the first signature and public key, validates their encoding, and calls the same function that OP_CHECKSIG uses to check the signature.
If validation succeeds, it will increment the signature counter. If it fails, this doesn't happen, so the loop will try this signature with the next key. But in either case, the public key counter is increased.
This means signatures need to be provided in the same order as the public keys. If a signature fails, try the next key. (Think about the ordering of signatures in a 2-of-3)
It will also check that the dummy null value exists, and optionally verifies that it actually is a byte of \x00.
At the end, it pushes a boolean indicating the return value.
The source code for this opcode can be found here: https://github.com/bitcoin/bitcoin/blob/master/src/script/interpreter.cpp

Thanks for giving me the push into the right direction to understand what exactly is going on!

authors: Роман Завальнюк


text:
Can i compose a multisig address where one of the signatories is a multisig address?
Can i compose a multisig address where one of the signatories is a multisig address?
In order to create a multisig address we need the pubkey of the signatories.
bitcoincli addmultisigaddress 2
'["pubkey1", "pubkey2",..."pubkey(n)"]'

Is anything stopping me from using a multisig address as one of the signatories?
Essentially: can one of the signatories be a multisig address (one that currently starts with a 3).

No this is not supported behavior. You can manually add the public keys from one multisig script into another, but a multisignature script can not be nested within another.

It was always supported by Bitcoin Script, and there is now tools to be able to use that safely (output descriptors with Miniscript support).

@AntoinePoinsot: I think the question asker is asking about the equivalent of a nested MuSig2 key within a script? If he is then that wasn't possible until Schnorr (or at least an equivalent MuSig2 scheme wasn't specified for ECDSA)

I don't think he's asking about a public key aggregation scheme specifically, just about having the feature "a 2-of-3 where one of the signer is actually a 2-of-3 (or another sort of multisig)". And it's precisely one of the things Miniscript makes possible: composability. Also, you wouldn't be able to have a threshold with Musig2 directly (only by using a Taproot tree) and he's using a 2-of-3 in his question. TL;DR: both work. You can either use Miniscript or have aggregated keys in separate Taproot leaves, but i'd argue the latter is significantly more involved at the moment. :)

Update to Claris' answer in April 2023:
With Schnorr signatures (enabled in the Taproot soft fork that activated in November 2021) you can now generate a single Schnorr public key that is the aggregate of multiple other valid Schnorr public keys and create a single aggregated signature from them. This aggregated Schnorr public key can be nested within another multisig scheme. However, your Bitcoin address needs to be a Taproot address (SegWit v1) starting bc1p.... rather than 3....


text:
A question on CNode class data members
Can any expert help me understand the need for the following members of CNode in src/net.h
std::deque<CSerializeData> vSendMsg;
std::deque<CInv> vRecvGetData;
std::deque<CNetMessage> vRecvMsg;

If I understand the source code correct, vRecvMsg is for storing the input messages and vSendMsg is a queue for output messages.

Why do we have different form of queues [i.e. CNetMessage and CSerializeData] for input and output?
What is the purpose of vRevGetData? Can you please help me understand also the concept behind the usage of this ?

The question is from 2015; some things have changed since then.
CNetMessage is a transport protocol agnostic message container. It contains the received message data (DataStream), time of message receipt, payload size, and other information. It is used to decompose messages from the network. See the following code:
bool CNode::ReceiveMsgBytes(Span<const uint8_t> msg_bytes, bool& complete)
{
    complete = false;
    const auto time = GetTime<std::chrono::microseconds>();
    LOCK(cs_vRecv);
    m_last_recv = std::chrono::duration_cast<std::chrono::seconds>(time);
    nRecvBytes += msg_bytes.size();
    while (msg_bytes.size() > 0) {
        // absorb network data
        if (!m_transport->ReceivedBytes(msg_bytes)) {
            // Serious transport problem, disconnect from the peer.
            return false;
        }

        if (m_transport->ReceivedMessageComplete()) {
            // decompose a transport agnostic CNetMessage from the deserializer
            bool reject_message{false};
            CNetMessage msg = m_transport->GetReceivedMessage(time, reject_message);
            if (reject_message) {
                // Message deserialization failed. Drop the message but don't disconnect the peer.
                // store the size of the corrupt message
                mapRecvBytesPerMsgType.at(NET_MESSAGE_TYPE_OTHER) += msg.m_raw_message_size;
                continue;
            }

            // Store received bytes per message type.
            // To prevent a memory DOS, only allow known message types.
            auto i = mapRecvBytesPerMsgType.find(msg.m_type);
            if (i == mapRecvBytesPerMsgType.end()) {
                i = mapRecvBytesPerMsgType.find(NET_MESSAGE_TYPE_OTHER);
            }
            assert(i != mapRecvBytesPerMsgType.end());
            i->second += msg.m_raw_message_size;

            // push the message to the process queue,
            vRecvMsg.push_back(std::move(msg));

            complete = true;
        }
    }

    return true;
}

After receiving the message, we push it to vRecvMsg and then put it in a queue to be processed. In ProcessMessage, we get the message data (DataStream) and process it according to the message type.
bool PeerManagerImpl::ProcessMessages(CNode* pfrom, std::atomic<bool>& interruptMsgProc)
{
    AssertLockHeld(g_msgproc_mutex);

    PeerRef peer = GetPeerRef(pfrom->GetId());
    if (peer == nullptr) return false;

    {
        LOCK(peer->m_getdata_requests_mutex);
        if (!peer->m_getdata_requests.empty()) {
            ProcessGetData(*pfrom, *peer, interruptMsgProc);
        }
    }

    const bool processed_orphan = ProcessOrphanTx(*peer);

    if (pfrom->fDisconnect)
        return false;

    if (processed_orphan) return true;

    // this maintains the order of responses
    // and prevents m_getdata_requests to grow unbounded
    {
        LOCK(peer->m_getdata_requests_mutex);
        if (!peer->m_getdata_requests.empty()) return true;
    }

    // Don't bother if send buffer is too full to respond anyway
    if (pfrom->fPauseSend) return false;

    auto poll_result{pfrom->PollMessage()};
    if (!poll_result) {
        // No message to process
        return false;
    }

    CNetMessage& msg{poll_result->first};
    bool fMoreWork = poll_result->second;

    TRACE6(net, inbound_message,
        pfrom->GetId(),
        pfrom->m_addr_name.c_str(),
        pfrom->ConnectionTypeAsString().c_str(),
        msg.m_type.c_str(),
        msg.m_recv.size(),
        msg.m_recv.data()
    );

    if (m_opts.capture_messages) {
        CaptureMessage(pfrom->addr, msg.m_type, MakeUCharSpan(msg.m_recv), /*is_incoming=*/true);
    }

    try {
        ProcessMessage(*pfrom, msg.m_type, msg.m_recv, msg.m_time, interruptMsgProc);
        if (interruptMsgProc) return false;
        {
            LOCK(peer->m_getdata_requests_mutex);
            if (!peer->m_getdata_requests.empty()) fMoreWork = true;
        }
        // Does this peer has an orphan ready to reconsider?
        // (Note: we may have provided a parent for an orphan provided
        //  by another peer that was already processed; in that case,
        //  the extra work may not be noticed, possibly resulting in an
        //  unnecessary 100ms delay)
        if (m_orphanage.HaveTxToReconsider(peer->m_id)) fMoreWork = true;
    } catch (const std::exception& e) {
        LogPrint(BCLog::NET, "%s(%s, %u bytes): Exception '%s' (%s) caught\n", __func__, SanitizeString(msg.m_type), msg.m_message_size, e.what(), typeid(e).name());
    } catch (...) {
        LogPrint(BCLog::NET, "%s(%s, %u bytes): Unknown exception caught\n", __func__, SanitizeString(msg.m_type), msg.m_message_size);
    }

    return fMoreWork;
}

In the case of vSendMsg, this is a vector of CSerializedNetMsg. CSerializedNetMsg structure is simple. It holds the message data (a vector of unsigned char) and its type. As its name indicates, it represents the serialized message. You can notice in the codebase that NetMsg::Make is commonly used to construct it. This function accepts a string parameter representing the message type and any other parameter except that will be used to compose the msg data.
namespace NetMsg {
    template <typename... Args>
    CSerializedNetMsg Make(std::string msg_type, Args&&... args)
    {
        CSerializedNetMsg msg;
        msg.m_type = std::move(msg_type);
        VectorWriter{msg.data, 0, std::forward<Args>(args)...};
        return msg;
    }
} // namespace NetMsg

Now, what I understand about the difference between CSerializedNetMsg and CNetMessage is:

CSerializedNetMsg seems lighter.
CNetMessage has additional members (m_time, m_message_size, m_raw_message_size)
Only one CNetMessage object for the same message will exist. CSerializedNetMsg has a specific method for copies.

Although they seem similar, they have specific characteristics for their purposes. For example, it is important to deal with received messages with CNetMessage, among other reasons, to know exactly the size of the data we receive to track the process queue size. Copy from CSerializedNetMsg can be useful when sending the same message for more than one node.
    m_connman.ForEachNode([this, pindex, &lazy_ser, &hashBlock](CNode* pnode) EXCLUSIVE_LOCKS_REQUIRED(::cs_main) {
        AssertLockHeld(::cs_main);

        if (pnode->GetCommonVersion() < INVALID_CB_NO_BAN_VERSION || pnode->fDisconnect)
            return;
        ProcessBlockAvailability(pnode->GetId());
        CNodeState &state = *State(pnode->GetId());
        // If the peer has, or we announced to them the previous block already,
        // but we don't think they have this one, go ahead and announce it
        if (state.m_requested_hb_cmpctblocks && !PeerHasHeader(&state, pindex) && PeerHasHeader(&state, pindex->pprev)) {

            LogPrint(BCLog::NET, "%s sending header-and-ids %s to peer=%d\n", "PeerManager::NewPoWValidBlock",
                    hashBlock.ToString(), pnode->GetId());

            const CSerializedNetMsg& ser_cmpctblock{lazy_ser.get()};
            PushMessage(*pnode, ser_cmpctblock.Copy());
            state.pindexBestHeaderSent = pindex;
        }
    });

About vRecvGetData, it was removed in #19911. Now we have m_getdata_requests as a member of Peer, when a node receives a GETDATA message, it stores the INVs into m_getdata_requests. It is used to control what a node requested of us (e.g., a transaction).


text:
About performing SHA-256 hashing on the public key
I am following this link, trying to perform SHA-256 on a given public key.
However, I cannot get the expected result.
I am using the sha256 library in python hashlib
The input is
0450863AD64A87AE8A2FE83C1AF1A8403CB53F53E486D8511DAD8A04887E5B23522CD470243453A299FA9E77237716103ABC11A1DF38855ED6F2EE187E9C582BA6

Mine output is
32511e82d56dcea68eb774094e25bab0f8bdd9bc1eca1ceeda38c7a43aceddce

The expected output on bitcoin.it is
600FFE422B4E00731A59557A5CCA46CC183944191006324A447BDB2D98D4B408

Do I miss anything?

The result you are getting (3251...)
is the result of sha256 on the string 0450....
The expected output on bitcoin.it is the result of sha256 on the bytes that are currently hex-encoded. To resolve this, hex-decode before hashing your input. In Python 2:

import hashlib
in_data = "0450863AD64A87AE8A2FE83C1AF1A8403CB53F53E486D8511DAD8A04887E5B23522CD470243453A299FA9E77237716103ABC11A1DF38855ED6F2EE187E9C582BA6".decode('hex')
my_sha = hashlib.sha256()
my_sha.update(in_data)
output = my_sha.digest().encode('hex')

That should retain output with the expected value.

Yep, that's tripped me up too. You'll need to use bytes objects - b"hexData" - for Python 3.x

Good point. I'm not sure how to write the code so it's compatible with Python 3 ("string".encode('hex') is replaced), so unless you do I'll just specify Python 2 here.

import codecs, then codecs.encode(codecs.decode("abcdef".encode("utf-8"), "hex")[::-1], "hex").decode()... It's quite counterintuitive IMO, as I'm so used to Python 2.7


text:
Is it possible to create pure lightning lockboxes (without touching the blockchain)?
I'm considering a scenario in lightning, where it would be extremely useful to be able to lock the funds in 2/3 lock
Is this possible on lightning network (even in theory)?

Every Lightning channel has two participants, funds are initially locked in a 2-of-2 multisig address and each participant has one of the two private keys associated with this address. Threshold schemes like FROST allow you to send funds to the equivalent of say a 2-of-3 with only one aggregated public key going onchain. One of the two (or both) participants in the Lightning channel could use a scheme like FROST to make their single public key associated with the Lightning channel into a 2-of-3 threshold. There are additional keys involved in the operation of Lightning channels once it has been set up but this answers covers incorporating an off-chain threshold scheme into the opening and closing of the Lightning channel.

I am not sure if I understand your question right. Do you want to have a 2 out of 3 multisig wallet? Theoretically this is very possible. Though practically this will be quite combersome. I guess the best way to achieve this would be to wait for BIP 118 to be activated and then the eltoo upgrade. This would easily enable multiparty channels. This could be used to achieve a scenario as you described.


text:
Where can I get some MuSig2 test vectors?
I am writing a MuSig2 library and need some test vectors. This should include the hash functions used.

I'll leave to a MuSig(2) expert to answer the question but I don't think these exist yet. The MuSig(2) authors haven't included it in a secp256k1 release and it has been described as "super-experimental" https://github.com/bitcoin-core/secp256k1/issues/997. Some people are using it on mainnet though https://twitter.com/MuunWallet/status/1459769042072440833?s=20

At the moment, your best bet is the the MuSig2 PR to secp256k1-zkp: https://github.com/ElementsProject/secp256k1-zkp/pull/131. It doesn't update the musig-spec.mediawiki yet and is only code so far. Still happy to hear your feedback there on the API design etc.

@MichaelFolkson In the linked secp256k1 issue #997, I didn't mean to say that it's super-experimental. I really meant the opposite and "Is musig2 "super-experimental" [...]?" was a rhetorical question: I assume that the other contributors don't think it's super-experimental. I personally feel very confident in the scheme. It has a security proof, and the core ideas have been verified by others independently (e.g., FROST)
The only thing that I think is experimental is the implementation, which is currently PR'd to secp256k1-zkp but maybe should be PR'd to secp256k1 as an experimental module.

@real-or-random: Thanks for clarifying!

As far as I know, there is not even a published specification for MuSig2. One is being worked on at https://github.com/jonasnick/bips/blob/musig2/bip-musig2.mediawiki, but it is incomplete.
There are some test vectors here.


text:
Where can I find taproot spend examples on testnet/signet?
I am in the middle of writing some Taproot libraries and would like some testnet/signet p2tr spends. Is there an easy way to find them? Most explorers don't have any filters like that. Even a list of 10 different p2tr's with various key-spends and tap-spends would be very helpful.

Not on chain, but have you seen https://github.com/bitcoin/bips/pull/1225 ?

I swear, I had the BIP0341 specification open and this PR wasn't included. Yes, this is exactly what I need because what was there before, I couldn't really understand. Thank you.

It was also only just merged.

As Pieter said in the comments there are Taproot test vectors that have recently been merged into BIP 341.
You can also find additional P2TR spends on signet, testnet on block explorers e.g. mempool.space. Any transaction that is spending from a tb1p address is a Taproot spend (tb1q is SegWit version 0 and tb1p is SegWit version 1 as discussed here).
Alternatively any output that starts with OP_1 (01) and is spent in a transaction is a Taproot spend.
There are a number of Taproot spends in recent signet/testnet blocks at the time of writing (less than a day before Taproot activates on mainnet).


text:
Tests for Pay To Taproot addresses?
I am having trouble finding examples showing a known WIF to a valid mainnet P2TR address.
I am working on a tool and want to see if my P2TR conversions/encodings are outputting the correct P2TR address.
For example, a known Private Key (empty string brain wallet):
L4rK1yDtCWekvXuE6oXD9jCYfFNV2cWRpVuPLBcCU2z8TrisoyY1
I am getting the following P2TR:
bc1pw74tdcrxlzn5r8z6ku2vztr86fgq0m245s72mjktf4afwzsf8ugs0gs8zu
Is this the correct P2TR for this private key. Any way I can test/confirm this is correct?

I've been working on a set of wallet-focused test vectors for BIP341: https://github.com/bitcoin/bips/pull/1225

If I use that WIF key directly as output pubkey (not recommended) I get bc1p5d9enu3v0yxyud4jk0pvxk3kmvrzymjpc6f0eq4ck44vr32qck7scr6tj5. If I use the procedure in BIP341 (tweaking it with H_TapTweak(xonly_pubkey))), I get bc1ps7qmhrpjnkwrraxwparm4drmvk4jdj6rlg4gtwm0q0je3r7afhnspef95z.

Meanwhile, BIP341 has been updated to link to a set of test vectors covering scriptPubKey computation, keypath spending, and control block construction.


text:
what is the best software based solution to secure the private key in Bitcoin
Would you mind telling me what is the best introduced approach to secure the private key in the wallet and a transaction which is done using Bitcoin? I do not want to use a separate hardware, I am seeking for the best software based approach to protect the private key.
Thanks in advance
Best Regards

Not sure what you mean by "best".  Put the private key on a piece of paper, lock it in a bank vault, surround it with armed guards and sharks with frickin' lasers?

Hi, Thank you. Yes, cold storage is a good idea, however my mean is securing the private key during the signing process of the transaction. I need to know what is the best way to protect the private key during this process?

Hm. Rather strange question you ask.
First of all: it depends mostly on the infrastructure you use for your bitcoin-client.
If your OperatingSystem is compromised, you probably can't guarantee security of your privateKey at all.
The Bitcoinclient supports a very strong encryption of the wallet (and ofc the private Key) since like 2 years natively.
I guess if you are paranoid and want extra security, use a VirtualMachine, install some hardened Linux-OS (DebianStable?), there make a TrueCrypt-Container with a hard password and install your Bitcoinclient there.
Ofcourse, as mentioned earlier, you can't be more secure than making an offline-wallet. Remebering your key instead of writing it down could also serve as the absolute security feature. - but i guess you don't want to go that far? :)
i saved my keys in a TrueCryp-container encrypted with sha512 and used a strong password - after this, backup copies of this container wherever you want, so you don't lose it. sha512 is not even crackable for the NSA.


text:
Is the idea of a "honey hole" such as in gold mining, present in bitcoin mining?
I understand the output of the SHA-256 hashing function is a random number between 0 and 2^256. Does the randomness of this output imply that there cannot be a, so to say, honey hole in the hashing.
For instance this reddit post, http://www.reddit.com/r/BitcoinMining/comments/1rysqh/is_there_something_strange_about_block_272754_and/
They point out that someone relayed two blocks very close together, approximately 10s apart. Could they have found an occurrence where two block solving hashes sit close to each other?
I guess this kind of doesn't make sense though. Once a block is found, or a transaction is accepted, and even for each 3 seconds(timestamp) the block header is changed. So you would really could never find a block with the same hash just one number off (same block header, with two blocks one with nonce and the other nonce+1), because the first block will not be included in the second one, the second will become a dead fork in the blockchain?

No, the SHA2 hashing function is not vulnerable to any known attack that significantly reduces the difficulty of finding a block, but it is a highly random process.
Randomness is funny sometimes. Statistically it it works out to be even over a longer time period or larger population, but if you focus in on a small set of samples sometimes you see a cluster or a gap. This is why the estimated hash rate of the network on blockchain.info is woefully inaccurate on anything less than a multi-day average, and bitcoin is based on a 2 week correction period instead of a smaller number.
(which is why the unlucky miner can't expect to find 2 blocks quickly "because" he had a drought, and you should not expect a gap to "make up for" earlier luck, it's the  population that matters.)

I think this answer is correct, but I'd like to add a comment regarding potential foul play with quick consecutive blocks by the same miner/pool. The vulnerability, already widely discussed, is where a block found by the miner is kept private and not broadcast immediately, giving the miner/pool a slight head start on the next block, at the potential risk of having the original block go orphaned.

I agree, that the luck will even out in the end, but that doesn't show that you couldn't find a block every hash right? You have the same chance each block, isn't there then a chance that you could find n blocks in a row? Though the chance is probably amazingly incomprehensibly low.


text:
Is it a problem that miners participate in a pools?
I am talking about miners centralization with large pools. Is it a problem?
I think there are some possible ways to prevent miners from pooling. For example add miner signature to the block header, so pool will not be able to disclosure their private key to outsource calculations. Miners should use something like insurance instead of pooling.

Can you be more specific about your proposal? I don't understand how it would work.  What's to stop a large pool from creating many keys, as if they were a large number of solo miners or small pools?

@NateEldredge currently miners calculate blocks with reward associated with pool address, that's why miners can't trick on pools. Sharing private key of reward recipient address is not safe – any miner will be able to get the whole reward.

Is it a problem that miners participate in a pools?

It can be.  The most widely-used mining protocol at present (and the historically popular protocols) give complete control over what gets mined to the pool operator.  Even if the pool operator has the best of intentions, they can be hacked and their hash rate abused for evil.  This is what allegedly happened when the mining pool GHash.io (with about 30% of hash rate) stole about $100,000 in BTC from the betting website BetCoin Dice.
But it doesn't have to be a problem.  One current mining protocol, GetBlockTemplate, theoretically allows pools to give individual miners the ability to select which transactions they include in their blocks, preventing the pool from centrally making important decisions.  Another system, Peer-to-Pool Pool (P2Pool), is a completely decentralized pool where there is no pool operator, just a bunch of independent miners collaborating without any one of them making decisions for the others.
Both of these protocols have problems. GetBlockTemplate (GBT) is bandwidth intensive, complicated, and not very flexible; P2Pool is complicated, less efficient than centralized pooling (it can't include as many fee-paying transactions in a block), and doesn't necessarily work well for very small miners.
A new alternative to both of these protocols is BetterHash which looks like it many work much like GBT was supposed to without the problems.

I  think there are some possible ways to prevent miners from pooling. For example add miner signature to the block header, so pool will not be able to disclosure their private key to outsource calculations.

This solution has been proposed many times before and it certainly could be added to the Bitcoin Protocol.  However, several experts worry that it would make centralization worse by only providing the benefits of pooling to the biggest individual miners.  Smaller miners who now couldn't join pools would be subject to increased block-finding variance and so would either be more likely to go out of business, or would simply rent hash rate from a large miner who would operate the equipment and payout rewards---increasing centralization in either case.

They should use some kind of insurance instead of pooling :)


text:
Can an attacker create many nodes that approve blocks without work?
There are currently about 7000 nodes in the Bitcoin network. The security rests on the fact that someone adding nodes would have have to expend CPU power to hash transactions.
What prevents someone from creating a large number (e.g. 10,000 to 20,000 or more nodes) and then execute the following attack.
The objective of the bogus network would be to appear to anyone, except for the legitimate nodes, as producing legitimate verification of Bitcoin transactions without actually doing the work. While the legitimate network would know something is wrong; to those outside it would appear that a majority were approving the transaction.
The bogus nodes run a stripped down version of Bitcoin which accepts new blocks from other bogus nodes, but not new legitimate ones. The bogus software is designed to make the blocks look as real as possible, chain together, but without requiring the true proof of work. The nodes could be bots on compromised computers.
The nodes might do something like:
1) Gather any transactions sent to the network, including ones with invalid signatures and perhaps other invalid information, including thefts by the schemers.
2) The nodes in concert generate the block header and hash, by hashing once with some pre-aggreed upon nonce. They don't bother to do the repeated search.
3) All the bogus nodes accept the new block, send their acceptance out to the network and start creating the next block

You're essentially talking about a "fork" - or a separate chain. It wouldn't and couldn't interface with bitcoin - merely work (at best) alongside it. Exchanges, merchants, and official bitcoin wallets could not be fooled into using this alternative. Someone would have to download a compromised client. As each block refers to the previous one, there would be no way for them to "inject" fake blocks in the way.

Wouldn't a client generally not check the entire blockchain?
Do clients know the current level of difficulty, load recent block headers and rehash them to prove that the hash value is small enough to pass the proof of work test?

@JonathanHarris: Clients do indeed check all of this.  They also verify the signatures on all the transactions in the block.

There is no acceptance to send to the network. Every node makes its own, independent decision of what blocks to accept based on what chain contains only valid blocks and has the most proof of work. This is the entire rationale behind proof of work and the reason it works.
The rule is this:
1) Invalid blocks are always ignored. Blocks must have only valid transactions. Blocks must chain to another valid block that itself chains only to valid blocks.
2) Among valid blocks, the one with the most proof of work behind it is the one a client considers the tip of the blockchain.


text:
Is OP_CODESEPARATOR disabled in Bitcoin and if so why?
OP_CODESEPARATOR is used in the Bitcoin script language to mark the beginning of signature-checked data. However, the given operator never found its real practical use case. So, is the given operator disabled in Bitcoin or can it still be used? If it is disabled, has any form of attack been detected that can be implemented using the given operator? Signing is a critical part, so I'm wondering if there might be some weakness in the system when using this operator, or is it disabled (if it is disabled) just because no one uses it?

So, is the given operator disabled in Bitcoin

No.

can it still be used?

Yes. It is still supported in bare scripts (but non-standard there), in P2SH redeemscripts, and care was taken to keep supporting it in segwit and taproot script even (though the functionality changed somewhat in the latter). While it's unclear why the opcode existed in the first place (the best theory I've seen is to support delegation, though that broke when the scriptSig and scriptPubKey execution split was introduced), it's also not harmful to have it, and in practice it allows creating a script that needs signatures that can only be used in some of its execution branches but not all.

Signing is a critical part, so I'm wondering if there might be some weakness in the system when using this operator

The existence of OP_CODESEPARATOR in pre-segwit scripts does worsen the quadratic hashing problem those scripts exhibit, by permitting many more distinct signatures over the same data that need separate hashing. It has been proposed to remove that feature (only for pre-segwit) for that reason, though that idea seems abandoned at this point.
Note that that is a DoS concern for validation, and doesn't affect the security of individual signers. Wallets can always actively choose to use patently unsafe constructions (e.g. give out an address corresponding to an OP_TRUE). Generally the goal of script design isn't to rule out unsafe behavior, only to avoid footguns or incentivizing unsafe or non-private behavior. OP_CODESEPARATOR doesn't seem to cause these (probably because its use is too niche anyway), so while it's unlikely that such a feature would be added today, there is little reason to go through the tremendous effort of developing and building support for a consensus change to remove it just because its use is unclear.

Thanks Pieter, all the best!


text:
Does the used standard (BIP44, BIP49, BIP84) define the type of addresses used?
I have a few questions regarding wallets and BIP44/49/89 standards.
In many places I found that BIP44 is used for traditional addresses (P2PKH) and that the path is m/44'/.... Also, if we want another type of address, say P2WPH, then we use different standard, it this case BIP84 and the path m/84'/..., and so on...
One of the places I found it would be the following answer: link.
Second place: link. Andrew Chow wrote the following:

By this logic, the xprv and xpub prefixes used previously then
indicate that legacy addresses should be made. Hence BIP 44 shows
xprv.

So I started to think that way too. That is, the type of the address is determined by the standard used. So:

BIP44 - traditional addresses (P2PKH), XPUB (it's defined in BIP32 but used here), m/44'/... derivation path
BIP49 - P2WPKH-nested-in-P2SH, YPUB, m/49'/... derivation path
BIP84 - P2WPH, ZPUB, m/84'/... derivation path

That's what I thought until I looked at golang's implementation of bitcoin wallet. Namely, it says there that it uses only BIP44 standard. Since the given wallet is used (I assume) for all addresses (not just P2PKH, P2WPKH-nested-in-P2SH and P2WPH, but for all as many as there are), I got a bit confused and started to think differently about these standards and how they are used. So I have a two questions.
1. BIP44 is just a derivation path standard, it doesn't decide what type of address you have to generate from it? You can generate any "type" of address you want and that's just how the golang bitcoin wallet works? It uses the BIP44 derivation path, but leaf-level tree keys are used for all address types.
2. Although BIP44 can be used for all addresses (if the answer to the previous question is yes), there are also standards that define other derivation paths such as BIP49 and BIP84 whose purpose is only to separate keys intended for a certain type of address to other derivation paths, nothing but that? But even if, for example, BIP84 says it's for P2WPKH, it's just a derivation path, nothing else. I mean, you can also use P2PKH here (it's weird, but you can) since it's a derivation path? Actually, they just wanted to kind of point out that if you use P2WPKH then a different derivation path is used, but you can still use the BIP44 derivation path?

Have you tried reading those BIPs?

@PieterWuille Hello Pieter. Yes, I have read it, and from what I understand I would say the answers to my questions are YES. In BIP84 it says the following: "This BIP defines the derivation scheme for HD wallets using the P2WPKH (BIP 173) serialization format for segregated witness transactions." and "With the usage of P2WPKH transactions it is necessary to have a common derivation scheme." From that I'd say it's just a derivation path with purpose to separate a certain type of address-P2WPKH, although BIP44 can still be used. However, I'm not sure because of the answers I found on this site.

To answer both of your questions, each BIP is intended for generating a specific address type like 1, 3, and bc1q.
BIPs 44, 49 and 84 define standard derivation paths that all wallets should use for defining P2PKH (and P2SH), P2WPKH-P2SH, and segwit version 0 addresses P2WPKH and P2WSH respectively, and you also might be surprised to know that BIP 86 uses the m/86' prefix for creating Taproot addresses. So the standards weren't exactly designed with extended public keys in mind - in fact, Taproot uses xpub (and presumably Xpub for testnet, although this was not shown in test vectors), and not zpub, or ypub.
Testnet addresses and keys for all these are derived in the manner specified in BIP44.

First of all, thanks for your reply. But nowhere in BIP44 does it say that this standard is intended for P2PKH, but that it is a standard that only defines the derivation path. Also, since it was wanted to somehow separate the derivation path for P2WPKH addresses, then BIP84 was created that suggests using that path for a given type of address (it doesn't say anything specific about addresses, just "hey if you want P2VPKH use this path"). We can still use BIP44 and that derivation path for all addresses. This is how the golang bitcoin wallet works. So the answer to my questions is YES?

BIP44 was written at a time when P2PKH was the only reasonable type of address to use for singlesig constructions (P2SH-P2WPKH, P2WPKH and P2TR did not yet exist) so it was probably just not considered at the time it was written. I think these days, BIP44 is typically considered to be P2PKH implicitly.

@PieterWuille Yes, I agree, but in the end it's just a standard for the derivation path, even though that path is implicitly considered to be used only for P2PKH. I mean it can be used for any type of address, the golang bitcoin wallet implementation only uses that path. As far as I understand. However I guess in the end it is best to use output descriptors.

@Filip That's possible. Ultimately, every piece of software chooses what they implement, and if things are unclear, they are up to interpretation. In this case I think the letter of the BIP44 document and what it's commonly interpreted to mean are different, which can be confusing. That's not unique to BIP44 by the way - for example, BIP32 also specified a derivation path for keys which almost nobody uses (BIP44 etc use a different one), and when one says "BIP32", it's not commonly understood to refer to that derivation path.

@PieterWuille Yes, you are absolutely right. Especially for people who are just learning things. I guess you sensed how much confusion arose in me, but I guess that's part of the process. In any case, we could say that the answers to my questions are YES, even though the addresses are implicitly determined by using some standard and deriving paths from that standard?

@Filip By the letter of the BIPs, BIP44 is not specific to any address type, BIP49 is P2SH-P2WPKH, BIP84 is P2WPKH, and BIP86 is single-key P2TR. BIP44 is however commonly understood to be implicitly P2PKH. There is no one answer to what things mean, as technology and context evolves, and so does the understanding. You could be explicit and say "BIP44 derivation paths" if you mean just the derivation path proposed in it and not the rest of the document or what it entails.

@PieterWuille Yes, I agree, I think so too. Thank you. All the best.

Also you might see some wallets using the 44 path for legacy P2SH (the one used before segwit and P2WPKH-P2SH was invented) - slight inaccuracy in my answer. Sorry about that.

@Zenul_Abidin Yes, I got it and, for example, in the description of the golang btc wallet (https://bitcoin.stackexchange.com/questions/75165/what-is-bitcoin-locking-and-unlocking-script) it says the following: "btcwallet uses the m/44'/<coin type>'/<account>'/<branch>/<address index> HD path for all derived addresses, as described by BIP0044." Thus, it uses one derivation path for all addresses regardless of its type. And it's legit since this standard doesn't define the addresses you have to use here (although implicit P2PKH).

@Filip My guess is that that README entry was written at a time when btcwallet only supported P2PKH and BIP44, and no other address types, and has just not been updated since. Looking at the source code, I see mention of BIP49, BIP84, and BIP86 too.

@PieterWuille I haven't gone into the source code (yet), but then it looks like they also support other derivation path standards and separate addresses by them. Possible. Maybe they even support output descriptors then.


text:
What types of transaction malleability exist and which one of them does SegWit solve?
I am little confused about what types of transaction malleability exist in bitcoin and which one of them does SegWit solve.
Here Antoine Poinsot talks about txid malleability and third-party malleability and says that SegWit solves only txid malleability. In the comment on that question, Murch mentioned first-party malleability. Furthermore, Master bitcoin 3rd edition talks about third-party and second-party malleability. In this answer Pieter Wuille says that SegWit doesn't prevent malleability (that is, in many ways the actual transaction data can still be changed by third parties).
Looking at all this, I'm a bit confused.
What I do know is that the transaction malleability comes from the fact that the digital signature does not cover the unlock script field (scriptSig), so if a third party (or anyone) changes its content (for example, adds something extra to the stack that will not affect the unlock validity of the referenced UTXO) it is possible to change the transaction ID and therefore if this transaction is accepted instead of the original one, it may invalidate all transactions in the descendant chain. Of course, this is only possible if we are working with an unconfirmed transaction and referencing its unconfirmed ID. SegWit solves this by moving the digital signature from the unlocking script (input portion of transaction) to separated witness data and therefore digital signature is not the part of transaction ID. So referencing this type of unconfirmed transaction is fine since its ID can't be changed (at least not by this way).
So my questions are the following:

What types of transaction malleability exist and what situations do they represent?
What type/types of transaction malleability does SegWit solve?
If SegWit does not solve all types of transaction malleability, how these other types are overcome, if there is a solution for them?
What type of transaction malleability does the situation I described represent? (I guess I'll find out by myself from the answer to the first question)

As you have identified, there are some inherent ways in which a transaction can be changed without changing its outcome. This is especially problematic when it changes the txid for its downstream effect, but it can for example also be used to modify a transaction’s feerate by increasing the overall weight.
First, second, and third party respectively refer to the “creator of a signature”, “a co-signer of the tx”, and “anyone that has a copy of the unconfirmed tx”.
Third-party transaction malleability refers to ways that anyone can change a transaction. For example a third party can invert the sign on the s-value in an ECDSA signature, or add some padding to the signature. Such modifications will make a transaction non-standard, but still valid to include in a block. On a non-segwit input this changes the txid, because the entire transaction is hashed to produce the txid. Transactions that only use segwit inputs prevent changes to the txid from all sources of third-party malleability. ECDSA signatures on segwit inputs are still malleable, but this only affects the WTXID not the TXID.
Generally, the original sender (“first party”) can always change a transaction, e.g. by simply signing again or by creating a replacement transaction. Murch probably shouldn’t have called that “first-party malleability”, since the term “transaction malleability” is inspired by “malleability” in cryptography, which refers to others being able to modify signatures after they were created.
The term “second-party malleability” was newly introduced in Mastering Bitcoin 3rd Edition. It refers to a situation in which a co-author of the transaction (“second party”) modifies the transaction after the sender has signed. This sort of effect needs to be considered in the context of multi-party protocols such as Lightning, coinjoins, or UTXO pools.
Schnorr signatures are not subject to third-party malleability.

Malleability is much wider than the class of problems second-layer protocols care about, and can even be desired. For example, Bitcoin transactions have sighash flags which allow signatures that e.g. only cover the outputs or inputs of a transaction - leaving the non-covered parts explicitly open for change after signing. Also scripts could choose to just take an input that gets ignored. What segwit did is make universally sure that transactions for which changes to the transactions which do not change its effect, also do not change the txid.

@PieterWuille What do you think about transaction malleability types defined by Antoine? It looks a little bit strange to me since TXID malleability is also a third party malleability.

In general, when we're concerned about txid malleability, it's not third party malleability (as generally, for typical transactions, with BIP62/BIP66 enforced, third parties cannot modify transactions without invalidating them), but concern about another signer re-signing. That's called second-party malleability in mastering bitcoin 3rd edition, but it's debatable whether that should be called malleability at all.

@PieterWuille Right, but generally speaking TXID malleability is also a third and second party malleability. I mean, they are both about changing transaction IDs. So Antoine in some way wrote it wrong. And segwit solves both of this malleabilities (as we already discussed before). Right?

@Filip The second class in Antoine's answer is not about txid malleability, but transaction malleability (as in second or third parties can change the transaction in general, even though in a way that doesn't change the txid). Such malleability is still a problem for some use cases post-segwit, though a much smaller one.

@PieterWuille Al right, thanks Pieter.


text:
Regtest automated peer discovery
I want to test peer discovery in regtest mode of bitcoin core, running in Docker.
However, I noticed in the source code of bitcoin core that there is a check if the address is routable before answering to a getaddr message. So I first create a network with routable addresses (IsRFC6598):
docker network create -d bridge --subnet 100.64.0.0/10 bridge_sharedrange

Then I launch four Docker machines with bitcoind installed (and using the bridge_sharedrange network, with different /16 addresses). The bitcoin daemon is then started in all of these four machines with:
bitcoind -regtest -daemon

Running ifconfig on these nodes show that they are running on 100.64.0.2, 100.65.0.2, 100.66.0.2, 100.67.0.2. From the first machine (100.64.0.2) I then execute the command:
bitcoin-cli addnode 100.65.0.2 add

From the third machine (100.66.0.2) I then execute the command:
bitcoin-cli addnode 100.65.0.2 add

So basically, the node with address 100.65.0.2 now has two peers (checked with bitcoin-cli getpeerinfo). I would expect now that

100.64.0.2 learned about 100.66.0.2 &
100.66.0.2 learned about 100.64.0.2

But that is not the case. They only know 100.65.0.2. Is this normal behaviour? I read on https://en.bitcoin.it/wiki/Satoshi_Client_Node_Discovery#Command_Line_Provided_Addresses that it might have to do with the timestamp of the addr, but I'm not certian this can be the cause, and how can circumvent that.
BTW, I also tested with
bitcoind -regtest -daemon -discover=1

Update: It seems this feature of discovering local LAN nodes is under discussion: https://github.com/bitcoin/bitcoin/issues/3802 , however I would think that the WAN rules apply to my situation, as I use RFC6598 addresses in my test set-up.

Bitcoin Core does not immediately broadcast a new addr message to nodes with each new connection it gets. On average, those broadcasts are every 30 seconds, but they could happen sooner or later than that as the delay follows a Poisson distribution. You should wait a few minutes before checking again. Furthermore, just because they know about the address does not mean that the nodes will actually choose to connect to them.

What are the checks a node does to 'choose' to connect to another node, or is it entirely random? I think it has something to do with both the protection against the sybil attack (https://en.bitcoin.it/wiki/Weaknesses) as well as the fact that not all addresses have a valid timestamp (https://en.bitcoin.it/wiki/Satoshi_Client_Node_Discovery).
However, I tried giving the nodes a different /16 address, and the issue remains (even after 30 minutes of waiting).

The nodes are selected randomly and must not meet the conditions checked here: https://github.com/bitcoin/bitcoin/blob/master/src/addrman.cpp#L33

Interesting, this led me to information about the Eclipse attack.  https://medium.com/mit-security-seminar/eclipse-attacks-on-bitcoin-s-peer-to-peer-network-e0da797302c2 and https://eprint.iacr.org/2015/263.pdf provide more detail. It seems it's not that easy to test peer discovery after all.


text:
Balance between minimum transaction fee and time using Electrum 3
Yesterday I made my first Bitcoin transaction using Electrum and I was able to configure the fees I was going to pay. As it was a non-urgent transfer I selected the minimum fee Electrum allowed me: "Within 25 blocks", the lowest sat./byte available from the slide bar.

If the transaction was supposed to be included in the next 25 blocks, I thought it will take around 250 minutes (~4hours) to be processed. But I don't know why, after 2hours the transaction already had 6 confirmations.
This minimum fee ended up being 0.0008 BTC (~10 € as the time of writing, which was ~14% of the sent amount).

Why did the transaction went so fast if the fee was for 25 blocks?
As it was a non-urgent transaction, couldn't I have selected a smaller fee in any way? I couldn't find a way to pay smaller fee in Electrum 3 --I could have waited for 24h with no problem to avoid this high fee.
Does this means that I paid much more for the fee than I could have paid, given the non-urgent requisition of my transfer?

It's all about bitcoin, the Electrum wallet is just a wallet, it doesn't minning block nor building block.

Why did the transaction went so fast if the fee was for 25 blocks?

You can't predict in which block exactly your transaction will be added.
The block is filled with arbitrary transactions. *

 * While larger fees improve the chance to be included in the block there is still have a chance to be added to the next block.

As it was a non-urgent transaction, couldn't I have selected a smaller
  fee in any way?

You can select any fees what you want(and 0 fees also), but there are no warranties what that transaction will be added to the block. Transactions that pay more fee take precedence.

I couldn't find a way to pay smaller fee in Electrum 3 --I could have
  waited for 24h with no problem to avoid this high fee.

You can set commision manually in the Electrum settings.

Does this means that I paid much more for the fee than I could have
  paid, given the non-urgent requisition of my transfer?

Transactions (with a small or zero fee) which stay unconfirmed for a longer time usually don't get priorized over transactions with a high fee.
Every miner is free to chose which transactions he is going to confirm in his mined block. Its all about how the node is coded to fill the block with TX's.
Since miners mine because they want to gain profit, they will always priorize transactions with a higher sat/B fee.
From me:

What happens with transaction with zero or low fees?

Generally, after a few days, the transaction will be removed from the mempool and your wallet shows your balance again, which makes your coins
spendable again via wallet.

Thanks, I didn't know I could set the fee manually checking it in "settings". Then I probably could have checked this: https://bitcoinfees.earn.com/
and choose a more convenient value.


text:
Website with trading and Ripple wallet?
I plan to start crypto trading with Ripple and other coins.
What litebit.eu type of site you recommend which support Ripple and many other coins and has wallet for Ripple too? If i use litebit i need to have gatehub too and i would like to have only one website.
It's hard to find such site because that would require to create account to all of these sites. :-(
That's why i ask here if anybody knows such site.
Thanks

I know Poloniex and Bittrex is good, i use Poloniex my self to trade currencies.. http://coinmarketlist.com/ <-- maybe there you find some other exchange platforms.. there are lot of platforms out there.
About wallet i don't know :(

Exchanges that are trading Ripple should at least have wallet support and allow you to trade XRP. Listing can be found at https://www.coingecko.com/en/coins/ripple/trading_exchanges#panel
Other than that best to run your own wallet and control your private keys to store large XRPs