text:
Running a local testnet
a newbie bitcoin researcher here.
For my first work I need a local bitcoin network on which I have as much control as possible. I have a windows machine.
For example - I need to control the number of nodes, network structure, mining difficulty and preferably any other parameter there is.
I downloaded bitcoin-core, installed it and ran its test mode, but it still asks me to download all blockchain history.
I had in mind a new bitcoin-core blockchain built from scratch, so I can control and understand everything.
Is it even possible, using the bitcoin-core software?
If not, how can I do this?
Thanks

Hi michaell_co, you might find this related topic interesting: https://bitcoin.stackexchange.com/q/17553/5406

Perhaps instead of running testnet you would prefer something like regtest? In order to do this, you can either run bitcoind like this:
bitcoind -regtest or add regtest=1 in your bitcoin.conf file instead of testnet=1. This will allow you to spin up a private blockchain and you will have control over when blocks get mined


text:
bitcoind 23.0 crashed mysteriously after 2 hours, what do I do next?
My new bitcoin node ran for about two hours and then crashed for no apparent reason. What do I do next to debug or fix the problem?
This is my first time running a bitcoin node. I have lots of experience running Linux services in general. I started by doing a fresh install of Ubuntu 20.04.1 on an laptop computer with 8 G of RAM and a 1TB hard drive. I downloaded the bitcoin core 23.0 .tar.gz file and extracted it into /opt . I created a separate "bitcoin" user without sudo privileges in order to run bitcoind. I forwarded port 8333 from my router to the laptop running the bitcoin node. I launched bitcoind as user "bitcoin" by running bitcoind -daemon with no custom configuration.
I watched as bitcoind started the intial block download. I could see it was using a bunch of bandwidth as expected, and waited until it had downloaded a few gigabytes of data onto disk. Then I went to bed.
The next morning I expected to see bandwidth still being used and a lot more blocks stored on disk. Instead, I saw no bandwith being used, and bitcoind was no longer running. The .bitcoind/bitcoind.pid file was still there but the process with the ID in that file was no longer running. The laptop computer is not configured to "go to sleep", and my SSH session from another computer was still connected and had not been interrupted. About 34 G of disk space had been used by the downloaded blocks before bitcoind crashed. I saw a .bitcoin/debug.log file with size 79M and last modification time about two hours after I had started bitcoind. I tailed that file, but saw no error message, only what looked like normal log messages. This is the last line in debug.log:
2022-08-28T05:11:25Z UpdateTip: new best=000000000000000012ef16fc2f2c7f6752a6affa988d98098f837260de9a8989 height=342602 version=0x00000002 log2_work=82.191226 tx=59049857 date='2015-02-08T21:31:10Z' progress=0.077785 cache=716.8MiB(5441897txo)

My guess is that bitcoind hit a bug and exited abnormally. These kinds of problems are a pain to debug and my sympathies go to the developers.
I am willing to take steps to help debug this problem if someone is willing/able to give me some guidance. For example if someone could point to an (official!) debug build that would render a stacktrace (I notice that the bitcoin executables are all stripped). And/or instructions to  run a debug build using gdb to capture the crash. Or maybe some extra debug parameters I could run with to log more info - if someone is interested in looking at the results.
Otherwise, if no debug guidance is available, would y'all recommend I downgrade to a non-crashing version? or wait till the next release?
Update 1: I mentioned above that no, my laptop computer did not go to sleep.
Update 2: As I mentioned in a comment below, I'm pretty sure I did not run out of memory. The machine has 8G of RAM, nothing else was running, and when I checked the log for any reported OOM kills by the operating system, there were none. I did see in the logs that at boot time the OOM killer was started.
Update 3: As suggested in a comment below, I tried running in foreground mode with debug logging turned on: bitcoind -nodaemon -debug . I did this on a fresh bitcoin data directory, to restart the initial block download from scratch. This time it ran only for an hour an a half before crashing, with this message:
2022-08-30T20:56:45Z UpdatedBlockTip: new block hash=0000000000000000175471627926fa912a59576262795b54ebd961c3167580c3 fork block hash=0000000000000000203dada33f89ef12aaa3995c390e0f575cf5d80901b79b49 (in IBD=true)
Bus error (core dumped)

You can try with -nodaemon -debug to increase debug output, and leave it running in the foreground. If the OS then kills the process, you can see why. Is it possible your system just ran out of memory?

@PieterWuille Thanks for the suggestion. Hopefully a developer would look at the debug log if I generated it. Without debugging, it was 79M, so with debugging it will be huge.
Regarding running out of memory - the machine has 8G of RAM, nothing else was running, and I checked the log and verified there was no OOM kill reported by the operating system. So no, it did not run out of memory.

I think I found a likely cause of the crashes. I saw some suspicious log messages:
[ 6067.607405] blk_update_request: I/O error, dev sda, sector 155782528 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0

This made me start to suspect disk hardware errors. So I ran this check:
$ sudo apt-get install smartmontools
$ sudo smartctl -H /dev/sda
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-46-generic] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
Failed Attributes:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   001   001   050    Pre-fail  Always   FAILING_NOW 16376

... and there we have it. Disk failure. All bets are off once the hardware starts failing.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How to Generate P2SH Address from XPUB Ledger
I'm trying to Generate unused receiving Addresses for a Segwit P2SH wallet in my Ledger. I tried https://pypi.org/project/bitcoinlib/ but without success, the addresses weren't right.
This is from my Ledger:
{
  "xpub": "xpub...",
  "index": 0,
  "freshAddressPath": "49'/0'/0'/0/6",
  "id": "js:2:bitcoin:xpub...:segwit",
  "blockHeight": 753361
}

Thanks in Advance.

Ledger is using incorrect version prefix. Most libraries/wallets/online tools use ypub format for P2SH address
To make this work you can use any of the following tools to convert you xpub to ypub:

https://www.blockonomics.co/views/segwit_xpub_convert.html
https://jlopp.github.io/xpub-converter/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How to determine which input address paid the fee?
I'm new to btc and have some experience on blockchains like ethereum. I understand that btc is based on utxo not balance model like other blockchains, but I have trouble finding which input address paid the tx fee, for example, this tx has two input addresses but which one of them paid the fee. This kind of confused me since on other chains finding fee payer is relateively simple and obivous.

This question simply makes no sense in the UTXO model. All that happened is that two UTXOs were spent, two new UTXO were created, and a fee was paid by the difference in value between them.
If you're asking which UTXO paid for it, I think the answer is "none", because UTXOs are not actors, or identities. They're just coins. Someone certainly paid the fee, but that's a question of identity or ownership, or perhaps even intent. And that information simply does not exist on chain; for your specific transaction, possibilities include:

Perhaps both input UTXOs were owned by the same person, who created the transaction, and paid a receiver, and sent change back to themselves. In that case, this sender paid the fee.
Perhaps the input UTXOs were owned by distinct people, who jointly paid a bill, and sent some coins back to one of them. The question of who paid the fee is now a function of what portion of the bill paid was allocated to each sender - not information that exists on chain.
Perhaps the receiving party of the transaction offered a service that involves paying the on-chain fees, by allowing the senders to deduct that value from the transferred amount. In this case, the receiver paid the fee.
Perhaps the transaction was created by the miner that constructed the block. As the miner is the one collecting the fee, arguably no fee was paid by anyone in this case.

Thanks for you answer.  It seems I didn't fully understand the utxo model before. Your explanation is very detailed and useful.


text:
Stop and Restart Service while BTC Core Syncing?
Is it possible to start and restart the Bitcoin Core service during the syncing process without loss of work?
What happens to syncing if the Server fails or shuts down, i.e. a power failure?
I would like to put it on a UPS but that requires stopping and restarting the BTC Core service. I am running the service on an Embassy One server

Bitcoin Core saves the current chain state to disk, so you can safely shut it down and it will continue where it left off after you start it again. Power failures are a bit more complicated because your disk can become corrupted, and I can find several posts from people experiencing problems with Bitcoin Core after their power went out, so I wouldn't assume it's safe against power failures.

thanks! I am going to install the UPS.


text:
Online wallet to redeem pre-fork paper wallet for both Bitcoin and Bitcoin Cash
I have BTC (pre-fork) and now BCH on a paper wallet. What's the best online wallet to transfer them into to ensure I retain both coins and also to be able to transact with them?

This is copied from  the Electrum website today:
Recommendation on how to redeem Bitcoin Cash
July 31st, 2017.
Electrum is a Bitcoin wallet created by Thomas Voegtlin in 2011.
Electrum is distributed by Electrum Technologies GmbH, a company
registered in Germany, using the website https://electrum.org
Electrum users who are not interested in Bitcoin Cash do not need to
worry nor to take any particular action. The following note is only
for users who want to access their Bitcoin Cash (BCC).
"Electron Cash" is a fork of Electrum for Bitcoin Cash. Electron Cash
is not endorsed by Electrum. It is open source, and binaries
(executables files) are available for Windows, OSX and
Android. However, when you run binaries instead of source code, you
have no guarantee that they match the source code. This is why wallet
binaries are usually signed by developers. A digital signature engages
the responsability of the person who signs.
The person who distributes the Electron Cash binaries has decided to
remain anonymous, and uses the fake name "Jonald Fyookball" in order
to sign Electron Cash binaries. Thus, if these binaries contain code
that is designed to steal your bitcoins, the author of the theft will
be anonymous and walk away safely with your funds.
This danger is exacerbated by the fact that the default behaviour of
Electron Cash is to silently copy all your Electrum wallets into its
own directory. Thus, if you run Electron Cash on a machine where you
previously used Electrum, all your pre-existing wallets will be
available in Electron Cash, and you will only need to enter your
password in order to expose your bitcoins to potential theft.
Therefore, users who want to run the Electron Cash binaries should do
it on a separate machine, that does not have their Bitcoin wallet. We
recommend to proceed as follows:

Install Electron Cash on a machine that does not have your
Electrum wallets.
Wait until the BCC hard fork has taken place, and a few BCC blocks
have been mined.
Move all your Electrum funds to a new Electrum wallet. This will
move only your BTC, and not your BCC, because the BCC blockchain has
replay protection. Wait until the transaction is confirmed.
Enter the seed of your (now empty) old wallet or private keys in
Electron Cash. Since the BTC have been moved to a new wallet,
entering your old seed in Electron Cash will not put your BTC funds
at risk.

Following these 4 easy steps you will be able to access your BCC
without compromising your BTC.

I would add "move your BCC to cold storage then wipe and restore the machine you used to run Electron Cash since you consider it possibly infected.

Easiest way is to import the same seed into BCH's Electron Cash wallet (an open-source fork of BTC's Electrum wallet) and then move BCH to a new wallet. You can first move the BTC to a new paper wallet, and then repeat the process for BCH.
Since 2017, Electron Cash has evolved into the most advanced open-source wallet for BCH: it supports importing all popular seeds and derivation paths, coin control, multisig, cold storage (offline signing), Ledger hardware wallet(s), CashFusion privacy, BCH's L1 native tokens ("CashTokens"), Satochip hardware key, and plugins like Flipstarter decentralized crowdfunding.
The other answer was written in 2017 and warned against running binaries signed by an unknown pseudonymous developer "Jonald Fyookball". It was a good word of caution at the time, but now the wallet has accumulated a solid 7-year long track record, and a realname developer is now signing off releases, so maybe it now deserves a little trust (but still, verify!).
The wallet now has reproducible builds which mitigate the risk of binaries not matching the (open) source code. Also, it is possible to download builds automatically performed by Github, without having to trust a particular developer (but then you're trusting Github's build process).
Also, many people have been participating in Electron Cash's wallet development, as evidenced by Github pull requests, and many users have been using the wallet without serious issues, as evidenced by Github issues.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Difficulty Adjustment Algorithms Comparison
Thinking about different difficulty adjustments for protecting new coins against attacks from big miners. 2017 when BCH forked from BTC, BCH added the EDAA (emergency difficulty adjustment algorithm) and later updated to another DAA (difficulty adjustment algorithm). Another approach of protecting the blockchain is DigiShield, the difficulty adjustment algorithm that is used by Dogecoin.
What is the difference between DigiShield and the DAA of BCH?
How do they differ from Bitcoin's original DAA?

Bitcoin DAA
Bitcoin's original difficulty adjustment algorithm (DAA) uses a very simple method:
If (height + 1) % 2016 != 0 then the next block's target will have the exact same target.
Else, the next block's target will be adjusted by a factor of time_difference / expected_time_difference of the 2016-block period, where time_difference is established simply by subtracting the beginning and ending timestamps, and max. adjustment is limited to -75% or +300% (a factor of 4).
Bitcoin Cash DAAs
Since then, Bitcoin Cash fork happened and it went through 3 DAAs:

Emergency DAA (2017), initial fork DAA, which was gameable by miners to accelerate block production and had to be changed soon after,
CW-144 DAA (2017-2020), which managed to maintain block time average but had a problem of difficulty oscillations due to miners just following their immediate incentives and switch-mining BCH/BTC with some of their hash-rate.
ASERT DAA (2020-now), which is stable and has performed well since 2020.

We will below discuss them all in more detail.
The Bitcoin Cash fork started with the Emergency DAA (EDAA), which was a simple modification to the original DAA:

In case the MTP of the tip of the chain is 12h or more after the MTP 6 block before the tip, the proof of work target is increased by a quarter, or 25%, which corresponds to a difficulty reduction of 20%.

done to enable a minimum viable fork. However, it performed horribly because miners could game it to speed up block production rate, which they did, and consequence was faster blocks on average, which resulted in BCH blockchain height getting ahead of BTC for about 2k blocks during the 6 months period of that DAA being active.
To fix this, the (now former) Bitcoin ABC team implemented CW-144 DAA and activated it in November 2017 network upgrade, and it was successfully maintaining average block time, but it had a problem of oscillations from miners switching some hash-rate back and forth between BTC and BCH to maximize their profitability. The problem was present because it was based on a simple moving average (SMA) where samples that "fall out" of the sampling window have as much impact as samples that get added, as discussed in this write-up:

In brief, the oscillation is due to cw-144 using a simple moving average (SMA) over the last 144 blocks to estimate hashrate. When e.g. a high-hashrate block leaves the SMA window, it reduces the difficulty by the same magnitude as a new block entering the window would increase it. Blocks leaving the window therefore sharply change the difficulty in a direction that incentivizes miners to repeat the hashrate of the departing block. I call this effect "hashrate echoes." Notably, these echoes do not require any malicious miner behavior to occur. They are merely the result of miners following the direct short-term incentives that the DAA gives them.

Finally, as a result of further research, "absolutely
scheduled exponentially rising targets" (ASERT) DAA was developed.
Led by BCHN team, BCH activated it in November 2020 network upgrade.
The ASERT DAA has performed well ever since and is the currently active BCH DAA. It belongs to exponentially-weighted moving average (EWMA) family, which independent DAA researcher known as zawy12 regards as "the only difficulty algorithm people should use":

WTEMA is an "infinite impulse response filters of 1st order". It's ideal in many situations. It very closely approximates relative ASERT by replacing e^x with 1+x which is very accurate for small x. Absolute ASERT is slightly better than relative ASERT because it prevents error caused by e^x function being imperfectly estimated and nBits imprecision. The error is linearly worse with larger N.

Note: WTEMA stands for weighted target exponentially moving average.
Later, an academic paper:

Kawaguchi, K., Komiyama, J., Noda, S., "Miners' Reward Elasticity and Stability of Competing Proof-of-Work Cryptocurrencies" (August 27, 2022)

further confirmed that the design is solid, and in fact beneficial to BTC as well since BCH can serve as a shock absorber for BTC:

Our results indicate that Bitcoin's stability heavily relies on external factors such as historical low hash-supply elasticity and interactions with other cryptocurrencies. Interestingly, despite their smaller size, Bitcoin's forks and their efficient difficulty adjustments significantly contribute to Bitcoin's stability. These findings imply that an algorithm upgrade to stabilize Bitcoin is highly recommended.

We can observe properties of all 4 DAAs by plotting historical blockchain data:

It is easy to spot the Emergency DAA anomaly.
Also, we can observe the oscillation effect of CW-144 in anomalous daily medians.
BCH now attracts only about 0.5% of available sha256d hash-rate, and increased variance is a consequence of that, and it is noticeable in the above plot.
I is interesting to examine the impact of BTC and BCH halvings on BCH block time:

where we can observe impact of sharp change in profitability causing miners to move some hash-rate from one chain to the other.
DigiShield DAA
We can again refer to zawy12's research, where he states the following about DigiShield v3:

This discusses the start-up and MTP delay problems in Digishield (e.g. Zcash). It works good, but it has two problems that arguably make it not a lot better than an SMA with N=75. "Better" means that for a given level of stability during constant hashrate, it responds faster to sudden increases in hashrate. With MTP and 16%/32% removed, and safe timestamp handling in place, it easily beats SMAs. With N=17 and a tempering factor of 4x in the normal Digishield, an SMA with N=17x4=68 has the same stabilit but slower response.

Just because someone has made something and given it a name doesn't mean that it isn't utter garbage.  Don't trust any review or analysis from anyone that can't give you clear and well reasoned list of criteria that it is necessary and sufficient for a difficulty update algorithm to achieve.
Some of the fad options that altcoins blindly copy don't even achieve the basic property of achieving the target block interval when hashrate is constant.
Instead they seem to obsess about falling back to the nominal rate when hashrate suddenly spikes, ignoring that slow blocks are a safe failure mode and completely failing to consider the security side effects of their decision. We've seen at least a few hyped difficulty update changes lead to actual attacks, not just theoretical weaknesses.
I have no doubt that its possible to make something that is pareto-superior to bitcoin's non-overlapping rectangular window, but I am equally sure that essentially all the other things you see hyped fail to do so. This is unsurprising because instead of starting with a meta-analysis to understand what properties are useful, they all seem to have been created by changing things and analyzing only one or two properties (or not even analyzing things at all).
Keep in mind the primary purpose of most technical changes in most altcoins is to create marketing bulletpoints that can be used to convince unsophisticated speculators to pay for the altcoin.  This marketing requires a lot of hype-- far in-excess of whatever hyper a theoretical 'perfect' difficulty update would deserve-- but not a lot of substance.

They used this simulator decide on a DAA for Bitcoin Cash. Although DigiShield is not a part of the DAA collection of it, a review of it can be found here and here.

Thank you very much for your answer. But what you think is better DAA or Digishield ?


text:
Post-quantum preimage resistance of HASH160 addresses
It is well understood that quantum computers could make finding 256-bit hash-collisions feasible, and that they could break elliptic-curve public key encryption currently used in Bitcoin. It is also well understood that 160-bit addresses are not collision-resistant, however that is not really a problem for common applications (P2PKH).
What about addresses where the public key was never revealed, would they be safe at rest? Currently HASH160 addresses have 160-bits of security against preimage attacks, but QCs could theoretically bring it down to 80 bits which would be in the "danger zone".
A successful preimage attack would not necessarily yield the original public key, but in the scenario where QCs would be capable of cracking 160-bit hash preimages then they could also trivially find a private key of whatever public key the preimage attack would yield, so some sort of constrained-size quantum preimage search would do?
The only literature I found discussing this is this multi-breakage section in Giechaskiel, I., Cremers, C., Rasmussen, K.B. (2016). On Bitcoin Security in the Presence of Broken Cryptographic Primitives.

It is worth noting that modern cutting edge quantum computers that would be able to handle such computations efficiently have not even acquired up to 100 qbits (source: https://www.youtube.com/watch?v=d_5u2qdKoUU) and at this barrier they cannot even break low bitlength RSA ciphers. There is probably a REALLY long time before this becomes a relevant threat, nevertheless I love the idea to protect against it. So go on with that please.

We could design a black box function to break both P2PKH and P2SH (and P2WSH, etc.) addresses in 2^80 single-threaded quantum computer cycles.
Assuming a clock speed on scale of GHz, this would take about 10 million years.
Important to note is that splitting the work and doing it in parallel is not as beneficial as with classic computers because it would offer only a quadratic speedup (Fluhrer, S., Reassessing Grover's Algorithm).
In other words, doing the work in 1 year would require building 100 trillion quantum computers because sqrt(100T) == 10M.
Therefore, we can say that breaking a 160-bit hash preimage is physically possible because 10M years is a finite amount of time and less than age of the universe.
However, it is still infeasible.
Breaking P2PKH
Output locking script template is:
OP_DUP OP_HASH160 OP_DATA_20 pubkey-hash-20 OP_EQUALVERIFY OP_CHECKSIG
so cracking it with a hypothetical quantum computer would require running Grover's algorithm to find some x such that hash160(x) == pubkey-hash-20.
Once we'd find x, we'd then apply a much easier Shor's algorithm to find the secret key.
The revealed keypair would most likely not be the original keypair, however it wouldn't really matter since the key authentication part of the Script would evaluate to true for any key that would satisfy hash160(x)=pubkey-hash-20.
Breaking P2SH
Output locking script template is:
OP_HASH160 OP_DATA_20 redeem-script-hash-20 OP_EQUAL
however the actual redeem script will be evaluated after authentication against the hash, and it must also pass validation.
Therefore, finding any random x such that hash160(x) == redeem-script-hash-20 won't do, because it will most likely be an invalid redeem script.
To address this, we could perform a Grover's search for a particular template.
We'd do that by designing a function like f(x) = 0x21 || x || 0xac and then crack the composite function hash160(f(x)) as our black box function.
The raw bytes appended with x in f(x) definition will make our redeem script match the pay-to-public-key (P2PK) template:
OP_DATA_33 x OP_CHECKSIG.
With Grover's algorithm we'd find x that satisfies hash160(f(x)) = redeem-script-hash-20, and then we'd apply Shor's algorithm to crack the secret key of x.
Finally, we could then spend the funds with the public key, signature, and the P2PK redeem script (redeem-script=f(x)).
The discovered redeem script would most likely not be the original redeem script, however it wouldn't really matter since the redeem script authentication part would evaluate to true for any redeem script that would satisfy hash160(redeem-script)=redeem-script-hash-20.
Afterthoughts / potential optimization
I think we could optimize it by embedding public key generation as another function in the composite so we won't need Shor's algorithm at all, because we'd find the secret key directly:

Define a function g(x)=p that maps a secret key to a public key.
Run Grover's on hash160(f(g(x))) = redeem-script-hash-20

some matching x will be a secret key that satisfies all the requirements, then we compute g(x) to get the public key, then f(g(x)) to get the redeem script, which will be acceptable way to spend because the hash will match that of the address.


text:
Why wrapping P2PK inside P2SH didn't catch on?
Call it a pay-to-public-key-script-hash (P2PKSH), like so:

Locking script:
OP_HASH160 OP_DATA_20 redeem_script OP_EQUAL
Redeem script:
OP_DATA_20 pubkey OP_CHECKSIG
Unlocking script:
<push signature> <push redeem_script>.

Either way, wallets would have to process a list of keys and compute a 20-byte hash and address corresponding to each key, but address generation would be slightly different.
Total size of prevout + input would be the same in either case:

P2PKH address (25 bytes):
0x76a914 || hash160(pubkey) || 0x88ac
P2PKSH address (23 bytes):
0xa914 || hash160(0x21 || pubkey || 0xac) || 0x87

Spending input script would also be slightly different:

P2PKH spend (66 + 34 bytes):
<push signature> <push pubkey>
P2PKSH spend (66 + 36 bytes):
<push signature> <push (0x21 || pubkey || 0xac)>

Note: || indicates byte concatenation.
Both would require support by applications such as wallets, and both would have the same transaction (input+output) size.
The P2PKSH option has some advantage in that it would make the address relatively smaller and in that it would make all 20-byte outputs indistinguishable from each other.
It wouldn't make sense to start using this kind of address now, but there was a window of opportunity in the period 2012-2016 so I wondered. If Bitcoin had shipped with P2SH right from the start, would we have only 1 address type?

P2PKH had been around since the inception of Bitcoin and was therefore the dominant address type. When P2SH was introduced, it required wallet developers to implement both the parsing of P2SH addresses and the method of spending them. There's no benefit to using P2SH-P2PK, so it doesn't make sense for developers to implement it and users to use it.
This is in contrast to Segwit v0 and Taproot, both of which introduce new single key address types, but have benefits for using them. We've also observed that switching to new address types can be difficult and is a long and slow process. Given that P2SH-P2PK would have a similar issues with getting more usage, and that there is no benefit to using it, it's clear why P2SH-P2PK never caught on.

Wasn't it all still small enough in 2012 so that it would've been possible to transition, with the argument of preseving privacy and having all types of outputs hide behind the hash? Was it ever discussed back before the idea of SegWit existed?

I do remember reading about the idea of P2SH-P2PK, but it was several years after 2012; possibly even after segwit, so it's possible just nobody realized the possibility, or they didn't care about publishing the idea. Privacy isn't much of a benefit, as the redeemscript gets revealing on spending anyway (something that didn't change until P2TR). And despite the fact that scriptSig+scriptPubKey size for P2SH-P2PK is equal to that of P2PKH, it actually isn't financially advantageous for the receiver, as all they're doing is making it cheaper to send to them.

That's right, it'd make spending costlier which would make a difference if you had lots of UTXOs you wanted to merge. Also yeah, privacy benefit is only for outputs at rest. Oh wow I looked up the string "P2SH-P2PK" and found that Armory actually supported those kinds of outputs! but that release was around the same time when SegWit was already a thing.


text:
What are the Trade-offs for Input Script Deduplication?
One possible optimization to compress bitcoin transaction size is avoiding repetition of redundant data between inputs.
As a consensus change, a new transaction type could allow signing a list of txid:vout pairs if they all share the same output script.
To pick an extreme example, there are 1.6 million outputs with script 1HckjUpRGcrrRAtFaaCAUaGjsPx9oYmLaZ. Spending one of these P2PKH outputs costs 40+72 vBytes (prevOut + scriptSig) per input. By aggregating the scriptSig across inputs, the transaction could omit 72 per input and only include it once per transaction. When consolidating all these outputs, an optimization like this could save network data on the order of 115 MvBytes. (At a fee rate of 10sat/b, this would translate into 11.5 BTC, which is about 80% of the aggregate value of those outputs)
What are the trade-offs and hidden problems of this sort of optimization?

Could you please amend the title to summarize the gist of your question?

May I ask you to adapt the name of this idea? I haven't heard it previously, so I assume you thought of it yourself. The problem with "Cross-Input Script Aggregation" is that A LOT of people will confuse it with Cross-Input Signature Aggregation which has been around for a while (the idea and name). I would also say that you are not describing aggregation, rather de-duplication. May I suggest "Input Script Deduplication"? I think ISD is not a reserved abbreviation in this space :)

@fjahr sure no problem

The problem with this approach is that it subsidizes address reuse and therefore hurts privacy.


text:
How is the Prev_block_id managed?
I found this schema.
I wonder how organize the relationship with prev_block_id(prev_hash)?
I think it would be like a below schema.(Here  limited to id(hash))
CREATE TABLE block (
    id VARCHAR(8) PRIMARY KEY,
    prev_id VARCHAR(8) NOT NULL,
    FOREIGN KEY (prev_id) REFERENCES block (id),
);

However, this schema would not be able to be INSERT due to self-reference constraints.

Which database engine vendors do you intend this to be able to run?

Sorry for the lack of explanation, I meant Postgres.

It looks like in postgres this is done via ALTER?

alter table block add column prev_id integer not null references block(id) - did you try this?

Haven't run it, but good idea.
In other words, don't put the references in the schema at first, and then run the ALTER statement at the time the genesis block is born, and create the prev_hash - hash constraint, right?

no, run the ALTER statement right after CREATE, and INSERT genesis as a self referencing block. Boom!!!

I ran your suggestion as a test, but Error saving new block: DatabaseError(ForeignKeyViolation, "insert or update on table \"blocks\" violates foreign key constraint \"blocks_prev_block_id_fkey.
As you suggested, CREATE(create id only) -> ALTER(ADD prev_block_id) is the same thing as the CREATE statement I posted in the beginning, what is the purpose?

um you gotta try inserting genesis and then set the constraint afterwards, it'll surely work

alter table block add constraint fk_prev_block_id foreign key (prev_block_id) references block(block_id)

Your method finally solved the problem. Thank you！

I’m voting to close this question because it's a database design/administration question, unrelated to Bitcoin.

Because you want to do this in PostgreSQL[which version?], this is what should be working for you:

Create the table.
Insert genesis as a self referencing block.
alter table block add constraint fk_prev_block_id foreign key (prev_block_id) references block(block_id)

Other engine vendors may require more simple or more complex code.


text:
What are rw_settings?
Here is the beginning of my log (v26 of the Bitcoin Core) on Windows :
2024-01-21T04:50:15Z Ignoring unknown rw_settings value onion-prev
2024-01-21T04:50:15Z Bitcoin Core version v26.0.0 (release build)
2024-01-21T04:50:15Z Qt 5.15.10 (static), plugin=windows (static)

Do you know what means the first line ? I can't find any information.


text:
Next Bitcoin Core client version
I'm currently using the 27.1.0 Bitcoin Core client version and I have a question about the next version.
Will it be the 27.2.0 or the 28.0.0 version ?
Update : The v28 will be available soon I guess, I saw one recently !

The version numbering scheme is explained at https://bitcoincore.org/en/lifecycle/#versioning

Versioning
Bitcoin Core releases are versioned as follows:
MAJOR.MINOR, and release candidates are suffixed with rc1, rc2 etc.
Major releases
We aim to make a major release every 6-7 months.
These will be numbered 22.0, 23.0 etc.
Maintenance releases
We will provide maintenance “minor releases” that
fix bugs within the major releases. As a general rule we do not
introduce major new features in a maintenance release (except for
consensus rules). However, we may add minor features where necessary,
and we will back-port consensus rule changes such as soft forks.
Minor releases will be numbered 22.1, 22.2, 23.1, 23.2 etc.

So it depends what bugs are found, if any, that need to be fixed within the major version release cycle.

Bitcoin Core version numbers are of the form MAJOR.MINOR. The major number goes up when features are added, the minor number goes up when bugs are fixed.
The latest version as of 2024-Aug-16 is 27.1, which is part of the major series 27.x. The next major version will be 28.0, and is scheduled to be released on 2024-Sep-30. It will start the 28.x series.
If bugs are found that are worth fixing in 27.1, there will be a 27.2. This can happen before or after 28.0 comes out. If so, it will have only the 27.x features and not the 28.x ones.


text:
Data about number of block mined + stale blocks per day in Bitcoin?
I am wondering if there is any easy way to get number of mined blocks per day + number of stale blocks per day for the last three month e.g.,?
I could not find such data on Blockchain.info
In Ethereum, however, Etherscan provides these, see: https://etherscan.io/chart/blocks and https://etherscan.io/chart/uncles
Any help about getting such data for Bitcoin?

Stale blocks are not propagated well in Bitcoin-- the very fast compact block and fibre relay very aggressively blocks the propagation of stales. As a result the only way to get a good visibility of stales is to have a data from a fairly large number of highly geographically distributed nodes.
Geographic distribution is important because for the case of a block race with geographically separated blocks you will usually only hear the one from a geographically closer source.  Connecting to a great number of nodes is not likely important, connecting to a decent number of geographically close nodes however is likely important.
As an aside, in ethereum only a finite number of 'uncles' are signaled so you are probably missing ones there too if the source are just depending on that.
In measurements on Bitcoin I've done recently I used data from 7 nodes spread around the world and still dropping any one of them would make me miss some blocks-- so I think it's likely that I was still missing some in those measurements.

To my knowledge, there is no high quality data source for this.
Blocks mined is fairly simple to check, as that is simply the main chain's progress.
Stale blocks is harder for Bitcoin and Bitcoin based currencies. Unlikely ethereum's uncles, references to stale blocks are not stored anywhere, and after a better chain exists, propagation will not take place, so many nodes do not see a stale block at all. The only way to collect data would be to run multiple nodes, connected to as many different peers are possible, and monitor the logs for chain tip reorganizations. As far as I'm aware, no one is currently doing this, and if they are it is not offered as a publicly available dataset.

Thanks! could i know how to run multiple listening nodes? and check the logs? any useful link to explain? Also, can i montior block & transaction propagation delays?

Multiple listening nodes just means running multiple nodes. Get a bunch of servers, run nodes on them. You will likely need some additional tooling to make sure they connect to different peers. Monitoring block and tx propagation will also require some additional scripts to check the logs/listen for incoming transactions.


text:
Weak Subjectivity Checkpoints trust assumptions
With AssumeUTXO it should be possible to sync to chaintip from a recent snapshot of the UTXO set, rather than from the genesis block (although there's background full validation after sync). This is similar to Weak Subjectivity Checkpoints, with the difference that checkpoints work as a new genesis block, with a snapshot, from where new nodes sync. In other words, archival nodes can discard data older than these checkpoints since it's no longer needed for syncing.
So what are the arguments against having Weak Subjectivity Checkpoints on Bitcoin, a la AssumeUTXO? It seems a different trust model since new nodes are forced to trust the latest Bitcoin Core checkpoint / snapshot for syncing, assuming old data was discarded.
However, we are already trusting the Bitcoin Core development process to not have introduced malicious code at any point in the past. This is indeed the trust assumption of Bitcoin. Hence adding the requirement of a periodic checkpoint + UTXO snapshot seems under the same assumption. Notice that any user running a node can audit the correctness of a new checkpoint / snapshot being added. If an invalid checkpoint / snapshot gets added without anyone noticing, then developers can do equally bad things right now.
How is this a different trust assumption than Bitcoin’s?

By definition there is no trust involved in bitcoin core you verify each block header with software that you yourself run.

You are comparing apples and oranges here. By weak subjectivity you are referring to the consensus rules of a "Proof of Stake" network. Assumeutxo is an IBD speedup specific to the Bitcoin Core client, that also verifies the rules of the Bitcoin network.
A user may choose to use assumeutxo, but they don't have to. Anyone can verify the Bitcoin chain since the genesis block without the UTxO set hash that might eventually be included in the Bitcoin Core client. It is not the case in the "weak subjectivity" system you refer to. In this kind of system in the end you have to rely on the provided checkpoints, you can't opt out of using them.
In addition, and this is often a point of confusion, note that a assumeutxo / -assumevalid user is not entirely trusting the provided data. For assumeutxo the client is still performing the regular IBD in the background. If it finds out the provided hash was incorrect it will discard it. Hence the user trusts the hash only for the duration of the IBD. For assumevalid if the node is provided an alternate chain with more work (strong objectivity!) it will discard the configured -assumevalid one.

So what are the arguments against having Weak Subjectivity Checkpoints on Bitcoin, a la AssumeUTXO?
[...]
How is this a different trust assumption than Bitcoin’s?

As explained above, assumeutxo is not a checkpoint. Assumeutxo allows you to temporarily trust a provided UTxO set while your node is syncing. A checkpoint would make your node refuse a chain that does not contain such block.
Sure if the blocks are deeply buried, the trust model of having a checkpoint is very different. The Bitcoin Core client already has buried deployments anyways. But if all you want is to speedup IBD, just use assumeutxo instead of introducing a network rule?
Related:

https://bitcoin.stackexchange.com/a/7830/101498
https://bitcoin.stackexchange.com/a/75735/101498

No, I don’t think I’m comparing apples to oranges. What I’m saying is that a checkpoint with a snapshot can be used as a new genesis block, and old data discarded. I also note why nodes cannot opt-out in this case and why it seems like a different trust model. But you didn’t answer my main question.

I might have extrapolated what you said. I didn't intend to. Weak subjectivity is often discussed in relation to "Proof of Stake" (https://blog.ethereum.org/2014/11/25/proof-stake-learned-love-weak-subjectivity) so i assumed from a quick read you were referring to this, my bad. I'll update the answer, it boils down to the same issue anyways.

It was coined in the PoS context in order to solve the nothing at stake problem, but it's related to Bitcoin as well (i.e. there are things in Bitcoin that depend on trusting people and having a social consensus). Weak Subjetivity Checkpoints are called like that because you trust someone to give you the snapshot. In our case I was talking about devs doing it, since we already trust them for more critical stuff

It's not just about speeding up IBD, the main point is that archival nodes can discard historic data that's no longer needed for syncing

Well you can always trust a snapshot if you are so inclined but making it mandatory is a fundamental change in the system.. The value proposition of Bitcoin is that you can verify the current state by replicating all the transitions since state 0, without having to trust any provided data. I think we are both talking about the same thing now, but the crux of our misunderstanding is about the leap from having an opt-in snapshot mechanism to making it mandatory.

You can not sync without trust however, you trust the developers as I have pointed. If there’s a softfork you also trust them to have implemented it correctly. You cannot opt-out this.

It is true that some amount of trust is required at various levels. But it is tunable at all those levels. While you may opt-in to the easy-and-more-trusted sync, it is still possible to use the super-paranoid mode of syncing. Let's not throw the baby with the bathwater and say "since some trust is usually involved let's just make it mandatory and more trusted". By this same token you could end up arguing that we don't even need miners to coordinate transactions, let's just have developers run a server that provides checkpoints since we already trust them!

I agree, although it would be opt-in as long as there are archival nodes. I just wanted to know how substantially the trust assumption changes if it wasn't opt-in. I think ZK-STARKs are the best way to do this since with them trust is no longer needed.

it should be possible to sync to chaintip from a recent snapshot of the UTXO set, rather than from the genesis block

It should be possible
However, it is easy to verify the Genesis block as that is only a couple of hundred bytes, is well known and published in numerous places. You could cross check by hand.
The current UTXO set has tens of millions of UTXOs totalling tens of gigabytes of data. That's a lot more difficult to independently check.
You could make and check a hash of the UTXO snapshot but there might be significantly fewer independent places where the hash of an arbitrary UTXO snapshot was published.
It might also create a precedent which criminals would exploit by persuading victims to use a UTXO snapshot hash from their fake websites etc.
Then you'd have to maybe rely on signatures from some set of trusted authorities. But this can also be problematic.

Well in AssumeUTXO the snapshot is published by Bitcoin Core contributors, reviewers, etc and it can be independently audited by anyone using a full node (just like AssumeValid).

However, we still trust Bitcoin Core without checkpoints in a similar way: we trust that the hardcoded genesis block is the correct one. Of course the genesis block is widely known and it’s easy to catch Core devs trying to change it. If we are able to attain social consensus over the true genesis block, then it follows that this is also the case for the checkpoint / snapshot, given enough time. In both cases we rely on social consensus.

I think this issue is dealt with through proof of work, you would waste your time trying to build a chain with as much work as bitcoin and catch up to it. If you had enough hashpower to do it you would be much more profitable just mining bitcoin.
The graph at the bottom of this page https://bitcoin.sipa.be/ puts it into perspective. Trying to re-write the entire bitcoin chain is literally Don Quixote level of delusion, you can re-write a block but you can't convince everyone that your one different block is the real one. It is just not realistic with proof of work and a dense full node graph. If no one ran their own node and it was a PoS system these would be very valid concerns.

If your client software has a different genesis block then your starting point is already a false chain, an attacker doesn’t need to give you an expensive PoW chain

Of course they do, if you start with a different genesis block you wont be able to connect to ANY nodes that have the real one.

When you have a different chain you have a fork, these forks can happen naturally sometimes through block reorganizations at the latest blocks  this basically represents miners fighting over block rewards. But that always settles with the actual dense node graph running the majority of the chain, they come to consensus on what is a reorg and what is not in real time and that has been going on since before the transaction scheme was even as secure as it is today.

If you start with a different genesis block you have literally no chance of getting your blockchain to look exactly like Bitcoins or one with as much work. It would take you 600+ days and by then the Bitcoin network would be 600 days further

What I meant is that you need to trust Bitcoin Core to have the correct genesis block hardcoded, that’s all

but that's not really true, if they did not they would not allow you to transact with the real bitcoin network and thus core would not be functional.

more so each user VERIFIES that Core has the correct genesis block


text:
Bulletproofs vs Merkle paths for inclusion proofs
In this blog post, Andrew Poelstra comments that bulletproofs can be used for efficient merkle inclusion proofs.
How would these proofs compare in size and verifying time to simple merkle paths? Take for instance proving a thousand elements in a set of millions. Can we use batch verification and/or aggregation of bulletproofs?
Although the bulletproof would prove inclusion in zero knowledge, could we reveal the leaves so that they match the proof?

It depends what you are verifying, signatures can be aggregated in ways that other types of data that might be included in a merkle math cannot.

It’s not clear to me what you want to achieve with the bulletproofs and how you would want to aggregate them. I’m wondering how what sounds like “batch validation” which is used in block processing fits together with merkle branches which are used to bandwidth-efficiently inform a light-client about the presence of a transaction in a block.

In the same way that I can prove that a transaction is included in a block with the merkle path, bulletproofs can also be used for that. Or for example to prove that an UTXO is included in a utreexo accumulator. I'm basically asking how efficient it's in comparison with using merkle paths.


text:
Is Bitcoin fee attack preventable?
A block on Bitcoin blockchain contains about 4000 transactions.
Since a block is added approximately every 10 minutes, in 3 months, about 51 million transactions can be added (4000 x 6 x 24 x 90 = ~51 mil)
A bad actor such as a government could post transactions for full capacity for 3 months with a fee of $50 per transaction (currently avg trx fee is $1.3), costing the government about $2.5 billion and causing a DOS (Denial of Service) for all other transactions (Assuming that "miners" prioritize transactions with higher fees).
Can Bitcoin network handle such an attack?
P.S. $50 fee is just something I made up for cost to government calculation and can be any number that is way higher than the average fee. Point is to make "miners" ignore all other transactions that have lower fees to cause a DOS.
Also, please assume that government would prepare for this scenario by first creating millions of addresses and placing some sats in them so that transactions during attack would look like legitimate transfers.

Recent blocks contain less than 2200 transactions on average.

I went with 4000 because that is what I could find about the # of trx that can be put in a block. My question still stands for 2200 too; it would even make the attack less costly.

Yes, whether its 4000 or 2200 tx/block, I think someone with a spare $2500000000 to burn up could kill Bitcoin for at least a while.

It is worrying. I hope to hear what bitcoin devs have to say about this.

First the attacker has to acquire $25 billion worth of BTC, that might push the exchange rate higher making BTC  more expensive to acquire. This might be noticed.

Why? They can generate, say 1 million addresses and put about 10 dollar worth of BTC in each at a cost of 10 million USD. Then they can start the attack by submitting transactions from 1 wallet address to other.

They need to have enough BTC to pay the $2.5 billion you calculated they would need in fees. And that's with current low fee rates that might not stay low if April 2021 is any guide.

Let us continue this discussion in chat.

If it were the case that an actor was to pull off such an erroneous attack any node in the network could choose to deny including transactions sent from their ip address or even public key if they wanted to. This is similar to if you had a node on the network spamming you with anything else, like empty or invalid blocks, you could just disconnect from them. If I have this wrong someone please correct me though.

@RedGrittyBrick "They need to have enough BTC to pay the $2.5 billion you calculated they would need in fees" Aah I see. I feel so stupid. Thank you! If you can add your answer below, I can set it as the answer

@Poseidon Attacker can use many different ip addresses and different public keys from all those generated that I mentioned. So attack would come from MANY different addresses and ips. RedGrittyBrick's explanation is satisfactory. Thank you as well

Even though they COULD come from many keys and address' it is incredibly unlikely and the efficiency of the transactions on the bitcoin ledger allow for this to be even more unlikely than in a situation with bloated on chain data storage. Thanks for the interesting question :) Oh not to mention that spam still looks like spam even if it is coming from multiple address' all you need to do is deny SOME of those address' and you have limited the spam greatly.

@Poseidon What do you mean by efficiency of transactions? Don't miners just pick the transactions with higher fees?

I mean the way in which transactions are stored then processed, if it was the case that we needed to spend lots of compute time verifying transactions it would be more possible for an attacker to spam the chain, in a situation where verifications are quick it is actually more expensive to spam over a longer period of time.

Compute time for verifying is not something I was angling at. I was just worried about legitimate looking attack transactions with higher fees that would prevent normal transactions from being processed. I got the answer for it above. Thank you

I don't think it is easily preventable but it is expensive and it would be difficult to prepare without being noticed and might be difficult to execute in a way that couldn't be detected and countered, at least partially.
Firstly $2.5 billion is a substantial sum to throw away unless you are a government.
You would initially need to buy $2.5 billion worth of Bitcoin since that is the amount you calculated you need to have available in BTC to spend in transaction fees. Large purchases of this type tend to push up the price of Bitcoin. (Reference).
Bitcoin transactions that spend $50 in fees to pay someone only some small amount such as $10 would be suspicious and could be blocked, you'd have to re-use a more significant amount, say $2000. Transactions that quickly recirculate that $2000 would also be moderately obvious and easy to block.
Bitcoin has survived brief periods of very high fee rates (e.g. average over $50/transaction in April 2021) - this spurred the development and adoption of higher level Bitcoin payment systems such as the Lightning Network (LN) which use the Bitcoin network relatively infrequently.
Bitcoin miners have invested in the future of Bitcoin and many of them might not want to assist someone evidently intending to do harm to the Bitcoin economy, even despite the short term financial gains offered. It seems likely that many would coopoerate in a defensive action once the attack was obvious. Many or almost all of the less convinced miners might have to follow along with defensive efforts to avoid being left on the wrong side of a fork. Or they would divide their effort between a lucrative fork and a fork with long term value.
Really the question boils down to, if you had $2500000000 and wanted to end up with $0 with the side effect of destroying Bitcoin - would there be a cheaper or quicker way? Would there be something else you could do with that money that would be more beneficial to you?
Ultimately this question is a bit subjective, but I guess there are some aspects of Bitcoin's operation and history that can be brought up in an answer in an objective way.

Footnote re transactions per block
blocks> blockchain -file blk03359.dat -blocks -format csv
   No.,Version ,Height,Date and Time   ,   Txs,Target  ,Id
     1,2C992000,772279,2023-01-16 20:24,  3218,17077CE2,000000000000000000041832D3A631058097F4D5DF55D985E8C664EA221F317D
     2,2CDF2000,772280,2023-01-16 20:52,  3510,17077CE2,0000000000000000000116ADA2621CB8F296852A50CD022AD1A905B6D827B600
     3,20006000,772282,2023-01-16 21:16,  2788,17077CE2,0000000000000000000506A596FF3A003DB3C4E4221FA38B8A500A19A102C847
     4,27FFE000,772283,2023-01-16 21:24,  2971,17077CE2,000000000000000000045811D897C1B593F7F709E0726E88E95E3B7B7AA7C379
     5,324F0000,772284,2023-01-16 21:42,  3320,17077CE2,00000000000000000004C114CA81064110AA59C2286D5FA0BA51E02F7BF9B481
[...]
   100,20000000,772448,2023-01-17 23:17,  2787,17077CE2,000000000000000000074C2DDD69DC4266EBCE407F4CB89906D0D5A4A6C277F1
   101,20000000,772449,2023-01-17 23:20,  3012,17077CE2,00000000000000000006D59307529DE12433207000992AA7087E6560641D1503
   102,20000000,772450,2023-01-17 23:23,  1317,17077CE2,000000000000000000066B6B40258410FF5FD5C9034BC5F046FFF82CAEB5A59B
   103,293F6000,772451,2023-01-17 23:34,  2957,17077CE2,0000000000000000000427BF57B0D9DBFD00286B5F5EA6DF508ACF56D021D08F

blocks> blockchain -file blk03359.dat -blocks -format csv | ^
More? perl -n -a -F"," -e "if (/0000/) {$t+=@F[4]; $n++;} END{print $t/$n;}"
2196.54368932039

So, <2200 tx/block on average

If there were someone with a HUGE trail of inputs spamming the chain we could identify said trail of inputs and run code that would hold off processing these inputs and prioritize transactions that don't look like spam. Comes down to the fact that obfuscating yourself on bitcoin is even more expensive than using it normally and the fact that we spend little to no time processing single transactions means if you wanted to sustain this attack it would likely end up costing way more than you would calculate it to cost. Denying incoming or outbound peers is a right held by every node.


text:
What is the asm section of scriptSig
I am trying to understand the output from bitcoin-cli. The sample transaction's txid is 061959f1a3360d3781a870b2d43f73f7105b194b22f3765fcb9b8f545f9c8317, from block 222,222.
The asm section of scriptPubKey (i.e., OP_DUP OP_HASH160 28dce60cf7ba4d749afce5fd9781a403d293b74a OP_EQUALVERIFY OP_CHECKSIG) is more understandable, it decodes hex and shows the assembly in a (sort of) human-readable format:
"vout": [
    {
      "value": 0.63918136,
      "n": 0,
      "scriptPubKey": {
        "asm": "OP_DUP OP_HASH160 28dce60cf7ba4d749afce5fd9781a403d293b74a OP_EQUALVERIFY OP_CHECKSIG",
        "hex": "76a91428dce60cf7ba4d749afce5fd9781a403d293b74a88ac",
        "address": "14j4eyoxaAwA4SHj9YcEyVJ7FsqyWvUh9B",
        "type": "pubkeyhash"
      }
    },
...
]

The question arises when it comes to the asm section of scriptSig:
{
  "txid": "b66e78c919e36a6c563ceb1b29cfec26f7dec3c3fc1b3631c84056f3ae147f2f",
  "vout": 1,
  "scriptSig": {
    "asm": "3045022100dceb566dec99cf195aba5d6313f1e95eb7bfc74c93a794c4bfd6dd9f4082d8a002203b495b70b917b3dffdcbe70fc6ff7de910d1697efccc14f3eea6944bda87d21c[ALL] 0445554717c4d3240d818f400ab66fd4de438f2fd9174641ea76480b95cd6e883ec274a10b0691d85ac2cb87dcb9eef58b3abb8ee4bd277c8d6fea09eace2bc24a",
    "hex": "483045022100dceb566dec99cf195aba5d6313f1e95eb7bfc74c93a794c4bfd6dd9f4082d8a002203b495b70b917b3dffdcbe70fc6ff7de910d1697efccc14f3eea6944bda87d21c01410445554717c4d3240d818f400ab66fd4de438f2fd9174641ea76480b95cd6e883ec274a10b0691d85ac2cb87dcb9eef58b3abb8ee4bd277c8d6fea09eace2bc24a"
}

As you can see, it is 304502206ee08c76923816e4ba287142e9f147fe9cd0f26e6bd58b9a43f2283b1c614f46022100d5de298b627407bc7d5ac0a40259cafb865c30c6a67db926c7284da96ff71abd[ALL] 040841958a405ca1c05de4dcf04dfdfd6e7de5e7cb106744977e3d99eab3e59a2b5bc2441e0ad179055c14200745feb2da2d1b4485087e3a9a2a88a6531a6d6b02, which is not decoded at all.
I checked the same transaction from blockstream.info. It's result is: OP_PUSHBYTES_72 304502206ee08c76923816e4ba287142e9f147fe9cd0f26e6bd58b9a43f2283b1c614f46022100d5de298b627407bc7d5ac0a40259cafb865c30c6a67db926c7284da96ff71abd01 OP_PUSHBYTES_65 040841958a405ca1c05de4dcf04dfdfd6e7de5e7cb106744977e3d99eab3e59a2b5bc2441e0ad179055c14200745feb2da2d1b4485087e3a9a2a88a6531a6d6b02
which looks more readable and I can understand that this scriptSig pushes two bytes arrays into the Bitcoin's stack.
So the questions are:

Why bitcoin-cli's output is like this?
How to interpret it? Especially, how to understand the [ALL] part in its asm?

EDIT:
While the answer below is quite informative, the issue I raised on Bitcoin Core's Github is worth taking a look as well.

The asm section is a symbolic representation of a Bitcoin script. For inputs, the scriptSig is the "unlocking script," proving that the input can be spent in this transaction.
In your specific example, the scriptSig asm can be broken down as follows:

304..abd is the DER encoding of an ECDSA signature. You can find more information about this structure here: What are the DER signature and SEC format
[ALL] is the sighash flag, indicating the signature is for all inputs and outputs. This is the most common sighash flag
044...24a is the public key of the input

bitcoin-cli does not give the human-readable OP_CODE names when translating from hex to asm, but different block explorers may do so. This explains why you saw a different representation when looking up the transaction with https://blockstream.info.


text:
How to get the private key & wallet address using a multi-wallet recovery seed phrase?
How can I „convert“ a 12-words-long multi wallet seed phrase (of the Blockchain.com wallet) to the wallets private key and the wallets address using any programming language?
The given multi wallet seed phrase is like I mentioned above from Blockchain.com. The derivation path is (to my knowledge) m/44'/0'/n'.
Help would be very appreciated :)

This can be done with the bcoin library.
'use strict';

const bcoin = require('bcoin');
const m = bcoin.Mnemonic.fromPhrase('abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon about');
const master = bcoin.HDPrivateKey.fromMnemonic(m);
const key0 = master.derivePath("m'/44'/0'/0'/0/0");

const ring0 = new bcoin.KeyRing({privateKey: key0});
console.log('Legacy: ', ring0.getAddress('base58'));

const ring1 = new bcoin.KeyRing({privateKey: key0});
ring1.witness = true;
console.log('SegWit: ', ring1.getAddress('string'));

Output:
Legacy:  1LqBGSKuX5yYUonjxT5qGfpUsXKYYWeabA
SegWit:  bc1qmxrw6qdh5g3ztfcwm0et5l8mvws4eva24kmp8m


text:
What is the technological achievement that is needed in order to get effective SNARKs into Bitcoin?
AFAIU SNARKs may be introduced into the Bitcoin protocol as a soft fork (e.g. a new segwit version number), and this would enable many new use cases, such as verifying very advanced computations that are executed off-chain, directly on the chain. Unfortunately, SNARKs are known to be computationally very expensive, or to require some level of centralized trust. How big is the scientific advancement that we need in order to get SNARKs into Bitcoin? Do we have recent significant technological discoveries that let us think that in the next 10 years it will be feasible to introduce SNARKs into Bitcoin?

SNARK is a specific implementation of a zero knowledge proof. There are lots of different implementations that try to avoid this scaling trade-off and it is likely that the ultimate solution for such a thing has not been discovered. You might look at zkSTARK or Bulletproofs for some generic zkproof models that are robust and come with their own sets of trade-offs but generally do not involve trusted setup as SNARK does. From what I understand most of the overhead here is in the size of the proof as verification tends to be fast. There are also privacy and quantum resistance concerns, IMO privacy is more relevant for now. I think the answer will lie in what is the most practical application of this and does it require a base-layer interaction or could we do all these proofs on a side-chain or l2?


text:
How to force close a channel at the original state, for a good cause
A few months ago I created a payment channel from my desktop node, on LND, to my mobile node, on Blixt. Then I tested a few payments between the nodes, and the final balance on the mobile phone is now about 20K sats (not that much). Unfortunately, I crashed my mobile phone, and now I am unable to perform a cooperative close of the payment channel. I do not have a backup of the Blixt wallet, because was used only for testing. I think the only option I have to recover the funds on the mobile node would be to perform a force close of the channel using its original state, when the Blixt node balance was zero. I know for sure that the mobile node won't be able to react and to broadcast a punishment transaction, because it is crashed, so I will be able to "steal" the 20K sats from my own mobile wallet. Unfortunately, I have no idea about how to do this technically.
Can someone help?

Just be sure that the channel wasn’t signed up with a watchtower, otherwise you might lose the other side of the channel as well


text:
How to differentiate between BTC transactions and BRC20 transactions on a blockchain?
How to differentiate between BTC transactions and BRC20 transactions on a blockchain using the RCP interface?
This is a transaction record that contains BRC20 tokens which I found on a blockchain explorer.
When I using the RPC to scan blocks and retrieve transaction records like this one, how can I use Python3 to determine whether the transaction is a BRC20 or BTC transaction?
{
    "hash": "331d02db3d6ee0cffae9052be5799bd7754c6ebb2020e422058b73334c59bf85",
    "ver": 2,
    "vin_sz": 1,
    "vout_sz": 2,
    "size": 359,
    "weight": 734,
    "fee": 7360,
    "relayed_by": "0.0.0.0",
    "lock_time": 0,
    "tx_index": 4705820269244017,
    "double_spend": false,
    "time": 1682913851,
    "block_index": 787738,
    "block_height": 787738,
    "inputs": [
        {
            "sequence": 4294967293,
            "witness": "03405d88c3c66893b950a42d3672f3009a9f15e674f248a1d1c435609440b91e426598d9505752c560b24ba6261f1b3c7e997941006c9ead7b0378ca7c92c65519418320117f692257b2331233b5705ce9c682be8719ff1b2b64cbca290bd6faeb54423eac061b237bd58701750063036f7264010118746578742f706c61696e3b636861727365743d7574662d3800357b2270223a226272632d3230222c226f70223a226d696e74222c227469636b223a2266726565222c22616d74223a2231303030227d6821c0117f692257b2331233b5705ce9c682be8719ff1b2b64cbca290bd6faeb54423e",
            "script": "",
            "index": 0,
            "prev_out": {
                "addr": "bc1pejeuynu5edq5fvfa8elpkem6zuzvrvqx20zundhpyuy3h3y8wumslwyd5h",
                "n": 0,
                "script": "5120ccb3c24f94cb4144b13d3e7e1b677a1704c1b00653c5c9b6e127091bc4877737",
                "spending_outpoints": [
                    {
                        "n": 0,
                        "tx_index": 4705820269244017
                    }
                ],
                "spent": true,
                "tx_index": 6896182349770388,
                "type": 0,
                "value": 10000
            }
        }
    ],
    "out": [
        {
            "type": 0,
            "spent": false,
            "value": 546,
            "spending_outpoints": [],
            "n": 0,
            "tx_index": 4705820269244017,
            "script": "51208873ed390476c5c7b6c825bae7eca675a46bc200cc7a82354b0debec19f83abc",
            "addr": "bc1p3pe76wgywmzu0dkgykaw0m9xwkjxhssqe3agyd2tph47cx0c827qd4yq9x"
        },
        {
            "type": 0,
            "spent": true,
            "value": 2094,
            "spending_outpoints": [
                {
                    "tx_index": 1274202281123575,
                    "n": 0
                }
            ],
            "n": 1,
            "tx_index": 4705820269244017,
            "script": "0014ea410b1701aeaa64dbf721cd55fce1231f2144a3",
            "addr": "bc1qafqsk9cp464xfklhy8x4tl8pyv0jz39rhhh0s7"
        }
    ]
}

How to analyze ? Witness component 1 , Witness component 2, Witness component 3.
If I want a complete analysis, what should I do.
The analogy is as follows:
{
  "p": "brc-20",
  "op": "mint",
  "tick": "free",
  "amt": "1000"
}

ID 331d02db3d6ee0cffae9052be5799bd7754c6ebb2020e422058b73334c59bf85i0
Length 53
Type text/plain;charset=utf-8
Timestamp 01 May 2023 12:04:11
Block height 787738
Size 133
Standard: brc-20
Operation: mint
Token: free
Amount: 1000

How to differentiate between BTC transactions and BRC20 transactions

Well, you can't because they are not disjoint sets, all BRC-20 transactions are also BTC transactions.
You identify BTC transactions that are also BRC-20 transactions in two stages:

Does it contain an Ordinals Inscription
Does the inscription contain a BRC-20 operation.

You look in the penultimate witness component for the Ordinals signature ("ord" etc)
You look in the inscription for JSON text content that matches the BRC-20 patterns.
In your example, just converting your "witness" from hexadecimal to ASCII produces
... ord ... text/plain;charset=utf-8 ... {"p":"brc-20","op":"mint","tick":"free","amt":"1000"}
But parsing it properly would be more reliable.

Parsing the Witness data

Hexadecimal data
Meaning

03
There are three witness components

40
Length of first component is 0x40 = 64 bytes

5d88c3c66893b950 a42d3672f3009a9f 15e674f248a1d1c4 35609440b91e4265 98d9505752c560b2 4ba6261f1b3c7e99 7941006c9ead7b03 78ca7c92c6551941
Witness component 1

83
Length of second component is 0x83 = 131 bytes

20117f692257b233 1233b5705ce9c682 be8719ff1b2b64cb ca290bd6faeb5442 3eac061b237bd587 01750063036f7264 010118746578742f 706c61696e3b6368 61727365743d7574 662d3800357b2270 223a226272632d32 30222c226f70223a 226d696e74222c22 7469636b223a2266 726565222c22616d 74223a2231303030 227d68
Witness component 2

21
Length of witness component 3 is 0x21 = 33 bytes

c0117f692257b233 1233b5705ce9c682 be8719ff1b2b64cb ca290bd6faeb5442 3e
Witness component 3

Different input types (for output types P2WPKH, P2WSH, P2TR etc) have different things in the witness lists. This can be seen in answers to What are the scriptPubKey, scriptSig, redeem script and witness for the various output types?
Parsing the penultimate witness component (component 2)
The Ordinals Inscription documentation says

A text inscription containing the string "Hello, world!" is serialized
as follows:
OP_FALSE
OP_IF
   OP_PUSH "ord"
   OP_PUSH 1
   OP_PUSH "text/plain;charset=utf-8"
   OP_PUSH 0
   OP_PUSH "Hello, world!"
OP_ENDIF

Here's what I came up with treating component 2 as bitcoin script:

Hexadecimal data
Normal Meaning
Commentary

20
Push the next 0x20 (32) bytes onto the stack

117f692257b23312 33b5705ce9c682be 8719ff1b2b64cbca 290bd6faeb54423e
Pushed data

ac
OP_CHECKSIG
(Note that nothing after this ends up affecting the final stack value and so has no effect on normal Bitcoin nodes. For them, the following might as well not exist)

06
Push the next 0x06 (6) bytes onto the stack

1b237bd58701
Pushed data

75
OP_DROP
(throws away data just pushed onto stack - presumably private data of inscriber?)

00
OP_FALSE
(This is the start of the Ordinals Inscription serialization)

63
OP_IF
(Since the previous operation added FALSE to the stack, script execution on normal bitcoin nodes will skip everything up to OP_ENDIF. The rest still has to look like valid script though.)

03
Push the next 0x03 (3) bytes onto the stack

6f7264
Pushed data = ASCII ord

01
Push the next 0x01 (1) byte onto the stack

01
Pushed data has value 1
(Ordinals indication that MIME type follows)

18
Push the next 0x18 (24) bytes onto the stack

746578742f706c61 696e3b6368617273 65743d7574662d38
Pushed data = ASCII text/plain; charset=utf-8
(I am surprised "application/json" wasn't used)

00
OP_0: Push integer 0 onto the stack
(Ordinals indication that inscription data follows)

35
Push the next 0x35 (53) bytes onto the stack

7b2270223a226272 632d3230222c226f 70223a226d696e74 222c227469636b22 3a2266726565222c 22616d74223a2231 303030227d
Pushed data = ASCII { "p":"brc-20", "op":"mint", "tick":"free", "amt":"1000" }
(This is an inscription in JSON format (despite prior MIME type) that contains a BRC-20 token minting operation.)

68
OP_ENDIF
(in this case, end of Ordinals Inscription)

Obviously you are looking for the script elements that are prescribed for Ordinals inscriptions. the last but one item above is the actual inscription. It can be parsed as JSON to give the BRC-20 data structure.  Obviously p's value, brc-20, is what you are searching for in the JSON data.

Related

What are the scriptPubKey, scriptSig, redeem script and witness for the various output types?

Thanks, you said it so well. I understand now.
But what if there are more than one in the witness? Just like this:
I see multiple strings in witness that are used, split :
"inputs": [
    {
        "sequence": 4294967293,
        "witness": "03405d88c3c03405d88c3c, 03405d88c3c,03405d88c3c",
        "script": "",
        "index": 0,

"witness": "03405d88c3c03405d88c3c, 03405d88c3c,03405d88c3c", looks like a bug in whatever blockchain explorer you are using. I suggest you work with the raw binary data, or at least raw hex, rather than this, or any other, blockchain explorer's JSON interpretation

Thanks, You're a genius.
How to analyze ? Witness component 1 , Witness component 2, Witness component 3.
If I want a complete analysis, what should I do.
The input field here is too small, I added the case to the question.

That's an interesting follow-on question but not one I have an answer to at the moment. It is something I will probably spend time on eventually.

@Mashaji See "Related" link I added to bottom of answer. Answers to that related question indicate what witness components are expected, and their meaning, for each different type of transaction input.


text:
What percentage of bitcoin supply sits on certain UTXO output type?
What percentage of bitcoin supply sits on certain UTXO output type?
Is there any dashboard that shows that? I found a dashboard displaying UTXO count but not the BTC supply per UTXO type.

Just like you, I’m only aware of a chart that shows the partition by count of the UTXO set. However, also on txstats.com there are individual stats per output type, which each include the amount of bitcoins held in outputs of that type.
As of 2023-08-17, these charts show:

Output Type
Amount [₿] (mostly rounded to 100s)
% of sum

P2PKH
8'089'100
43.3

P2SH (includes wrapped segwit)
4'454'400
23.9

P2WPKH
4'267'400
22.9

P2PK
1'733'700
9.3

P2WSH
 876'600
4.7

P2TR
35'100
0.2

Non-standard
2'600
~0.0

P2MS
57.49
~0.0

OP_RETURN
40.47
~0.0

Update:
Because the chart didn’t exist, I made one:

Thank you Murch! Anyway one could query this from own node?

I guess you could use dumptxoutset and then use https://github.com/theStack/utxo_dump_tools to get a more manageable format, calculate the results from there?


text:
Using Bitcoin as a trusted clock?
I'm looking for a way that a computer could get a rough estimate of the current date and time with a very high confidence level, for example in order to verify the expiry of a contract by means of a computer program.
NTP is a popular protocol for synchronising computer clocks across a network. Despite providing highly accurate time measurements, NTP doesn't sign timestamps, which, in the context of my project, is a lack of reliability in that the nodes are susceptible to DNS spoofing or man-in-the-middle attacks#, which are able to make the victim think it's communicating with someone else than who it's actually talking to, and thereby manipulate the victim's perception of current time.
After some deliberation I realized that the blockchain can be taken as a trustworthy log of past timestamps, the latest of which give a rough estimate of the current time. Thus –I thought– I could write a program that starts a Bitcoin node, requests from its peers the latest blocks and fetches the timestamps from their headers... done! Now, I'm pretty confident this method is robust against the attacks discussed in #, but I don't know how to formalise the method. Here's my idea.
In order for the method to equally apply to old and new users, let's assume that my computer starts a brand new node each time it wants to check the current time. Each time, it will DNS-lookup some node IPs and subsequently request the latest blocks from them, check their validity and, if positive, read the timestamps. This gives raise to the following question.

What is the quickest way for my node to know that the blocks received from its peers have been validated by the network? Can it do so without downloading the full 400+GB blockchain nor compromising the robustness against DNS spoofing and MITM attacks?

My guess is that I could verify all the transactions, Merkle roots and headers of the blockchain provided by my peers; but still what would guarantee me that the branch that I'm looking at is backed by the network?

Despite the generic feeling of the question-marked questions, my sole goal is to turn the blockchain into a trustworthy timekeeping log (a "clockchain", if you would), not running a Bitcoin node per se. It would be ideal to find a specific solution to the problem but the only approaches I have come up with are as generic as the questions up there.

Possibly related: https://github.com/opentimestamps/restamp.

the blockchain can be taken as a trustworthy log of past timestamps, the latest of which give a rough estimate of the current time.

Very very rough. Instead of tens of milliseconds with NTP you are talking about errors of the magnitude of hours with Bitcoin block timestamps. It can be five or six orders of magnitude worse!
If you are considering Bitcoin network adjusted time from Bitcoin Version messages, I don't think that has any guarantees about accuracy or even monotonicity?  It certainly doesn't take into account latency and jitter. I don't think it does much to counteract timestamp spoofing either. Apart from averaging a handful of sources.

What is the quickest way for my node to know that the blocks received from its peers have been validated by the network?

There is no way to know.
Bitcoin nodes all perform their own validation. They never assume anyone else has performed validation for them. No Bitcoin nodes ever trust any other nodes. Bitcoin nodes effectively assume all other nodes are possible liars and proceed accordingly.

NTP doesn't sign timestamps

NTP isn't a notary but it does have provision for securing reliable time.
https://en.wikipedia.org/wiki/Network_Time_Protocol#Secure_extensions

[NTP?] nodes are susceptible to DNS spoofing or man-in-the-middle attacks

On the other hand, Bitcoin nodes have to counteract possible Eclipse attacks etc.


text:
Is it possible to download only a part of the blockchain?
I am doing some investigation related to the bitcoin transactions graph and I don't need the whole blockchain, because it would be too much data (and I don't have much space). I just need a part of the blockchain to construct a graph with the transactions and then do some analysis on it. Is it possible to download only a part of the blockchain? Let's say I want only up to 3 million transactions. Or something like the transactions from 2015 and 2016, or the most recent 3 million transactions (these are not requirements I have on the data, it's just to exemplify).

If you write software yourself, with a custom implementation of the P2P protocol, this is certainly possible. Just send getdata messages for just the blocks in the range you want.
No such feature exists in Bitcoin Core however, as its purpose is fully verifying the state of the chain, it will always start at the very beginning. If you want blocks somewhere from 2015 you can of course just stop the software once it has passed that point.
Note that Bitcoin Core does support a pruned mode, in which only the last N blocks are kept on disk (but they're still all downloaded and verified, just discarded after verification). If you want access to the last 3000000 transactions, and lack diskspace for the full chain, it would be an option to synchronize in pruned mode.

I will investigate those getdata messages. I am new in the bitcoin technology so I did not know you could just send messages for specific blocks. Could I do that also to ask for some transactions? I think the idea could be to start from a random transaction and ask for the previous and the previous and so on.
A problem I have with using Bitcoin Core is that my PC is not very powerful, so I guess it could take too much time to download and verify the blockchain, even if I use pruned mode.

The P2P protocol does allow asking for specific transactions, but it's only for transactions the peer has recently announced to you. Nodes do not generally keep a database of all transactions ever that you can just query, as it's not needed for general operation, so even nodes which do have such an index do not expose it through P2P commands.


text:
What is the purpose of version field in version messages exchanged during establishing a connection?
I thought that the version messages that are exchanged when establishing a connection between two nodes and the version field within them indicate what each node supports. Each version supports some new feature and then, depending on which versions are exchanged in these messages between the nodes, a smaller version of the protocol is used and each node knows what it can "ask" other nodes.
However, based on this answer and comment:

Protocol version 70014 simply means the sendcmpct command exists, not
that compact blocks are actually supported.

...it seems to me that my assumptions are wrong.
Therefore, what is the purpose of the version field in the version message

A node establishes which "language" it will speak with other nodes per the protocol version. The protocol version defines which messages exist and what responses are deemed acceptable to those messages.
It is still up to the nodes to establish which features they support by using the available vocabulary.
E.g. consider a node with a complete copy of the blockchain vs a pruned node. Both nodes may speak the same version of the protocol, both will advertise the NODE_NETWORK_LIMITED service, but only one will advertise NODE_NETWORK. Asking the pruned node for an old block would be a violation because they did not advertise NODE_NETWORK, even though their protocol version enables them to understand perfectly what was requested.

Yes. The example that Pieter Wuille left in the comment (link is above) is also good and explains it well

The purpose of version field in version messages exchanged during establishing a connection is for the nodes to see which features they can negotiate with each other. The fact that some node supports a certain version does not necessarily mean that it supports all the features related to it (all the features introduced in that version or versions before). In order to see if something is really supported, some negotiation mechanism is necessary.
For more information see correspondence between @PieterWuille and myself here.
You can see all the methods used for negotiation here.


text:
Why are indexes in cmpctblock and getblocktxn messages calculated as differences?
If I want to send transaction indexes in either cmpctblock or getblocktxn messages, they are represented as differences. For example if I want to send indexes 0, 54, 65, 67 and 1236, they are represented as 0, 53, 10, 1 and 1168.
Why was this done this way? Why not only indexes are passed, but their differences? Is this due to some optimization or something else?

Numbers up to 252 are encoded as a single byte, while larger numbers take 3 bytes. Differential encoding is effectively a form of compression, reducing the size of the data without losing information (the same set of transactions is communicated). Size of encoding matters in compact blocks, as its purpose is reducing latency. One aspect of that is aiming to fit everything in as few IP packets as possible (typically around 1500 bytes on the internet).


text:
Does the whole message become invalid when the invalid INV type occurs in the message?
If an INV type unknown to the node appears in an INV message, a GETDATA message or some other message where the INVETORY structure is used (e.g. MSG_WITNESS_TX for non-segWit nodes, MSG_WITNESS_TX in an INV message for segWit/non-segWit node, some random INV type, etc.) , is the whole message then considered invalid or only that "row" in the message is skipped/ignored, while the other rows and the rest of the message are processed?

In Bitcoin Core, as of version 26.0, INV entries of unknown type are ignored and skipped (and optionally logged, if "net" debugging is on); later entries in the same message are still processed.
On incoming INV messages, MSG_WITNESS_TX is treated the same as MSG_TX, and MSG_FILTERED_BLOCK, MSG_CMPCT_BLOCK and MSG_WITNESS_BLOCK are treated the same as MSG_BLOCK.

Does it apply to all messages in which the given structure (INV) appears or are there exceptions?

I ask because only MSG_TX and MSG_BLOCK should appear in INV messages, but what if MSG_WITNESS_TX or MSG_WITNESS_BLOCK appears. They are a known type to the segWit node, but should not appear in the INV message. Are such rows ignored or the message is considered as invalid and rejected?

Updated my answer.

@MudjaAdjum My answer is specifically about Bitcoin Core 26.0, which does support segwit. "updating to segwit" has no meaning here; you need to talk about specific versions of specific software. There is no guarantee that any other version/implementation behaves similarly.

I meant Bitcoin Core. When I wrote "if it was not updated to segWit" I was referring to Bitcoin Core versions before segWit, and their behavior when they received in the INV message, for example, MSG_WITNESS_TX.

But even within versions of Bitcoin Core, the situation is more complicated. E.g. (random example) it appears that 0.14.0 would not treat invs with MSG_CMPCT_BLOCK like MSG_BLOCK while 26.0 does. There are no specifications for how a node behaves when arbitrarily wrong data sent to it, so different pieces of software (even different versions) may behave differently.

MSG_WITNESS_TX and MSG_WITNESS_BLOCK were introduced with the SegWit update. Nodes that have not updated to SegWit do not know this INVENTORY type. Since you wrote that if a node receives INV entries of an unknown type, they are skipped, I wanted to add a note that if a non-SegWit node received such an entry in the INV message, it would be just ignored since for that (non-segWit) node MSG_WITNESS_TX is unknown and MSG_WITNESS_TX == MSG_TX is not true.

@MudjaAdjum Yes, my point is that you have to be specific about what versions of what software you're talking about. E.g. 0.13.0 also treats MSG_WITNESS_TX as MSG_TX, but didn't have segwit active (that only happened in 0.13.1).

You are right. I should have denoted the version I was talking about. My bad


text:
Is a header message used to notify peers of a new block?
As far as I know, when we receive a new transaction or block, we send an INV message to all peers to inform them about it. If the peers want that data, they send us a GETDATA message. In several places (in BIP152 for example) I read and saw (picture below taken from BIP 152) that notification can also be done by sending a HEADER message? However, bitcoin wiki and all other sites say that this message is a reply to a previously sent GETHEADERS message.
bitcoin.wiki:

The headers packet returns block headers in response to a getheaders
packet.

bitcoindeveloper:

The “headers” message sends block headers to a node which previously
requested certain headers with a “getheaders” message. A headers
message can be empty.

Can you explain to me whether the HEADER message is also used for notifying peers about new blocks, why and in which cases?

Since the introduction of BIP130 ("sendheaders"), a node can request that new block announcements are sent through a HEADERS message rather than through an INV. This avoids a roundtrip, in case the receiver first wants to verify the PoW of the header before downloading the actual block data.


text:
What methods are used to negotiate between nodes about the features they support?
The version number that is exchanged when establishing a connection between nodes only indicates which features are potentially supported by the nodes (a feature that can be negotiated). In order to establish what they really support, some negotiation mechanism is needed. The two I know are:

via an additional message (sendcmpct, sendheaders, etc.)
via service flags

Are there any others?


text:
At what point does a node start downloading blocks in IBD?
What I know so far is that when the node starts for the first time or after some time, the Initial Block Download (IBD) begins. The node chooses one peer (the header sync peer) from 8 outgoing connections. Initially, it sends a getheader message only to this peer and receives headers from it. As the timestamp of receiving headers approach current timestamp, the node starts sending getheader requests to the remaining peers. Headers received from other peers can differ and even create two chains starting from the genesis block.
What I'm interested in is at what point does node start requesting for blocks? Is it during IBD or does it first download headers from all peers, create a complete header tree and only then start downloading blocks for the chain with the most cumulative power and move the active chain tip?
By the way, if checkpoints are applied to header sync, if I get from these other peers in IBD some chain (headers) that starts before checkpoint, how will my node accept it? In case my sync peer gave me wrong chain, I will stay forever blocked from getting the best chain.

Whenever there are headers in our header tree, and peers which we know to have blocks in that header tree that we don't have, we start asking them for those blocks.

@PieterWuille Ok, thanks. What do you think about this what I wrote for checkpoints? If my node choose bad peer, we would remain permanently blocked in that kind of attack.

There are various timeouts, at some point other peers get asked too. If a node gives us invalid blocks/headers, the network code may disconnect the peer too, etc.

What I'm interested in is at what point does node start requesting for
blocks? Is it during IBD or does it first download headers from all
peers, create a complete header tree and only then start downloading
blocks for the chain with the most cumulative power and move the
active chain tip?

There are two different processes.
The first is IBD. It is divided in two phases (caused by header pre-sync). In first we select one peer as the sync peer and send him one by one getheaders requests. As the timestamp of received headers begins to approach the current timestamp, we start sending getheaders messages to other peers as well. Peer responses with corresponding headers, we validate them and store their tiny identifiers (nothing is stored in block header tree at this moment).
Only after the peers send us all the headers they have (all header messages with less than 2000 objects) we move to the second phase.
In the second phase, we send getheaders messages only for headers that are part of the chain that has enough cumulative work. There are two conditions: that cumulative work must be within one day of the current active chain tip and greater than a certain predefined value. Since the first condition at the very first moment is a genesis block (and some low difficulty blocks), it will be easily bypassed by malicious peer, however, the second condition will not. At this phase, it does not matter which peer you are requesting (whether it is sync peer or some other), it is only important that we ask some peer for which we know it has given header.
More to read about header pre-synchronization you can find in this post.
The second process works, something like, on events. Whenever a new header is added to the tree (event) this process is triggered. If various necessary conditions are met, a request for the block will be sent to one of the peers that is considered to have it.
When a node is just started, since in initial synchronization we only "talk" to one peer, then as sync peer sends us headers, we validate them and put in the tree, getdata requests for these blocks will be sent to that sync peer immediately.
Therefore, no! We do not store all headers first, create complete tree and then download blocks. Everything is done simultaneously. However, we do download all headers in the tree for pre-sync first, before doing anything else.
For this and much more other information check this post.

The pre-syncing process will prevent low-difficulty headers from being accepted, and without headers, no blocks will be downloaded/accepted either.

@PieterWuille Fine, but pre-sync at the begin is on genesis block. Low difficulty chain that we get from malicious peer will have cumulative work that is bigger than genesis, so it will be accepted?

No, there is a minimum amount of accumulated chain work hardcoded in the client, updated at every release. No headers chain below that value is accepted. See https://bitcoin.stackexchange.com/questions/76018/how-does-headers-first-prevent-fill-disk-attack/121235#121235: "(which means: enough to within one day of the active chain tip, and more than the preconfigured minimum chain work)".

@PieterWuille Thanks, I've edited the answer. Is it okay now?

We do not wait for the headers sync to complete before requesting blocks (but pre-sync does need to complete, because no headers are added to the block header tree during presync).

@PieterWuille Here you wrote that header presync only denotes first download of headers from some peer and verifying that the chain has enough work, so redownload and checking whether they match is not part of presync? However, as I know we need first to redownload all of these headers again and if they ALL match, then they are one by one stored (not need to again request headers, we got them in redownload) in header tree. As they are added to the header tree, a getheader message for the blocks can simultaneously be sent for them.

@PieterWuille What do you mean by presync here but pre-sync does need to complete, because no headers...?

Sorry, confusing naming. The concept of "headers presynchronization" consists of two (sequential) phases, the presync phase and the redownload phase. Nothing is added to the block header tree before the presync phase completes, but once that happens, during the redownload phase, blocks can start be requested immediately (before header synchronization completes). The if the ALL match refers to the all the headers in the redownload buffer, not all headers in the entire chain.

@PieterWuille Ok, so we take entire header chain from some sync peer (not store anything, just small IDs). Calculate work and if its not enough, we reject. Otherwise, we start second phase. This is first phase (pre-sync). In second phase (redownload phase), we redownload headers in number of redownload buffer (for example 100 headers). If they match, we start header sync for these 100 headers (validation and storing in tree). It also trigger block sync for them. Also, since first 100 headers in redownload buffer match, we start with redownloading another 100 headers and checking their match.

@PieterWuille If some matching in current redownload buffer fail, or header/block validation of them fails, further redownload is suspended. Is this correct? Also at what moment we start to ask another peers for headers (not just sync peer). I know its when headers timestamp start to approach present timestamp, but is that during pre-sync phase or during redownload?

The redownload buffer as of Bitcoin Core is 14441 headers FWIW (significantly larger than what fits in a single headers message). I wouldn't say "we start header sync" only when the redownload buffer matches - the header sync process is the entire thing (from presync phase to redownload to eventual adding to block header tree). Header synchronization (the entire thing) is started by sending getheaders to a peer. We initially send it to one peer, but once the best header that actually made it into our header tree (regardless of how it got there) is less than 24h old, we ask everyone.

@PieterWuille Are you with "best header" referencing to active chain tip or to header that is last (by its number) in some of the sequence (does not have children but does not need to be active chain tip; maybe it does not have valid corresponding block or some of its ancestor is invalid)?

The best header is just the best entry in the entire block header tree; it does not need to be the active chain tip).

@PieterWuille I'm really sorry to spam and bother you. But the terminology is not the clearest for me. What does the best header mean? The best in the sense that it is just the last one by sequence/serial number in the header tree that is still considered valid (it can become invalid if during block synchronization its block or a block of some ancestor is permanently invalid; then some earlier header by serial number that has not yet been declared invalid becomes the best). Therefore, it does not have to be part of the chain with the most work, nor to be an active chain tip, just to be last?

The best header is the one with the most accumulated work, and not known to be invalid or descendant of a known-to-be invalid block. It doesn't need to be the active chain tip, but it is the one that would become the active chain tip if we had all block data, and validation succeeds.

@PieterWuille Thanks for all this help. You have done a lot for me, I appreciate it.

I realize I made a mistake in the comments above. We only start requesting blocks from a peer when they have announced/sent to us a header whose work exceeds the minimum chain work (the same hardcoded constant needed to get out of header presync). I've updated my other answer to reflect this.

@PieterWuille Oh, okay, no problem. So everything stay the same (entire story) related to header sync, pre-sync, redownload, the moment when we ask another peers for their chain (not just sync peer) etc., just the difference is that we will not request for block immediately after the header is added to the tree, but when we download enough headers so they became part of the chain in chain tree that has enough chain work? Did I understand correctly?