text:
How does circular rebalance on LN work?
How does circular rebalance on LN work? (note: circular rebalance is term used in RTL, I use lnd). Let me describe the situation that happened to me:
I have two channels, one with Alice, one with Bob.
The situation before circular rebalance from my point of view:
Me    (200) sats outbound -----[400]----- (200) sats inbound         Alice
Me    (200) sats outbound -----[200]----- (0)   sats inbound         Bob

Then I initiated a circular rebalance of 100 sats on the the second channel with Bob and I was asked to choose "receive from peer" where I chose Alice
The situation after the circular rebalance from my point of view (supposing no routing fees):
Me    (300) sats outbound -----[400]----- (100) sats inbound         Alice
Me    (200) sats outbound -----[300]----- (100) sats inbound         Bob

My overall outbound capacity increased by 100 sats while overall inbound capacity stayed at 200 sats. Did I just make 100 sats? Did I cheat Alice?
Solution (It was my misunderstanding, I'm answering here because Rene's answer is really good and I want to keep it):
Basically, my outbound capacity decreased on the channel with Bob by the expected amount but the commit_fee on that channel rose by 100 and since I thought that my outbound capacity + commit fee are my disposable funds I thought of them as a overall outbound capacity which is not obviously the case.
What actually happened:
The situation before circular rebalance from my point of view:
Me    (200) sats outbound {100} commit fee -----[500]----- (200) sats inbound         Alice
Me    (400) sats outbound {100} commit fee -----[500]----- (0)   sats inbound         Bob

The situation after the circular rebalance from my point of view (supposing no routing fees):
Me    (300) sats outbound {100} commit fee-----[500]----- (100) sats inbound         Alice
Me    (300) sats outbound {200} commit fee-----[500]----- (100) sats inbound         Bob

if your channel's are initially as you described (just depicted here again with better readability using total cpacity of the channel in brackets in the middle of the channel and the respective local balance on the side of the peer)
You -(200) --------[400] -------- (200) - Alice
You -(200) --------[200] ---------- (0) - Bob

And you create a circular rebalance of 100 sats the result schould (in an ideal world ignoring routing fees) look as follows:
You -(300) --------[400] -------- (100) - Alice
You -(100) --------[200] -------- (100) - Bob

Thus your total local balance should have stayed the same as the total capacity of the channels should not have changed. Of course you pay the routing fees for such an operation so in reality the result might look a little bit more like
You -(298) --------[400] -------- (102) - Alice
You -(100) --------[200] -------- (100) - Bob

This means that you had sent out 100 set to Bot Bob and others have probably decreased the amount to forwards a little bit so that you will only receive for example 98 on your channel with alice.
The situation that you discribe in which you magically have 500 sats instead of 400 can / should not happened with a rebalancing operations. Are you sure that:

noone sent you 100 sats on the channel with Alice at the same time
You did not earn 100 sats of routing fees on that channel with alice at the same time?

Thus I would kindly ask you to reproduce the operation..

I reproduced the case and you are right. I did not gain 100 sats to my outbound capacity. My confusion arose from the fact that I added the commit_fee to my outbound capacity. So outbound capacity + commit_fee != channel capacity?

Basically, my outbound capacity decreased on the channel with Bob by 100 sats but the commit_fee on that channel rose by 100 and since I thought that my outbound capacity + commit fee are my disposable funds I thought of them as a overall outbound capacity which is not obviously the case.

https://bitcoin.stackexchange.com/questions/83305/how-is-node-capacity-determined-on-1ml-com helps you to get a better understanding of capacity and balance (aka outbound capacity). The commitment fee is explained at: https://bitcoin.stackexchange.com/questions/89232/why-is-my-spendable-msat-much-lower-than-msatoshi-to-us/89235#89235 it can be high and changes. So in general you should have something like capacity = commitfee + local Balance (outgoing capacity) + remote Balance (aka: incoming capacity). Though the fee is only taken from your balance if you opened the channel

Specific and unregular thing about it is that my channel with Bob after the rebalancing is like this:
capacity + circular rebalance amount  = commitfee + local Balance (outgoing capacity) + remote Balance (aka: incoming capacity)


text:
Negative fees on Lightning network and fees on incoming liquidity
Why there aren't negative fees on lightning network protocol and the ability to set incoming fees? Would there be any benefit? Let me describe the situation. There are nodes where the most of the liquidity goes to (the terminal type) and the nodes where the most of the liquidity goes from (the initial type). To rebalance it back is sometimes costly. A lightning node operator would maybe be willing to pay people to send satoshis from certain terminal nodes (setting incoming fees on negative) and send them to the initial type nodes (setting outcoming fees on negative).

Inbound Fees is a proposal currently discussed in the Lightning world, so it may become a possibility in the future. For a recent status see this thread: https://twitter.com/lucasdcf/status/1615413064148328448

I am not sure if it is possible to give a satisfying / definite answer to your question. As far as I can tell lightning developers have been very cautious in the early days (and still are) with everything that might make people loose funds. Negative fees are an obvious possibility to loose money as people would route along that channel and you would have to pay for that if you don't actively manage your node and know what you do.
On a bit more opinionated level and from a mathematical point of view I agree that it is a little bit strange that the cost function is just not permitted to have negative values and I agree that a lot of things with respect to liquidity management, routing and payments would probably be much easier if we did not have to make that somewhat artificial distinction.

Negative fees would create the problem of finding the shortest (cheapest) path in the network, since a fast algorithm to find the shortest path in negative-weight graphs was elusive.
Not anymore:
https://www.quantamagazine.org/finally-a-fast-algorithm-for-shortest-paths-on-negative-graphs-20230118

Adding on a little later, here. There was some discussion on Twitter about this, regarding nodes used for the purpose of surveillance. Better statistical on-chain data can be obtained by processing more transactions which negative fees would encourage. So, based on my limited understanding, a negative fee could incentivize the monetization of users' on-chain activity records.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How to find the wallet balance?
I'm trying to find the balance of specific addresses in a wallet. I basically have the master key which I use to generate addresses but I wonder, is it possible to find the derived addresses that have any balance and how can I query it (the balance of the addresses, and thus the wallet)?
I know I can generate infinite derived(child) addresses from the master key so, am I supposed to keep generating addresses and check every single one to see if it has any balance? Are you supposed to store both the master key and the addresses that you generate to receive payments?

If you have the master private key for an HD wallet, then software can easily generate derivative addresses, and check the balance of those addresses against the UTXO set.

am I supposed to keep generating addresses and check every single one to see if it has any balance?

Wallet software will have a set number of addresses it generates and checks for a balance upon wallet initialization (maybe 50? this will vary and is configurable in some software). The wallet will probably include some logic like "look 50 addresses past the last used address found". You can design this to fit your need, for example if you were a merchant accepting payments on a busy website, you might need a higher bound, in case 50 customers in a row generate a payment address but then don't complete checkout.

Are you supposed to store both the master key and the addresses that you generate to receive payments?

The wallet software will do this for you, but ultimately you do not need to explicitly keep a list of every private key / public address derived from your master key in order to access your funds later. If you have the master private key saved you will be able to recreate all derived keys from it.

Thanks for the clarifications. I'm developing the "wallet" software so I guess I'm responsible to keep track of addresses. As far as I understand the point is that you need to generate addresses "blindly"/sequentially until you find the funds you are expecting.

@Books yea, thats pretty much it, I wouldn't really call it 'blindly', but calling it sequential makes sense.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
How can I avoid paying transaction fees?
A bitcoin transaction requires paying a fee to the network, and that fee changes depending on network congestion. During busy times the fee rate may increase substantially, so how can a user avoid paying these fees in the first place?
Is there a way to transfer/import bitcoin from one address to another in order to avoid transaction fees?

(I'm posting this question because I was writing a reply to a similar question, and halfway through writing that reply, the question was voted closed)

Is there a way to transfer/import bitcoin from one address to another in order to avoid transaction fees?

No. It is not possible to move btc between addresses without publishing a transaction to the blockchain. However, there are some ways to accomplish a similar result (transferring/altering the ability to spend some coins), but I think every one of those ways will employ a different trust model than actually transacting on the blockchain. As a user, you must balance the level of trust you are comfortable with, against the need to save on transaction fees.
Some examples:
Off chain transactions using a centralized service:
This is how an exchange moves btc between accounts without incurring transaction fees. For example, when you buy/sell on a bitcoin exchange, a transaction is not actually published to the blockchain. Rather, the exchange will simply update their internal database, which keeps a ledger of all user account balances. As a user you are trusting that the exchange has a full reserve of all outstanding user $ and btc balances, and that they will not make an error in keeping track of your account balance.
Off-chain transactions by exchange of private keys
You can give someone bitcoin by simply giving them a copy of private keys for a funded address. In this situation, the recipient will need to trust that you do not own any other copies of the private keys, or that you will not spend the coins yourself if you do have a copy saved.
(as an aside, if you are sharing keys with a friend, and the coins are stolen by a third party, it may be impossible to prove to your friend that you did not simply steal the coins- and vice versa).
Off-chain transactions by exchange of private keys, which are provably not backed up
This is made possible by specialized hardware, such as that produced by OpenDime. In this case, as long as you trust that the OpenDime hardware functions as advertised, you can give someone a funded OpenDime stick and they do not have to worry that you might have a backup of the keys for those coins.
A signed transaction that is communicated off-chain
In this case, you could sign a completely valid bitcoin transaction that sends coins to your recipients address, and then send someone the raw tx data, without publishing it to the blockchain. The recipient would be able to publish the data to the blockchain themselves, but they will have to trust you to not sign and publish a different transaction using those same outputs in the meantime. This method would also only 'delay' the spending of the tx fees.
(I'm honestly not aware of any actual use-case for a scheme like this. It's probably a dumb idea that requires a lot of trust, but it is possible nonetheless)
Payment channels
In this case, two transacting parties can lock some amount of btc into a special multi-sig address, and then trade a limitless number of partially signed transactions back and forth off chain, with no tx fees, and no need to trust the other party will not steal your bitcoin. The parties involved can only exchange value up to a maximum amount (the amount they locked into the channel), but the channel can be bi-directional, so payments can be sent back and forth basically endlessly otherwise. This method still requires a transaction fee to open and close the payment channel, but that fee now account for a vast number of 'off-chain' transactions, instead of just one or two on-chain.
An implementation of this idea is the lightning network, it is currently live on the bitcoin testnet.

"trusting that coinbase ... will >not< make an error in keeping track"?

@dave_thompson_085 Yes, Coinbase is reputable, but ultimately with a centralized service you are trusting that their system will not make a mistake, and also that they will not just outright steal your funds. Think of it like a bank: their servers could make a mistake and report your account balance as zero, but if you are a customer of a reputable bank, that issue probably won't arise.

But you didn't say 'not' in your post; you said "trusting that coinbase ... will make an error".

@dave_thompson_085 Oh, you're right! Good eye, my mistake.

Could you perhaps add a sentence or two on why a transaction must pay a fee? It feels that a lot of the duplicates are asking about sending with zero fee, and while alternative approaches are addressed, the answer doesn't weigh in on why a transaction must have a fee in the first place.

@Murch yes I think thats a good idea, I came across this answer again not long ago and was thinking it could use some improvement anyways. I'll make it a point to do so soon.

If you run a full bitcoin node - for example bitcoin-qt, and it's in sync with the network, you can send bitcoin without a fee. It's not advisable however, since transactions stay in a memory pool waiting to be mined until a miner picks them up and adds them into a transaction block. Miners prioritize transactions with high fees attached - for obvious reasons. So you can end up with a transaction stuck in a transaction memory pool for days, sometimes weeks, after a few weeks it typically gets dropped (and returned to your wallet), although that depends on your wallet node configuration.
If it gets stuck there, you need to get it dropped from the memory pool. This is a nasty manual process, and your bitcoins will neither be in your wallet, nor spent during this period.
Generally however, when the blockchain is less busy, it will catch up and clear the memory pool. Eventually, and you will have succeeded in transfering btc for nothing...

If you don’t care about your privacy, just use https://pro.coinbase.com/…
While they don’t allow to send transactions without paying fees, they will pay the fees for you…

Worth noting: privacy is not the only concern when using GDAX, they are a custodial wallet so the user will be engaging with a very different trust model (relevant: see Off-chain transactions using a custodial service in my answer above)

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
synchronizing block headers
When I opened the bitcoin client, I found that it was "synchronizing block headers". Does this mean that the client first downloads block headers and then downloads block bodies? What kind of file does the header of the downloaded block is stored in?

Yes, downloading and validating the headers first has several advantages

You know in advance that the chain you're downloading blocks for is in fact the most-work chain (= the one you would ultimately end up accepting, assuming it is valid). This means you don't waste time on potential forks that have been resolved.
Since you know what blocks you will be downloading, and which peers have them, you can parallellize the download (spreading the download over multiple peers).

In Bitcoin Core, the block headers are stored in the blocks/index/ directory in the datadir, as a LevelDB database.


text:
Can I prioritize my transaction in Bitcoin?
How can I prioritize my transaction in Bitcoin? Does Bitcoin has any feature that can provide me fees required in case I want my transaction to be sent as high priority?

In bitcoin core existed embedded tools to estimate fee cost and confirmation time in blocks.
you can read the description with help command
bitcoin-cli help estimatesmartfee

estimatesmartfee conf_target ("estimate_mode")
Estimates the approximate fee per kilobyte needed for a transaction to begin confirmation within conf_target blocks if possible and return the number of blocks for which the estimate is valid. Uses virtual transaction size as defined in BIP 141 (witness data is discounted).
Arguments: 1. conf_target (numeric) Confirmation target in blocks (1 - 1008) 2. "estimate_mode" (string, optional, default=CONSERVATIVE) The fee estimate mode. Whether to return a more conservative estimate which also satisfies a longer history. A conservative estimate potentially returns a higher feerate and is more likely to be sufficient for the desired target, but is not as responsive to short term drops in the prevailing fee market. Must be one of: "UNSET" (defaults to CONSERVATIVE) "ECONOMICAL" "CONSERVATIVE"
Result: { "feerate" : x.x, (numeric, optional) estimate fee-per-kilobyte (in BTC) "errors": [ str... ] (json array of strings, optional) Errors encountered during processing "blocks" : n (numeric) block number where estimate was found }
The request target will be clamped between 2 and the highest target fee estimation is able to return based on how long it has been running. An error is returned if not enough transactions and blocks have been observed to make an estimate for any number of blocks.
Example:
bitcoin-cli estimatesmartfee 6

That is one of the purposes of transaction fees.
Since block space is finite, miners will select transactions with the best Feerate or fee/kB ratio. Of course, this is variable based on volume of transactions and the market rates.
See Fee Rates | Transaction fees

If you haven't send your transaction already, most wallets have an "economy", "normal" or "priority" transaction fee, which will automatically estimate what the average fee at the moment is, and will adjust the fee/kB ratio accordingly.
Remember that this ratio varies all the time, depending on how congested the Bitcoin network is. Here's a couple of sites I use to see how congested the network is: https://bitcoinfees.earn.com/ and https://bitcoinfees.info/ If you'd really want to send your transaction through, you could probably see what the highest tx fees are and manually enter a high number in the fee/kB field, if your wallet offers you to set your tx fee manually.


text:
How to use Wireshark to identify Bitcoin/Lightning P2P packets
I've seen that Wireshark can detect protocol packets like Bitcoin and Lightning Network, which are identified in the protocol column.
However, my Wireshark is not classifying those packets, it just says tcp/ip or http/json.
Does anyone know how to solve that? I want to identify the packets.

Run 2 nodes with different ports on regtest
bitcoind -regtest=1 -port=18333 -datadir=/home/satoshi/node1

bitcoind -regtest=1 -port=18444 -datadir=/home/satoshi/node2

Add node1 as outbound connection in node2
addnode "127.0.0.1:18333" "add"

Run Wireshark and select adapter for loopback traffic capture. You will see some packets with 'bitcoin' protocol.
Example: This is INV message sent by node2 to node1 when I generated a block using generatetoaddress 10 bcrt1qugkqq2a3lhf58e4pclcf52dgd4e02hpve3vhzs

For reference, Wireshark has included the ability to decode Bitcoin protocol packets since version 1.10.


text:
What's happening in regtest mode? - Wireshark analysis

Motivated to watch data exchange on Bitcoin Newtwork, especially on regtest, and with hints from David Harding I decided to analyse traffic on regtest with Wireshark.
I've installed and setup (I've got a list of interfaces) Wireshark and start capturing from all interfaces available and immediately run one regtest node with following connection specification:
...
Bound to [::]:11111
Bound to 0.0.0.0:11111
...

I put a tcp.port==18444 filter (in the style of this description) and apply.
Problem: There's no traffic shown!
Question: What am I doing wrong and what interface should I capture from?

There won't be any network traffic to capture unless you have at least two nodes running.

The regtest mode is basically for running a node in a test network where all peers have to be added manually and it is extremely easy to solve blocks. As such, there won't be any network traffic unless you start multiple nodes and connect them using the addnode RPC command.
Alternatively, maybe you meant to run this in -testnet mode instead of -regtest mode?

You are right. With 2 regtest nodes traffic is visible in fact.


text:
Question about MAST tree in book "mastering bitcoin"
On page 174, 3th edition, book "mastering bitcoin", what does the following statement means? thanks
"we could also have four commitments for the same cost."
Do the four commitments means commitments we need to prove which condition we choose? or commitments to four conditions?

It seems to me that the text should read "we could also have four conditions for the same cost".
MAST is a binary tree, which means the number of possible scripts (conditions) grows exponentially with the size of the inclusion proof, or equivalently, the size of the inclusion proof grows logarithmically with the number of possible scripts. For example, in a balanced tree:

1 commitment (merkle root) gives you 1 possible script
2 commitments (merkle root and 1 branch step) give you 2 possible scripts
3 commitments (merkle root and 2 branch steps) give you 4 possible scripts
4 commitments (merkle root and 3 branch steps) give you 8 possible scripts

What the original text is saying that 3 and 4 possible scripts both require 3 commitments.
It's important to note (and the very next section in the book goes over this) that using an imperfect/unbalanced binary tree can be beneficial if you expect some of the conditions to be used more often. By putting these shallower in the tree you use less block space on average.

thanks. according to your answer, four commitments means four commitments to conditions(4 possible scripts)?

No, "commitments" refers to the parts of the inclusion proof. The number of possible conditions is then 2^(commitments - 1).


text:
Taproot Script Path Spending can't pass signature verification
I am trying to verify the witness of a P2TR transaction, but the signature verification failed, could you please help?
The raw transaction:
010000000001016dacbb729930a8ba699a4188880d1b3525f430e52ad0c01fb4f6eae7987febd50300000000fdffffff014a01000000000000225120e2bc122cdb981a985074e182db9120e52c79b96710d517f84ab186c6ab092c4a0340318bd919ead99017116322d9b861fef0e1f7d26a115dcc7b48fba34bfd5be023ef6d5ab364861b99bd0950ff74b3e93e50c6fd6d17751336f0cf435bb0738c5d8e2064a5ee88fba503623726eabb0aa54bcced4eb2328d30dd05c8088bf5ed26fc41ac0063036f7264010118746578742f706c61696e3b636861727365743d7574662d3800487b2270223a226f72632d3230222c20227469636b223a226d656d65222c20226964223a2233333838303736222c20226f70223a226d696e74222c2022616d74223a2231303030227d6821c064a5ee88fba503623726eabb0aa54bcced4eb2328d30dd05c8088bf5ed26fc4100000000

The scriptpubkey of input: 5120dde70c00d3aec1533bd7f8b6f704ec21e6ed0a0de317a0ccf8199abde58d3062 , and the value: 4677.
Above are arguments, and I generate a sigmsg from those arguments:
000101000000000000009d9a573f31fa9eb2787db1c5d2162e36d76d21b0ecfa8c1e65617a4f5ca4a54bb4e368d5ab0ab9129e0c72ac46249d5bb8dc63b28638ebbc35bb69f62bae4ef4e342cce0133cd546aca2580d6f5bc5a184674e8dadda449fcfd073f503a44fe423e9829bfb4e23fbd3c4848baa035af15d73bcb83e510f7f097f90a21a4280d2386a04a2c1c490d8709f4b2bdf3d0d95d2aca0b6052ecc4a4c6a93fa33fba6ff0200000000

Then sighash: 54383836474379391a2ef0151bdbd9dc0e3b0e770f5052c433952082b09a2d7d.
The merkle_root: ec24ababd925ccbc309ce2684694fa32d17a8263b0a3f2d2999febfdfc7e5d28.
And the tweaked_pubkey: dde70c00d3aec1533bd7f8b6f704ec21e6ed0a0de317a0ccf8199abde58d3062.
In the end, I use signature 318bd919ead99017116322d9b861fef0e1f7d26a115dcc7b48fba34bfd5be023ef6d5ab364861b99bd0950ff74b3e93e50c6fd6d17751336f0cf435bb0738c5d to verify with sighash and tweaked_pubkey, but failed.

If it's a script path spend, the signature in it should be verified against the public key at it appears in the script, not the tweaked pubkey (the latter is only for key path spends).

@peter Thank you. I tried the internal pubkey 64a5ee88fba503623726eabb0aa54bcced4eb2328d30dd05c8088bf5ed26fc41, but still can't pass the schnorr signature verify.

Finally, I found the problem.

The signature is generated with the default sig type, aka 0x00. But I use the sighash_all, aka 0x01 to generate the sighash.

I missed the extension part. Which is <tapleaf_hash_of_script> <0x00> <0xFFFFFFFF>, and should be appended at the end of sigmsg.

As Peter commented, we need the internal pubkey to verify the script path spending.

Not the internal pubkey, but the pubkey in the script. In this case they are the same, but they don't have to be.


text:
how private key is stored?
How is the private key or master seed managed on iOS or Android?
In copay, using Secure Enclave? And as for Android device how is the private key or master seed managed?

There are too many third party wallets for this to be answerable, and if you want to know about a specific wallet (that is not Bitcoin Core), you will need to ask them about their implementation


text:
A question about full nodes, lightweight nodes and merkle trees
What is the purpose of full nodes storing merkle roots? Why don't they save the transactions only? The hashes could be calculated only when a lightnode asks for them and the merkle roots could be saved only be the lightnodes. Am I missing something here?

I suspect you are confusing merkle roots and merkle trees.
A merkle tree is the binary tree of hashes. The leaves are individual transactions, which are hashed, and then the hashes hashed together until only one hash remains. That remaining hash is the merkle root. A merkle branch is a particular path down the merkle tree from the merkle root to a transaction.
Full nodes store the merkle root because it is part of the block header. The block header contains the merkle root so that the block hash itself will also commit to the transactions contained within the block. While the merkle root itself isn't generally used elsewhere, as part of the block header, it frequently needs to be sent out to other nodes which are downloading blocks. It is expensive to have to recompute the merkle root every time, so instead we just keep it so that blocks can be sent out quickly.
Furthermore, by storing the complete block header, when the transactions themselves are discarded in pruning, the block headers are kept and so that the header chain is still tracked. Without the transactions, it would not be possible to verify this header chain again if the merkle root were not being stored.
Full nodes do not store the entire merkle tree or any merkle branches. These are unnecessary for normal operation. When a SPV client asks for a merkle branch, it is computed on the fly.

I suspect you didn't undestand the question. I don't confuse trees and roots. Regarding the final panagraph of your answer, I thought that a full node stored the whole array of transactions and the merkle root. Why does it store the root? Why is it not calculated on the fly like branches?

That's because the root is part of the block header, and block headers are frequently needed (far more often than the transactions). They could be computed on the fly, but it would slow things down tremendously, while saving just around 18 MB (at time of speaking). Futhermore, this isn't possible at all for pruned nodes (which still need the headers of old blocks, despite not having their transactions anymore).

@PieterWuille If you read the question, I talk about FULL nodes not needing to store the merkle root, not prune nodes. I know that block headers contain the merkle root,what I am asking is WHY the need it.

Pruned nodes can be full. There is no requirement to have all blocks and transactions in order to validate future blocks. The term full nodes refers to the fact that they validate fully; not whether they keep all historical data around.

So why doesn't a prune node calculate the merkle root from the transactions of the first block it stores and continues the validation from there? Why does the blockchain have to store that?

The Merkle root needs to be part of the block header (whether it's stored by nodes or not) for security; otherwise blocks wouldn't commit to their transaction contents. It doesn't technically need to be stored by nodes as it could be computed on the fly, but as I explained, this would massively slow down all operations involving block headers, and only save 18 MB for non-pruning full nodes (out of 250 GB). It's simply not a worthwhile optimization.

why the merkle root must be saved and not calculated on the fly during the verification/validation process

The merkle root is a part of the block header. When mining, it is the block header that miners must construct and then hash, in an attempt to find a valid block. So when mining, a miner must commmit to the transactions that will be included in their blocks. By this mechanism, after a valid proof-of-work has been found, a miner cannot alter which transactions are included in the block.
In other words, the merkle root ties the transactions in the block to the proof-of-work, thus securing the history and ordering of transactions in a way that is relatively simple and computationally non-intensive.

This doesn't answer my question. Why does the block header have to store that? Why isn't it calculated on the fly by the transaction array on the full block? The value would be the same either stored or unstored.

I do think this answers the question: Without storing it in the header, there is no cryptographic commitment tying the transactions to the proof of work. The header is what the miners hash to find a valid proof of work, so it include the merkle root. Without the merkle root, the miners could just find a valid PoW and then append any block of transactions onto it. So even though the root can be computed on the fly, that doesn't solve the issue of needing a cryptographic commitment for transactions. That is the whole point of blocks: to commit to an ordering and history of transactions.

The hash of previous block header could calculate the merkle root of the transactions of the previous block from the transactions of the previous block. The merkle root can still be included in the cryptographic function, without being stored on the header. Am I wrong?

No, if the merkle root is not included in the header, then the previous block's hash would not provide any sort of commitment. The hash (proof of work) is calculated from the block header alone, there is no way to "add the transactions to it later".

Please read again my comment.

@evannemo it might be better to continue this conversation in this site's chatroom ('the mempool'). You last comment does not really make sense: the hash of the previous block is just that block's PoW, it cannot 'calculate the merkle root of the transactions of the previous block'. The merkle root is either included, or not

Why it cannot calculate the merkle root of the the transactions of the previous block? (I don't have enough reputation to continue this on the chat, sorry).

Without the merkle root being explicitly included in the header, the transactions in that block will not be committed to. So there would be no way to verify that a certain set of transactions is the set that is included in a certain block. This commitment must happen before a valid PoW is found, what you are suggesting is to somehow add/calculate it afterwards. The whole point of the merkle root is to make this commitment in advance.

I’ll add a second answer, that is an analogy instead of a technical explanation. As analogies go, it isn’t perfect, but I think it works well enough.
Consider a situation in which you must fill in a lottery card, by picking 5 numbers. This lottery card is then stamped by the lottery authority, so that you cannot change your numbers later to claim a false win. Obviously, it is critically important that you commit to the numbers before the lottery draw takes place. If you could change the numbers later, then the lottery would be broken.
This is similar to bitcoin mining: selecting numbers is like selecting transactions, and then the ‘stamp of approval’ is like the merkle root being committed to the header that hashes to a valid proof of work. Finding a valid proof of work is the mechanism by which an ordering of transactions is selected.
So, without the merkle root being included in the header, the proof of work would not be related to the transactions at all. It would be like miners were just submitting blank lottery cards, that they could change the numbers on later. Since the point of a blockchain is to commit to an ordering of transactions, it is important to commit to the transaction ordering prior to the valid block being found. There is no way to do this afterwards, in a cryptographically secure manner.

The proof of work can calculate the merkle root from the transactions and then it can include it in the hashing function. The algorithm remains the same. The merkle root is just not stored, but calculated every time on the fly. Am I wrong?

Doing that would break several functions, without any real gain. The extra bandwidth and storage required to include the merkle root in the header is very small, compared to the benefit of being able to do things like header-first validation, light-wallets, etc. Note that nodes do recompute the merkle root for each block anyways, as one check to ensure a block is valid. But including the merkle root in the header provides many additional benefits (rather than using it in the PoW computation, but not explicitly storing it, as you suggested).

Full nodes do not maintain the blockchain only for servicing lightweight nodes. The primary reason is for its own security so as to ensure that no blocks that it has received over the network have been tampered with and they all meet the consensus requirements.
In order to verify that the blocks are linked going back to the genesis block, the full node will have to calculate the block header. Block header of a given block contains the previous block header hash and merkle root amongst other fields. The merkle root acts like an identifier to all the transactions included in the block as changing even a slightest component of the transaction can completely change the merkle root. So a full node verifies the transactions, and calculates the merkle root from those transaction hashes. Then this merkle root is used amongst other block header fields like previous block header hash to calculate the current block header hash. When verifying the next block in the blockchain, the node will ensure that the previous block header hash field in the block matches the one it had calculated for the last block. This allows the full node to ensure that all the blocks are linked in a chain without any tampering.
Moreover, the full nodes sync with one another headers first. In simple words, when a node boots up (at start of even after a pause) it would sync with the rest of the network syncing the headers of the blockchain first. And merkle root is part of the block header.

All right. But this still doesn't answer why the merkle root must be saved and not calculated on the fly during the verification/validation process.


text:
What is the complete process and purpose of a block validation using merkle trees?
I need the whole details please. Like does that happen only in the current block? Or at past blocks do? And how exactly does this happen? Do full nodes send the proper hashes and then a local lightweight node calculates them until it reaches the merkle root?
An example with a step-by-step process explanation would be ideal. I need all the techincal details described as simple as possible

Possible duplicate of How does an SPV wallet use the headers that it downloads?

Related: How do SPV wallets learn about incoming transactions?

"does that happen only in the current block? Or at past blocks do?" none of your links answer this. they also don't describe the whole process in details.

Which "process" are you referring to? Are you asking how do full nodes validate the merkle root? Or are you asking how the merkle root is useful to SPV wallets?

@AndrewChow Both. Not the maths. The purpose and the process. Does this happen to past blocks or to the next block to be added on the chain?


text:
Convert EC PRIVATE KEY from .pem into WIF
I have a openssl generated private key in the following form:
$ cat priv.pem
-----BEGIN EC PARAMETERS-----
BgUrgQQACg==
-----END EC PARAMETERS-----
-----BEGIN EC PRIVATE KEY-----
MHQCAQEEIDTijUe0yc3orw5+hElM2WZPWWawO6yTymJZagaStSxIoAcGBSuBBAAK
oUQDQgAE81+mTl3Pyzy41kCg8xgnV3lokrYJ/iUAGda0JUpx99aExBk1kD9Heera
5ndWARwUTa6T1Cfi7c1U6Nf81IEAQQ==
-----END EC PRIVATE KEY-----

How can i convert this (preferably using bash/Perl/python) into a WIF that can be imported into (preferably) Electrum? Also Electrum confuses me a bit, it says
WIF keys are typed in Electrum, based on script type.

A few examples:
p2pkh:KxZ...->
1Dck

What does this mean? Wallet import format does not say anything about typing

How can i convert this (preferably using bash/Perl/python) into a WIF that can be imported into (preferably) Electrum?

OpenSSL .pem files contain base-64 encoding of the values encoded using DER. In case of private keys they use PKCS#8 explained in RFC5208. To extract the key itself, you first have to decode the base-64 string and get the key out by reading the DER encoding (the posted example is missing 1 byte since the sequence length is 0x74 but the remaining bytes that come after it is 0x73 bytes). If you want to use OpenSSL itself, try using this command:
openssl pkey -in priv.pem -out priv.key

This should give you the key bytes. Now you have to convert them to WIF (base-58 with checksum). Use your favorite bitcoin library (I don't have any python suggestion but there are many you could find on GitHub) that has byte-to-Wif functions.

What does this mean? Wallet import format does not say anything about typing

Your one private key gives you one public key that could be used in both compressed and uncompressed form in many different output scripts which then you will see as a different type of address (P2PKH script -> address starting with 1; P2WPKH script -> address starting with bc1; ...). So when the private key is imported in a wallet, the wallet either has to look for all the output script types (like what Bitcoin-Core does) or ask the user to explicitly include the script type (which is what Electrum does).
In other words when you import a private key in Electrum with p2pkh:<key> the wallet only checks the address starting with 1 (legacy or base58) and when import it with p2wpkh:<key> it only checks the address starting with bc1 (native SegWit or bech32) and so on.
As a side note, there is BIP-0178 that proposed extending WIFs to address this and also Electrum briefly used a different way similar to this BIP but stopped because of incompatibility with other clients.

DER encoding contains more than just the key itself, it includes things like the curve parameters. WIF just encodes the private key itself. So after decoding the base64 string, you still have to extract the actual private key. That's just a matter of looking up how DER encoding works and then finding the private key.

OpenSSL supports 8 privatekey PEM formats, only two of them PKCS8, and the example in the Q is not PKCS8 format, it is SEC1. (And as you note, corrupted.) To convert to DER, you need pkey ... -outform der or ec ... -outform der. But you don't really need to; any language that can compute base58check or similar can decode base64 and select bytes.

Yes, you can import the private key generated by OpenSSL to Electrum, Bitcoin Core, Mycelium and some other Bitcoin wallets after converstion to the Wallet Import Format.
There is the bash script for the converstion: https://gist.github.com/GregTonoski/438992249df6e4bd613f9758421ff38a
Usage example:
$ openssl ec -in private_key.openssl.pem -text | grep -A 3 'priv:' | tail -n 3 | tr -d -c [:xdigit:] | xargs bash private_key_into_bitcoin_wif.sh > private_key_bitcoin.wif

The resulting text in the private_key_bitcoin.wif is accepted by Bitcoin wallets.
Side note: in case of Bitcoin Core the import works only for the non-descriptor type of wallet which is created like that: $ bitcoin-cli -named createwallet wallet_name="your_wallet_name" descriptors=false (the default setting is: descriptors=true).

Easiest way, use python

from bit import Key, PrivateKey, PrivateKeyTestnet

# Import PEM file
key = Key.from_pem('path/to/pem_file/here')

# Export to wif

WifFormat = Key.to_wif()

print(WifFormat)


text:
How do I switch a pruned node to "neutrino mode"?
According to Lightning Lab's documentation, Bitcoin Core can be run in "neutrino mode" to serve block data without storing transactions.
I've been running Bitcoin Core in pruning mode and everything is okay. Now I would like to enable this mode to serve data for lnd. The documentation says that adding these parameters will set up Bitcoin Core to run in "neutrino mode":
blockfilterindex=1
peerblockfilters=1

When I add them, Bitcoin Core gives this error:

Error: basic block filter index best block of the index goes beyond
pruned data. Please disable the index or reindex (which will download
the whole blockchain again)

Of course, it is necessary to re-download the blockchain. So my questions are:

Will this configuration cause Bitcoin Core to store the entire blockchain, or will it just keep the headers and continue pruning the blocks?
How can a reindex be triggered?

Are you running a sufficiently recent version? If you are running an older version, you would need to have the full blockchain archive to serve compact client-side block filters.
Originally blockfilterindex and pruning were exclusive features. This was only changed recently when PR#15946 allowed serving compact client-side block filters from pruned nodes. The feature was included in the Bitcoin Core v22.0 release.
You should be able to trigger the reindex by stopping your Bitcoin Core, and then starting Bitcoin Core with the -reindex startup parameter once.

thanks! I'm trying this at the moment, still redownloading the blockchain

Bitcoin Core doesn't run in Neutrino mode, it can only serve compact filters. Neutrino is a Lightning Lab's implementation of BIP157 and BIP158 basic filter type. It is a standalone backend, and you can run lnd without Bitcoin Core.
Moreover, the only way to compute the missing filters is to download all blocks again, compute the filters and then prune again.

Thanks, I updated the question


text:
How can we find the hashrate distribution of different pools?
we can see that different websites propose some graphs in which the hash rate distribution is depicted, like the attached Figure.
1) I want to know that how do these websites approximate this distribution? Are these websites understand the generator of the block by reading the Coinbase Transaction?
2) If AntPool generates a block, does AntPool inform others that it was me who did that?!!

Yes, most pools leave a watermark somewhere. 2) They export the data via API's

Thank you, guys. Why do the pools insert a watermark in the Coinbase transaction? What is the benefit of this?

@tesoke It's their signature. Mining pools attract more miners if they solve more blocks. The blockchain is immutable, so by leaving this record they're leaving their footprint in Bitcoin history. Perhaps there's more to it but as far as I know it's just that.

Thanks, @Jose Fonseca. Therefore, it seems that if a miner inserts a wrong watermark intentionally, that does not belong to him, nobody can detect this. For example, if an independent miner inserts watermark of AntPool although he does not work in that pool. Then, nobody can detect this, Right?

Yes. The miner who solves the block is free to insert anything they want. Most aggregators get the data from the mining pool API's though. There is no incentive to falsely sign. If you have the chance to solve a block, you'd likely insert some legit data to leave your mark on the chain.


text:
How to run and access Blockchain.info platform on a remote server?
I've installed Blockchain.info local machine on a server and got an API key for it. But it seems it's designed to work only locally and on the port 3000.
I need to call APIs from another server with a certain IP. Unfortunately, I could not find anything about how to access to BlockChain.info machine and its APIs by calling a server IP instead of localhost or 127.0.0.1, and how to change the default port 3000 to something else.
May you help me please?

I suspect you have misunderstood what you are installing. Blockchain.info is not open source and there is no ability to run their software on your local machine.

@Anonymous: https://github.com/blockchain/service-my-wallet-v3

That’s not their website or service, that’s a proxy to their API.

What I need is using their API to built our own applications and website. May you guide me how can I do it using their services please or give me a link to a tutorial?


text:
Where does a Client in Bitcoin Read From?
I can understand how Bitcoin replication mechanism work profoundly (i.e. the write operations). However, I don't understand the read operation from a client's point of view (a client is called a user in Nakamoto's paper).
For instance if a web-based client wants to check his balance, which node does he read from? Does he read from a single node (perhaps that node is not updated yet) or several nodes?
Links, citations and references are much appreciated.

Terminology
Using non-standard terminology leads to confusion. Your question uses terms like "client", "web-based". "read" and "write" in ways that are not commonly used when discussing Bitcoin.
Web based client
To me a web-based application is an application that is presented to a human user using a web-browser that uses HTML and CSS perhaps with Javascript and which acts as an HTTP client to communicate with a HTTP server. This is a client-server model of communication. Bitcoin itself primarily uses a peer to peer model. Full-node Bitcoin applications such as Bitcoin-core do not use HTTP they use a peer to peer protocol usually based on TCP port 8333.
This sort of web based application isn't a Bitcoin node and does not participate in the Bitcoin network. It typically communicates in a proprietary way with some 3rd party service who own some Bitcoin. The user of such a web based application owns no money as far as the Bitcoin network is concerned.
Bitcoin was invented to avoid the need for this sort of 3rd party intermediary who has to be trusted.
Balances
Each full node accumulates all the data it needs to calculate a balance for a human. The Bitcoin network itself has no knowledge of balances. Nodes don't need to know balances to interoperate or to validate transactions.
For example, Bitcoin core can present a person with a "balance". But to calculate that it peruses the transaction journal, adds credits and subtracts debits for each "address" of interest. Obviously this total can be cached in persistent storage for efficiency.
So the node read from by a person interested in a balance is typically the local node they run themselves.
For people who don't own Bitcoin themselves but who give money to third parties in return for an IOU denominated in Bitcoin, there is no Bitcoin node that knows money is owed to them and such people have to use proprietary software to consult with the business that issued the IOU. Usually these businesses provide software that creates the illusion that their customers have direct control of Bitcoin.

Things to remember

Bitcoin is cash, not a bank account.
No single person, organisation or consortium owns the Bitcoin network
No single person, organisation or consortium controls Bitcoin standards
No one is in charge of the Bitcoin system
bitcoin.org is just an information site about Bitcoin, with links to various wallet software. It doesn't develop, intermediate, serve, or interact in any way with Bitcoin wallets software except guiding users who want to download one (ref PW comment below).
The system was originally designed so that each person keeps their money under their own control using only software they run on their own computer. It can still be used this way and many do.
This software is called a "wallet" - even though it never contains money. It allows the user to create Bitcoin transactions.
Other businesses sometimes call other things "wallets" even though they don't give the customer custody of their own money as a standard Bitcoin wallet does.
Miners don't really have any special privileges or rights.

Got it. My confusion was about the third party thing. So, a client connected to bitcoin.org is buying BTCs through a third party. Hence, the transaction complies to the third party rules. This way the client is supposed to trust that third party (at least that it is updated), right?

bitcoin.org is not involved in bitcoin transactions.

bitcoin.org is just an information site about Bitcoin, with links to various wallet software. It doesn't develop, intermediate, serve, or interact in any way with Bitcoin wallets software except guiding users who want to download one.

@RedGrittyBrick, I don't know if this is a new feature, but bitcoin.org are selling on this link https://bitcoin.org/en/buy. they are selling through moonpay!


text:
Why Block Chain not Transactions Chain?
I believe that, for some people, the term Block is a sort of holy notion. However, there exist yet some block-free blockchains such as Libra (Sometimes referred to as Zuck Buck).

Unlike previous blockchains, which view the blockchain as a collection
  of blocks of transactions, the Libra Blockchain is a single data
  structure that records the history of transactions and states over
  time.

It seems that the Block concept is a sort of tradition, neither more nor less. Would you please clarify the reason behind the invention of Blocks?

While this question is a bit different, the answer to your previous question covers a lot of aspects of this question as well: https://bitcoin.stackexchange.com/q/91436/5406

Would you please clarify the reason behind the invention of Blocks?

When determining who owns what money, the ordering of transactions becomes important. If I only have $10, and then tell both Alice and Bob that I paid $10 to them, who will rightfully own the $10? Some sort of clock/timestamp is needed, in order to determine who received the payment first, and thus has rightful claim to spend it in the future.
If you just trust some central party to tell you who owns what, then this isn't really much of a problem. The central server can update the ledger according to the order in which they see transactions published, and all the users can follow along.
However, it becomes very tricky to establish a reliable timestamp/ordering of transactions across an untrusted network. How do you know which history is correct? How do you know all of the other nodes will agree with what you think is correct? These problems are formalized as the Byzantine General's problem, and the invention of Bitcoin provided a novel solution to it.
Since we need a way to order transactions, each block can be thought of as 'one tick of the clock', and with each tick of the clock, the state of the ledger is updated. This sort of step-by-step update of the ledger is integral to the function of a monetary network, it is difficult to imagine how a ledger might be maintained if there were not some sort of method to order transactions chronologically.
So in a very general sense, this is 'why blocks were invented'. If you want to just trust some network admin(s) to tell you the state of the ledger, then perhaps you can forgo blocks entirely, but you will also forgo many of the important and interesting properties of the Bitcoin network (as is the case with Libra).

Thank you. There exists no FIFO ordering on Bitcoin (even Merkle tree does not guarantee FIFO). Because, suppose that I spent Bob's 10$ first. Later, the transaction of Alice got chosen to be mined before Bob's transaction (for any reason). So, Alice will get the money. If I did not misunderstood you, you mean that the cause of blocks is not ordering but double-spending, right?

@Mohamad-JaafarNEHME Ordering of transactions is what prevents double spends, the two are related.


text:
What is the meaning of 'change' and 'reserve' in a Bitcoin Core wallet dump?
I'm using libwally to derive addresses from an extended private key. I shared the seed with Bitcoin Core (loaded it in a blank wallet with sethdseed), and noticed that there are 2 addresses in the dump of the wallet file under the same derivation path, one is labeled reserve=1 and is the same that I derive with libwally, the other is labeled change=1 and is different.
I have a few questions about this:

what is exactly the meaning of reserve and change? My guess is that reserve are normal reception addresses and change are, well, change, but I'm still wondering if there's some meaning more specific to Bitcoin Core.
I thought that change addresses were derived with a different path, like the reception addresses all end with ../0/{index} and change with ../1/{index}, or is Bitcoin Core doing things differently of say Electrum?
Besides I wonder how to generate 2 different key pairs with the same master key and path, could I have generate the change address with libwally too?

[EDIT] After doing it one more time I only have one address with the same path in the dump file of Bitcoin Core, so I guess I made a mistake last time, I'll explain it here in case it helps:
To create a wallet with a seed you choose yourself you need 2 commands, createwallet and sethdseed. Besides the required wallet_name, createwallet also takes an optional blank argument. By default it is false, and the command will use a random seed to generate a key pool. But since we want to use our own seed, we need to set it to true. If you dump the newly created wallet, it will be blank, no master key, no key pool, nothing. Then only you can sethdseed with a 32B seed encoded in wif format.
I was having 2 keys under the same derivation path because I first created a wallet with a random seed, and then added my own with sethdseed, so I was having 2 key pools in my dump file. As Andrew said in the answer, it is impossible to have 2 different key pairs with the same master key and path.

what is exactly the meaning of reserve and change? My guess is that reserve are normal reception addresses and change are, well, change, but I'm still wondering if there's some meaning more specific to Bitcoin Core.

reserve means that the key is in the keypool and has not been used yet. This is used for both change and receiving addresses as there are keypools for both.
change means that the key is used for change addresses.

I thought that change addresses were derived with a different path, like the reception addresses all end with ../0/{index} and change with ../1/{index}, or is Bitcoin Core doing things differently of say Electrum?

They are. Bitcoin Core uses m/0'/0'/i' for receiving addresses and m/0'/1'/i' for change. However if you have an older Bitcoin Core wallet, this split doesn't exist and there's only one keypool. But if you are using sethdseed, that shouldn't effect you.

Besides I wonder how to generate 2 different key pairs with the same master key and path, could I have generate the change address with libwally too?

There aren't 2 different key pairs with the same master key and path. Bitcoin Core doesn't do this, and that is not possible with BIP 32 (or any BIP).

Thanks for the answer, I made a test again and this time I indeed only have one address with the same path in BC wallet, I think I messed up last time and didn't add the blank option to createwallet. I was thinking it was indeed impossible to have 2 key pairs with the same master key and path, so I'm relieved my wallet didn't break the laws of mathematics :)


text:
What dependencies are needed to build the dogecoin client?
What are the actual dogecoin client requirements?  Their list doesn't seem to match exactly with what I have available, and I had to go back and forth with apt to even install what I have so far, and I'm by no means confidence that I have the correct dependencies
thufir@dur:~/dogecoin$ head doc/readme-qt.rst -n 20Dogecoin-Qt: Qt4 GUI for Dogecoin
=================================

Build instructions
==================

Debian
------

First, make sure that the required packages for Qt4 development of your
distribution are installed, for Debian and Ubuntu these are:

::

    apt-get install qt4-qmake libqt4-dev build-essential libboost-dev libboost-system-dev \
        libboost-filesystem-dev libboost-program-options-dev libboost-thread-dev \
        libssl-dev libdb++-dev libminiupnpc-dev

then execute the following:

thufir@dur:~/dogecoin$ sudo apt-get install qt4-qmake libqt4-dev build-essential libboost-dev libboost-system-dev libboost-filesystem-dev libboost-program-options-dev libboost-thread-dev  libssl-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libboost-dev is already the newest version.
libboost-program-options-dev is already the newest version.
libboost-filesystem-dev is already the newest version.
libboost-system-dev is already the newest version.
libboost-thread-dev is already the newest version.
libqt4-dev is already the newest version.
libssl-dev is already the newest version.
qt4-qmake is already the newest version.
build-essential is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

You seem to be missing a word in your introductory sentence in "I'm by no means confidence that I <?> the dependencies"

have the correct...dependencies.   You can edit it, Murch, also :)

somehow or another, I got this to compile.  it was some combination of:
install libboost1.48-all-dev libdb++-dev
then install without version number
then compile, compile fails
blocked packages
start over
install without version number
compile works
I don't remember, but it seemed random.
Much frustrating.
If I knew for a fact the correct dependencies, I would say so.  but, the dependencies for the client and the digger (minerd) are, or might be different, and I was compiling both.
All I'm certain of is that the documentation wasn't correct.
Ubuntu 12.04LTS  (probably didn't help, old version)


text:
Is there a way to determine the optimal size of a batched transaction to save on fees?
I've been looking at other questions about fees and batching, but it seems no one asked if there is some way to determine the optimal size of a transaction to save on fees (assuming all inputs are spending from segwit native UTXOs for simplicity).
To expand a bit on what I have in mind: I need to pay various amounts to different people (let's consider their number can be anywhere between 1 and infinite), and I guess that by making one "big" transaction paying them all at once instead of 1 transaction for each of them I can save on fees. But is it the more people I can add up in the same transaction the better, or would I save less at some point if my transaction keeps getting bigger? Is there some model to calculate the "optimal" size for my batched transaction, or maybe it doesn't make sense?
Here's my best guess for now: adding more people means really adding outputs, which makes the transaction grow linearily (assuming all ouputs are pretty standard and roughly of the same size), so assuming the fees are split between each receivers the more people you can onboard on the same transaction the cheaper it gets for everyone.
But at some point the sum of the amounts of the ouputs will grow bigger than the one input I added at first and I'll need to add another input. If I have relatively big ouputs to spend this is probably ok, but if I only have small outputs and/or I'm adding relatively big amounts in the outputs at some point adding one more output could need to add one or maybe more inputs, making the growth in size transaction not worthing the save in fees, so I'd rather just stop it and send the batch at this point.
[EDIT] I came across this article that seems to confirm what I was thinking, and that all things being equal it will always save fees to add a new output.

But at some point the sum of the amounts of the ouputs will grow bigger than the one input I added at first and I'll need to add another input. If I have relatively big ouputs to spend this is probably ok, but if I only have small outputs and/or I'm adding relatively big amounts in the outputs at some point adding one more output could need to add one or maybe more inputs, making the growth in size transaction not worthing the save in fees, so I'd rather just stop it and send the batch at this point.

The addition of another input does not matter, as it would have to be included in the next transaction too anyway.
In any way, a second stage transaction will always be more costy than a single batching transaction as you would duplicate the (non-witness!) meta bytes (version, n° of input, n° of outputs, locktime, etc..).

Ok, so it means that no matter what it is always better to add one more recipient if possible? I know it's a bit of the scope of my question, but shouldn't I also consider the change output I'll get? I mean if adding one more output mean that I will get a very small UTXO or even dust as a change, that I'd need to consolidate after that, maybe I'd rather not batch one more output, what do you think?

I think that the change output is actually an argument for batching transactions. The only alternative to batching is using multiple transactions, which implies many change outputs (and more inputs in later stages).

What you save now by minimizing no of input, maximizing no of output, will become expenses later on for the output owner, where the number of input getting more and more, and the value of each input getting smaller and smaller.
If the output owner is someone else, then continue with minimizing no of input, maximizing no of output. Hot potato fee. I just minimize my $expenses, these outputs belongs to someone else, they will have to take care of the spending expenses later on.
If you said that "But at some point the sum of the amounts of the ouputs will grow bigger than the one input I added at first and I'll need to add another input." then what you should actually consider is consolidating your output into bigger input when the fee is low. Bigger input means more $fund in-flight during transaction.
If the output owner is you, then you should consider a different strategy.

I need to think through thoroughly about what will be the different strategy b4 i edit the answer again.!


text:
Running bitcoin node in docker behind NAT
A running bitcoin node doesn't accept incoming connections when it's running behind NAT in a docker container.
To confirm the problem is related to docker I edited NAT settings on my router. If router's WAN port 8333 redirects to a machine running full node as a process incoming connections are possible. But if WAN port 8333 redirects to a machine running full node in docker container - I get inaccessible from outside bitcoin node.
To test accessability that I used this URL https://bitnodes.earn.com/#join-the-network (the website auto detects your external IP address and auto fills the box; only thing left is to press the button). If the node is accessible the test finishes green, if not - red.

Banning happens for a reason.

Root cause.
The root cause of the problem is default settings in docker container. More precisely, network settings mode which has bridge mode by default. In bridge mode all incoming connections are seen from inside docker container as coming from the same IP address (which is 172.17.0.1). There are multiple hard forks out there so a foreign node running BCH or other forked bitcoin stuff would connect to your node and try to propagate invalid transaction(s). After few attempts your bitcoin node puts such misbehaving incoming connection in a blacklist. Having one connection banned, your node effectively bans all other incoming connections as they are seemed to come from the same IP address.
Confirmation from a technical point of view (via RPC).
To confirm such situtation we can setup RPC to call bitcoin node. Configuration options such as -rpcconnect and -rpcauth can be put into bitcoin.conf file so I skip them here. The command
bitcoin-cli listbanned

shows list of blacklisted misbehaving nodes where we can see only one address.
In my case the output was like that
[
  {
    "address": "172.17.0.1/32",
    "banned_until": 1572109811,
    "ban_created": 1572023411,
    "ban_reason": "node misbehaving"
  }
]

After a successful diagnosis it's possible to clear blacklist with the following command to temporarily clear blacklist
bitcoin-cli clearbanned

Having blacklist cleared we can run another test here https://bitnodes.earn.com/#join-the-network and confirm the node is now visible from outside.
Solution.

Network settings in docker.
The most proper solution.
In my case, running on a Synology NAS, I had to recreate the container to switch off the bridge mode.
Schedule (eg, via cron) a task to clear banned.
Whitelist the only address docker accepts connections from.
Couldn't find a way to permanently implement it.

PS. Can someone tag the topic with a 'nat' and 'docker' tags. It looks like I need some reputation to make it happen.


text:
produce a public key from private key with ecc, with clear description about k=k*g
According to this answer: https://bitcoin.stackexchange.com/a/63996/100526
When he wants to generate a public key from the private key, he has used K=k*G. but I can't understand it's result.
If we assume our private key is (according to the answer) :
0F479245FB19A38A1954C5C7C0EBAB2F9BDFD96A17563EF28A6A4B1A2A764EF4

and according to Secp256k1 G point is:
- Compressed form (prefix 02)
02 79BE667E F9DCBBAC 55A06295 CE870B07 029BFCDB 2DCE28D9 59F2815B 16F81798

 - Uncompressed form (prefix 04)
04 79BE667E F9DCBBAC 55A06295 CE870B07 029BFCDB 2DCE28D9 59F2815B 16F81798
483ADA77 26A3C465 5DA4FBFC 0E1108A8 FD17B448 A6855419 9C47D08F FB10D4B8

So to calculate the public key I used this site: https://www.boxentriq.com/code-breaking/big-number-calculator
I used compressed form and private key:
0279BE667EF9DCBBAC55A06295CE870B07029BFCDB2DCE28D959F2815B16F81798  *  0F479245FB19A38A1954C5C7C0EBAB2F9BDFD96A17563EF28A6A4B1A2A764EF4
and result is :
25d355ebb23e6ce0fd5463bf40e6da2ccc4e2e8ce654db6a8a8e5a275f0d2266005569112cfda60d14f9b6d0c0218cc072047a3b2fcf97aee95a437b4bb6cce0

but in that answer the result is :
02E8445082A72F29B75CA48748A914DF60622A609CACFCE8ED0E35804560741D29

what's wrong?

Read this. You have to multiply key by the point G which is a special process to make it irreversible.

@CodingEnthusiast : I know it. So I wrote in my question.
private key * G.
Is this wrong way to multiply those like  two hex numbers?

@Hamid You have clearly not read it. G is not a number, and the multiplication here is nothing like big integer multiplication. It's an elliptic curve point, and you need elliptic curve multiplication.

@PieterWuille Thank you. Could you please give me a good resource that explaining this clearly? I was totally confused :(

The entire article CodingEnthousiast linked to is about elliptic curve multiplication. It's not a simple operation.

@PieterWuille As a developer I wanted to create a function for this to calculate public key from private key and I thought it's like 2 * 2 = 4. As you told it's not like this. Is there any online tools that we give it private key and it calculate K=k*G ?

As a developer, see https://bitcoin.stackexchange.com/questions/25024/how-do-you-get-a-bitcoin-public-key-from-a-private-key . For an online tool, google online calculator secp256k1 key -> https://paulmillr.com/ecc/ which gives the specified result for your example.

Does this answer your question? How do you get a Bitcoin Public Key from a Private Key

@PieterWuille,  Thank you for your help. I added an answer to clear about k=k*g.

@dave_thompson_085 Thank you for you help. I read all your links and I added an answer to fix my fault.

@CodingEnthusiast Thank you for you help, I lost this page and it was the lost ring. I added an answer to fix my fault.

It was my fault.

G is a point on the Elliptic curve. It means G has a number for X and another one for Y. so uncompressed G in Secp256k1 is G(x,y):
- Uncompressed form (prefix 04)
04 79BE667E F9DCBBAC 55A06295 CE870B07 029BFCDB 2DCE28D9 59F2815B 16F81798
483ADA77 26A3C465 5DA4FBFC 0E1108A8 FD17B448 A6855419 9C47D08F FB10D4B8

X = 79BE667E F9DCBBAC 55A06295 CE870B07 029BFCDB 2DCE28D9 59F2815B 16F81798

Y = 483ADA77 26A3C465 5DA4FBFC 0E1108A8 FD17B448 A6855419 9C47D08F FB10D4B8

And 04 to show this is an uncompressed form of G.
This point is Base point or Generator point that it used in k=K*G.
* is not like * in math like 2*2 = 4. It is a sign to show one of operators on curve.
we have 3 operators on a curve :
-Addition
-Negation
-Doubling

Addition is for add two points. According to we have a point like P1(x1,x2), and another point like Q1(x2,y2), so we can't use 2+2 = 4, we need some other calculation for P+Q.
The result of P+Q is another point on the curve that we can call it R.
Doubling is double a point on the curve, and the result of this is a point on the curve too.
so P+P=R.
When we say k*G, we want to multiple our private key to a point on the curve.
We need to do these :
Convert our private key to binary base.
With a for on all bits we start to check:
For each of this bits we do some doublePoint and addPoints.
This is a predefined role and we have to follow it.
But how we can calculate a addition of two points?
λ = ( yq – yp ) / ( xq – xp)
xR = λ2 – xp – xq
yR = λ(xp – xR) – yP

With this. first we need to calculate λ, it is our slope.
Then we can calculate our x and y for new point. and the result is our
p+q = R

and what about doubling?:
λ = (3x2) +a / (2y)
xR = λ2 - 2x
yR = λ(x - xR) – y

with these, now we can calculate K=k * G.
There are many other tips to calculate Public key from Private key.
This one is a good implementation in PHP:
https://github.com/BitcoinPHP/BitcoinECDSA.php/blob/master/src/BitcoinPHP/BitcoinECDSA/BitcoinECDSA.php#L350

When talking about math using a curve, when we use terms like addition and multiplication, we don't mean normal additional and multiplication. We are referring to operations on the curve that have properties similar to addition and multiplication. To get the correct results, you have to use the actual operations on the curve, not normal addition and multiplication.

Ok thank you,I got it. but in how we calculate it in the computers view? I know about it if we have a point like A and another point like B, with add these two we will have C, and if we reverse c in X line we will have another point. Or about multiplication in the Math view. I want to know how can I write a function that get Private key as input and give us public key as output? All we have in computers like 2*2=4 and I can't find relation between these.

@HamidNaghipour You need a tool that specifically does secp256k1 point addition at least. Then you can do multiplication by repeated addition. For example, you can compute 9k by adding k to itself (giving 2k), then adding that result to itself (giving 4k), then adding that result to itself (giving 8k), then adding k to that result (giving 9*k).

after many researches on this topic, I got it. Now I'm looking a way how to create a function that does point addition. Could you please edit your answer? I want to accept it with these extra information. Thanks a lot.

Thank you for your help. I added the answer that I needed it.

could you please answer my question:
https://bitcoin.stackexchange.com/questions/95676/how-to-getheaders-from-blockchain


text:
can i solo mine with minepeon?
i'm solo mining but it just doesnt feel right that i don't get any accepts while solo mining.
is this right? (see picture below; the top pool with the blacked out adress)

settings:
bfgminer options: -S all -c /opt/minepeon/etc/miner.conf --no-longpoll --coinbase-addr 1NCWZT4hbfPw5LMpPsPauXYdVB9LYKrZM7 --icarus-options 115200:1:1 --icarus-timing 3.0=100

While solo-mining the miner falls back to attempting to create real-difficulty shares, a.k.a. blocks. Hence you won't see a share until you actually find a block. The only reason that pools use lower difficulty shares is in order to determine which miners are actively attempting to mine a block. Since you are the only miner there is no need to keep tabs on the miner, it's already clear who's going to get the reward.
tl;dr: this is perfectly fine, as long as the miner is showing a hashrate you'll find a block eventually, although with your current hashrate it'll take 193 years in expectation for you to find a block.

thanks :) 193 years expected. but the change that i find an block the first minute is there. super tiny, but it's there! :) i'm glad to hear that my mining time didn't go to waste.

There's always luck involved in this, so it might just work out for you :-)


text:
How do I get money on and off of harborly?
I signed up for a harborly account but I'm still not quite sure how it works. I would like to buy some bitcoins, but how do I get money onto the site?

You cannot store your fiat currency directly on harborly. They're a lot like coinbase in that they allow you to sync your bank accounts with your wallet. Instead of sending money to them, then buying and selling, whenever you buy or sell they directly credit or debit your synced bank account.


text:
Is there someway to determine the current network fee from the bitcoin network/daemon?
Is there some way to programmatically extract the current consensus of the transaction fee from the bitcoin network, either by using the daemon or through some other means?

Check the medium fee of last n blocks? I believe the transaction fee is currently not very variable.

Check the medium fee of last n blocks? I believe the transaction fee is currently not very variable. The reference implementation "ensures" that all transactions that are smaller than 1000 bytes, have all outputs over 0,01 BTC and have a large enough priority will be processed, regardless of the transaction fee. Otherwise 0.1 mBTC is paid for every 1000 bytes of transaction (rounded up).
The priority of a transaction can be calculated by [1]
priority = sum(input_value_in_base_units * input_age) / size_in_bytes

With respect to http://meta.bitcoin.stackexchange.com/questions/633/please-post-answers-as-answers-not-as-comments?cb=1, I posted my comment as an answer.


text:
Can Bitcoin save the U.S. Post Office
I noticed this article and was thinking about the possibilities of the U.S. Post Office seriously considering Bitcoin.
http://www.bloomberg.com/news/2014-02-07/could-bitcoin-be-answer-for-profit-starved-post-office-.html
Too Soon?   Too many people don't understand crytocurrencies?

There are some fundamental legal problems with this, as the Post Office is a creature of the Constitution and thus a creation of the United States government. It's not at all clear that the Post Office has authority to engage in brokering financial transactions. It's even doubtful that Congress has the power to allow the Post Office to do that.
That said, for the sake of argument, let's assume the Post Office starts brokering Bitcoin transactions. What then? Well, the point of Bitcoin (oversimplification ahead) is that it is not controlled by a government, is anonymous, and is secure. How many Bitcoin users are going to transact Bitcoin with a government agency whose job it is to route documents to people all over the planet, but is notorious for losing, mishandling, or misdelivering its cargo? I don't see it.
For that matter, how many American citizens or politicians will support inserting the Post Office into the volatile and rapidly evolving crypto-currency world? The two things the Post Office has done predictably over the last few years, other than deliver other people's mail to me, are lose money and ask for taxpayer bailouts. Add in the complexity of a major financial institution and the possibility of a market crash, and you have the makings of a major disaster (think something like LTCM + national debt horror rolled into one).
This doesn't mean I see some nefarious scheme here to "take over" Bitcoin or something. It just means this sounds like a hairbrained idea thought up by a bureaucrat who doesn't really know what Bitcoin actually is.

The postal service already brokers lots of financial transactions, thanks to its money order product. So I don't see constitutional obstructions to them entering this market.

@NateEldredge You're right about the money orders. My point is that the legal objections exist and are real, not that I think there's much chance they would actually prevent the PO from going forward with this idea. I think the constitutional objections are valid and that the same logic means that the PO has no business selling money orders, either. That said, obviously the PO does sell money orders, so anyone bringing a legal challenge to this proposal would have a tough row to hoe in Congress and the courts.


text:
Getting historic exchange data through an API
I'm trying my hands on a trading bot, but I need a website to pull trade data for min. the last 3 days. This can be from any major exchange. Eg. the exact same as this:
https://www.bitstamp.net/api/transactions/
But this only allows retrieval of transactions max. 1 hour old (or am I wrong?)

If you keep pulling data for 3 days, you'll have the data you need.

yeah, but isn't there an already made solution?

Have a look at the API of Bitcoin Charts. They have trade data available for all the major exchanges but delayed by 15 minutes. You can then fill up the gap using the available exchange APIs.

ClueDex is the best solution when you don't want cached or delayed data. You can get raw ticks from any time period back to October 2017. Or, you can get OHLCV candle data from back to the same time frame and in almost any period--even create your own i.e. 15-second or 30-second periods, or whatever. Retrieve all OHLCV candles in a single query and specify the number of candle data you wish to have returned.
Most people want OHLCV candle data as this plugs into the TA-LIB functions for analysis. Not every API delivers this, and ClueDex does so without a delay or caching.

If it's just for bitcoins, have a look at bitoincharts


text:
Newbie running full node (Bitcoin Core): Commands to import wallet with passphrase from external USB stick? And how to sweep paper wallet?
I need help with:

Which console commands (Linux) to use to import external USB stick wallet (with passphrase) created by an earlier version of (Linux) Bitcoin Core?

How to sweep paper wallet (with passphrase)?

For the first time, I want to transfer from a Bitcoin Core wallet to an exchange to sell.


text:
Can miners single handedly perform 51 attack toward the network?
Is this true that miners alone cannot do 51 attack even one or a group of miners posses >51% of hashrate of the network?
As far as I know, every block should be accepted by nodes to be added to the blockchain. In other words, nodes are responsible for maintaining the whole blockchain, so they can choose which blockchain is the best one.
If a group of miners decide to perform an attack such as double spending a coin, nodes can block those miners and so their blocks would not be a part of the blockchain. So any party who owns more than 51% of hashrate can not do any malicious thing, unless it controls 51% of both nodes and miners. Is this claim correct?

I think nodes without economic relevance (or with redundant economic relevance) don't really cause 51% attacks, but eclipse attacks.

51% attack
My understanding is that the 51% attack only requires more than 50% of the total hashpower. It doesn't require control over a majority of nodes.
Nodes each choose the chain with most work as their active chain. If one miner can produce more work than the rest of the miners combined, they can produce a chain with more work than any alternative chain.
Attack detection and reaction
So far as I know, the broad ecosystem of wallet software is not able to explicitly detect an attack of this sort and automatically apply countermeasures consistently.
the human operator of the software may be able to apply some measures such as block invalidation, but I believe every person operating a wallet or other node would have to do this manually. It is likely that people who choose a wallet for ease of use do not actively monitor bitcoin news and would not react in a timely manner. Since there is no established system of coordination between ordinary users, the network would presumably fragment. At least initially. It seems likely this would develop into a major issue with significant long term technical, community and reputational consequences.
The countermeasure to the 51% attack is not a reactive one but more of a preemptive one - it is considered very difficult for one person or organisation to suddenly amass 51% of the network hashpower.
I guess if the more vigilant members of the Bitcoin community detect some gradual centralisation of hashpower rising towards 50%, the response is possibly to engage more people in diversifying hashpower and perhaps develop new measures that aren't currently automatic.

and even if nodes know that one party is attacking it they choose the longest chain? Can't they choose a smaller chain in order to prevent the attack? In other words, choosing the longest chain is a part of the protocol? or just a sign to check which chain has the chance to more healthy?

Another point, adding a block to the block chain at the final step is nodes' job, is this ok? so if nodes detect that one party has gained 51 percentage of the hashpower they can reject its blocks, so their chain would not become the longest one. Is this conclusion right?

@AmirrezaRiahi Yes, yes they can. But do they want to? The goal of the protocol was to introduce an economic incentive to avoid misbehaving parties. If that incentive doesn't work (exemplified by the occurrence of an attack), it means the assumptions underlying the system were wrong, and maybe the conclusion is that PoW in general just doesn't work. You can't patch it up by replacing it with the consensus of users overriding the hashrate - if that was ok in the first, why were you using PoW in the first place, as its goal is specifically avoiding the need for trusted decision makers.

I think the possibility of the users of the system deciding to reject a hashrate majority consensus-valid chain does factor in to the equation, but only to the extent that it is hardly more than a mutually assured destruction mechanism. Yes, if absolutely need be, the users can decided to reject the PoW-decided chain, but it likely be a very catastrophic event (only slightly less bad than the alternative of the attack going through), so much that it becomes undesirable for able parties to even attempt the attack.

@PieterWuille I think in this situation the attack would be much harder and more costly. Because it have to control both nodes and miners other than only miners. Think of PoS which removes miners and has only nodes (I'm not very familiar with PoS in high detail but I just know the concept) and the attack only needs to control 51 percent of stakers. I think this is why PoW is much safer than Pos.

In PoS stakers and nodes are still separate groups. The alternative with just nodes is traditional consensus algorithms like PBFT - but you can't compare these algorithms, as they operate in a very different context (needs trusted parties!).

@PieterWuille why this would be catastrophic? Because there is no secondary chain to consensus?

@PieterWuille I'm not familiar to PoS, would explain why it needs trusted party or a link for explanation?

The Bitcoin ecosystem (humans) needing to come to consensus to discard the result of PoW seems catastrophic to me. Yes, it may be needed in order to avoid a 51% attacker preventing progress to the chain for example, but it's making it evident that PoW just doesn't work, as its entire point of existence is avoiding the need for humans to decide which version of history is the correct one. That combined with the fact that it inevitably requires political consensus about this decision to be made, which would raise questions about whether it can happen again.

So to be clear my point here is: when we're talking about Bitcoin nodes, we're assuming nodes which faithfully follow the most-work valid chain, even if that chain is produced by an attacker in some form. While yes, indeed, node operators could jointly decide to discard that result and switch over to another chain, and functionality for doing so even exists in Bitcoin Core (the invalidateblock RPC), I'd argue that if you're talking about a system where this mechanism actually needs to be used and is relied upon, it's not anything like Bitcoin as we know it anymore.


text:
Geographic distribution of Bitcoin users
I haven't found recent question about this subject. I wanted to know if there was a site or a database with the geographic distribution of Bitcoin users freely available ? I know that there are several limitations but I've seen research papers (and Chainalysis) estimating this. The main goal is possibly to have time series of volume of transactions per country.

Geographic distribution of Bitcoin users

No one can know the geographic distribution of Bitcoin users with any certainty.
For example, a user in France might access a Bitcoin node in the USA. The node might not know the location of the user. In addition, the user might use a VPN that disguises their location.
As another example, how would anyone collect the location of a Bitcoin user who moved their money into a paper wallet and relocated?
The locations of nodes is not a reliable indication of the locations of users.

The main goal is possibly to have time series of volume of transactions per country.

I doubt this is straightforward. If a user in France and a user in Japan provide a 2 of 3 multi-signature input to a transaction that pays someone in Mexico, what countries' volumes does that transaction contribute towards?
Given my first point, this is moot anyway.

Related : How much bitcoin in Canada?.

This isn't exactly what you want but Bitnodes has a breakdown of reachable full nodes by geography. As RedGrittyBrick discusses this breakdown doesn't necessarily give an accurate indication of volume of transactions by geography and indeed Bitcoin Core developers have in the past (and continue to) made it as hard as possible for a network observer to assess from which full node a particular transaction was initially broadcast from.


text:
Can I create an entirely new m-of-n multi-sig wallet with the same seed phrases as an existing one by increasing n and discarding the new seed phrase?
If I were to create a 2-of-4 multi-sig wallet using the 3 seed phrases of an existing 2-of-3 wallet, plus a new seed phrase that I instantly destroy, would I then effectively have a new completely independent 2-of-3 (technically 2-of-4) multi-sig wallet that uses the same seed phrases as the existing wallet?

You would have to transfer any funds from the 2-of-3 multisig address to the 2-of-4 multisig address on the blockchain. But yes you could do this. I don't know why you would do this! Assuming the 2-of-3 can be spent using 2 of the 3 private keys (A, B and C) the 2-of-4 can also be spent using 2 of the 3 private keys because you threw away the fourth. Maybe to obfuscate the fact you are using a 2-of-3 on the blockchain? The potential benefits of doing this are minimal if not zero.

One possible reason would be if you had securely created, stored and backed up your existing seeds for a wallet containing bitcoin from a KYC exchange, but wanted to start accumulating non-KYC bitcoin in a new wallet while avoiding having to manage new seeds..

@evo_race: Ah ok, fair enough. Yeah it would be a different address but function exactly the same. If you spent from both addresses you would leak the fact that they were both using the same public keys. I don't know if a KYC exchange would notice. But while you haven't spent from the 2-of-4 address the public keys wouldn't be visible on the blockchain.

Thanks - I didn't appreciate that the wallets could be linked on the blockchain through their public keys.

Addresses can be linked once you've spent from them. Kinda fits into the address reuse thing. It isn't clear it is a multisig, it is a 2-of-4 multisig or what pubkeys it is using until you spend from that address when all of that is revealed.

Taproot and things like MuSig2, FROST improves on this.


text:
Is it possible/feasible for nodes to redact information (for example inappropriate images) in their blockchain and remain part of the network?
With the freedom to run nodes being critical to bitcoin's success, I'm concerned that if someone successfully writes highly inappropriate or abhorrent information/images as a valid transaction then many people will be put off running their own node and acting as a host for this content, thereby creating a threat to bitcoin as a whole.
I believe this issue could be successfully mitigated by allowing nodes to redact (i.e. make unavailable) select parts of the data on their blockchain. Is it possible or at least feasible for a node to do this while otherwise being able to participate fully in the bitcoin network?

So first of all this has already happened e.g. "Child Abuse Content Found On Bitcoin Blockchain, Users Must Be Protected" (reference, Coin Telegraph).
If your aim is to maximize censorship resistance and restrict the ability of state authorities to prevent certain transactions from being mined then you can't prevent any consensus valid, high fee rate transaction from being mined.
However, the OP_RETURN limit of 80 bytes is tiny in comparison to typical image sizes that can get up to 1-2MB for a high quality photograph. So for an image (abhorrent or otherwise) the best one can do is split an image into tiny chunks and spread it across many OP_RETURNs and many transactions. There are many people who dislike Bitcoin for various reasons and would do anything to try to prevent its growth so it is inevitable that bad actors will do this. But no single transaction or even a single block can contain an entire image. It becomes somewhat farcical if you take this concern to extremes by splitting an image into say bytes and storing a single byte on every page of a website. It would be impossible to detect this and visitors to the site or hosts of the site could not be reasonably held liable.
correction: As RedGrittyBrick points out in the comments since the SegWit soft fork it has been possible to get a >2MB image into a single Bitcoin transaction and an example is this transaction 0301e0480b....

Michael, when you write "for an image (abhorrent or otherwise) the best one can do is split an image into tiny chunks and spread it across many OP_RETURNs and many transactions. " were you unaware of  "the network mined its largest block, nearly 4 MB in size, containing just 63 transactions. One of the transactions was a 3.94 MB Ordinal inscription featuring an image of a wizard," reported here. Search tx for ASCII "image"

https://mempool.space/tx/0301e0480b374b32851a9462db29dc19fe830a7f7d7a88b81612b9d42099c0ae

I am having difficulty reconciling this single transaction containing a > 3MB image with the assertion that a < 1-2MB image must be split across many transactions.

@RedGrittyBrick: Yeah you're right and hence my answer is incorrect. I wasn't aware that was possible. Submitted directly to a miner and not paying a fee. Starting to think that Taproot relaxation was a bad idea now :)

For what it is worth, my opinion is that most of your answer is right and blockchain polluters will likely fragment and obfuscate objectionable content if nodes start censoring them.

If you ignore policy rules, taproot didn't change anything w.r.t. embedding data. Whatever people do now to include a 2+ MB image in a transaction was perfectly legal before the taproot softfork too (it must be; if it wasn't, it would not be a softfork).

Thanks Pieter. Gosh it is hard to follow what relaxed limits are enabling what. I thought (mistakenly) that this transaction was using the BIP342 relaxed script limit.

@MichaelFolkson The standard ones do, but the non-standard ones obviously aren't subject to any policy limits.

https://github.com/bitcoin/bips/blob/master/bip-0342.mediawiki#cite_note-9 This is consensus rather than policy though right?

@MichaelFolkson Yes, but it applies to different outputs. P2TR applies to v1 outputs, P2WPKH/P2WSH rules apply to v0 outputs. Before the introduction of P2TR one could just have used a v1 output + spend thereof, which was unemcumbered then. And even today, if one is willing to ignore policy rules, one could use (say) a v13 witness output + spend thereof. It's anyone can spend, but perfectly consensus valid (and must be, to keep it available for future softforks).

And even if one wants to stick to existing output types for some reason (but still not care about policy), it has since segwit been possible to create a very close to 4MB transaction (it would need to split it up in 520-byte witness stack items, or 10000-byte scripts, though, but that barely affects the efficiency).

Is it possible/feasible for nodes to redact information (for example inappropriate images) in their blockchain and remain part of the network?

Sidenote:
Although OP_RETURN was intended for embedding small amounts of data, I believe almost all cases of embedding of images in the last 12 months use OP_FALSE OP_IF <multiple data pushes> OP_ENDIF inside an otherwise normal script in a witness component. This avoids limitations imposed on OP_RETURN and gains the Segwit discount on transaction fees.
Censoring transactions:
Pruning nodes discard blocks that have been validated and whose content is no longer necessary for maintaining UTXO state. It must be feasible to be selective and have specific criteria to discard data that is unwanted. For Bitcoin Core I imagine it might necessitate rewriting large blockfiles or some strategy such as overwriting portions with say zeroes and updating indexes accordingly.
Not relaying unconfirmed transactions of this sort has been suggested. As Michael Folkson commented, large images (e.g. several MB) already can't be relayed by Bitcoin core nodes because of its default policy rules. publishers of large images have to make direct arrangements with miners.
However, the above still requires that nodes download and validate new blocks, regardless of content, in order to maintain UTXO state. So the goal cannot be completely achieved.
I believe it is also computationally infeasible for normal nodes to distinguish inappropriate images from acceptable images. Furthermore, People who embed such images can take countermeasures to prevent their detection.

I'm pretty sure existing Core default policy (e.g. datacarriersize?) would already prevent the transaction you highlighted being relayed.

Well this specific transaction wasn't paying a fee either so it definitely wasn't relayed via P2P network default policy rules. But a fee paying equivalent of that transaction.

I can only speak for my self but downloading and validating all transaction data would be fine - it's being forced to then knowingly store and host particular obnoxious data on my node that I'd be against. I'd be happy for the redaction to be manual as I can't see it being a common occurrence

@evo_race. I'm not a Bitcoin wallet developer but I believe it would be possible to develop a wallet that does not permanently store images embedded in this specific way. Either a new wallet or a patch to (possibly a fork of) an existing wallet. In my (completely unimportant) opinion, this might ameliorate the issue to a limited extent for a while and be acceptable to some as a partial solution.

@Michael, many thanks for pointing that out, answer updated accordingly.

@evo_race: You don't have to do anything. You don't have to verify transactions let alone store parts of them if you don't want. OP_RETURN makes it clearer that a transaction is including data whilst with some other transactions it is less clear what they are doing. If you want to do full verification you have to keep UTXOs around in case they are ever spent. But if you don't want to store OP_RETURNs or say SegWit witnesses post verification you don't have to.