text:
Can Bitcoin nodes vote to increase the money supply beyond 21 million?
Is it possible for 51% of the Bitcoin nodes to pool and vote to increase the money supply limit? Is that allowed in the protocol?

The reward schedule is a consensus rule in the Bitcoin network. Consensus rules are enforced by each full node independently. Node operators choose which consensus rules they enforce per the Bitcoin implementation they are running.
If a subset of the nodes were to increase the block subsidy, this would be a hardfork: a non-forward compatible consensus rule change.
Blocks following the new rules with a higher block subsidy would be in conflict with the original rules and invalid to any nodes that did not update to the new rule set. Depending on whether either one, the other, or both resulting p2p networks have sufficient hashrate, this would potentially split the network irreconcilably, as nodes allowing the new higher reward would get disconnected and banned for sending invalid data by nodes enforcing the original rules. Since the old rules remain a subset of the new rules, a majority of the hashrate siding with the old rules would wipe-out the chain with new rules as nodes with new rules would follow that chain. Every new block with new rules would interrupt the node topology again, though.
It is irrelevant how many nodes are participating in the rule change. Nodes enforcing the original rules would never follow the chain with new blocks, the only way to switch over to the new rules would be by switching to new client software that implements the new rules. The node count would also only indirectly affect the survival of the competing chains as that would chiefly depend on which ruleset garners the majority of the hashrate, which in itself would depend on the rules that are adopted by the economic majority and user population.

REQUEST: If you downvote because you disagree, please specifically
point out where the answer either (1) got the facts wrong, or (2) made a
reasoning error. Just saying you disagree with the conclusions may be
expressive, but does nothing to improve the answer.

First a technicality: It does not matter what 51% of the nodes do. Despite common opinion, , bitcoin nodes have very little say in rule enforcement and changes. You know this must be true since you can currently set up a node for less than $25 and run it for less than $1 per month. Additionally, it is not hard to make one physical node look like a huge number of logical nodes to the bitcoin network.
Practically speaking, the rules are set by an interplay between the miners and a few influential participants. If a segment of miners that controlled a significant part of the total hash power on the network implemented a rule change, and a few influential participants (like, say, Coinbase.com) followed along, then the change would effectively become a new bitcoin consensus rule. In fact, according to the current rules about how the rules change, all changes are based solely on votes cast by miners.
So instead I will answer the more general version of your question: "What prevents the bitcoin consensus rules from being updated to allow more than 21 million total coins?"
This is a very interesting question, and it is rarely discussed seriously.
Today, the thing that prevents the coin limit from being modified is: incentives. Miners control the rules, and large miners (actors who control large parts of the hash power) are currently very profitable due to the current combination of (1) the current price of bitcoin, (2) the current cost of mining a bitcoin, and (3) the current block reward. As long as large miners can make excess profits while mining with the current consensus rules (including the 21 million coin limit rule), they will continue to do so.
But miners will only mine when it is profitable to do so, and this fact will eventually create incentives to increase the bitcoin money supply.
Bitcoin users want both low transaction fees and high network security (aka hash power). Today they can have both because block reward massively subidizes mining operations. For example, in the most recent block mined when I wrote this answer, the miner earned $305,000 from the block reward and $155 from transaction fees.
Today we get both low fees and high security because almost all of the costs of mining blocks are paid for by block rewards.
But this will not always be true. The 21 million coin cap - by definition - limits the total number amount of block rewards payable. Each block reward is effectively a one time sale of a limited resource, and the proceeds from each sale are used to pay for hash power.
As we run out of block rewards to pay to miners, either (1) fees will need to rise, or (2) network security will need to go down (or some combination of those). A lot. Really, really a lot.
Realistically this will not happen. A bitcoin network with massively higher fees or massively lower security is not viable. So as fees start to rise and/or security starts to fall, we will eventually reach a point where something has to be fixed. But of course the only possible fix is to increase (or at least mitigate further reductions in) the block reward.
Remember the 21 million coin limit is not a rule itself, it is a result of the of the ever-decreasing block reward. If you increase (or even stop decreasing) the block reward, then you inherently increase the total number of bitcoins that can mined, and you debase the currency.
So that is how it will happen that the 21 million coin absolute limit myth will dissolve. It will make total sense then. There will be a few loud people screaming "but if we raise the limit then we completely destroy the original basis of value of the system!"  And they will be right. But no one will care because the only other options will be to massively raise the transaction fees or massively lower the network security, and either one of those options would destroy the value of the system.
Does all this sound familiar? Remember that the number of possible dollars was once limited by a rule (the number of gold atoms available). When that rule was changed, pretty much everyone went along with it for the same reasons as they will when the bitcoin rules are changed. Bitcoin is subject to the same inflationary politics as (almost1) every other currency.

There is exactly one unit of money that truly has a fixed supply. I'm going to write an article about it soon, so sign up for updates at josh.com.

I understand that you strongly disagree, but I want to hear counter arguments with supporting information about why you disagree. How about moving this to a better forum where we can explore your objections point by point? A stackexchange public chat room perhaps?

The Taproot rule change was literally was voted on by miners. They cast their votes by putting special values into their mined blocks that indicated if they approved of the change. The only way to vote was to produce a mined block - which by definition only miners can do. Google BIP9 to see how this voting mechanism works.

RE Cherry picking: I just checked the last 100 or so blocks and the average block reward is more than 100x bigger than the transaction fees with many bigger than 1000x.  This is a page that displays the most recent blocks in a table with reward and fees broken out...
https://btc.com/btc/blocks

Let us continue this discussion in chat.


text:
Bitcoin Fraud/Recovery Protocol Proposals
Does Bitcoin plan to implement any Fraud/Loss Prevention in their protocol? I just read a story of how Brokerage Exchange Shapeshift, and Internal Employee stealing private keys from their customers.  https://news.bitcoin.com/looting-fox-sabotage-shapeshift/
Also heard about Bitfinex and Binance having similar issues.

Maybe allowing nodes to create new/recover Bitcoins that were stolen? And expiring tokens from the thief?

Restricting consumption of any bitcoins with a certain publickeys/IP address?

When using credit card, I had thousands of dollars charged. My credit card company said not worry, and reverted the charges.
I can see SEC or FDIC having something similar come to Crypto, and hopefully bitcoin community can address issues in their next protocol. Otherwise, government regulators may interfere.

I asked similar question about insurance few weeks ago, but now asking about protocol actual solution proposals for Bitcoin Fraud/Money recovery

similar how tellers often help internally with bank thieves, employees with admin privilege at Bitcoin wallet companies, can steal private keys, etc https://www.theverge.com/2013/12/19/5183356/how-to-steal-bitcoin-in-three-easy-steps

why is this voted down? trying to learn

As the answers say "does Bitcoin plan to" suggests there is a plan by an individual or group of individuals who decide what happens to Bitcoin. The short answer to this is no because "Bitcoin" is a protocol, network and currency and doesn't plan to do anything.

The question is framed in a way that does not align with reality.

Does Bitcoin plan to implement any Fraud/Loss Prevention in their protocol?

Bitcoin doesn't exist as a person, corporation or other entity
The question implies the existence of a person or organisation named Bitcoin.
As Mark H pointed out, there is no entity named Bitcoin who owns the Bitcoin protocol. There is not even any group of people who can claim exclusive control of Bitcoin.
Yes there are all sorts of people who think about fraud and Bitcoin, but they are not a well defined group with a well defined membership. Anyone can think about this and make proposals.
However there is another reason why this is irrelevant:
Cash is not Banking
Bitcoin is digital cash and this is fundamental to any answer to this question. Fraud prevention in cash is a different problem to fraud prevention in banking. Satoshi Nakamoto's Bitcoin Whitepaper begins

Bitcoin: A Peer-to-Peer Electronic Cash System
Satoshi Nakamoto
satoshin@gmx.com
www.bitcoin.org
Abstract. A purely peer-to-peer version of electronic cash would allow online
payments to be sent directly from one party to another without going through a
financial institution.

Note the repeated use of the word cash there. Note the rejection of any role for any kind of financial institution.

What do other systems of cash do?
Now we can ask ourselves, who designs and creates US dollar bills (banknotes) and what fraud prevention measures do they use?
The answer is the US Treasury Department's Bureau of Engraving and Printing. The fraud prevention measures they take are, so far as I know, entirely measures by which one can detect forgeries. They include special papers and inks, special printing methods and so on. These are all features built in to the currency.
Note that the US Treasury Department's Bureau of Engraving and Printing do not provide a service where you can phone them up and say Mr X swindled me, please trace Mr X and remove five $10 bills from Mr X's pocket and then place them in my pocket. That is not the job of the US Treasury Department's Bureau of Engraving and Printing.
In this system of cash, it is the police and the courts who remove cash from the pockets of swindlers and return the cash to the pockets of victims. It is their job, not the job of those who design the bills (banknotes).
Solutions?
Note that there is nothing in the design of the US dollar bill that in reality facilitates the work of US police in determining which party is telling the truth or tracing the location of Mr X or working out where he put those bills. The serial numbers are some potential help but few of us write down the serial numbers of money we receive or pay. Transaction Ids in Bitcoin might play a similar role.
So what do users of cash like the US Dollar do?
The answer is as old as cash. Escrow and the courts.

Conclusion
The job of the people who design the cash is to prevent fraud by preventing forgeries. It is not their job to intervene in disputed transactions or to move money between citizens.

Does Bitcoin plan to implement any Fraud/Loss Prevention in their protocol?

"Bitcoin" doesn't plan anything. It's a network of software nodes run by individual operators, who each decide whether or not they accept transactions as valid based on the transaction's inclusion in the valid chain with the longest proof of work.
There is no "their." Bitcoin doesn't belong to anybody. If someone has a proposal to improve bitcoin they must publish the idea and attempt to have the overwhelming majority of bitcoin validators run software which incorporates the changes.

Maybe allowing nodes to create new/recover Bitcoins that were stolen? And expiring tokens from the thief?

In regards to having some kind of backdoor, as you suggest, to revert bitcoin transactions, I think that almost all bitcoin node operators will decisively say NO! The fact that bitcoin transactions are irreversible is a feature, not a bug.
Bitcoin eliminates buyer fraud. Once a transaction has been confirmed past a few blocks, it is then absolutely irreversible. Whether a recipient of some funds would like to give a refund is a separate matter and is for dispute between the respective parties only - it is of no concern to other bitcoin users. As a consumer, you might find the ability to reverse credit card transactions "desirable," but businesses accepting credit card payments certainly don't. Many businesses suffer significant losses because of credit card fraud, and will be delighted to accept irreversible transactions.
The matter of seller fraud must be addressed by other means such as third-party escrows.
As for exchanges not being able to internally manage their data security - this is their problem only and is not of concern for the rest of the bitcoin network.

Restricting consumption of any bitcoins with a certain publickeys/IP address?

The fact that you can't censor certain transactions outputs from being spent is also a feature and not a bug. Bitcoin is intended to be for everyone. Any tool intended for censoring "fraudulent" transactions will inevitably be used for other nefarious purposes.

I can see SEC or FDIC having something similar come to Crypto, and hopefully bitcoin community can address issues in their next protocol. Otherwise, government regulators may interfere.

Bitcoin is governed by mathematics and individuals deciding whether or not they accept a global source of truth. Anybody can attempt to contribute to this truth by mining bitcoin blocks, and they do not require any permission. There is not going to be any regulation that targets bitcoin itself - such thing is unenforcible and entirely a folly. Regulators can only hope to take on businesses which deal in exchanging legacy fiat for bitcoin, which is unrelated to the bitcoin protocol.
And there is also no "their next protocol". Bitcoin is already here, it works, and its rules are being enforced by tens of thousands of unrelated people who each assume that every other party in this network is potentially malicious.The protocol can and is being improved, but it will certainly not be getting these kinds of backdoors. It is unlikely that any kind of "hard-fork" will ever be accepted by the majority of bitcoin users.
Welcome to the new age of personal responsibility.

interesting all things you say above are true; I have a feeling regulators will be involved and will enforce changes on corporate entities, who abide by rules in sec  or have risk fines/penalties, or with company of 1000+ employees, someone will disclose non-compliance issues,

however these rules cannot be enforced in individual levels , hopefully lightning network finds a solution, because if bitfinex, binance, and shapeshift don't even have personal responsibility to enforce security with highly encrypted systems, good luck to mom and pop regular people who aren't computer experts;  bitcoin may never reach large market corporate penetration, and 'governments will never let a crisis go to waste, to exert control next time an exchange has money lost, appreciate the answer, thanks

If a transfer was a fraud or not is not decidable by a decentral network. If you put an instance in between then you created a new banking system with all it's corruption and political influence. So imagine someone transferred 1 BTC to another person against cash and claims afterwards the BTC was stolen. Whom to believe and who should make the decision? It's like with cash today, if it's gone you can go to police and only in case you have good evidence you have a chance to get anything back. If you implement your proposal in Bitcoin then it's not Bitcoin anymore, it's maximal an altcoin then.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How do you retrieve your wallet balance with HWI?
How do you retrieve your wallet balance with HWI?

You can't.
HWI is stateless - it has no way to query the blockchain and does not store any information about your  hardware wallets, so it is unable to know or retrieve your hardware wallet's balance. Furthermore, hardware wallets are essentially offline devices and also have no way to know or retrieve its balance.
This is why a software wallet must be used with a hardware wallet as the software wallet will watch your hardware wallet's addresses, learn about its UTXOs, and be able to create transactions.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Where is the bitcoin source code is the 21 million hard cap stated?
I'd like to be able to see the 21,000,000 number.

Does this answer your question? What actually caps Bitcoin's supply at 21 million?

I don't think that actually answers OP's question; it goes into how consensus rules define the limit, but doesn't show the source code that actually effects this.

Where is the bitcoin source code is the 21 million hard cap stated?

Nowhere, because there isn't actually a 21 million cap rule.
The relevant code is this:
CAmount GetBlockSubsidy(int nHeight, const Consensus::Params& consensusParams)
{
    int halvings = nHeight / consensusParams.nSubsidyHalvingInterval;
    // Force block reward to zero when right shift is undefined.
    if (halvings >= 64)
        return 0;

    CAmount nSubsidy = 50 * COIN;
    // Subsidy is cut in half every 210,000 blocks which will occur approximately every 4 years.
    nSubsidy >>= halvings;
    return nSubsidy;
}

// consensusParams.nSubsidyHalvingInterval = 210000.

It computes the maximum subsidy a miner can claim at a given block height, and this function effectively controls Bitcoin's inflation schedule. The often-stated "21 million" limit is just the approximate result of summing all coins this function permits to be brought into circulation, over all time.
The code, in short, implements the following:

The first 210000 blocks (roughly 4 years) allow up to 50 BTC subsidy each.
Then 210000 blocks with 25 BTC each
Then 210000 blocks with 12.5 BTC each
...
After 10 halvings, the subsidy becomes 0.04882812 BTC rather than 0.048828125 (which would need half-satoshi precision, and the code uses integer division which rounds down).
After 33 halvings, the subsidy becomes 0.

If one sums up all the subsidy values over all halvings, the result is 20999999.9769 BTC, not 21000000 BTC. Though, due to various minutiae, that's also not the real limit; this answer goes into more detail.

Now, the code does also contain this constant:
/** No amount larger than this (in satoshi) is valid.
 *
 * Note that this constant is *not* the total money supply, which in Bitcoin
 * currently happens to be less than 21,000,000 BTC for various reasons, but
 * rather a sanity check. As this sanity check is used by consensus-critical
 * validation code, the exact value of the MAX_MONEY constant is consensus
 * critical; in unusual circumstances like a(nother) overflow bug that allowed
 * for the creation of coins out of thin air modification could lead to a fork.
 * */
static constexpr CAmount MAX_MONEY = 21000000 * COIN;

which, as the comment explains, does not actually control the total supply (as that's rather the result of summing all the subsidy), but is used as a sanity check in many places.

What about this https://github.com/bitcoin/bitcoin/blob/master/src/consensus/amount.h. I came across it this morning whilst browsing through doxygen.

I've cited this file already in my answer above.

i wanna add additional info towards what Pieter wrote:
it would not be wise to change into more than that for shown MAX_MONEY value before the year 2100 and only into the next million, if even. for synching into 22nd century (22000000) next.
(but synching code had also been written in the past. so maybe its not even a good idea towards that. making changes at all from here on towards that value. Ever! otherwise: "out of synch" problems)
this ensures its not growing larger than what the century it exist inside is capable to offer in limitation capabilities for growth itself.
however you will loose structural integrity towards that build like it is today.
meaning: an extension may not be an advantage here! but its too far away to tell for sure yet, so its more a warning!!!
a shorter number is like a triangle vs. a square that would be the next bigger number if the shorter is the triangle.
transfering only 3 lines takes up lesser space in bytes than transfering 4 lines the same size a line.
we are more on an advantage with the number 21 here cause 2+1 in crosssum gives us the principle of the 3 lines i mentioned above as well.
sure you can increase further depending on further centuries it develops into.
but you can also work yourself downwards to its beginnings. when 21 wasnt even in place yet towards the millions shown today.
choice is yours! (to point on forking into own chain ideas and values solutions for that)
and regarding: sanity checking, it wasnt named that way without reason. cause it would be insane to meddle with that entry by people in the wrong mind for the burden it takes to keep it the way it is and should be! to make sure things running stable (the most important aspect from a developers perspective i would say anyways).
(a static and constexpr for this variable also points strongly on this fact)
but where it will be growing into more. its up to future generations to deal with that problem i guess.
still it cannot hurt to have mentioned that i think still important relationship towards that number. 21st century tech build for 21st century like it is right now! an alone important monetization aspect for the network itself. and maybe most important future anchor as well!
there is also a bit more to it. but from my experience regarding its source code. the century linking does the job best! while anything how it became that way. was how it grew into that.
long enough time to figuring out from now on. if we should make a change for it or not in the future! i would call this a fair deal as well towards the yet unknown future in general understanding!

Hi CrashOverride, I’m having a hard time following your train of thought. Could you perhaps try to read your answer as if you were looking at it with a fresh set of eyes to ensure that you are providing enough information to establish your line of argument? Also, answer posts on Bitcoin Stack Exchange are meant to provide their own standalone answer to the question. It seems to me that you are trying to discuss Pieter’s answer, but our platform is a poor fit for discussions.

i did what you asked of me. it may be a long explanation. but it is solid. detailed and simple written to understand it by normal coders. you do not need to be a pro coder to understand what i wrote about. but the problem is, it was simple explained. and this is the main reason why people struggle with it.
i wasnt trying to discuss his answer. i was extending it and going into more details about the number limitation per se. i was pointing on a potential change but with a warning. and i explained why i think it has its reasons for that warning. but also for a potential change.


text:
SHA-256 hashing algorithm
Where in the Bitcoin github repo is the SHA-256 hashing algorithm defined? Is it a methematical formula?

The SHA-256 algorithm is not defined in the Bitcoin GitHub repository, it is defined in the USA Federal Information Processing Standards  (FIPS) publication 180-2 (see Wikipedia: SHA-2).
The implementation in Bitcoin Core is in src/crypto/sha256.cpp,  src/crypto/sha256.h
and related source files and/or dependencies.

If I was looking for a definition that might include a mathematical formalism that describes the algorithm, I would read the FIPS document.
If I was looking for working C++ code available under an open source licence, I would look at the above files in the Github repository.


text:
How can I use the RPC API on Mac?
This might be simple question but I would highly appreciate if anyone can bring their wisdom on this.
If you install bitcoin-core software through https://bitcoincore.org/, it gives you dmg file that automatically install the bitcoin core (bitcoin-qt). I was assuming that you can still use JSON-RPC-API just with bitcoin-qt since it still communicates with the network and the GUI and RPC console. However, even when I enabled RPC server, when I use curl command which is specified on this wiki page: https://en.bitcoin.it/wiki/API_reference_(JSON-RPC), it doesn't return any result.
Does anyone know if it is still possible to use JSON-RPC API with just bitcoin-qt installed?   If not, is there a way to install bitcoind/bitcoin-cli without going through IBD since bitcoin core already has all the blocks installed? It seems like unnecessary to redownload all the blocks while they are stored locally. If this is possible, how can I achieve this? Ideally, I think I should be able to use JSON-RPC-API just by installing bitcoin-qt from bitcoincore.org.

When I check GUI, I can see "Enable RPC server" checkbox is clicked.

However, even when I enabled RPC server, when I use curl command which is specified on this wiki page: https://en.bitcoin.it/wiki/API_reference_(JSON-RPC), it doesn't return any result.

You need to configure a username and password for the JSON-RPC if you don't use bitcoin-cli (in the bitcoin.conf file, using the rpcuser and rpcpassword options, or alternatively the more modern and secure rpcauth option).

Does anyone know if it is still possible to use JSON-RPC API with just bitcoin-qt installed?

Yes, just enable the RPC server in the GUI, or start it with the -server command-line option.

If not, is there a way to install bitcoind/bitcoin-cli without going through IBD since bitcoin core already has all the blocks installed? It seems like unnecessary to redownload all the blocks while they are stored locally.

You can also install bitcoind, which has the RPC server enabled by default, but lacks the GUI.
If you run them with the same data directory you can even alternate between the two without redownloading. You can't run both simultaneously though, but there also shouldn't be any need, since bitcoin-qt also contains an RPC server.
bitcoin-cli is independent, it's only an RPC client to communicate with either bitcoind or bitcoin-qt.

If this is possible, how can I achieve this?

Just install the relevant binaries. The DMG installer only contains bitcoin-qt I believe; if you want the other ones, you need to get the .tgz file.

How does the curl command should look like if you are using rpcauth option? RPC server seems to be enabled but still returning no result.
Also, where do I get .tgz file fore bitcoin-cli?

See https://bitcoincore.org/en/download/, under macOS tar.gz for the download.

The curl command would be something like curl --user USERNAME:PASSWORD --data-binary '{"jsonrpc": "1.0", "id": "curltest", "method": "getblockchaininfo", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:8332/'. The USERNAME and PASSWORD are your username and password that you entered in the rpcauth tool (https://github.com/bitcoin/bitcoin/blob/master/share/rpcauth/rpcauth.py) to generate rpcauth= config line.

Thank you so much for the help


text:
Calculating addresses from Bitcoin transaction data
What would be the easiest way to retrieve addresses that are used in specific transaction? For instance, if you look at following transaction which is retrieved by getblock of JSON RPC with argument of verbosity 2 (https://developer.bitcoin.org/reference/rpc/getblock.html),
....
"tx":[
      {
         .....
         "vin":[
            {
               "coinbase":....,
               "sequence": ...
            }
         ],
         "vout":[
            {
               "value":...,
               "n":...,
               "scriptPubKey":{
                  "asm":...,
                  "desc":...,
                  "hex":...,
                  "type":...
               }
            }
]

How can I know receiver's address by looking at "scriptPubKey"? One of the challenges was there were different types of scriptPubKey so that it was kind of hard to parse them. Does anyone know best way to retrieve public address from any kind of scriptPubKey, so that the result will be matched with the addresses that are used by block explorers?
P.S. I think some of the newer transactions actually contain "Address" field inside the scriptPubKey, but older transactions do not. I want to have a generalized method that can handle any kind of transactions.

Do you mean a way to decode scriptPubKeys yourself from hex data to addresses, or a way to get it from the Bitcoin Core RPC output? Because if an output has an address, the RPC output will list it. But addresses only exist for scriptPubKeys that match certain patterns; others just don't have one.

I was referring to a way to decode scriptPubKeys to addresses. For example, if you retrieve block height 13367 from RPC, it won't give you address explicitly even though the type is pubkey. However, if you use blockchain.info API, it contains address: https://blockchain.info/rawblock/000000008be5151fa84c4eeec998e61cbcfdaf8123d610381d7e3c8c25af57d2
I am wondering how I can do this without API's help.

Also, do you mind if you can explain what you meant by address does not exist for certain types of transactions?

That's because blockchain.info is using an outdated notion of "address". Pay-to-pubkey outputs don't have addresses corresponding to them, and never had any (it's not possible to pay to such outputs by paying to that address). However, historically, at some point they were lumped together with P2PKH (pay to pubkey hash) outputs for the same public key (which do have addresses). This is currently considered a historical mistake, as it was confusing addresses (a way to specify who to pay) with identities (which public key is involved). Since the introduction of multisig, these are distinct.

Addresses don't exist at the Bitcoin protocol level; only scripts exists. Some scripts are very common, so human-readable ways of communicating them were introduced, known as addresses. Nothing in the protocol however restricts people from creating payments to scripts which don't have a corresponding address. One prominent example is pay-to-pubkey (P2PK)... no address was ever defined for such outputs, but they were at some point common on chain, so explorers lumped them together with another type of script (which does have an address) for the same key.

That was a lot so I needed some time to digest. Thanks for the explanation. If my understanding is correct, "address" is just a concept we made to identify account in more efficient way. By looking at script data retrieved from RPC, is it possible to derive some sort of address, or any kind of unique identity of the sender and receiver? I am assuming there should be a way to derive public key from transaction data, but I would prefer to store them more efficiently since public key is long.

Also, how is blockchain.info is deriving "address" even for transactions to multisignature wallet or other special types of transactions?

LIke if I get public key hash from any type of transaction, and perform base58check encoding, would that represent "address"? To be clear, I want to come up with efficient way to store unique identity of each "account".

In short, the answer to your question depends on what you mean by "address". Historically, there are two different concepts with that name which I'll refer to as payment destinations and public key identifiers:

If you want payment destinations, these are already reported in the Bitcoin Core RPC output in the "address" field.
If you want the dated concept of public key identifiers as some block explorers still use, my hope is to convince you that this is terrible practice that you shouldn't try to imitate or persist at all.

These aren't well-established names, nor are they concepts I want to introduce a name for. They're just what I use in this answer to distinguish the two.

For background: Bitcoin transactions pay to scriptPubKeys, these are scripts that encode the conditions under which the sent coins can be spent. When a receiver R expects a sender S to pay them, they need some way of communicating to S ('s software) what scriptPubKey they expect S to construct so that S can access them.
Early in Bitcoin's history (before 2012) there were only two common types of scriptPubKeys, pay-to-pubkey (P2PK; where the scriptPubKey directly stores the public key of the recipient) and pay-to-pubkey-hash (P2PKH; where the scriptPubKey only stores a hash of the public key of the recipient). However, only a human-readable notation was introduced for P2PKH. P2PK was still in use, e.g. in mining software where no script needed to be communicated to any other party (it was just miners paying themselves effectively), but any "payment" use that involves one party asking to be paid by another party used P2PKH.
However, despite there not existing a way to instruct someone to pay using P2PK (=payment destination), people still needed a way to describe who/what a payment was for (=public key identifier). For listing who a P2PK output was for, the choice was obvious: just use the address for the corresponding P2PKH payment destination.
Clearly, a payment to a P2PK scriptPubKey with key P and a payment to a P2PKH scriptPubKey with key hash equal to the hash of P are spendable by the same person, so as a way to determine who a payment is for (=public key identification), this approach works. However, it does create a confusion of the form that someone might see a P2PK output, compute its corresponding P2PKH address, and then create another payment to that same address (=payment destination), and perhaps expect to see the same scriptPubKey appear.
Now, P2PK and P2PKH scripts are not the only types of scripts that Bitcoin supports (even from the very beginning). Another possible script is bare multisig: a scriptPubKey that contains multiple people's public keys directly, so everyone can see who it is for. This saw some limited use, and some infrastructure was introduced to make this work, including reporting these types of outputs as having multiple addresses in the public key identification meaning: one address for each of the public keys involved. Bitcoin Core's RPC output in fact worked this way for a while; there was no "address" field, but an "addresses" field, which could contain multiple elements. For bare multisig scriptPubKeys, it'd list one address for each public key. For P2PK and P2PKH scriptPubKeys, it'd just list one address (the one corresponding to the public key's P2PKH payment destination).
However, no address (in the payment destination sense) type was ever defined for bare multisig, so there was no way to instruct anyone to create such payments. Instead, in 2012, following the P2SH softfork, a very different payment destination address format was introduced: one where the receiver only reveals a hash of a script to the sender, and the individual public key(s) are not revealed to the network at sending time. The public key identification interpretation of addresses fails completely here. We have a new payment destination address type from which nothing about the participating public key(s) can be inferred at all.
And this concept caught on: P2SH (and its successors) actually ended up widely adopted, and bare multisig pretty much died. There were good reasons for this:

more private due to not needing to tell the sender the public keys about all the participants in the multisig scheme (which is none of the senders' business)
cheaper for the sender (as they don't need to encode all those public keys on chain in their transaction), moving the cost of using complex logic to the receiver, who is the party that decided to use the complexity
much more convenient (a single payment destination string, rather than some possible complex structure containing multiple public keys)

After P2SH, even newer script extensions were added, with corresponding address (=payment destination) encodings. The table below hopefully summarizes them:

scriptPubKey type
Introduced
Use as payment destination
Use as public key identifier

P2PKH
2009
Legacy base58check format
Yes, contains public key hash

Bare multisig
2009
-
Sort-of, with multiple "addresses" per output

P2PK
2009
-
Yes, public key hash can be computed from public key.

P2SH
2012
BIP13 base58check format
-

P2WPKH
2017
BIP173 bech32 format
Yes in theory, contains public key hash, but not commonly done.

P2WSH
2017
BIP173 bech32 format
-

P2TR
2020
BIP350 bech32m format
-

So overall:

Bitcoin transaction outputs contain scriptPubKeys, which can be anything, and are not restricted to any particular set of standard scripts.
Some types of scriptPubKeys have an associated address format that can be used to instruct parties to pay to that scriptPubKey.
Some types of scriptPubKey allow determining which public key (hash) the payment is for, but that's increasingly not the case for modern scripts.
Historically, the P2PKH address format was used as a way to represent "who" a scriptPubKey is for, as opposed to encoding a specific scriptPubKey (which involved representing P2PK using the P2PKH addresss format, and bare multisig even as multiple P2PKH addresses). This doesn't make sense anymore: the introduction of P2SH and its successors show that in general, one cannot identify who from a scriptPubKeys, and doing this just reinforces a misunderstanding.

This is a thing of beauty. Thank you so much. Most of them make sense now, but I am still not sure how block explorers generate addresses for all kind of transactions then. Even though they are using an old notion of "address" (payment destination), they are able to accumulate balance with unique addresses for all kinds of transaction, right? I guess I want to understand how can people assign unique address to account by just looking at the chain data, like block explorers are doing.

I apologize if this is a stupid question or if you already explained it in the above post, but I need to clarify my understanding.
Now I think I understand what you meant by "payment destination" and "public key identifier" since the transaction is paying to "script", and sometimes the requirement to unlock that script is more complexed than represented with one public key. However, I need to find a way to represent a unique representation for payment destination.

@xor If there is no well-defined address for a scriptPubKey (in the payment destination sense), you're out of luck. If you want to be unambiguous, you could just hex encode the scriptPubKey or so, but there is no way that's both unambiguous and well-established. Certain block explorers just don't show anything at all if there is no address (including for P2PK outputs, e.g. https://blockstream.info/nojs/tx/0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098?expand).

If my understanding is correct, all of the scriptPubKey types that can be used as "payment destination" has an "address" field when retrieved with JSON-RPC. Is my understanding correct? Secondly, if I want to derive a payment destination for the "P2PK" type, if I just hash the public key and use legacy base58check format, will I get the same "address" as the "P2PKH" when the public key is the same? Finally, even though the transaction is paid to the same person in "P2PKH" and "P2WPKH" (meaning that a private key owner can spend both outputs), address fields will have different values, right?

If an an address exists for a scriptPubKey in the payment destination sense (as in: sending to that address will result in the same scriptPubKey), then the JSON-RPC interface will report that address. P2PK scriptPubKeys don't have such an address. You can in theory take the public key in a P2PK output and hash it, and encode that as a P2PKH address, but that's only meaningful if your goal is identifying public keys, because sending to that P2PKH address will not send to the same scriptPubKey, and possibly even burn funds. Further, for e.g. P2SH/P2WSH/P2TR no such hack is possible.

And I very seriously hope you're not going to turn P2PK outputs into P2PKH addresses. It's a nonsensical historical practice that confuses what address mean.

My usage of the term "payment destination" and "public key identifier" are about what you want the address to mean. If the point is to show something that can be sent to, simply nothing exists for P2PK or bare multisig. If the point is identify which public keys are involved, simply nothing exist for P2SH, P2WSH, or P2TR. You should ask yourself what you're trying to achieve with your "unique representation".

I see. I will think more about what is more meaningful representation. Maybe using hexcode of scriptPubKey is most unamgiuous option as you mentioned.


text:
Does each bitcoin node do Continuous Integration? Is this merely the process of the node syncing with the network? What is involved with it?
I notice the folder in the bitcoin core github repo for CI, and would like to know more about it.
I have read the entry in Wikipedia for Continuous Integration.

Continuous integration is a paradigm used in software development. In the context of the Bitcoin Core software repository, it specifically refers to the automated testing setup: all changes and proposed changes to the source code are subjected to a set of automated tests, to make sure nothing breaks.
It has nothing to do with the operation of Bitcoin nodes, and users of the software generally do not interact with it at all. It's just part of the software repository so that changes to the tests can be made simultaneously with changes to the source code.

Who carries out these automated tests?

Bitcoin Core currently uses Cirrus CI as CI provider, see https://cirrus-ci.com/github/bitcoin/bitcoin.

Continuous Integration (CI) is an automated development practice used by the Bitcoin Core team. It involves automatically integrating code changes from multiple developers into a shared repository, followed by automated builds and tests to check for bugs and issues. This ensures the stability and reliability of the Bitcoin software. The "CI" folder in the Bitcoin Core GitHub repository contains configuration files and scripts used for CI automation.

Who are these 'multiple developers'? How do you become one of these developers?

@EoinMcQuinn Literally anyone who chooses to contribute to the project.

I have noticed one or two pull requests on github carrying the 'CI failed' label. What does this mean and who would have been able to add this label to the pull request?

When a pull request on GitHub is labeled as "CI failed," it means that automated tests and checks for that code update didn't pass. This label helps highlight that there are issues with the code that need to be fixed before the changes can be merged. Contributors, maintainers, or automated systems can add this label. It's a signal that the code needs adjustments before it can be accepted.


text:
Will my node relay a transaction if it is below minimum tx relay fee?
If I create a transaction which pays less fees than the minimum transaction relay fee of my full node, will my node relay the tx?

No, your node will only broadcast transactions that it accepts into its own mempool. If your transactions pays a lower feerate than your own node’s minimum transaction relay feerate ("minTxRelayFee"), your node will not accept your transaction and therefore not broadcast it either. If you configure your node to use a lower minTxRelayFee, your node will accept it. If any of your peer nodes have also configured a lower minTxRelayFee, it will forward it, but it will still likely not get mined unless it finds its way to a miner that also has set a lower minTxRelayFee, and that miner includes it in a block.
Given that there have been over 70 blocks worth of transactions bidding at least 1 sat/vB for the past year, it is unlikely for your transaction to get picked into a block any time soon, even if there were some miner that has their node configured with a lower minTxRelayFee.


text:
What are the advantages/disadvantages to using the same Wallet Address?
I'm very curious about the reasons people keep their wallet addresses private. I don't understand, some people just post them publicly and nothing bad ever seems to happen.
I am involved with a project that uses randomly generated new wallet addresses every time you click the button. I am not sure if this is native to the Blockchain or just a feature of both wallets I use, but everyone seems to like it and idk why.
Personally, I want to just smash the button until I find an address with some aesthetic/memorable strings. But I have no idea if this is a bad idea, wastes resources, etc.
I'm curious about the longevity of these addresses. Is each address permanently assigned to my wallet? How does that work? I thought on Bitcoin, your whole "identity" was your wallet address, so who keeps track if that changes every time you click? I know there is some insane number of possible hashes, but what happens if it runs out? And can't someone still try to "hack" you or whatever if they have your old address? So why make a new one?
I want to make sure I keep this relevant to Bitcoin, so I'm also interested in learning more about if people commonly use different BTC addresses for diff things, or like shell layers where you have a few wallets you share with people and then forward all your coins to a secret wallet to hodl, and then send them to yet a third set of wallets before spending them on anything.
Also, people talk about privacy a lot, but if I know your wallet address, can't I see your total BTC holding (at that wallet) as well as any & all transactions you've made and what wallets you transacted with? In theory, couldn't someone track down and identify identities of core wallet users (say a porn company that accepts crypto) and then make a list of every wallet that has paid them, and then try to make further deductions and dox the customer base? Maybe this is impossible but it is a thought that keeps me up at night and I would like some clarity.
tl;dr: why no same hash?

A Wallet does not have one address these days, the HD type you are talking about can have many thousands.

Questions about altcoins are off-topic here, but the answer is essentially universal: if you re-use addresses, especially because of a desire to think of them as a long-term identity, you're effectively giving up any hope of even rudimentary on-chain privacy. Privacy is hard, and certainly not a given even without address reuse, but with it, there is little hope to be begin with.
Another frequently brought up argument, though cargo-cult in my opinion, is that address reuse also has meaningful security implications. The reasoning is that if public keys aren't exposed, they aren't subject to attacks to compute the corresponding private key (the hash would need to be reversed first, which is presumed to be (even) hard(er)). My counter is that most interesting use cases of Bitcoin already involve sharing public keys with non-trusted parties, so we need to make sure that things remain secure even with revealed public keys anyway - and there is no reason to assume that isn't the case.
There have been some efforts to move away from the term "address" for this very reason - addresses do have a notion of permanence / identity associated with them, and some believe that thinking about them differently (e.g. by calling them "invoice identifiers") may improve the situation. I believe efforts like this are well-intended, but the time to change the terminology was about 10 years ago. Now it's so ingrained it's unlikely to swap it out.
That said, most wallet software these days does a pretty good job at not reusing addresses, e.g. by giving out a new address every time one is requested, and making it hard to list historical ones. Most reuse I believe is due to services/"hosted" wallets where users have accounts, where changing addresses is less common than in wallet software proper.
There is no financial impact on the user from address cycling: using a new address for every incoming payment has no higher (or lower, for that matter) fee associated with it than always using the same one, due to the fact that Bitcoin uses a UTXO model. The downsides of address cycling are purely in convenience, user experience, and implementation complexity:

Wallet software needs to be more complicated to maintain possibly large sets of addresses. While the UX likely makes it seem that the address changes every time, you do want it to actually keep around the ability to receive coins to all old addresses and spend those, because who knows how long the sender keeps it around before using it (or worse, reuses it).
Backups are more complicated by the fact that newly generated addresses need to be recoverable. This is what led to the development of deterministic key wallets, which generate all keys/addresses from a single seed or master key. That means that one backup suffices for all future addresses too. A downside, as mentioned in RedGrittyBrick's answer, is what is called the gap limit: wallets will only look ahead a limited number of "future" addresses, which may result in missed recovery if a backup is restored with 100s or 1000s of previously given-out-but-never-received-to addresses.
User understanding is complicated, especially when presented with block explorer websites, which use an "address balance" model. The more private wallet-is-a-collection-of-otherwise-unlinkable-addresses model effectively used by constantly-refreshing-address-wallets doesn't match neatly to this address balance model, leading to complications in understanding, and sometimes lost funds (because people e.g. backup only the private key to "their" single address, not knowing they also need the private keys for addresses that the wallet automatically sends change to.

Gap Limits

I want to just smash the button until I find an address with some aesthetic/memorable strings. But I have no idea if this is a bad idea, wastes resources,

That could create problems when recovering the wallet from a seed-phrase. Newly restored wallets will generate a fixed number of addresses and look for transactions at those addresses. If there is a gap of many addresses with no transactions the wallet will stop looking. This is called the gap limit. If you mash that button too many times you will create a gap bigger than the gap limit and your restored wallet may not see all  the transactions it needs to know of in order to calculate the amount of money under your control (your "balance")
Address Lifetime

I'm curious about the longevity of these addresses.

Addresses never expire. They last forever.
However, occasionally Bitcoin introduces a new type of transaction such as Segwit/Bech32. These replace older types. It is conceivable that eventually older types will not be popular and fade into disuse. Wallets might then stop supporting older types. this seems to me to be unlikely in the lifetime of anyone reading this.
As Pieter pointed out in his comment. People sometimes do thing such as transfer money to a new wallet and throw the old one away along with its private keys for the old addresses. This would mean that from then on, money sent to the old address would not be spendable by the intended recipient. People in general would have no way of telling that an address is one for which nobody has the private key. The Bitcoin network would still accept transactions that paid to the old address.
Address ownership

Is each address permanently assigned to my wallet?

No. They are just random looking numbers that can be produced from a pair of numbers called a private key and public key. No one other than the address "owner" needs to know anything about who generated an address. To the rest of they world an address is just a random number. One that has a mathematical relationship to another secret number which allows the existence of that relationship to be proven without divulging the secret.
No one keeps track of who owns every address used in Bitcoin.
Large numbers

what happens if it runs out?

There are too many to ever run out. Even if ten billion people generated a billion a second for a billion years.
Hacking

can't someone still try to "hack" you or whatever if they have your old address?

Addresses are of no significant use to hackers.
Address re-use is a privacy issue.
Privacy

if I know your wallet address, can't I see your total BTC holding (at that wallet) as well as any & all transactions you've made and what wallets you transacted with?

That's why you generate a new address for each transaction. Other people can't easily tell from the transaction journal (blockchain) which addresses are in the same wallet.
Insomnia

it is a thought that keeps me up at night

Maybe try not to buy goods and services from unscrupulous or dishonest people and businesses?
Maybe look into Bitcoin-mixers / Bitcoin-tumblers and wallets that support them?

Addresses within a wallet certainly last forever, but the wallet might not. E.g. over long periods of time, a receiver may migrate to a new wallet software/provider. So as a sender you shouldn't assume a once-given address will last forever, even though every wallet software essentially should try its best to make the receive address work as long as possible.

@Pieter, thanks, good point, answer updated.

I just want to clarify that I am not worried about my own purchases or anything. I have no malicious intent but the insomnia is more related to just the theoretical possibilities.
I have heard of tumblers and will look into this more, I did not know they even involved wallet support, thank you.


text:
Why does it cost so much to forward all my received payments to my other wallet?
I am using bitcoin-rpc and every time I create an invoice for the client to pay, I create a new address. The payment is okay, I get the money after a cron.php sees that the transaction has three or more confirmations, but then when I try to send my money back to my big wallet, I see 200—300 usd in fees.
bitcoin-cli sendtoaddress "MY_WALLET" 0.1 "" "" true
I see on blockchain that everything was sent from each and every address that I have created
ADDRESS1 ... 0.05
ADDRESS2 ... 0.08
ADDRESS2 ... 0.028

... and so on
How can I send everything from one big address and pay only 20$ usd fee?
I think I pay per vbyte for what I have read ... but I want to send internally with low or no fees or something. Can I send coins internally to one big wallet?
I mean don't think that big payment gateways that do the same thing and create an address for every invoice but pay such amounts of fees.
Thank you.

i think it is estimate_mode and as imputs i have 30 40 50 100 ... a lot , explain more about UTXO or a link ...

Can i send coins internally to one big wallet?

No.
By internal you mean without communicating with the Bitcoin network and without involving external miners.
All Bitcoin transactions which move money from one set of Bitcoin addresses to another Bitcoin address are external in the sense that they must be recorded on the blockchain and therefore must incur transaction fees.

ok, i understand, but how to implement a payment systems when i transfer all i got into my big wallet and after that i lose a lot of money? if i use a big payment gateway service, they send with 10 ... 20 usd fee ... how they do it ?

Each on-chain payment you receive creates a new Unspent Transaction Output (UTXO) for your wallet. When you forward the received payments to your other wallet, each of those UTXOs is consumed in an input of your consolidation transaction. This will happen whether you receive payments to different addresses or reuse the same one.
Given that you will need to spend a number of UTXOs corresponding to the count of payments you receive and Bitcoin transactions bid on the amount of blockspace they occupy, you have several options to reduce the total cost:

Blockspace-efficient output types
If you have been using P2PKH addresses (i.e. "legacy addresses" that start with 1…), you could significantly reduce weight of your inputs by switching to a more modern output type. P2PKH inputs weigh 148 vbytes each, while P2WPKH (native segwit) inputs weigh 68 vbytes, and P2TR inputs even weigh only 57.5 vB.
Low feerate
Consolidation transactions tend to require a larger amount of blockspace than most transactions. The command that you cite
bitcoin-cli sendtoaddress "MY_WALLET" 0.1 "" "" true
does not set a feerate. This means that your transaction will use the default feerate estimate which aims to get a transaction confirmed very quickly. In this case, you are just moving money from your left pocket to your right pocket, so you actually probably do not need the transaction to be confirmed urgently. Instead of using a feerate high enough to get a confirmation in the next half hour, you could use a feerate that is sufficient for the transaction to confirm within the next day or week. This could easily reduce the overall transaction fee by 50–90% depending on, or di the prevalent feerates. You can find a view of the mempool as well as a feerate estimate for example on mempool.space.
Off-chain payments
In the time since you wrote this question, it has become a lot more viable to receive payments off-chain, e.g. via the Lightning Network. Especially if you are receiving many small payments, you may consider running a Lightning node to receive your payments which would allow you to receive payments without generating a new UTXO with every single payment.

I mean don't think that big payment gateways that do the same thing and create an address for every invoice but pay such amounts of fees.

As far as I know typical approach is to have root P2WPKH bitcoin wallet and to generate child wallets for each payments. Benefits of P2WPKH is lower transaction fees. Because the witness data is stored outside the main block, less block space is used, resulting in lower fees.
Then you need to create many-to-one transaction to withdraw, where inputs are the unspent UTXOs related to your child wallets and one output is the destination address. Required fee for your withdraw transaction = raw transaction size * fee rate in sat/vB. You can use public API (for example, https://blockstream.info/api/fee-estimates) to get current fee rate.

Hi dmitrdv, welcome to Bitcoin Stack Exchange. Thanks for chiming in on this topic. I am writing to point out that it’s a common myth that witness data is not part of the block. While non-segwit output types used the input script to authorize payments, segwit transactions have a new section called "Witness" that holds a witness stack for each input. The witness is part of the transaction, and each transaction is recorded in full in the block that confirms it.


text:
Is it possible to prompt another wallet to add to a PSBT and finalize it?
Is there anyway to request a wallet to complete a PSBT?
Let's assume I want my client to send a transaction with follow structure:

output[0]: OP_RETURN with bytes 636363636363
output[1]: SEGWIT output, with 1000000 satoshi to my address

The scenario I'm thinking is that I generate a PSBT, generate these two outputs, and send the PSBT to my client. He import it in his wallet, wallet add inputs (and output to his address if necessary, as change box) then sign and send the transaction.
Is this scenario possible? if not, any alternatives?

Hi, I attempted to better summarize your question in the title. Please feel free to roll back or improve the title if it doesn’t match your intended question.


text:
How can I doublecheck the value stored in a paper wallet?
I have a paper wallet with some bitcoins that I bought in 2017. It says Balance 0.03614145 BTC but when I scan the QR-code with the private key, it says Balance €163.18; shouldn't the balance be more than that?

Paper wallets are useful means to store Bitcoin in a secure fashion that allows the value of a wallet to be transferred if the owner chooses to transfer ownership (holding) of the paper wallet and is where the idea that the owner of the private keys, that is, a person who can spend the Bitcoin is the owner.

Hi michelle, could you please clarify what you mean with "it says"? Is this in both cases the wallet that you imported the key from the paper wallet into, or is the bitcoin balance stated on the paper wallet printout?—Rather than just tagging the topic, we prefer when the titles of topics capture the gist of the question. I’ve edited your question’s title to summarize what you are asking about. Please feel free to edit it further or roll that back, if I misunderstood the intent of your question.

shouldnt the balance be more than that?

At today's exchange rates (₿1:€47117), your ₿0.03614145 is worth €1702 not €163

when i scan the QR (private key)

The result of this depends what exactly you mean.
Importing an address (not private key) into a non-custodial wallet and rescanning the blockchain is a way of checking the balance. You only need import the private key if you want to spend money or if the wallet is intended to be a cold-wallet.
Putting a private key into a randomly selected untrusted website invites theft.

What kinds of wallets are there?

Paper wallets are not useless if they are left on the bench unopened with the Private Key secure, but otherwise after they are spent the wallet can still be refilled.


text:
Monitoring a sequence of bitcoin transactions
Me and a person I met on the internet have agreed to a sequence of two btc transactions, but neither of us trusts the other party. We would like to hire a person to monitor the sequence on the blockchain for a fee. The idea is we both deposit an amount in this person's wallet. After the transactions have completed, this deposit is paid back to both parties. Is this a good idea to resolve the "trust stalemate". Could it be resolved automatically using a smart contract? How do we find a trusted third party that would do this for us for about 50$ ?

The standard way to set up an escrow relationship like this is by creating a 2-of-3 multisig address with keys of the two parties and the escrow. As long as the two parties agree, they can pay out the money any way. If there is any dispute, the escrow can move the funds together with the justified party. Often the output script for the 2-of-3 multisig address contains a timeout condition that returns the funds appropriately in absence of other transactions. I know that this pattern is used by marketplaces that sell bitcoins for other currencies to coordinate the release of funds from seller to buyer, IIRC there is also a decentralized marketplace that uses a similar pattern for arbitrary transactions. It might have been BISQ or OpenBazaar.

Ok thanks who offers such a escrow service (standard setup you describe)?

I don’t have a recommendation for you from the top of my head.


text:
UniSat's `getPublicKey` returns internal key or output key when using Taproot?
According to UniSat documentation here, I can call window.unisat.getPublicKey to get the public key of the current account.
My address type is Taproot. After calling this function, I get a compressed public key in hex encoding (66 hex digits). I'm wondering if this is the internal key or the output key?
In addition, how to derive the Taproot address? I know a SegWit address is derived by concatenating the following:

Human readable part. bc in mainnet.
Separator 1.
Some data. Always a single bech32 digit (representing SegWit version) + some extra data.
Checksum or correction code.

But I don't know what the 3rd part should be in the case of Taproot. I've heard it's a special form of public key, but how can I derive this from the return value of UniSat's window.unisat.getPublicKey?

internal key or output key
I think this post https://bitcoin.stackexchange.com/questions/117580/how-to-extract-the-internal-key-of-a-simple-p2tr-address, is a better answer for you.

you get back the internal key, which can be used to generate a raw taproot address with bec32m, then you can use the default bip341 tweak of the internal key to get the tweakpubkey and bech32m it, to get the taproot addressed used by the wallets.


text:
Is there really a thing called "Bitcoin address"? Since all I need to spend a UTXO is to provide a scriptSig
When I want to spend a UTXO, I need to and only need to provide the scriptSig (or input script, or unlock script). In another word, the scriptPubKey (or output script, or lock script) of a UTXO simply defines the way to unlock this UTXO. And it doesn't have to consist of someone's pubkey hash (the RIPEMD160 hash). I can even define the scriptPubkey as follows: anyone that finds the original content that can be hashed to 0123456789abcdef can spend this UTXO. In this case, there's actually no receiver address.
So is there really such a thing called "Bitcoin address"? Or is the "Bitcoin address" only inferred based on the content of scriptPubKey? Since most P2PKH, P2SH, P2WPKH, P2WSH, P2TR transactions use a fairly standard form of scriptPubKey.

Addresses do not exist on chain, and at the protocol level even nothing equivalent to an address balance exists. All that exists are UTXOs created by transactions, which carry a scriptPubKey.
Addresses are merely instructions for sender wallets to construct the scriptPubKey of outputs in order to pay the receiver. While it's currently generally possible to infer the used address back from the scriptPubKey, this is really backwards, and not something that will work with e.g. silent payment addresses.

Thanks for asnwering! So the addresses shown in the Bitcoin explorer are inferred from the scriptPubKey, right?

Indeed, and that inference process only works for the current typical address formats. In fact, many (older) explorers show a P2PKH address even when the scriptPubKey is actually P2PK, for historical reasons.


text:
Bitcoin Mining Pools Centralization
I read a couple of articles and watched some YouTube videos, but couldn't find a super convincing argument for why centralized mining pools are not a huge issue. I get that from a Game Theory perspective, a rational agent would be more incentivized to use computational power in an honest way rather than attacking the network (risking losing all of the wealth in the network). However, isn't it still possible that some economically irrational agent takes over the top 5 mining pools and control which blocks are being added to the chain (possibly double spend without getting caught)?

First you need to consider what a mining pool is doing, if a pool has enough hash-rate it is likely for them to open multiple stratum servers (and full nodes) for the large amount of miners to connect to. So already they are in some sense by nature of bandwidth limitations encouraged to decentralize their own block submission. That means that every single stratum server and full node would need to become compromised for an attack to occur.
The other thing to keep in mind is that these pools are constantly dealing with actors going on and off the pool, this implies that if there was any corruption with the operation, every single miner who has logged on to the pool can easily switch and submit shares to a different pool. This could be at a moments notice, so any attack would be easily negated by people realizing there was an attack.
The other thing to note is that these pools are so large BECAUSE they are decentralized and not the other way around. Over time lots of different miners set up their own infrastructure and simply point it to the pools for which they consider are doing a diligent job hosting. It is presumable that if any were to become compromised they would lose hashrate QUICKLY and this loss of hashrate means loss of profitability for the pool maintainer, if we suppose that maintainer has a lot of ASICs they will be ultimately forced to capitulate into putting their hashpower on an alternate pool (and be forced to earn pooled rewards as opposed to solo blocks from maintaining the pool) or eventually selling their own hardware due to profitability decline from massive hashrate leaving their pool.
The game theoretics driving proof of work are very resilient and are unlikely to be cheated by any individuals in a truly decentralized organic network like Bitcoin.
Synopsis of my thoughts: It would be better if people decentralized /distributed their hash-rate amongst many more mining pools but as it currently stands the top pools are being picked for being good performers and good actors and are inherently decentralized on the inside otherwise they would not have so much hash-rate.

Thanks for the response! I still have some follow up questions though.

How the miners who logged in to the pool would notice if there was a corruption during the operation? Doesn't mining pool control which transactions are being added to the new block?
Can we still say it is possible to double spend on certain transaction if top 5 miners make secret plan? If so, is it possible to detect this?

I will answer in reverse order lol. Double spending is only possible on exchanges that are not auditing properly, there has never been a recorded double spend of a single UTXO on bitcoin that I know of, it is against the consensus rules. The mining pool controls which kinds of transactions to accept but does not (generally) actively monitor which specific transactions are flowing in and out, doing so without automation would be impossible and doing so with automation would look like this node is censoring transactions (so most would just stop using the node leading it to find less solutions).

Miners may notice that they are not being properly paid by the pool in which case they will point to a new pool, this is actually quite common as pool maintenance is difficult and prone to greedy operations that arent audited by miners. I am sympathetic to this as I GPU mine and deal with pool related profitability issues directly, but that being said the only combat to it is for more people (like me) to spin up solo operations or alternative pools (as I am working on) but unfortunately I am not even a proponent of Bitcoin hashrate as I have not got 240v power yet.

Once I do aquire a BTC ASIC I intend to point it at a pool with lower hashrate than the top 5, I have found that generally this leads to increased rewards when you have low hashrate compared to the network as the top pools need to feed the top hashrate miners. Low pools need to compete for lower hashrate miners and therefore "feed" them more equitably compared to large pools. This is all from personal experience and much of it is open to debate however it represents my feelings on the matter.


text:
Why does the mined block differ so much from the block template?
I am looking at the specific block at height 680175.
At the time when the block was not mined yet, I received the block template via bitcoin-cli getblocktemplate '{"rules": ["segwit"]}' with 1451 transactions:
{
    "capabilities": [
        "proposal"
    ],
    "version": 536870912,
    "rules": [
        "csv",
        "!segwit"
    ],
    "vbavailable": {},
    "vbrequired": 0,
    "previousblockhash": "00000000000000000008465bf3145011298dddb04f026e077d8deb5b9ac5cf7f",
    "transactions": [
        {
            "data": "..",
            "txid": "f0fcac28b1dec7b8e67f73229052639c58f7f175bdb87ea068f499a4728bb35d",
            "hash": "2a4aa6c2b5a6bee592e3690a843ee63b2598315da01954c05c4c0aa77d57af34",
            "depends": [],
            "fee": 160000,
            "sigops": 5,
            "weight": 953
        },
        ..,
        {
            "data": "..",
            "txid": "e0ae05fa82edb0dee7e9d11ca9471bf3ee1a90e19ac8abf1013511338613b814",
            "hash": "e0ae05fa82edb0dee7e9d11ca9471bf3ee1a90e19ac8abf1013511338613b814",
            "depends": [],
            "fee": 103950,
            "sigops": 4,
            "weight": 1388
        },
        ..,
        {
            "data": "..",
            "txid": "8d55bd874acea1fbc03b98e82d585c39021be083645dde83e8306171e32258d5",
            "hash": "8d55bd874acea1fbc03b98e82d585c39021be083645dde83e8306171e32258d5",
            "depends": [],
            "fee": 16425,
            "sigops": 8,
            "weight": 904
        }
    ],
    "coinbaseaux": {},
    "coinbasevalue": 745954254,
    "longpollid": "00000000000000000008465bf3145011298dddb04f026e077d8deb5b9ac5cf7f3670",
    "target": "0000000000000000000bef930000000000000000000000000000000000000000",
    "mintime": 1619115663,
    "mutable": [
        "time",
        "transactions",
        "prevblock"
    ],
    "noncerange": "00000000ffffffff",
    "sigoplimit": 80000,
    "sizelimit": 4000000,
    "weightlimit": 4000000,
    "curtime": 1619117253,
    "bits": "170bef93",
    "height": 680175,
    "default_witness_commitment": "6a24aa21a9ed82637c9b63e2165105a672c4f6edcfa0187d961843a8d2b9c7a6bceeda109a65"
}

When comparing the block template to the mined block, there are some differences:

The version of the mined block is 0x2fffe000 instead of 536870912.
Only 194 transactions from the block template are in the mined block, e.g. f0fcac28b1dec7b8e67f73229052639c58f7f175bdb87ea068f499a4728bb35d.  The other 1257 transactions are missing.

The transaction 8d55bd874acea1fbc03b98e82d585c39021be083645dde83e8306171e32258d5 was mined in a later block.
The transaction e0ae05fa82edb0dee7e9d11ca9471bf3ee1a90e19ac8abf1013511338613b814 was not mined at all.

Does anyone know how these differences can be explained?
What would have happened if a miner found a block hash for the received block template and submitted it? Would the mined block then have a different version and the missing transactions (including the transaction that was not even mined yet)?

The version of the mined block is 0x2fffe000 instead of 536870912.

Many miners employ a strategy known as overt Asicboost. This strategy requires modifying the version number like the nonce, so we often see blocks that have strange version numbers.

Only 194 transactions from the block template are in the mined block, e.g. f0fcac28b1dec7b8e67f73229052639c58f7f175bdb87ea068f499a4728bb35d. The other 1257 transactions are missing.

Each node has a different view of the set of unconfirmed transactions that depends on how long they have been running, where they are located, and any custom rules that they are using. It may be that your node has not been running for as long as the miner's node and so it has not seen all of the same transactions that the miner has seen. It may be that your node is using different rules than the miner is, possibly due to different versions of the same software, or possibly because some miners may be using custom software. Additionally, some miners offer a service where they will prioritize transactions after being paid out of band, so such transactions can end up in a block when they do not appear in a block template produced by your node.

What would have happened if a miner found a block hash for the received block template and submitted it? Would the mined block then have a different version and the missing transactions (including the transaction that was not even mined yet)?

There is no "current template" for the entire network. If there were, then there wouldn't need to be a blockchain as that would mean every node is actually in sync with every other node. As such, when a miner finds a block, they broadcast the block in full. When nodes receive a block, they verify it without creating their own block template. There are no requirements that the block contain the transactions a node expects it to contain. The only requirements are that the block contains valid transactions and that the block header is valid. Whether your node has seen those transactions before and whether it would include them itself are irrelevant.

Thank you for your answer! So, is it allowed to change the version, even if it is not in the mutable array? .. A transaction that I received in the block template was never mined. What happened to this transaction?

Ah, good point, I didn't address the funny version business. Have a +1.

@andy Miners have complete authority over the contents of their blocks. You include a fee in your transactions to incentivize them to include it. If a transaction is not included, it is left in mempools, and thus can be picked up again in later blocks.

@Andy Yes, the version can be changed, within the consensus rule restrictions. As stated previously, there is no requirement for blocks to conform to the block template.

Going out on a limb here, but I would guess that your node is not running 24/7.
Full nodes only rebroadcast transactions once when they first hear about them. If your node is only online intermittently, it would only hear about new transactions getting submitted to the network. However, each node builds its block templates from the mempool that it locally maintains.

via jochen-hoenicke.de
Especially in the past day, where blocks were mostly clearing transactions broadcast much earlier, your node would not be aware of the majority of the transactions mining nodes were tracking in their mempools and including in their blocks. Your node would only learn about most of the transactions as blocks get found and announced.
This is substantiated by your block template only collecting ₿7.45954254 in mining reward, whereas the block you link to collected an additional ₿1.68 in fees for a total reward of ₿9.14729350, i.e. the miner was simply aware of more valuable transactions to include in their block than you.

The version of the mined block is 0x2fffe000 instead of 536870912.

Only a few of the version field bits are constrained. Some miners have been using the remaining field to generate additional entropy, and to employ a mining optimization called Overt ASICBoost. Those uses result in various different values for the version field, which do not impact validity of blocks, though.

Thank you for the answer! Yes, my node was not running 24/7. I did the initial sync and then wanted to look at a block template. So, if I would have started mining with the block template that I received, are the chances the same compared to nodes that run 24/7?

Your chance to succeed per hash that you try would be the same, but your block reward would be lower as described.


text:
What does KeyError: 'zmq-pub-channel-opened' mean?
Can anybody tell me if this is an issue when loading the zeromq plugin for Core Lightning?
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py: Traceback (most recent call last):
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:   File \"/usr/local/lib/python3.11/dist-packages/pyln/client/plugin.py\">
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:     result = self._exec_func(method.func, request)
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:   File \"/usr/local/lib/python3.11/dist-packages/pyln/client/plugin.py\">
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:     ret = func(*ba.args, **ba.kwargs)
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:   File \"/usr/local/lib/python3.11/dist-packages/pyln/client/plugin.py\">
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:     return self._exec_func(self.child_init, request)
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:   File \"/usr/local/lib/python3.11/dist-packages/pyln/client/plugin.py\">
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:     ret = func(*ba.args, **ba.kwargs)
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:   File \"/home/main/.lightning/reckless/plugins/plugins/zmq/cl-zmq.py\",>
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:     Setup.check_option_warnings(options, plugin)
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:   File \"/home/main/.lightning/reckless/plugins/plugins/zmq/cl-zmq.py\",>
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:     for endpoint in Setup._iter_endpoints_not_ok(options):
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:   File \"/home/main/.lightning/reckless/plugins/plugins/zmq/cl-zmq.py\",>
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:     endpoint = options[endpoint_opt]
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:                ~~~~~~~^^^^^^^^^^^^^^
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py: KeyError: 'zmq-pub-channel-opened'
2024-06-09T14:54:53.169Z INFO    plugin-cl-zmq.py:

I'm running the example-subscriber.py file and I never see a shutdown event logged. I did add shutdown to the notification list.
python3 example-subscriber.py --zmq-sub-shutdown=tcp://127.0.0.1:5555

Here's how I'm starting the plugin:
lightningd --regtest --daemon --plugin=/home/main/.lightning/reckless/plugins/plugins/zmq/cl-zmq.py --log-file=/home/main/.lightning/logs.txt --zmq-pub-shutdown=tcp://127.0.0.1:5555

Hi, I updated the formatting in your post, and edited the title to give a sense of what the question is about. Please feel free to further edit the title, if you feel that I didn’t capture the intent of your post.

KeyError is a generic python error indicating that a key does not exist in a dict. In this case, there is a dict named options, and the code was trying to find a key named zmq-pub-channel-opened in options, but it does not exist, so Python threw a KeyError. This seems like a bug in the ZMQ plugin and I would suggest opening an issue in its github repo.

Will do. Thank you for the response.


text:
Bitcoin Energy Consumption and Carbon Footprint a good long term investment?
How can Bitcoin a good long term investment, given one transaction is equivalent to carbon footprint of 986,125 VISA transactions?
If they use the Lightning network which takes transactions Off the Blockchain, wouldn't that defeat whole purpose of having ledger transaction?
https://digiconomist.net/bitcoin-energy-consumption/

why is this voted down? I am trying to learn

I have read somewhere that every Bitcoin transaction is "equivalent to the power consumption of an average U.S. household over 22.86 days.". Is that true? Isn't that horrible?
tl;dr: Most people don't value Bitcoin as a payment mechanism, and even if they
were, most payments are not recorded on-chain. So calculating the energy cost of
a transaction is as impossible and nonsensical as it would be for gold.
It's true in a similar way that the lack of pirates is causing global
warming.
Yes, this is a valid estimation of Bitcoin mining's energy use divided by a
valid estimation of on-chain transactions, compared to a probably valid
estimation of household electricity use.
However, connecting these three figures in this way doesn't yield any insight
beyond shock and awe. That's because Bitcoin's primary value is not as a payment
network, so the number of on-chain transactions is not an interesting metric for
it. This can be seen easily: On one hand the number of onchain
transactions
has always been limited — and that limit has been pretty much hit constantly
since at least 2019. On the other hand, the value of Bitcoin, as well as the
estimated number of its users, has been rising over many orders of magnitude
since 2009. How can this be possible?

In recent years, long term store of value has become Bitcoin's main use case. That is, people use it more for storing their savings than for paying for goods and services.

Its current usage is thus more comparable to gold, which is also rarely used in transactions. I have never seen anybody calculate the energy cost of gold mining plus handling divided by the number of physical gold shipments. Probably because that's as misleading a number as the equivalent in Bitcoin.

Just as with gold, there are ways to make even small transactions quickly and cheaply with Bitcoin. Most real-world transactions use one of these off-chain methods. Probably the most popular today are custodial services like exchanges. Sending bitcoin from one customer to another on a service like this just requires changing two numbers in their database and is the equivalent of gold in a bank changing owners. It's hard to say how many such transactions there are, since there are many "bitcoin banks" and their numbers are proprietary, but it's safe to say they dwarf on-chain transactions by orders of magnitude.

Unlike gold, there are methods of transacting Bitcoin off-chain that require less trust in intermediaries, such as the Lightning Network or Blockstream's Liquid federation. With a growing demand in transactions, such alternatives are refined and gain popularity, without this being reflected in the above numbers. With the Lightning Network, it's even difficult to estimate how many transactions are being performed, since there is no central ledger recording them all.

But I have read that Visa transactions are much more economical!
That's because these are very different things. Visa transactions are just a few
companies changing a few numbers in their databases. It's basically just a fancy
way of signalling an intent to spend something. It's not final. There might not
be a real transfer of value happening at all (all the "money" is still debt in
the same bank, just on another account).
On the other hand, a Bitcoin onchain transaction is a real transfer of a bearer
instrument, comparable to a gold shipment from one central bank to another.
Those are also quite expensive, I imagine. It's total overkill if you want to
buy a coffee. For that, it might be enough to just update some numbers in a
(Bitcoin) bank. Or use the Lightning Network.
If you have a little time, Nic Carter has explained it much better than me.
If you have more questions regarding Bitcoin's carbon footprint, I recommend the FAQ at netpositive.money

interesting, if the real value is store of value, why are they trying to create Lightning network? seems like an additional benefit they want

Blockchains don't scale. Lightning is a technology for enabling trustless payments/transactions that needs the Bitcoin blockchain as a base layer, but can scale much further. Other benefits include potentially better privacy and immediate finality. This will enhance Bitcoin's functioning as a medium of exchange. Beside the technical complexity, however, this is limited by economic problems, ATM, mostly volatility. So SoV comes first, MoE is still developing.

just accepted answer and gave points, feel free to thumbs up question also, thanks

No, bitcoin is inherently not an investment.  There are three distinct concepts, but any one instrument could be more than one of these:

Saving, or Store of Value

Bitcoin may well be this, at least in the medium term

Speculation, or betting on the change in value of something

Bitcoin is principally this

Investment, or spending money to increase productive capacity

Bitcoin is very much not this.  In fact, on a global scale, it is the opposite.  By using so much of our carbon buffer, productive capacity and waste disposal capacity it is literally reducing our useful productive capacity.

By buying bitcoin, you are providing price support for the mining operations that you detail.  If you invest, you help everyone have more stuff. If you buy bitcoin you help everyone have less stuff.

I agree with your investing vs saving distinction, but I don’t think buying bitcoin necessarily leads to a net reduction in useful capacity: the lower friction and less censorship when compared to legacy financial networks might make for a net increase of throughput of economic activity, as new forms of economic interaction become possible using bitcoin. The effect you described may well exist, but it is also possible it could be outpaced by the gains of these new avenues of economic interaction.

@chytrik with current transaction fees, where do you see this low friction?  We are talking about 1% of our CO2 production, it should be demonstrable on that scale.

tx fees are only one factor relevant in quantifying ‘friction’, but in any case, as example, lightning payments allow on-chain fees to be amortized across a potentially unlimited number of payments. The tech is still young but the potential of it is mind-blowing in terms of facilitating low-friction payments.

The friction of bitcoin investment is 1% of current CO2 production.  What is this lower than?

there is no other network which provides the same utility (censorship resistance, over-the-wire transfers, independently verifiable, etc), so I’m not sure it is relevant to compare it to a legacy system in the way you’re suggesting. However, I also think the real measuring stick here is ultimately opportunity cost, not energy. I think that legacy systems do incur a higher opportunity cost per unit of security, though an exact quantification of this is cost extremely difficult for certain (what is the opportunity cost of running the legacy financial system?).

I am specifically questioning the "lower friction" claim made in the first comment to my answer.  That requires a comparable investment that has lower friction than bitcoin, and I am not aware of one.

Ah, I understand your point better now. As example then: If I want to send ~$1 worth of value to someone in a far-off country, then a lightning network payment is lower friction than any other method I can think of. It is as simple as communicating a number to the counter-party privately.  Much lower friction than any other payment method (wire transfer? PayPal? Dollar bills sent in an envelope??). Bitcoin does for money what the internet is doing for information: ease of transaction! And so people will transact more, and in new ways (again, similar to the change the internet is effecting).

The friction of bitcoin investment is 1% of current CO2 production. < You are off by quite a bit here. Global energy-related CO2 emissions were 31.5 Gt in 2020 (https://www.iea.org/articles/global-energy-review-co2-emissions-in-2020). Bitcoin mining caused at most 35.89 Mt (https://netpositive.money/calculator/) That's about 0.1 % of JUST the energy-related CO2 emissions.

Bitcoin's energy consumption is not a long-term problem.  It is a short-term red herring being used by those afraid of Bitcoin's eventual sweeping adoption.
There are already plans being developed to make Bitcoin's mining inefficiencies (energy consumption) independent of terrestrial sources of energy.


text:
How does a HD wallet work in practice?
EDIT - I have refined the question in the post to be more succinct.

If I derive multiple child/grandchild key pairs from the original master public key, I understand each of these is its own specific address. However, if grandchild key pair 1 contains 1 BTC, and grandchild key pair 2 contains 2 BTC, could I send the 3 BTC total as a single transaction? If so, would this appear on-chain to leave from the child key pair address from which both are derived?

If the master private key was in cold storage, but one of the child private keys was not, could the wallet be defined as a cold wallet?

Thanks

Does this answer your question? How to create off-chain transaction, the one that exchanges uses

If I derive multiple child/grandchild key pairs from the original master public key, I understand each of these is its own specific address. However, if grandchild key pair 1 contains 1 BTC, and grandchild key pair 2 contains 2 BTC, could I send the 3 BTC total as a single transaction? If so, would this appear on-chain to leave from the child key pair address from which both are derived?

To answer this, it would be useful to see how a transaction appears on-chain. E.g. see here
In this random example, you can see that the transaction in question has 2 inputs. Each of these inputs comes from a different bitcoin address which may or may not come from the same "wallet" (as a wallet can have more than 1 address - in fact, it is very common, and most individuals will generate a new address to receive every transaction). To fit your example, these could both come from the same HD wallet, with each address representing one of the grandchild key pairs. This demonstrates that 1 transaction can be "sent" from more than 1 address.
Equal to this, you will also see 3 outputs. However, you will see the 3 outputs correspond to only 2 different bitcoin addresses. This is due to the UTXO bitcoin protocol, which you can read about in more detail separately. For our purposes, this shows that an input, or output, does not necessarily correspond to new address. 2 of the outputs go to the intended recipient, a single address likely generated to specifically receive this transaction, while another portion goes to the second address. This is likely (although not certainly) a "change address" and belongs to the original sender (who controls both of the input addresses i.e. has their private keys). Once again, this is due to the UTXO protocol, and you can read why this can occur separately.
To conclude, I think you have confused HD wallets as simply a wallet having more than 1 address (whereas the HD is more about how the private keys are derived/made). In your example, the transaction can leave from more than 1 address (and hence have more than one input), as will appear on-chain. This is nothing to do with HD wallets specifically, as many wallets can have more than 1 address, but it is about how transactions work/appear on-chain.

However, if grandchild key pair 1 contains 1 BTC, and grandchild key pair 2 contains 2 BTC, could I send the 3 BTC total as a single transaction?

Yes, but not in the way you describe it.
Bitcoin, at the protocol level, does not "belong" to a key. That is, there is no balance per address or per key as a concept.
Instead, think of them as coins. Every time you get paid, you receive a separate new "coin". Each coin has a little program (the locking script) that determines who is allowed to spend it. In practice, your coins are ones with a locking script that requires a signature with a key you have.
The protocol permits creating transactions that spend multiple coins at once. It does not matter whether those coins all have the same address, or different addresses derived from the same master key, or even if they have completely unrelated addresses entirely. They're just multiple coins, and each coin individually requires a signature.
in fact, nothing prevents multiple people from constructing a single transaction that spends multiple coins owned by different people in the same transaction. This is called a CoinJoin transaction.

If so, would this appear on-chain to leave from the child key pair address from which both are derived?

All HD wallets do is change the way an individual wallet derive their keys. Other nodes or parties on the network do not know or care the keys are constructed in a related way. As long as they do not know the master public key, your keys are indistinguishable from random keys. The only thing visible on chain is that the transaction spent coins that were previously sent to distinct addresses, but this is the case for almost all transactions on the network.

If the master private key was in cold storage, but one of the child private keys was not, could the wallet be defined as a cold wallet?

This is not possible with non-hardened keys. Given a child private key and the master public key, you can compute the master private key, and from that, all other child private keys.
Thus, you scenario is equivalent to the non-cold system having access to all private keys, and the whole setup is very much not cold.
With hardened keys, knowing the child private key does not reveal the master private key, but you lose the advantage of being able to compute child public keys without access to the master private key.
in either case, I would not call this a cold setup. I would reserve that term for setups where no coins are ever on an online system.