text:
Why can I not dump private keys from a descriptor wallet?
I created a new wallet in Bitcoin Core v23.x,
as expected it created an empty wallet
version 169900
descriptors true
format sqlite.
It seems it created 4 streams of 1000 keys each.
Anyway I told it create some P2PKH, P2SH-nested P2WPKH, P2WPKH, P2TR addresses.
It created a bech32m address, but it won't let me access private keys.
Whatever the address type, it's telling:

error code: -4
error message:
This type of wallet does not support this command

Why is this happening and where to go from here?

As the error message says, the type of wallet you are using does not support the dumpprivkey command. You are using a descriptor wallet, and individual private keys cannot be exported from the wallet.
You can get the master private keys by using listdescriptors true. This will output all of the descriptors stored in the wallet along with their corresponding private keys. Because Bitcoin Core uses BIP 32 derivation, instead of thousands of individual private keys, you will get descriptors which contain the master private key from which you can do derivation on to get the individual child keys.
Descriptor wallets specifically disabled export of child private keys because it is unsafe. Because unhardened derivation is used by default, a child private key and a parent extended public key is equivalent to having the parent extended private key. However it is not immediately obvious to the vast majority of users that this is unsafe, so child private key export is disabled.

When did Core switch to non-hardened as the default? Is there a reason non-hardened keys is the default? https://bitcoin.stackexchange.com/a/50247/26873

@m1xolyd1an Since descriptor wallets were added, and only for descriptor wallets. As descriptor wallets make it easier to import and export wallets via descriptors, we decided to move to use the standard derivation paths that allow for such exports.

Since "Descriptor wallets specifically disabled export of child private keys because it is unsafe.", how do I sign a 2-of-3 PSBT transaction whose one of its public keys is of an address derived in such a descriptor wallet - without exporting its child private key?  Looks like I will need to import the multisig adddress' descriptor into the wallet first, and it will need to have a masteror child private key, so I'll end up export it anyway. Or am I thinking about it in a wrong way?


text:
How can a private key be imported to a descriptor wallet?
If the Bitcoin Core Wallet Migration Tool is supposed to be creating descriptors for sparse keys (as it's supposed to support pre deterministic wallets), then why doesn't Bitcoin Core allow using importprivkey on a descriptor wallet?
I'm trying to, and it's telling error code -4 (type of wallet does not support this command).
How can I import a private key to a descriptor wallet?

The migration tool has not been implemented yet, and it is not for manually importing keys. Rather it takes an existing wallet and migrates everything in that wallet.
As the error message says, descriptor wallets do not support importprivkey. You will have to use importdescriptors with the private key you want to import as part of the proper descriptor.
For example, if you wanted to import the P2WPKH address for a private key, you would make a descriptor of the form wpkh(<privkey>). Then use getdescriptorinfo to get the checksum and create wpkh(<privkey>)#<checksum>, and then use importdescriptors to import that descriptor.

getdescriptorinfo would just tell wpkh(): Uncompressed keys are not allowed

You shouldn't use uncompressed keys in wpkh; they aren't valid.

@PieterWuille understood, thanks, now I got checksum.

Importing the descriptor would then answer error code -4 Cannot import descriptor without private keys to a wallet with private keys enabled, but the descriptor was created out of a dumped privkey and the wallet is an newly created descriptors wallet

@Mercedes getdescriptorinfo does not give you the descriptor with the private key. It sounds like you are taking the returned descriptor from getdescriptorinfo and importing that directly, when what you actually want is to get the checksum field of getdescriptorinfo's result and construct your descriptor with private key manually.

OMG THAT WORKS like a charm tyvm

@AndrewChow Could you please revisit your answer? wpkh returns "Method not found (code -32601)"

@MartinBraun wpkh is not an RPC, it's a function in the output descriptor language, which you'd use in commands such as importdescriptor.

@PieterWuille Thank you, now it makes more sense, although numerous things still confuse me: Should I pick the checksum in the descriptor key, or in the checksum key when building my descriptor for import? Also, the importdescriptor command requires a timestamp. listtransactions can only work for imported addresses, is there even a way to get my timestamp without checking the blockchain externally? importdescriptor also requires an internal flag, so if my address will be a change or receive address. Since this address has funds, I assume I will make this an internal address?

Those questions are too extensive to answer in a comment here; I'd suggest you read the documentation (https://bitcoincore.org/en/doc/24.0.0/rpc/wallet/importdescriptors/ and https://github.com/bitcoin/bitcoin/tree/v24.0.1/ e.g.) more, and open question(s) for things that aren't clear.

@PieterWuille I get that, I will figure out how I can get the timestamp, but the checksum part seems to be apart of the problem. getdescriptorinfo returns two checksums (one in the descriptor itself, one in checksum). Could you please tell me which one I should pick?

The checksum it reports is the checksum for the descriptor you provided. The descriptor it returns is a full, normalized, descriptor including the checksum (for that normalized descriptor). You can either use your input + checksum, or you can use descriptor. The latter will miss private key information if the former included any.

@MartinBraun (for the sake of completeness) [on the current master branch] `internal' is an option with a default and the only mandatory data besides the descriptor is the timestamp.

I read all these comments.  Why is the end user (a new/typical bitcoin gui user who has their older private key[s]) expected to do all of this to "simply" import an older key?   After studying some more articles, I now have a greater appreciation for why backwards compatibility was broken.  But why am I having to hunt around for all these explanations?  No compatibility chart on the home page?  No import wizard in GUI (or templates)?

@JonGrah Importing and exporting has never really been a feature meant for most users - it's never been directly available in the GUI. There is an open PR that would make importing available in the GUI, and make it a lot easier to import descriptors, but it has not garnered any review.

To give a bit more context to Ava Chow's answer: the reason why this isn't any easier is that the concept of "importing a private key" is an insufficient method for describing what a wallet should do, and the wrong way to think about wallets in the first place. While it is possible to convert wallets with keys imported to them to approximately equivalent descriptor wallets, this does not mean that it's the best way when importing things. Descriptors and descriptor wallets allow you to be precise about what you want to import, and the importdescriptors RPC lets you do that.
In more detail, this is roughly the philosophy behind the two types of wallets supported by Bitcoin Core:

Legacy wallets: A legacy wallet is a collection of private keys, and loosely structured additional information such as scripts and addresses. Any output which can be spent or observed with those keys/scripts/addresses is considered to "belong" to the wallet. If you import a new key to a wallet, this immediately results in treating payments to the P2PK, P2PKH, P2WPKH, P2SH-P2WPKH, and possibly more, addresses derived from that key to be watched. This is inefficient, hard to describe and reason about, and just doesn't scale well with new wallet constructions being added (e.g. P2TR), and even harder to deal with when multisig across devices or participants is added.
Descriptor wallets. In a descriptor wallet, the outputs considered part of the wallet can be described exactly using a simple "language" which contains all metadata about how the keys are used. If you want just P2PKH addresses derived from a key, you can say that. If you want P2SH-wrapped P2WSH multisig across multiple devices you can say that too, and it will work exactly the same.

In short: the old model of how we thought of wallets wasn't manageable anymore, and "importing a key" only made sense in that kind of thinking. In the new model, you don't import keys, you import a description of exactly what you want the wallet to do.

I think I understand all of your point, but I'm still not sure why importprivkey isn't reasonable high level sugar for a larger sequence of commands that import one descriptor wrapping one single isolated key. Isolated keys being wrong doesn't destroy existing isolated keys. I'd suppose it's easier to detect legacy behavior and warn against it by accepting importprivkeys than by having users use the new toolset in a conceptually wrong way, unable to warn of legacy behavior until importdescriptors is seen importing a single isolated key.

The question is just: what way of wrapping the private key? P2PKH? P2WPKH? P2SH-P2WPKH? P2SH-P2WSH-P2PKH? P2TR? There are many ways of constructing a scriptPubKey whose spendability depends on access to a given private key. The practice of watching all of them just doesn't scale. Descriptors allow one to be explicit about all of this.

I take your word for it doesn't scale. Maybe some link can be added explaining further.

Also worth a read: https://achow101.com/2020/10/0.21-wallets#:~:text=Benefits%20of%20Descriptor%20Wallets,scripts%20necessary%20to%20sign%20them.

@Mercedes You don't need to take my word for it; I gave the explanation above: there isn't just one way of turning a private key into an address. Descriptors add the necessary metadata (whether that means just fixing whether it's P2PKH or P2SH-P2WPKH or whatever, or including other keys if it's multisig, or ...).

So is there unlimited ways to turning an old private key into an address?  There are only so many variations among the Core apps.  So can't a locally run script account for these different variations?  Or templates offered for the common variations?
the minimum assumption is that the user has private key and the matching public address.  And has some idea when the original address/wallet was created.

@JonGrah In practice, as long as you're only talking about single-key addresses (and not multisig or other more advanced things), there are only a few variations, and for recovery purposes it's easy enough to just try all of them. The problem is that that number increases over time, so for common usage you absolutely don't want to just try all - that's a growing multiplier to the cost every time new address types are introduced.

Here are example commands which can be used with bitcoin-cli to import wallet using WIF into new wallet:
wallet_name=mywallet
wif=5MyWifPass
bitcoin-cli -named createwallet wallet_name=${wallet_name} descriptors=true disable_private_keys=true
bitcoin-cli getdescriptorinfo "pkh(${wif})" # Legacy: pkh(), for P2SH-SegWit use sh(wpkh())
descriptor=$(bitcoin-cli getdescriptorinfo "pkh(${wif})" | jq -r .descriptor)
bitcoin-cli -rpcwallet=${wallet_name} importdescriptors '[{ "desc": "'${descriptor}'", "internal": false, "timestamp":"now", "label": "'${wallet_name}'"}]'
bitcoin-cli -rpcwallet=${wallet_name} listdescriptors
bitcoin-cli -rpcwallet=${wallet_name} listreceivedbyaddress 1 true

The above needs to be run in shell.
It assumes you've rpcuser and rpcpassword setup in your ~/.bitcoin/bitcoin.conf, otherwise add -rpcuser=youruser -rpcpassword=yourpass to the command.
Related:

[HOW-TO] Import privkeys into a Bitcoin Core descriptor wallet

The how-to link worked great! https://bitcointalk.org/index.php?topic=5498210.0

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Any idea why Bitcoin Core signature is untrusted?
It's suppose to be a trusted signature, right?. I'm downloading Bitcoin Core from bitcoin.org
gpg --import laanwj-releases.asc
gpg: key 90C8019E36C2E964: 51 firmas no comprobadas por falta de claves
gpg: clave 90C8019E36C2E964: "Wladimir J. van der Laan (Bitcoin Core binary release signing key) <laanwj@gmail.com>" sin cambios
gpg: Cantidad total procesada: 1
gpg:              sin cambios: 1

sha256sum --check SHA256SUMS.asc
sha256sum: bitcoin-0.20.0-aarch64-linux-gnu.tar.gz: No such file or directory
bitcoin-0.20.0-aarch64-linux-gnu.tar.gz: FAILED open or read
sha256sum: bitcoin-0.20.0-arm-linux-gnueabihf.tar.gz: No such file or directory
bitcoin-0.20.0-arm-linux-gnueabihf.tar.gz: FAILED open or read
sha256sum: bitcoin-0.20.0-osx64.tar.gz: No such file or directory
bitcoin-0.20.0-osx64.tar.gz: FAILED open or read
sha256sum: bitcoin-0.20.0-osx.dmg: No such file or directory
bitcoin-0.20.0-osx.dmg: FAILED open or read
sha256sum: bitcoin-0.20.0-riscv64-linux-gnu.tar.gz: No such file or directory
bitcoin-0.20.0-riscv64-linux-gnu.tar.gz: FAILED open or read
sha256sum: bitcoin-0.20.0.tar.gz: No such file or directory
bitcoin-0.20.0.tar.gz: FAILED open or read
sha256sum: bitcoin-0.20.0-win64-setup.exe: No such file or directory
bitcoin-0.20.0-win64-setup.exe: FAILED open or read
sha256sum: bitcoin-0.20.0-win64.zip: No such file or directory
bitcoin-0.20.0-win64.zip: FAILED open or read
sha256sum: bitcoin-0.20.0-x86_64-linux-gnu.tar.gz: No such file or directory
bitcoin-0.20.0-x86_64-linux-gnu.tar.gz: FAILED open or read
sha256sum: WARNING: 20 lines are improperly formatted
sha256sum: WARNING: 9 listed files could not be read

gpg --verify SHA256SUMS.asc
gpg: Firmado el mié  3 jun 10:59:52 2020 WEST
gpg:                usando RSA clave 90C8019E36C2E964
gpg: Firma correcta de "Wladimir J. van der Laan (Bitcoin Core binary release signing key) <laanwj@gmail.com>" [desconocido]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 01EA 5486 DE18 A882 D4C2  6845 90C8 019E 36C2 E964

related: https://bitcoin.stackexchange.com/q/50185/5406

When you import a key, you can mark it as trusted. Usually, you would do this after meeting the owner of a key and verifying that they actually control the key, e.g. at a CryptoParty. Since it is infeasible for each user to verify every other user in person, PGP/GPG leverages a "web-of-trust" to establish connections between keys. E.g. Alice has met Bob and trusts Bob's key. Bob has signed Carol's key, so Alice transitively trusts that Carol's key to some degree. There are some issues with the reliability and assumptions of this approach, but that's how it works.
Anyway, what GPG is telling you here is that the package you have downloaded was indeed signed by the key you checked against, but it warns you that you have not verified the authenticity of the signing key.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
"New rules activated" warning after upgrading to 0.20.0
I've upgraded a node from the previous release. It's up and running but for some reasons I get this warning.
How can it be related to the upgrade?
$ bitcoin-cli -testnet getblockchaininfo | jq
{
  "chain": "test",
  "blocks": 1764182,
  "headers": 1764182,
  "bestblockhash": "0000000000004a0e9cc1fbc2a21938feef1a26cd480483b3c498ae2ef16b46f8",
  "difficulty": 65536,
  "mediantime": 1591710673,
  "verificationprogress": 0.9999998260147338,
  "initialblockdownload": false,
  "chainwork": "00000000000000000000000000000000000000000000015bbad4a75d4239391e",
  "size_on_disk": 27411142847,
  "pruned": false,
  "softforks": {
    "bip34": {
      "type": "buried",
      "active": true,
      "height": 21111
    },
    "bip66": {
      "type": "buried",
      "active": true,
      "height": 330776
    },
    "bip65": {
      "type": "buried",
      "active": true,
      "height": 581885
    },
    "csv": {
      "type": "buried",
      "active": true,
      "height": 770112
    },
    "segwit": {
      "type": "buried",
      "active": true,
      "height": 834624
    }
  },
  "warnings": "Warning: unknown new rules activated (versionbit 28)"
}

Miners have been using the version field of the blocks' headers as an "extra nonce" (*).
However, this version field has been used to signal readiness from miners to deploy a soft fork (update to new backward-compatible rules).
The version field being used to synchronize the update to new consensus rules is the reason you see the warning.
The version field value not being enforced by the consensus, and the development of new techniques that increase miners' revenue is the reason they can "disgress" the protocol.
(*) In this specific case, it's known as version rolling and is likely part of using overt asicboost.

EDIT - Some reference of discussions about how to deal with this in bitcoin-core :

BIP320 implementation
Discussions on an issue
GUI warning removal


text:
Ironically, encryption is not an important part of bitcoin (?)
The above statement is taken out of a book which I've been reading: Mastering Bitcoin 2nd Edition.

...as its communications and transaction data are not encrypted and do
  not need to be encrypted to protect the funds...

Just started about bitcoin, going through the third chapter and when I come across the above lines, I merely struck by one question, and that is, if the communication happens between (peers' / wallet to bitcoin network) is not encrypted...then I am just thinking a scenario
X is sending BTC to Z (obviously by mentioning Z's address in txn)
Y (bad guy) cuts in the communication which is not encrypted (as per the above reading) and tampers the transaction that X has been sent ( by Just putting his address in beneficiary )
I mean, honestly, it is a very naive scenario that anyone could think of and I know there would be something in bitcoin (which I am not aware of) could prevent this from happening, so, giving clarification on how this kind of attacks being handled or mitigated in Bitcoin would be much helpful for this post.

Related: https://bitcoin.stackexchange.com/q/95564/5406

Cryptography and Encryption
In the wider world, cryptography is used for a variety of security related purposes:

Authentication (proof of identity, or proof of ownership)
Non-repudiation (author cannot later deny authorship of data)
Confidentiality (secrecy - third parties can't see the data)
Integrity (proof that data has not been changed or tampered with)

When we say "encryption" without clarifying further, we are usually talking about the use of cryptography to ensure confidentiality (secrecy of data, but not secrecy of a transmission having occurred - the latter might involve steganography or private channels etc).
Bitcoin's peer-to-peer protocol doesn't need secrecy but it does need a form of authentication and integrity proof that is provided by digital-signatures.
Traditionally digital-signature algorithms used RSA which involves encrypting a hash of data but not encrypting the data itself. Bitcoin uses Elliptic Curve cryptography which produces digital signatures without involving encryption of a hash.
So the author is perhaps pointing out that the data sent by Bitcoin peer-to-peer protocols is not encrypted for confidentiality and does not involve encryption.
Terminology
The word crypto-graphy literally means hidden writing. There is a certain irony in using this term for elliptic curve cryptography which cannot directly be used for hiding the text of messages - what many people mean by encryption is the process of converting normal readable plaintext into unreadable ciphertext.

If by traditionally you mean "in RSA". Looking at the variety of digital signature schemes that exist, ones that reduce to encrypting a hash are rare. In particular, ECDSA, the digital signature scheme used in Bitcoin, does not use anything that could be useful for encryption.

Standard transactions are digitally signed using a private key. If a middle man tampers with the transaction, then the digital signature verification would fail, because the only way to produce a valid signature is to have the original private key. This would cause the transaction to fail and the network would reject it as invalid.
This signature algorithm is called ECDSA and a common address type that requires signature verification is a P2SH(P2WPKH) (Pay to Script Hash - Pay to Witness Public Key Hash).
Also, it is important that the transactions are not encrypted, otherwise the rest of the network would not be able to verify that the transaction was valid (and prevent double spends, etc.).

"if a middle man tampers with the transaction, then the digital signature verification would fail, because the only way to produce a valid signature is to have the original private key".......................one can possibly keep the signature intact and modify the beneficiary can't they?

@Explorer_N no, the signature is made over the relevant parts of the transaction, including the recipients address. Modifying any of that data will cause the signature verification to fail.

@Explorer_N Related question discussing about modification of recipient: https://bitcoin.stackexchange.com/questions/90028/what-is-stopping-me-from-modifying-the-mempool-of-my-node-may-it-be-pos-or-pow

Thanks to all the contributors; I realized I was bit hasty..After continue reading the book (6th chapter now!) during my spare time I come across this passage

A digital signature serves three purposes in bitcoin (see the
  following sidebar). First, the signature proves that the owner of the
  private key, who is by implication the owner of the funds, has
  authorized the spending of those funds. Secondly, the proof of
  authorization is undeniable (nonrepudiation). Thirdly, the signature
  proves that the transaction (or specific parts of the transaction)
  have not and cannot be modified by anyone after it has been signed.

Third purpose answers this question.


text:
Can I recover my money in a Coinbase Wallet?
I just want to ask if there is possibility to recover my money in a Coinbase wallet. I forgot my recovery phrase but I remember my wallet address and TXID.

According to coinbase

Wallet is a user-custodied digital currency wallet and DApp browser. This means that with Wallet, the private keys (that represent ownership of the cryptocurrency) are stored directly on your device and not with a centralized exchange like Coinbase Consumer. You do not need a Coinbase Consumer account to use Wallet and you can download Wallet from anywhere in the world.

You cannot recover control over money in a user-custodied wallet using only a Bitcoin-address and a Transaction ID.
If you have somehow lost your wallet (e.g. phone dropped into ocean) - without the recovery phrase or some other equivalent sort of backup, your money is gone forever.

If by "Coinbase Wallet" you actually mean "Coinbase Consumer" - that is a custodial-wallet (or equivalent service) and your only possible way to recover control over your money is to communicate with the help-desk of Coinbase.


text:
How Bitcoin wallets and it's transfer process work?
I am very new on cryptocurrencies, and like to understand it's transfer process. I think if I want to buy Bitcoin, I must download a Bitcoin wallet, then find a person who has Bitcoin. Then he/she must transfer that bitcoin from his/her wallet to my wallet. Then I will pay the price either in cash or with credit card to him/her bank account.
After that I can see this Bitcoin in my wallet. OK? But I have some questions:

How many wallets a person can download and use?
If I used a wallet in the above example, then I wanted to change my wallet, should I download a new one and do a transfer from my previous wallet to new one? I mean there is no unique ID for people that they can use it in different wallets to reach one unique account?(Like our stack account that we can reach to it by Chrome, Firefox, Edge, etc, considering the browsers are wallets)
How crypto exchange websites work? If I want to buy or sell 1 Bitcoin, should I pay the money/coin to their bank_account/Bitcoin wallet, and the other person should do that too, then the website will pay it to us? I mean is it a 2-step process or buyer/seller do a direct transfer?

First, a wallet is a software application that usually manages your keys, addresses, and transactions. When someone sends you Bitcoin, it is generally sent to one of the addresses that the wallet software manages. There really is no limit to how many wallet software applications you can run, and not really a limit on addresses either (except for disk space, but that's not really a concern etc.).
Your wallet software should allow you to create new addresses that you can use to receive Bitcoin.
Exchanges work by managing your Bitcoin in their internal wallet software, and you exchange anonymously with others by placing and filling orders on the global order book. The exchange manages the Fiat (dollars) currency exchange internally as sort of an escrow.

How my personal wallet will contact the exchange's internal wallet? Is that mean I must send my coins to the exchange's wallet then they will send it to another person's wallet who likes buy my coins?

@user3486308 You generally send bitcoins to an address provided to you by the exchange. But, the exchange then at regular intervals (may be hourly or daily depending on volume) consolidate all their user's holdings in their cold/hot storage addresses. Howver an internal database is maintained that credits the bitcoins to your account. When you sell on an exchange, since the other guy also has an account on the exchange, the exchange can merely change the credits at the backend without making an on-chain transaction. It is only when you withdraw the money will they make an on-chain transaction.

How many wallets a person can download and use?

As many as they want.

If I used a wallet in the above example, then I wanted to change my wallet, should I download a new one and do a transfer from my previous wallet to new one?

Yes that is often the best method, but that will incur a transaction fee.

I mean there is no unique ID for people that they can use it in different wallets to reach one unique account?

Yes there is a unique piece of data that can be moved between different wallets and which gives users of that wallet control over an amount of Bitcoin. This is the private-key. It can often be generated from a seed-phrase, sometimes called a recovery phrase.
You could have ten wallets each containing the same private key. All of those ten wallets would show you the same "balance" and any could be used to spend all or part of that "balance". The other nine wallets would find out about the transaction and adjust the "balance" they show. However having your private key in ten places is less secure than only having it in one (plus an offline backup of some sort).
You can alternatively import private keys into a new wallet but you need to understand the difference between the early type of private key and those used in hierarchical deterministic (HD) wallets - an "extended" private key from which others are generated. When moving a private key from an HD wallet into another you need to make sure the new wallet is configured to use the same "derivation paths". This can be complicated so it requires care.

How crypto exchange websites work?

In my view, exchanges are mainly for exchanging one currency into another. For example USD into Bitcoin or Bitcoin into Litecoin etc.
A member of an exchange will deposit some money into their exchange account and can then typically transfer some to the exchange account of another user of the same exchange without any fee. Amounts transferred this way are not recorded in the Blockchain. The exchange remains the "owner" of the money.
Exchanges also typically allow members to transfer money to non-members. These are typically a transfer from the exchange's wallet to the other persons wallet and this transaction is recorded in the Bitcoin blockchain and involves a transaction fee.

How my personal wallet will contact the exchange's internal wallet?

They don't.
Wallets broadcast intended transactions to a few nearby nodes. They pass it on. Eventually a miner incorporates that transaction into a new block and broadcasts that block to nearby nodes. They pass it on. Eventually your wallet and the recipient's wallet hear about the new block. Eventually the wallets mark the transaction as confirmed and update the calculated "balance" they show their users.

It is worth remembering that Bitcoin are not stored in wallets or in the Blockchain. Wallets store private-keys. The Blockchain stores historical records of transactions. A transaction is a record of a transfer of control over some amount of money where Bitcoin is the unit of measure.
When you give money to an exchange, you no longer have money, you have an IOU.


text:
How to you recover a stolen bitcoin lost to an investment scam?
I once lost my bitcoin to an investment platform and it took a while before I managed to get my bitcoin back. My question is I want to know how many people have lost their bitcoin and how do you manage to get them back?.

How to you recover a stolen bitcoin lost to an investment scam?
how do you manage to get them back?

Generally you don't.
Take MtGox as an example. It is believed that a very large amount of money was stolen. so far as I know, none of that has been recovered. A small amount was discovered that the MtGox staff had lost track of and last I heard this was to be returned to the customers. If so some customers recovered some of their money. This was only possible because the exchange was based in a country relatively free of corruption where administrators were appointed and there was a police investigation.
Another example would be Quadriga-CX where the owner either died or faked their death and was the only person who had access to the private keys. So far as I know no customers got any money back.
These were initially genuine businesses and probably not investment scams from the outset. They were large exchanges with many customers who lost money. For smaller exchanges in countries where law enforcement is poor, there is even less chance of recovering stolen funds.
There are probably large numbers of people who gave money to scammers who created a website that was only around for a few months before the scammers moved on and set up new ones. Those victims are very unlikely to have the resources to do anything and even if informed police will likely feel the amounts are too small to warrant spending time on them.  In a dispute over returns on a speculative or failed investment it might not be obvious that there is evidence of a crime. Some events may be disputed and may fall under civil law (e.g. contract dispute) rather than criminal law (e.g. fraud)

I want to know how many people have lost their bitcoin

Nobody knows. No-one tracks this globally. Not everyone reports loss to the police. Local police might have data and might report it nationally but they often might not know if the claims are accurate. So at best, some countries will have an estimate of total cryptocurrency crime but they might not break the figures down in the way you might want.
People also lose money for reasons other than fraud. They don't make backups then lose their computer or phone, they forget they had a wallet or an account or forget the password. Sadly people die and their family often are unable to find their keys even if they know their dead family member had Bitcoin.

You can't know how many people have lost their Bitcoin because bitcoin is not lost. If you have your private keys, you have the Bitcoin, so the Bitcoin change from one hand to another hand (EG. if you send your Bitcoin from your wallet to another person).
The problem with most platforms is that they are in control of keys, this means that they have the Bitcoin, not the person that are investing.
And the only way that you have to get them back is taking back control of the private keys.


text:
what channel balances are implied by routing hints?
I understand that as part of a lightning invoice routing hints can be provided that provide information about private channels that can be used to reach the destination. I believe these are the r fields in BOLT 11.
I also found a link to the LND implementation that appears to only include hints for channels that have sufficient balance to route the entire amount by themselves. So this means that at least for LND a routing hint implies a certain minimum channel balance.
I've got the following questions

is this specific to LND or do other implementations do the same?
if so, it could make sense to provide routing hints even for public
channels, just to signal that they have sufficient balance. Is this
allowed and do implementations do this?
if a certain destination wants to receive 100 satoshis and has 10 private channels that have a
remote balance of 99 satoshis each, does this mean no routing hints will
be provided, because no single channel can route the entire amount by
itself, so the payment will fail, even though the payment is feasible
using a multi path payment?

is this specific to LND or do other implementations do the same?

Other implementations also send these.

if so, it could make sense to provide routing hints even for public channels, just to signal that they have sufficient balance. Is this allowed and do implementations do this?

It might make sense, I've thought about ideas where you could signal which channels you'd prefer payment through even if they're public. From the senders point of view they wouldn't be able to tell what is most convenient for the recipient, so this could help alleviate issues.

if a certain destination wants to receive 100 satoshis and has 10 private channels that have a remote balance of 99 satoshis each, does this mean no routing hints will be provided, because no single channel can route the entire amount by itself, so the payment will fail, even though the payment is feasible using a multi path payment?

LND tries to provide at least 20 channels to facilitate payment to a private/hard to find node. If the number of high quality channels is lower than 20, LND will select as many low quality channels as possible (without doxxing other private nodes) to meet that 20 channel requirement.
To simplify, it will provide all those 99 sat channels.
Another idea of note - all hophints are designed to do is add more edges to the senders's graph, in case their graph doesn't have the information required to route to the recipient.
I wrote a short blog about hophints a few months back, maybe you'll find it interesting!
https://write.as/arshbot/everything-you-need-to-know-about-hop-hints-in-the-lightning-network

thanks for your answer! Somehow I missed the second pass in the source file that I linked although it's very clear now. Sorry for that. So to summarize: no, routing hints do not with certainty imply a certain minimum channel balance, however the first routing hints are more likely to have sufficient balance, because those are added first. Interesting blog post.


text:
How to actually handle the xPub gap limit when receiving Bitcoin deposits?
I've read the other questions involving this, including How to solve Blockchain 20 address gap-limit problem? and Blockchain gap limit which don't actually remedy the issue in a practical sense, only further explain why it exists.
I understand why the gap exists, how it works, and how Blockchain.info handles it. (Both with callbacks, checkgap API requests, returning an error when the gap reaches over 20, etc)
My question is, what can I, as a developer, do to prevent exhausting this gap? Is this even a valid method for handling a larger website's Bitcoin deposits from users? Do I just have to hope that 20 users don't generate an address for fun, sending operations into a standstill?

Any website that is handling user funds should be running a node to ensure they have an accurate view of the network. Depending on an API introduces trust of a third party into the situation, which is an unnecessary risk.
As a developer, you can then simply customize the gap limit to suit your business’ needs. In many cases this likely just means increasing the gap limit to a larger number.


text:
Can you have a cryptocurrency with a single source of truth?
Can you have a cryptocurrency that's based on TRUST? (how dare I use this dirty word haha)
Everyone who participates in the system agrees to trust this 1 party only, because they find them trustworthy. And only they are allowed to add blocks to the blockchain. Or they can just process transaction on a 1st come 1st serve basis and eliminate the need for blocks, just timestamp each transaction :)
It would make things way easier but it will make the cryptocurrency centralised?
Is that even legal to do? Can someone create their own centralised cryptocurrency and let people buy/use it?

Is that even legal to do? Can someone create their own centralised cryptocurrency and let people buy/use it?

What is legal/not-legal can change drastically depending on what jurisdiction you live within. If you have legal concerns, you should talk to a lawyer.
Generally though, there is a history of powerful governments shutting down centralized digital-money systems (eg Liberty Reserve), which culminated in the invention of the Bitcoin network (which has no central point of control/authority which can be shut down). This is similar to the evolution of file sharing: tools that were still at least partially centralized were shut down over time (eg Napster), until eventually someone invented a tool that couldn't be shut down (bit torrent).

Everyone who participates in the system agrees to trust this 1 party only, because they find them trustworthy

Even if the operator of the currency is perfectly benevolent and fair (ie, trustworthy), there is still a massive risk inherent to having a centralized point of control: what happens if the control falls into the wrong hands? (eg a rogue employee, hacker, government, etc). This is a massive risk, that only increases as the value secured by the network increases. What happens if the government policy changes? A war breaks out? The centralized entity running the network goes bankrupt? What if a natural disaster takes out their office? There are a lot of ways things could go wrong, and even if the chance of something happening is small, the impact could be massive.
On the other hand, the Bitcoin network has no hierarchy, no admin, no central point of control or authority. On the contrary, the network is governed from the bottom up, making it much more robust against many types of attacks.

Or they can just process transaction on a 1st come 1st serve basis and eliminate the need for blocks, just timestamp each transaction :)

This would basically amount to an .csv file with limited write access, which is rather uninteresting in light of the above.


text:
What stops miners from manipulating "target" difficulty in the block header?
I am relatively new to understanding the fundamentals of the Bitcoin network, and I couldn't find the answer to my question.
I understand how mining difficulty/target is decided and calculated in the block header. (hashrate of previous 2015 blocks...)
Since the larger the target is, the easier it is to mine the new block,
what stops a miner from cheating by giving himself a large target, so that he can mine this new block very fast? Does every node check the target value when they are validating a new block?
Thank you

The difficulty target in the block header has to match the currently required difficulty. This value is derived and checked by each node individually from the timestamps and difficulty statements of previous blocks. Then, the block hash of a new block has to adhere to this given difficulty target. If either of the two requirements is not met, other nodes will reject a block as invalid.
So, if a miner ignored either of the rules, they'd of course be able to do that, but then their blocks are simply not accepted by the Bitcoin network.

Thank you! I was thinking it should be checked by other nodes also

Every node enforces all rules independently, otherwise they'd have to trust other nodes to provide correct information to them.


text:
In Electrum, how does one watch/poll a wallet for an incoming payments to a certain address via API?
I want to create a simple way to accepting bitcoins on my own, using Electrum. I don't except big volume. I suppose, I'd have at most a handful of incoming payments in the beginning.
That is, I generate an address, a new one for each checkout where Bitcoin is selected as a payment method. And then I'll need to watch an address, via API of Electrum, for an incoming payment of a certain amount. The 1st confirmation, at least, which will have to arrive within, I suppose, 1 hour.
How can it be implemented? I don't need the code, but a high level explanation: what API to use, potential caveats and suggestions in general.
P.S.
I'm aware of this - https://electrum.readthedocs.io/en/latest/merchant.html , but it won't work for me. I'm a developer, therefore I can create a simpler solution and which I also could customise however I like.

I have worked with Electrum 3.3.8 daemon through JSONRPC interface.
A vary simple explanation how it woks in my case:

I send to Electrum "addrequest" command which contains also amount of BTC, memo, timeout.
Get the address of payment in answer from Electrum and display it for payer person.
Store address in database for further checking.
Periodically checks (by cronjob) statuses of addresses by
"listrequests" command.

If status is "Paid" and there are more than 1 confirmations, then payment accepted.

listrequests isn't it, because I won't request a payment from someone, but I'll instead give an address to him. Therefore listrequests won't work

When you send "addrequest" comand, Electrum provide to you an address and Electrum starts to monitor this address.

Ты делал через addrequest, но я делаю по-другому. Что я и описал. addrequest мне не подходит

The easiest way to accomplish this would be to use a hosted service like Coinbase Commerce and leverage their API. Commerce generates a wallet that only you control and then uses your public keys to generate and monitor addresses on your behalf. Every customer is given a unique address and the service monitors the network for payments to that address. When a payment is detected, it sends a webhook notifying you of the unconfirmed transaction and for each subsequent confirmation.

Are you unable to read?


text:
How much time do miners spend validation?
Before mining, miners have to validate previous block and transactions which exist in mempool. Then, they pick up some validated transactions from pool and start mining on top of validated block using its hash.
But, they are competing with each other and really busy I guess.
Here is my question. How much time do miners spend validation in general and start mining?

Are you looking for an answer in nanoseconds?

Does this answer your question? Appending txns to merkle tree on PoW efficiency, true hash rate

@RedGrittyBrick
No. I was guessing that miners need to validate manually. Like, go to the Blockchain.com and check the last block and unconfirmed transactions and check by themselves, manually. So, I was imagining that it takes 3 or 4 minutes for them. But from what you are asking me, my guess is completely wrong.

But, they are competing with each other and really busy I guess. Here is my question. How much time do miners spend validation in general and start mining?

None at all. While they are validating new information, they are continuing to build blocks based upon the older information. Miners do not have any reason to wait to start mining except on their very initial startup. But once they are operating, they do not stop to verify information.

First of all let's get a few misconceptions out of the way:

I was guessing that miners need to validate manually. Like, go to the Blockchain.com and check the last block and unconfirmed transactions and check by themselves, manually. So, I was imagining that it takes 3 or 4 minutes for them.

No, miners run full node software, just like you can run yourself, and they query this full node for block candidates to mine. If mining involved going to a centralized website, we could get rid of mining entirely, and just make blockchain.com decide what the next block is. The whole point is to never trust any other party, but be able to perform validation yourself. blockchain.com is a third-party website, operated by a private business, and has in no way a special position in the Bitcoin protocol or its operations. Using the website for anything but debugging purposes means trusting their interpretation of what's going on in the chain. Anyone can run such a website (and in fact, many competitors exist, like mempool.space, blockstream.info (disclaimer: my former employer), blockchair.com, blockcypher.com, ...).
So what this also means: deciding what the next block candidate is going to be isn't done by the mining hardware, but by software. That means one does not interfere with the other - they happen in parallel. The hardware is always mining on something, and the software is always verifying new transactions/blocks as they come in. From time to time, the software will give an updated block candidate to the hardware, based on its own best understanding of what blocks are transactions are available.
Now, how much time is spent on validating blocks and transactions? A significant amount of time, but it's important to point out that most of it isn't actually on the critical path for constructing new blocks. The process, in theory, goes approximately like this:

New transactions are continuously received by the miner's node. These transactions are validated on the fly, and stored in the mempool (a place for unconfirmed transactions; every full node has its own independent one). Typical transactions take less than a millisecond to verify, but larger/more complex ones can exceed this. It also heavily depends on the hardware used, of course.
At every point in time, the mempool is kept consistent with the best block the node knows about. That means that creating a new block template is very fast: just take the top transactions from the mempool (typically, for economical reasons, the ones with the highest fee per byte), as many as fit in a block (1 million bytes before BIP141; 4 million weight since). These transactions are already validated, so they don't need to be validated again at block candidate production time.
From time to time a new block will be received. Typically, all, or almost all, transactions in that block will already be in the mempool when it is received. The P2P protocol is even optimized for this (see BIP152 compact blocks), avoiding the need to even repeat those transactions when transmitting the block. Because of this caching, block acceptance is very fast, but occasionally it may be slow if lots of unknown transactions were included in the block, for whatever reason. In this case, it make take 100s of milliseconds, or even multiple seconds on slower hardware, to validate a block.
When a new block has been received and validated, the hardware must be informed as quickly as possible about a new candidate on top of this. Every millisecond before this is done means wasting the hardware's resources: it's working on a candidate on top of the previous block, which is unlikely to be accepted by the network (nodes typically only accept the earlier one of two competing blocks, unless the later one is extended first). In order to speed this up, some miners will in fact make their hardware switch over to working on an empty block on top of the newly received block, as soon as it arrives, and before waiting for it being validated. This is risky: as they don't know whether the received block is valid yet, this may mean switching over to working on a candidate that builds on top of an invalid block, which would be a waste of time itself. Around the time of BIP66's activation, this caused an entire chain of invalid blocks to be built, because miners started building on top of a block produced by an unupgraded other miner, without knowing it was invalid.

Most of this the same for every node. Non-miners obviously don't spend time on building candidate blocks, but apart from that, the effort spent on validating blocks and transactions is the same for everyone running a fully-validating node, miners and non-miners alike.

I believe your question points to the Verifier's Dilemma.
Basically, validating all transactions is beneficial for the system but not doing it or doing it partially might provide an advantage to the miner that chooses to skip it.
For Bitcoin it's close to a non issue as the cost of verifying transactions is negligible and the overall reward for doing so benefits all (including the miner)
Section 2.3 from the paper I linked :

Verifying a transaction at both steps requires negligible
additional work compared to mining a new block. For
example, the Bitcoin transaction in Figure 1 has only
4 opcodes. Miners can do this extra check without
expending any significant computational work. The
cost of validation outweighs the miner’s risk of having
their earned Bitcoins in new blocks be discarded, if the
invalidity is detected in the future.

For more complex transactions (smart contracts) other mechanisms need to be setup in order to incentivize execution / validation. Ethereum partially solves this with gas but vulnerabilities are still present (See Section 3 of the paper).
While there is incentive to skip the validation, there is also indirect incentive to abide by the rules :

common good of the system / blockchain validity
Fee / block reward

If the blockchain holds many invalid transactions / blocks because miners don't validate then the reputation of the system would intuitively go down which in turn would decrease the interest of mining for such system.
The block producer gets the block and fee reward. Ensuring that your block is valid is in your interest.
I hope that answers your question !


text:
Is there a way to convert a P2PKH address into P2TR?
As far as I understand, Taproot enhances privacy by allowing someone to only reveal a specific key path, which in my understanding means that it's fairly flexible - is it possible to construct a P2TR address that'd be spendable by someone with (knowledge about the possibility and) "access" to an existing P2PKH address?

Please don't vandalize your posts. By posting on the Stack Exchange (SE) network, you've granted a non-revocable right, under the CC BY-SA license (https://creativecommons.org/licenses/by-sa/4.0), for SE to distribute the content (regardless of your future choices). The question prompted an answer that may be interesting for other users in the future and removing the question makes it harder for those users to understand the context.

Short answer: no, at least not via the key path (and using a script path would be less efficient than using P2WPKH instead)
Longer answer: no, and even if it were possible, this would be a terrible idea. The address a receiver gives you is how they expect you to pay them. Giving them money by paying in a different way is not something you should expect them to honor. Specifically, their wallet software may not support taproot, making the money unspendable for them. Even worse, their wallet could include a hardware security module or other security mechanism that can't just easily be upgraded to support taproot.
If they want you to send to a taproot address, they'll give you one.

Aren't you supposed to be able use an arbitrary redeem script? Surely there's a way to P2TR wrap P2SH-like script (which itself can wrap P2PKH).

Yes, but that'd be very inefficient. You can't construct a P2TR address that uses the key path (as you need the X coordinate of the public key, not just its 160-bit hash) given just a P2(W)PKH address.

Well, yes, a receiver could craft an arbitrary script and communicate that script's address representation in order to get paid. The sender unilaterally making unsupported assumptions how they may change the instructions to pay the receiver will certainly lead to conflict, if not outright lost funds.--Imagine you're running a restaurant in the US and when you bring the bill, the customer pays in Euros instead. Would you accept that?

The analogy I've used in the past (when this question came up about converting P2PKH to P2WPKH) was that it's like someone "paying" you by at night coming to dig a hole in your lawn and bury an enveloppe with money there. Sure, the money is there, on the receiver's property. But the receiver might not even notice it's there unless you tell them. Similarly, with such converted addresses, the receiver likely won't even recognize it as a payment to them.


text:
Can you use un-tweaked public key with P2TR?
Given some public key, is turning it into a working P2TR address and then going back from P2TR to get the same pubkey possible? I've read about the tweaking but don't really get it - do you need to tweak the pubkey?

It depends on what you mean by public key.
P2TR outputs encode an "x-only pubkey", that is a public key with just the X coordinate. The Y coordinate is implicitly the even one (on elliptic curve like secp256k1, every X coordinate has either exactly 2 corresponding Y coordinates, or none; and one of those two will always be odd, and one will be even).
Older mechanisms, such as the ECDSA signing used in older Bitcoin scripts, use a public key where the Y coordinate is identified exactly. Either by sending it in full, or sending just its parity.
So, if you would use a public key directly, without tweaking, as a P2TR address, then it is indeed possible to recover the public key's X coordinate from it. Which of the two Y coordinates it was will be lost.
Is this safe to do? In simple cases, yes. BIP341 recommends always tweaking, even when there are no scripts involved, because of interaction with certain other protocols that could be built on top. But if all you're going for is single-key signing, you could in theory get away with using keys untweaked.


text:
How is data in a blockchain stored? Structure of block and blockchain
I'm trying to build a blockchain-based application and I have a problem with understanding some crucial issues.
What exactly is a block? I mean, I've seen application where blocks were just String structures and a blockchain was an ArrayList of these String structures. But my question is, does it work like that in real blockchains? Are blockchains just ArrayLists or maybe they are stored in some files that are distributed among a P2P network?

You can parse the raw database which is stored in blk*****.dat files with blockchain parser and see the structure of blockchain data, so it would be the most fastest way to learn how it works.

How is data in a blockchain stored

So far as I know, the Bitcoin specifications do not mandate any particular structures for storing the Bitcoin transaction-journal (blockchain) in persistent storage (e.g. Hard disk).
Any Bitcoin node is free to store the transaction-journal in whatever form is convenient. For example it might be decomposed as a set of tables in third-normal form in a relational database in "uncooked" storage outside any filesystem. The node might include in the stored data a complex set of indices for fast retrieval of certain types of data (e.g. Bitcoin-addresses).
Bitcoin Core stores the transaction-journal in groups of consecutive blocks in a series of large files whose format I believe closely matches the form used in the Bitcoin network protocols. But there is no prohibition on other software products doing something quite different.

Are blockchains just ArrayLists

That is just one way of representing the data. Note that the ArrayList is a feature of specific programming languages. It may not exist in some languages and quite different data structures may be more appropriate in that case.
The data itself needs to keep track of blocks and those blocks contain data that references data within other blocks. Hence the name. But there are many different ways of implementing these ideas which could all work in a Bitcoin node.

maybe they are stored in some files that are distributed among a P2P network?

Normally, in my field of work, the term "distributed" implies that not all nodes hold all the data. It implies that there is some planned and coordinated division of data among nodes so that specific nodes are designated to be responsible for specific subsets. This isn't really a fundamental design feature of Bitcoin.
I believe that initially Bitcoin used a model that I would more strongly associate with the word "replicated" than "distributed". Initially all nodes each had a full and complete copy of the entire data-set. There was no system to coordinate the intentional distribution of subsets of data among cooperating storage nodes.
Currently nodes, even full nodes, can discard blocks that are no longer of interest to them. This means that many nodes do not contain a complete set of data. Importantly however, any distribution of the data between nodes is essentially unplanned and uncoordinated.
You could argue that there is distribution of data between SPV clients and the servers they rely on for transaction-journal data. However I'm moderately sure this isn't really the sort of distribution of data that database engineers generally mean when they use the term "distributed".

See also descriptions of Bitcoin Core's storage format in these answers:

What is the database for?
What are the keys used in the blockchain levelDB

Thanks, you helped me a lot. But I still have some doubts in my specific situation. I have a website with several forms, that every single form is treated like a transaction. When all forms (transactions) are filled in, a new block containing data from the forms is created. Can you recommend me a good way to store this data?

@mmaciek: Firstly, this isn't the place for that. See area51.stackexchange.com to find places that discuss this kind of software design question. Secondly it depends on application-architecture, platform, toolset, throughput, volume, response-time requirements, skillset, resources, budget and many other things.

At its core, a blockchain is a linked list with a twist.
In a normal linked list, each item points to the previous item (these are the "links" which allow us to traverse back through the list.) In a computer, these links are references to memory locations, but in a blockchain, they are references to the hashes of preceding blocks.
Most importantly, the block of data that is hashed to produce a blockhash includes within it the blockhash of the previous block. You can't change the previous blockhash inside of the current block without changing the hash of the current block... and this means you obviously can't replace a block in the chain without visibly breaking the chain of blockhashes that come after.
This is the critical difference between a linked list (or ArrayList) and a blockchain:

in a linked list, it's easy to replace any element by modifying two
elements, it and the element after it, and the change is not visible
in later parts of the chain
in a blockchain, to replace a single
element, you must replace all elements after it, corrupting the whole
chain in an obvious visible way

Now, in most blockchain implementation, including bitcoin, what I said above pertains only to block headers, not to whole data blocks. For efficiency, the block headers comprise the cryptographically chained list, and the data is stored in separate chunks. So how are they related?
Each block header also includes within it a hash of one data block. This way, if you're looking at block 2045 of data and want to authenticate it, you merely have to hash it and see if that hash matches the hash in block header 2045. Nobody can give you a false block of data without corrupting the hash and, as noted above, they can't change the hash in the header without corrupting the whole chain of headers.
That's the typically used blockchain structure:

A chain of block headers, each of which includes the hash of the
preceding block and also includes the hash of one data block.
A matching collection of data blocks, each of which has a hash that is
included in one of the block headers.

(You could dispense with the block header/data block separation, but it can make lots of things more efficient to separate them.)
If you have the first block (sometimes called a "genesis block") and the validation rules (including what hashes are used and how they made), you can validate everything else yourself, without having to trust some other system to give you honest data.
If designing your own blockchain, you obviously have a lot of leeway as to what goes into the data blocks, how they are stored and distributed, etc.  The choice of networking and storage protocols are not what make it a blockchain.
So, back to your original question: what is a block? It's just a chunk of data.
How is it stored? However you like, but for Bitcoin, they are just chunks of data disseminated via a P2P network of nodes, and many nodes will store those chunks in a database on disk. In fact, there are several bitcoin implementations which store them in different ways! But they can all agree on what's valid, because of the blockchain.

If you are an engineer you can have a look at this code here: https://github.com/mrqc/bitcoin-blk-file-reader
It shows how to parse the blk files.

Are blockchains just ArrayLists or maybe they are stored in some files that are distributed among p2p network.

Yes. The blockchain is stored in files that are shared in every full node of the network. Each full node has a full copy of the blockchain (files). Each file contains a certain amount of blocks.
Each block contains a header and a list of transactions. In the header, you can find information about that specific block, as well as which block it is linked to. Transactions are the atomic unit of the blockchain, and they contain information such as what Bitcoins are being spent, how they are being spent, and to where they are being sent.
If you really want to go into details, just go to the wiki page of Bitcoin:
For block headers:
https://en.bitcoin.it/wiki/Protocol_documentation#Block_Headers
for transactions:
https://en.bitcoin.it/wiki/Transaction
Also, check out this guy's work. He explains very well the Bitcoin blockchain, and how to parse the blockchain into a database to create a block explorer.
https://learnmeabitcoin.com/technical/

Full nodes can be pruned, in which case only recent blocks are stored (old ones are still downloaded, verified, and processed, but thrown away after some time). The "full" aspect refers to validation.

Thanks man. You always give me great inputs and updates. I still have a lot to learn.

Ok I almost got it, but one more thing. Every block is an independant file and all these files have some kind of reference to the previous file (block), or there is one blockchain file which is updated every time when a new block is added?

A single file contains many blocks because blocks have a size limit of 1 MB, and the files have a limit of 128MiB. You can find these files in a folder called 'blocks', and they are formatted like this:  blkxxxxx.dat (xxxxx is the 5 digit consecutive index of the file). Every block has an id which is calculated through the information in the block, and part of that information is the id of the previous block. In this way, the block knows after what block it comes in the blockchain.

I don't know exactly how the latest blocks interact with the blk.dat file being created right at the moment, but I guess it works similar to how rotating-log-files work. But again, I don't really know the exact mechanics of it.

block size after segwit soft fork allowed blocks to be up to 4 MB. Check out this new question: https://bitcoin.stackexchange.com/questions/98810/whats-the-blocksize-limit-after-segwit-and-how-do-legacy-nodes-deal-with-segwit


text:
What are some affordable options for running a full node?
I would like to start running a full node so I can start building bitcoin/lightning applications, but my laptop doesn't have the storage to do so. Does anyone have any suggestions on how I can affordably run a full node? I was thinking of running it on a remote server, but I'm also not sure of what I should look for in a service.

This would be off topic here as recommendations for this stuff tend to go out of date fairly quickly, but you should be able to run a Bitcoin node with a small VPS, storage is usually the most expensive part - Most entry level offerings + additional storage will suffice.

@RaghavSood thank you for your response, I'll have to take a look at different VPS options

There are 1TB SSD's on Amazon in the $150 price range. The latest Raspberry Pi 4 has 4GB RAM ($60). I'm currently running bitcoind AND lnd on such a configuration with zero issues.

Please do NOT run bitcoind on a remote server (I'm assuming cloud). This defeats the whole purpose of a decentralized network, not to mention privacy of being able to check your balances or broadcasting your transactions. Your cloud provider can always keep your ssh logs and any activity you perform on that server.
A better cheaper solution (other than pruned nodes, which I would generally refrain from recommending as it prevents you from validating the whole chain) would be something along the lines of Raspberry Pi as also suggested by others. You can follow the instructions here:
https://github.com/rootzoll/raspiblitz
If you are not too tech savvy, you could buy a preconfigured Raspberry Pi with raspiblitz
https://thecryptocloak.com/product/build-a-node/

You can reduce storage requirements by running Bitcoin-core in pruning mode

How can I run bitcoind in pruning mode?

If I'm running bitcoind in prune mode, do you know if I will still be able to run lightning on top of it?

@MatthewCruz See running lightning in prune mode

You can run it locally in prune mode like @redgrittybrick suggested, or you can look into renting a cloud server.
For example yesterday I created an AWS EC2 instance with extra storage and I played around with geth to deploy some smart contracts to testnet. You can do the same with bitcoind etc.
You will connect to these remote servers with ssh and execute terminal commands without wasting any hard drive of your local pc.

Note: running a node on a cloud server involves trusting that cloud server in crucial ways, so it is entirely preferable to run a node on your own hardware. Also note: altcoins are strictly off-topic here.

That's a really good point, with your own hardware you have full control over it. Aand got it, my bad!

The cheapest configuration consists from RPi3 or its copy Opange Pi equipped with SATA shield for external HDD. I've used my old HDD which left after laptop upgrade with SSD.
I bought extension shield with metal case so everything is assembled together and the case has a fan.
This hardware is working for almost two years. During first year it had Core + LND, now it serves Core + Elects and fairly stable (for this cheapest configuration).


text:
Why should BIP157 compact filters be processed in-order?
In BIP157, under "Client Operation", we have the following paragraph:

Starting from the first block in the desired range, the client now MAY download the filters. The client SHOULD test that each filter links to its corresponding filter header and ban peers that send incorrect filters. The client MAY download multiple filters at once to increase throughput, though it SHOULD test the filters sequentially. [...]

I'm wondering about the bolded part: why should the client test the filters in sequence? Why can't it test them as it receives them? I'm having a hard time coming up with a good reason to test them sequentially. Anyone knows?

A HD wallet doesn’t know what indexes of a seed to scan for until the ones prior have been used, which can’t be discovered out of order. To allow for out of order scanning the address gap limit would need to be prohibitively large, and might still result in re-scanning filters if the wallet is particularly active.

Hmm, but I was under the impression that the wallet would have to scan up to the gap-limit anyway, in case some of the indexes end up being "skipped", eg. 3 addresses are generated, but only the 3rd ends up being used.

Thinking about this more, it makes sense if what you mean is that we need to scan up to highest_used_index + gap_limit, such that everytime a higher index is used, we scan further. Is that the idea?

Yes, that's exactly the point: the address body is expanded as time progresses and you might not have derived sufficient addresses to find all your transactions if you start with later blocks. On the other hand, if you just set an enormous gap limit, scanning the blocks for matches will take forever.

Especially in the case of systems like BIP157, having a large gap limit just reduces inefficiency as the filters get dirtier unnecessarily. Scanning them only sequentially isn't of particular concern regardless as they tend to be processed rather quickly once received (or cached).


text:
How is BIP 158 wallet rescanning supposed to work with xpub keys?
A light client implementing BIP158 downloads filters and checks its addresses against that filter. If one matches, it downloads the block.
Now, let's say we have an HD wallet (BIP32), and/or an xpub key. How would we decide whether to download the block, based on the filter? Do we have to iterate over all possible derivations/addresses (potentially thousands) for each filter? Is there a more efficient solution?

No, you essentially have to do exactly that: test the block filter for each address that might have gotten used by your wallet. That might not be quite as bad as you seem to think, though.
Deterministic wallet backups usually come with a master secret and the derivation path. If you have both these pieces of information, only one derivation path needs to be searched.
Then, your wallet would usually generate a number of addresses in advance as a form of lookahead. The size of this buffer is often referred to as the gap limit. If any of the wallet's addresses appear in a transaction during the rescan, the buffer is refilled with additional new addresses before continuing to scan from that point on. The underlying assumptions are that addresses are generally used in the order they are generated, and that personal use of a wallet does not generate a lot of unused addresses. Therefore, light client wallets often employ a fairly small gap limit by default.
Some wallet backups additionally store the creation date of the wallet, which allows a rescan to ignore prior blocks than the corresponding blockheight.

Ok, so if I get this, given you have the derivation path, eg. m/44/..., you have to check all derivable addresses within that path, up to the gap limit? Or can you stop searching if the first derivable address isn't a hit?

Since the user might give out the first address and then never get paid to it, that would make it likely that you don't find the funds associated with the wallet. You'd want to derive "the first gap limit" addresses in the path and scan the blocks in order for hits.

Gotcha. Re-reading your post makes me understand why it's called a gap-limit, since it's the maximum number of unused addresses between two addresses that are used. Hence every time you get a hit, you look-ahead for another gap-limit addresses.

Yep, you got it! :)


text:
Does every transaction have the same signature?
I want to know whether the bitcoin network generates the same signature for each transaction, because people have only one signature?

No, according to the bitcoin network, the system generates a different digital signature for each transaction. Because the sender generates a digital signature from a transaction message and his private key.
A signature is an intermediary that proves you have a password without seeing the private key due to the math algorithm.

You're correct that every signature is different and that it is generated using the transaction message and private key. However, I don't understand what that's supposed to have to do with allowing users to generate new keys.

@Murch I was just giving extra information. However, now I updated my answer. Thanks for mentioning.

This is much better, but equating a private key to a password is not quite right. How about "A signature proves that you have the private key without revealing anything about the key itself."?

Bitcoin uses the Elliptic Curve Digital Signature Algorithm (ECDSA). While people in the physical world use their unique signature to authenticate when accepting documents or contracts, digital signatures are cryptographic commitments by the owner of a private key corresponding to one specific message (which in this case is the Bitcoin transaction).
As the inputs to the signature generation algorithm are the message and the private key, different messages produce different signatures.

What's the format of the Bitcoin transaction in the Signature that everyone keeps referring to as a message?
Also, aren't some high entropy numbers used in a signature, to keep them different even for the same message from the same address?
Thank you.

Yes, the signature generation additionally requires rolling a random number, the nonce value k which is used to derive the r-value. The signature does not include the message, but it commits to the sighash which is a digest of the whole transaction.


text:
Bitcoin Core Desktop Wallet retaining my coins due to Gap Limit being exceeded
trying to send coins am unable to enter address. have to load from clipboard, cannot copy and paste, address bar goes red, disables send functions. i explore reasons, seems 20 unused address gap limit is probably the reason. researched solutions. only feasible solution seems is sending funds to unused addresses, did so with 11 addresses to no effect -  i am still almost 80 over. looks like i exceeded 20 over a year ago. money sat in desktop wallet unusable for a week now. Block chain is up to date. cannot see any solutions here or anywhere else. 1. can anyone offer a solution? i am on Ubuntu  - maybe there is a terminal gap limit extension solution i am unaware of? 2. how is this possible that an automated system can withhold my own money from me over a technicality which isn't ever flagged to users? is this even legal? will it ever release my funds if i can't sort this problem out and do i have redress if not? seems a ludicrous situation and design flaw. any suggestions or help would be appreciated, thanks.

Bitcoin Core uses a gap limit of 1000 addresses, I very much doubt this is the problem. Can you provide the first few characters of the address you're trying to send to? Specifically, does it start with "1", "3", "bc1q", or something else?

thanks for the 1000 gap limit info, i wasn't aware of that. seems the problem is fixed now - was at the other end of the transaction, not my end. bitcoin core is still working fine. thanks for your help anyway :-)


text:
Bitcoins purchased in 2013
I purchased Bitcoin in either 2012 or 2013 and at the time noted down 2 sets of 34-character addresses each with a 12 character password, both 34 character strings start with a "1".
I'm not holding out much hope that I can retrieve them, but can anyone explain what these details might be.
I can't remember logging into anything at the time.

When you write "12 character password" did you actually mean a 12 word passphrase?

I'm afraid that if you only have the addresses and not the corresponding private keys you're out of luck here.
To recover a wallet or an address you need to have either the wallet.dat or the private key to a Bitcoin address.
The password you made back then was likely used to encrypt the wallet.dat.


text:
How do I submit hashes to a stratum URL?
I have block hashes mined from bitcoind. I want to submit it to a pool. How do I do that?

If you are mining with bitcoind then you are your own pool. If you find a valid block then you have to submit it to bitcoind. You got the block template with bitcoind's "getblocktemplate" and now you submit the complete block with bitcoind's "submitblock".
If you want to mine in someone else's pool then you use the stratum protocol to get the work and then you submit the work results back with the stratum protocol. You don't need bitcoind when you are mining in someone else's pool.
You can't mix the two. You can't work in your own pool and send the block to some other pools to get paid there. They will not pay you for working on your own blocks in your own pool.


text:
How does miner order transactions?
I understand that miners can do whatever they are pleased to do. Either include 0 transaction or 1000 transactions in the block before mining. But I assume majority of the nodes run the same set of code, Bitcoin client. So, I assume there has to be some logic implemented in the code.

How does miner order transactions if two are unrelated? e.g., A pays C, X pays Y. I assume changing the order in the block will change the hash, so I am curious how Bitcoin code combines them in which order.

How does Bitcoin client code decide when to start the nonce + mining process? I understand it has some kind of batch process, so it doesn't start the mining until it can assemble some transactions into a block. But, where's the code to assemble those? Some say miner does whatever they want, but at the end of the day, nobody writes their own client code. There has to be some common code that implements this logic.

The sequence of blocks is critical (and accommodated as each block is chained to a specific prior block), but the sequence of transactions within a single block is not

But still don't understand what do we mean by order within a block doesn't matter. Do we compute hash based on the set of data after sorting?

How does miner order transactions if two are unrelated? e.g., A pays C, X pays Y. I assume changing the order in the block will change the hash, so I am curious how Bitcoin code combines them in which order.

Transactions can be ordered in a block in any way a miner wishes to so long as a transaction which spends an output created by another transaction in that block is placed after its parent. This order does matter for the block hash, and so once the order is decided, it is fixed for that block.
Many miners use Bitcoin Core's block template functionality to build their blocks. This will choose the transactions to include, and an order to include them in. The way Bitcoin Core chooses transactions is to first group related transactions together as packages, then order packages by the feerate paid by the package. Then packages are simply selected in order from highest to lowest until the block weight maximum is reached. (The actual selection is a bit more complicated than that, but this is the general idea of what it does.)
The result is that many blocks simply have their transactions in decreasing feerate order, with transactions that are dependent on each other being placed in order consecutively.

Do we compute hash based on the set of data after sorting?

The merkle root is computed with transactions in the order they are found in the block. So once an order is chosen, it is the only correct order for that block. Changing the order will change the merkle root and thus the block hash.

How does Bitcoin client code decide when to start the nonce + mining process? I understand it has some kind of batch process, so it doesn't start the mining until it can assemble some transactions into a block. But, where's the code to assemble those? Some say miner does whatever they want, but at the end of the day, nobody writes their own client code. There has to be some common code that implements this logic.

The actual miner and the node (that receives blocks and transactions and builds the block template) are separate software. The node, once it is synced, can build a block template at any time. Depending on how long that node has been online, the template it creates may not be ideal, but it is still usable. There is no requirement that blocks must have more than 1 transaction, so it is perfectly fine for a node to be creating block templates even when it is not aware of any unconfirmed transactions.
So a solo miner might just start their node, wait for it to sync, then start their mining software. There is not any smart determination of when to start mining. Miners generally want to start as soon as possible, as any block reward (starting immediately) is better than no block reward (waiting for transactions).

Hi Andrew sorry for digging this out, but I have a question. You said miners are most likely to start mining as soon as possible even if block is empty. But doesn't that mean each time new transaction is added into a block they have to start over? Because hash will change with even slightest change to a block. Im assuming every time someone is broadcasting a transaction to them they have to include it in their block and thus in their proof of work?

@gosu94 No, there is no requirement that a miner must try to include every new transaction. Miners can very quickly exhaust a nonce range so there isn't really any need to be interrupted every time a new transaction comes in. They'll get those transactions the next time they ask for new work which won't be very long afterwards anyways.


text:
How does the Bitcoin p2p network work? Does it need port forwarding?
But does every peer node in the bitcoin p2p network need to have ports forwarded in order to participate? Otherwise, how does the p2p network work in the Bitcoin network?
I heard that there are some technologies for implementing peer-to-peer networks such as UPnP and Hole Punching, does Bitcoin use any of these technologies for its p2p network?

Possible duplicate of What is the NAT traversal technique used by bitcoin

Nodes that participate in the Bitcoin network run a computer program such as this one: https://github.com/bitcoin/bitcoin, which dictates how everything works.
To answer your more specific questions:

Yes, bitcoin once used UPnP, here are some more details: https://dirkmittler.homeip.net/blog/archives/3340 but in modern versions it does not (thanks for specifying this the comment section).
Bitcoin uses TCP, and hole punching over it is not reliable.. Here's another place where this is briefly touched upon Can exist bitcoin network with all nodes behind NAT?

No modern versions include UPnP.


text:
Why is the data format for block height in coinbase scriptSigs inconsistent for heights 1-16?
BIP-34 claims that in coinbase transactions, the scriptSig should contain the block height. It says the format of this is "serialized CScript", i.e. a byte indicating the length of the number (n), and then n bytes in little-endian format.
This works if you look at e.g. the most recent block as of this writing—block 640037—whose coinbase scriptSig begins with these 4 bytes: 0x0325c409. 0x03 indicates the length, and then we have (0x25=37 * 256^0) + (0xc4=196 * 256^1) + (0x09=9 * 256^2), or 37 + 50176 + 589824 = 640037.
For obvious reasons mainnet and testnet will never need to encode heights 1-16 in their coinbase. But regtest nodes will. When I run a fresh regtest node (v0.19.1) and generate a couple blocks, I get coinbase scriptSigs that look like this:
height 1: 0x510101
height 2: 0x520101
...
height 16: 0x600101

Clearly it's not serialized CScript, at least in the format described in BIP-34. If it were, those first bytes would indicate data lengths of 81, 82, and 96 respectively, clearly ridiculous. The actual encodings of 1, 2, and 16 in serialized CScript are 0x0101, 0x0102, and 0x0110. It looks to me like the format is height+80 followed by 0x0101. And for some reason this only persists for the first 16 blocks, as 17's scriptSig is 0x01110101.
I tried looking through the bitcoin source code, but I don't write much C++ so it's hard to tell what's going on. No documentation—in the source (as far as I can tell) or the PR for BIP-34 or in the BIP itself seems to indicate that heights 1-16 would have a different format from every other height.
Why is this happening?

The actual encodings of the numbers 1-16 are 0x51, 0x52, etc. This is because the numbers 1-16 are special in Bitcoin script and have their own explicit opcodes. OP_1 is 0x51, OP_2 is 0x52, etc. The correct way to write numbers with their own opcodes is to use those opcodes instead of using a pushdata opcode followed by the number.


text:
Is block reorg works in this way?
i need an expert to clarify is my understanding correct to block reorganization. I can't find answer can persuade me, sound like only i dunno what is block reorganization :(.
say my bitcoin network has total 10 nodes, and 10 of 10 are all miners
say i had successfully mined a block#10, and that is another competitor also mined a same height of block at the same time and then i shut down my node.
after few days, i restart my node, so now what event will be trigger in my bitcoin program? i imagine possible steps would be like this. below "I" refers to my bitcoin node.

i found my largest chainwork is block#10 with hash 0000123.

i broadcast getheader message to all my peers, with max query blocks = my block height + 100 to grab next 100 block headers after my block.

say 5 of my peers return me result. i wonder now is bitcoin performs PBFT on returned result? say at least 3 peers return me same result, then i assume that this result is correct.

ok, now result return from point 3 told me that there no longer got block added after my blockhash, i know my block has been orphaned.

next, i lower my block height to 9, then i ask my peers again, and peers response me yes that is block#10 which hash is 0000456 and prevhash same to my hash in block#9.

before start to download all blocks start from hash 0000456, i need orphan my block#10, so i remove it from chain and revert the transaction to mempool.

and now i feel safe to keep download until meet most chainwork block.

i need your clarification. thanks.

Similar question answered here: https://bitcoin.stackexchange.com/a/98681/3667

thanks, it is very informative

Your understanding is basically correct. Keep in mind two things:
First, chain length isn't really the correct metric but accumulated chainwork. For practical discussion we can imagine for now they are the same thing.
Second: the entire purpose of Bitcoin full node software is to find the chain with most accumulated work. The whole meaning of its life is to discover that chain and synchronize its database with that chain (specifically, the UTXO set aka list of unspent coins aka "the ledger").
So it does not matter who mined what block. As long as it is running, the full node will try to stay in sync with its peers by sending locator messages (getheaders) and responding to new block announcements.
When a full node receives a valid block header that has more chainwork than its own tip, it attempts to download whatever blocks are necessary to verify that chain. If those blocks are valid and require "rewinding" the current chain, the node will do so, iteratively.
Consider this set of chain tips:
A-B-C-D-E-F
       \
        G-H-I

If your node synced blocks A through F then some time later learned about block header I, it will first download and verify headers G through I, then if they are valid download blocks G through I. If those blocks are valid, the node will disconnect F from its own chain, then disconnect E, then connect blocks G through I from the common ancestor D.
To address your points specifically:

You don't really "request +100 blocks" what you do is tell your peers about your own chain tip and ask them if they know about any chains with greater work.

"orphaned blocks" are blocks with no parent and this does not really ever happen on Bitcoin any more. I think you mean "stale blocks" here, but even then if peers do not respond with a greater-work chain, your node does nothing and assumes it is on the correct chain tip.

You would never "lower" your chain unless you were doing a reorg as I described above. In fact, if there were two blocks at height 10 you still wouldn't do anything until one of those chains got extended to height 11. THEN -- you become aware there is a greater-work chain out there and your node will attempt to download and verify it.


text:
How is the UTXOs of each partially-spent TxIn determined?
My current understanding of a transaction is as follows:

a transaction (Tx) can have multiple inputs (which are previous unspent transactions = UTXOs), let's call them the TxIn
a Tx can have multiple outputs (which thus become UTXOs before they will themselves become spent by being used as TxIn), let's call them the TxOut
the TxOut have a specific order N in the transaction: TxOut[0], TxOut[1], etc.
each TxIn references the number N which corresponds to the index of the TxOut referred to in point 3 above, so this makes the link between a specific TxIn and a specific TxOut
each TxOut specifies a specific bitcoin amount (which represents the UTXO amount that can be later spent)
each TxOut references one or multiple (in the case of MultiSig) bitcoin addresses (public keys) that can spend the amount
however, and this is what I don't understand, the TxIn does NOT specify the amount of bitcoins that are used

If the TxIn does not specify the amount that is used, how do we determine the spent amount for each TxIn, and thus the UTXO of the part of the TxIn that remains unspent?
Example, a transaction with 2 inputs and one output (assuming no fees):

TxIn[0] whose UTXO before spending is 1 BTC
TxIn[1] whose UTXO before spending is 1 BTC
TxOut[0] whose value is 1.5 BTC

How do we know which TxIn, and how much of which, was spent?

was it TxIn[0] for 0.75 and TxIn[1] for 0.75?
was it TxIn[0] for 0.5 and TxIn[1] for 1?

What am I missing? Because given the above, it is not possible to determine the spent proportion of each input.

a transaction (Tx) can have multiple inputs (which are previous unspent transactions = UTXOs), let's call them the TxIn

More accurate would be to say "unspent outputs from previous transactions".

a Tx can have multiple outputs (which thus become UTXOs before they will themselves become spent by being used as TxIn), let's call them
the TxOut
the TxOut have a specific order N in the transaction: TxOut[0], TxOut[1], etc.
each TxIn references the number N which corresponds to the index of the TxOut referred to in point 3 above, so this makes the link between
a specific TxIn and a specific TxOut

Correct, we call the compound of transaction id and output index txid:vout the "outpoint" of a UTXO. As you describe, each input spends exactly one specific UTXO.

each TxOut specifies a specific bitcoin amount (which represents the UTXO amount that can be later spent)
each TxOut references one or multiple (in the case of MultiSig) bitcoin addresses (public keys) that can spend the amount

It's a bit easier if you think of each output as being locked to a set of conditions that need to be fulfilled for spending. In single-sig the spending condition is "pay-to-(witness-)public-key-hash". This lock is satisfied by a signature of the private key corresponding to the specified address. (The address in turn is derived from the hash of a public key which forms an asymmetric key pair with said private key.) In multisig the spending condition is "pay-to-(witness-)script-hash". The funds are locked to the fullfilment of a specific script. The address in this case is derived from the hash of the script. Spending such a UTXO requires 1) revealing the script that hashed to the address, 2) fulfilling the conditions specified in the script. In the case where that script is a multisig construction, it would specify a set of public keys some subset of which is sufficient to unlock the funds. However, these scripts can take various other forms as well. I'd nit that "each TxOut references one or multiple bitcoin addresses that can spend the amount" is both an overgeneralization (because not all scripts must be multisig) and an inaccuracy (because the funds are locked to the script, not to multiple addresses).

however, and this is what I don't understand, the TxIn does NOT specify the amount of bitcoins that are used

If the TxIn does not specify the amount that is used, how do we
determine the spent amount for each TxIn, and thus the UTXO of the
part of the TxIn that remains unspent?

There is no way to partially spend a UTXO. Spending UTXO is binary, either they are unspent or they are spent. Since UTXO can only be spent in full, any remainder from a payment gets collected into one or multiple outputs in which the sender pays themselves (see e.g. How does change work in a bitcoin transaction?).

Example, a transaction with 2 inputs and one output (assuming no
fees):

TxIn[0] whose UTXO before spending is 1 BTC
TxIn[1] whose UTXO before spending is 1 BTC
TxOut[0] whose value is 1.5 BTC

How do we know which TxIn, and how much of which, was spent?

was it TxIn[0] for 0.75 and TxIn[1] for 0.75?
was it TxIn[0] for 0.5 and TxIn[1] for 1?

What am I missing? Because given the above, it is not possible to
determine the spent proportion of each input.

The answer is that both TxIn[0] and TxIn[1] get spent completely and the transaction is created with two outputs: TxOut[0]: 1.5 BTC to recipient and TxOut[1]: 0.5 BTC to sender.

Ah thank you for your detailed explanations. One of the pieces I was missing was that a UTXO cannot be partially spent, and that the remainder gets 'sent back to oneself' in a vout. Makes much more sense now.


text:
What are the chances of hitting VALID mnemonic phrase out of bip39, if someone picks 12, 18, or 24 words from the word list randomly by himself?
So, here is the statement: "Not every combination of words from bip39 wordlist is valid."
What are the chances of hitting a VALID mnemonic phrase out of bip39, if someone picks 12, 18 and 24 words from the word list randomly by himself?

Hi crypto enthusiast, interesting question. It might help if you add a source for the statement so that answerers can investigate the context in which it was made.

This question does not need any more context.

The checksum is 1 bit for every 3 words so 1/16 for 12 words, 1/64 for 18 words and 1/256 for 24 words

from https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki:

The following table describes the relation between the initial entropy length (ENT), the checksum length (CS), and the length of the
generated mnemonic sentence (MS) in words.

 CS = ENT / 32 MS = (ENT + CS) / 11

|  ENT  | CS | ENT+CS |  MS  |
+-------+----+--------+------+
|  128  |  4 |   132  |  12  |
|  160  |  5 |   165  |  15  |
|  192  |  6 |   198  |  18  |
|  224  |  7 |   231  |  21  |
|  256  |  8 |   264  |  24  |

so for 12, 18, 24 words, the entropy of the checksum is 4, 6, 8
or 1 in 16, 64, 256

Any private key in existence, I am guessing a few million created by now, in 2256, are your chances of guessing someones mnemonic/private key. 2256 is a very, very large number.

If you mean "valid" being the valid checksum word at the end of the phrase,
A valid mnemonic is defined as:

The checksum is then the first n / 32 bits of the SHA256 hash of the entropy. This is just concatenated to the end of the entropy. The mnemonic is then encoded by dividing the entropy into groups of 11 bits and using the resulting 11 bit number as an index into a list of 2048 words

So I would think getting a valid checksum for any arbitrary input would be 1/2048

For 12 words (128 bits) 128/32=4 bits = 1/16; for 18 words (192 bits) 1/64; for 24 words (256 bits) 1/256. The checksum is never 11 bits. (Also: pneumonic means relating to the lungs, or breathing; mnemonic relates to memory.)


text:
OP_SHA256 - Hashing the string representation
I'm building my own Bitcoin script interpreter for learning purposes.
I've compared my results with https://siminchen.github.io/bitcoinIDE/build/editor.html
And the script OP_16 OP_SHA256 which results from:
0x4A44DC15364204A80FE80E9039455CC1608281820FE2B24F1E5233ADE6AF1DD5

That is the hash of the string representation of '10' (16 in hex)
Instead of the hash of the bits
(which is: c555eab45d08845ae9f10d452a99bfcb06f74a50b988fe7e48dd323789b88ee3)
Does the bitcoin script interpreter hash the string representation of any hex value, instead of hashing the raw number?
edit: Is it a normal behavior, to prevent things like SHA256(1) == SHA256(01)?
Thank you all for your answers.

No, the correct result should be c555eab45d08845ae9f10d452a99bfcb06f74a50b988fe7e48dd323789b88ee3.
I think this interpreter is just wrong. Looking at the source code, it has other deviations from the protocol (e.g. OP_MUL, which doesn't exist).

Thanks! I'll avoid that one and find something else to double-check my code.


text:
How do mining rewards get verified?
I understand that a miner who adds a new block can pay the reward to their address per the coinbase transaction, but who verifies that claim? Say, for example, I add a block, but instead of claiming 6.25 bitcoins, I claim 50 bitcoins. Who is going to verify that my claim is wrong? Thanks.

I understand that a miner who adds a new block can pay the reward to their address per the coinbase transaction, but who verifies that claim? Say, for example, I add a block, but instead of claiming 6.25 bitcoins, I claim 50 bitcoins. Who is going to verify that my claim is wrong? Thanks.

You cannot claim 50 bitcoins, it is not possible. There is no way to form a block that claims 50 bitcoins because, under current rules, only 6.25 bitcoins can be claimed. What you would form is an invalid block.
Nobody could possibly conclude that your block claimed 50 bitcoins because no rule allows such a block. So there is no way anyone could ever allow you to pay 50 bitcoins based on that block.
There is simply no way to form such a block as no rule allows such a block. You might just as well form a block that claims 50 bricks of gold and try to get them somehow. Both tasks are equally impossible under current bitcoin rules.
All data exchange is untrusted. Every participant must check every piece of information for compliance with every system rule, otherwise they simply cannot understand it. Something that violates any rule is just ignored by every honest participant who processes it.