text:
SHA256 Implementation Gate Equivalent
I needed help in understanding how to calculate the GE for a SHA256 implementation. I need to calculate the GE of individual functions in the SHA256 hashing algorithm.
Could anyone please direct me to an appropriate paper/book that explains this in detail?

Here's the SHA2 spec: http://csrc.nist.gov/publications/fips/fips180-4/fips-180-4.pdf Are you asking about how to implement this in an FPGA, or how many gates it will take to implement it?

Don't cross-post


text:
The bitcoin mining algorithm from a programmer's viewpoint
This page: Blocks said mining is actually to solve a mathematical problem, but reading Block hashing algorithm doesn't give much help.
I also tried reading bitcoind source code, but reading code takes much more time than reading documentation:)
And I have written a simple json-rpc client to call getwork() method to fetch the "data", but what should I do next to this "data"?
Anyone could explain the mining process in programmer's view?

but what should I do next to this "data"? Here's a short reference implementation. https://github.com/jgarzik/pyminer/blob/master/pyminer.py

Also, explain the mining process in programmer's view is a pretty broad question. What specific problem are you trying to solve?

@NickODell Thanks for your code! Now I understand how to create new blocks. But is each submitted block valid (bitcoin network will send bitcoin to this person)? I found Bitcoin wiki said only 6 blocks will be created every hour, Assume that 100 people get the block header by using getwork at same time with same difficulty, I think much more than 6 blocks will be created by these people. Am I right?

Answering in order: No. No.

possible duplicate of I am unable to figure the getwork api

@NickODell Sorry for my delay, I have something emergent these days. If not every submitted work is valid, how can bitcoin network tell which submitted solution is valid? Thanks

The Mining Algorithm is as follows:

Step 0 - Retrieve the hash of the previous block from the network.
Step 1 - Gather a list of potential transactions known as a "block". This list of transactions comes from the peer-to-peer bitcoin network.
Step 2 - Calculate a hash for a block of potential transactions along with a random number.
Step 3 - If the hash is more than the currently set difficulty level, then you have mined that block.
If not, start over from Step 1. Any additions to the list of transactions from step 1 along with change in the random number from Step 2 mean that there's a chance that the criterion will be met in the next go around.

From a programmer's view, the pseudo code might look something like this:

P := The hash of the previously mined block
B := A block of transactions
H := A hash function
D := Difficulty Level

0 Retreive P
1 Construct/Modify B
2 IF H(P, B, Some Random Number) > D END
3 GOTO 1

I should warn you that there are a few inaccuracies in that description, but for the most part, that should be good enough. And here are a few more useful clarifications:
What's a hash?
A hash is a function that converts data into a number within a certain range. The hash has the property that knowing it's output is essentially unpredictable (within the given range). The specific hash function used for bitcoin mining is SHA256 applied twice.
How does the difficulty level work?
This unpredictable nature of the hash function means that putting in random data (the transaction + the random number) will essentially produce a random number within a certain range. Further restricting the range of the desired output affects how likely one is to find it in a single round. This creates a way to probabilistically determine how often a solution will be found based on the number of times the algorithm can be run on the network. Specifically, when you hear the term "gigahashes" or "terahashes", this refers to the number of times Step 3 can be run. As the number of hashes per second across the entire network grows, the network automatically raises the difficulty such that a solution will be found within about 10 minutes.
What happens when a block is mined?
When a block is mined, the miner sends the block to all other miners on the network as evidence that it has found it. This block contains a list of transactions, the found hash, the specific random number, and a reference to the previous hash. As each miner receives the newly mined block, it removes all transactions that it is currently mining that exist within the block (because they've already been confirmed in the block chain) and broadcasts the block to other miners that do the same thing. The propagation happens pretty quickly.
Note: the original miner of the block gets a "miners' fee", which is a reward consisting of any unspent coins from transactions in addition to a "coinbase" reward. The coinbase reward started out at 50 bitcoins and halves after every 210,000 blocks (about once every 4 years). The coinbase reward will eventually get so small that it will be miniscule compared to miners's fees.

The reward is halved every 210000 blocks.
-> Just saw that now and think it is neat: http://bitcoinclock.com/

Oh, and the Difficulty is adjusted every 2016 blocks by assuming that the network will continue to operate with the average hash rate of said last 2016 blocks and setting the new difficulty such that this hashrate would result in an approximately 10 minute block cycle.

I could be wrong, but I believe that the block also contains the hash from the last block. And that is how blocks are chained backwards, leading to the term blockchain.

@Murch, the difficulty is actually adjusted every block based on how long it took to find the previous block, and at this point, the difficulty can only go up. Every 2016 blocks, the difficulty is adjusted with respect to the last 2016 blocks and it can go either up or down. This helps guard against sudden drops in computing power, which would otherwise result in the time to find a block increasing to more than 10 minutes.

@Tarandeep-Gill, that's correct! The hash of the previous block is hashed along with the list of transactions. This actually makes me think that my explanation of hashing is a bit incomplete -- The specific hash function is not SHA256 applied twice as stated, but rather that function applied to various parts of the block (including transactions and the hash of the previous block) in different ways. This is one of those "inaccuracies" I warned you about. I wonder if you can find the others...

Are we talking about network difficulty or some other difficulty? Because the network difficulty does not change every block, it exclusively changes every 2016 blocks. See for example: https://en.bitcoin.it/wiki/Difficulty#How_often_does_the_network_difficulty_change.3F

I think omitting the hash of the previous block from the explanation is a bad idea even if it makes the answer easier to read. People could be like "Aha" that's where the name blockchain comes from when they read this answer as suggested by @TarandeepGill

@AntonA., I think it's important not to conflate answers that don't necessarily have anything to do with each other. While I agree that this information is useful, the term "blockchain" is not specifically mentioned in the question or my answer. This information my be more useful here: http://ethereum.stackexchange.com/questions/459/what-is-a-blockchain-what-is-the-concept-behind-it

Any hash is a valid hash.  The question is if you're hash meets our criteria.  What you hash is actually a couple of things (we'll come back to this) that lined up together make a string that is a string of a specific length.  You then hash that entire string.
Think of your resulting hash as a number.  What we want is that resulting number to be less than a target number.  So it's like rolling a billion sided die, and coming up with a number under the target number.  That target number is considered "difficulty".  As more people are rolling the die, we lower what that target number is to reduce the likelihood any one die-roll will hit.
So importantly, some of those things that we line up in the string that we hash are things that we're allowed to adjust.  Hashing isn't exactly like rolling the die because if we has the same thing over and over we get the same result over and over.  However even a minor adjustment in what we hash can have a major impact on what the resulting hash is.  The primary item that we can manipulate is called the "nonce".  Basically we pick a nonce to use in the string, hash the whole string, and see what we get.  If the hash isn't a hit, we modify the nonce and try again.  The nonce itself isn't very large so there's a limited number of variations of the nonce that you can try.  The next item we can change is a timestamp.  We're allowed to wiggle the timestamp too.  Every wiggle of the timestamp allows us through the entire set of possible nonce values all over again.  We repeat this process of exhausting the nonce possibilities, then wiggling the timestamp over and over until the resulting hash is lower than the difficulty target number.
When that happens, we then proclaim to the world that this timestamp plus this nonce works to solve the block.  Others validate it to be true and it's added to the block chain.  A block is considered "Validated" once it's a certain number of blocks "deep" in the block chain, meaning it's a historical block compared to the current block.  Validation is a little bit of a misnomer here because it's not that the block isn't already known to be valid.  What we're validating is the proof of work, meaning once that historical block is buried far enough the amount of effort involved to create that history is insurmountable for someone else to try to create a different variation of the history.  They would have to create their own version of that block (the only reason being to write in their own version of the transactions, i.e. steal coins) then solve it themselves, then solve the next block and the one after and so on and "catch up" with everybody else.  This means they would have to out-race the world in the dice rolling game.  Maybe once in the history of the universe someone might get lucky on the nonce with two or three blocks back to back, but with the 120 blocks that most mining pools and exchanges require now?  Not going to happen, ever.

The purpose of solving a 'puzzle' is to (a) delay the mining of the block to avg 10 minutes and (b) to incur real-world costs for mining a block (spending cpu power, thus energy).
The costs are there to prevent a Sybil attack (putting many miner machines at work to do a 51% attack).
The delay is put in to allow a good block to propagate around the globe to all other miners, without giving the miner who just minted the new block a headstart advantage.  For that, the blocktime (10 minutes) needs to be order of magnitudes larger then the propagation delay (a few seconds).
So the kind of puzzle is in a sense irrelevant, it could as well be a giant Sudoku.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Repeatedly sending coins from an address to itself
Let's say I create an address that has a balance of x bitcoins, where x is the minimum transaction amount before prompting a fee. Hypothetically this address is used as a permanent cryptographically secure mechanism that sends itself x coins at certain points which have some significance.
Will miners mine this transaction over and over or will it eventually be ignored by the network?
Will this transaction be readable by parsing the blockchain, or is it handled differently?
Will this transaction ever eventually incur fees and thus never be put into a block (since the wallet will never own more than x BTC)?

Why on earth would I be downvoted for asking this question?

I didn't downvote, but the question is fairly confusing, it's not clear what you're asking and the premises suggest misunderstandings of how Bitcoin works.

I want to send myself x amount of coins continuously and use a blockchain parser to get that data (timestamp, block #, for carbon dating). Will I eventually be unable to send this transaction into the network due to fees or will this work as expected as long as I put in the minimum coin amount to avoid fees?

Or simply, is there ANYTHING AT ALL within the bitcoin protocol (besides pruning in the future) which would pose as a serious problem for this application?

I am really curious what the motivation for such a scheme would be.

It's not a DDoS, simply a way to associate the nth transaction with a timestamp using the blockchain as a proof-of-existence. The transaction will have 1 input and 1 output both concerning the same address. The wallet will contain x Bitcoins where x is the minimum amount necessary to send this transaction to the network every y seconds, where y is the smallest amount of time required by my application. y will probably be in the range of a few days, so I'm assuming the x value will be limited by input_value_in_base_units rather than input_age or size_in_bytes.

Nodes don't check the history of a coin or address to determine whether to include or propagate it, so it will be possible to send coins from an address to itself multiple times.
However, the priority of a transaction depends on how aged its inputs are. If you try to do it too frequently, you'll be using young inputs, lowering the priority and making a fee necessary.

I see. So even if you mine a new block of 25 BTC, and used those 25 BTC to send to yourself every so often, eventually you'd end up paying a fee on that transaction due to the coin age? That makes sense. Is there anywhere I can learn more about the exact timing of that eventuality or do I have to sort through the source code to figure this one out?

@bvpx: There is some information at https://en.bitcoin.it/wiki/Transaction_fees. I'm not sure it's up to date but at least it can give a hint what to look for in the code.

@bvpx: I'd like to reiterate that it's not how many times you do it it matters, it's the interval. With 25 BTC you likely won't run into this problem, but with a smaller amount, you won't need to pay a fee if you make a transaction per day (giving the coin time to age before sending again), but you will need to pay a fee if you make a transaction every 10 minutes.

@bvpx The network doesn't discriminate against coins that are old, only coins that are "new." Coins can become new by being spent.

I see. So there is an equation that I can use to figure out how many coins I need to put into the wallet to achieve no fees at a certain interval. For example, if I wanted to send a message every 30 minutes I could then calculate priority = sum(input_value_in_base_units * input_age)/size_in_bytes and estimate around 1-3 for input_age and measure size_in_bytes to find the values that fit this model.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
How do I uniquely identify a transaction (to protect against malleability)?
Transactions I broadcast can be malleated by anyone watching the mempool (more info here).
This rules out using txid for transaction identification.
How can I identify my transactions to make sure they are tracked by my application, even if the txid changes by the time the transaction is mined in a block?

Tracking the UTXO:s it spends and the outputs it creates.

This is an interesting problem, one that has been studied and discuss within the bitcoin community quite a bit.
The basic way to do this is to keep track of a normalized TXID alongside of the actual TXID used in the protocol. Then to calculate the normalized ID of a transaction, you serialize it:

With the inputs' txids replaced by their normalized transaction
Removing the scriptSigs (as those are the most malleable parts of a transaction)

The only caveat is that that this doesn't need to be done for coinbase transactions, as the scriptSig is not malleable (since it's already in a block) and there are no real txids in the inputs. You also don't want to remove the scriptSig from the coinbase transaction because it contains the height of the block (per BIP34), which guarantees the TXID (and normalized TXID) will be unique. Hence, for coinbase transactions, the normalized transaction id is the same as the transaction id (nTXID == TXID).

Note: this is a lot of work to keep track of normalized IDs for every transaction. It has not been implemented in Bitcoin Core yet, and implementing your own version of this may be time consuming.
If you can guarantee the set of outputs in all transactions within your system will be unique (i.e. you are using new receiving addresses and new change addresses for every transaction), then you might be able to use a hash of the set of outputs as a unique identifier. This would be more specific to your use case, as a temporary fix while Bitcoin Core doesn't yet have support for normalized transaction ids.
Some further resources:

https://github.com/bitcoin/bitcoin/pull/3656
What is the normalized transaction hash of a transaction?
When computing a normalized transaction id, when do I finalize the sha256 hash?

I'm not sure if hashing the outputs would help, since anyone could create a hash of the same outputs to attack my system (it would cost them bitcoin, but not much - my application will always send the minimum amount of BTC). What about hashing the inputs?

@bvpx, very good point. Well, inputs are the parts that are malleable, so it's hard to include the input data. Keep in mind that inputs have the (prevout txid, prevout index, scriptSig, and sequence number). The prevout txid and scriptSig are occasionally malleable, and the other two aren't all that unique. No good option :(

You can still use the txid to identify it in your database. You just have to make sure you wait enough confirmations to trust the transaction, and that it can't be "malleated" by other parties. If you receive 2 transactions which have same inputs/outputs but different transaction ids, then you simply keep one (maybe the first one) and discard the other. In other words, identify transactions however you want, just remember not to trust unconfirmed ones.


text:
what is the primary factor responsible for the increase of block size?
Blocks are made up mostly of transactions.
I understand that the larger the transaction output is in coins, the greater the transaction size must be because it must list all previous transaction inputs that lead up to this transaction and add up to the net output of this specific transaction.
Is there any other factor I might not have accounted for that increases transaction size or block size?

The primary driver of Block size is the number of transactions. Below is the structure of a block and the size of each part in the block.

Magic no, 4 bytes
Blocksize, 4 bytes
Blockheader, 80 bytes
Transaction counter, 1 - 9 bytes
Transactions, -many transactions

The only part of a block with a variable size is the transactions part.
See: https://en.bitcoin.it/wiki/Blocks
If you're exploring the scalability of Bitcoin, see: https://en.bitcoin.it/wiki/Scalability#Storage

thanks for the quick and concise answer. I have one more question: why does the blockchain store "size" metadata in each block? Can't any bitcoind client quickly add up the amount of space taken up by that particular block?

Are you talking about the Blocksize field which is 4 bytes? If you're okay with the answer, don't forget to mark it as a valid answer ;)

Hey one more question: the "size" metadata in each block represents the size in bytes that block measures, correct? Not megabytes or bits or something else?


text:
if a coinbase reward isn't fully spent, can the coins ever be recovered?
If a pool mines a block and is mistakenly rewarding miners with less coins than they should be given, is it possible for others to recover those coins somehow? Or are they lost forever?
For example, if the reward should be 50 BTC and the pool pays everyone a total of 5 (because the script has a bug and everyone's payout is divided by 10), is it possible for the other 45 BTC to be recovered?

I can't imagine there could be a way to recover the coins after the fact. Actually, it could be that a block with such a mismatch would be considered invalid, and the other miners wouldn't build on top of it. In that case the original pool gets nothing at all.

No, the remainder of the reward is lost.
The bitcoin protocol includes a check which rejects a block that claims a bigger reward than 'current block reward + transaction fees'. Thus, inevitably follows that any rewards unclaimed in the coinbase transaction cannot be claimed at a later time.


text:
how exactly was the original coefficient for difficulty determined?
I'm assuming difficulty has a "target" of 10 minutes because the coefficient satoshi chose becomes multiplied by a variable to help the network remain in proportion. IE, if the network hashrate is 2x the previous network hashrate, bnProofOfWorkLimit is multiplied by the previous difficulty and also by 2, which is the measurement of how much faster the network is producing hashes.
This is pretty simple to understand. However, I've been wondering about how the original difficulty was chosen so that the target is 10 minutes.

Is it based on some hard physical limit of computing?

I've come across this code in chainparams.h on line 113:
bnProofOfWorkLimit = CBigNum(~uint256(0) >> 32);

What does ~uint256(0) mean in C++? I know that ~uint256(0) >> 32 means 1 / 2^24 in human-readable terms.
Why is the ~uint256(0) function used instead of simply 256?
Why was this chosen to be expressed as 256 >> 32 rather than 1 >> 24? Is there a good reason for this?

Any direction to learn more about this would be appreciated as well.
Note: I am not asking why the amount of time chosen was equal to 10 minutes, I am asking how the number 1 / 2^24 was arrived at to estimate 10 minutes in terms of how long it takes to calculate a sha256 digest.

I'm not sure it was. When the network started operating (and Satoshi was pretty much the only one mining), blocks weren't found every 10 minutes. For example, the first 2016 blocks were found in 24 days rather than 2 weeks. Normally this would cause the target to go up but it can't go above the hardcoded max target, so only in block 32256 in December 30 2009 we started seeing the retarget mechanism kicking in and blocks arriving every 10 minutes.
However, seeing that it did come out close to 10 minutes, it's possible Satoshi figured out the hashrate of his own machine, and chose the parameter as a round number close to what it would take to find a block every 10 minutes on his hardware.
I think you did not understand this line of code correctly. Hash values are represented as 256-bit unsigned integers. uint256(0) gives the representation of 0 in this data type. ~ is logical not and ~uint256(0) inverts all bits, giving the highest possible integer, 2^256-1. Right shifting by 32 bits gives 2^224-1, so that each hash has a chance of 1/2^32 to be lower than the max target.
The value is then converted to the more compact "bits" form, which also causes it to be rounded to (2^16-1)*2^208.
Doing 256>>32 or 1>>24 would have resulted in 0 since integer types can't hold fractions.

how does a value of 2^224-1 for bnProofOfWorkLimit result in each hash having a chance of 1/2^32 to be lower than the max target?

There are 2^256 different possible hash values, roughly 2^224 of them are lower than the max target, so a uniformly random one has a chance of 2^224 / 2^256 = 1/2^32 chance of being lower than the max target. PS I'm not intimate with the code enough to know exactly what bnProofOfWorkLimit is, but I would guess it translates to the max target as described above.

Got it, I wasn't sure about the maximum possible hash values. What is the significance of a hash being "lower" than the max target anyway? How does this all play into the difficulty equation? It's an incredible challenge to understand how all of this works at the lowest level starting from nothing and just reading the code.

@bvpx: Well, being lower than max target isn't very meaningful, other than being useful in calculations, and being a requirement for a traditional mining pool share. What matters is being lower than the current target. There actually isn't such a thing as "difficulty", the target is computed from the previous target and time to find 2016 blocks, and difficulty is defined as the ratio between the max target and the current target. The first target was of course the max target, and can never go above it.

@bvpx: The thing is, once we know that the probability to be lower than the max target is 1/2^32, and that the difficulty is D, we know that the probability of a hash to be lower than the target and hence a valid block is 1 / (2^32*D). Of course, since the max target is 2^224-2^208 rather than 2^224, this is off by a negligible amount.

I see. So the difficulty couldn't retarget to lower than 1 since it was already at the limit. When more miners joined the network in late 2009, difficulty rose. However, I'm wondering how the 10 minute window was calculated. Is a 1/(2^32) chance to find a block significant in some way, maybe because of the speed of light the transfer of data is limited in hardware so it isn't possible to calculate a hash faster than expected? I'm guessing other alt-coins change the block timer by simply multiplying or dividing this value by whatever they want to approximate a new block timer.

@bvpx: No, 1/2^32 is not significant besides being a round number and giving a result that matched what was desired. From the total network hashrate (which at first was Satoshi's hashrate) it is easy to calculate what is the target needed to keep blocks at one per 10 min. Alts change the interval between blocks not by messing with the max target, but by changing the retarget calculation.

The first difficulty was just simply the hardcoded mininum of 1. This didn't correspond to ten minutes initially, it was actually too high for the networks hashing power. The difficulty increased for the first time almost a year later on the 30th of December 2009.
For an overview of the difficulty adjustment you can refer to this table: Bitcoin Difficulty Adjustments


text:
Received bitcoin but can't send
I am new and using multibit 0.5.14. I have received 2 bitcoins in one transaction, and 0.005 in another. However, after over an hour they are still not spendable. I checked the status and it says "seen by 4 peers Pending/Unconfirmed. When I check on blockchain.info, there are 17 confirmations for that block. How do I know if something has gone wrong? Is there something else I should do to make my bitcoin 'spendable' or do i just have to wait it out longer?

What does it say about your specific bitcoin address at blockchain.info?

Have you restarted the client?  There is a chance that your client experienced some form of network hiccup and isn't updating it's block chain.  Close the application and restart it.

You can click Tools-> Reset Blockchain and Transactions.


text:
Does an increase in hash/s decrease the time it takes to obtain a block or does it increase the possibilities of solving one?
I am a total noob, and I've been reading the wiki for a long time and have yet to understand the basics... My question here is that if I were to mine at 100GH/s (example), is it guaranteed that I would find a block after a determined amount of time? Or does it just increase the possibility that I am the first to find it?

Since solving a block is a probabilistic math problem, having more hash power will increase the probability that you will find the right hash and solve the block. You cannot be 100% sure of the estimation given by the profitability calculators.


text:
Converting from Private Key to WIF, what am i missing?
They say a image speaks louder then 1000 words so:

As you can see from the image above, this should work ?
What am i missing here?
This is my encode function,
http://cryptocoinjs.com/modules/currency/coinstring/
I can confirm the private key spitted out is in correct format for if i do a dump i get:
$ ./src/d-cli dumpprivkey WRSybD94sWugjAQ89zyvbLsnw76ovCYEGo
BxkN86fdFBN75Aat3WCDs9w6NRHCMJJvvrDgb44Vcc23f96ohFZ5

How does your encode() function work? You probably aren't encoding it as base58 correctly or you are encoding the wrong thing (e.g. hex string instead of the bytes themselves).

sorry i edited my question to tell about my encode function, http://cryptocoinjs.com/modules/currency/coinstring/ and i can confirm the spitted out key is in correct format, the PREFIX is 100% correct as you can see by the image and code reference, really struggling with this one :(

You can see on the referenced encode() function that im sure im doing everything correct?

the issue was in the encode() function thanks kindly

The issue as suggest by Andrew was in the encode function.
After using https://www.npmjs.com/package/wif everything worked perfectly as intended.

and here is the list of address prefixes: https://en.bitcoin.it/wiki/List_of_address_prefixes


text:
How do you switch your bitcoinjs integrationfrom TransactionBuilder to transactions-psbt.js?
the code below works:
const btc = require('bitcoinjs-lib');

var tx = new btc.TransactionBuilder(net);
inputs.forEach(o => tx.addInput(o.txId, o.vout));
outputs.forEach(o => tx.addOutput(o.address, o.value));
tx.sign(0, owner);
var body = tx.build().toHex();

but it's complaining that TransactionBuilder is deprecated... sigh...

Deprecation Warning: TransactionBuilder will be removed in the future.
  (v6.x.x or later) Please use the Psbt class instead. Examples of usage
  are available in the transactions-psbt.js integration test file on our
  Github. A high level explanation is available in the psbt.ts and
  psbt.js files as well

and apparently the signing methodology has also changed (grr):

DEPRECATED: TransactionBuilder sign method arguments will change in
  v6, please use the TxbSignArg interface

I've read the documentation on Psbt and understand the general concept but have not found a guide that helps with the transition.  I tried to simply change tx.sign() for tx.signInput() but it pukes
I've also looked at the test suite for the library but it's too complicated for me to understand.  How do I convert my code to the new mechanism?

You're only signing the first input? tx.sign(0, owner); Usually every input has to be signed somehow.
I also struggled with the switch to Psbt. Biggest difference was basically requirement to have a full transaction hex of every input for non-segwit inputs. Segwit inputs can be simpler but they can also be created using the full hex of the tx using non-segwit command so don't have to add extra logic differentiating between inputs. You can no longer just provide txid and vout, just add on the hex of tx that created each input for your object.
I really just followed examples https://github.com/bitcoinjs/bitcoinjs-lib#examples
So this is my take on a very general tx with psbt that I use for now:
import * as bitcoin from 'bitcoinjs-lib'

// not all parts of inputs have to be defined
// inputs - array of all data needed for inputs
// outputs - array of all data needed for outputs

const network = bitcoin.networks.testnet // or bitcoin.networks.bitcoin

const psbt = new bitcoin.Psbt({ network })
psbt.setVersion(2)    // default
psbt.setLocktime(0)   // default

// ************** create inputs **************

inputs.forEach(input=> {
  psbt.addInput({
    hash: input.txid,  // txid number
    index: input.vout,  // output number
    sequence: input.sequence, // often 0xfffffffe
    nonWitnessUtxo: Buffer.from(input.hex, 'hex'), // works for witness inputs too!
    redeemScript: input.redeemScript, // only if there's redeem script
    witnessScript: input.witnessScript // only if there's witness script
  })
})

// ************** create outputs **************

outputs.forEach(output => {

  psbt.addOutput({
    address: output.address, // only if not op_return type output (no address for those)
    value: output.value,
    script: output.data  // only if there's string (utf8) data to embed in op_return
      ? bitcoin.payments.embed({ data: [Buffer.from(output.data, 'utf8')] }).output
      : undefined
  })
})

// ************** signing **************

inputs.forEach((input, index) => {

  // sign regular inputs that can be simply signed
  if (!input.redeemScript && !input.witnessScript) {
    psbt.signInput(index, input.keyPair)

    // give error if signature failed
    if (!psbt.validateSignaturesOfInput(index)) {
      throw new Error('Signature validation failed for input index ' + index.toString())
    }
  }

})

// ************** finalizing inputs **************

inputs.forEach((input, index) => {

  // sign regular inputs that can be simply signed
  if (!input.redeemScript && !input.witnessScript) {
    psbt.finalizeInput(index)
  }

  // for p2sh or p2wsh script inputs
  if (input.redeemScript || input.witnessScript) {
    psbt.finalizeInput(
      index,
      getFinalScripts({
        inputScript: input.inputScript,
        network
      })
    )
  }
})

// ************** make tx **************

const tx = psbt.extractTransaction()

const virtualSize = tx.virtualSize()
const txid = tx.getId()
const hex = tx.toHex()

console.log('tx virtualSize:', virtualSize)
console.log('tx txid:', txid)
console.log('tx hex:', hex)

And for some reason finalizeInput for custom scripts is not exactly built in yet but I found it in examples and just added as a function I import or put somewhere in same file.
function getFinalScripts ({ inputScript, network }) {
  return function (
    inputIndex,
    input,
    script,
    isSegwit,
    isP2SH,
    isP2WSH,
  ) {
    // Step 1: Check to make sure the meaningful script matches what you expect.

    // Step 2: Create final scripts
    let payment = {
      network,
      output: script,
      input: inputScript,
    };
    if (isP2WSH && isSegwit)
      payment = bitcoin.payments.p2wsh({
        network,
        redeem: payment,
      });
    if (isP2SH)
      payment = bitcoin.payments.p2sh({
        network,
        redeem: payment,
      });

    function witnessStackToScriptWitness(witness) {
      let buffer = Buffer.allocUnsafe(0);

      function writeSlice(slice) {
        buffer = Buffer.concat([buffer, Buffer.from(slice)]);
      }

      function writeVarInt(i) {
        const currentLen = buffer.length;
        const varintLen = varuint.encodingLength(i);

        buffer = Buffer.concat([buffer, Buffer.allocUnsafe(varintLen)]);
        varuint.encode(i, buffer, currentLen);
      }

      function writeVarSlice(slice) {
        writeVarInt(slice.length);
        writeSlice(slice);
      }

      function writeVector(vector) {
        writeVarInt(vector.length);
        vector.forEach(writeVarSlice);
      }

      writeVector(witness);

      return buffer;
    }

    return {
      finalScriptSig: payment.input,
      finalScriptWitness:
        payment.witness && payment.witness.length > 0
          ? witnessStackToScriptWitness(payment.witness)
          : undefined,
    };
  }
}

I haven't tested every kind of input yet but this is basically what I use for p2wpkh and p2wsh inputs. I try to find examples spending exactly the type of input I need usually.

Here is an example (native Segwit, you need to modify "witnessUtxo" appropriately for a p2pkh, see this):
const bitcoin = require('bitcoinjs-lib');
let testnet = bitcoin.networks.testnet;

let keyPair1 = bitcoin.ECPair.fromWIF('privateKey1', testnet);
let keyPair1wpkh = bitcoin.payments.p2wpkh({pubkey: keyPair1.publicKey, network: testnet});
let keyPair2 = bitcoin.ECPair.fromWIF('privateKey2', testnet);
let keyPair2wpkh = bitcoin.payments.p2wpkh({pubkey: keyPair2.publicKey, network: testnet});

const psbt = new bitcoin.Psbt({ network: testnet })
let inputdata1 = {
  hash: "hash of the previous transaction" ,
  index: index,
  witnessUtxo: {
    script: Buffer.from('scriptPubKey_hex', 'hex'),
    value: 98e4
  }
};

psbt.addInput(inputdata1);
let output1 = {address: keyPair2wpkh.address, value: 97e4};
psbt.addOutput(output1);
psbt.signInput(0, keyPair1);
psbt.finalizeAllInputs();
const raw = psbt.extractTransaction().toHex();
console.log(raw);

Let me first of all say I am a complete newbie so I might be wrong. Regarding the sign method deprecation, what has been already deprecated is not the method tx.sign() itself, but the way we passed the parameters. In order not to get the DEPRECATED message for the sign method you should pass an object of the form
{
  prevOutScriptType: string;
  vin: number;
  keyPair: Signer;
  redeemScript?: Buffer;
  hashType?: number;
  witnessValue?: number;
  witnessScript?: Buffer;
}
(see https://github.com/bitcoinjs/bitcoinjs-lib/blob/master/ts_src/transaction_builder.ts line 74)
so concretely in your case
tx.sign({prevOutScriptType:'p2pkh', vin: 0, keyPair: owner})

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Mining profitability sounds too good, where is the catch?
I am a beginner looking into Bitcoin mining. I don't want to waste time (or electricity) so I started looking into hardware mining rigs, for example:
Saturn
priced at $3000 USD,
275GH/s, consumes ~400Watt
(or even Monarch from butterfly labs which claims 600GH/sec @ 350W, priced $4700)
I put these numbers + my local electricity price + current difficulty into a decent calculator,
and I get >7,000 $USD profit for 3 months time frame (or >18,000$ for the Monarch). Sounds a bit too good to be true.
I know BTC/USD rate might fluctuate, and that the miner will produce heat, but still. - Where's the catch/mistake, if at all?

possible duplicate of Why does mining profitability tend towards zero?

possible duplicate of Do these new ASIC miners really pay for themselves in 5 days?

Keep in mind that difficulty also evolves over time: the more hashes the network produces, the higher the difficulty becomes.  As of this writing the difficulty is 148,819,199.  By my computation, each additional 275GH/s rig that joins the network will increase the difficulty by about 40,000; if 4,000 of them are sold, the difficulty will more than double (not counting all the competing mining rigs that may join the network).  And of course, in a few more months, there may be even more powerful rigs available.  So you should expect difficulty to rise, perhaps sharply, and this will decrease your profitability over time.


text:
Why are Transactions sending out randomly?
I see in the sourcecode this two lines:
// Do this infrequently and randomly to avoid giving away
// that these are our transactions.

in the method ResendWalletTransactions. Why is this necessary? If a client send out his transactions every 30 minutes and the receiver put these transactions in his own queue and send this transaction list every 30 minutes to all his neigbors I don't see why someone should recover the own transactions of the client.
The full beginning of the method is something like:
void CWallet::ResendWalletTransactions()
{
    // Do this infrequently and randomly to avoid giving away
    // that these are our transactions.
    static int64 nNextTime;
    if (GetTime() < nNextTime)
        return;
    bool fFirst = (nNextTime == 0);
    nNextTime = GetTime() + GetRand(30 * 60);
    if (fFirst)
        return;

I don't see why someone should recover the own transactions of the client.

You are possibly misreading something as there's no "recovering" of transactions.  This has to do with the timing of when the re-broadcast of a transaction that hasn't confirmed occurs.  Please clarify what your concern is.

Because it doesn't re-broadcast any non confirmed transaction, it re-broadcast only your own non-confirmed transactions. So you will be the only node on the entire network who is doing the re-broadcast every exact 30 minutes and based on this anyone can get what IP is originating that transaction.
A few lines below your code example you find this:
BOOST_FOREACH(PAIRTYPE(const uint256, CWalletTx)& item, mapWallet)
{
...
}

This will loop on the transactions of your own wallet (mapWallet), not all the network transactions.


text:
I sent my bitcoins from LocalBTC to a unsynced bitcoin wallet program, the bitcoins are not showing up. How do i get them back?
I sent bitcoins from localbtc wallet to a unsynced program, it says 0 in the balance of the program and I really want them back.

Possible duplicate of http://bitcoin.stackexchange.com/questions/6056/cannot-access-my-bitcoins-because-my-wallet-is-still-synchronising-what-can-i-d

I corrected the tags, I assume you are using the standard client, is that correct?

You have two options: Either wait until your wallet has sychronized with the network – your bitcoin will show up when the block was processed that validated the transaction with which it was sent to you. The other option is that you can export the corresponding private keys to a client that doesn't store the blockchain and access it that way.

The easiest way is to open your wallet program and export the private key for the address(es) to which you sent the money. You would then import these private keys into a wallet program that doesn't require downloading the blockchain to work. Your best bets are BlockChain.info's web app and Electrum. Your balances should show up properly and immediately in those wallets and you'll be able to spend them freely.


text:
How many TXO have been created in total?
I've seen some statistical data about the Bitcoin UTXO number (around 61M currently). I was wondering how many TXOs (including spent and unspent) there were in the complete history of Bitcoin?

Possible duplicate of How to get total transactions on the network with RPC

@JBaczuk: These two questions are related but different in that the linked question is asking for the total transaction count and this one here is asking about the total transaction output count. Voting to leave open.


text:
How does LND make sure that it does not reuse the same route for a retry after the first payment failed?
Suppose LND node A wants to send a payment to another node D. The pathfind algorithm based on Dijkstra found a route through B and C and initiates the payment. It fails, as B -> C does not have sufficient balance/bandwidth.
Will LND just fail the payment or try another time with a different route? How does it remember the failed route?
Follow-up question: Can node A figure out which edge failed?

Edit:
To be more clear, in my example, all channels have 0.5 BTC as capacity, and all of them have an equal balance distribution of this capacity (0.25, 0.25), except for B -> C, which has (0, 0.5), so B currently cannot send any payments to C.
The balance state (here: (0, 0.5)) is private, but the capacity of all channels is public.
As the routing computation is done in A (source routing), the algorithm initially has no knowledge of the balances. If A just wants to send 0.001 BTC, the routing algorithm will find the route, but it cannot reach the target.

In fact, finding a route in LND can be described as the process of finding the "shortest path" on a weighted graph. In LND, the weight of each channel is composed of three parts, fee, CLTV_DELTA and the probability. You'll understand the first two factors if you are familiar with the Lightning Network, and the last factor is the probability that this channel can afford the payments from a local perspective.
If a route fails, then the node with insufficient capacity will actively include this information in the return message, which can only be unlocked and seen by the payer. For example,  Then the payer will reduce the probability of this channel in the local view and find the route again (you can treat it as setting the weight of this edge to very large).
You can find detailed information here.

So I went ahead and looked at the LND implementation here: https://github.com/lightningnetwork/lnd/blob/master/routing/pathfind.go and this is the comment block above the function findPath:

findPath attempts to find a path from the source node within the ChannelGraph to the target node that's capable of supporting a payment of amt value. The current approach implemented is modified version of Dijkstra's algorithm to find a single shortest path between the source node and the destination. The distance metric used for edges is related to the time-lock+fee costs along a particular edge. If a path is found, this function returns a slice of ChannelHop structs which encoded the chosen path from the target to the source. The search is performed backwards from
  destination node back to source. This is to properly accumulate fees that need to be paid along the path and accurately check the amount to forward at every node against the available bandwidth.

If this is the case then I don't think your scenario would play out. It appears LND wouldn't return the path A --> B --> C --> D because it does not have sufficient balance/bandwidth.

Yeah, that is the general aim of findPath, but as long as bandwidthHints is not set, it has no "available bandwidth". bandwidthHints usually is only set for the outgoing channel of A. I also extended my problem description.
If it actually sets bandwidth hints from e. g. failed routes, that would exactly be what I am interested in!

The distribution of the balance in a channel is not public information, so lnd wouldn't know that the funds are all on the side of C in B->C before attempting to route the payment through the channel.


text:
How to tell if a transaction is valid?
So I have been reading up on bitcoin and blockchain for awhile now, but I have yet to fully understand it. I understand that bitcoin leverage on DLT, which uses cryptography to verify transaction. This is what I have understood so far:

When I make a transaction, this transaction is added to a pool of unconfirmed transactions
This pool of unconfirmed transactions forms a block and miners will solve a mathematical problem as verification
The first miner to solve it will have the block verified and added on to the chain

However, how do they know if my transaction is valid? What if I have 1 BTC but decide to send 2 BTC, how does the solving of mathematical problem verify that I do not have 2 BTC? What if I just purchased 5 BTC from Coinbase using fiat, how will anyone know that I have 5 BTC?
I believe this should not be a very difficult question but I cant seem to understand it fully. Hope someone can point be to the right direction and enlighten me thanks!

When you spend Bitcoin in a transaction, the transaction must reference at least one existing unspent transaction output (UTXO) and provide a valid signature proving that you are authorized to spend it. The transaction then consumes the UTXOs in the inputs, and assigns value to newly created UTXOs in the outputs.
Other network participants running a type of Bitcoin software called "fully validating nodes" have followed along the full history of all Bitcoin transactions (the "blockchain") to maintain a complete set of all UTXOs in existence (the "UTXO set"). You can think of the blockchain as the journal of all transactions, and the UTXO set as the ledger of all Bitcoin balances.
UTXOs are uniquely identifiable by their "outpoint" which is composed of the transaction id that created the output and their position in the transaction's output list: txid:vout.
This means that each full node can look up whether the UTXOs the transaction spends in the inputs exist in their UTXO set. They also check that the transaction is correctly constructed, including whether the transaction fulfills the locking conditions of the UTXOs, whether the signatures are valid, and that the transaction does not create new money by assigning more to the outputs than the inputs brought in.

When you say the transaction "consumes the UTXO", does this mean that all UTXOs (even if same value) are unique - no 2 UTXOs are the same in the UTXO set?

Yes, exactly! Each UTXO is uniquely identifiable by their outpoint, a value derived from the transaction that created the UTXO and the position of the UTXO in that transaction's output list. I've amended my answer to include this point.

Got it, a final clarification: I am assuming this UTXO Set includes all transactions on the chain, as well as the pool of unconfirmed transactions - so what is the purpose of mining (hashing of block header to find a result smaller than the block target)? Since UTXOs are unique and already prevents double spending and verifies transactions? How does mining fit in here?

Since Bitcoin is a distributed network, it is non-trivial to make everyone receive the same transactions at the same time. A malicious users can leverage the latency to create two transactions that spend the same inputs, e.g. submitting one directly to a merchant and another to the rest of the network. The merchant would think they got paid, the rest of the network would disagree. A block however, can only include one of the two transactions, so it allows the network to converge on one ground-truth.

Would'nt the creation of two transactions that spend the same inputs be disallowed in the first place - since UTXOs are unique and will be rejected by the validation nodes?

Yes, that's right, but it's possible for some nodes to see the original version of the transaction first, while other nodes see the competing transaction first.


text:
Sidechain vs layer 2
I want to know what's the difference between "sidechain" and "layer 2".
Is it the same thing ?
Thanks

The definition of a sidechain is more universally agreed than the definition of Layer 2. Some people might argue that sidechains are a subset of Layer 2 scaling solutions. Lightning Network is definitely a Layer 2 scaling solution.
Personally I would say the difference between a sidechain and Layer 2 is that Layer 2 scaling solutions such as Lightning are trust minimized protocols. They are designed such that trust in your channel counterparty (or anybody else on the network) is minimized.
Whereas sidechains (e.g. Blockstream's Liquid) do not seek to minimize trust, they instead make the design decision to lock funds up into a multisignature (in Liquid's case, a 11-of-15) onchain where the parties in that multisignature are trusted or relied upon to get the funds out of that sidechain. Sidechains make this choice because it provides greater efficiency (less reliance on timelocks), greater flexibility (you don't need to set most of the rules at inception), reduced fees (you don't need to pay routing fees like you do on the Lightning Network) and reduced capital constraints (you aren't constrained by the capacity of Lightning channels).

A key point for me would be that sidechains have their own global consensus mechanism, usually a distinct blockchain.

I'd go further and say that in order for something to be a sidechain, it has to be a chain. Regarding Layer 2, I think the term is grossly overused, but if I had to give it a consistent definition it would be "anything built on top of a blockchain". Something that's a blockchain itself (like a sidechain) would therefore be just another layer 1, not a layer 2.


text:
What happens to transactions included in extinct or invalid blocks?
I have read that a block contains several transactions.
An attacker can forge fake a block.
The blockchain is safe because the system will choose and keep the longest chain.
So the fake block will be lost.
This is great but what happens if my transaction was included in the fake block?
Let's suppose I am buying a car. Should I wait for 6 confirmations to be sure? That would mean that I have to wait one hour!

what happens if my transaction is in the fake block?

It is also in the memory pool (mempool) of pending transactions at other nodes run by more honest people and will be included in a new and valid block in the normal way.

Let’s suppose I am buying a Tesla car. Should I wait 6 blocks to be generated to be sure? But I have to wait 1 hour!

Yes.
There are faster payment methods for Bitcoin - e.g. Lightning Network (LN) and arguably perhaps escrow services. But they have their own trust model and characteristics you need to understand when using them.

An attacker can forge fake block.

Yes, but an invalid block would be rejected by all other participants that see it, and not relayed. Thus, such a block has little to no impact.

This is great but what happens if my transaction is in the fake block? Let’s suppose i am buying a car. Should i wait 6 blocks to be generated to be sure? But I have to wait one hour!

A transaction is a signed payment order to the Bitcoin network.
As long as it remains a valid instruction set to the network, it will be tracked and distributed by network participants to include in a block. From the perspective of nodes following the best chain, being included in a block that is not part of the best chain is equivalent to not being included in any block at all. Thus, an invalid block including a transaction has no impact on any nodes that recognize the block as invalid–they still consider the transaction unconfirmed and they will continue to consider it for inclusion in another block.

The fake block is ignored and you have to wait until the next block, 2 blocks, 4 blocks, or 6 blocks, depending on how many confirmations the miner wants to be mined. But, if you are using Lightning, this can't happen unless you are taking out of lightning. The invalid block will not be added. If the miner will mine your block (ignoring other factors such as transaction fee, willingness of miner to fill block fully (this is not a problem anymore), average block time (if the difficulty hasn't changed recently enough, only really a problem if it goes down, it would help if it's predicted to go up), and the number of transactions sitting already), there is already a 62% chance that the confirmation will happen in 10 minutes. Although you have to wait, just be patient.


text:
When does a miner get the reward?
Let's suppose that a miner mines a block and adds it to the blockchain. When it gets the reward ? I suspect that first of all the other nodes have to verify that its proof of work is correct and it's not cheating. Right ?

Miners don't receive rewards from anyone else - they give the reward to themselves as the output of the coinbase transaction, which includes the block subsidy and any fees from the txs in that block.
They then broadcast this block to everyone else. Each node that receives the block independently validates it to ensure it meets consensus rules (has a valid proof of work, does not contain any invalid transactions, has the correct subsidy and fees charged, etc.).
If any node believes the block is incorrect, it will reject it and continue to mine or wait for another valid block at the same height.
If the block is considered valid, it is added by each node to its own copy of the blockchain. Once 100 blocks have been mined extending the block in question, the coins in the coinbase output are considered mature and may be spent by the miner of the block.


text:
Does every node have a whole copy of the blockchain?
As I read in this blog article, in a blockchain "each node has a copy of all transactions ever made". This to me seems impossible, bitcoin has existed for a long time, and the memory size of the entire blockchain would be enormous.
ps: if I have to pay for a service using bitcoins, do I become a node of the blockchain?

To join the network, a Bitcoin full node will work its way through the network history, independently verifying the state of the network through time. So yes, each node will have downloaded and validated the entire transaction history, but a node can choose to use 'pruning mode', where it will discard unneeded transaction history after validation is complete.
The ever-increasing size of the blockchain is a heavy consideration when considering network and protocol updates. Bitcoin requires a distributed network in order to maintain its function, so it is important that the software can be run on inexpensive hardware, so that more people can participate.

ps: if i have to pay for a service using bitcoins, do i become a node of the blockchain ?

There are other ways to interact with the network. Many wallets are lite-wallets, which validate the chain via headers instead of the entire block. This is a less robust security model, but it works well for many people.
You generally do not need to pay a service in order to interact with the network (as a full node, a lite-wallet, etc).

Full nodes by default store the entire blockchain. The current size of the blockchain is ~250GB, so it is feasible for individuals to have their own copy and verify all of the transactions. There is also the option to run Bitcoin in pruned mode, which reduces the number of transactions that need to be stored. This greatly reduces the storage space demands and it can still verify all the transactions.
However, not every node is a full node. There are lightweight nodes known as SPV nodes that don't verify the entire blockchain. Instead they can verify block headers and rely on a full node for the state of the blockchain.
More info on SPV nodes: https://gist.github.com/ariard/1034cd7624805d53334e80d4712fb8ee
If you're paying for a service using Bitcoin, its very likely that the service provider is running a full node for you. You run your own node when you run the bitcoin software yourself. It's a a worth while experience and if you're interested in Bitcoin and want to help maintain its integrity you should consider running your own node.
What/how to run your own node: https://bitcoin.org/en/full-node#what-is-a-full-node

re: "Every full node has the entire blockchain" -  this isn't true. Many full nodes run in pruning mode, allowing them to discard the vast majority of the block data, if they so choose.

@chytrik you're correct, I edited my answer and I believe it is right now.

Pruning has nothing to do with the Merkle tree. It's simply downloading and verifying the whole chain normally, but deleting old blocks once they have been verified.

@PieterWuille okay I edited my answer again. I was under the impression that if a transaction was pruned and needed to be verified the transactions would then need to be generated from the merkle root. Is just the block header sufficient for that?

Transactions are pruned after they are verified. They don't need to be verified again. The Merkle root can be used to prove to lightweight (non-validating) clients that a transaction was part of a block, but that is unrelated to pruning.


text:
Fork and Longest Chain Rule
At the same time, A and B succeed in resolving the "Proof of work" challenge and both add a new block to their local blockchain, then they propagate the new blockchains to other nodes. I have 2 questions:
1) If a node X receives first the blockchain from A, and it approves it, when it receives the blockchain from B, it simply reject it ?
2) If a node X receives first then blockchain from A, and then the B's blockchain in which also a node C has put another correct block (so is longer then A's blockchain), does the node X reject A's blockchain (applying the longest chain rule) and use the other blockchain as the correct one ?
2) Due to latency a miner node may not have the complete list of transactions, so it may start to build an incorrect block (for example some transaction misses). If the miner recognize is building an incorrect block (because for example, it received other transactions while it was building the block) it simply throws away the block and start building another one?

If a node X receives first the blockchain from A, and it approves it, when it receives the blockchain from B, it simply reject it ?

It will keep the block locally as an alternate chain, but not immediately discard it.

If a node X receives first then blockchain from A, and then the B's blockchain in which also a node C has put another correct block (so is longer then A's blockchain), does the node X reject A's blockchain (applying the longest chain rule) and use the other blockchain as the correct one ?

Correct, once any one chain has accumulated more difficulty, the minor chain is discarded.
Note that here accumulated difficulty is a measure of how much hashpower was spent to produce the amount of work required for that chain. This is generally in sync with the chain being longer (a chain with more blocks will have more difficult in most cases), but it is not a guarantee.

Due to latency a miner node may not have the complete list of transactions, so it may start to build an incorrect block (for example some transaction misses). If the miner recognize is building an incorrect block (because for example, it received other transactions while it was building the block) it simply throws away the block and start building another one?

This is really a separate question, but the answer is mostly "ASICs are fast, it doesn't matter". Miners are constantly producing update block templates with altered timestamps, secondary nonces in the coinbase transaction, and transaction lists. These are then passed to the actual machines doing the mining, which can burn through the nonce space in the header in a matter of seconds. A miner will simply wait for the old block template to be non-viable and create a new one with the better transactions.
Of course, if a miner does find a valid block on a slightly out of date template, they will broadcast that block - it makes no sense to give up a guaranteed 12.5 BTC for the possibility of slightly higher fees in a block you might not find before another miner.

The rule is most difficult, no longest.

@Anonymous Absolutely, I've updated the answer. Thanks!


text:
Can attackers change transactions during a 51% attack?
If in a 51% attack transactions can be reversed, isn't it possible that the chain with more hash rate also changes transactions (for example if x sent 5 btc to y, they would change it to x sent 500 btc to y)
and then would confirm such malicious transactions more often than the corresponding good transactions that were confirmed on the honest chain.
I have this notion that transactions get n number of confirmations if after the block with that transaction was added, n more blocks were added.

No. A 51% attack cannot modify transactions or those transactions become invalid. Blocks do not dictate whether a transaction is valid. Transactions have their own validity rules and if a block includes an invalid transaction, then that block is invalid, regardless of the Proof of Work.
Changing a transaction's details as you suggest requires producing a new signature. Only the original creator of that transaction can do that. So attackers cannot just arbitrarily change people's transactions as they do not have the private keys necessary to create those signatures. Thus any modifications they make will result in invalid transactions.
All that an attacker could do is remove transactions from history and insert their own transactions. They could only replace their own transactions. The danger of a 51% attack is that the attackers can replace their own transactions paying someone else with a double spending transaction that pays themselves thereby undoing a transaction.

Every transaction other than the coinbase transaction must draw bitcoins from somewhere that ultimately trace to a coinbase transaction. Undoing the block with that coinbase transaction renders that transaction invalid. So a 51% attack can render any broadcast transaction invalid by rolling back any block that contains any coinbase transaction that any bitcoin moved in that broadcast transaction ultimately traces to.


text:
How the bitcoin network solves the double-spending problem by adding all the transactions that happened at the same time into one block
According to my research, I understand if person A sends 2 BTC to person B and at the same time person A sends the same input to himself before the input is considered as spent. So, now there will be a different route for each transaction while broadcasting (where nodes will check transaction chain to verify the history of ownership by the previous transactions) in the bitcoin network and might be the wrong one got confirmed first. Obviously, the second one will be rejected and the network will give contradiction while consensus. So, according to my understanding bitcoin network solve this issue by adding all the transactions that happened at the same time into one block of the block-chain. Please let me know if I am wrong. I will appreciate your guidance. Thanks
So, My question is: How the bitcoin network solves this issue by adding all the transactions that happened at the same time into one block?

according to my understanding bitcoin network solve this issue by adding all the transactions that happened at the same time into one block of the block-chain.

This is not true.
Same time
You should remember that the Bitcoin network doesn't really try to keep accurate time. Blocks are not always in sequence by timestamp in the block. So long as they do not seem to be more than about 2 hours wrong, it will be accepted. So the notion of "happened at the same time" is ill-defined.
There is a limit to the size of a block, so it would often not be possible anyway.
Choosing transactions
Miners choose transactions for incorporation into a block. They choose transactions from a pool of unconfirmed transactions they have received. This pool is called the mempool.
Which transactions miners choose is entirely up to them. They could choose on the basis of highest fees that the transactions will pay to the miner. They might choose on any other basis that their operator feels makes sense for them.
What might be causing some confusion is that if an unconfirmed transaction B spends the output of another unconfirmed transaction A, I believe miners will find it necessary to include A in any block that contains B.
Double spending
A miner cannot include in one block two transactions that spend the same input. Such a block would be rejected by everyone else.
If a miner includes in a block a transaction that spends an output already spent by a transaction in an earlier block, the later block will be rejected by other nodes. So miners don't do this if they can avoid it.


text:
How is it legal to award miners with coins made out of nothing?
I understand that block rewards are made out of nothing but say a miner wants to cash out on all of their bitcoins... this will lead them to obtain, let us say, the USD equivalent of all the bitcoins the miner has from transactions + whatever the miner earns as block rewards.
So what this is doing is giving the miner all the USD they legally own and also some extra money that they generated from their block rewards. How is this legal because no entity is paying the miner and the miner is technically making money out of thin air?
I know this is kind of a complicated query but I do not understand how it is okay to make money out of thin air like that.

Bitcoin isn't making money out of thin air (mining is a free market wherein energy is turned into network security), you must be thinking of the Federal Reserve.

Miners spend energy to create blocks/getting rewarded bitcoin, it's not out of thin air. As new miners try to create new blocks and get the block rewards, the energy needed to create a new block increases and it gets harder and harder to create a new valid block. This difficulty is adjusted every two weeks. For example, at the start of the year the difficulty was 5,618,595,848,853 and now is 12,876,842,089,682, which means it costs more than double to create a bitcoin now than at the start of the year. If you try to mine bitcoin with your laptop/desktop/smartphone you will get less bitcoin for the electricity that you spent than you would get if you bought it at an exchange.
The cost of creating one bitcoin with an electricity cost of $0.05/Kwh, for this era, is apr. given by:
Cost($/BTC) = difficulty[trillion] * energy_efficiency[J/Gh] * 4772

So, for an antminer S9, the current cost would be about $6144 per bitcoin.
Also keep in mind that the cost of mining bitcoin doubles every four years (halvening event).
Some resources you might want to check:

Bitcoin difficulty - https://btc.com/stats/diff
Energy efficiency of mining equipment (ASICs) - https://www.cbeci.org/methodology/


text:
How do you recognize that a transaction is the replacement of another?
Let's say I have a transaction with txid=A which has bip125-replaceable=True. If I bump the fee the original transaction will be removed from the mempool and replaced by the new one, so how do I know that the new transaction is, in fact, the same as the old one just with a higher fee? Does the txid stay the same? If so, than the answer is obvious. But if it changes, e.g. txid=B, then this is a completely new transaction. How can I establish the link to the old one?

The txid will change, because the transaction has a different effect. It will have more/larger inputs or less/smaller outputs, possibly combined with other changes.
The link between the two transactions is that they conflict. This means that they directly or indirectly depend on spending the same predecessor output. As every output can be spent exactly once, both cannot end up in the same chain.


text:
With Legacy bitcoin addresses deprecated, will Bitcoin Core no longer be able to sign messages from an address?
Bitcoin Core can only sign messages from legacy bitcoin addresses, in my experience. With legacy addresses being deprecated in the software, does this mean that Bitcoin Core will no longer be able to  sign any messages from the new descriptor wallets verifying someone holds the keys to their bitcoin addresses?
If this is true, is there any work being done on adding message signing functionality for bech32 addresses?

Bitcoin Core can only sign messages from legacy bitcoin addresses, in my experience.

Yes, that's correct. Message signing is currently only supported for P2PKH addresses.

With legacy addresses being deprecated in the software, does this mean that Bitcoin Core will no longer be able to sign any messages from the new descriptor wallets verifying someone holds the keys to their bitcoin addresses?

This is conflating two things: legacy wallets and legacy addresses. They're independent. If you want to, you can create/import a descriptor for P2PKH addresses in a descriptor wallet, which can generate P2PKH addresses, for which message signing will work. You can also use a legacy wallet to construct P2SH or P2WPKH addresses, which don't support message signing.

If this is true, is there any work being done on adding message signing functionality for bech32 addresses?

Yes, there is an ongoing though somewhat stalled effort to define and implement a generic message signing standard that would work for any address type, BIP322.


text:
I sent Bitcoin to an address but the recipient hasn't yet received it
This is the transaction ID:
81f0babd0a3c6d9be17db45a62932679b05bac6a30df1abfb47c2eee83d9f0ff. The address is not wrong but the receiver couldn't get the payment. It has been 2 days.

Possible duplicate of https://bitcoin.stackexchange.com/q/89673/5406, https://bitcoin.stackexchange.com/q/92058/5406


text:
What is the SPV Mining Exploit?
Can anyone explain this simply? I am not finding enough explanation exactly on this type of attack. If possible, add some additional reading to your answer so that I can understand more.

I am not sure what exploit you are referring to. Perhaps this https://www.reddit.com/r/Bitcoin/comments/7752jc/trying_to_understand_spv_vulnerability_as_stated/ is relevant? A majority miner can inflate coinbase transaction and then spend those additional coins fraudulently with you - your SPV node just checks whether the chain is ok.

Are you referring to this? https://medium.com/coinmonks/bdos-blockchain-denial-of-service-385c8c56b401?

"SPV mining" refers to a bad practice where a miner starts hashing a new block on top of an unverified parent block. If the parent block turns out to be invalid due to a double spend or newly activated soft fork, the new block will also be invalid.
The term SPV in this case is used because the miner only verifies the headers of the incoming block and then immediately starts mining the next block.
This has happened on main net before: when BIP66 activated in 2015, a short chain of blocks was mined on top of an invalid block that did not enforce the new rules.
This question may be a duplicate:
What is SPV mining, and how did it (inadvertently) cause the fork after BIP66 was activated?


text:
What is the benefit of having a fixed target block interval?
By design Bitcoin targets a block interval of 10 minutes.
How does the blockchain benefit from maintaining a bounded block time?

Does this answer your question? Why was the target block time chosen to be 10 minutes?

I am more worried about why the time is fixed, not specifically why it is 10 min?

Block mining time is not fixed. 10 minutes is just average value which is aimed by difficulty adjustment algorithm to balance two contradicting requirements:

low transaction time requires low block mining time.
low probability to mine several different blocks simultaneously requires higher block mining time.


text:
How does bitcore handle transactions in a stale block situation?
I know stale blocks can occur naturally when two miners produce blocks at similar times. How is this handled in bitcore?
For example, when a chain reorganization occurs in the node, does bitcore change the blockHeight of transactions in the transactions table?

You might be thinking of stale blocks. Orphan blocks have no parents, and don't occur anymore since headers first synchronization.

Conceptually, every full node only has one "main" chain. There is only one block at each height in that chain.

Oh, I just realized that you mention Bitcore, not Bitcoin Core. If that was on purpose, my answer is probably obsolete, because I wrote about Bitcoin Core. Please let me know, then I'll delete it.

Thank you for all response,
I mean orphan block in bitcore. in bitcore repository issue, micahriggan replyed to the question(https://github.com/bitpay/bitcore/issues/2659)

If there are multiple chaintips that match in total work, Bitcoin Core will consider the one it heard first about the best chain. Only if another chaintip accumulates more work, it'll switch to that chaintip. As only one block can be part of the best chain at the same height, the competing block is essentially irrelevant unless it happens to become part of the best chaintip.
Example:
After block 99, there were two competing blocks found at the same time. Our full node first hears about block 100A (the asterisk marks the block our full node currently considers the chaintip), and a peer announces chaintip B with a different block at height 100. Bitcoin Core will not update its chaintip, but still consider 100A the best chaintip.
           100A*
98 ← 99 ↢
           100B

It's very unlikely that both 100A and 100B get a successor block at the same time. Let's say that the chaintip B is extended first.
           100A*
98 ← 99 ↢
           100B ← 101B

Upon learning about the new best chain, our full node will first revert block 100A: all of the transactions that got confirmed in block 100A get added back to the mempool and marked unconfirmed.
            100A
98 ← 99* ↢
            100B ← 101B

Then the full node follows the best chain by applying the two blocks 100B and 101B.
           100A
98 ← 99 ↢
           100B ← 101B*

Any transactions that had been confirmed in 100A might have now been associated with 100B or 101B instead, or they may still be in the mempool.

SeungJun linked above in a comment to the same question in the bitcore repository that got an answer from micahriggan. The answer states that bitcore removes information about the old block and processes the new block. It also links to the function handleReorg in the code.


text:
How do you handle returned goods in e-shopping?
The blockchain is immutable. So, using bitcoin for shopping websites can cause problems. Customers often want to return purchases in retail scenarios. But Bitcoin is immutable, so we can't reverse the transaction.
Somebody please guide me how to handle the problem of returning goods in bitcoin-based e-commerce.

Even in conventional systems, a return is a separate transaction from the original purchase. Both are recorded, the original purchase transaction need not be amended so long as the system associates both transactions together. In short, returns are not the same as a transaction cancellation.

Since I saw this as "not applicable to Bitcoin" in the review queue, I've edited it to highlight how the question does not change at all by applying it to Bitcoin. TBH, I don't understand why this is being closed under the "not applicable to Bitcoin" reason.

Even in conventional systems, a return is a separate transaction from the original purchase. Both are recorded, the original purchase transaction need not be amended so long as the system associates both transactions together. In short, returns are not the same as a transaction cancellation

Questions like this are off-topic on this forum, however I'll take a stab at answering your question as you seem to have a lot of misunderstanding.
The bigger question you need to ask yourself if why you want to implement your e-commerce application on a 'blockchain'? It would be a gross neglect for basic engineering design principles as blockchain is nothing but a slow and expensive database. I mean I would understand if you wanted to create a Dapp if you are selling drugs and other illegal substances (like SilkRoad) since you wouldn't want a single point of failure and lack of attribution as an owner. However, if you are creating an e-commerce application that is perfectly legal you can create it on a normal database system.
Blockchain is one of the many components that make Bitcoin successful, not the only one. A blockchain is used in Bitcoin to prevent double spend attacks as transaction included in previous blocks get precedence over the latest transaction if they spend the same output, and achieving consensus on a global scale where all participants have the authority to accept changes.
Private blockchain is nothing but an old wine in a new bottle. There are plentiful example of distributed databases with multiple write access available today like Google Spanner or Paxos. So, coming back to the question you need to ask yourself, what are benefits that the "blockchain technology" is providing you that is not possible with existing 'traditional' solutions in the market?

A few comments: "other PBFT consensus blockchains": PoW is not PBFT (and PBFT is a specific instance of a class of consensus algorithms called BFT; PoW is not BFT either).

Voting to change the state of the database in (P)BFT is equivalent to a block that contains an (possibly illegal) transaction. The immutability is about the history, not about the final state. If you maintain a log of all updates in PBFT, you also have an immutable history. If you make it hashlinked, it's arguably a blockchain even.

Last: Paxos and Spanner are not (P)BFT algorithm. BFT consensus keeps working even in the presence of malicious actors, but need 2/3 majority vote (and 2/3 honest participants). Traditional consensus algorithms like Paxos assume only malfunctioning/unreachable as failure modes, but not intentional obstruction. This is fine as they're run by trusted parties (all servers owned by Google), and it reduces the vote to a 1/2 majority.

@PieterWuille Thanks for the comments. Updated for (1) (it was a poor choice of words from my part) and added for (3). On (2), what I was referring to is as follows: There is transaction that happens with company A, that pays B and C. All three have write access to the blockchain (with one other company). At some point, A, B and C realize that the current state with the transaction is not beneficial to them. So they completely erase the hashlinkled 'blocks' previous to that and vote on the state that does not contain the transaction. The previous state with that transaction is completely erase

Right, my point is more this: in PoW the assumption is that a majority of the hashrate is honest (and indirectly because they act in their own selfish interest). In BFT, the assumption is that 2/3 of the signers are honest and will not sign a conflicting version of history. In both cases it is possible to revert the state if, but only if, those assumptions are broken. The distinction is not in what is possible or not, but under what assumptions that can happen.

@PieterWuille Exactly my point in terms of how it can happen. In PoW, to change a state they will need to expend huge amount of compute and electricity costs. This means that the transaction value to be excluded has to be much more (tangible and intangible) than the cost involved to mine all blocks subsequent to it. In PBFT, all they need is a vote to alter the state, which is easier.

No, you misunderstand. They cannot just vote to go back to an earlier state; that would be a violation of the protocol, and would be rejected by validators. Just as in PoW creating a low-work branch that branches off an ancestor would be a violation of the protocol and be rejected. A possibility is resetting every validator in the network, but that's not part of the protocol; it's replacing it with something else.

Perhaps you mean that they can vote to create a new state, without reverting, that changes the state back to what it was. That may or may not be possible, but it depends on the consensus rules. In something Bitcoin-like, where state transitions depend on things like transactions having valid signatures, even in a PBFT-like context, this would not be possible.

@PieterWuille "and would be rejected by validators": I'll try to explain my understanding better. There are 4 validators, A, B, C, and D. The first three want to change the state of the system (exclude a transaction) as it is more beneficial to them. Since they have 75% majority, they can easily do it by vote. However, in PoW although there can be similar collusion by miners, but they would still need to expend energy to build the hashes again. In PBFT there is no cost.

No that's not how it works. Either those 3 try to do it by pretending a previous state transition (the one that included the transaction?) never happens, and reverting to an earlier version of history. This is in violation of the protocol, and detectable - the 4th node will treat it as malicious and ignore those messages or something. The alternative is that they sign an update that brings the state back to what it was before the transaction (without pretending the history never happened). This is likely not allowed either - it's similar to a Bitcoin where everyone trust miners.

Let us continue this discussion in chat.

What I understand from discussion is that we can use 'voting' to cancel a transaction, maybe this is the reason why @Ugam calls it a "but a slow and expensive database". This is not logical in case of public blockchain because there can be many nodes. The other solution is to use a private blockchain and once we settle down with the transactions we can transfer them to public blockchain. This would help to ease the cost factor also. This is what I read in an article.  However real-world solution required.

What are validators? Never heard about this term in the context of blockchain. Please provide some link.


text:
What are the potential attacks against ECDSA that would be possible if we used raw public keys as addresses?
According to this answer about why addresses are hashes rather than public keys there are potential attacks that are possible if you have the public key rather than the address, what are these attacks? If one was writing a cryptocurrency would you advise against the addresses being the public key encoded in base58 with a checksum on the end? why?

The theory
It is assumed that in order to forge an ECDSA signature you need to compute the private key for a given public key first (this operation is known as the "discrete logarithm" (DL), and its hardness is the basis for ECDSA's security). In order to do so, you must actually have the public key.
Once you have the public key, it is assumed that you need at least 2128 operations to compute its private key. That's an enormous amount (if every computer in the world could do one relevant operation per clock cycle it would take over 100 million years; in reality it'd be orders of magnitude more than that). However, that is assuming that there are no fundamental breakthroughs in algorithms to compute the discrete logarithm, or quantum computers. A sufficiently powerful quantum computer (not anything anywhere near what already exists) may be able to do this computation much faster.
By using an address that contains a hash of the public key rather than the public key directly, the actual public key is not revealed to the world until the output is spent by its owner. Barring any (spectacular) vulnerabilities found in the used hash functions (SHA256 and RIPEMD160), even a quantum computer cannot trivially find a public key from a hash function. However, the 160-bit hashes used are still considered relatively weak in this case (280 operations on a sufficiently powerful quantum computer).
In short: the argument is that by using public key hashes, the ability for someone with a DL break, or a hypothetical quantum computer, to steal coins is made harder.
In practice
What I write in this section is my own opinion, and presumably not everyone agrees with it.
I believe this (often repeated) advantage of hashed public keys is marginal at best, and a false sense of security at worst. There are several reasons for this:

The argument only applies until the output is (attempted to be) spent. Once someone tries to spend a pay-to-pubkey-hash output, they reveal the full public key. With minor cooperation from miners, the original transaction could be delayed temporarily to give the hypothetical quantum computing attacker time to find the private key and steal the coins.
Address reuse was, and still is, very common, and apparently hard to avoid. Whenever addresses are reused, their public key is revealed on the first spend, making all future ones vulnerable still.
Almost all interesting things that people do with Bitcoin (including multisig, 2FA, escrows, payment channels, BIP32 accounts, ...) involve sharing public keys with other parties. It's an illusion to think that in such a world any security is gained by using public key hashes - as public keys are revealed all the time still, often without people even knowing about it.
Even if you restrict yourself to carefully not rely on any of these techniques and keep all your public keys actually secret until spent, there are over 5 million BTC (my own research) stored with publicly known public keys. I cannot imagine that BTC retains any value if those become practically vulnerable to theft (or even just credibly believed to be vulnerable).

This doesn't mean we have a problem. Sufficiently powerful quantum computers are far away - if they're feasible at all for the enormous number of q-bits needed to solve these problems. This gives us time to slowly migrate to schemes that are actually quantum resistant (by not using ECDSA or similar cryptography at all). This is not yet done, as currently existing quantum-resistant schemes come with very large keys and signatures, and various other caveats. This makes them very unappealing fow now, but research on them is proceeding rapidly, and, if needed, they exist.
Should you use public key addresses?

If one was writing a cryptocurrency would you advise against the addresses being the public key encoded in base58 with a checksum on the end? why?

Advice on other cryptocurrencies is off-topic here, but the Taproot proposal for Bitcoin would effectively do this. Its outputs (and thus addresses) contain a full public key, because it comes with numerous advantages (it's smaller, cheaper, and makes a number of more advanced protocols on top a lot easier).
The Bech32 address format is used for these types of outputs, which has a number of advantages over Base58 (easier to transliterate/compare, stronger error detection, more extensible, smaller QR codes, ...).
Disclaimer: I'm a co-author of both the Taproot proposal and the Bech32 standard.
TL;DR: public keys should be public.


text:
How does a miner put his vote for certain BIP?
I understand that a miner is the only one who can vote for a certain BIP, but I'm wondering, how do they technically do that? How do they submit their vote? I found out that they  are doing it through the coinbase transaction, but I couldn't find any details on it.
Thanks!

See my answer here to address some of your misconceptions. TL;DR: miners signal support in blocks for certain rule changes in order to coordinate activation, not to determine whether it is accepted or not.
As for the actual mechanism used to signal, a number have been used in the past:
Time based: BIP16, BIP30
Early softforks (up to mid 2012) used a simple time based activation mechanism: node software that implements these proposals enforced the new rules on all blocks with a timestamp after a certain date. In the case of BIP16, this date was determined (and modified) in response to miner signalling, but this signalling was just for human interpretation; nodes took no automatic actions in response to it.
Specifically, the signalling used here was to put a string with a support message in the coinbase transaction's scriptSig field, which is otherwise free for miners to put anything in.
Block-version signalling: BIP34, BIP65, BIP66
A later generation of softforks used the block header's nVersion field for signalling (up to the year 2015). Each of those used a subsequent version numbers (BIP34 used version 2; BIP66 used version 3; BIP65 used version 4).
Whenever 750 out blocks number N-1000..N-1 (so 75%) had the proposal's version number or higher, block N would be subject to the proposal's rules. Whenever 950 out of blocks number N-1000..N-1 (so 95%) did, the next block would be required to also signal for it - resulting in a final lock-in.
Versionbits-based signalling: BIP68/112/113, BIP141/143/144, BIP91, BIP341/342
The rollout of relative locktimes (BIP68/112/113) and Segregated Witness (BIP141/143/144) used a different mechanism, which had its own document, BIP9. It specifies using one specific bit of the block header's nVersion field for each proposal, and a finite state machine to determine when to signal and when to activate. Its purpose is/was to permit multiple concurrent proposals to activate, without needing to have one completed before the next one can roll out. This was a downside of the previous mechanism, as it would be impossible to activate the proposal with version 4 without also signalling for activation of the proposal with version 3.
Due to various reasons, segwit was not entirely uncontroversial, and eventual activation happened through a meta-proposal, BIP91. BIP91 itself used BIP9 to activate, which then on its turn made signalling for BIP141/143/144 mandatory, resulting in its activation in August 2017.
BIP341/342 activated in November 2021, using a modified versionbits-based activation mechanism with lower activation threshold, and minimum activation height, in what has been called the "speedy trial" approach.
Disclaimer: I am a (co-)author of several of the listed documents in this post (BIP9, BIP30, BIP66, BIP141/143/144, BIP340/341/342).

Thank you for your answer. Good luck in your future work on BIPs

For 100 blocks, a miner will include its yes or no, number of votes is proportional to the percentage of the hashing power of the network. If you have 20%, you’ll likely be able to vote approx 20 times. Developers make a proposal and miners vote on it. If the majority vote yes, the miners implement the code. Majority in this case is when over 55% of the miners vote for a specific change.
BIPs and this kind of infrastructure for cryptocurrencies in general, has sparked some interesting debates within the community. Some developers fear miners will vote based on vested interest rather than the good of the network.
Additionally there is a growing fear in the community towards the concentration of hashpower (which translates into voting power) as mining pools and big scale mining operations continue to grow. This system is still relatively new, so it will be interesting to see how BIPs progress and ultimately how they affect the development of other cryptos as a result of its successes and failures.

Thank you for your answer, but I edited the question and added "technically" word. I'm interesting in how do they technically vote? Where they add their "yes" or "no"?

They put yes or no (a piece of code) inside the coinbase transaction

As part of the scriptSig?

take a look at the third point of this site and it wil explain it for you [https://medium.com/@dongha.sohn/bitcoin-7-coinbase-and-priority-f5e9f47f1c9b]

Sorry but link is 404

It works fine, use a VPN, and if it will not work, let me know and  i will hyperlink a screenshot of the site

I downvoted. I think this answer misses the point: what miners do in typical softforks is not voting, it is signalling readiness. If the community wants proposal to happen, they can demand it from miners (see BIP148 for example), but things go smoother if miners deploy faster rather than later. See BIP8, BIP9, for modern activation mechanisms.

@PieterWuille i Upvoted your comment beacuse it will help the community, thank you


text:
How do SPV clients acquire transactions from other nodes and verify that they are part of the blockchain?
I explored SPV clients as a subject from a lot of different resources, but there are a few of things that are keeping me confused:

I found out that SPV clients have no mempool even though SPV clients are participating in transaction verification. So, what happens when the node verifies a transaction? Is a SPV client checking only its own outgoing transactions?
Does a transaction that a SPV client is trying to verify have a block hash as a reference where the transaction came from? Is that how a SPV client knows in which block to search for the target transaction when the SPV asks a full node for part of Merkle tree?
A SPV client asks a full node for hashes inside the Merkle tree, so it can reconstruct the part of the tree it is interested in and checks that way if a transaction is really in the block that the transaction is referring to. Is this correct? And if it is, is there anything more these two clients share?
If I'm on the right path, how do bloom filters help in increasing of level of privacy since SPV clients still needs to ask a full node for a transaction by its hash?

Thank you.

I found out that SPV has no mempool, but although SPV client is participating in transaction verification. So, what happens when node verifies transaction? Is SPV client checking only owner's outgoing transactions?

That depends on what you mean by "verification". SPV clients cannot and do not verify the validity of transactions themselves; they can only verify that they were included in a block, relying on the assumption that miners would not waste their energy on creating a block with invalid transactions in it.

Transaction that SPV client is trying to verify has block hash as a reference where did a transaction came from? That's how SPV client knows in which block to search for target transaction when SPV asks full node for part of Markle tree. Is this correct?

Transactions do not (and cannot) contain a hash of the block they're contained it; that would be a cyclic dependency. I don't understand the rest of this question.

SPV client asks full node client for hashes inside Merkle Tree, so he can reconstruct the part of the tree of his interest and that way check if transaction is really in the block that transaction is referring to.

When using BIP37, the client sends a Bloom filter of which scripts/addresses/utxos they're interested in, and then requests blocks. The server responds with filtered blocks, containing only the matching transactions, plus efficient Merkle proofs that those transactions are actually in a block with the claimed hash.

If I'm on a right path, how does bloom filters help in increasing of level of privacy since SPV client still needs to ask full node client for a transaction by its hash?

BIP37 bloom filters are primarily a way to improve efficiency - it means that with little bandwidth overhead, the server knows what to not send to the client. It was intended to optionally offer some moderate level of privacy, because a client can choose to send an overly broad filter, resulting in false positives. It has since been discovered that this basically offers no privacy at all, due to correlation the server can still observe between requested things.

Thank you for your answer, you got me on the right path of understanding SPV now.