text:
Could mining centralization be countered by checking if the node connects to many ASICs instead of one?
In a mining pool, many ASICs and GPU/CPU from all around the world connect to the pool's node. Is it possible to force every node's client software to check if a node is connected to just one computing unit, before the node is allowed to mine?
Or am I missing something about how mining pool node works?
If I am right, some would say we cannot know if the client or hardware are tampered in such a way that the client will always say there is just one computing unit.
How about putting the client and hardware in a Trusted Execution Environment (TEE) so that it cannot be tampered with?

No, this is not possible. Miners must be allowed to join the network without permission, because otherwise the position of a ‘decision-maker’ becomes necessary. This doesn’t necessarily mean the decision maker is just a single individual (or group of individuals, etc), but in any case it presents an additional barrier of entry to the network.
Consider: if I were a miner, why would I allow my node to ‘give permission’ to any other miner to join the network? By not allowing anyone else to join, I would have less competition when mining. So I would be smart to do anything I could to prevent others from joining.
Further, how does the network come to consensus on which nodes are allowed to mine? This is a non-trivial problem, in fact it is mining itself which allows the network to maintain consensus. You would need to create a whole new system and layer of consensus which would precede the act of mining.
Also worth mentioning: even if it were possible, running a node is inexpensive, so there is no stopping a single entity from just running many nodes in order to run many mining hardware devices.
As you mentioned, there is also no way to force a user to use a specific software implementation. A trusted execution environment is not a solution in this case: forcing something like that would centralize the decision of what runs inside the TEE, which is in stark contrast to the permissionless and open source nature cryptocurrencies. Without these properties, the value proposition of a cryptocurrency like bitcoin is destroyed. What if the decision maker or hardware manufacturer is compromised? Situations like that present huge existential risks to the network.

How about putting the client and hardware in a Trusted Execution Environment (TEE) so that it cannot be tampered with?

That's basically the only way you could accomplish this.  But it means that the code would have to be signed, and you'd need some centralized authority to do the signing.  That authority would have total control over how Bitcoin operates.
However, one of the major design goals of Bitcoin was to avoid centralization, so that decisions about how the currency works are made by consensus, not by any single authority.  (You can see some text right in the genesis block expressing Satoshi's skepticism of central banks.)  So you could design a currency that can only be run on TEEs, and it could have your desired mining restrictions, but I think you'd find that most of the current Bitcoin userbase would have no interest in using such a currency.

Hi, I don't understand the code signing part. What code needs to be signed by authority?

@Danny: I guess it depends what kind of TEE you have in mind.  One possibility is hardware that will only execute code if it has a valid digital signature created by a specified key (or set of keys).  Then whoever holds those keys gets to decide what software the device can execute, and the question is who holds them.  You could also have a device where the software is "burned in" and can't be changed - then you have to decide who gets to make the devices.


text:
Why choose the longest chain before checking for double-spending?
Can we make the following changes to the current POW protocol to better avoid double-spending:
a) For full/mining nodes that receive two chains, check if there are identical addresses that occur on both chains that perform double-spending. If there are any, ignore the chain that would result in the node doing a reorganization (reorg), choose the other chain;
b) If there is none, choose the longer chain;
c) For nodes that receive only one chain, accept it if it does not require a reorganization.
Wouldn't this be better in avoiding double-spending than the current protocol, i.e. accepting the longest chain blindly? Shouldn't we check the block contents from both chains before deciding which one to accept?

What do you mean by "that perform double-spending" exactly? Do you mean that the two chains have two different transactions that each spend the same output? If so, are you suggesting every node just stick to the one it saw first?

If there is a double spend attempt, there is no way for a node to know "which chain (transaction) actually came first and is legitimate". Some nodes may hear about the 'legit' chain first, some may hear about the double-spend chain first.
So, if the rule states that nodes work to avoid reorgs in case of a double spend, then we would expect the network to split at the occurrence of a double spend. Consider: what happens if there is a double-spend attempt, and the first chain your node hears about is the one that is soon-to-be the shorter chain? You would be stuck on an orphan chain, while the rest of the network marches onward. This is obviously not good, as network consensus and reliability is important.

c) For nodes that receive only one chain, accept it if it does not require a reorganization.

If the node only receives one chain, then it will never require a reorg. Only a node that receives an alternative, longer chain will reorg.

..accepting the longest chain blindly? Shouldn't we check the block contents from both chains before deciding which one to accept?

I would not say chains are accepted blindly, acceptance requires the chain to be valid according to all of the network's rules. Your node will check the contents of new blocks, but as I mentioned above, there is no way to reliably distinguish a 'legit' transaction from a 'double spend' transaction, at a network-consensus level.

Wouldn't this be better in avoiding double-spending than the current protocol, i.e. accepting the longest chain blindly? Shouldn't we check the block contents from both chains before deciding which one to accept?

Where do you read that the longest chain is accepted blindly? Of course, no chain with double spending in it is valid - that would completely defeat the purpose.
What does happen (at least in Bitcoin Core) is that the check for double spending and signature validity only happens after a particular chain becomes the most-work chain. But it isn't treated as the accepted best chain until it is fully validated. If a double spend or invalid signature is found in a candidate best chain, that chain is discarded, and the selection algorithm will revert to the second most work chain, and fully verify that.
The reason for delaying the double spending and script checks until a block is in the most work chain is to save resources. These checks are many times more computationally expensive to perform than all other validation rules combined, so we perform them lazily.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Are the huge transaction fees mostly mistakes or some other market indicator?
I am trying to understand why there are numerous days in Bitcoin's history that show huge transaction spikes.  See this blockchain.info chart.

The Jan 10th, 2012 spike is due to a mistake by this guy.  I can't explain the other spikes.
Are these spikes mostly or all due to human error? Or are they some other indicator, possibly market value, global usage or popularity?  Maybe some think that paying a high fee is supporting the network.
The number of transactions can obfuscate the actual worldwide usage due to things like mixing services, automated transactions, spam.  Maybe the fees spent can rule those out and show the global adoption rate assuming those types of transactions would avoid fees as much as possible.
Bonus points if you happen to know what specifically caused any one of these individual spikes.  Having a list would benefit Bitcoin history story telling.

As mentioned, raw transactions are incredibly easy to screw up since you must explicitly pay yourself if you want change.
Here's a few scenarios besides unbalanced outputs which will produce huge transaction (Tx) fees:

forgetting to pay change
forgetting to convert to Satoshis in your code
misplacing the decimal (perhaps due to rounding errors)
swapping the fields using BCI or Bitcoin Core (this used to happen easily)
setting the Txfee variable in the core software on top of the built in Tx fee
a miner could spend time mining the current blockchain plus an additional non-broadcasted transaction with a high fee. If the miner succeeded, it would then broadcast and cause a spike in the graph. The miner wouldn't make any extra money, and it would be wasting time not contributing to a pool.

Thanks, it's nice to see the scenarios which could lead to large transactions.
Would it be possible to "game" the chart by paying yourself and collecting the large transaction fee via mining?  Of course it's a risky bet since you aren't guaranteed the transaction fee.

Interesting question. But yes, a miner can do this without risking very much. The key would be to keep the transaction and NOT propagate it or let any other node know about it. When it finds a block, voila, it looks like someone made a mistake, but the miner got it right back anyway.

"without risking very much".  That is intriguing.  It sounds like you are inferring that the miner could intentionally orphan a high transaction fee transaction.  I don't think orphans would affect this graph though, they are forgotten.

@Scott: The proposal is simpler than that. The miner does not post his transaction on the peer to  peer network at all, so no other miner can include it in a block. However, he includes it in the blocks he works on himself. When he eventually finds a winning nonce for the block, his huge-fee transaction is included. Until then, nobody else knows it exists.

@NateEldredge I always wondered whether this occurred as a means of writing off "lost" BTC for accounting purposes

Speaking as a programmer, it's VERY easy to screw up the transaction fee. If you're not using a known bitcoin client and are making your own, it's quite easy to accidentally leave out an output or calculate things wrong. The blockchain won't accept your transaction if you're spending more than the inputs, but will happily accept transactions if you're spending much less than the inputs.
All the errors basically end up going the same way (that is, in favor of the miners), so I'd imagine it's people like me who are testing something or trying to be too clever. This, by the way, is why you should be using testnet for software that's not ready.

Actually, nodes won't relay your transaction if you have more than 10000 times the required fee.

Good point about using the testnet.  I believe most of these spikes are due to programming error.  There have to be other reasons and possibilities though, right?

@NickODell - do you have a reference for the "nodes won't relay your transaction if you have more than 10000 times the required fee"? I can't find anything like that.

It would seem using a large Tx fee is an attack vector in that miners will all realign to try and solve the block at height n, which is quite different to the normal situation where it's usually two chains competing for length. The outcome requires 100% "rational actors" trying to mine the block height n and the ability of the nodes to detect the errantly/anomalously high fee.
A description from the blog of the Freeze Attack, postulating a ฿50 Tx fee:

The freeze problem occurs if someone publishes a transaction with fees
  much higher than the block subsidy. I don’t remember who described the
  attack first. Suppose that, by mistake, a transaction is published
  with 50 BTC in fees. The transaction is included in a block at height
  n. If everyone acts rationally in his own interest, then the best
  choice for the remaining miners is to try to mine a competing block at
  the same height n including the high-fee transaction, to collect the
  fee for themselves. All the miners having solved the block at height
  n, now move on mining at height (n+1). But they won’t choose each
  other branches until one branch is sufficiently longer so that it is
  better to switch to it and abandon their own branch rather than try to
  keep the block with the high fee. This case is different from the real
  block competition case we see periodically on the blockchain, where
  the miners are generally split between two branches. Here there are
  multiple branches competing. If there are 10 “top” miners each having
  10% of the network hashing power, then 10 different branches will
  compete. The analysis for this case is similar to the Gambler’s Ruin
  problem analysis present in the Satoshi paper, but with a fixed
  constant monetary incentive not to switch. Since the incentive to
  switch grows exponentially with the branch length difference, any
  initial constant is diluted. In the special and rare case that all the
  miners have exactly the same hashing power, then the network diverges,
  and this is equivalent as having two miners having exactly 50% of the
  hashing power each. So in principle the freeze on transaction problem
  is just a temporary disruption in the network, but not a fatal halt.
  Nevertheless, since during the freeze period each miner is mining on
  his own branch, it also means that moving forward with blocks is a lot
  slower. Assuming 10 miners having 10% of the total hashing power each
  (+/- 3%), the network becomes 10 times slower. I simulated it with a
  50 BTC tx freeze fee, and 10 miners, and it takes approximately 6
  blocks to converge, so the freeze time is approximately 60 times the
  block interval, or 10 hours. If the distribution is approximately 25%
  of the hashing power for each top miner, the freeze time is 4 hours.

Very interesting attack, have a +1. I'd dispute the "100% rational actors" line though, since a rational actor would likely also consider other factors than the block reward in a past block, such as the lessened security of the blockchain, which has a subjective value which may or may not be priced at larger than this block reward.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Does a transaction always get verified/processed by ALL full nodes in the network before being added into the block by some mining node(s)?
If not, then why is it often claimed that the blockchain scaling issue is due to the fact that all tx has to be verified by all full nodes?
As long as a mining node found the correct nonce, the tx will be added to the block and the blockchain advances. It does not matter if the tx is not yet verified by some full nodes in the network.
(Yes, it will most probably get verified by all full nodes after it got into the block and the block propagates throughout the network. But this is not relevant as the blockchain already advanced (assuming there is no fork).)

Does a transaction always get verified and processed by ALL full nodes in the network before it is added into the block by some mining node(s)?

Technically no, but only because of the way you've phrased the question. In a decentrilized network, there is no way to know what ALL nodes have done.
However, EACH particular full-node (including miners) will verify every transaction in a candidate block before updating their own blockchain state to the new head. Since no one will include the invalid block, it doesn't accomplish anything to produce one.

Does a transaction always get verified and processed by ALL full nodes
  in the network before it is added into the block by some mining
  node(s)?

No, the miners add the transaction to a block by mining the block. The block is then sent to a node, which validates the block, and if it passes, then it broadcasts it to other nodes.

It does not matter if the tx is not yet verified by some full nodes in
  the nerwork.

The transaction must be validated by all the nodes eventually, or it will become extinct and forgotten. "The longest chain wins" means that the chain with the most blocks will be considered valid. This means that (simplifying) if a block was validated by some nodes, but two blocks were validated by more nodes, the two blocks become the new valid blockchain, and the other is discarded. The whole point of proof of work (mining) is to provide a means to establish consensus on which transactions are valid within the network, so eventually all full nodes who follow the rules of the network must validate every new block.

Thanks. I know everything you said here, but still think it doesn't really answer the question. Indeed all tx will eventually get verified, but when there is no fork, blockchain do advance before the latest transactions get verified by all full nodes, right?   When there is fork

Indeed the blockchain grows for any nodes that know about the new block.  But, it does not grow for the nodes who do not know about it. Each nodes stores a copy of the longest blockchain that they know about.


text:
On blockchain scalability bottleneck
It is often said that the major bottleneck of blockchain scalability is that every (honest) node has to process every (valid) transactions. But the progress of a blockchain is how fast it reaches consensus, and this does not seem to require all nodes to verify all transactions as the block containing the longest chain can spread throughout the network before every transactions in the block is verified by all the nodes.

But the progress of a blockchain is how fast it reaches consensus, and this does not seem to require all nodes to verify all transactions as the longest chain can spread throughout the network before every transactions on it is verified by all the nodes.

All nodes having received a block does not mean that the network has come to consensus. Consensus occurs when all nodes on the network agree on what the blockchain is. This can only happen once every node has verified all blocks and their transactions. So verifying all blocks and transactions is actually a bottleneck.

Yes, I was saying that consensus is reached when (the block containing) a longest chain spread throughout all the nodes, not just any block. All node will agree on the longest chain based on its cumulative work.
When a node is validating a block, its acceptance does not require that all transactions in the block must have reached the node beforehand, just that the block and its content follows all the rules.

The validity criteria includes that all the transactions in a block are valid.

@Anonymous , Yes but I am referring to the validation of each transaction when a node receives it, not the one during block validation. (normally each tx is verified twice in the blockchain network)

They’re really not. Validation happens once and is cached if not in a block.

@Anonymous yes but when a block containing the transaction arrives, the node validates all tx in it.

Not necessarily. BIP 152 Compact Blocks allows for blocks to be relayed before they are fully validated. It is possible to have received and then relayed an invalid block because some parts of it were valid. This has happened in the past. If you assume that all nodes have verified all of the transactions in a block before that block is relayed, then yes, when the block has propagated to all nodes consensus has been achieved. There is no need for every node to receive and verify all unconfirmed transactions for consensus to be reached. Bitcoin Core even allows for this by using blocks only mode


text:
What is a good resource to learn about bitcoin
I’ve got to this point reading various blogs, articles, other stackexchange threads and watching Youtube videos on the subject and still have a ton of questions. I don’t expect the questions below to be answered here, they are just for illustrating my level of understanding and what I’m confused about. What I’m really asking here is to be pointed to an online resoursce (tutorial, lecture, course, etc), that can help me answer ALL these questions below:
1) Ok so I keep hearing about creating a wallet. And just about every resource I’ve come across goes into talking about using something like blockchain.info or coinbase or some site. Now to my understanding, the wallet is essentially made up of two components: the private and public key. So creating a wallet (account) on a site means I’m pretty much allowing the site to know what my private key is, correct? Don’t want an ethical discussion here. Technically speaking, they have my private key correct? If so, how is a bitcoin wallet created this way different to online banking except with no insurance, because it’s some random 3rd party site? I must be missing something here, surely...
2) Next I hear about mining. Being rewarded for solving complex mathematical puzzles and what not. “Complex”? Really? Are they like...differential equations or NP-hard problems? So just, trying a new nonce? Is that really all there’s to this “complex puzzle”? Trying random numbers till something works?
3) Then I hear about specialised mining hardware, and mining farms, and what not. Guessing it’s all to do with (2) above. But what about the actual ‘verification’ bit with transactions? Is that automated too now or?
4) “When a group of transactions is verified and the proof or work done, a new block containing these transactions is added to the block chain once it receives sufficient confirmations”. Oh dear lord, where do I even begin asking about this one. Let’s see...which transactions does one ‘select’ to verify and put in the next block, what happens when someone else creates another block containing some of the transactions you’re attempting to verify as well, what actually does a block contain, is it a hash of all transactions or, wtf confirms a block, is that even a thing, who does that, what’s their incentive? Head exploding People invest in this without knowing all these details, really!?
https://m.youtube.com/watch?v=6HFtk29k7-Y
5) “When you send bitcoins to an address, the transaction is broadcast to the network”. What network, how is it broadcast, is there some API that gets called with a request or? Is there some software I MUST download to become a node in this network?
6) Merkel trees. What now?
Ok, so hopefully that gives a good indication of where I’m at. Need help.

This question covers far too many topics to be answerable properly. Moreover, al the questions you have posed have already been answered on this site, it's simply a matter of searching. Beyond that, you could look at Mastering Bitcoin.

Take a look at https://bitcoin.stackexchange.com/questions/22796/i-am-new-to-bitcoin-how-can-i-get-started. Pretty much everything you've asked has been asked before, and there are some great answers on here. Hunt around a bit and do some reading, then feel free to return and ask any remaining (more pointed) questions :)

@Raghav Scouring the web looking for bits and pieces of info across various sites (including this) is how I’ve ended up in this mess to begin with. Hence the question about a single good resource that answers everything from start to finish. Will have a look at mastering bitcoin like you suggested though.

@chytrik Thanks but I’m asking for a single reference, not a link to another questions that itself has suggestions for various links, none of which actually indicate that they answer the precise questions I’m asking.

@AjLearning I honestly suggest just using the search bar on this site, you will find answers to the questions you posted (even if each answer is on its own question). This site is design to be ‘one Q : one A’ for each post, this helps keep it searchable and useful. Expecting all your specific Qs to be perfectly answered in one spot is kinda unrealistic. If you are technical, the mastering bitcoin book is a great resource, I second that recommendation.

@chytrik thanks for the suggestion. I’m familiar with stacexchange sites and the Q&A format, have used these a lot, just not for bitcoin. Like I said though, searching for bits and pieces is how I’ve ended up in this spot. I’m used to referring to a single point of reference for the truth (docs.microsoft for instance for Microsoft technologies). Surely Bitcoin proponents wouldn’t expect people to just go scouring everywhere for information? It’s like a lecturer saying: “no book or lectures for this course, all up to you to find all the info, see you at the final exam”.

@chytrik Will have a look at the madtering bitcoin book though. Seems like understanding the Bitcoin protocol will help resolve some of my questions.

I'm voting to close this question as off-topic because as it shows the unwillingness to follow advices. If OP is willing to pay, then I am happy to provide him with the requested "microsoft" like replies. Even after explanation of logic of bitcoin.SE OP is still looking for a one stop shop, to fit his (egoistic) needs. Really?

@pebwindkraft I would gladly accept “there isn’t a one-stop resource” as a worthy answer...unless of course you are too egoistic to post such a remark for bitcoin?

I would recommend bitcoin book and bitcointalk forum as starting points. The book is very technical, but has everything you need to know. After 2nd+ reading things will get much easier. Forum has huge amount of useful information, but keep an eye on outdated material. It is also good to know whats going on with current BIPs to have some vision about upcoming protocol changes, etc.

The bitcoin book is starting to look like what I’m after. I’ll carry on with it and accept this answer if it suffices.


text:
Why is BIP32 root key different for different standards(purposes)?
When generating key on https://iancolemen.com/bip39/, the BIP32 Root Key changes as I change the derivation path tab, e.g. BIP32, BIP44, BIP39 etc. Why is this the case? Isn't BIP32 Root Key the m at the beginning of the derivation path and the different standards are subsequent purpose that should not affect the root key?
Contrary to the above, when I generate addresses with the following code in go, the master key printed always matches the BIP32 Root Key on iancolemen.com. And changing the purpose does not affect the address generated in the end, it is always a BIP32(legacy) address, instead of a segwit which I would expect from BIP84.
m, err := hdkeychain.NewMaster(seed, chainParams)
if err != nil {
    panic(err)
}
fmt.Println("bip32 root", m.String())

purpose, _ := m.Child(hdkeychain.HardenedKeyStart + 84)
coin, _ := purpose.Child(hdkeychain.HardenedKeyStart + 1)
account, _ := coin.Child(hdkeychain.HardenedKeyStart + 0)
receiving, _ := account.Child(0)
index, _ := receiving.Child(0)
fmt.Println("address", index.Address)

EDIT: The wrong address printed in go was a result from not properly encoding the addresses to bech32 and by default they are interpreted as legacy addresses. The keys themselves are alright.

If you look closely, you'll notice that BIP 32 and BIP 44 actually have the same root key and it begins with xprv. Then BIP 49 and BIP 141 also have the same root key and it begins with yprv. BIP 84 has a root key that's different from the others that begins with zprv. Now if you were to take all of those root keys and Base 58 decode them, you'll find that they actually are identical except for the first 4 bytes and the last 4 bytes.
The last 4 bytes are because of the checksum used in Base 58 Check encoding.
But why are the first 4 bytes different? They are different because BIP 49 specifies for extended keys for nested segwit addresses should begin with the prefix yprv and ypub. Even though this is specific to BIP 49, this has been more generally applied to all extended keys for nested segwit addresses, thus you see it used for BIP 141. Then BIP 84 specifies that extended keys for native segwit addresses should begin with the prefix zprv and zpub.
These prefixes are what cause the different root keys as changing the prefix will cause the entire serialization to change as well.
By this logic, the xprv and xpub prefixes used previously then indicate that legacy addresses should be made. Hence BIP 44 shows xprv.
However BIP 32 is what specified the xprv and xpub prefix, and really this prefix is supposed to be independent of the derivation paths used and the addresses to be derived. BIP 32 is solely concerned with keys, not addresses. So xprv and xpub are used for general BIP 32 key derivation stuff too, which can make this confusing. But that's also why the BIP 32 tab shows xprv.

any reference on how the pub and prv strings are generated that you can kindly point me to?

The serialization of extended keys is specified in BIP 32: https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki#Serialization_format. For the different prefixes, there are just different version bytes and those can be found in their respective BIPs.


text:
Can not decode transaction. Script was manually verified successfull
I can not decode transaction in Bitcoin-QT.
The error is: TX decode failed (code -22). But blockexplorer site decode it.
Then I try to send transaction from blockcypher.com site, and the error is:
Error validating transaction: Error running script for input 0 referencing c806c9ae2ac9c71fad307c9fedeca2133edb195cae9e924424885f57a63ba9a9 at 0: Script was NOT verified successfully..
Please help what is wrong. I check transaction structure by hand and check signatures. It's all ok.
I upload signed and unsigned transactions on pastebin.
Unsigned transaction: https://pastebin.com/XfE73U57
Decoded unsigned transaction: https://pastebin.com/RH2tNrNn
Signed transaction: https://pastebin.com/K15c29tK
Decoded signed transaction: https://pastebin.com/Mfs0bXgR
I check signatures and structure of transaction by hand, and they are all correct. Inputs are unspent. So what is the problem? Please help me.

when looking at https://www.blockchain.com/btc/tx/c806c9ae2ac9c71fad307c9fedeca2133edb195cae9e924424885f57a63ba9a9?show_adv=true, I can see that the transaction's input spent the funds already to other addresses. What do you get with listunspent?

pebwindkraft, I little bit confused about what are you asking for, because the 5 outputs of this transaction are "unspent"

I thought the error message indicates s.th. with the first transaction and it's input zero. Before going into further analysis of sig and scripts, I wanted to make sure, that this "outpoint 0" really contains funds. However, when I was looking at blockchain.info, I thought I can see, that the funds might have been sent already. So the output of "listunspent | grep -A9 -B1 $tx_id" would have helped to verify current status for the error message... If you like to, you can edit your original question and copy&paste output of the command, or use pastern again.

All inputs of my transaction are the unspend outputs from the one transaction c806c9ae2ac9c71fad307c9fedeca2133edb195cae9e924424885f57a63ba9a9. The outputs of this transaction are unspend (5 outputs). I try to edit my question so to clearify the problem that I really misunderstand.

@chupacabra something happened? similar error

Your transaction has 4 extra bytes at the end.
Just remove the 01000000 and it should be fine then.

I think the unsigned raw transaction is not setup correctly. The unsigned tx has 5 times the pubkey script in the tx_in[0-4] section. It should have only one for each signature. How I understood the system, the way to go is to set all input scripts to nothing, and run through a loop for each of the five inputs, creating 5  separate signatures.
The first loop will have the pubkey script for tx_in[0]. All others remain empty. Then you calculate a valid signature for tx_in[0]. Then in the next loop you again set all to nothing, and only for tx_in1 the pubkey script. And so on... I try to display it this way:
set all scriptSig fields to empty, and length of field to 0
Loop 1
  only tx_in[0] with pubkey script, generate sig[0]
Loop 2
  only tx_in[1] with pubkey script, generate sig[1]
Loop 3
  only tx_in[2] with pubkey script, generate sig[2]
Loop 4
  only tx_in[3] with pubkey script, generate sig[3]
Loop 5
  only tx_in[4] with pubkey script, generate sig[4]

This will give you the signatures, now you place each signature in its scriptsig field, and adopt the length fields accordingly.
I have seen similiar info here and here.


text:
What hierarchical deterministic derivation path is used for FIDO authentication on a Ledger Nano S?
What hierarchical deterministic derivation path is used for FIDO authentication on a Ledger Nano S? Does it ask for a different key each time or is it always the same key?
If I connect to Google for 2FA can it ask for proof of whatever key path they desire or does it have to be a particular key?
I'm wondering how much privacy I lose if I use a Ledger Nano S that is not exclusively for a particular accounts 2FA. E.g. What happens if I use it for multiple Google accounts, other companies, or have crypto currency. Could Google discover a Bitcoin balance if the same Ledger is used for 2FA?

On Ledger the derivation path appears to be 0x80553246:
https://github.com/LedgerHQ/app-u2f/blob/3204324cb72eb00ad1a1919b5c0312edb3324de6/include/config.h#L22
However since the derivation path is not part of SLIP-44 it might not be compatible with other brands of HW wallets.


text:
Do I have to download all blocks to see my bitcoin using bitcoin-qt?
I saw a youtube video about bitcoin wallet and he said you can't see your bitcoin or send or receive until the bitcoin-qt download all the blocks.

In order for your client to show your bitcoins, you'll at least have to download and process up to the block that validated the transaction with which your money was sent to you. (The client can't show you, what it doesn't know about.)
Bitcoin-QT will allow you to send bitcoins after that, even when it is not synchronized completely yet. Since likely your funds have only been sent to you in the near past, you will have to download almost the complete chain to reach that stage.

For bitcoin-qt, the answer is yes.
For Multibit, the answer is no.

Only if you use a client that downloads all the blocks.  Multibit and some others don't.  Also, you can check your balance on sites like blockchain.info (though you're putting your faith in their honest reporting).

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
What is the best way to test new consensus protocol?
I wanted to test a new consensus protocol. However, the bitcoin core is a large codebase (>100k lines). Is there any small codebase which is designed to test new consensus protocols and includes only essential features of a cryptocurrency?
Goal: Run a testnet with a few AWS-EC2 instances across the world for payments using UTXO model.

What is the goal or scope of your test? What are the key differences between it and bitcoin (or other cryptocurrencies) that you hope to test?

@chytrik: I have updated the question. I just want to change the consensus protocol of Bitcoin to support high throughput for payments.


text:
How do I move the blockchain file location on Windows?
I tried to edit the storage information for the blockchain. Neither the Windows 7 nor Bitcoin qt could do it (it says that the app can't open the configuration file).
So I tried to edit the -datadir in in Bitcoin qt. The app doesn't give me a response for the command (I thought it had worked but the blockchain was yet being download in my desktop).
I don't know how to create the symbolic link that someone has wrote in another question. Can someone give me some information that I may had misread?

Go to Settings > Options and click the Reset Options Button. Note that this will clear all of the options shown there, so note down what they all are so you can set your options again later.
In the Dialog that appears, click Yes and Bitcoin Core will shut down. Now go to the Bitcoin data directory and copy the entire thing to wherever you want it to be. The data directory on Windows is located at %APPDATA%\Bitcoin. Move the contents of the data directory to the location you want Bitcoin Core's data to be.
Start Bitcoin Core again. You should be prompted to choose the data directory. Choose a Custom data directory and set it to wherever you moved the data directory to. Then set your options again.

The simplest way worked haha, thanks Andrew.

Will reseting options affect pruning?


text:
Is there anyway to produce the transaction's hash created by sender?
It seems that when someone wants to send Bitcoins, he creates a transaction and the transaction_id will be transaction's hash. after block confirmation this id will be change and we can find out this id in the transaction explorers, I want to access the transaction id created by sender(first id) to verify the transaction by r, s and sender's public key that saved in block chain, but to verify this transaction I need the hash that created by sender, there is too many references that I can use to calculate the final transaction id but I need the id that created by sender to verify transaction. I wrote some implementation according to Bitcoin's wiki but it was wrong.
I need an exact reference or a function that shows how to calculate the transaction id that calculated by sender.
for example in a random transaction  with id
adb1b4723f24841b671b1e979571929f7fc1db9f3884eee2e3fec782bb3ff0a6
we can find the r = 0x0082a19fd532b25c79831c8a8e4802cd35ccabe41469c1c46386ce3fa9f60fde and the s = 0x04fa4e1593c94c130b0ebb1a57d390367d797191af0cb201b3a68fe683bc393b
the public key of sender is pub = (0x6bb502c0eb0a69caec1d649259f5ec3286bef74e2d1cf4cf2b3657be7242d5f9, 0x63f0ab38e0cb9399dcb386e64a67188d0acdc49363583893190e3f950fae5c38) if we run the verification algorithm it will return false if we take h = tx_id  = 0xadb1b4723f24841b671b1e979571929f7fc1db9f3884eee2e3fec782b
because r_prime = 0x9fc47f1afdb0f2012e373bb6e55e7f208afd5c162d538f36d6420daaf800f21 and it is not equals the r!
I test this senario with other transactions but the results was the same!

Hi Mohsen, a transaction does not change by being confirmed. There are some ways how third parties can modify the hash of a transaction with non-segwit inputs, but I suspect that your problem may stem from the misunderstanding that a transaction id always changes by getting included in a block. Could you please edit your post to clarify your question in light of this information?

Hi @Murch thanks for the comment. the post is updated.

It is not clear what you are trying to do in your example. It sounds to me as if you assume that the message signed by a signature is the txid, but that is not the case.

@Murch, maybe it is because of my misunderstanding. I should read the documentation of Transaction of bitcoin with more details. Thanks for your comments BTW.

Perhaps the chapter on signatures in the recently published 3rd Edition of Mastering Bitcoin can clarify: https://github.com/bitcoinbook/bitcoinbook/blob/6d1c26e1640ae32b28389d5ae4caf1214c2be7db/ch08_signatures.adoc#L52 (the whole book is available in that repository).

Thanks for the reference, it will be useful for me.


text:
Why does this transaction results in only 2 unspent output (and no spent output)?
This transaction is from one of the latest block. However I don't understand why it seems to be just dividing a single transaction without being spent and transferred to someone else.

I don't understand why it seems to be just dividing a single
  transaction without being spent and transferred to someone else

Only Unspent tx i.e UTXO could be spent, and here in your case this  tx is UTXO which after spending to addresses
13MDsVCs3AvSxZxjWUwty2gthPAXkYf6wb - (Unspent) 0.00928291 BTC
1CPp812W35rarR45CS6EYxdC4JazR4H9Wn - (Spent) 0.00231133 BTC

now becomes spent tx

Why does this transaction results in only 2 unspent output (and no
  spent output)?

This is because lets say this is  your address: 1CPp812W35rarR45CS6EYxdC4JazR4H9Wn where you want 0.00231133 btc, so its one of the output address.
So after sending 0.00231133 to your address leftover balance i.e 0.00928291 need to be stored some where so in that case  13MDsVCs3AvSxZxjWUwty2gthPAXkYf6wb becomes your change address where remaning btc left after spending utxo is sent. Thats why you have 2 unspent op

I swear to god when I was posting this questions, the page was showing that both outputs are unspent. Now one of them became spent. Is there some kind of delay here?

no there is no delay as such, what happened is when you posted the question 13MDsVCs3AvSxZxjWUwty2gthPAXkYf6wb was unspent only, but later in tx: a314d0b1f3f08a07cccca64865d4fe4654b3fe86dabd019438895d2dd0f75d0d this address was used as Vin which result it to be spent in tx: 7253e818ba712f98654ca1fc03191027e590dcac06eda36e7b13dc835422b67a

Understood. the output only became spent if it is used in the subsequent tx. But let's not talk about the subsequent transaction and just look at this tx (7253e818ba712f98654ca1fc03191027e590dcac06eda36e7b13dc835422b67a) alone. Why will the user want to make a transaction when the outputs are not spent immediately? (I assume that no amount is sent to anyone else at the end of this transaction)

Its because your transaction is “unconfirmed” until the next block is created. Once that block is created and the new transaction is verified and included in that block, the transaction will have one confirmation. Approximately every ten minutes thereafter, a new block is created and the transaction is reconfirmed by the Bitcoin network. It is common for six confirmations to be required which takes about an hour.

I don't get the link here. Why will the user want to make a transaction when the outputs are not spent immediately? Only to be truly spent on the subsequent transaction? is it like a test run to see if he can do transaction? but that will cost him transaction fee no?

Thanks @cryptoKTM, I figured this out: the biggest misconception I had with regards to unspent transaction output is that, the word 'unspent' actually is according to the recipient's perspective, not to the sender's perspective.
This means that if a transaction results in multiple outputs that says 'unspent', it means all of these outputs have not been used up, or spent by any recipient. It DOES NOT mean that the sender (who has the inputs' addresses) did not spend and transfer the bitcoin successfully.


text:
Are BIP39 mnemonics generated with a French list compatible with software that expects English words?
BIP39 specifies different word lists in different languages. But I wonder if the word list generated by a software that gives me French words (e.g. Copay on iOS) can be used to re-create the same wallet using another software that expects English words.
I looked at the section From mnemonic to seed of the BIP39 specification.  Given that the mnemonic sentence itself is fed through the PBKDF2 function, rather than first looking up the indices of the words in the dictionary, I guess that the answer is no, and a French list can't be easily converted into an English list. Is this correct?
Doesn't this drastically reduce the interoperability of wallets using different word lists?  Specifically, how can I ensure that the word list I have for a wallet can be used in another software than the one it was generated with?

The short answer is no.
BIP 39 hashes the complete set of wallet BIP 39 words (typically in order specific sets of 12, 15, 18, 21, or 24 words).  The complete set of BIP 39 words in different languages is documented at https://github.com/bitcoin/bips/blob/master/bip-0039/bip-0039-wordlists.md.  Results of the BIP 39 related hashes shall differ radically if even if the list indices for different spoken languages match. This means internationalization interoperability between the BIP seed words between the different spoken languages supported by BIP 39 are impossible.
In other words, if you pick English stay with English BIP 39 seed words. If you pick French BIP 39 seed words, stick with French. Absolutely do not mix and match languages. Otherwise, the BIP 39 standard will be useless for restoring your funds in a HD wallets.

Given that the mnemonic sentence itself is fed through the PBKDF2 function, rather than first looking up the indices of the words in the dictionary, I guess that the answer is no, and a French list can't be easily converted into an English list. Is this correct?

That is correct. You can not convert a mnemonic generated using one wordlist to another since the mnemonic decodes to different byte arrays. For example the third English word is 3 bytes (able) while the third French word is 8 bytes (abdiquer). And consequently the seed (that you use in BIP32 functions) is going to be different.

Doesn't this drastically reduce the interoperability of wallets using different word lists?

Usually when wallets implement BIP39 they implement all of it including all the wordlists that are linked in the protocol improvement. So they can remain compatible

Specifically, how can I ensure that the word list I have for a wallet can be used in another software than the one it was generated with?

You have to check those wallets individually to see what they have done, either by checking their source code on GitHub or their website, or asking the community about for help. Lack of compatibility doesn't stop there.
Usually there is a page explaining it clearly. For instance Electrum uses an entirely different scheme although it still supports BIP39, or LND and wallets based on it use unique scheme called aezeed


text:
Can you make a water heater by mining Bitcoin?
I read somewhere that you can actually cut down your heating bills by using bitcoin mining based water heaters or air heaters . Is that possible? Can someone give detailed economics on this ?

Related: https://bitcoin.stackexchange.com/q/2283/516

More or less... computer chips produce heat, so sure, it is theoretically possible. First you should know that, compared to a heat pump, a miner's efficiency is zero: heat is the byproduct, not the main goal, so it would take much more energy for the same amount of heating than a decent electric heat pump. But since you are generating Bitcoin in the process, we hope it pays off (wouldn't cut down your bills, would increase it instead, but you would have Bitcoins to pay for the increase, hopefully).
The management of the setup and the economics would be hard: an AntMiner S9 consumes 1200W (and consequentially, can heat water by this same amount). You would have to replace the casing and air cooling system with a kind of water cooling system, using water from your hot water tank as cooling fluid.
Then there is the problem: if you water gets to the desired hot temperature, it won't serve anymore as a good cooling fluid. Then you have two options: if you live in a place where bitcoin mining pays off the electric bill, you can mechanically switch the cooling fluid from the hot water tank to an alternative water circuit, this one designed to dissipate the excess heat (an ordinary PC water cooler radiator, for instance), so that you won't have shut down your miner.
On the other hand, if bitcoin mining is economically unsustainable per se, you would have to shut down the miner when the water is hot. In this case, you would have to be sure that the cost of operating the miner minus the bitcoin it produces is lower than operating a regular heat pump (which is more energy efficient, as I said). Not to mention the cost of equipment and installation and R&D to get everything right is much higher with a miner than with a heat pump designed for this purpose.

Sure if u mine with some reference blower style GPUs you will automatically heat up your house with no effort involved

Welcome to Bitcoin SE!  I hate to complain to a newcomer, but this is a pretty weak answer.  You provide no explanation of what a "reference blower style GPU" is, don't mention the futility of bitcoin mining with a GPU, and provide no information of the costs of running a miner vs the cost of traditional heating.  It doesn't do a good job of answering the question.  I recommend that if you take the time to answer a question, you take the time to make that answer useful.  You'll earn a lot more rep that way.

authors: Серёга Дмитриев


text:
getrawtransaction seems to only work for certain transactions
I am working through mastering bitcoin. I am confused by getrawtransaction. I would like to look at the first transaction in block 277316.
from bitcoinrpc.authproxy import AuthServiceProxy, JSONRPCException

# rpc_user and rpc_password are set in the bitcoin.conf file
p = AuthServiceProxy("http://%s:%s@127.0.0.1:8332"%(username, pw))

# look at specific block
blockheight = 277316

# get the hash of that block
block_hash = p.getblockhash(277316)

# get the block
block = p.getblock(block_hash)

# get the transactions in the block
transactions = block['tx']
first_transaction = transactions[0]
print p.getrawtransaction(first_transaction)

Gives this error:
bitcoinrpc.authproxy.JSONRPCException: -5: No such mempool transaction. Use -txindex to enable blockchain transaction queries. Use gettransaction for wallet transactions.

It seems like it only wants to look in the mempool. How do I use the -txindex parameter it suggests?
If I run the below code, it works fine. This is also a transaction from the same block (64th transaction) and clearly not in the mempool. What is the difference between this and the how I get the transaction id in the above code?
print p.getrawtransaction('0627052b6f28912f2703066a912ea577f2ce4da4caa5a5fbd8a57286c345c2f2')

getrawtransaction is only guaranteed to work for mempool transactions, unless you're running with the -txindex command line flag, or txindex=1 in your bitcoin.conf file.
In practice it works for all transactions that have at least one unspent output remaining, but it's relatively slow, and functionality that may be removed at some point.

Thanks. Is there a better way of doing this since you mentioned this is slow?

Use a specialized indexing service, there are several projects doing that. Bitcoin Core's intention is validating blocks and transaction, with the least amount of resources. Maintaining an index to query arbitrary transactions does not fit into that.


text:
Difference between downloading bitcoire core from bitcoin.org and compiling from Github
I'm following instructions to install Bitcoin Client in Andreas Antonopoulous' Mastering Bitcoin.  In Chapter 3, the author details options of installing the bitcoin core from bitcoin.org or compiling it from the source code.  From a practical perspective, what is the difference?
What do I lose by just installing it from bitcoin.org?  (Since that seems easier than making sure that I have all required dependencies in Linux)

The main advantage of building from the source is that you have the option of manually inspecting the code to make sure you trust that no one has inserted any nasty logic (in practice this won't be feasible unless you are already familiar with the code).
A second advantage is that if you want to get the latest and greatest improvements for your bitcoin software. The releases on bitcoin.org occur approximately every two months so if there was a change you'd like to use before the official release and building from source will let you do that.
If you were to checkout the source on branch 0.15 and tag v0.15.0.1 and build that then theoretically you'd be running the equivalent logic to the latest release (as of 23 Sep 2017).


text:
How Do I Calculate the profit of the value of my bitcoin?
Lets say I buy $50 of a bitcoin for 3916.74, And then the price goes up to $4200. How Do I calculate how much I have made in profit?
As in What type of formula do I use to see how much more my $50 is worth now that the price is higher than when I bought it?

I'm voting to close this question as off-topic because it's a basic math question.

The simplest way to calculate would be:
averagecost = ((BTC[t0] x USD[t0]) + (BTC[t1] x USD[t1]) + (so on))/total_BTC
where t0,t1 and so on are different time when you bought.
The above gives you the average cost for all your BTC
Now to know the profit, use the below formula:
profitloss = (current_BTC_price - averagecost) x total_BTC
If profitloss is negative, then you have a loss.
If profitloss is positive, then you have a profit.

First of all, let a number of bitcoins be bought and (x1) is how much you paid for it.
Now the price of 1 bitcoin changes
Then you use the formula:
x2= (New value of 1 bitcoin)*(Number of bitcoins you have)

Now you just find the difference between x1 and x2. This will give you your profit/loss.
All you have to remember is the initial value. In your case it is 50.
Therefore,
x1=50
your bitcoins, in this case, = 50/3916.74 ~ 0.0128
x2= 4200*0.0128
      =  53.76
so your profit x2-x1= 3.76 is your profit

Be a lot nicer of someone could break this down simpler than the two answers shown, For example.. let’s say you buy $50 at $7500, and the price goes to $9000. What’s the profit margin? Should be $1500 right? That seems pretty simple.

But instead, it’s like $63 or something. Would love a formula that isn’t as ridiculous as the ones posted above.

@ElSols: No, $53.76 is correct. The profit margin is not the increase in the exchange rate, but the increase in the value of your own holdings.


text:
Is it safe to skip the initial block verification at startup?
The longest step in bitcoin core initialization is "Verifying last 6 blocks at level 3". The number of blocks checked at startup went down from 288 in 2014 to 6 now. Therefore, in order to speedup bitcoin core initialization I am thinking about lowering this number further to 2-3 with the checkblocks option.
However, I wonder how risky this is? In case bitcoin core shuts down incorrectly, isn't it unlikely that all the last 6 blocks become corrupted? Isn't 2-3 blocks enough?

I think Bitcoin Core should load the entire UTXO set into memory or some sort of SSD cache during the initialization...


text:
How to verify if a transaction is signed correctly?
Given an arbitrary signed raw transaction, how can we easily verify if all inputs are correctly signed (assuming all inputs are existent/unspent and the fee is higher than zero)? Bitcoin core's RPC command testmempoolaccept will check if all inputs are available to be spent in the mempool/blockchain so it's impossible to test transactions that have parents not yet broadcasted.
I am aware that for this kind of check, the scriptPubKeys of all inputs needs to be known and therefore only the signed raw transaction by itself is not enough for this kind of check. Still, the scriptPubKeys could be passed to the transaction instance or verify method. I was looking for some nice way to do this in python/javascript but was surprised how difficult this task is:

Peter Todd's python-bitcoinlib requires each input to be validated manually with some flags indicating the type of ScriptPubKey.
1200wd bitcoinlib, has a transaction.verify() method but it seems to only verify the basic transaction types: coinbase, P2PKH, P2SH(P2PKH), P2SH(multisig), ...
BitcoinJS has a method for this, psbt.validateSignaturesOfAllInputs() but converting a raw transaction to PSBT is not possible/easy.
Bitcore's transaction.verify() does mostly some basic sanity checks and doesn't verify the signatures.
BitcoinJ's transaction.verify() does also basic verification and the script execution, as python-bitcoinlib, has also to be done manually with verification flags.

Pubkey script alone is not enough to verify a transaction, you'll need:

Pubkey script, to evaluate the scripts
Amount, in order to check if the sum of inputs is bigger than or equal to sum of outputs
Index (in TxOut list) and tx hash of the transaction being spent
Block height so that you can set the consensus rules

After gathering all the above (should be found in the node's UTXO database) you can start evaluating if the provided script is valid which isn't limited to signature verification but running/evaluating the script that includes checking correctness of the script, OP codes, OP count, script size,...
A lot of the verification can be found in validation.cpp file.
This is also how I do it in Bitcoin.Net library.

Forgot about block height, which is important for nLockTime... Thank you for sharing code from your project but I would prefer using some more battle-tested code for now, Seems like this task is more easily accomplished in C derived languages (although I was trying to steer away from them), just reminded me that it might also be possible with libbitcoin: https://github.com/libbitcoin/libbitcoin-system/tree/master/include/bitcoin/system/impl/machine


text:
What valid transactions are not broadcastable?
Many valid transactions are not broadcastable by default in bitcoin core to avoid attacks that could harm the network. These would rarely be found in a node's mempool but could, however, be found in blocks because they don't violate any consensus rules.
What transactions fit these criteria?

Another one that comes to mind is a transaction with too many unconfirmed ancestors

Hi Pedro, our platform tends to work poorly for questions that preconceive the answer. You may get more interesting answers if you reduce this post to the question portion and post your suggested answer as an answer post. Self-answers are encouraged! This would give other users a better chance to write a comprehensive answer rather than responding to the answer you already preformulated in your question.

Thanks for the suggestion @Murch , I moved it that part to an answer.

The following transactions are not broadcastable with the default bitcoin core configuration but could be found in new mined blocks:

Transactions whose fee is smaller than 1 sat/vB (and RBF bumps that increment less than 1 sat/B).
Transactions with size greater than 100 vkB (and smaller than ~ 1 vMB)
Transactions with outputs amounts smaller than 566 satoshis (dust limit)
Transactions that are considered non-standard (I will try to list them because this is a very broad category)
Transactions with scriptSigs larger than 1650 B (and smaller than 10kB)
Transactions with more than one OP_RETURN output or an OP_RETURN output bigger than 83 B.
Transactions with scriptPuKeys of bare n-of-m multisigs, where m > 3.
Transactions with scriptPubKeys or ScriptSigs that make the transaction malleable.


text:
What bitcoin core options can be changed without a restart?
I often experiment with new bitcoin core options (like zmq, debug and fee options) and I would like to avoid restarting my node for them to take effect.
If I am not mistaken, changing the RPC password, for example, can be done without restarting the node but most options require a restart.
What are the options that require (and don't require) a restart to take effect?

Debugging print options can be changed without a restart.

Thank you for the answer, won't accept it though, because I am hoping for a more complete answer.

That's the sum total of options you can change.

Are you sure those are the only ones? I'm disappointed, I was hoping there would be more.

The configuration options are things like how much memory can be allocated, how much caching to use, if peers should be sending transactions or not. Being able to configure these on the fly would come at an absurd cost of complexity, for very little in the way of use cases. What would you like to be configuring on the fly?

For now: zmq URLs and watermarks, maxmempool, mempoolexpiry, minrelaytxfee, maxuploadtarget

Wallets can be loaded/unloaded at runtime now (which is arguably changing the "wallet=" option).

I'm not sure I'd even consider loading wallets on the fly to meaningfully be different than taking down the node and adding it to the configuration. The majority of the time it will require a rescan, which hogs cs_main, and you desync from the network.

@PieterWuille that just reminded of 'addnode' which can be done with the corresponding RPC command.


text:
Distribution of tx fees
Do tx fees issued by users follow a particular distribution ? I'd guess they follow the normal distribution, but I'm not sure. More precisely, how tx.fee/tx.size is distributed in the mempool ?

I don't think that the fees would follow a normal distribution, but I haven't done the math. You can find a visual representation of the mempool at https://core.jochen-hoenicke.de/queue/#24h.


text:
Change in hash rate fraction of mining pools
There are pages that show how the cumulative hash rate of network changes over time however, I wasn't able to find any that shows how hash rate fraction of pools change over time. Is there a page that traces this metric?

You could try this page for the current share and keep a record?
Bitcoin Hashrate Distribution


text:
The ratio between propagation delay and block interval
Is there a certain website or a study that gives an estimation for the
ratio f = Propagation Delay/Block Interval for the Bitcoin network ?
I know the Block Interval is roughly 10 minutes however, I don't know the propagation delay. I define it as the time it takes for all nodes to hear  about a mined block after it's been broadcast.

Probably related: https://bitcoin.stackexchange.com/q/10821/31712

As mentioned by Ethan Heilman here:

According the paper 'Information propagation in the bitcoin network.' by Decker and Wattenhofer. The mean time for a node to see a block is 12.6 seconds, after 40 seconds 95% of the nodes have seen that block.

From my personal experience as a developer, this delay is real and noticeable. Some block explorers catch transactions ~20-30 seconds before others. It may be scary at first but its normal.

This is unrelated. Transaction relay is deliberately slowed down in order to improve privacy (to hide the source of a transaction, random delays are added). This is not the case for block propagation which is very optimized to be as fast as possible.

He asked about propagation times.

I assume he means propagation time of blocks. Transaction relay time is not relevant for network convergance. But you're right, it is a guess.


text:
Blockchain size problem
According to my understanding, the ledger/blockchain is stored on every computer (node) that partakes in the network. The beauty of this system is that it is decentralised and backed up by every single participant in the network. Furthermore, the security of the blockchain comes from there being so many copies, as long as there are many copies, then it's unfeasible for anyone to try to "hack" it, as too many other people have copies which would be automatically checked against your fake one to reveal the deception. On average a new block is created every 10 minutes. Right now there is a limit of 1 MB per block. So, if the limit isn't changed, then the growth rate is limited at:

1 MB every 10 minutes (on average)
6 MB every hour (on average)
Approximately 144 MB per day
Just a bit over 1 GB per week
A bit less than 4.4 GB per month
A bit less than 52.6 GB per year

How is it feasible for someone with capped internet to participate in the blockchain network as it starts to grow over time? For example, the Internet in third world countries is limited, expensive and slow. Are there any solutions out there or research being carried out which resolve the scalability problem which is currently an inherent property of Bitcoin?

I don't see why you think there's a problem. The growth rate is constant (assuming the block size doesn't change). And Internet speed is going up while storage costs are coming down. Why would you think it's not sustainable as is?

Thanks for the response. I think that what you have described best fits a model scenario where all blockchain participants are living in a developed country but it is unsustainable for those who are not. On my trips to Southern Africa, I have found that the cost of data and hard drive storage is very high unlike in Europe - I should have perhaps added this point to the question.

I don't see how that changes anything. If those people can't participate today, how is it a scalability problem? And if they can participate today, why wouldn't you expect their storage costs to continue to come down too?

Unless of course we are envisaging the Bitcoin as a digital currency better suited for those in developed countries. If a node is unable to host the full blockchain, I suppose that it will need to depend on another one with the full transaction history. In the long run, won't we see a loss of "financial sovereignty" and "reduced security" in these regions?

Perhaps it isn't best described as a "scalability problem". Wrong use of words. I guess what @amanusk described below would apply in such scenarios. And as he has also declared, the real scalability problem is how many transactions can take place every 10 minutes.

SPV clients are thin clients that do not need to hold the full blockchain. They use other nodes on the network to validate transactions. SPV clients are available for Android and require a lot less disk space.
Of course it is more secure for you and for the network to run a full node.

This is an incorrect use of the term scalability. What you're trying to say is that there is a capacity problem. Scalability is about how the resource costs of a system increase with increased capacity, and in the case of Bitcoin, scalability is terrible because of the points made in OP's question: every participant needing to see everything means that increased capacity comes at a huge cost factor.

I see. I was not aware this is refereed to as a capacity problem. Removed it as it is not directly related to OP's question.


text:
Double spend problem on a delay tolerant network
I recently read about the Digitally project, an overlay payment scheme for use on mobile phones, whose goals are to extend mobile phone payments to areas with poor connectivity and reduce transaction fees.
In summary: DigiTally enables two people to do a payment transaction by simply copying short strings of numbers between their phones. This doesn't need either smartphones or a network connection. Either phone can upload the transaction later, once it gets a network connection.
I see the benefit of such a system working in underdeveloped economics, wherein branchless mobile banking is a way of life. Alice and Bob can engage in an offline transaction which is later verified and added to the blockchain once either of them get internet access.
Are there any technical implementations on Bitcoin or any other cryptocurrency which allow for asynchronous transactions such as the above? If not, then how can we imagine it would look like?
Is there a solution to the double spend problem that works on a delay tolerant network?

Yes - in a nutshell...

As explained in the referenced paper [4] "SMAPs: Short Message Authentication
Protocols", the security issue is the delay-tolerant networking. Bitcoin already requires a minimum of 6, or better 8, blocks before a transaction is sufficiently secured. The trade off is between delaying the settlement and lowering the risk.

I’m sorry but could you please clarify, and thanks for introducing a new dynamic to the discussion - how would bitcoin work on a delay tolerant network?

Transaction is secured when is stored into the blockchain. If we are offline our transaction is not settled until one of us go online and notarize the transaction to a Bitcoin node. This aspect adds a delay to the usual time that we must wait for the settlement. More delay is equal to less security.

Is it possible to temporarily debit the senders account and credit the receivers account so that the funds are available for use immediately? If so, what are the risks that could arise?

Funds are not available until settlement. Risk is the same of cheques. https://en.m.wikipedia.org/wiki/Cheque?wprov=sfla1


text:
How do I import an Electrum wallet into BitcoinJ?
I'm trying to create a BitcoinJ-based app that watches an existing wallet (created in Electrum) and does things when transactions come through against that wallet.
As I understand it, I need to create a BitcoinJ wallet by feeding it my Electrum wallet's "Watching Key", and it appears that, to do this, I need to somehow load my Electrum wallet into BitcoinJ so that I can call getWatchingKey() on it.
The problem is that I cannot sort out how to import my Electrum wallet into BitcoinJ - I have BIP32-formatted public and private keystrings (from testnet, so they start with tpub and tpriv respectively), but I simply cannot sort out how to convert these strings into a Wallet object in BitcoinJ. Any help would be greatly appreciated.
EDIT
Alternatively, I could import the Key by using the Seed...but I'm supposed to supply the Seed's creation time (in Unix form)...what is that information used for? I don't have it, so I've been working under the assumption that I cannot use the Seed.

As stated in https://bitcoinj.github.io/working-with-the-wallet:
"The “watching key” in this case is the public form of the BIP32 account zero key i.e. it’s the first key derived from the master."
I also thought it was the master public key, but it is the first (or 0th) derived key.
Hope this helps.
EDIT:
To answer your last question, the seed's creation time is used as a hint to BitcoinJ as to when it should start searching for transactions. If you have access to your first transaction on your Electrum wallet, use that approximate time.

Is there a way to get this Zero Key from Electrum? Your answer, while helpful, does not actually answer my question (not trying to be rude, I truly appreciate the help)

Sorry. I don't know the answer to that one.

I face up the same problem(still I can not reach my balance) when I was trying to have access to my wallet through a BitcoinJ application.
Despite the fact that my bitcoin address had created successfully from my private key, still I was not able to have access to my funds. I do not know why.
Here is my code, hope it will help you:
 //create my ECKey
        DumpedPrivateKey dumpPrivKey = null;
        try {
            dumpPrivKey = new DumpedPrivateKey(params, "your private key. You can reach it by calling the dumpprivkey RPC command in Bitcoin-Core console");
        } catch (AddressFormatException e1) {
            // TODO Auto-generated catch block
            e1.printStackTrace();

        }
        ECKey myKey=dumpPrivKey.getKey();
        Address bitcoinAddress=myKey.toAddress(params);

       System.out.println("My bitcoin address is : " +bitcoinAddress); //the
    Main network address
    Coin coin =wallet.getBalance();
    System.out.println("My bitcoin balance is: " +coin); //the balance


text:
Clarification of some getaddr() function inputs and outputs
In the Bitcoin system to discover new nodes, each node communicates with its peers by sending getaddr() to the peers. Based on the bitcoin implementation:
//! the maximum percentage of nodes to return in a getaddr call
#define ADDRMAN_GETADDR_MAX_PCT 23

//! the maximum number of nodes to return in a getaddr call
#define ADDRMAN_GETADDR_MAX 2500

Based on Eclipse Attacks on Bitcoin’s Peer-to-Peer Network ∗

The peer responds with n randomly selected addresses from its tried and new tables, where n is a random number between x and 2500 and x is 23% of the addresses the peer has stored.
The reason behind this is to eliminate an eclipse attack (aka netsplit), where an attacker monopolizes the environment of a node by sending it compromised addresses.
The receiving node accepts a maximum of 1000 addresses because it is limited to a buffer size that can only hold 1000 addresses.
What is the 23% or the 2500 addresses based on?
Why is the buffer size limited to 1000 addresses?


text:
Is there a maximum balance that a single address can hold?
Is there a maximum number of satoshis that a single address can control at a single time? I ask because when I generate regtest blocks to a single address, it's balance gets stuck at 1494999998350 satoshis (14949.99998350 BTC), and doesn't increase anymore from that point. All balance for that address is from coinbases, not vouts.

Nope. Not 1 bit. A balance (in this context) is the sum of all UTXO's for a given address. There is no limit on the number of UTXO's or their total amount (from the perspective of how the blockchain works and the blockchains limitations).
Software interpreting the balance may have limitations on consuming, processing and/or displaying a number beyond a point. Int32 vs Int64 for example: https://stackoverflow.com/questions/9696660/what-is-the-difference-between-int-int16-int32-and-int64
Here's an address with a balance of 138,660 BTC at the time of this post:
https://longhorn.bullpay.com/?gateway=2&data=3D2oetdNuZUqQHPJmcMDDHYoqkyNVsFk9r
Note: This answer is specific to the question and specific to Bitcoin. For the sake of brevity situations where "creating" tokens or currency on another chain in amounts that could theoretically "break" anything is not covered.

In addition, the reason the balance tops out is simply because the subsidy goes down much faster on regtest.


text:
Can I mine cryptocurrency using Javascript?
Is it a good idea to mine a Scrypt-based cryptocurrency, like Dogecoin or Digitalcoin, using Javascript instead of having paid ads.
The script would mine using all computers visiting my website.

Are these computers yours to run inefficient software on? (ie are you coding with nefarious intent?)

Could the people downvoting this explain their criticism please? While this may be a bad idea for reasons mentioned in the answers, it is a valid question. So, if you downvoted because of the potential nefarious intent of the asker, instead please explain why it's a bad idea in an answer, or upvote any you find helpful. If you downvoted for another reason, feel free to explain your reasons in order to give the asker a chance to fix his question.

Even if your visitors didn't mind the slowness of your website, you would make orders of magnitude less by mining than paid ads.

Today this might not be viable for technological reasons, but ideas like this actually show solid foresight because you can bet things like this will begin to appear over the next 5 years.

I see where you are coming from and I wouldn't be surprised on some systems if the user has less of a bad experience than ads (some sites filled with ads are simply unbrowseable for some).
But, Javascript is not a quick language and you wont generate any coins worth generating, no pool will allow you to direct this sort of traffic to them in any significant quantity and if you run your own p2pool the bandwidth and other resources will far outweigh the coins mined.
As a rough but educated guess that you'll probably generate about 1-2kh/s per user while they are on your site, meaning you'd need 250,000-300,000 simultaneous viewers to match a modern scrypt mining rig, take into account overheads and you're probably looking at more like 400-500,000 viewers.
In other words, don't do it. Your users will hate the site, it will lag video and audio and generally create a poor browsing experience meaning users won't stay on your site meaning you wont get the required hundreds of thousands of viewers. You may also get flagged for malware by Google etc. and like I say, pools wont accept it.

I don't quite understand your question.  Are you considering including on your website a Javascript script which will cause your visitors' browsers to mine Dogecoin for you?
This is not a good idea.
First, it will likely be extremely ineffective.  Mining Dogecoin and other scrypt-based coins generally require a high-powered GPU and the ability to run code directly on it.  As far as I know, you can't do this through Javascript.  You can implement the mining algorithm in Javascript, and run it in the virtual machine, but it will run many orders of magnitude slower than on proper hardware.  I would not expect this to earn you any significant amount of money.
Moreover, it will annoy your visitors.  Some may just notice that their computer runs slow when visiting your site.  Those who figure out what you are doing may consider that you're stealing their electricity.  Some would call this malware.  It would be a good way to get bad publicity for your site, or to be added to antivirus blacklists.
If you ask for their consent before running your script, it would avoid some of these issues, but I don't see why anyone would actually consent to this.

Eh, I'm torn on this. On one hand, I would rather not have my computer use extra power. On the other hand, I would also rather not look at advertisements. You can say that mining uses my computer's power inefficiently, but the same is true of advertisements: they use my time and attention inefficiently.

@NickODell I'm sure javascript couldn't use any more cpu mining than a video ad.

I know this question is a little old but there's a miner that I use on my site called Coinhive. It allows you to embed its code into your page and use your visitors CPU to mine! It works too! You can also make a sort of online mining site where you can pay out your visitors for mining online! A recommendation, however, is to not do this secretly. Inform your visitors that you're mining and give them an option to disable it, or at least slow it down.

How profitable is this method? Is it a viable alternative to placing ads on a website?

@jimmyplaysdrums It depends on how much traffic your site is getting, and how long your users are generally going to spend on your site. As Coinhive says, if you're a tiny blog with 10 visitors a day that spend 5 minutes on the site, you won't be making much. However, if you host an online game on your site that your visitors generally spend a couple of hours on, it will be making more. Again, it all depends on what your site is for.


text:
Does the inability to spend only part of a transaction result in "too many" extra transactions?
I understand that I can only spend the amount of a full transaction associated with an address. Often this will result in change being sent to a change address that I provide. One question at this point is, why does the receiver bother sending back the change. Do you send way more than the original cost for an item?
Say they do, don't they have the same problem, not having a transaction for the exact amount so that they too might end up sending too much to you, so that a cycle of change happens - what could have been one transaction ends up being many which seems like this is a burden on the system?

Why does the receiver bother sending back change?

This assumption is wrong. Nevertheless, you are correct saying, that you need to spend the whole transaction input (UTXO) balance on your address, but the person who makes the change is the sender, not receiver.
Example, You have unspent balance (UTXO) of 1 Bitcoin and you want to pay 0.2 BTC for coffee. You create a transaction spending 1 whole Bitcoin to two separate receivers. 0.2 BTC (tx output) goes to merchant and 0.7 BTC you direct back to your address (the change). The difference between UTXO and sum of tx outputs is called fee, in this case 0.1 BTC.

Does the inability to spend only part of a transaction result in “too many” extra transactions?

No, since it is still 1 transaction with just two output in case there is a change. No extra transaction is made.

Thanks. I was under the impression, just like in the real world that if I had a 10 dollar bill and I gave it to pay for a 1 dollar purchase, it would be the receiver giving me change. But I in the case of Bitcoin give the change to either a new address or the original address, right? I was really baffled -- it seemed for a moment like this was at least an awkward feature of Bitcoin -- was it that dumb of a question?

@Jeff Many newcomers are having trouble understanding this, specifically non-programmers, otherwise it makes perfect sense. The real world application would be: I have 10 dollars, purchase for $1, so I tear 1/10 of the note, which goes to cashier and 9/10 of banknote is still mine. Not dumb question, but you might have found the answer just by googling.

I did indeed google and while I found a lot of info about change addresses and maybe even some examples that implicitly showed your answer none that I saw came out and said, "You might think that the receiver has to give you change, but that is wrong..." Thanks again, I was baffled for a bit.


text:
importing all blkXXXXXX.dat
I've moved my bitcoind process to a new server, I made a backup of my .bitcoin folder (I'm using linux). I would really like to avoid downloading the whole blockchain again.
What's the right way to import all the blk00000.dat files? Is it enough to move the .bitcoin folder over the new one?
EDIT: I see the Loadblock option but I suppose I should run it after move the file outside.Anyone can confirm this? Is there a better way?

Unfortunately, I don't know that myself, but these questions cover similar topics and might help: 1, 2, 3, 4

Have you tried --rescan?

-rescan is for finding missing wallet transactions in your existing block chain. It doesn't affect the blockchain itself.

just copy chainstate and blocks to new server.

I've just posted this in response to another question, but will post again as using bootstrap.dat is by far the best way to get a node up and running.
If you are downloading the blockchain from scratch I strongly recommend using the bootstrap.dat file, which allows your computer to simply verify blocks and import them without needing to download for days or even weeks on end.
The main issue with downloading from the network is that your client will only download one block at a time. It starts with block #1 and will not try to download anything except for block #2 after block #1 is verified, and so on. As you can imagine, with network latency this can become a rather painful process, as you have endured.
The bootstrap.dat file is widely available on bittorrent, torrent files can be found here: http://sourceforge.net/projects/bitcoin/files/Bitcoin/blockchain/
Direct torrent link: http://sourceforge.net/projects/bitcoin/files/Bitcoin/blockchain/bootstrap.dat.torrent/download
Downloading the bootstrap.dat file is just as secure as downloading the blockchain from peers, but since all the data is already there, the only latency you suffer between blocks is going to your hard drive and finding the next block.
You simply need to place the bootstrap.dat in your %appdata%\Bitcoin folder, (on linux/unix in your .bitcoin folder) the file will be renamed bootstrap.dat.old when the data import is complete. Using this it took me only 4 hours to catch up with the network, as opposed to the 1-2 weeks it took me before reformatting on the same computer.

Technical tidbit here, the bootstrap.dat file is actually just a concatenation of the blkXXXXX.dat files. You can literally build your own bootstrap.dat by doing something like cat blk*.dat >~/bootstrap.dat

Copy these files to a exact Bitcoin Data folder on your Ubuntu (~/.bitcoin/) and then run client to download the latest blocks available. To synchronize with the blockchain run bitcoind with command -reindex bitcoind -reindex.

On the Windows, I just replaced the directories and it worked fine.

Thanks, under linux seems to cause some issues. I'm using bitcoind, not bitcoin-qt but I don't think there are any differences.

It should work if you copy the blocks and chainstate directories.  You should also copy wallet.dat if you want to keep your old wallet.

If you are not changing the program as such, you should be able to place the blockchain into the same folder as the original.  We do this quite successfully in Windows.  We have also (for other currencies) zipped up a blockchain and had it downloaded onto a client which had sync problems - in order to fix the problem.  Similarly it should work the same way for Linux.

I would strongly recommend using bootstrap.dat or creating your own as described here: http://bitcoin.stackexchange.com/questions/18568/how-do-you-generate-a-bootstrap-dat-file


text:
Why some transactions disappear from the mempool?
Just before block 537562, I had the following transactions in my mempool:
 'a4ca904361952c8a27bcf1fd36e5bc8ccdc5d42ceb72cfba60c017fb9cbbb0e5'
 '69718e60b6e7d20d5e56e097b2de9789007fb230d0205a90a1411ef8086672a0'
 '346b93674d91bfc5b5568f7e77dc969b5c87351135a1c75d551c1418533164f2'
 'e8fddfc634f2e3d535ad81d7dce314e543047bc0281854d9eded4d107fa4f525'
 'da2fd9ed1a8ba7612b885377dc7018d8d0c35ad4cdd03a24829b9a0b8edbf123'
 '3318d59ac00f5dbdd223fcf9ab953e0eee2765b4d1d72285c33c75121e888eee'
 'fc4aea4f7f3ae94bf65c6cc23b73e594f7c849ab5f1362ac78d157cf4af8bc8a'
 '1e7b6ebefd9cd4983f598831ed8082bcdab6b7857f2d302e7d4952e4f145f8e7'
 'f0f20dcf5142895d19140b60830bd9f15b664294d8535c3dcc09ebe3398a2fa3'

These transactions simply disappeared from the mempool later without ever being mined. Why did this happen? At a time when fees are so low, I doubt they were the target of a replace-by-fee. What are the possible reasons for a mempool transaction to disappear? and is there a way to investigate what happened to these specific ones?
EDIT: these transactions had a feerate above the default minrelayfee of 1 satoshi/Byte
EDIT2: I admit that I was not aware of question How do transactions leave the memory pool? , but my question is still different. I am interested in finding out why a specific transaction has disappeared from the mempool without ever being mined. Now it's a bit late to see if the transactions that I gathered were replaceable or not (nLocktime equal to 4294967294 - 0xffffff-1 - if I am not mistaken), but I will repeat my analysis taking this into account in the future.

Possible duplicate of How do transactions leave the memory pool?

Posted in https://old.reddit.com/r/BitcoinBeginners/comments/9afuer/why_some_transactions_disappear_from_the_mempool/ and kindly answered by /u/bitusher

@PieterWuille I edited my question to try to explain why my question is bit different, although there is some overlap. Thanks for pointing out the other question.

At the very least you van try resubmitting the transaction. The result may teach you something.

Those transactions were not mine... I simply took all the mempool transactions before block 537562, and subtracted all transactions that were mined in the next 9 blocks. I was left with the transactions I posted which have disappeared from my mempool and apparently from the blockexplorer's mempools too. But yes... I can take the whole transaction info and check where the conflict is and if I don't find any conflict, resubmit it...

Skimming the source code, I found the following reasons for a transaction to be removed from the mempool:

It was included in a block
A conflicting transaction (i.e. one which spent at least one of the same inputs) was included in a block
Expired: it has been in the mempool longer than the time specified by -mempoolexpiry (default 336 hours)
The memory pool size limit was reached (set by -maxmempool).  Lowest fee transactions are removed first.
A blockchain reorganization occurred (i.e. a different branch of the chain became the best), and this transaction spent coins that are now immature, i.e. the block where those coins were created is no longer at least 100 blocks deep
A blockchain reorganization occurred, and the new chain is shorter, such that the block height mentioned in this transaction's locktime has not yet been attained
A blockchain reorganization occurred.  Transactions from disconnected blocks have to be added to the mempool.  If it's already full, in this case, oldest transactions are removed to make room (not lowest fee in this case).
Under replace-by-fee, it was replaced by one with a higher fee

If someone knows of others that I've missed, please edit to add them.
If we could find the transactions themselves, we might be able to guess why they were removed.  For instance, maybe we could look at their inputs and see if they were spent by some other transaction.  But when I looked up the first few of those transaction IDs at blockchain.info, I got no results, so it may be that the block explorers have dropped them as well.

Thanks for the answer, is there any resource to check for these accidental forks or blockchain reorganizations as you called it? Would like to see if anything happened during that time.

Look in your debug.log file.

Do you have any template text that would appear in debug.log? so I can grep it?

I'm not sure offhand, but each new block added to the chain gives an UpdateTip message.  If you see one whose height isn't one more than the previous such message, that would indicate a reorganization.

I know that this is old question, but I have noticed the same behaviour.
In most cases, I have noticed that transaction are removed due to RBF.
eg. txid 6762…0284 was deleted from mempool, because it is replaced by bffa…22a2.
As of this writing bffa…22a2 is unconfirmed so it could still be replaced as well.


text:
Claim Bitcoin Gold from my paper wallet
I have asked similar questions in the past, But I didn't get a clear answer yet. I have my bitcoins in a paper wallet, Everywhere I am reading that exchanges like yobit,hitbtc etc will support bitcoin gold if the bitcoins were in their wallet before the fork happened. Which in my case is still in my paper wallet. So now, to which wallet I should import/transfer so that I can claim bitcoin gold.

You can't. The blockchain has just forked, but the clients are not ready. They still haven't added replay protection, so the "specs" is not ready  too. When they release the client, you'll be able to import your paper wallet private key and claim your Bitcoin Gold.
Now, just don't fall for "claim your bitcoin gold" scams.

Well, the Bitcoin Gold (BTG) blockchain explorer is up and running now, but when I entered a public address from one of my Bitcoin paper wallets that had Bitcoin on it at the time of the alleged fork/snapshot in October, an error message is coming back effectively stating that not only are there no BTG funds on the wallet, but that there never were any on it. I also attempted to sweep the same paper wallet through the BTG wallet for the Coinomi wallet application, and the exact same error message appeared.
I did remove the actual Bitcoin from my paper wallets following the time of the alleged fork/snapshot for BTG, but the BTG balances - if there were any balances - should show up through the blockchain explorer. I know that with the Bitcoin Cash fork, my BCH coins were all showing up on the separately respective BCH blockchain explorer both before and after I had removed the actual Bitcoin from them.
So my first guess here is that this whole Bitcoin Gold thing is a nice little scam and that there never was any so-called "snapshot" made of any public addresses; and that no BTG was ever allocated to those public addresses. If there had been, then at least a "history" of those funds being on the wallets should show up through the blockchain explorer regardless of whether maybe the funds could have gotten stolen through some alleged "replay attack."
SCAM, SCAM, SCAM, Bitcoin Gold! I'll never think any different at this point...

Or, it might just be a bit buggy still (or you made an mistake somewhere along the way). Time will tell I suppose...

It's my understanding that if you had BTC during the snapshot that the BTG developers took in October you will be able to claim an equivalent amount of BTG using your private key of the address your BTC was in at the time of the snapshot. The Coinomi wallet claims to be able to allow you to do this but we won't know for certain until tomorrow after the BTG chain starts getting publicly added to.


text:
Is it possible to get all transactions from a single address with bitcoind?
Is it possible to get all transactions from a single address with bitcoind?

For an address that is in your wallet, you can use the listreceivedbyaddress command. You would use the command as follows to get the transactions related to an address <address>:
bitcoin-cli listreceivedbyaddress 0 true true <address>

The 0 means list all transactions, including unconfirmed ones. The first true indicates that the address should still appear in the output even if there are no transactions; it just won't have any transactions listed under it. The second true means to include watching only addresses in the output. This only matters if your address is a watch only address.
If the address is not in your wallet, you cannot get all transactions for an address. This is because that would require maintaining an address index which Bitcoin Core does not do. Address indexes require more computing power and are not useful to the functioning of the node.

I have written a program in Python3 that allows you to search for any address on bitcoin core whether it belongs to your wallet or not.
Here is the github link:
https://github.com/ORP967/Bitcoin_Core_RPC_par_address
Let me know what you think or if you have any improvements you might have.

yes. try this command:
bitcoin-cli -regtest getrawtransaction  1.
This command will give you all transactions for the required address.

That is not at all what getrawtransaction does. It gives you the raw transaction for a specific transaction. It has nothing to do with addresses.

You can use command "listtransaction".
http://chainquery.com/bitcoin-api/listtransactions
Good luck =)

No, that only lists transactions that affect your own wallet.


text:
bitcoind is refusing to connect to my other server
bitcoind is refusing to connect to my other server

By default bitcoind only binds RPC to the loopback interface, so you can only connect to it from localhost.  This is for security.
You can allow connections from a specific other machine with -rpcallowip, or in general with -rpcbind, but beware that this substantially increases the attack surface.  Moreover, the data sent and received is not encrypted, see Why is JSON-RPC over SSL "strongly discouraged"?.
It's recommended instead to use VPN software to set up a tunnel between the two machines.  In such a setup, bitcoind will see the RPC connection as coming from localhost.

I will try this soon. Thanks. I am running bitcoind on a VPS. I am not sure exactly what you are saying in the last part, but do you mean you recommend running bitcoind and the site on the same server?

@2523454: You can do that, and it might be simpler, but it's not what I'm talking about.  I'm saying you can set up a virtual private network such that communications from the remote server are forwarded over an encrypted connection, and it's the VPN software on the bitcoind server that actually talks to bitcoind.  Pretty standard network admin stuff.

I already am running it on a VPN. I have 2 VPNs, one for website one for daemon.

I suspect you mean VPS (virtual private server) not VPN.  You would need to set up a VPN between the two VPSes.  How to do this is beyond the scope of this site.

Yes, I meant VPS. I'll try the -rcpallowip flag in about a hour and accept this if it works.


text:
estimatesmartfee outputs very different results
Around block 564815 (2019-02-26 23:40 UTC), the mempool was empty. I noticed that my two nodes are giving very different estimates for transaction fees to be used:
Node 1

Version: 17.1.0
Number of peers: 21
Synched for: about three weeks
Prunned: 3GB

// see how many transactions are there in the mempool
 >bitcoin-cli getrawmempool | wc -l
 4027

>bitcoin-cli estimatesmartfee 5 {
    “feerate”: 0.00021195,
    “blocks”: 5 }

>bitcoin-cli estimatesmartfee 5 ECONOMICAL {
    “feerate”: 0.00009715,
    “blocks”: 5 }

Node 2

Version: Bitcoin core 17.0.0
Number of peers: 8
Synched for: about 5 hours
Prunned: 3GB

# see how many transactions are there in the mempool
>bitcoin-cli getrawmempool | wc -l
3999

>bitcoin-cli estimatesmartfee 5
{
    “feerate”: 0.00008823,
    “blocks”: 5
}

>bitcoin-cli estimatesmartfee 5 ECONOMICAL
{
    “feerate”: 0.00001000,
    “blocks”: 5
}

The node that has been synchronized for only 5 hours and is less well connected seems to be estimating the fee better than the one that has been synchronized for three weeks. Why is the "best" node the one that gives the worst fee? Why are the fee estimates so different and what can be done to make them more accurate?

The node that has been synchronized for only 5 hours and is less well connected seems to be estimating the fee better than the one that has been synchronized for three weeks. Why is the "best" node the one that gives the worst fee?

Just because the fee is lower does not mean it is better.
The fee estimator tries to estimate the fee the balances between cost and time to be included in a block. It does this by looking at the fee rates of transactions that it receives in its mempool and seeing how long it takes for transactions at a given fee rate to be included into a block. It becomes more accurate as it sees more transactions and more blocks because it has had more data to analyze.
So your two nodes are giving wildly different numbers because of their uptime. Node 1 has been online for much longer. It has seen more unconfirmed transactions become confirmed thus its estimator has had more data to observe.
However Node 2 has not been online as much so it doesn't have as much data. It hasn't seen as many blocks and as many transactions become confirmed. Since it has a lack of data it resorts to the fallback values. These values are the minimum relay fee rates which are not likely to actually be the true fee rate. The minimum relay fee rate is not the minimum to be included in a block, it is the minimum for a node to even relay your transaction to another node.

what can be done to make them more accurate?

Just wait. Leave the nodes online. The estimators require data and the only way to do that is to wait. They become more accurate over time.

I was wondering if I copied .bitcoin/fee_estimates.dat from the node with higher uptime to the node with lower uptime, it would make them equal? If not, is there any way of exporting the feerate estimate data from one node to another? And how long is enough for the fees to become accurate? I assume there is a cap after some days of running, no?

It seems that the trick I suggested really works, as morcos said here: https://bitcoin.stackexchange.com/questions/59854/where-are-the-new-smart-fee-estimate-data-saved-can-the-file-be-re-used
I was wondering what happens if a node hsa been running for a very long time and then it's stopped and restarted one month later. Are the estimates from one month ago decayed so much they are negligible or they still bear a heavy wait because it was "just before it restarted"? Also, is there a way to deserialize the information in fee_estimates.dat? I'd like to take a look at the information inside.