text:
How do I to get the r, S and Z values from a raw transaction (version 2)
For example I have this raw transaction
0200000000010177858f84b8534b69d5ce161ce709854f18d3a37b607abf570f301eb3d6e0d2520100000017160014f8513401ea5e9dcd57597e8a736f162572d19079feffffff020f68a5000000000017a914d55600283b297e12a0a8e1a92da7c03c0bcb6c528780412942000000001976a914ed5268cb6853e1934f5ec3c1eb637e1ac8ca5fa688ac0247304402207db22867e47fd73b43d44d2faf51f492c7e13b0356e64928e4ee5ccb8c5654b3022046b569f15fae82c6c9ffe91e98c63004974aab0582e7296ef2cec555fc329ee30121037ef7c159605d43e78d4c1ad53b53e60e46bcc504ad9bca6c33fd889fcc324eea00000000

How do I extract the r, S and Z values from the signature?
I'd be happy about a step-by-step explanation.

Your question is somewhat unclear. I presume you are using Bitcoin Core and want to extract the r and S values from the ECDSA signature(s) in your raw transaction. I don't know what you mean by "sig Z". The edited question is clearer now. I learned about the Z-value only after answering the question and thus don't address it here.
Bitcoin Core offers a decoderawtransaction RPC that you can use to decode the transaction into JSON.
bitcoin-cli decoderawtransaction 0200000000010177858f84b8534b69d5ce161ce709854f18d3a37b607abf570f301eb3d6e0d2520100000017160014f8513401ea5e9dcd57597e8a736f162572d19079feffffff020f68a5000000000017a914d55600283b297e12a0a8e1a92da7c03c0bcb6c528780412942000000001976a914ed5268cb6853e1934f5ec3c1eb637e1ac8ca5fa688ac0247304402207db22867e47fd73b43d44d2faf51f492c7e13b0356e64928e4ee5ccb8c5654b3022046b569f15fae82c6c9ffe91e98c63004974aab0582e7296ef2cec555fc329ee30121037ef7c159605d43e78d4c1ad53b53e60e46bcc504ad9bca6c33fd889fcc324eea00000000
{
  "txid": "de527fc755b4307c1f615cceda469c0a8057b0f29e73f86b543a98ce5d978462",
  "hash": "08b7d919d467558638a7f50e8011a3dfe62769d7f01a35d5b87b66712fe62740",
  "version": 2,
  "size": 249,
  "vsize": 168,
  "weight": 669,
  "locktime": 0,
  "vin": [
    {
      "txid": "52d2e0d6b31e300f57bf7a607ba3d3184f8509e71c16ced5694b53b8848f8577",
      "vout": 1,
      "scriptSig": {
        "asm": "0014f8513401ea5e9dcd57597e8a736f162572d19079",
        "hex": "160014f8513401ea5e9dcd57597e8a736f162572d19079"
      },
      "txinwitness": [
        "304402207db22867e47fd73b43d44d2faf51f492c7e13b0356e64928e4ee5ccb8c5654b3022046b569f15fae82c6c9ffe91e98c63004974aab0582e7296ef2cec555fc329ee301",
        "037ef7c159605d43e78d4c1ad53b53e60e46bcc504ad9bca6c33fd889fcc324eea"
      ],
      "sequence": 4294967294
    }
  ],
  "vout": [
    {
      "value": 0.10840079,
      "n": 0,
      "scriptPubKey": {
        "asm": "OP_HASH160 d55600283b297e12a0a8e1a92da7c03c0bcb6c52 OP_EQUAL",
        "hex": "a914d55600283b297e12a0a8e1a92da7c03c0bcb6c5287",
        "reqSigs": 1,
        "type": "scripthash",
        "addresses": [
          "3M92sq9ssFaNbEwF47uteVKJsbw125juS7"
        ]
      }
    },
    {
      "value": 11.10000000,
      "n": 1,
      "scriptPubKey": {
        "asm": "OP_DUP OP_HASH160 ed5268cb6853e1934f5ec3c1eb637e1ac8ca5fa6 OP_EQUALVERIFY OP_CHECKSIG",
        "hex": "76a914ed5268cb6853e1934f5ec3c1eb637e1ac8ca5fa688ac",
        "reqSigs": 1,
        "type": "pubkeyhash",
        "addresses": [
          "1NdqsNEXYx9RnpoPqdkfW7hmZiDuX8BrgK"
        ]
      }
    }
  ]
}

From the JSON response you can tell that you transaction has one input and two outputs. The transaction has a Nested-P2WPKH input and a P2SH and P2PKH output.
You'll find the ECDSA signature in the input. Since the input spends a SegWit output the signature is a witness element (in the JSON a part of txinwitness).
The witness contains two elements.
txinwitness:
1. 304402207db22867e47fd73b43d44d2faf51f492c7e13b0356e64928e4ee5ccb8c5654b3022046b569f15fae82c6c9ffe91e98c63004974aab0582e7296ef2cec555fc329ee301
2. 037ef7c159605d43e78d4c1ad53b53e60e46bcc504ad9bca6c33fd889fcc324eea

In Nested-P2WPKH (or in general all script templates from the P2PKH family) the first witness element is the ECDSA signature and the second is the corresponding public key. All newer ECDSA signatures (since the activation of BIP-66 in summer 2015) must be strictly DER encoded. For more information on the DER-encoding see for example: Why the signature is always 65 (1+32+32) bytes long?
Decoding the DER encoded ECDSA signature (first witness element) yields:
DER marker:       30
Sig length:       44
r-value marker:   02
r-value length:   20 (32 byte)
r-value:          7db22867e47fd73b43d44d2faf51f492c7e13b0356e64928e4ee5ccb8c5654b3
S-value marker:   02
S-value length:   20 (32 byte)
S-value:          46b569f15fae82c6c9ffe91e98c63004974aab0582e7296ef2cec555fc329ee3
SigHash:          01

Both the r-value and the S-value are big-endian integers.

I believer Z is what some people call the integer calculated from hash of the message in ECDSA algorithm. ie. double SHA256 of the modified transaction.

To Add onto the previous answer by 0xb10c, Z is the signature hash. It is the hash of the message being signed, and for a transaction, that is the hash of the transaction with empty ScriptSigs. For Witness transactions, this is the hash of the transaction, as the Scriptsig is already empty.


text:
Is it normal to get 60 confirmations for a transaction in blockchain
Is it normal to get 60 confirmations for a transaction in blockchain and still it is not received and confirmed, can you help I m worrying as this is my first transaction, did I forget anything

Confirmations never stop. Confirmations is simply the number of blocks that have been mined on top of the block that contains your transaction. It will keep incrementing as long as blocks are being mined.

Is it normal to get 60 confirmations for a transaction in blockchain and still it is not received and confirmed

All wallets I'm aware of show even unconfirmed transactions, and definitely show confirmed transactions with even 1 confirmation. If your wallet is not showing the transaction you are:

Connected to a node that is not fully synced for a light wallet.
Using a wallet that is attempting to scam you
Behind some kind of internet restricting proxy or firewall that prevents a wallet from communicating with nodes
Using a Bitcoin full node directly that is not fully synced.

Many thanks but it already took 7 hours for confirmation, is it normal or I have already lost this amount unfortunately, I ve used copay is it safe, as my status is still pending, what are the steps I need to check, thanks again for big help

Are you sending or receiving? Copay is a fairly large wallet, and not known to be a scam. They are unlikely to be affected by the points mentioned above. If you look up the txid on an explorer, and it shows confirmed, then the transaction is complete. Try moving your keys to another wallet, such as electrum, and see if it works.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is Blockchains 12 word recovery passphrase process
Hello thank you for looking.
I am trying to discover a simple way of creating blockchains 12 Word Recovery Passphrase, they have a so called help page on this subject which is of very little help and profoundly obscure, your help would be greatly appreciated.
Kind Regards
Richard Norman

You secret key is a set of 256 random ones and zeros (bits). This is very difficult to remember. So, we need to represent those ones and zeros differently.

Create a checksum of the private key by taking the first (entropy-length/32) bits of its SHA256 hash.
Add the checksum to the end of the random sequence.
Divide the sequence into sections of 11 bits.
Map each 11-bit value to a word from the predefined dictionary of 2048 words.
The mnemonic code is the sequence of words.

Source: https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki

Your question is not clear as to whether you are interested in recovering a wallet or if you would like to write software implementing a mnemonic HD wallet.
If the latter is true, BIP 39 is the original proposal, explaining mnemonic recovery phrases as a seed for HD wallets, and references some examples in various languages to get you started. Chapter 5 of “Mastering Bitcoin” by Andreas Antonopoulos offers a detailed technical explanation of HD wallets and mnemonic recovery phrases.
As for recovering a wallet from a mnemonic phrase, many popular cryptocurrency wallets offer this option, and can be found through an Internet search (I can not recommend any and would advise you to do your own research on which best suits your needs).

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
BTC not received after sending
I have a transaction sent from Coinbase on 12 Dec 2017.  i can see it on the blockchain, fees taken but the rest is unspent.
How can I force the transaction to complete.
Why didn't it complete
Answera in laymans terms please as I'm new to this
Thanks

Unspend just means that the part you received (this part is called a transaction-output) is not spend yet. Your transaction is complete.
It would show spend if you e.g. payed someone with your transaction-output. Then they would have an unspend transaction output.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Does finding neighbors in Bitcoin consider locality and hop count?
When the list of neighbors is being gathered by a new nodes in the network, does it consider the distance between the nodes to find closest possible, or it just takes some random nodes? In other words, are neighbors in bitcoin network, close to each other or they are usually departed from each other by many hops?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Unconfirmed Transactions, help please
I sent some bitcoin to another adress, i paid regular fee and it already more than 28 hours that its not confirmed. Can I do anything?
Why does it need that much time to confirm?
On this page (https://bitcoinfees.earn.com/#delay) there was written like 19 hours waiting time. But nothing happened.
Here is my transaction
https://blockchain.info/tx/4c4402a5439fdfcf2cd3d9bbf0c36fbae91a3ce08504da317938eed2d1a125e1

There is not mutch you can do. The easiest solution would be to wait until the transaction confirms.
Your transaction is not eligible for Replace-By-Fee. With RBF it's possible to replace a transaction by nearly the same transaction just with increased fee. To indicate RBF the sequence of one of the inputs needs to be lower than MAX_INT - 1 (in dec: 4294967295 - 1 or in hex: 0xFFFFFFFF - 1). You can check that for your transaction here.
Since you mentioned that the transaction is to an other address of yours, you could try Child-Pays-For-Parent. This works by spending the unconfirmed transaction output in a new child transaction with much higher fees to pay for the parent transaction and its self. This depends on your wallet software.
Here would be a guide for Electrum.
I the end it comes all down to the patience you have. Your transaction will confirm eventually. I would not advice somebody without technical expertise to experiment with different wallets and export/import private keys, just because the transaction does not confirm.
Fee estimation is still a difficult topic, especially in times with many transactions waiting to be confirmed. It's nearly impossible to say that a transaction will confirm in the next 19 hours.


text:
any limitation to use "bitcoind - zapwallettxes" to reverse unconfirmed transactions?
I read the following lines in some post about how to reverse a bitcoin transaction:
"How to stop/reverse a Bitcoin transaction without confirmations:
Run bitcoind and with -zapwallettxes.
This makes the wallet "forget" any unconfirmed transactions, thus enabling you to reuse their inputs."
My question is: Is this method only applicable to specific wallet? I use blockchain.info wallet and have some unconfirmed outgoing transactions not yet confirmed.

Run bitcoind and with -zapwallettxes.

bitcoind is the daemon version of the original Bitcoin wallet maintained by Bitcoin Core. bitcoin-qt would be the GUI version.
The option -zapwallettxes is specific for the Bitcoin Core wallet. It 'forgets' spend outputs and allows you to respend them. However this does not imply that other bitcoin nodes on the network accept a new transaction with the same (already spend in your first transaction) outputs.
-zapwallettxes is only affects your local wallet.
Your transaction can not be reversed. Probably the best thing to do is wait. Your transaction will confirm eventually.

Yes, this is wallet specific, not every wallet will have this command.

What wallet can use this command?


text:
How can users have confidence that there are no backdoors in paper wallet generators?
I'm thinking of issues like the randomization not being truly random. I appreciate that some users could examine the code themselves, but what assurances do less technically skilled users have?

There's really no way for a nontechnical user to know. In fact, most technical users do not have the specific technical knowledge required to evaluate cryptographic code (including the random number generator). Therefore, it comes down to a matter of trust. If the user does not directly trust the software author, they may sufficiently trust somebody else who has evaluated the technical aspects of the software.

Thanks! Is there any formal or quasi-formal system of 'peer review' for code of this sort? Or places where people put their testimonials about having reviewed code?

I don't know of any formal scheme for publishing peer reviews, particularly for, shall we say, the wild west of cryptocurrency software.

This is a time when very rapidly, people like you and me (Luddites in a highly Technical, complex virtual environment) are either going to swim or ... do something else less uplifting...
Having played 3+million hands of internet poker in my career, I do however know 1 or 0 things about RNG's; trust spread over a large, human net, cast upon the backdrop of the Internet - - - Trust no one, trust nothing, kill first and kill last. It is just that kind of world in the real, it's only natural these aggressive attempts to deceive and defraud other humans pervades the virtual realm as well.
Yes, there is a well-known repository of Peer Reviewed code. It can be found at www.github.org. It's free, it's open, and it's everyone's for the pilfering, plagiarizing, pluralizing and profiteering. It's also just vastly interesting, even more incomprehensible.
The cut dry answer to your question is: One should NOT trust Paper Wallet Generator services because services like this are centralized, often closed systems that commonly operate contingent upon a for-profit business model. You have immediately given incentive to the bad actors that would hack your Bitcoin Protocol, realized in it's truest form - A Trust-less Digital Currency, store of value, and pioneer of transparency in a proprietary world.
Bitcoin's most essential virtue is precisely this notion of requiring, unlike all previous systems introduced to the human race, a purely open-source technology. Meaning even an idiot like I could tinker with the deepest layers of coding in the protocol, if I so desired, if I so deemed it pertinent to my wallets security, or to satisfy my curiosity. It is, in it's highest form, a purely P2P, distributed network of individuals, self-governing their every transaction. It is very unfortunate indeed that we continue to see the growing trend where would be cryptographers counterfeit the gift of Satoshi by relinquishing trust to centralized, closed third parties that have every mal/ability to coffer their coin.
You have no assurance other than your best knowledge. To the best of your knowledge, you would never willingly hand your money, your trust, over to a faceless, inhuman, fallible Paper Wallet Generator Service - one that is susceptible to various forms of attack from within, and without. You would much rather understand your world, your finances and your future, and place all of your trust in yourself. THAT is what bitcoin is all about.
It's a sad reality but it's an exquisite solution, so I recommend you rethink who and why you entrust your wallets security. The pinnacle of cryptography, of bitcoin, is a system that requires you to know your code. I mean honor, I mean integrity, not necessarily that of the intractable, deep math encoding the client, either. Especially prescient is that knowledge that, if you don't trust yourself with your own paper wallets randomization, you can only trust that given sufficient time, every hack will be realized. Without hands on security measures put in place by the capable individual, infinite time*computer integrity = 100% failure.

Great answer. I will do some more reading

I'm very glad it was of use to you. Best of luck

authors: user1205901 - Слава Україні


text:
Can the Bitcoin blockchain eliminate the need for Trusted Computing in Smart Contracts?
Let me preface this by saying that I'm a novice, but this is my rough understanding of how Bitcoin's blockchain works.  (This is largely based on blog posts like this one.)  If person A wants to transfer a certain amount of Bitcoin to person B, he signs a message authorizing the transfer using his private key.  A minor can examine the message, and using a program that performs a certain mathematical operation, can verify that person A really did sign the message.  But there's the issue of, how do we know that the miner really did do the computational work needed to verify the signature?  This is solved by something called a "proof-of-work" protocol, by which the miner can demonstrate that he did the verification computation.
I was reminded of all this when hearing about the notion of "external state contracts", a type of Smart Contract described briefly at the 35 minute mark of this video and in more detail in this Bitcoin wiki article.  In an external state contract,  certain disputes about objective features of the world can be resolved automatically, instead of trusting a third-party human arbitrator to decide fairly.  For instance, contract could specify that in the event of a dispute about a sports bet, a computer program (with source code specified in the contract) would be automatically run that consults a sports database, determines who won the bet, and the bitcoin is automatically transferred accordingly.  But the Bitcoin wiki article describes an issue with this: how do you determine that the computer program that's supposed to resolve the dispute has actually been run, and if so whether the output of the program is being reported fairly by the computer that ran it?  The article suggests using one or more "Trusted Computers" known as oracles, which both parties trust to run the code properly and not misrepresent the output.
But it occurs to me that the issue with external state contracts is analogous to the issue I described above that's solved with the blockchain.  So why can't you solve it the same way in this case?  That is to say, just as proof-of-work protocols can be used to demonstrate that a miner ran the code that determines whether a signature is authentic, why can't you make miners run dispute-resolution code for external state contracts, and then determine whether they really ran the code using proof-of-work protocols?  That would eliminate the need for a Trusted Computer.
Is there any significant difference between the two cases that would make this unworkable?  Has this been done already?
Any help would be greatly appreciated.
Thank You in Advance.

...how do you determine that the computer program that's supposed to resolve the dispute has actually been run, and if so whether the output of the program is being reported fairly by the computer that ran it? ...
But it occurs to me that the issue with external state contracts is analogous to the issue I described above that's solved with the blockchain. So why can't you solve it the same way in this case?

If I understand you correctly, the question you're asking is: Why can't you extend the blockchain trust/coordination system to handle the problem of resolving disputes about the validity of a computation's output (or validity of an observation, like "who won this game?").  After all, if the blockchain gets everyone to agree on a transaction record, why not on arbitrary claims about the world?
The answer, in short, is that the blockchain restricts the "claims" that it "proves" to ones that, by construction, it can provide (strong) evidence for.  And those claims rely on special cryptographic properties that don't (as far as we know) generalize to arbitrary claims about the world.
More concretely, what the blockchain is claiming (and getting network coordination on) is something like, "These transactions happened in this order."  And it proves it by saying, "Transaction T30 must have been seen by someone who had already seen transactions T29, T28 ..., and spent 9000 units of work (say, computations of SHA-256) on it."  Which in turn is proven by how SHA-256 has a known computation difficulty, and the proof-of-work for Transaction T30 was generated uniquely from the previous transactions.
That's a very different problem than proving that a certain (family of) computation(s) has a certain result.  For that problem (as Nate Eldredge mentions), you will want to use Zero-Knowledge Proofs, an active area of research.  But those a) only apply to "pure functions", not arbitrary questions about the Real World, and b) require a certain asymmetry between computation and verification -- i.e., it must be a lot easier to verify them than to do the computation, or else you've saved nothing: just do the computation yourself instead of trusting others' reports!
And it's ever further off from the problem of "proving" real-world observations.  So yeah, the best we have cryptographically is to designate an address as trusted on certain matters, and then check the validity of their signatures.
In terms of verifying real-world claims, the best we have is to check all the implications of that claim, which is just old-fashioned fact-checking.  For the ballgame example, that would be checking things like "What do the newspapers report about the game?  What do the league records say?  Who played next in the tournament?"  Which is fine in practice -- like resolving things at the bookie.  But it doesn't provide a cryptographic proof, which would have to be robust against attackers e.g. falsifying news reports and such.
Okay, I've rambled enough now.  Did I answer your question?

Let's take a simple example: person A and person B make a bet on whether the billionth digit of pi is odd or even.  They both run the same program, and person A claims that the output is that the digit is odd, whereas person B  claims it's even.  Clearly one of them is lying, but how can we resolve who wins the bet?  You can trust one or more third-party arbitrators to run the program themselves, but that defeats the purpose of programmable clauses in smart contracts, which is to remove the need for trusted third-party arbitrators in resolving contract disputes.

That depends on how you define "the digits of pi" or "according the program" for purposes of the contract. One way would be to put a pi-digit-calculating program into the blockchain's contract.  (Note that the scripting language is loopless and Turing-incomplete so it would probably be long.)  Then, it's verified like any other script: everyone who sees the blocks containing these grinds through the program to check who gets the payoff, and they reject transactions predicated on the wrong answer.
This is why you usually just want to designate an address of a trusted party to make the call.

As for whether smart contracts are supposed to remove the need for trusted 3rd party arbitrators, I don't think they are. In some cases, the resolution of the contract is so mechanical that it can eliminate the need for the trusted party.  However, when someone has to make a judgment, the purpose of these smart contracts is only to provide a reliable escrow infrastructure -- so that the arbitrator can only give the money to one of the main parties, and not e.g. run off with it.

The problem with using blockchain to verify external world transactions is that the protocol requires that everyone is able to verify that any given block is valid.
So, at any given time, any miner can get back to the block #2, and verify that all the transactions were valid in that block. That is obviously only possible when the transactions don't reference an external input. Because if txs reference external input, the input might've been changed in the meantime, and you can't do that any more.
Why is it important to check the validity of the blocks? Because that's what blockchain is - the longest series of blocks since the first one. Your bitcoin client, when it wants to figure out which blockchain is the proper one, downloads all the blockchains that people claim are "the one", then weeds out the ones containing improper blocks, and then chooses the one that is the longest.
That's why there are no blockchain-based currencies that support external inputs and people use oracles for that. That's why Ethereum has all cool contract mechanisms but those contracts can't reference external outputs :)
You might find some of the answers in Orisi.org/distributed oracles whitepaper.

I'm afraid your first paragraph is not accurate.
The proof-of-work that a miner presents doesn't certify "I have verified the signatures."  Verifying a signature is a cheap computational task, and every client on the network can (and does) verify for itself that every signature in the block chain is valid.  They don't have to rely on a miner to verify the signatures for them.  Indeed, if all we needed to do was verify that every transaction was properly authorized, we wouldn't need miners at all.
Mining solves a different problem, called the "double spend problem": there could be two different transactions, both properly authorized, that are in conflict because they attempt to spend the same coins.  The community needs to be able to reach a consensus as to which of the two should be accepted.  The choice can be made arbitrarily, but then in order to keep the community committed to that choice, the protocol requires proof of work to certify that this transaction is accepted.  If the community (or someone within it) later wanted to go back and change its mind about which transaction to accept, they would have to redo all the work that had been done in the meantime, which is intended to be prohibitively expensive.  The computational work that's done is useless in and of itself; the proof of work certifies nothing except that some work was done.
What you are describing later sounds to me more like a zero-knowledge proof.

OK, then can zero-knowledge proofs be used to eliminate the trusted computing issue described in the bitcoin wiki article?

@KeshavSrinivasan: I don't know.  Actually, the more I read, the less likely it looks.

OK, do you know of any other way to automatically resolve a dispute about the output of a computation, without relying on trusted computing?

@keshav: I do not.


text:
Keeping the blockchain index after Bitcoin-Qt upgrade
Can this be done or do everyone updating their client everytime a new version is announced on bitcoin.org has to re-do the whole process over again?
On OS X, (probably with the other two OS's as well) blockchain data is kept on a seperate directory in ~/Library/Application Support/Bitcoin ; however a new version of the client needs to reindex the whole thing over again- which is a super time-consuming process..

Version 0.8 of the reference client changed the database format completely, in order to scale better. That meant a one-time rebuild that everyone had to go through, as converting the old database would have been error-prone.
With that change, the transaction index was made optional, and off by default. This is because for normal operation (verifying and processing blocks and transactions, and maintaining a wallet), such an index is not necessary.
An option -txindex (or txindex=1 in bitcoin.conf) was introduced to re-enable this transaction index (which is necessary for the getrawtransaction RPC call). It has to be specified when the database is created though, so at first run of a post-0.8 version, or when explicitly rebuilding the index (with -reindex).
This type of database format change is not expected to happen frequently.


text:
Alternative uses of a blockchain-mechanism?
Bitcoin's blockchain is basicly just a mechanism to record timestamped entries. Bitcoin uses it as a transaction ledger, but it could be used for many other things.
For example, one could create a chain for a decentral chess-game, where each move is an entry in the blockchain. All nodes can verify if the players turns are by the book, and in the correct order, and decide who is the winner.
Can anyone think of other examples of alternative chains?

Provably fair games can utilize it. For example, Satoshi dice published the hash of their daily secretkey hash list in the blockchain. In that way, they can prove that they haven't altered they secret key list. Or you could do a simple "guess a number" game that kept an encrypted copy of the result in the blockchain. Later reveal the secret key and
Essentially any kind of system that relies on the escrow of a small amount of information could use the blockchain for that escrow. You could even use it to as a communication channel if your messages were small enough.

Just a note: You cannot really add arbitrary things like text into blockchain.
You can "comunicate" by exact values and by bitcoin addresses, yes, but there is only so much that you can encode in that.

Actually I'm fairly sure that there is field for an arbitrary string to be added to a transaction. Not all clients provide an interface, but I'm fairly sure it is there. I will add a link in a few minutes if I find it.

Where would it be stored? In the script? Miners reject transactions with non-standard script with additional information.

Yes, in the script. I was thinking of transaction with a message from bitcoin.it. If miners would reject it (despite being a valid transaction) that might change things. I'll take away my downvote just because I've never tried it and it might not be possible in the real world. But that would require an edit to the answer, I guess. (Side note: I had guessed that blockchain.info was implementing their "public note" feature with this, but I was wrong about that.)

I have tried it :D not only the miners will not accept it, but most importantly the regular clients will not even relay it and throw it away, so it never propagates through the P2P network. If you want to add random stuff to the blockchain, you will have to use Namecoin that is full of "rubbish" in the blockchain :D

see http://bitcoin.stackexchange.com/questions/4486/transaction-with-slightly-changed-script-is-never-relayed - my question that more or less dealt with that

@KarelBílek Im talking about a new chain. This new chain could have completely different limits on transaction size, so "not enough space" is not an issue.

Oh. In that case, again, just use Namecoin. You can save any arbitrary things into Namecoin blockchain if it is under I think 500 bytes (theoretically more, but there is bug in the client and nobody is developing Namecoin for about a year.)

Not sure why this is voted down. http://bitcoin.stackexchange.com/questions/4707/what-method-does-my-wallet-use-to-encode-messages-in-the-blockchain You can't embed a text message into the blockchain itself, although it will be received by the recipient and be stored on blockchain.info


text:
Why do transactions need to be spend fully?
If I want to send you 75 and I have 100, a transaction of 100 will be made, in which I get 25 back as change. Why is this?
See also Change

When the output of a transaction is used as the input of another
  transaction, it must be spent in its entirety

Excellent question, I thought this is just how most clients worked, didn't know that this was actually a protocol requirement.

Related question

Not exactly a duplicate, he just wanted to know why they need to be spent fully, while the main answer doesn't address that at all (it just says "they do"). The new answer addresses it, but I'd still be reticent to close as a duplicate.

All bitcoins that you hold in a wallet are received direct from generated coin (issued as the block subsidy) or value assigned to you by the previous holder.   That assignment does not have a value field, only authorization for the value to be spent by you.
Many other accounting systems use the approach where account balances are used, rather than aggregating all transactions like how Bitcoin works.  Bank account systems work like this.
Work is underway to bring this capability to Bitcoin.


text:
Can a transaction be dependent on another in the same block?
Say I'm creating a multi-sig wallet with a friend to bet on the election. Can my deposit to the multi-sig wallet be dependent on the my friends deposit to the wallet?
Say I'm Alice, and my friend is Bob. Our shared multi-sig wallet is S. Would the following work?
Block N:
TxA
  Input1: A.amountX
  Input2: B.amountX
  Output1: S.amountY
  A.sig
  B.sig

Edit: Received good answer below, to summarize use Partially Signed Bitcoin Transactions (PSBT) BIP-174, it relies on Sighash Flags, this enables parties to sign more than just their own input.

To accomplish this you could create a single transaction that you both supplies inputs to, and that creates an output paying to your multi sig address. In order for the transaction to be valid you will both need to sign it, and of course neither of you should sign it unless the other has supplied an input with the agreed upon value.
See: BIP 174 Partially signed bitcoin transactions

Very interesting. How does the bitcoin logic allow the unlocking of the input based on a signature that signs both inputs? Does the verifier check for each input to see whether one of the unlocking scripts satisfies it? but the unlocking script (with signature of all inputs) depends on multiple inputs (both Alive and Bobs) being there?

I did further research, I think what I'm looking for is called a coinjoin.

@miketery re: signatures: you'll want to learn about 'sighash flags', they are used to designate which parts of a transaction the signature covers. re: coinjoins: that term generally describes a more specific type of multi-party transaction, that seeks to improve the on-chain privacy of participants, I wouldn't really use that term to describe what you're trying to achieve.

I found this https://wiki.bitcoinsv.io/index.php/SIGHASH_flags (hopefully this is the same in BTC?). What does BIP-174 add if N people can agree on a X inputs and Y outputs, and then use SIGHASH_ALL?

@miketery BSV is an altcoin, I would definitely not recommend using that reference. Try perhaps https://raghavsood.com/blog/2018/06/10/bitcoin-signature-types-sighash ?   re: BIP174: it is a standardized protocol for passing partially constructed bitcoin transactions between users/software. Using software that employs it will make collaborating on a multi-party transaction much easier.


text:
Problem with Berkeley DB while building Bitcoin on Windows from Source Code?
I'm trying to compile Bitcoin from source code in Windows 7.
When I run ./configure, I get the following error:
checking for Berkeley DB C++ headers... no
configure: error: libdb_cxx headers missing

I have followed all the steps mentioned in the source code folder doc/build_msw, but still get the same error.
Additionally, I have tried following these instructions for building Bitcoin on Windows:

How do I build Bitcoin source in Windows 7?
Building headless Bitcoin and Bitcoin-qt on Windows

I have installed berkeleydb 4.8.30.msi.
What else can I try to resolve this issue?

Have you tried googling the error message?

Yes, I have tried everything, but still its give me the same error

follow this links to resolve the issue https://bitcointalk.org/index.php?topic=149479.0   and also installed with berkeleydb 4.8.30.msi, with command also installed

Please put sufficient information into your questions so that they can be answered by someone that is interested in trying to help you. This means that you need to provide background information, tell us what you have tried, and show us that you have put in reasonable effort to solve your problem yourself. Please don't add additional information into the comments, but edit your question to improve it. For additional information see [ask].

Have you tried installing berkleydb from source, as the forum page you linked tells you to do as step 2.2?

yes, I have done with source as well as with exe db-4.8.30.msi, still it will give the same error again and again

There doesn't seem to be a file in doc/ named build_msw. Are you building the latest version of Bitcoin?

I am usings bitcoin 0.9 version. and It will contain build-msw.md file. If I am use 0.10 version which is a latest code,  then how I can configure all dependencies for windows.

When you run the configure script and compile berkleydb, does it produces any output or error messages?

when I compile berkeley DB with its source code, It will compile perfectly. But when I run ./configure command from bitcoin code it will give me error that "checking for Berkeley DB C++ headers... no
configure: error: libdb_cxx headers missing"


text:
Is it possible to create bitcoind regtest network on Ubuntu (which itself is on Virtualbox)?
I am getting into Bitcoin development and it seems that bitcoin regtest mode is perfect for testing our applications.
I am trying to create a network, following Fred Tingey's answer but, unfortunately, it's not working.
I need suggestions, what information can I provide?
I am trying to use docker, but I have to first create 64-bit Ubuntu machine.
Thanks

Sure: link

I think I am not properly creating additional data directories. I can't connect to the network, the connection times out. @ShaunWilson

Well the instructions in Fred's answer appear to be mostly correct, he does seem to list inconsistent port numbers. I'll see if i can clean it up.

Thanks a lot. What I am struggling with is how to test if the network is working properly? Also, to create additional directories, is it just copy and paste? @ShaunWilson

I cleaned up his answer, but it'll take a while for peer review to catch up to it.
Yes, you can simply copy/paste folders to create more nodes.
Once they are running and connected you can use the command-line interface to issue commands like getpeerinfo, for example http://sourceforge.net/p/bitcoin/mailman/message/32012030/

Can you also post your edited answer here please? @ShaunWilson

It appears that Fred Tingey's answer was mostly correct, but contained incorrect port numbers in the samples he provided. If someone were to copy and paste it all it would not have worked correctly.
I've modified his answer, it is awaiting peer review.
The use of Ubuntu, Docker and VirtualBox shouldn't affect your ability to configure regtest.
You may have problems connecting from the host to the guest, if you do I suggest perhaps asking questions on superuser or ubuntu exchanges instead.
Hope that helps!
=== Here's the edited version:
In order to run multiple nodes in regtest mode on a single machine you will need to sandbox each node.
In this example I have three nodes, they are named Alice, Bob and Cory. Since Bitcoin is a Peer/Mesh network, my goal is to connect each nodes so that changes made to Cory are ultimately visible to Bob (without necessarily requiring a direct connection between the two.)
Step 1: Create folders for each node.
These folders will contain our data and configs for each node. The names are not important, they can be renamed later, so you can name them "Alice", "Bob" and "Cory" for now. Remember where you create them, you will need their paths for Step 3 below.
Step 2: In each folder, create a bitcoin.conf file
In the config files Alice will be configured to connect to Cory, and Bob will be configured to connect to Alice. In addition to configuring outbound connections, this is also where we specify regtest=1 and server=1 settings.
bitcoin.conf for Alice
regtest=1
server=1
 # Alice connects to Cory
connect=localhost:18333
bitcoin.con for Bob
regtest=1
server=1
 # Bob connects to Alice
connect=localhost:18444
bitcoin.conf for Cory
regtest=1
server=1
 # Cory connects to nobody

Step 3: Start each node.
Note each node must have a different port number since only one node can bind to a port number at any time.

Alice on port 18444
Bob on port 18555
Cory on port 18333

You can choose your own port numbers of course, just be careful not to use a port number that is already in use by something else.
The command-line options will tell bitcoin what port number to use, where to store data, and where to find its config file:
Cory:
"C:\Program Files\Bitcoin\bitcoin-qt.exe" -conf="C:\Cory\bitcoin.conf" -datadir="C:\Cory" -port=18333 -rpcport=8333
Bob:
"C:\Program Files\Bitcoin\bitcoin-qt.exe" -conf="C:\Bob\bitcoin.conf" -datadir="C:\Bob" -port=18555 -rpcport=8555
Alice:
"C:\Program Files\Bitcoin\bitcoin-qt.exe" -conf="C:\Alice\bitcoin.conf" -datadir="C:\Alice" -port=18444 -rpcport=8444
This will launch three instances of the Bitcoin Client, and the folders for each node will begin to initialize with regtest data.
Step 4: Testing
You can use the Help->Debug Window->Console option to interactively execute commands on any of the nodes. For example, you can mine 101 blocks with the following bitcoin console command:
setgenerate true 101
You should (after a few seconds) then have been awarded coins.
These mining activities result in coinbase transactions - and only the last one of 50 BTC is available to spend. Now you can send some BTC to one of the other nodes.  After a bit more mining such a transaction will be confirmed.
You can also use the console commands getpeerinfo and getinfo to check the network is set up correctly.  One of them (Core wit port=18333) should have 'sync=true' in its peerinfo output, and all nodes should have the same "block height" regardless of which node is used for mining.

you don't need the -conf arguments btw... at least on linux. And you will need to add -rpcport=XXX to deconflict the rpc ports

You have to use -listen=1 if you're setting -connect otherwise it won't accept incoming connections.

you have 2 Bobs in command line examples


text:
Why is my raw transaction non-canonical, getting error -25, and being rejected?
I'm trying to port TX generation logic to a different (and unpopular) language.  I've been beating my head against this way too long and should have walked away but, well, you know, closure, and stubbornness.
Here's my code so far.
Blockchain.info, Electrum, and Bitcoin Core all decode my attempt at a TX into something reasonable looking, such as:
{
"txid" : "49c210ae472c5b5e39447e1f6d9bc020cd0f0075cc03d919afb0857a964e1f41",
"version" : 1,
"locktime" : 0,
"vin" : [
{
"txid" : "b097384c42a3be2730db3e3720a1806c76172b6b62b2b5ee007c2c6fd295cadf",
"vout" : 1,
"scriptSig" : {
"asm" :     "30460221009f478737296e39bbcff2ef7c6f013acc25bea75941acd8573bcea83ca910018b022100e9bff518297fec344d6e7e42cabc5ec793c4be507af68210bdde803a8a2958ed0104d8f39341451e2e66a00ef010c815f1284bc5e3a187476aab319b2d127b7e219c9b9e68bff311c63474242a9baab34f7ddec05de2c45bd140a74a64621ccb42cb",
"hex" :   "4930460221009f478737296e39bbcff2ef7c6f013acc25bea75941acd8573bcea83ca910018b022100e9bff518297fec344d6e7e42cabc5ec793c4be507af68210bdde803a8a2958ed014104d8f39341451e2e66a00ef010c815f1284bc5e3a187476aab319b2d127b7e219c9b9e68bff311c63474242a9baab34f7ddec05de2c45bd140a74a64621ccb42cb"
},
"sequence" : 4294967295
}
],
"vout" : [
{
"value" : 0.00439999,
"n" : 0,
"scriptPubKey" : {
"asm" : "OP_DUP OP_HASH160 7847eb9e366653aeb8857d541236fe4fd90c57e7 OP_EQUALVERIFY OP_CHECKSIG",
"hex" : "76a9147847eb9e366653aeb8857d541236fe4fd90c57e788ac",
"reqSigs" : 1,
"type" : "pubkeyhash",
"addresses" : [
"1BxzF8rgXtuiPuSN8azdMJgryzQSWt4Uoj"
]
}
}
]
}

Trying to sendrawtransaction in Bitcoin Core, I get "error -25".  Reading through other StackExchange posts and Googling, the possible causes are listed as:

"Reading the source code, this error is returned when AcceptToMemoryPool fails, but not when it fails because the transaction is invalid. Does debug.log emit anything when this happens?"
"You get that obscure RPC error when your tx is using outputs that bitcoind never heard of."  Nothing appears in ~/.bitcoin/debug.log.

Trying to send it with blockchain.info's slash pushtx service (oh, come on, is there seriously not a white list for sites I can link to?), I get "Script resulted in a non-true stack: []", which implies that I made a mistake in my signing logic somewhere.
I've walked through these:

http://procbits.com/2013/08/27/generating-a-bitcoin-address-with-javascript

http://www.righto.com/2014/02/bitcoins-hard-way-using-raw-bitcoin.html

Redeeming a raw transaction step by step example required

And others, including various Ruby, Python, Java, and JavaScript implementations, but I need more than 10 reputation to post more than 2 links.  I cannot for the life of me figure out where I'm deviating.
I know parts of those are out of date.  The private key I'm starting with (from Electrum) is not flagged as compressed (33 bytes after DecodeBase58Check with the last being 0x01), so the publicKey in the transaction I'm trying to draw from shouldn't be compressed either.
The TX I'm trying to spend from (b097384c42a3be2730db3e3720a1806c76172b6b62b2b5ee007c2c6fd295cadf, second output, aka 1) with this TX has an output script which is Pay to Public Key Hash (d9495c762aed3dba15eec648beb55a8a43b8d1bd) according to blockchain info.  The full scriptPubKey of the prevout is:
OP_DUP OP_HASH160 d9495c762aed3dba15eec648beb55a8a43b8d1bd OP_EQUALVERIFY OP_CHECKSIG

My private key, decoded from WIF, and run through ecdsa::pub_from_priv(),  ecdsa::pub_encode(), matches that value.
It seems like getting the right stuff hashed before the hash is signed into the signature would be the hard part.  I'm appending 0x01 to the signature after it comes out of ecdsa::Sign().  Before the TX is signed, it gets 01000000 appended to it.  scriptSig (the Script in the input for the TX I'm building) is "OP_DUP, OP_HASH160, PUSHDATA, Bitcoin address (public key hash), OP_EQUALVERIFY, OP_CHECKSIG" before it is signed then it becomes a varstr of the sig and the pubKey, and that decodes to reasonable looking bytecode:
scriptSig:
0: OP_PUSHDATA 0x304602210094c538663c149f40929bb787d6174104a694181d063943a745e558b17d09e276022100b1812105ea6d7a8206c9019303a6459a9a2b2524b364debe69405b5d8b90c6c301
74: OP_PUSHDATA 0x04d8f39341451e2e66a00ef010c815f1284bc5e3a187476aab319b2d127b7e219c9b9e68bff311c63474242a9baab34f7ddec05de2c45bd140a74a64621ccb42cb

scriptPubKey:
0: OP_DUP
1: OP_HASH160
2: OP_PUSHDATA 0x7847eb9e366653aeb8857d541236fe4fd90c57e7
23: OP_EQUALVERIFY
24: OP_CHECKSIG

Using the old scriptPubKey as the scriptSig is approximated and assumes a simple pay-to-signature rather than actually pulling the exact scriptPubKey out of the TX being drawn on, but in this simple case, they do seem to match.
Here's the raw, (incorrectly?) signed TX:
0100000001dfca95d26f2c7c00eeb5b2626b2b17766c80a120373edb3027bea3424c3897b0010000008b48304502202647239b48610693967a24c7c976f0df903891113c56f50b6e3368c3f73eefb0022100e372ab8bf76ab35cba1ce6c1890fb6b27bb4d6d54f080180b46a6ecc2ae09248014104d8f39341451e2e66a00ef010c815f1284bc5e3a187476aab319b2d127b7e219c9b9e68bff311c63474242a9baab34f7ddec05de2c45bd140a74a64621ccb42cbffffffff01c0b60600000000001976a9147847eb9e366653aeb8857d541236fe4fd90c57e788ac00000000

Here's the code:
https://gist.github.com/scrottie/15f2fca963d164306dcb
PrivateKey upon request (there's $1 in there).
If anyone is able to dump that and figure out where I went wrong, I'd be ever so grateful.

(1) Signature is invalid (2) Signature should be in "Low-S-Value-Format" (3) USE TESTNET FOR SUCH EXPERIMENTS!!!

Sorry about the lack of testnet use. If I can get this to work, I'll try to create a good walk-thru for that, but existing ones are for the mainnet.
I plugged in the private key/from address/to address/BTC amount/prev TX hash from http://bitcoin.stackexchange.com/questions/3374/how-to-redeem-a-basic-tx and get the exact same sig hash:
9302bda273a887cb40c13e02a50b4071a31fd3aae3ae04021b0b843dd61ad18e
Comparing byte by byte, it's identical up until there.  Then the random K has the sig changed.  Color me miffed.

Really wish I knew what the K was in that example so I could completely reproduce it...

I can't find anything in Google on "Low-S-Value-Format".  Are you simply referring to endianness?  Do you have a reference for that?  Thanks for your reply.

Search for "ECDSA signature malleability"

%&@!
Fixed it with this change:
$privateKey = $privateKey->as_hex();  $privateKey =~ s{^0x}{} or die;
warn sprintf "doing: python sig.py  '%s' '%s'\n", $privateKey, to_hex($s256);
open my $python, '-|', 'python', 'sig.py', $privateKey, to_hex($s256) or die $!;
my $sig = readline $python;
warn "python says for sig: $sig\n";
$sig =~ s{[^0-9a-fA-F]}{}g;
$sig = from_hex($sig);

And adding this file as sig.py:
import ecdsa
import ecdsa.der
import ecdsa.util

import sys

privateKey = sys.argv[1]
s256 = sys.argv[2]

# print("privateKey len: ");  print(len(privateKey.decode('hex')))
# print(privateKey)

# print("s256 len: ");  print(len(s256.decode('hex')))
# print(s256)

sk = ecdsa.SigningKey.from_string(privateKey.decode('hex'), curve=ecdsa.SECP256k1)
sig = sk.sign_digest(s256.decode('hex'), sigencode=ecdsa.util.sigencode_der) + '\01' # 01 is hashtype
print sig.encode('hex')

I'm new here and I'm apparently trolling myself.  I couldn't get the modules on CPAN that interface with openssl to work so I had one option for EC library, something written in pure Perl.  That apparently has problems, even though it's unit tests pass.  Or I was using it wrong.
That resulted in output that made https://blockchain.info/pushtx say "Transaction Submitted".  And then coins moved.
Thanks to everyone who stopped to give this an eyeball.  I owe you a beer.  I'll try to play it forward with very, very good comments and docs.

Sorry, I do not use perl language and do not have perl environment installed to debug your code
Have a look to How to redeem a basic Tx?
I do not see step #13 where you have to append 01000000 = SIGHASH_ALL to the data you sign in your makeRawTransaction method

I have looked at How to redeem a basic Tx, Protocol specification, the description of OP_CHECKSIG on bitcoin.it (limited URLs for me until I get more StackExchange points), and https://en.bitcoin.it/w/images/en/7/70/Bitcoin_OpCheckSig_InDetail.png.
pack("V") or pack("L<") in Perl is like pack("<L") in Python.  Otherwise it follows pretty closely.
SIGHASH_ALL is on this line:
my $myTxn_forSig = makeRawTransaction( $outputTransactionHash, $sourceIndex, $scriptPubKey, $outputs ) . "01000000";
Thanks for taking a look at it.


text:
Why does Bitcoin-Qt use hidden private addresses?
As I found out from this question default wallet contains 100 private keys but only makes one visible.
But what I don't understand is why Bitcoin using those "hidden" addresses? I've been using my main address ABCDE and after some point I've noticed that my balance is spread on different private addresses which I can see in "Receive coins" tab. Why couldn't it just use my main address?

This is done so as to ensure you have all of your keys backed up. BitcoinQT uses a new address for each transaction - taking them from the pool of unused addresses. However, as there is a pool of those addresses always kept in the file, even if you back it up every 100 transactions, you can still recover all of your money. If you would only be keeping the addresses you use and geenrate new ones on the fly, you would need to back them up each time you send coins.
The main client does not reuse the same addresses to increase your privacy - one would not be able to tell who you would be sending money to, and which address you would own. It's privacy by default, rather than on demand.

Thanks, I understand why it is not generating new addresses on the fly. But why it uses new address for each transaction? Why not just use one "main" address?

To obfuscate your balance - when sending a transaction the remaining balance will be sent to a new address, thus not showing everyone that you have X bitcoins in one address. This allows for higher anonymity.

You aren't forced to obtain a new address for each transactions but in most instances without doing that there would be no way to tell who made the payment.  For things like donations where you don't care who paid, using a static bitcoin address works fine.


text:
What is the expected time until the next block is found?
https://bitcoin.stackexchange.com/a/3908/659 says:

Luckily, you'll only have to wait five minutes on average for a block to be found (blocks are one every ten minutes, so the expected delay until the next block is five minutes).

Upon reading it, I instantly felt that this was incorrect, and that the expected time until the next block is always 10 minutes (assuming the network hash rate matches the current difficulty).
If the blocks were generated exactly every 10 minutes, than it would be true that the expected time would be 5 minutes, but they're not.
So what's the real expected time?  Please make a plausible argument for why it is so.

10 minutes. Assuming, as you say, that the current network hashrate matches the difficulty - otherwise it is proportionally higher or lower.
Block finding is a Poisson process. The rate parameter λ is chosen so that 2016 blocks on average are found every 2 weeks, and since in such a process the expected number of events is proportional to the time span, it means λ is 1/(10 minutes).
The time until the next event (the next block found in our case) follows the exponential distribution with mean 1/λ, which is 10 minutes. This distribution is memoryless - the amount of time spent waiting for a block has no effect on the amount of time left to wait. When a block is found the expected time until the next block is 10 minutes (which is intuitive), and it is also 10 minutes at any other time.
If blocks arrived once every 10 minutes like clockwork, the expected time until the next block at a random point would be indeed 5 minutes - but then we wouldn't have to worry about expectations, we'd know exactly when the next block will be found.
In the future, the decreased generation reward might cause the total hashrate to fluctuate as transaction fees accumulate, causing reduced variability in block finding times, and reduced expected waiting time.

Don't forget that the block finding process is a factor of difficulty, not total hashrate, so at any given point it will be lagging difficulty in one direction or another. Block times should average out to 10 minutes over the long haul, but at any given moment the average block time for a day, week, etc could be quite a bit higher or lower :)

@DavidPerry: Yes, I specified that I'm assuming that the difficulty matches the hashrate. When the hashrate changes this will of course not be accurate.

Pick any random point in time. The average time from that time to the next found block is 10 minutes. Also the average time from that time back to the previous block is also 10 minutes. Also, the average time from the previous block to the following block is also 10 minutes! Try to get your intuition to accept that!

@DavidSchwartz: That's wrong. The average time from the last block to the next block is 20 minutes, and follows the negative binomial distribution. There's no escaping the additivity of expectation.

@MeniRosenfeld are you saying that the average time between blocks is 20 minutes?  That would seem to contradict the fact that blocks are generated on average every 10 minutes (again, assuming constant hash rate and matching difficulty).

Meni is right! This is not a contradiction because the 10 minute average block time is averaged over blocks. Whereas if you pick a random time, you're averaging over time. Averaging over blocks, every block interval is equally weighted. Averaging over time, longer block intervals get a higher weight (because they span more time) and hence the average is higher.

So can we say that the expected time between blocks is 10 minutes, but the expected time between the previous and the next block (right now) is 20 minutes, because 'right now' is likely to be a bigger than average gap between blocks?  It's certainly counter-intuitive if that's the case.

@ChrisMoore: Your last comment and David's last comments are correct. It's like the problem with 2 buses, one with 10 passengers and one with 20. If you ask a random bus driver how many passengers were on his bus you get 15 on average, but if you ask a random passenger how many passengers were on his bus you get 16.67 on average.

Also consider this: If you wake me up in the middle of the night and ask "How long do you think will be the interval between the last block and the next?" I'd say "20 minutes". If instead you wake me and ask, "How long will be the interval between the last and the next? Oh and by the way a block was found just now", I'd say "10 minutes". The interval between last and next is the sum of the time since the last plus the time until the next; the time until the next is always expected to be 10, but the time since the last is only expected to be 10 in absence of information to the contrary. ...

The latter question is basically what you ask when you take a random block and inquire how long it takes from the time it was found to the time of the next block, which gives you the average block generation rate.

I'm glad I asked this question.  It's more interesting than I thought it would be.  Thanks for your answers!

Minor correction: The time between the last block and next block, at a random point in time, follows the Erlang distribution with shape parameter k=2. This is the continuous limit of the negative binomial distribution.

@SamuelLi: Hm, right, I guess I was a bit loose with the terms there.


text:
How do I "Verify my wallet"
I'm completely new to the BitCoin world and I've purchased a couple of coins through eBay. The seller has messaged me saying this:
"Donation sent to 1Nf9iW******Yx4cPKB ...Please verify wallet and your 3.00 bitcoin will be released. Thank you for your business!"
What exactly does this mean? I've navigated through the wallet application and can't seem to find anything.. Any help is more than appreciated, thanks for reading.

Hi Mark and welcome to StackExchange! The reason that your question was closed is probably that it has to do with the practices of a single eBay seller and it is therefore not very likely to help other visitors.

Also, to the people who closed the question: please add a comment when closing, especially if the user is new to the site.

Do you own the address specified by the seller (as in, did you provide them the address)? If so, the seller probably wants you to confirm "yes, the address is correct" or something like that. If you don't own the address, I think you should ask the seller what exactly does he mean by that.
Generally, you don't verify a wallet. All the Bitcoin addresses verify their integrity by themselves (they have a checksum). Someone might ask you to verify that you are an owner of an address by making you sign some message with it, but that is quite rare.
I did a bit of search on the address you provided in part. Generally, unless you want to really hide that you own the address, there is no need to obscure any information about the addresses. In that case you wouldn't give any information. As it stands, I think you meant this address: 1Nf9iWh318w45Q1BY6cebPFp6YnYx4cPKB. From what I can see, you received a donation of 0.103BTC, so I'm guessing the seller sent you a bit of Bitcoins so you can check if the transaction went through and you received it correctly before sending you the full amount. If you can see the transaction in your wallet, you should go ahead and tell the seller that the address is correct and they can send the full amount.

This is probably a test transaction by the seller to confirm that you are able to receive funds at that address.
If you are using the Bitcoin.org client, a transaction for that test transaction should appear under transactions and it should get at least one confirmation (or preferably several) before you confirm that you received it.


text:
For Application Development is bitcoind RPC the Only Way?
I am looking at building an application using Bitcoins, and being a newbie the only real resources I have found regarding how to interact with the network is to issue RPC commands to the bitcoin-qt client running as a service.
I've played with this a bit and I am curious if there are other clients that do the same thing? Or is this client the only game in town?

You have many options but Bitcoind is a popular one due to its initial ease.

Use Bitcoind
Use an API such as Coinbase or Blockchain.info be careful as these can be slow and not as secure.
Use one of the many libraries made for various languages. Bitcoinj for Java, Libbitcoin for C++, Gocoin for Golang, and for other languages like Python and Javascript you can always find tools in a github search. This is the search for Python libraries in python.

No, it's an important part, but it's not the only option.  And it depends entirely on what type of application you want to build and where you are located geographically.

If you want a lot of flexibility but don't want to run your own
Bitcoind, look into Blockchain's wallet API.
If you are in the US and you want to accept Bitcoin, Coinbase has a rich API as well.  BitPay has an API also, although I'm less familiar with it.
Other sites exist that let you get quotes, inspect blocks, transactions, etc. through REST-formed requests.

Running bitcoind exacts a certain toll for hosting fees as well as managing security.  It's what I've chosen for my business, but it's not the only approach for everyone.

While @ChrisW's answer is good for situations where you need basic functionality, you have to remember that you'd be relying on third-parties. Which means when they go down, also your application goes down.
It depends on what you need to do, but if you're just looking to send and receive bitcoins and checking balances there's for example bitcoinj which is a Java implementation that you could use within your application without having to rely on a full-node (bitcoind).
This would be a lot faster and easier in my opinion if you're just building a wallet application for example, because it's not a full node and doesn't require you to download the entire block chain, but just the headers.

authors: გოგა გიორგი ღლონტი


text:
Can I reuse blockchain/index files with multiple wallets?
With the bitcoin-qt client, I'd like to have multiple wallets use a single set of blockchain data files.
Why? Downloading a fresh copy of the blockchain can take days, and each copy of the blockchain seems to be ~ 5GB large.  This feels somewhat wasteful if coins are spread about multiple wallets.
Note: I do not want to run multiple instances of bitcoin-qt simultaneously as I'm sure this would result in a bad time.
I was hoping this could be accomplished using the -datadirs option and symlinking blkindex.dat and blk000X.dat files to a central location.  But it seems like I'd have to create new symlinks in each datadir everytime a new blk000X.dat gets created :(

Is there any reason why you want to do something like that? Maybe you don't need to do it this way.

You can swap wallets around and synchronise them up with the same block chain data files - you would just need to replace the wallet.dat files in the Bitcoin's data directory. The client would only synchronise once with the network and later be able to synchronise your wallet files independently. Keeping up with which wallet is synchronised and which isn't might be a bit of a hassle though. I don't know of a client that can operate multiple wallet.dat files simultaneously.

hobonickels wallet will manage multiple wallet.dat files simultaneously


text:
Sweeping a bitcoin private key using PHP
Is there a PHP script where it validates a bitcoin private key and also gets the public address from the key?
My goal was to:

verify the private key
Get the public address using that private key
Check the balance using an external api
Sweep if balance > 0

I know I can test it by sending some bitcoins to it, or use electrum to attempt the sweep, but let's say I have 100 private keys I need to verify, how would I go about that?

What do you define as a valid private key in step 1.? When the key is in the secp256k1 ECDSA range? That would be nearly all  256-bit numbers. (https://en.bitcoin.it/wiki/Private_key#Range_of_valid_ECDSA_private_keys)

What format do your private keys have?

Question title edit: How do I empty the accounts of 100 people who's private keys I've hacked

No MVP PHP code, but some bits that could help.

verify the private key

Assuming you want a private key to be in the secp256k1 ECDSA range. You could check if the key is any number between 0x1 and  0xFFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFE BAAE DCE6 AF48 A03B BFD2 5E8C D036 4140. Take a look in the bitcoin wiki entry to private keys about the key range.

Get the public address using that private key

The library BitcoinECDSA.php could help you out here. Especially
$bitcoinECDSA->setPrivateKey($privateKey);
$bitcoinECDSA->getAddress();

3 + 4.Check the balance using an external api and sweep if balance > 0

This step can be easily combined. If you want to rely on a third party you could use the Block.io PHP API and Libary. They have an example PHP code for sweeping.

FYI, this needs the PHP  gmp module.


text:
How are sigops calculated?
How are sigops calculated on a transaction? For example in the (randomly selected) following transaction:
https://btc.com/bc9384919ad5d08b2c66e31f29e7c63572c398a87631c03a4ce9e94ff1cbe62f
It says there were 5 sigops. What is the rule for calculating this number?

Each OP_CHECKSIG and OP_CHECKSIGVERIFY is counted as 1 sigop.
Each OP_CHECKMULTISIG or OP_CHECKMULTISIGVERIFY, is counted as 20 sigops if they are found in an output or scriptSig and not part of a redeemScript, i.e. not P2SH.
If OP_CHECKMULTISIG or OP_CHECKMULTISIGVERIFY are in a redeemScript (i.e. it's P2SH), then the number of sigops is the number of public keys in the multisig redeemScript. If there is more than 16 public keys, then it is always accounted as 20.
For non-segwit inputs, these numbers are multiplied by 4, so 4 sigops for OP_CHECKSIG and OP_CHECKSIGVERIFY, and 80 sigops for OP_CHECKMULTISIG and OP_CHECKMULTISIGVERIFY.
However segwit inputs don't use these scripts explicitly. So a segwit input where the witness program is version 0 and the program is 20 bytes (i.e. P2WPKH), the number of sigops is 1. If the witness program is version 0 and the program is 32 bytes (i.e. P2WSH), the witnessScript is interpreted and its sigop count is as above as that uses the opcodes explicitly.

So in your example transaction, the input spends a P2SH-P2WPKH and the output has an OP_CHECKSIG. Since the input is P2WPKH, it is counted as 1 sigop. The output as an OP_CHECKSIG, so that's another 1 sigop. However because that output is not a segwit output, its number of sigops is multiplied by 4. Thus the output is actually 4 sigops. Summing those is 1 + 4 = 5 sigops in the entire transaction.

Why is OP_CHECKMULTISIG and OP_CHECKMULTISIGVERIFY counted as 20 sigops if they are found in output?

"then the number of sigops is the number of public keys in the multisig redeemScript" -> Hmm if it's >16 then it's accounted as 20, if i remember correctly.

To add to Chow's answer:
How sigops are counted changed with the activation of SegWit in 2017. For example, the sigop count in pre-SegWit scripts (P2PK, P2PKH, ...) was quadrupled from 1 to 4. See the sigops section in BIP-141: https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki#sigops


text:
Trying to understand parent fingerprint in HD wallets
I'm trying to implement a barebones HD wallet generator in C#.  I'm starting with a mnemonic and derivation path m/0' and want to generate BIP141 p2wpkh-in-p2sh addresses.  Obviously for the Master Node the fingerprint is 0x00000000, but when I get to depth 1 (0'), I need to calculate the fingerprint of the parent key.  My question is, is the formula RipeMD160(parentPrivKey) or RipeMD160(SHA256(parentPrivKey))?  I've been looking at Ian Coleman's implementation in javascript (https://iancoleman.io/bip39/) and he seems to do the latter one.  However, on this site under "Parent Fingerprint", it says it's the former.

It's ripemd160(sha256(parentpriv)) also known as hash160(parentpriv) source. There is also a python implementation here

Wrong. It's ripemd160(sha256(compressed parent pub))[0:4]. Your link also says it clearly: Hash160 (RIPEMD160 after SHA256) of the serialized ECDSA public key

Can't use comments because of reputation, but both the question and the answer is saying that the fingerprint is based on private key. It should be the hash160 of the PUBLIC key. for even if you are deriving a private key you should use the parents PUBLIC key for generating the fingerprint. This is important to make both a derived public and a derived private key having  the same fingerprint so you can relate them to each other.

A number of HD wallet fingerprint test vectors for curves secp256k1, nist256p1, and ed25519 are provided at the bottom of https://github.com/satoshilabs/slips/blob/master/slip-0010.md#test-vector-1-for-ed25519. Notice all top tier fingerprints are 00000000.
For https://github.com/satoshilabs/slips/blob/master/slip-0010.md#test-vector-1-for-secp256k1 Chain m/0H, the fingerprint is computed from its parent's public key as follows:
% echo 0339a36013301597daef41fbe593a02cc513d0b55527ec2df1050e2e8ff49c85c2 | bx sha256 | bx ripemd160 | cut -c 1-8
3442193e
% echo 0339a36013301597daef41fbe593a02cc513d0b55527ec2df1050e2e8ff49c85c2 | bx bitcoin160 | cut -c 1-8
3442193e


text:
How did Extranonce2_size change?
From stratum,I know that the coinbase is composed of "Coinb1 +Extranonce1 + Extranonce2 + Coinb2".
How did Extranonce2_size change ?
Will the Extranonce2_size change when I connect to a pool?

Miner sent message mining.subscribe to pool
mining.subscribe("user agent/version", "extranonce1")

Pool reply with
[[["mining.set_difficulty", "subscription id 1"],
["mining.notify", "subscription id 2"]],
"extranonce1",
extranonce2_size]

Pool set size of extranonce2

Thanks for your reply.I would like to know in what situation the Extranonce2_size will be changed.


text:
How is mining profitable?
The miner that creates a block gets the reward and all the fees that are in that block.

What happen with other nodes that does not create a block. In their case, will not they  get any reward and also any fees? How do they survive?
As I understand, when a block is created, all the miners start again finding the solution to create the next block. Can the miners choose which transactions validate? If it is like that, all the blocks in that time frame, are they the same?

Thanks!

What happen with other nodes that does not create a block. In their case, will not they get any reward and also any fees?

They will not get any reward for that block.

How do they survive?

By finding other blocks. Blocks are mined, on average, every 10 minutes. That means that a miner has a chance to find the next block if they don't find the current one. Blocks are fairly frequent, and the miner who found one block is not necessarily going to find the next one.

Can the miners choose which transactions validate?

Miners can choose which transactions they want to include a block. They do not get to choose which transactions are valid. It is not the miners' job to dictate what transactions are valid. All transactions that they include in a block must be valid, and they can choose any valid transaction to include.

If it is like that, all the blocks in that time frame, are they the same?

No, they are not. Miners may choose different transactions. The transactions may be in different order. Miners also have full control over the coinbase transaction which is the first transaction in the block that pays them the block reward. This coinbase transaction can be modified by the miner, and typically is modified in order to have more possible blocks for them to try.
Furthermore, the coibnase transaction for different miners will always be different as they will want to pay themselves, not someone else. Thus the blocks will always be different.


text:
Can I change a transaction description in Electrum while the transaction is still unconfirmed?
I inserted a wrong description into a transaction made with Electrum per mistake, so I double clicked the description in and changed it to another value, while it was unconfirmed. My question is, will this last (final) value be visible once it is confirmed?

Descriptions are a feature of the electrum wallet, and do not appear on the Bitcoin chain - someone using another wallet (or even you restoring the keys into another electrum wallet) will not see the description.

So i must always mention the right description at the moment of the transaction itself?

The description has nothing to do with the transaction - it is only for your personal records, it doesn't matter if you leave it blank, put in something random, or put in what you actually wanted to. It has no effect on the transaction and its completion,


text:
Where can I get all historical trades BTC price data?
Where can I get all historical trades BTC price data?
(There are some similar questions, but I am looking for high-resolution data, while the previous questions asked for lower resolution data, such as daily prices.)

Related: https://bitcoin.stackexchange.com/questions/748/where-can-i-get-historic-data-series-of-bitcoin-prices

http://api.bitcoincharts.com/v1/csv/
Each of these files have every trade executed on the given exchange (except the last 15 minutes or so).

Thank you! http://api.bitcoincharts.com/v1/csv/mtgoxUSD.csv.gz worked

any idea what does last field represents  ?

It's a Unix timestamp, the number of seconds since Jan 1st 1970.

that is first column

Amount of BTC traded? It's been a while since I've looked at those. They have a timestamp, a price, and an amount. I just can't remember what order they're in.

Thanks last column values around 1 mostly, so is it that much sold or newly generated, how do I know how much was volume for sell / buy

You have to use the timestamp. The volume in a minute/hour/day would be the sum of the amount of all trades in a range of seconds a minute/hour/day long. Like, the minute volume for Jan 1 1970 00:00 to 00:01 would be the sum of all trades with timestamps 0 to 59. Since every trade is a buy and a sell, the volume is the same both ways.

so one record looks like 1393887724,646.001000000000,0.600000000000
so time is Mon Mar 03 15:02:04 PST 2014 price is right compared with other sources, and last might be volume of trade as you are saying, but what does that trade mean actually is it (number of sell - number of buy / time avg) ?

Tyler, the direction of the trade, i.e. whether you buy or sell from the orderbook, is absolutely essential! This is what makes the price! Unfortunatelly it seems impossible to decode the direction of the trade from this data :-( Or is it there?

You might want to try http://www.cryptodatasets.com/
It has free raw data and sampled data from Bitfinex and HitBtc (currently).
P.S. it's my website

You can also use their JSON API:
http://bitcoincharts.com/about/markets-api/ to get updates to your already downloaded historical data.

Founder & developer of https://tardis.dev here. It's an API I've built that provides high resolution cryptocurrency historical raw market data — tick-by-tick trades, full order book depth snapshots & delta updates, quotes, liquidations etc. There were already services providing similar data, but were very expensive or didn't have all the data I needed - like tick by tick order book updates etc.

Check out www.kaiko.com, we offer tick-by-tick trade data for 2000+ currency pairs, over 25 of the largest crypto exchanges. Same with order book data, on a historical basis. And if you are interested, we're hiring!

for 2300 EUR!! You must be mad

You can get free intraday data for bitcoin and other cryptos here http://www.coinintraday.com
good luck))

Are you affiliated with this site? If so, please disclose that affiliation in your post, because it may otherwise be considered spam :)

authors: Константин Ван


text:
Security and recover issues in blockchain.info wallet
I have a wallet in blockchain.info. I'm quite new to bitcoin and would be happy if someone would explain me :
(1) what should I do in order to save my bitcoins if something happens to blockchain.info. Can I access to my bitcoin with ANY wallet, if I know my address and private key, or if I know my address and password? if so, how do I find my private key?
(2) I've also noticed a backup option in blockchain.info wallet. If I use it, can I get access to my bitcoins with this backup, with every wallet? should I save a new backup after every transmission of bitcoins from my blockchain.info wallet?
(3) Wha'ts the best way to prevent a situation where a malware "reads" my password when I enter it?
Thanks!

(1) what should I do in order to save my bitcoins if something happens
  to blockchain.info. Can I access to my bitcoin with ANY wallet, if I
  know my address and private key, or if I know my address and password?
  if so, how do I find my private key?

You should backup your wallet, using one or more of the options blockchain.info provides.

(2) I've also noticed a backup option in blockchain.info wallet. If I
  use it, can I get access to my bitcoins with this backup, with every
  wallet? should I save a new backup after every transmission of
  bitcoins from my blockchain.info wallet?

With wallets that are compatible with the format, such as Multibit, yes. I think it's safe to say that there'll always be some tool on the Internet you can use to read the file, but downloading the Multibit client (even if you don't plan to use it right now) wouldn't hurt.
As for how often to backup, I'm unsure of blockchain.info's setup...if you only have one address, then you only have one private key, and you don't need to rebackup. If you have multiple addresses, then you need to backup periodically or even after every transaction. You can see if the backup file is any different to see if you actually needed a different export.
Blockchain.info has the following instructions for accessing your backup, even if blockchain.info is permanently offline:

Download the Multibit desktop client available for Windows, Mac OSX
  and Linux.
Launch the application. Choose Tools -> Import Private
  Keys.
Click the "Choose Import File" button and navigate to your
  wallet.aes.json file. Change the file format to *.aes.json and open
  the file.
Enter the wallet password and press unlock. If required
  enter the second password and press unlock again.
Click The Import
  Private Keys Button to complete the process. Your balance will be
  restored once the blockchain has downloaded.

(3) Wha'ts the best way to prevent a situation where a malware "reads"
  my password when I enter it?

Make sure there's no such malware when you enter your password. One way to be sure this is true, is to work from an offline Linux Live CD. E.g. see How to set up a secure offline savings wallet and/or bitBuntu. You can decide for yourself, based on how likely you think it is that you're infected and how much money you're dealing with, whether that's worth the trouble to you.


text:
Withdraw and wallet lock race conditions?
I'm working on a site that has wallet-like functionality in ruby/rails. I'm wondering the best way to process withdraws. My first inclination is to do something like this:
def process_withdraw
  client.wallet_passphrase(ENV['wallet_password'], 10) # unlock the wallet for 10 seconds
  client.send_to_address(address, amount) # send coins
  client.wallet_lock # re-lock the wallet
end

However, I'm concerned that with many concurrent withdraws I could run into race conditions. For example, if "Process A" unlocks the wallet and sends coins, but then "Process B" jumps in and unlocks the wallet but before it can send coins, "Process A" locks the wallet. So when "Process B" tries to send, the wallet is locked.
Is this a legitimate concern? Do I need to re-lock the wallet or is it safe enough to just wait for the 10 second timeout? Is it worth writing some sort of queue system for my client?
Thanks a lot in advance!

What is the password on the wallet protecting you against? If your server is compromised then they have your code and your password anyway, you're just making life hard for yourself. There's really no situation where having a locked wallet is helpful to you here. If you insist, look into making a locking system, process A locks, does it's actions, then removes the lock. Process B queues and waits for the lock to be removed before continuing.

I'm considering running the site on Heroku in production with the dogeoin daemon on ec2 (since heroku doesn't allow write access to the file system.) So the thought was if attackers could somehow get the dogecoind username/password correct, they still wouldn't be able to unlock the wallet. I think you're right though, I'm probably making this too difficult.

Long password + forced SSL on the RPC port, or just use a VPN and don't expose the port altogether. If they did have RPC access but no encryption password, they could just wait for the 10 second window and snake your funds out then.

Yea, great point. I just won't worry about re-locking the wallet. Thanks a lot!

If the wallet unlock is global (that is, it affects a wallet that exists outside the method, which it clearly does), then yes, that is a race condition. Generally, this is handled with mutexes. A mutex provides "mutual exclusion", that is, only one thread of execution can enter an area protected with the mutex.
From a software design point of view, forcing everything that calls the send_to_address method to do so in a mutex is very bad design. It seems you should make a wallet class that encapsulates all the functionality you need to perform on wallets that manages the locking/unlocking of the wallet inside the class.


text:
BTC stuck with zero confirmations
I sent BTC from my blockchain.info wallet to my other wallet 6 hours ago. It is still at 0 confirmations.
Further Detail:

the network  propagation is very low at 0-1%
I used the recommended fees setting(the middle setting)
I sent a couple cents from the same wallet to the same receiving wallet and it went through to 3 confirms after about an hour or so.
my wallet is :
and the hash is:

So, now I'm stuck and can't find any info online that I understand to help me with this.  Any help will be very greatly appreciated.

possible duplicate of Is it possible to add a miners fee after you send your bitcoins?

Rewards aren't necessary here! Its a free and open Q&A site. I think the only rewards allowed here are the built-in bounties.

Looking at the blockchain.info treechart, didn't the final (already confirmed) transaction spent the output created in the unconfirmed transaction? How would it be possible that the final transaction could go through before the used input was validated then?

sorry, I didn't really understand what your saying, I'm too novice : (

When you look at blockchain you will see that your transaction contains 0.0001 BTC which is not zero fee, so it is very likely that your transaction will be confirmed.
You have various options if the transaction doesn't get confirmation:
Reset your blockchain and wait until it's up to date again (keep your device on power and connected to internet). After the rescan, stuck transactions should be gone and your BTC free to spend again.
OR
Relay stuck transactions to a miner that accepts non-standard fees. You can use the trusted peer option for this, just remember to clear that option after your problem is resolved. Also see https://en.bitcoin.it/wiki/Free_transaction_relay_policy.
From: UnstickLowFeeTransactions
Note: There is certain time after blockchain.info clears the transaction if it isn't taken by any miner +- 72 hours. I would recommend you wait this time.
OR

Download electrum or any bitcoin client
Dump your private keys from blockchain.inf and import them. Use Pywallet to delete that transaction from your wallet.
Create whole new bitcoin wallet at blockchain.info and sent entire transacion again. (Wallet mustn't be created under same account).
send your transaction again.

Conditions for confirmation of zero transactions:

Transaction is smaller than 10 thousand bytes.
All outputs are 0.01 BTC or larger.
Its priority is large enough (see the Technical Info section below)

https://en.bitcoin.it/wiki/Transaction_fees
Source of my post:
https://bitcointalk.org/index.php?topic=274185.0
https://bitcoinarmory.com/download/troubleshooting/

Condition 2 is not met, one of the outputs was 4.2mBTC.

@murch Does this mean that I did something wrong? Its been over 48 hours now and still no coins. I'll just wait I guess, thanks a lot everybody for your responses, I greatly appreciate it.

just sit and wait ;)
https://blockchain.info/en/tx/5bf37f4f40b4f47c9f5f66e4de2f788ccb366c038c8a1184f68f1984d4bc8e9f
Estimated Confirmation Time  Very Soon (High Priority)

Hi, thanks for your supporting comment.  It's been over 48 hours now, may I please bounty or pay you to help undo my transfer, It would be very appreciated, thanks for any consideration.

Not a good answer or explanation.


text:
Mining costs, impact on long term viability of BTC
Given the rapidly rising difficulty level and electricity consumption, is the protocol sustainable as-is, and if not what is most likely to happen in the evolution of BTC?

As Bitcoin expands there will be much greater numbers of transactions while the cost of maintaining the network will remain relatively constant, if not decrease, as power-inefficient GPUs and higher consumption ASICs are replaced by more efficient ASICs. The cost per transaction will thus drop.
Also, keep in mind that it isn't replacing nothing, it's replacing centralized accounting which has a significant cost per transaction.

Comments have been moved to chat; please do not continue the discussion here. Before posting a comment below this one, please review the purposes of comments. Comments that do not request clarification or suggest improvements usually belong as an answer, on [meta], or in [chat]. Comments continuing discussion may be removed.

As of 2023 the energy usage is already mother huge: > 0.3% of world electricity production. By design (proof of work). This can become even deadlier problem if it's usage picks up even more, and the price goes way up.
The problem is that proof of work distributes the fixed block reward (which is huge - currently about 9 billion $ per year at bitcoin price $20000) randomly among miners according to their hash rate, setting an incentive for them to increase the hash rate until the block reward barely covers the costs, which are mostly electricity. So the global energy cost spent on "mining" tends to be a major part of block reward, which is proportional to its price up to the halvings, and that's little dependent on technological advances in mining or otherwise.
I don't think it's possible to make an actual prediction what happens because that depends on many external factors that influence the demand for bitcoin and thus it's price. But the problem I described has the consequence that as long as Bitcoin is using proof of work as it does right now, it's price cannot go up indefinitely. So there are basically two scenarios:

the price stays low enough so that the mining reward doesn't get much higher than it currently is or at least not long enough for the mining to adapt to that price. Note that admits a slow raise because of the halvings. Or:

if it doesn't stay low (some predict the price goes up to > $100000, e.g. because commercial usage goes up), the mining resource usage will be that high that politics has no choice but to intervene, e.g. by pushing it's price down again by outlawing it's commercial usage until proof of work is exchanged to something else, like Ethereum did. (This is pure speculation, but if you consider the effect of rumours about ETFs and the SEC attacks on bitcoin price in 2023, you can imagine what a full blown ban on commercial bitcoin usage by the EU and the US would do, possibly enforced by sanctions. In fact the EU already considered something like that, but did not (yet?) do it.)

Just for the fun of future readers: my personal opinion is that it'll go up, politics will intervene in two years, and there will be a modification of POW that will be pushed as the "official" Bitcoin and the unchanged POW-Bitcoin will live on under the radar at low prices. But that's the part where I just guess.
If you like even more details than it's possible to give here - I wrote a blog that contains this topic.
Update: it turns out you can easily verify that this matches the current numbers: the current mining consumption is more than 50% of the block reward, and the block reward is also the upper bound for the money spent on electricity as it pays for the mining.

Reducing the number of miners will reduce the cost/transaction. As the reward decreases the number of miners will also decrease. In the near term (2-4 years) ASIC increases in speed and reduction in power will stabilize. As mining becomes less profitable there will be fewer people mining thus stabilizing the transaction cost.
One needs to look at BTC from the standpoint of a currency, not from the standpoint of a miner--looking to get rich mining.
Yeah, I mine.

It appears to me that electricity costs to mine blocks are rising at an unsustainable rate. Charts of difficulty and network hashes are growing exponentially. This is unsustainable. Currently miners generate revenue of approx. $30 on every transaction. Who pays for this? BTC holders, that's who. (Source: Blockchain.info charts) While it appears that fees we pay to transact are low, we actually pay this much higher amount mainly through inflation right now, due to mining of new BTC.
The big claim in favor of protocol sustainability is that the market will find a balance at the right transaction fee level. I would argue that already it is being shown that effective transaction fees (mining inflation + fees) are too high - mainly because of the underlying costs of producing blocks, of which electricity is a growing component. An unsustainably growing component. And that the low fees we pay to transact are actually an illusion hiding the real costs simply because so much investment money is flowing into the illiquid BTC base, preventing the inflation effect that would otherwise reduce the BTC price being so apparent.


text:
Can the two biggest bitcoin mining pools lock out all others?
I just noticed that the two biggest bitcoin mining pools already have more than 52.5% of the hashing power (as of 11/12/23 Foundry USA had 27,7% and AntPool 24.88%) and am wondering about the consequences of this. Is it true that if they decide to cooperate and only continue blocks produced by themselves or the other pool, that nobody else could ever again squeeze in a block sideways, because they will always produce a longer chain that will invalidate anything the others do, or am I missing something?
If so, could the others do anything about that? Finding out who created a block is probably not quite possible if they don't want you to and distribute the new blocks over their members, right?
UPDATE: I'm especially interested in what would technically happen if they do that and the hash rate relationship continues for a while. I'm aware that there are human factors in this - e.g. their miners could move to other mining pools if they think they've turned evil. But that could also play out the other way around - if, e.g., governments would try to take control and explicitly order the pools to do that for some real or pretended reason. And if my hunch is right, they would be the only mining pools who were able to make any profit, so there would be a big incentive for miners to move into these pools. But that's just an example - the point of my question is especially what would / could technically happen, as the human factor is rather speculative.

Since there has been two answers to the original question already, i think your UPDATE should be a follow-up question instead. Especially since it seems to be partially responding to our answers.

It's certainly not an ideal situation. However them cooperating to censor outsiders' blocks doesn't make it impossible for anyone else to "squeeze in a block" ever again. In order to prevent anybody else from mining a block, they need to keep maintaining a majority of the hashrate. They can't prevent another miner from challenging them.

To answer your UPDATE. As long as they maintain a majority hashrate they can indeed continue reorg'ing out blocks from other pools. You are right that in this case they would be the only ones able to claim block rewards. But an underlying assumption in discussing an evil hashrate majority is that they would censor transactions: otherwise there isn't much harm to users of the system.
In this case they would necessarily have to give up on the fees of the censored transactions, that other block producers would presumably include. Also note how as the block subsidy decreases and transaction fees become a larger part of the block reward the opportunity cost for a censoring 51% attacker increases.

Thank you for your answer! Of course, but what happens technically if we assume they are able to keep the majority for a while - see my update of the question? If nobody can mine blocks but them, I'd suppose they will suddenly grow very quickly, btw.

Edited to answer your followup question.

A mining pool (or several cooperating mining pools) can only prevent others from mining if they maintain a majority of the hashrate. That can only happen if miners continue mining with them. Since miners can switch pools very easily and they probably don't want to participate in an attack on the system they're heavily invested in, it's not an unreasonable assumption they'd leave the pool. We've seen a mass miner exodus last year when Poolin had a liquidity crisis, and about 80% of its miners left within a few weeks.
It's also unlikely a mining pool could launch such an attack in secret. All mining pools currently label their blocks, so if anyone stopped it would immediately be suspicious. Even if that weren't the case, (at least some) miners in a pool necessarily know which blocks are mined by that pool, so there's really no way to keep that a secret for very long.

Thank you for your answer! Sorry, I was a bit imprecise in my question - please see my update. An attack like that could conceivably be carried out in the open by governments ordering the mining pools to do that, with most people not objecting to it. So I'm curious about the technical aspect.


text:
Collisions when adding an address entry to the new table
What does the bitcoin protocol do to handle collisions when adding addresses to the new table?

What addresses and what new table?

The addresses that are in the addr message are placed into the new table of bitcoin core.

Surely addr entries can't collide because there is no expectation that each peer has no peers in common. If two peers report a common addr entry wouldn't you just keep the common value in your table?

In case of a collision of addrs in the new table, the existing address will normally stay (and nothing changes) - unless

the existing entry is already present somewhere else in the new table (an address can be in up to 8 different buckets/positions at the same time), or
if the existing entry is of bad quality ("terrible")

In these two cases the existing entry will be replaced by the new one.
There are several criteria by which a node can be terrible, roughly:

if its timestamp is too far in the future or too far in the past from the current time.
if we unsuccessfully tried to connect to it too often in the past

See Addrman::IsTerrible() for details / exact limits.
It may also be worth to mention that the behavior in case of collisions in the tried table is very different from this ("test before evict").