text:
How bitcoin application in production retrive UTXO's
AFIK, UTXO of an address cannot be retrieved using bitcoin-core, without importing the address in the wallet.

How are application in production able to get UTXO ?
Does all the application use Indexed database like Insight or Abe to get UTXO ?
Is there any way to get UTXO from bitcoin core without importing the address into the wallet?

Most services run their own services which track the UTXOs that they need. They use Bitcoin Core as an edge node which forwards all of the valid blocks and transactions to their internal software that adds them all to a database. This could be a software based on insight or Abe or something homegrown. In this way, they don't need to implement consensus and validation logic, just deserialization and database storage.

Is there any way to get UTXO from bitcoin core without importing the address into the wallet?

Bitcoin Core has a scantxoutset command that you can use to get the UTXOs for a particular scriptPubKey (from an address, descriptor, raw script, etc.).

My major concern was having privatekey in the Bitcoin Core wallet, So I did generate the address outside the Bitcoin core from third party library or HSM and used importaddress RPC method to import the address as watch-only address and then Bitcoin core handled all UTXO monitoring, Then use listunspent to get the UTXO of the imported address. By this approach privatekey is cold from the core.
Thanks @yyforyongyu for suggesting this.

This is a valid approach for using Bitcoin Core as a cold wallet. In the future I suspect many Bitcoin will be lost as the result of inheritance where the priv-key must be passed from one generation to another, but usually, the priv-key will be in the offline wallet to sign transactions and only the password will be necessary. The transactions can be hand loaded into mempool of the online wallet to gossip.

Most services operating large scale wallets only use Bitcoin Core for networking information, and maintain a separate utxo set for wallet purposes.
There are many existing options to build a full utxo set that can be queried for arbitrary addresses, such as the insight project.
Moreover, recent versions of Bitcoin Core come with the scantxoutset RPC call, which allows you to locate unspent outputs for a given address or locking script.

Speaking as a formal wallet engineer, what we did back then,

Once in a while, we would batch generate lots of bitcoin addresses and import them into bitcoind.
Can't speak for ALL applications, for us, we let bitcoind manage all its UTXO set.
Depending on the situation,

if the address is newly generated by you, then no, you have to import it first because, well, it's new, you have to tell bitcoind to use it.
if it's a used address, I believe you can parse its UTXOs from all the block data, though we never did that.

Thanks, Does having address imported in the wallet suggested for production ?, What are the safety measure that has to be taken?

Yes. You probably need to periodically import a batch of addresses into the wallet, as a production site, however that's more of a topic of software engeering. As for safety measure, if you are generating those addresses, it's recommended to use HD wallet so your private keys won't get exposed.

Using an HD wallet does not prevent exposure of your private keys, especially if you are importing them into Bitcoin Core

Import addresses as watch-only; 2) Use an HD wallet, and save the master private key offline; 3) Construct raw transaction online, send it to offline signing. How's my private key exposed?


text:
Are Bisq offer ids picked or partially picked by humans?
Two recent Bisq "offer ids" are:

"emzlols-7639fae0-cd4d-48cf-acc5-abc97dab89db-124"
"aixcp-6efe1e8d-90bf-4c25-b021-53616cad1109-124"

(Emphasis mine. These are fully public, so I don't see a reason to redact/censor them here.)
I can't help but notice that the first says "lols" in it. Is this a pure coincidence, auto-generated randomly, or are these id strings somehow picked by the offer makers/sellers? At least the first part before the first dash?

isn't that the username of the person that created the offer?

Hello @bisquit, if you found an answer helpful, please mark it as the accepted answer to your question so others will know too.

Bisq Offer IDs are randomly generated and consist of 3 parts separated by "-":

Prefix: between 5 to 8 characters in length
Randomly generated UUID: 5 sections of 4, 2, 2, 2, & 6 hex values separated by "-"
Bisq Version: "." separators removed

In your first example, emzlols-7639fae0-cd4d-48cf-acc5-abc97dab89db-124, emzlols is the Prefix, 7639fae0-cd4d-48cf-acc5-abc97dab89db is the Randomly generated UUID, and 124 indicates a Bisq Version of 1.2.4
The IDs are created in a method called getRandomOfferId() in the bisq.core.offer.CreateOfferService class as of Bisq v1.2.4.
The Short ID for an offer is its Prefix.

So it has nothing to do with the username?

Yes, it has nothing to do with the username.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Converting Bitcoin compressed address to uncompressed or vice versa?
Is this possible with just an address; if so how?

Peter Wuille's comments are canon, and in case you haven't seen this tool, for learning  https://iancoleman.io/bitcoin-key-compression

A 1... address encodes the hash of a public key (specifically RIPEMD160(SHA256(key))). Since cryptographic hashes aren't easily invertible, the address alone cannot be used to convert between compressed and uncompressed keys. If however the owner of this address has spent coins belonging to it (or has otherwise published the public key), the public key is now known and can be used to construct an address from both its compressed and uncompressed version.
You should never actually do this though, for reasons explained by Pieter in the comments.

Even if it is possible after seeing the public key, it is a terrible idea. The receiver may not be able to spend coins sent to the converted address, and may not consider coins sent there to be a valid payment. The advice in matters like this should be: as the sender, you should never ever attempt to convert someone else's address. As the receiver you can "convert" by upgrading to wallet software that gives out such addresses in the first place.

@PieterWuille By "may not be able to spend the coins" you mean simply that the receiver may lack a convenient method of spending them, or that they truly have no way of spending them? And how would that happen?

For example if their private key is on a security module or HSM, which only supports signing in certain ways.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Mainnet node sync really slowly with 12 tb seagate external hdd says data rate is 6gb/s
Hi my node is syncing really slowly I have the datadir in an external hard drive connected through usb. Says the data rate for this seagate 12TB is 6Gb/s but the syncing is slow in comparison to when I have synced nodes in the past with internal nvme drives but 2TB is low so I am moving up to a larger hard drive and went with the external.
Any thoughts or someone who experienced similar? Even when communicating with RPC it is slower than normal I thought maybe it was because it was syncing but I am beginning to question it. Thanks for whatever input.
Also my internet is 1gbps upload download and it is connected through ethernet did a speed test to verify.

Hard disk drives have been known to sync much slower than SSDs due to their slower read/write speed and higher latency. Even a SATA (non-NVMe) SSD will do a much better job. This also applies to RPC calls that need to read from the disk.
6 Gb/s (768 MB/s) is just the theoretical limit of the interface your drive is connected through, no hard disk drive can actually read or write at this speed.

ahh damn thats kind of what I was starting to realize oh well, thanks

also if have any memory recommendations seems hard to find larger SSDs like 8TB+ that aren't insanely expensive am thinking about combining nvme's somehow

Have you considered combining a HDD with a smaller SSD for optional fast storage? The Bitcoin blockchain is currently a little over 500 GB, so even just a 1 TB drive will last you for some time.

No but i will, I ran out of space before thats why I was trying the HDD but I might have messed up the configuration somehow or double synced somehow

was also an ethereum and btc node on the same computer neither are fully synced right now but i think like 80% each its about 961G I have 3TB so maybe I can use them with the HDD

Increasing the database cache can dramatically speed up synchronization, especially on disks with slow I/O like spinning ones.


text:
How do bitcoin nodes rollback state when a higher POW chain is introduced
Let's say the current chain consensus has a total POW of X... this is agreed upon by all 10k+ BTC nodes.
A new "magic" node joins and it has mined a chain (privately) with total POW = 2*X. The new nodes chain is on the order of hundreds of thousands of blocks as is the current consensus chain...but they are totally non-overlapping, how would the "magic nodes" new higher POW chain propagate through the network (or would it?)

There is no limit to how far a node will reorg given a chain with a larger proof of work, as long as the chain also starts from the hardcoded genesis block (which means they cannot be completely "non-overlapping").
As far as I can tell, the propagation of the new chain would go something like this*: The nodes first coming into contact with the "magic node" will start downloading and validating this new chain, until they validate far enough that its proof of work has exceeded their original chain's POW. This might take minutes to days depending on the distribution of mining difficulty in this new chain, how many transactions are in each block (if any) and also on the node's processing power and bandwith. At this point the node will start relaying this new chain to its peers and they would start validating it. Thus the new chain will slowly but surely propagate through the network until all nodes are synced.
* Based on some assumptions about how Bitcoin nodes should work.

Got it. I'm interested in the technical details of "coming into contact" ... My understanding is that the bitcoin nodes only validate against a peer header chain upon initial start or restart... then they depend on receiving propagated blocks from neighbors. So there seems to be an edge case if no node in the network reboots (and thus no node ever loads the new header chain from the "magic node")

A node must be able to accept conflicting chain headers at any time there might be a reorganization, not just when it's restarted.


text:
Segwit transaction in golang
I tried to create transaction which spends p2sh(p2wpkh) output:

from wif I derived compressed pub key: "035476e55521c97cf0489e6c57893a99d67e84d306a69e6b6904f324c11b3cef8d".
created witness program: "813798fcae182d1a36c6b4bb5f69e03bb9ec4124".
created script: "0014813798fcae182d1a36c6b4bb5f69e03bb9ec4124" and sigscript from script which should be specified to spend this ouput: "160014813798fcae182d1a36c6b4bb5f69e03bb9ec4124".
added tx input: {hash: "863fea23ba1c176bda5e360b0be25cacc405877aa3e46a3e1c60196c3b6cd7b6", vout: 0, signatureScript: scriptSig, witeness: nil}
created pkscript for address "14s9ocrbTX3ZCybtcN1jmtdxxJwKuyM3Em" (p2sh-p2wpkh): "a914809800f62119afdcf2317882eda5efd7a9a2163487".
added txout {amount: 11000, scriptPubKey: pkscript}
calculated txSigHashes := txscript.NewTxSigHashes(tx)
decoded utxo subscript into bytes: "a914bb4d332112bde9787ac2da5954ee5a2c5fbada6987".
derived witness signature: wit := txscript.WitnessSignature(tx, txSigHashes, 0, amount, subscript, txscript.SigHashAll, wif.PrivKey, true)
result: "3045022100bae83fd9e87751350a09bc4c120b29decadb13fb389541c3025a8982906602d5022005dc688901fd27b749cfd4ce14a62587f267a070984654271454e88cc3f9ed4e01 035476e55521c97cf0489e6c57893a99d67e84d306a69e6b6904f324c11b3cef8d".
added witness signature to tx.TxIn[0].Witness
Serializied signed tx: "01000000000101b6d76c3b6c19601c3e6ae4a37a8705c4ac5ce20b0b365eda6b171cba23ea3f860000000017160014813798fcae182d1a36c6b4bb5f69e03bb9ec4124ffffffff01f82a000000000000434104b2534d287f2d9fa908319557298591143fecbd3f6497d6d11f283b80c3248c35f580ff376805418c3610256bc102c0beb3e9f745d493aed81127f5d4a4921bebac02483045022100bae83fd9e87751350a09bc4c120b29decadb13fb389541c3025a8982906602d5022005dc688901fd27b749cfd4ce14a62587f267a070984654271454e88cc3f9ed4e0121035476e55521c97cf0489e6c57893a99d67e84d306a69e6b6904f324c11b3cef8d00000000"

Even if I followed all the rules to create this transaction any try to push it into mainnet (via Blockcypher and other services) failed because of this error: "Error validating transaction: Error running script for input 0 referencing 863fea23ba1c176bda5e360b0be25cacc405877aa3e46a3e1c60196c3b6cd7b6 at 0: Script was NOT verified successfully"
So my question is where have I wrong done? Which step is wrong and how to do it in a right way?

In step 9, are you signing with the entire value of the input under amount?

yep, the amount is equal to 12780 satoshis

Does this "outpoint 0" really contains funds? When looking at blockchain.info, I thought I can see, that the funds  for tx 863fea23ba1c176bda5e360b0be25cacc405877aa3e46a3e1c60196c3b6cd7b6  might have been sent already. Can you do a "listunspent | grep -A9 -B1 $tx_id" ? (where $tx_id equals the tx number from step 4). You may edit your original question and copy&paste output of the command...

I found that witness didn't include serializied script: that func for some reason just ingnored writing script into buffer, that means algorithm specified here(https://github.com/bitcoin/bips/blob/master/bip-0143.mediawiki, specification paragraph) ignored 5th step. So I removed if statement, and instead of (https://github.com/btcsuite/btcd/blob/master/txscript/script.go#L496) pasted writing has160 of pub key and that worked for me


text:
Relation between multi-sig and single-sig addresses (in BCCAPI library)?
I had a multi-sig address M (starting with 3) that I was trying to send a small amount of coins to. I was playing around with BCCAPI and the coins got sent to an "ordinary" address A (starting with 1)
I had supplied M to the library but somehow the signed tx had the address A
I know who controls M but no one seems to control A. I am trying to see if the funds can be recovered by someone controlling M.
Is there any relation between the two addresses? I'd prefer not to post the actual addresses yet. However, the fact that I used BCCAPI might give some clue.
Here is the code (in Scala) that I used:
val M = "3..." // multi-sig address
val receiver = com.bccapi.bitlib.model.Address.fromString(M, netParams)
val so = new com.bccapi.bitlib.model.ScriptOutputStandard(receiver.getTypeSpecificBytes())
...

The call to receiver.getTypeSpecificBytes() strips off the version byte (the '3'), and ScriptOutputStandard() always creates a P2PKH script from this raw (with no version byte) 20-byte hash. The bitcoin that was sent to the P2PKH address is probably irretrievable.
You probably meant to do this to create a P2SH script:
val so = new com.bccapi.bitlib.model.ScriptOutputMultisig(receiver.getTypeSpecificBytes())

Also, just FYI I don't think BCCAPI is being supported any longer. I don't see any updates to it since late 2012, and the then maintainer has since moved on to create Mycelium. You may want to consider an alternative Java Bitcoin library such as bitcoinj.

The bitcoin that was sent to the P2PKH address is probably irretrievable is the part I'm interested in confirming. Its a small amount so if its futile, I won't waste my time. I used this exact method to create ordinary transactions without problems. It only happened with multi-sig. I'm hoping that the public key in the output might be one of the public keys of the multi-sig.. But thats probably not true.

If your "3..." M address is in fact a valid P2SH address created by a working wallet or library, then there is no way to recover any funds sent to the A address created by your code above. The private keys for the A address cannot be determined even if you have the keys for the M address. Sorry.... :(

In that case I should consider myself lucky that the amount lost is small. A lessen well learnt.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
What is type 'pubkey' in a scriptpubkey output from getrawtransaction?
Here is an output from getrawtransaction:
{
  ...
  "vout": [
    {
      "value": 12.50000000,
      "n": 0,
      "scriptPubKey": {
    "asm": "03d81b3d2ac76e322dcb2e713cb8fdeaf68cb83406c0e4d28dcd10a880ef172a6a OP_CHECKSIG",
    "hex": "2103d81b3d2ac76e322dcb2e713cb8fdeaf68cb83406c0e4d28dcd10a880ef172a6aac",
    "reqSigs": 1,
    "type": "pubkey",
    "addresses": [
      "mwzWcEU4kBkJPSTgB8LSBfvyaEjdXmyGh7"
    ]
      }
    },
    ...
  ],
  ...
}

What is a pubkey type? I could not find any reference to this.
Here is the output of decodescript on the script:
{
    "result": {
        "asm": "03d81b3d2ac76e322dcb2e713cb8fdeaf68cb83406c0e4d28dcd10a880ef172a6a OP_CHECKSIG",
        "reqSigs": 1,
        "type": "pubkey",
        "addresses": [
            "1HUZKBP5wAK3cKz4TZN4MkieiF8vbyNvpE"
        ],
        "p2sh": "3469piMXSTikVAdrSC791pQGxQb8pNXhua"
    },
    "error": null,
    "id": null
}

The script for an output describes the condition that must be satisfied for the money to move.
It means that the public key is contained in the scriptPubKey. Usually, the scriptPubKey only contains a hash of the public key, and relies on the scriptSig to supply the public key.
Explanation of the asm field in decodescript:
"asm": "03d81b3d2ac76e322dcb2e713cb8fdeaf68cb83406c0e4d28dcd10a880ef172a6a OP_CHECKSIG",

The first element is a compressed public key. The second element is OP_CHECKSIG. To spend this output, someone must put a signature in a scriptSig.
See also the P2PK bullet point here: https://bitcoin.stackexchange.com/a/35458/2306

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
What is time warp attack and how does it work in general?
I am trying to understand the recent time warp attack on verge but since the attack is universal across all POW block chain i figure I could also ask here since people here are .. more knowledgeable.
I don't understand the detail of code.
But i assume something like, in bitcoin, there is a difficulty parameter that is decided for every 2016 blocks. Perhaps a integer. Such that all 2016 blocks mined during that that period all needs to have a hash that is less than that integer.
This difficulty is recalculated whenever 2016 blocks are mined. Everybody goes over this entire 2016 blocks, and take the earliest time stamp, take the latest timestamp, calculate how long it took to mine those blocks, and decide on a new difficulty integer for the next 2016 blocks.
But how does time warp works? I read something along, miners starts submitting blocks with large drifting timestamps (i think it is 2hrs for bitcoin?), but 2 hours is relatively small compare to the average time it is going to mine 2016 blocks (300+hrs). How would add or subtract 2 hours in the blocks cause mining to difficulty to decrease dramatically?
For a miner to drastically decrease difficulty, wouldn't he need:

Majority hash power so he can construct all 2016 blocks.
Change all timestamp in this 2016 blocks in a way such that it causes
dramatically decreased difficulty
profit... (but by that time people probably abandoned the chain anyway)

But once we get to point #1 we are.. kind of in trouble already, it's
just that having time warp made it worse???
Thanks!

The Bitcoin Protocol (consensus rules) has two relevant rules for the timestamps in block headers:

A node will not accept a block whose timestamp is more than two hours in the future.
A node will not accept a block unless it has a timestamp greater than the median of the previous 11 blocks.  In Bitcoin, we call this Median-Time-Past (MTP).

As you mention in your question, difficulty changes are calculated based on the times of the first and last blocks in a 2,016-block difficulty period.
Given the above rules, if all miners agreed, they could simply increment the clock the minimum amount of one second MTP for the first 2,015 blocks and then set the time to two hours in the future.  That would basically give them just a small decrease in the difficulty, but think about what happens to the MTP when they add that last slightly-future datapoint: the actual median doesn't change much at all.  Actual time stamps are in seconds, but here's a set of 11 timestamps in days delta from present time:
[-13, -13, -13, -13, -13, -13, -13, -13, -13, -13, 0]

The median of the above is -13, meaning that after miners create the slightly-future-time block at the end of the difficulty period, they don't need to move their timestamps forward more than the minimum one second---so the next difficulty period starts out -13 days.
At the end of the next difficulty period, miners again move the timestamp forward as far as possible, so the protocol thinks it took 28 days to mine the blocks---half the expected speed---and so decreases the difficulty by about half.  Now the values used for MTP look like:
[-27, -27, -27, -27, -27, -27, -27, -27, -27, -27, 0]

So miners can continue keeping timestamps far in the past and repeat the attack, lowering difficulty every period until the point where it takes them less than 2,016 seconds to produce 2,016 blocks, at which point they can't lower difficulty any further because the MTP function requires time increase by a minimum of one second over the median every block.
Now, your main question was how can this attack work without collusion by a majority of miners.  Now that you've seen how the attack works with all miners participating, it should be clear than selecting the median time can allow an attacking miner who's lucky enough to find blocks reliably to prevent median time from jumping forward to an honest value.  For example, imagine these are the times of the previous 11 blocks, in block chain sequence:
[-27, 0, -27, 0, -27, 0, -27, 0, -27, 0, -27]

If you sort those numbers to find the median, it's -27 even though 5/11ths (45%) of the hashrate is mining accurately.  But wait, doesn't that mean the attacking miner has 55% of the hashrate?  Maybe not, for a large miner with about 30% or more of the hashrate could obtain an advantage over other miners using a selfish mining attack, or the miner could simply threaten to attempt to make stale ("orphan") other miners blocks who have accurate time stamps, causing those honest miners to earn less income.
I myself don't consider the attack particularly likely on Bitcoin because it's slow to execute and publicly obvious, but it's something protocol designers do need to keep in mind for when they change parameters, as those changes could make the attack easier to execute.

re: "it's something protocol designers do need to keep in mind for when they change parameters": a perfect example of this is the recent attack against Verge (XVG), where a different algorithmic setup allowed a timewarp attack to be more potent. David has another great answer on this here: https://bitcoin.stackexchange.com/questions/75438/what-prevents-similar-time-warp-attacks-in-bitcoin-as-happened-to-verge

It seems wrong for a node to accept: [-13, -13, -13, -13, -13, -13, -13, -13, -13, -13, 0] ? Isn't the last block "more than 2 hours in the future" from the previous block ? It seem it's 13*24=312 hours in the future. The protocol's notion of time comes from the timestamps of the blocks and the time reported by peer nodes.

@dbkeys by "2 hours in the future", I mean by the node's own clock (e.g. its host system's clock).  For example, it's 2018-10-03 11:00 UTC as I write this; my node will accept any otherwise-valid block right now with as timestamp up to 2018-10-03 13:00, but will not currently accept an otherwise-valid block with a timestamp of 13:01.  If we wait a minute, then my node would accept the 13:01 block.

"they could simply increment the clock the minimum amount of one second MTP for the first 2,015 blocks..." It's not clear to me what exactly is going on. What clock is being incremented 1 second, (is it the system clock, or the timestamps ) ? how many times is it being incremented one second ?

"Technically there's an off-by-one error there, but that's not important here." -> I think it's important. If the difficulty re-target were overlapping (as they would if it was based on the last 2016 blocks instead of 2015), the timewarp attack wouldn't be possible. Of course miners could still hold off timestamps in the past but it would be much less effective.

@AntoinePoinsot edited to correct this, thanks so much for your comment here and for walking me through it in more detail in a chat!

I was reviewing the Testnet 4 BIP yesterday and Pieter made me realize that the minimum timestamp increase of blocks is not 1 second per block, but 1 second every sixth block.

@Murch I've described that in other places, e.g. https://bitcoinops.org/en/newsletters/2019/03/05/#the-time-warp-attack , but I think it would be good for somebody to go into detail on that, so I created https://bitcoin.stackexchange.com/questions/123698/how-many-blocks-per-second-can-be-sustainably-be-created-using-a-time-warp-attac

A time warp attack occurs when a miner manages to put an incorrect time-stamp into the blocks in order to reduce the subsequent mining difficulty.
By inserting wrong time-stamps a miner, or a group of miners with majority of the hash rate can deceive the difficulty adjustment algorithm (DAA) that deals with adjusting mining difficulty, thereby pushing it to lower it's value, causing extraction of the blocks to become faster and therefore make mining more profitable.
As per the consensus rule, Bitcoin won't accept a block with a time-stamp more than 2 hours in the future, and it won't accept a block unless its time-stamp is greater than the median time-past (MTP) of the previous 11 blocks.
A miner can exploit this consensus rule to mine block in a difficulty adjustment period of 2,016 blocks by incrementing the timestamps of the blocks mined by merely 1 second greater than the MTP of the past 11 blocks for 2,015 blocks and then insert the time stamp as current time + 2hours.
This will not bring any drastic effect in the first time epoch. However, for the subsequent difficulty adjustment period, the miner can manipulate the time stamp of the first block (block 2,017) and set it to be 1 second greater than the MTP (block 2,011). This will deceive the DAA into believing that the first block of the second difficulty adjustment period (2,017) was mined ~13 days ago, and then insert the timestamp of time final block of the second difficulty adjustment period to be current time + 2hours, and thus it would appear that it took ~28 days to mine 2,016 blocks in the second difficulty adjustment period, which will lead to a decrease in the difficulty of ~50%.
It is important to note that DAA takes into consideration the timestamps of the first and the last blocks in a difficulty adjustment period to adjust difficulty.

The time-stamp for the first block in the second difficulty adjustment period(block 2,017) should be minimum 1 second greater than the timestamp of the median of the previous 11 blocks(block 2,011)i.e 6th previous block, which is ~13 days in the past, while the time stamp of the last block of the second difficulty adjustment period is C+2 (Current time + 2hours), thus deceiving DAA to believe that it took ~28 days for all the blocks in the second difficulty adjustment period.


text:
Sending bitcoins to same address
I am using the BitcoinJ API to create my wallet.
When I create a transaction, I provide the change address to which change should be sent.
For example: let there be 1 bitcoin at address A;
then I create a transaction to send 0.1 bitcoins to address B and the change (0.9 bitcoins) back to address A.
In the BitcoinJ wallet, I see that the change (0.9 bitcoins) is not received back. To try to fix that, I created a custom coin selector to explicitly select unconfirmed transactions which are generated by myself---but still I do not see the change (0.9 bitcoins) in my wallet.
This is the transaction view https://www.biteasy.com/testnet/transactions/db66d5058e5773f353447ffa2174a417867e9aa134113f01359576f5e6f2b4c6
And this is my wallet https://www.biteasy.com/testnet/addresses/mnj3589DHMoCyhxDSvWKubjkUoTenmrGkE.
According to my address on Biteasy, there is a balance of 4BTC,
but my BitcoinJ wallet shows only 0.069BTC
Now my question is: Why can't I send return change to same address it is being sent from?

You can send bitcoins to the same address you are sending from, if you create a raw transaction manually, or if you use a wallet that does not create a new change address each time a transaction is made.
One reason why this new address process exists is to protect privacy. If you consider that the outputs of a transaction with change, where one of the outputs points back to itself, you will immediately be able to determine how much you paid, and factually, that you paid that amount to someone else, keeping the change.
If on the other had a different address is used for change, an independent observer cannot determine which of the two amounts was change and which was the payment.
However what I suspect you are doing is the classic mistake of thinking that the difference between what you are paying to the other address and what was in the originating address is automatically sent back to you.
I cannot emphasize this enough - the difference is sent to the miner as a mining fee. You have in effect not explicitly told the transaction what to do with the difference and it is therefore assumed that this is a fee that you are paying, and is collected by the miner who mines the block containing your transaction.
In order to avoid this, each transaction must have a minimum of two explicit payments, and one implicit payment:

Input amount
Payment to address 1 (must be less than input)
Payment to address 2 (must be change minus the fee that you want to pay)

The input must be explicitly greater than the sum of payment 1 and 2, and anything left over is implicitly the miners fee


text:
How does bitcoind store blocks?
I want to parse blockchain manually and extract the UTXO set (for various reasons). Is there any way to understand how the blocks are stored without going through the C++ code?
Some specs will be great.

There is a lot of information available, just not consolidated. After searching for sometime, I found the best answer from this link.
The format of each file is:
4 bytes: Magic bytes
4 bytes: Encodes size N of next upcoming block.
N bytes: Block encoded in standard format (with header)
--
above repeats

Quotes from the original link:
4 | 4 | 80 | TxData | 4 | 4 | 80 | TxData | 4 | 4 | 80 | TxData | ...

First 4 bytes: magic bytes (identifying which network you are on)
Second 4 bytes: the number of bytes of the remaining block
Next 80 bytes: block header itself
Next NumBlockBytes - 80 bytes: Transaction data in this block [ numTx | Tx1 | Tx2 | Tx3 | ... ]

What I don't understand is why are the magic bytes repeated? We could very well have used:
4 | 4 | 80 | TxData | 4 | 80 | TxData | 4 | 80 | TxData | ...

What I don't understand is why are the magic bytes repeated? <<<< because Core programmers use the same serialization routine for transferring blocks by network and storing on disk. You can store blocks on your own hard-drive in any format. Nobody cares.

Yup, makes sense. I was wondering, if some space could be saved if bitcoind didn't store those bytes.

I want to parse blockchain manually

If you mean blockchain is a RAW on-disk data, which contains all information about the blocks and transactions (../blocks/blk*.dat files), so you can try blockchain parser. This is a standalone script that can parse that files. No installation needed, no dependencies.

Is there any way to understand how the blocks are stored without going through the C++ code?

The blockchain parser by the link can give you the way to understand how the bocks are stored in a simple intuitive view.

extract the UTXO set

For extraction of the UTXOs (for constructing UTXO set) you need to understand that this data did not stored permanently on disk.
There are two ways for building the UTXO set:

Parse the chainstate frozen dump (../chainstate/*.ldb files)
Construct UTXO set from parsed RAW data (../blocks/blk*.dat files)

The first way is faster, but not so usefull for deep understanding of how blockchain really works.
The second way is what I'am trying to do now. This also called blockchain verification by third-parties scripting. I made the script for UTXO set building from RAW dumps, but my implementation on Python is very very slow.
I hope my answer may help you in your explorations.


text:
Why does P2SH-P2WPKH use DER encoded signatures?
If we simply assumed 32 bytes of r and s each, then signatures can be 64 bytes. Now they are on average 71 bytes in DER plus an additional byte to declare their size, effectively making them use 72 bytes of space. Similarly, if we assume public key is always compressed then we can simply keep 2 stack items of 64+32 bytes. I guess it is to do with the way OP_CheckSig is implemented.

Segregated Witness was already a large collection of changes that impacted several parts of Bitcoin (P2P, validation, fee estimation, sighashing, script execution, ...).
Many small improvements could easily have been added to the proposal (including the signature encoding, the superfluous stack pop in OP_CHECKMULTISIG, the signature algorithm, various script extensions, new sighash flags, ...). However, we wanted to avoid letting the scope grow too large, to avoid bikeshedding and review burden. Instead, a script versioning system was introduced that would make it possible to later pretty much enable adding in any other change as a softfork).
So the answer is that DER is still in use in P2WPKH simply because it was in use in P2PKH and it didn't change. However, I expect that future proposals will come soon enough to change it to pure 64-byte signatures.


text:
Why are Lighthouse campaigns limited to 684 pledges?
I'm curious to why is there a 684 transaction limit for each campaign when using the lighthouse crowd funding app. If I was wanting to raise 25 BTC for a project, the minimum that someone that would have to fund me is .036 (about $10 today). This leaves out the ability for people wanting to fund $1-$5. Just curious to why is there that limit and why wont it enable smaller amounts? I was contemplating on using lighthouse or nxt's monetary system.

Lighthouse's Frequently Asked Questions (FAQ) says:

Currently you cannot accept more than 684 pledges for your project. This is due to limitations in the Bitcoin protocol - the final contract that claims the pledges is a single Bitcoin transaction, and thus is limited to 500 kilobytes by the Bitcoin block size and protocol relay rules.

Note: "500 kilobytes" is a typo---I just confirmed this via email with Lighthouse author Mike Hearn.  The correct size for standard transactions is 100 KB.  Mike is also thinking about bumping up the standard size in his Bitcoin XT project to allow miners to opt-in to the larger size.  (That hard limit, for now, is slightly less than 1 megabyte---a whole block.)
Of that 100 KB, a small part is the output (TxOut) that pays the receiver of the funds (you, in your case).  If you use a standard P2PKH or P2SH pubkey script, the output will be 34 bytes or less.  There's also a small part dedicated to the transaction metadata (such as the transaction version number); the size of the metadata varies a bit by how many pledges you get, so lets round it up to 20 bytes.  That leaves 99,946 bytes left for pledges.
99946 / 684 = 146 bytes per plegde (rounded down).  A pledge is a single input (TxIn) that pays the output described above.  The input is (if necessary) pre-spent to ensure it has exactly the pledge amount---that means there's no change output to worry about.  The Lighthouse design document doesn't explicitly say what type of input it spends, but I'll assume it only spends P2PKH inputs using compressed public keys, which are:

36-byte outpoint
up to 72-byte signature
1-byte signature hash
33-byte compressed public key
2 bytes of data pushes

... for a total of 144 bytes.
The Lighthouse FAQ also says in that same section:

A future version of Lighthouse could optimise its usage of transaction space to bump up the maximum number of pledges by another few hundred.

I suspect the optimization Mike is implying would be using P2PK inputs, which would be:

36-byte outpoint
up to 72-byte signature
1-byte signature hash
1 byte of data pushes

... for a total of 110 bytes.  If implemented, this would allow 99946 / 110 = 908 pledges per transaction, or about a 30% increase.  There are a few other ways to optimize, such as requiring smaller signatures, but these provide rapidly diminishing marginal gains.


text:
Is there any open source implementation of Bitcoin wallet encryption in Java?
I intend to establish an efficient and secure scheme for protection of wallets. For this , I need source code that implements wallet ecryption as mentioned in https://en.bitcoin.it/wiki/Wallet_encryption

BitcoinJ is the Java open source library used by many popular wallets, including Bitcoin Wallet for Android, MultiBit, GreenAddress GreenBits, and lots more.
It supports some sort of encryption of private keys, although it may not be AES-256-CBC (but it probably is).
HD Wallets
It's worth noting that BitcoinJ was recently upgraded to support creating and using BIP32 hierarchical deterministic (HD) wallets.  This is an option, and the old method of generating separate random keys is still available.
When used in HD mode, as the wallets listed above do[1], it won't store the private keys directly, but will simply generate them on demand from the HD root seed, so it's only the HD root seed that needs to kept encrypted.  Also users are strongly encouraged to keep a paper backup of the root seed, meaning that digital backups of the encrypted key are usually not needed.
[1] MultiBit currently doesn't do it, but the in-beta-test MultiBit HD does.
Bitcoin Core Wallet
In addition, Bitcoin Core developer Jonas Schnelli is working on splitting off and replacing Bitcoin Core's backend wallet code, so you may not want to start on any major projects without coordinating with him.
One of the proposed objectives for his project---although I haven't heard his take on this---is adding HD support to the wallet code.  (In any case, HD support will likely be added as an option; at least one core dev would like to keep random-derived private keys for some applications because they've been much better studied than HD derivation.)


text:
Bitcoin nodes voting
I'm just watching Adam Back & Gregory Maxwell: Blockstream and Sidechains interview, and wondering how the decision of further Bitcoin core development are made. I know about Bitcoin Foundation, but I do not think that board members represent majority of voting power, when it comes to decision making for bitcoin core development, altough there are some core development members.
Who does make the decision which feature of bitcoin to include?
What will be the procedure of voting (who and how) would decide whether or not to include Sidechains?
Does the participant nodes (roughly more than 6300) vote as well?

Who does make the decision which feature of bitcoin to include?

The Bitcoin Core developers, particularly the ones with commit access, make the ultimate decision about what features to add to Bitcoin Core---but they do listen to community feedback, particularly polite and knowledgeable comments left on GitHub pull requests.

What will be the procedure of voting (who and how) would decide whether or not to include Sidechains?

Sidechains will initially be introduced as federated sidechains, which is described in Appendix A of the sidechains paper.  Federated sidechains don't require any changes to Bitcoin Core or the protocol, so there will be no voting.
Merged-mined (DMMS) side chains can be added as a soft fork.  If this is done the same way as the BIP34 soft fork or the upcoming BIP66 soft fork, it'll be miners that vote.  They'll cast their vote by upgrading to the version of Bitcoin Core that includes the new SPV-proof redesignated op code.
However, the only-let-miners-vote method is mainly intended for non-controversial forks.  (We need miners to vote, because only they can enforce a consensus rule change (fork) in a minimally-disruptive way.)  If adding the SPV proof op code is controversial, we'll probably need to do something different to ensure minimal disruption.  (See point below.)

Does the participant nodes (roughly more than 6300) vote as well?

They can.  Each node independently verifies each block on the block chain, giving them the power to enforce or not enforce any rule they want.  For example, if you don't like the upcoming BIP66 soft fork, you can modify the code of your node to reject all version 3 blocks.
If you do this by yourself, it'll likely mean that your node will never accept another block again since everyone else is on a chain that includes version 3 blocks.  It'll also mean that people won't want to connect to your node because your node looks like it has fallen behind everyone else.
But if a lot of node operators make that same decision to reject version 3 blocks, it'll become confusing about what's going on.  And if a few miners join you, you'll have a forked block chain, which will create more confusion.
During the confusion, it'll be hard to tell who controls what bitcoins, and double spends may become easy.  This makes Bitcoin unreliable, and nobody wants to use an unreliable financial system---which is why we work very hard to avoid creating such confusion.  That's why the only-miners-vote system is only used for non-controversial forks.
It's my guess that Greg, Adam, and the rest of the Blockstream co-founders plan to use federated sidechains to introduce us all to two-way pegs so that we can see what they do.  Then, as we become comfortable with them and begin want the additional benefit of distributed trust from merged mining, we can work through any controversy surrounding adding an SPV proof op code.

The actual stated development process can be found here. That said, there's become a rule where enough core developers need to "ACK" it like seen in this pull request.
Will there be a procedure for voting? If core devs feel it is safe, tested, useful, well written, match coding style, etc. then it gets merged into bitcoin. This is usually not easy and the bitcoin development archives and the numerous pull requests that haven't made any headway should make this obvious.
Will Sidechains get in? The big thing to note about sidechains is that at least a few of the core developers are actively working on it. This means they're going to try their hardest to get it in. So there's a better chance than most features people usually propose.
Do nodes vote? They can, of course, refuse to upgrade their bitcoin client to the new version that has sidechains, for example. That's entirely their choice. But actual getting into the bitcoind code, they do not vote on. People can and do, fork bitcoin and use that as their node (Luke-Jr and Eligius being the most prominent example)


text:
Bitcoind suspend when Synchronizing blocks
My bitcoind started synchronising blocks. The bitcoind version is 0.93. When the bitcoind found an orphan blocks in the synchronising process, it seems stop and did not work. the message from the log like this:2015-01-08 06:08:19 ProcessBlock: ORPHAN BLOCK 184, prev=0000000000000002d79cd7069cedd05f49716354590bd5518b18a513b3b4d513
I need restart it and it would continue synchronising. This case happens many times. sometimes the bitcoind found orphan blocks in the synchronising process, it would continue work.  I don't where is wrong. Any ideas? Big thanks.
note: my system is CentOS

please, tag the OS you are using: linux, windows, etc


text:
can I use elements of the blockchain to protect the integrity of a leaderboard without central servers?
For a World of Warcraft community feature conception contest I've come up with something that involves a leaderboard. However, World of Warcraft addons are purely individual, so a centralize platform where a leaderboard is usually saved is not possible. Because of this, it would be trivial to fake the sending of leaderboard updates to the community.
I've been reading about the blockchain recently, and it seems like this might be a way to fix this. The blockchain can protect the integrity of Bitcoin transactions, so it seems like it might protect this. However, I'm not 100% sure.
My main concern is to avoid people abusing the leaderboard by giving themselves extra points. However, Bitcoin transactions are usually between 2 parties. Could the blockchain work with only positive transactions that only involve a single party?

Technical explanation: The general concept is that users can get points for completing certain challenges and compete with each other on a constantly updated leaderboard, similar to how some multiplayer games have a list of best players (cfr. Diablo 2). There are no rewards linked with the leaderboard beyond bragging rights.
The problem, however, is that leaderboards are susceptible to manipulation by malicious players unless they have enough verification that people actually achieved what they said they did. Most games fix this by having the leaderboard being hosted on a remote server where the game tells the server what happened and the server checks if the client could have done that.
However, in World of Warcraft, addons are local-only: you cannot configure an addon as a server to receive messages and do central validation. You can send messages from one addon client to other clients via a central addon message channel, but you cannot say "only send messages to this client", because the channel is not secured and any addon can register to listen to a channel. In addition, addons are written in human-readable LUA code and may not be obfuscated.
Because of this, it's easy for a malicious user to manipulate his addon and just spoof the messages. Or even write an addon that will send an autocomplete for every newly generated challenge. This is the problem with distributed leaderboards, they are easily manipulated because there is no verification of integrity.
However, from what I understand, Bitcoin has managed to fix this issue through the blockchain. Transactions are publicly logged, community consensus is required for transactions to be completed and the final state can by verified by anyone. However, the Bitcoin blockchain handles transactions between 2 parties, so there is always 2 sources for each transaction. Because of this, it's unlikely for any single user to be able to manipulate the blockchain.
However, with leaderboards, there are no multiple parties in a transaction. Each transaction is just a player receiving challenge points locally based on something that happens on their machine. So there is no second party that helps against manipulation. This is one of the reasons why I'm concerned the blockchain might not work for this.
Is there an aspect of the blockchain that can help defend against this? And if not, (although somewhat off-topic) is there a different method that could help defend against forgery?

I neither understand what your problem is nor how you think bitcoin can contribute to fix it. Describing it in more technical terms probably will help.

Proof-of-work (i.e. hashcash) sounds like a solution to your problem, not so much Blockchain. What you need is a deterrence against ease of access and proof-of-work deters very well as users would have to prove that they've done some "work" to mutate the scoreboard.

@renlord i'm editing my question to clarify what I'm after.

The core problem described mimics a Byzantine generals problem which is a consensus problem. Bitcoin solves this by proof of work. In your case, proof of work may be overkill and games often use lockstep or even some paxos algorithm to resolve these problems.

@renlord I see. I know this is starting to get off-topic, but could you write an answer based on my minor edits I just made so I can at least give you some rep for helping me?

In general, as mentioned in the comments. Your problem in these games resemble a Byzantine Generals Problem where you may experience non-compliant/malicious actors in the network. To achieve consensus in this context, there are multiple avenues.
Firstly, there's Bitcoin's approach. Bitcoin enables every full node to perform independent verification of the Blockchain. The Blockchain is simply a store of a series of past events leading to the current state. Where "past events" (i.e. transactions) are stored in Blocks and the "chain" refers to the hard requirement that each subsequent block is required to be dependent on the previous block based on a set of requirements. Access to the write permissions to the Blockchain is governed by means of Proof-of-Work, where you must burn CPU cycles to create a new block which implicitly grants you a write permission to append to the Blockchain. A key difference here is that the Proof-of-Work need not be done in a collaborative manner as it is a Poisson Process, meaning my chances are just as good as anyone else's, assuming that everyone has the same hash power. In other election algorithms, you may have to negotiate collaboratively and it is not necessarily fully byzantine-failure tolerant.
So how can some elements of this be applied in your gaming context is left for you to decide. But in principle, yes, elements of Bitcoins are applicable in the context you've described, but again I'd reiterate, it is somewhat "overkill". Maybe in the context of the game, you'd set the leaderboard to have some condition that requires some proof of some action that proves that the player has actually done something to mutate the leaderboard? Otherwise, it'd probably be simpler to do it in Lockstep as in RTS games.

As PoW power is dominated by who invests the most money in hardware and electricity, you're effectively allowing people to spend money to influence the leaderboard. That's almost certainly not what you want. Use a traditional distributed consensus protocol instead.

I'm not sure whether the blockchain protects against fraudulently submitted proofs in this matter. Suppose Alice, Bob, Carol, Dave and Eve are playing. Eve sends a fake "Earned X points" message to Alice, Bob, Carol and Dave, where she actually didn't earn one at all. Wouldn't all 5 calculate the same block based on this message?


text:
Coins sent while wallet closed
I opened my wallet after almost a year and all of a sudden I see 2 transactions saying I sent coins to someone.  The address it says I sent to is 1PAZw2aNgurapsxUVm8ctZDD76v7JiwQkC however I never sent this.  So, if someone is stealing my coins how do they do it while my wallet is closed (and incidentally it is also encrypted)?
I have done several deep scans...my guess is that I had tried out a mining pool and that software was the culplrit...needless to say I deleted it and did multiple deep scans.  But if my keys are compromised is there any way to change them?
Nate thanks for the reply...is there a way to change my private keys are get new ones?

If someone gets access to your private keys, they can create a transaction that spends your coins.  This transaction can be created and sent from any computer in the world.  There's no need for your computer to be involved, and it's irrelevant whether your wallet software is open or not.
It's possible that some person or malicious software had access to your computer at some  point during the last year (or before) and copied the file which contains your private keys.  If it was encrypted, they would need to decrypt it, but if your passphrase is simple enough, that may not be very difficult.  Their software may also have been running earlier and recorded your passphrase at some point when you typed it, or extracted the private keys from the computer's memory while the wallet software was running.  For whatever reason, they may have waited some time before spending them.
If there were other copies of your private keys somewhere else (such as in a backup, or on an old computer or phone that you gave away or had stolen), then that is another place they could have got them.  In that case your computer needn't have ever been involved.
But you should probably assume at this point that your computer is compromised.  The safest course of action is to back up all your data files (but not the operating system or application programs), wipe the system and reinstall it.

If you want to get a new set of addresses, then I would try:

Make sure all untrusted software is uninstalled (mining software or otherwise).
Send all your coins to a different (trusted) wallet for temporary storage.
Close Bitcoin-QT.
Delete your wallet.dat file.
Reopen Bitcoin-QT (it will make a new wallet.dat file for you).
Get a new address from Bitcoin-QT and send all your coins keys back to that address.

It's not good idea to make a new wallet on once infected system - try to do the same with USB-Linux distribution or create a brain wallet with different system (or online creators) and send whole amount to it. Then install new OS and use your bitcoins again safely.


text:
Can't withdraw from blockchain.info: "Private key does not match bitcoin address [my address] = [probablyprivkey1] | [probablyprivkey2]"
I have some BTC (and I really need money now) on my Blockchain.info account, I can't withdraw them (just send them to another BTC address) as it tells me "Private key does not match bitcoin address [my address] = [probablyprivkey1] | [probablyprivkey2]"
The address was created on blockchain.info, not imported.
The incomes are from Apr 03, 2013, and although I was trying to withdraw them for some time, now I've realized that is the last time those bitcoins will be a really large amount of money for me — 0.3651BTC (I'm living in Poland, so it is ~360.59PLN).
I haven't found an answer on forums, so I decided to ask about it myself.

I don't understand. Why is it that you can send your Bitcoins to a different address, but you can't withdraw? Aren't the two the same?

Their support team is good - open a ticket if you haven't already. Perhaps it relates to something they've changed in the back-end. I've used BCi a lot and never seen that error.

Or.. go to the BCi address book and find the address with the funds in it, then go to settings/advanced and export the private key for that address. Then import the key into a light desktop client such as MultiBit and send using that.

@NickODell I meant that by withdrawal I meant sending.

@GeorgeKimionis Tried. No response yet. I know I should wait, but I'm in quite a hurry and I wanted to know whether sombody already got the money back maybe. Thanks for a suggestion.

@George well, the problem appeared back in 2013 or the beginning of 2014, and there are some forum threads on the Internet from this time, among others one was from December 2013. It was probably a rather larger problem. Thanks for a hope.

@George You seem to talk about some old Blockchain UI... there's no 'Advanced'; I have MultiBit, and tried importing wallet.aes.json but it requires .key

I'm not familiar with MultiBit but if you open that .json file in a text editor you'll see the private keys (it needs to be an unencrypted export). Then follow this guide https://multibit.org/v0.5/help_importASingleKey.html to import the relevant key(s).

@George Hmmm... So I have to just modify .key... I was just thinking it's binary without actually checking. Thanks, I'll try it!

https://gist.github.com/ArchieT/51f8d2541d58da7479c8f778160925fc Here I tried https://github.com/prof7bit/wallet-key-tool, same error. First with a wrong password, and then with a correct one.

welp. now it's worth 11kPLN, about my yearly expenses on everything except room rental.

new UI of blockchain info wallet allowed me to generate a signature, but an online checker just said "Invalid data" to it. And trying to send money results in error. Trying to export results in "Unsupported Key Format".

my username on GitHub changed from ArchieT to mkf, so adjust the above gist links


text:
Can miners reject transactions from an address?
It is alleged that the stolen Bitstamp coins are in this address. In theory, could miners agree to reject all transactions from this address? And if so, would all miners need to agree or just more than half?

related: Is it technically feasible to “shun” particular bitcoin wallets?, Is it possible for the miners to collude to blackmail a single wealthy address?

Technically it is possible to blacklist addresses, and it needs just the majority of the mining power to agree.
The problem is arriving at the agreement: Nobody stands to gain from witch-hunts, so a consensus to block an address would have to be carefully built on solid proof, which is hard when there is no central organs with jurisdiction, nor a way to get a statement from the culprit.
Also, blacklisting addresses, just like tracking tainted coins, has a strong implication on what we view Bitcoin to be: One of the sources of Bitcoin's value is that bitcoins are considered fungible – that each individual does not need to care about the history of a bitcoin, because a bitcoin is a bitcoin. If we start treating bitcoins differently in accordance to their transaction history, it will make the bitcoin economy more complicated and less useful.

People will treat them differently, and are. Bitcoin is not fully fungible. No-one wants dirty bitcoin even less than dirty dollars. Alleged stolen bitcoin are anathema to exchanges and have less value than other bitcoins.

Yes, but I don't think the effect will be pronounced: 1) Exchanges are bound to disagree on perceived taint of coins. 2) It is unlikely that there will be broad consensus to blacklist addresses, so coins will be able to move, hence be mixed with other coins. 3) Through mixing taint either a) spreads or b) dilutes. 4) Punishing users for receiving unwittingly tainted coins would be an undeserved collective punishment for which I can't fathom broad support. 5) Taint will dilute over time. 6) Bitcoins will remain fungible for most uses except directly selling stolen coins on regulated exchanges.

TL;DR – Most users will be too lazy to track or in disagreement about taint for it to take a measurable effect. (Unproven opinion.)

I think miners can't refuse any transactions once they're in a block, except those which are invalid right now.
If everyone agreed, there could be a centralized service that would broadcast alert messages to miners, alerting them that the transaction shouldn't be confirmed. But I am not sure this is a good idea. It would definitely go against the decentralization of bitcoin.


text:
On average what percentage of the total block reward comes from transaction fees?
Question
Hello. At the time of this question the reward per block is 6.25 bitcoins. But if I understand correctly that does not include transaction fees. On average, in addition to the 6.25 bitcoins, how many additional bitcoins do miners currently get that are derived from transaction fees and not new bitcoin being created?
More context
I know it varies since the number of transactions varies, as does the cost of the individual fees added to each block, but even some insight on the range of variation in size would be appreciated.

The block reward consists of two parts, the block subsidy and the transaction fees. The block subsidy is currently 6.25 ₿ per block and will halve at block 840'000 in about three months. The transaction fees depend on the current demand for blockspace and can therefore be fairly variable. The average feerates collected in the last 24h were 0.33 ₿ per block. About a month ago, miners briefly collected block rewards with transaction fees of over 7 ₿ per block:

via blockchair.com
If you know the average feerate in a block, you can easily estimate the transaction fees:

via mempool.space
Since blocks can have up to 1'000'000 vB in virtual size, 1 s/vB translates to 10 m₿ of transaction fees. So, ~38 sat/vB here indicates that the block collected about 380 m₿ = 0.38 ₿ in fees.

Thanks. Wasn't aware the numbers can jump so much transaction fees per block can exceed the block subsidy. Moments of congestion makes sense. Thanks.


text:
Can private keys remain offline for their entire life?
I am new to Bitcoin and this question is based on 3 assumptions that may be incorrect.
Assumption 1:
If I understand correctly, the Bitcoin address is generated algorithmically independent of the network, so I should be able to create a public address and private key offline correct?
Assumption 2:
I can receive funds through the network using my public address without ever having to make my private key known to anyone correct?
Assumption 3:
Now when it's time to spend my funds, offline I can create a Bitcoin transaction authorizing the transfer from my address to another location and sign it with my private key entirely offline. I can then transfer that transaction to a separate computer with internet access and then publish that transaction to the Bitcoin network to be mined/verified in the blockchain?
Conclusion:
If all 3 points are true then I should be able to keep my private key from the internet for it's entire life correct?

Yep, only nitpick is on assumption 3, the transaction is not just "pay from my address to other address(es)", but "pay from transaction(s) xyz to other address(es)". What this means in practice is that you'll get the transaction ready on an online computer, sign it on the offline computer, then publish it from the online computer.

Yes, all three assumptions are correct. You can with some effort keep your private keys offline for their entire life.
A description of how to sign transactions offline is found here: How to sign a transaction using only an offline computer?

Adding a bit of detail on Assumption 3 along the lines of Tim S.'s comment: A computer that is offline would obviously not get updates from the network, so it would also not know which transaction outputs are available for spending. This could be mitigated by importing the blockchain data to the offline computer (for example with the bootstrap.dat).
However, it would be much easier to create a two wallet setup, where one online wallet knows only the addresses, and the offline wallet has the private keys. You then create the transaction on the online wallet, which has the transactional information and sign it on the offline wallet. That's what is described in the above linked question as well.
Update in 2017: The creation of a transaction on the online device and signing on the offline device is essentially what hardware wallets such as Trezor do.

There are also solutions like Ledger Wallet in which private keys are kept on a hardware chip and never leave the hardware; it performs signing on-chip.


text:
What is the best way to download a "recent" UTXO set snapshot?
I am using bitcoinj to listen for payments. I need a way to maintain the entire UTXO set in a local database.

Can bitcoinj be used to download every block (I think it downloads only recent blocks and headers for older blocks)?
I don't need to use bitcoinj or even Java for creating or maintaining this database if a better solution exists. Is there anything like this?

Ideally what I want is a wallet that downloads only the UTXO set without downloading the entire blockchain (due to bandwidth limitations) and maintains it from then onward. This needs to be the entire set, not just for the addresses I own.
Additionally I need an API that allows for following queries:

Given a UTXO, check if it is present in the set.
Given an address, get me all the UTXOs for it.

I think that it is currently not possible to validate a given UTXO set without downloading the entire blockchain, so some solution in this regard would also be appreciated.

Yes, by default bitcoinj downloads full blocks, from the Genesis. Only if you connect a wallet it uses optimizations like checkpoints and filters.
Just downloading the UTXO set (in a trust free way) isn't possible, because the Bitcoin protocol isn't designed for that. You can get UTXOs from block explorers and Electrum servers, but it will be without crypto proof.

How much time does it take currently with a i7 k7700 CPU and decent bandwidth?

If you put aside the issues of trust, validation of the data, keeping it up to date and so on. If you just want a one-off snapshot for some academic purpose, you could ask a friend who uses Bitcoin Core to give you a copy of their chainstate folder - but you'd have to extract data from the LevelDB data structure.
Other wallets such as Electrum might also have some persistent data storage that mainly consists of UTXO data.

As Andreas wrote, Just downloading the UTXO set (in a trust free way) isn't possible.  I would like it to be possible, and so I've thought about how to do it...
Each block identifies transactions from previous blocks that were in the UTXO set and are no longer in it and also fully specifies all the transactions that have been added to the UTXO set by the current block.  This means that IF you already have a full and verified UTXO set and someone wants to know if a UTXO set as of N blocks ago is valid, you can provide an answer.  Whether or not they trust you is a different issue.  If your set is valid, then you can undo the additions and subtractions of transactions as indicated by the N blocks, calculate a hash, and then send that hash to the other person.  If they get the same hash on their set, it's safe for them to assume that you used the same UTXO Set to get the hash that they used (ie., they match).
There may be software that can do this already.  I'd like that.  But there's no proof that either of you has the correct UTXO set.  I'd like to see that fixed for everyone, but it requires a little bootstrapping.  I have a plan for it:

Create software that will render a SHA256 Hash into something that is memorable and unique.  This is a bit more complicated than it sounds.  Whatever the rendering produces (music, art, animation, a combination, ...) must have (at least) 256 distinct binary properties that will be easily remembered.  Anyone could use this software to see if the hash of the UTXO Set at a recent block height produces a memorable rendering.
Make it a thing that sometimes a memorable rendering is produced.  That can serve as a global checkpoint for the blockchain.  If enough people (judgement call) know about the memorable rendering, then new installations of full nodes could get (from anyone who has it) the UTXO set at that height, compute the hash, render it, and compare the result to the one everyone remembers. I imagine that would be readily available from around the Internet, but it's safer to talk to someone in real life to avoid the danger of Sybil attacks.
Update full node software to provide the UTXO Set as of a block the requesting node specifies (to mak.e the previous step possible for fresh installations)
Update the full node software to allow the user to specify a block height known to have produced a UTXO Set with a memorable hash, ask for the UTXO set as of that block from peers, create the rendering from the hash of the answer, and display it to the user.
Allow the user to indicate that the rendering matches and if the user so indicates, then skip downloading the blockchain (except for any parts that aren't included in the canonical UTXO set - if any) to save bandwidth and disk space, and just start with the UTXO set that came in as the answer.

For the same reason that mined bitcoin that is less than 100 blocks deep cannot be spent (ie., a reorg of 100 blocks is unlikely - possibly not even supported?), this strategy allows us to proceed with a minimum of 100 blocks of history instead of the entire blockchain.
This places the responsibility of ensuring the integrity of the starting point UTXO set on the human being using the full node software.  It would have to be experimental for a while, until it proves that blocks zero through X are no longer necessary (because the UTXO set at block height X was memorable enough to be recognizable by mere humans using this new feature), thus reducing the blockchain requirement for an always-increasing amount of storage to an amount of storage that will presumably level off.
Part of the appeal to me of this idea is that the random SHA256 hash of a UTXO set can be rendered into something memorable, and the chances of that happening are low but not zero, just like mining itself.  Different rendering algorithms could compete.  I just thought of one...
The 256 bits of the hash plus the 32 bits of the block height could be used to indicate which cells in a 16x18 game of life are on.  Let the game of life run.  Watch it.  Remember how it stabilizes.  Some games will eventually clear the board.  That's memorable.  How it clears the board would also be memorable.
I started looking for a canonical UTXO set (ie., one whose structure is standardized, so that the same set of UTXOs would have the same representation as data).  Converting the SHA256 hash plus 32-bits of the block height into a Game of Life to see how it plays out can be pretty easily done with Golly or any other software that implements John Conway's "Game of Life."
So I have the same quest as OP, but Andreas has a point. You have to run a full node and create it yourself.  However, once you've done that, you could possibly make it available to others (me!).  He also points out that there's no way to prove it's correct except by running a full node and repeating all the work of updating it for every transaction.  Proving the result is correct without all that work is the goal of the idea I present in this answer.

Take at a look at this program. It will dump the entire current UTXO database to a CSV file that you can crossreference.  https://github.com/in3rsha/bitcoin-utxo-dump

@ Dave Scotese: I agree with your idea of having a generally agreed UTXO set (Not sure why u suggested running the resulting checkpoints hash into a game of life though). Anyway, I think the problem is not that it can't be done rather it comes down to consensus (very similar to the problem solved by blockchain itself). Suppose you did honestly used your algorithm and provided the world with a memorable hash that defines the UTXO set at some block height checkpoint which you place in some repo for download.
Now if Mr. A downloads it and checks using his own database and agrees with you.
You don't expect Mr. B to agree with you simply because Mr. A agrees.
You see, whoever needs to use this UTXO set needs to verify it for himself.
It still comes back to that Byzantine Generals problem which blockchain solved.
I honestly think for it to work, it has to run on its own separate blockchain so that UTXO set miners can mine it for some kind of financial gain.
And yes I completely agree with you that having such a UTXO set will solve the problem of having to download the entire blockchain before you can validate transactions.

authors: Андрей Белянин


text:
How does ENS prevent frontrunning attacks?
Namecoin uses commitments to hide the domain. ENS uses simple hash, which is not hiding.

The only way to get the domain name on the ENS is to win at the target name auction.
So there is no need to hide the name.
ENS uses the name hash for converting arbitrary length string into bytes32.


text:
Litecoin transaction fee structure?
I am exploring litecoin and I am confused about transaction fee.

What is the minimum recommended fee per kb?
Is the fee structure enforced by the miners or only by the clients?
If I somehow craft and send a transaction with zero fee, what will happen? (will it not be accepted by the network or will it be accepted and eventually confirm?

The client calculates the priority of the transaction, which is defined as coin days destroyed (i.e. number of coins times coin age) divided by transaction size. If it is above a certain level, the transaction may be sent for free. Otherwise, a 0.001 LTC per KB fee is enforced by the client. In general, this also happens to be the minimum recommended fee per KB.
Some miners choose to enforce these fees and won't include a transaction in a block they mine if it does not have enough fees. Other miners don't care and include any transaction in their block.
As you can see in an example block, most transactions of any decent size don't require a fee. Generally, if you send 1 LTC which you did not receive very recently and is not composed of a lot of small input transactions, you do not need to pay a fee. If you try to send a transaction with zero fee when it needs one, miners might skip over your transaction, and you'll need to wait until some miner is willing to include it in their block. If you grow tired of waiting, you can broadcast another transaction using the same coins (but this time with a fee). It will be included in a block, and the previous transaction will be regarded as a double spend and thus be rejected and disappear from the network.

Is this still up to date?


text:
Does bitcoind store timestamp at which private key was imported?
If we import a private key using bitcoin-cli importprivkey <key>, is there anyway to find out when the import was done?
I am hoping a timestamp is included.

No, bitcoind does not. bitcoind does store a timestamp for keys however this timestamp is used to determine when to start rescanning. importprivkey does not allow you to specify this timestamp, so in order to be sure to not miss any transactions, the timestamp is set to 1 indicating that the entire blockchain will be rescanned.
The importmulti command allows you to specify the timestamp which by default is the time at which you imported the key. However, as mentioned earlier, since the timestamp is needed for rescanning, if it is not set to be early enough, the automatic rescan done at import time will not catch all of the transactions the key is involved in.
If you want to see the timestamps, you can use getaddressinfo to get the information for the address corresponding to the private key you imported. getaddressinfo will give you the timestamp in unix time.

We can assume address was added at the time of private key import.. Will it tell me how to get that timestamp? Even if there is no command, I can write my own parser for wallet.dat outside of bitcoind, as long as the data is stored.

As I said, the timestamp that it stores is not the timestamp of the import. The timestamp of the import is not stored, only the key "birthdate" is stored, and that depends on the import call used and what you specified in the import. It might be the timestamp of the import if you used importmulti.

It was a fresh wallet and I had used importprivkey. What about the timestamp of the corresponding address that I get from getaddressinfo? Will that be approximately equal to the timestamp (Unix time) of the private key import?

No, it will not. There is only one timestamp that is stored for each key (and thus address). That timestamp is the same one I described in my answer and comments.


text:
Is there any wallet that that sets fee based on input amounts (not tx size)?
Generally the fee is based on tx size in the wallets I am aware of. Is there any wallet that allows setting fee based on a fixed fraction of total input value (say, 1/1000th of the inputs value)?
Please include wallets used by developers also (Bitcoinj, bitcoinjs, etc).

Most wallets that allow you to specify a custom fee would support this, technically. The fee is simply totalInput , totalOutput, so you can easily adjust it a fair bit.
However, miner's select txs based on sat/byte. In the system you described, some txs will end up severely underprices, while others will be massively overpriced.
Wallets currently just take the size of your tx, multiply it by a fee rate, and deduct that from the change outputs. This ensures that you are paying a fair fee for the tx size.

Is there any wallet that allows setting fee based on a fixed fraction of total input value (say, 1/1000th of the inputs value)?

I don't know of any that do (besides by allowing you to manually set the fee), because it doesn't make sense.
Prioritization of your transaction in the network is based on their weight (~size), not the value being transferred. The value being transferred isn't even always possible to infer from the transaction (for privacy reasons). A wallet that would set a fee based on amount would be letting the user seriously shoot themselves in the foot.


text:
Are there any services that can reasonably guess what IP address a transaction originated from?
Are there any services/agencies running a large number of nodes so that they can guess the IP address of a transaction's origin?
I guess they would need to run tens of thousands of nodes. However it'll be useful in tracing stolen coins. Surely there must be some.
I have seen talks and papers claiming to do so but not found any such service.

I see this as a non-issue. By using Tor the IP address of the originator is sufficiently obfuscated.
See my two other posts for some discussion and related resources:
How can I setup Bitcoin to be anonymous with Tor?
What are main advantages of Bitcoin over Monero?

When you (a node) receive a transaction, you are not sure if you are receiving it from a node forwarding a transaction to other nodes on the network or if you are the first node in the web to receive the transaction to your mempool.
There are services which are engaged in meta data analysis that are likely revealing a surprising amount of information from studying mempool generation and the block chain. Exactly what they are discovering is still very obscure. At least one study by Chainanalysis has determined about 4 million coins are off the market indefinitely. (Source: http://fortune.com/2017/11/25/lost-bitcoins/ ) This service offers Blockchain analysis to exchanges, government agencies and financial institutions, publishes some findings from time to time for publicity, but is otherwise inaccessible to the individual as the amount of resources they invest into analysis requires significant compensation, though I suspect if they ever discover a use for their data that could be marketed to the masses they would likely develop a service for it.
The fact that there is no one out there marketing a way to track bad actors, either in service to the government or public, indicates that the technology does not yet exist, or the laws don't provide a means by which such analysis could be put to productive use. Some people still publish addresses and stories of bad actors to forums for community awareness, essentially providing an ad hock address blacklist. This is so easily circumnavigated by services such bitcoin gambling sites and address washing services that I don't believe anyone is curating a 'master blacklist' of the reports. Currently blockchain analysis is focused on detecting KYC/AML violators, money launderers, and in general providing some kind of evidence that a company is performing their due diligence in preventing bad actors from using their services, which here means a token effort and expense that reduces their liability in the case of a damaging law suit.


text:
Large number of dust inputs sum to nice round numbers. How and what wallet used?
Consider the bitcointalk donation address 17RTTUAiiPqUTKtEggJPec8RxLMi2n9EZ9.
This has many incoming transactions with large number of dust inputs (hundreds) that sum to nice rounded numbers as in the following examples:
ba930f48aae32b8915a6e413ed297060291fec7fc916cf57b0751e0f77c41df2 (inputs sum to 0.05)
c0f25cc0f008ac935b3a38fecdea42e062ba3e3e7395161a2ad1b38754ee31c0 (inputs sum to 0.05)
1f4e0b21e6edd0dc1148116200426f390880e3e33ae0723de8415836874f7a3e (inputs sum to 0.5)
These can't be coincidences. What wallet is being used here? Does it solve the "subset sum" problem for each transaction to ensure that the sums are round numbers? I know the problem won't be hard for such small sizes but still needs solving.

Another explanation might be that one of the input transactions is intentionally created to cause the sum to be a round number.

I believe that's the question. Selecting a large number of inputs to perfectly sum  a round figure is non-trivial.

Why would someone do that? And out of curiosity who is the sender?


text:
Is it possible to run a simple attack on a previous version of the client in testnet mode?
I need to simulate an attack locally or in testnet (if possible) for academic purpose, on an old and bugged version of the bitcoin core client. I am in posses of test BTC in the last version. I want to know an easy way to exploit one this listed vulnerabilities(https://en.bitcoin.it/wiki/Common_Vulnerabilities_and_Exposures), if this has been done before or if im speaking nonsense.

Hi Similarion, it is not clear what you need, could you please add a few more details so people have a better idea what to look out for in an answer?

Added infos hope it helps


text:
BitFinex API call returns 400 bad request
Trying to get authentication with BitFinex to work. Docs here: https://www.bitfinex.com/pages/api

Say the client wants to make a request to POST
https://api.bitfinex.com/v1/order/new
With a payload of { "request":"/v1/order/new", "nonce": "1234", "option1": ... }
The nonce provided must be strictly increasing.
To authenticate a request, use the following:
payload = parameters-dictionary -> JSON encode -> base64
signature = HMAC-SHA384(payload, api-secret) as hexadecimal
send (api-key, payload, signature)

These are encoded as HTTP headers named:
  X-BFX-APIKEY X-BFX-PAYLOAD X-BFX-SIGNATURE

Here is the c# code, which follows the above example, but generates a HTTP 400 Bad Request. Any ideas how to fix?
        long nonce = DateTime.Now.ToUnixTimestampMS(); //returns a strictly increasing timestamp based number e.g. 1402207693893

        string path = "https://api.bitfinex.com/v1/balances";
        string paramDict = "{\"request\": \"/v1/balances\",\"nonce\": \"" + nonce + "\"}"; //ie. {"request": "/v1/balances","nonce": "1402207693893"}
        string payload = Convert.ToBase64String(Encoding.UTF8.GetBytes(paramDict));

        //API Sign
        HMACSHA384 hmac = new HMACSHA384(Encoding.UTF8.GetBytes(APISECRET));           //My API SECRET
        byte[] hash = hmac.ComputeHash(Encoding.UTF8.GetBytes(payload));
        string hexHash = BitConverter.ToString(hash).Replace("-", "");

        NameValueCollection headers = new NameValueCollection();
        headers.Add("X-BFX-APIKEY", APIKEY);                                            //My API KEY
        headers.Add("X-BFX-PAYLOAD", payload);
        headers.Add("X-BFX-SIGNATURE", hexHash);

        //POST data
        try
        {
            //create post request
            HttpWebRequest request = (HttpWebRequest) HttpWebRequest.Create(path);
            request.KeepAlive = true;
            request.Method = "POST";

            //add headers
            request.Headers.Add(headers);

            //write out payload
            byte[] byteArray = System.Text.Encoding.UTF8.GetBytes(paramDict);
            request.ContentLength = byteArray.Length;
            using (var writer = request.GetRequestStream()){writer.Write(byteArray, 0, byteArray.Length);}

            //read reply
            using (var response = request.GetResponse() as System.Net.HttpWebResponse)
            {
                using (var reader = new System.IO.StreamReader(response.GetResponseStream()))
                {
                    //get reply (JSON)
                    string responseContent = reader.ReadToEnd();
                }
            }
        }
        catch (Exception e)
        {
            //always throws an exception here
            Debug.WriteLine(e.Message);
        }

Finally discovered the fix: hexhash has to be lower case:
string hexHash = BitConverter.ToString(hash).Replace("-", "").ToLower();

Such a weird API... Thanks for both your question and (self) response

I am having the same issue in python, but the hexdigest is already in lower case...

did you ever get the order/new to work ? I am still getting bad request despite running properly on other calls such as balances etc.


text:
How do I compose a transaction with OP_RETURN in Python?
Currently trying to figure out how to use Python to construct a transaction with OP_RETURN. I tried to encode the message myself, and had no luck. I found a function, OPReturn(), on the internet, but when I attempt to use it, I get the error (from the Blockchain.info broadcast API): Exception: None Standard Script Output OP_RETURN 594f4c4f53574147
Code:
# coding: utf-8

from bitcoin import *
import binascii
from test import *

priv = sha256('brain wallet')

pub = privtopub(priv)

addr = pubtoaddr(pub)

inputs = unspent(addr)

message = "YOLOSWAG"
FullLen = format(len(message)+2,'x').rjust(2,'0')
MessageLen = format(len(message),'x').rjust(2,'0')
ID = binascii.hexlify(str(message))
snd = "6a"+MessageLen+ID

outputs = [{'value': 50000, 'address': addr}, {'value': 0, 'script': snd}]

fee = 10000

tx = mksend(inputs, outputs, addr, fee)

dt = deserialize(tx)
ins = dt['ins']

#print addr
#print ins

for ind, elm in enumerate(ins):
    print elm
    tx = sign(tx, ind, priv)

#print tx

print(pushtx(tx))

Note that this code will only work on messages of 75 bytes or less.

@NickODell isn't it 40 bytes? (EDIT: you mean 75 bytes total I just realised?)

What version of Python? I'm pretty sure you're trying to do this on Python 3.x where the struct "q/Q" field is no longer available. Pybitcointools will require Python 2.7 (in fact virtually all Bitcoin Python stuff is version 2.x due to the way it handles strings as bytes without binascii and such.

I have forked the pybitcointools library to return a properly formatted OP_RETURN hex string, or insert the OP_RETURN into a raw hex transaction.
My fork can be found here. The code is as follows:
from bitcoin.pyspecials import safe_hexlify, from_string_to_bytes, from_int_to_byte, from_string_to_bytes

def mk_opreturn(msg, rawtx=None, json=0):
    def op_push(data):
        import struct
        if len(data) < 0x4c:
            return from_int_to_byte(len(data)) + from_string_to_bytes(data)
        elif len(data) < 0xff:
            return from_int_to_byte(76) + struct.pack('<B', len(data)) + from_string_to_bytes(data)
        elif len(data) < 0xffff:
            return from_int_to_byte(77) + struct.pack('<H', len(data)) + from_string_to_bytes(data)
        elif len(data) < 0xffffffff:
            return from_int_to_byte(78) + struct.pack('<I', len(data)) + from_string_to_bytes(data)
        else: raise Exception("Input data error. Rawtx must be hex chars" \
                            + "0xffffffff > len(data) > 0")

    orhex = safe_hexlify(b'\x6a' + op_push(msg))
    orjson = {'script' : orhex, 'value' : 0}
    if rawtx is not None:
        try:
            txo = deserialize(rawtx)
            if not 'outs' in txo.keys(): raise Exception("OP_Return cannot be the sole output!")
            txo['outs'].append(orjson)
            newrawtx = serialize(txo)
            return newrawtx
        except:
            raise Exception("Raw Tx Error!")
    return orhex if not json else orjson

Note that the module name has been changed to btc (from bitcoin) in my fork.
To run this, you'll use os.chdir("c:/python/pybitcointools") (or whatever directory it's been downloaded to. Then from bitcoin import *. Now, let's use msg = 'The enemy of my enemy is my friend' and rawtx = "01000000016e3cd2b24fcf49259db29888ec5fe6521070041cb8c7bb2017537046f9e00f2b0000000000ffffffff0168d61100000000001976a91469bbbb16301e40b9fb67130e1aa53a2281d60af088ac00000000".
mk_opreturn(msg, rawtx) returns:
01000000016e3cd2b24fcf49259db29888ec5fe6521070041cb8c7bb2017537046f9e00f2b0000000000ffffffff0268d61100000000001976a91469bbbb16301e40b9fb67130e1aa53a2281d60af088ac0000000000000000246a2254686520656e656d79206f66206d7920656e656d79206973206d7920667269656e6400000000

This is the raw Tx with the OP_RETURN properly inserted. Run the function without the rawtx parameter and it returns the string 6a2254686520656e656d79206f66206d7920656e656d79206973206d7920667269656e64

Did you mean to answer this twice? PS. I sent you a PR on github. :)

Yes and no :). I felt it needed a separate answer because I'm trying to answer the "how to easily use OP_RETURN with Python using code" question. This answer is more self-contained I feel. Good catch on the PR BTW

The script is simply (so to speak) taking a raw transaction and splicing in a hexadecimal number which represents an additional Tx output. So it's looking for ffffffff (sequence) and appending 6a hex encoding of your msg (up to 20 bytes)
The message needs converting to hex (which the code does).
The code works fine for me, although I've yet to try broadcasting. Based on your quoted error

I get the error (from the Blockchain.info broadcast API): Exception: None Standard Script Output OP_RETURN 594f4c4f53574147

I'd have to say BCI is reacting to theMessageLen part of the code. There should be an OP_PUSHDATA1 (0x4c) between OP_RETURN (0x6a) and the 1 byte msg length, 0x08.
Try: 6a4c08594f4c4f53574147
Alternatively, try another service to push the raw Tx, eg:

https://insight.bitpay.com/tx/send
https://brainwallet.github.io/#tx

Doesn't my line ID = binascii.hexlify(str(message)) already take care of that?

@rsmoz let me have a look for you. Can you confirm the URL for the script and the Python version you're on?

2.7. I'm about to update the code with some changes I've made.

I added the push code, didn't work. Tried Bitpay's tx send, told me my transaction was dust, which I remembered meant the transaction was too small. Made it larger, then used the Bitpay send and it worked! https://blockchain.info/tx/e44d541fd4cc0c2390a91e7e4d35436524badddd314e905d9113f383ba2a0c62

@rsmoz good stuff! I'm interested to know if it was the 550 Satoshi dust limit or the push op code that did it. I'll edit my answer to remove the unhelpful binascii comments I made

Hmm, still having some issues. Now that I've got a lot of inputs to deal with, the transaction isn't being assembled properly.

I wanted to add a clarification: when assembling a 40 byte OP_RETURN Txn the code is: 2a6a28 40 byte data, where 2a says push next 42 bytes, 6a is OP_RETURN and 28 says push the 40 bytes. This is the required format for Bitcoincore.

You might find our python-OP_RETURN library helpful, either to use out-of-the-box, or to look inside to see how we are building OP_RETURN transactions.
https://github.com/coinspark/python-OP_RETURN
It also has a neat feature to store arbitrary-sized data in the blockchain using multiple chained transactions with OP_RETURNs, and to retrieve that data using a single 12-digit reference number.

What does the reference number correspond to?

It's in the format: [estimated height]-[partial txid]
[estimated height] is the block where the first transaction may appear. When storing we set [estimated height] to 1+(current height). When retrieving we try that block then some later and a few from before in case of reorgs.
[partial txid] contains 2 adjacent bytes from the txid:
position=2*([partial txid] div 65536)
([partial txid] mod 256) is the byte at position
([partial txid] mod 65536) div 256) is the byte at (position+1)
You can't use a txid for the reference since most nodes don't index by txid.

@Gideon Greenspan -- I tried to use your code but got this error: 'OP_RETURN.py", line 456, in OP_RETURN_bitcoin_cmd if not len(port): TypeError: object of type 'int' has no len()'

Set OP_RETURN_BITCOIN_PORT using an integer in a string rather than an integer.


text:
How do I copy the blockchain to a separate computer running Bitcoin Core?
I'm running bitcoin-qt 0.8.6 on a different computer that has the whole blockchain downloaded. I tried putting all the blk.dat files and the index from that computer  the blocks folder in bitcoin-qt v.10 on a separate computer but it wouldn't load. I tried reindexing it but it still wouldn't load. I'm not sure if the chainstate is related to the process because I tried placing the original in the new bitcoin qt application support folder but it would cause it to crash before opening or everything would be erased upon opening.
I'm running OSX Mavericks on the original computer and Yosemite on the new one.

@valkyrie ! Since bitcoin-qt is such a complex piece of software that the easiest answer to your question is: can't you let it run the usual way of downloading the blockchain per se ? Another point is: do you really have to run different versions of the software ?

Related: https://bitcoin.stackexchange.com/q/10787/516

The answer to the linked thread above fully applies since the blockchain doesn't care what architecture or operating system that is running it in terms of reading the core data directory. See this useful post: https://bitzuma.com/posts/moving-the-bitcoin-core-data-directory/


text:
Sample proof of work in Ruby, what am I doing wrong?
Given the following block, how do I verify the proof of work? I got this data from bitcoin-cli getblock and parsed in to a Ruby hash called data:
{"hash"=>"00000000000000001354e21fea9c1ec9ac337c8a6c0bda736ec1096663383429",
 "confirmations"=>2, "size"=>529935, "height"=>347928, "version"=>2,
 "merkleroot"=>"14b2c974a4f9ab92a68f03a491965063d1f8a2e8b2268fdb4cbce92262bb09c0",
"tx"=>"..removed for space convenience...", "time"=>1426554998,
"nonce"=>3284264479, "bits"=>"18172ec0", "difficulty"=>47427554950.6483,
"chainwork"=>"000000000000000000000000000000000000000000056f68a6406c831f16c1bd",
"previousblockhash"=>"000000000000000007c464352935f12c3695f7de4eee8390f444233682abf8aa",
"nextblockhash"=>"000000000000000016e3530b3a7cdaaa495e6eb70aa8f3f15d02fd078235e931"}

I have tried the following:
version = "02000000"
prev = data["previousblockhash"]
merkle = data["merkleroot"]
time = data["time"].to_s(16) #convert to hex
bits = data["bits"]
nonce = data["nonce"]

hashable = version + prev + merkle + time + bits + nonce

answer = Digest::SHA256.hexdigest hashable

The problem is my answer is this:
b7dc9db86cdf47bf1b13274f88b14211a1ce58c52ac1e41c987b4ad65cf50f4f
but the given answer (in data["hash"]) is
00000000000000001354e21fea9c1ec9ac337c8a6c0bda736ec1096663383429
What am I doing wrong?
Thanks for any help,
Kevin

A few things that need to be fixed here:

Endianness - The data given to you in JSON is almost completely in big endian but bitcoin uses little endian to serialize block headers before hashing.
You need to double hash the result with SHA256, meaning apply it once on the header and then apply the algorithm again on the result of the first hash.
Conversion to binary before hashing. You can't just hash the ascii characters, as those are just an encoding for the underlying binary, which is what is meant to be hashed.

I'm not very familiar with Ruby, but I made a short python script that you can use as a reference.
import hashlib
import binascii

def rev(string):
    revsed = ""
    for i in range(int(len(string)/2)):
        revsed = string[2*i : 2*i+2] + revsed
    return revsed

def DoubleHash(string):
    bin = binascii.unhexlify(string)
    hash = hashlib.sha256(hashlib.sha256(bin).digest()).digest()
    raw = str(binascii.hexlify(hash))[2:-1]
    return rev(raw)

# Straight from JSON

version = "02000000"
prev    = "000000000000000007c464352935f12c3695f7de4eee8390f444233682abf8aa"
merkle  = "14b2c974a4f9ab92a68f03a491965063d1f8a2e8b2268fdb4cbce92262bb09c0"
time    = "55078076" # hex(1426554998)
bits    = "18172ec0"
nonce   = "c3c1e61f" # hex(3284264479)

header = version + rev(prev) + rev(merkle) + rev(time) + rev(bits) + rev(nonce)

print(header)
print(DoubleHash(header))

The results:
02000000aaf8ab82362344f49083ee4edef795362cf135293564c4070000000000000000c009bb6222e9bc4cdb8f26b2e8a2f8d163509691a4038fa692abf9a474c9b21476800755c02e17181fe6c1c3
00000000000000001354e21fea9c1ec9ac337c8a6c0bda736ec1096663383429

Thanks for this, it looks like either your prev didn't reverse properly or I'm misunderstanding what you're doing:  last 5 of prev: bf8aa  your rev of prev in header: aaf8a

You have to list the bytes in reverse order. Each byte is two hex characters. The previous block is 0000...f8aa. You just have to list the bytes in the reverse order as aaf8...0000. i.e. 1234 byte-reversed is 3412.

ok, i see, so it's both byte reversed and Endian reversed?

Endian reversed just means you need to list the bytes in the opposite order. Remember to mark answers as correct if they answered your question.

sorry for not being clear, i understand what endian reversed meant, i was attempting get confirmation that two distinct processing steps are required 1) reverse endian (swap the hex bytes) and 2) reverse the string

Reversing the endianness is the same thing as reversing the bytes in the string.