text:
What exactly is hashed and signed in a segwit transaction
I have an unsigned raw segwit transaction on my regtest network:
02000000011333183ddf384da83ed49296136c70d206ad2b19331bf25d390e69b222165e370000000000ffffffff0200e1f5050000000017a914a860f76561c85551594c18eecceffaee8c4822d787f0c1a4350000000017a914d8b6fcc85a383261df05423ddf068a8987bf02878700000000
Pretty format:
Version: 02000000
Flag:
Input Count: 01
Input 1 Previous Output Hash: 1333183ddf384da83ed49296136c70d206ad2b19331bf25d390e69b222165e37
Input 1 Previous Output Index: 00000000
Input 1 script length: 00 (empty for now)
Input 1 scriptSig:
Input 1 sequence: feffffff
Output Count: 02
Output 1 Value: 00e1f50500000000 (100M satoshis / 1 BTC)
Output 1 public key script length: 0x17 (23 bytes)
Output 1 public key script: a914a860f76561c85551594c18eecceffaee8c4822d787
Output 2 Value: F0C1A43500000000 (899990000 sats / 8.9999 BTC)
Output 2 public key script length: 0x17 (23 bytes)
Output 2 public key script: a914d8b6fcc85a383261df05423ddf068a8987bf028787
Witness Count:
Witness 1 length:
Witness 1:
Witness 2 length:
Witness 2:
Locktime: 8c000000 (block 140)

I can sign it using the above data (concatenated from top to bottom):
$ bitcoin-cli -regtest signrawtransaction 02000000011333183ddf384da83ed49296136c70d206ad2b19331bf25d390e69b222165e370000000000feffffff0200e1f5050000000017a914a860f76561c85551594c18eecceffaee8c4822d787F0C1A4350000000017a914d8b6fcc85a383261df05423ddf068a8987bf0287878c000000

{
  "hex": "020000000001011333183ddf384da83ed49296136c70d206ad2b19331bf25d390e69b222165e370000000017160014b93f973eb2bf0b614bddc0f47286788c98c535b4feffffff0200e1f5050000000017a914a860f76561c85551594c18eecceffaee8c4822d787f0c1a4350000000017a914d8b6fcc85a383261df05423ddf068a8987bf028787024730440220434caf5bb442cb6a251e8bce0ec493f9a1a9c4423bcfc029e542b0e8a89d1b3f022011090d4e98f79c62b188245a4aa4eb77e912bfd57e0a9b9a1c5e65f2b39f3ab401210223bec70d670d29a30d9bcee197910e37cf2a10f0dc3c5ac44d865aec0d7052fb8c000000",
  "complete": true
}

But, when I try to do it manually, I get a different result for the signature in witness. I think I may be signing the wrong data, so what would the serialized hex string look like that gets hashed (HASH256) and then signed to produce the signature in the Witness data (Witness 1 below)?
Final Tx (signed using signrawtransaction)
Version: 02000000
Flag: 0001 (indicates segwit)
Input Count: 01
Input 1 Previous Output Hash: 1333183ddf384da83ed49296136c70d206ad2b19331bf25d390e69b222165e37
Input 1 Previous Output Index: 00000000
Input 1 script length: 0x17 (23 bytes)
Input 1 scriptSig: 160014b93f973eb2bf0b614bddc0f47286788c98c535b4
Input 1 sequence: feffffff
Output Count: 02
Output 1 Value: 00e1f50500000000 (100M satoshis / 1 BTC)
Output 1 public key script length: 0x17 (23 bytes)
Output 1 public key script: a914a860f76561c85551594c18eecceffaee8c4822d787
Output 2 Value: F0C1A43500000000 (899990000 sats / 8.9999 BTC)
Output 2 public key script length: 0x17 (23 bytes)
Output 2 public key script: a914d8b6fcc85a383261df05423ddf068a8987bf028787
Witness Count: 02
Witness 1 length: 0x47 (71 bytes)
Witness 1: 30440220434caf5bb442cb6a251e8bce0ec493f9a1a9c4423bcfc029e542b0e8a89d1b3f022011090d4e98f79c62b188245a4aa4eb77e912bfd57e0a9b9a1c5e65f2b39f3ab401
Witness 2 length: 0x21 (33 bytes)
Witness 2: 0223bec70d670d29a30d9bcee197910e37cf2a10f0dc3c5ac44d865aec0d7052fb
Locktime: 8c000000 (block 140)

Edit: Code for signing: GIST - ec_sign_hex.sh. This takes the final hashed hex string and a 32-byte hex private key as parameters. The private key I'm using is 72b90220501a0c27a499535c010de9e2fa190c0d287a940a3886bcb2cbcccdb8, and it's executed as:
$ ec_sign_hex 02000000011333183ddf384da83ed49296136c70d206ad2b19331bf25d390e69b222165e370000000000feffffff0200e1f5050000000017a914a860f76561c85551594c18eecceffaee8c4822d787F0C1A4350000000017a914d8b6fcc85a383261df05423ddf068a8987bf0287878c000000 72b90220501a0c27a499535c010de9e2fa190c0d287a940a3886bcb2cbcccdb8

#!/bin/bash
## Command Line parsing
#######################

if [[ $# -lt 2 ]]; then
    echo "Usage: $ ec_sign_hex <input-hex> <priv-key-hex>"
    exit 1
fi

inputHex=$1
privKeyHex=$2

## Create .pem and .pub files
#############################
pubKeyHex="$(openssl ec -inform DER -text -noout -in <(cat <(echo -n "302e0201010420") <(echo -n "${privKeyHex}") <(echo -n "a00706052b8104000a") | xxd -r -p) 2>/dev/null | tail -6 | head -5 | sed 's/[ :]//g' | tr -d '\n')"
asnFormatKey="30740201010420${privKeyHex}a00706052b8104000aa144034200${pubKeyHex}"
echo "-----BEGIN EC PRIVATE KEY-----" > tmp.pem
echo $asnFormatKey | xxd -r -p | base64 | fold -w 64 >> tmp.pem
echo "-----END EC PRIVATE KEY-----" >> tmp.pem

openssl ec -in tmp.pem -pubout -out tmpPub.pem &>/dev/null

## Sign message
#  sign:
    openssl pkeyutl -sign -inkey tmp.pem -in <(printf $inputHex | xxd -r -p) -out tmp.sig
    echo "Signature"
    echo "####################"
    echo ""
    openssl pkeyutl -sign -inkey tmp.pem -in <(printf $inputHex | xxd -r -p) | xxd -p #-hexdump #| xxd -p
    echo ""
    echo "####################"
#  verify:
    openssl pkeyutl -verify -pubin -inkey tmpPub.pem -sigfile tmp.sig -in <(printf $inputHex | xxd -r -p)

rm tmp.pem tmpPub.pem tmp.sig

Update: I believe the above script isn't working because pkeyutl defaults to a sha1 fingerprint before signing.

Are you following BIP143? It has a number of examples.

It'd be helpful if you could post the code you're using to sign.

@NateEldredge Just added a link.

Slightly related, do you know where I could find the bip that defines the original hash generation for p2pkh utxos? Not SegWit.

Figured out a few things:
Segwit Signatures
For segwit transactions one must sign the following (see BIP 143):
Double SHA256 of the serialization of:
     1. nVersion of the transaction (4-byte little endian)
     2. hashPrevouts (32-byte hash)
     3. hashSequence (32-byte hash)
     4. outpoint (32-byte hash + 4-byte little endian)
     5. scriptCode of the input (serialized as scripts inside CTxOuts)
     6. value of the output spent by this input (8-byte little endian)
     7. nSequence of the input (4-byte little endian)
     8. hashOutputs (32-byte hash)
     9. nLocktime of the transaction (4-byte little endian)
    10. sighash type of the signature (4-byte little endian)

For the example above, a P2SH-P2WPKH, that would be:
hash prevouts - hash256(1333183ddf384da83ed49296136c70d206ad2b19331bf25d390e69b222165e3700000000)
6623eab09650c6a6a98617d581c5d1fc26c6f5f158820e68ee636be93b433cee

hashSequence - hash256(feffffff)
18606b350cd8bf565266bc352f0caddcf01e8fa789dd8a15386327cf8cabe198

hashOutputs - hash256(00e1f5050000000017a914a860f76561c85551594c18eecceffaee8c4822d787F0C1A4350000000017a914d8b6fcc85a383261df05423ddf068a8987bf028787)
6a7522acd2fc865421d71893d71027bc3ebe0839d25dabc6277aef64b24b2370

hashPreimage
020000006623eab09650c6a6a98617d581c5d1fc26c6f5f158820e68ee636be93b433cee18606b350cd8bf565266bc352f0caddcf01e8fa789dd8a15386327cf8cabe1981333183ddf384da83ed49296136c70d206ad2b19331bf25d390e69b222165e37000000001976a914b93f973eb2bf0b614bddc0f47286788c98c535b488ac00ca9a3b00000000feffffff6a7522acd2fc865421d71893d71027bc3ebe0839d25dabc6277aef64b24b23708c00000001000000

nVersion:     02000000
hashPrevouts: 6623eab09650c6a6a98617d581c5d1fc26c6f5f158820e68ee636be93b433cee
hashSequence: 18606b350cd8bf565266bc352f0caddcf01e8fa789dd8a15386327cf8cabe198
outpoint:     1333183ddf384da83ed49296136c70d206ad2b19331bf25d390e69b222165e3700000000
scriptCode:   (OP_DUP OP_HASH160 <pubkey hash> OP_EQUALVERIFY OP_CHECKSIG) where <pubkey hash> is from redeemscript from previous output
              1976a914b93f973eb2bf0b614bddc0f47286788c98c535b488ac
amount:       00ca9a3b00000000 (where amount is the total utxo amount)
nSequence:    feffffff
hashOutputs:  6a7522acd2fc865421d71893d71027bc3ebe0839d25dabc6277aef64b24b2370
nLockTime:    8c000000
nHashType:    01000000 (SIGHASH_ALL)

The hashPreimage is double sha256 hashed to produce:
ef7d163bfe970439476d15537e1373bc23bb6282b9e145115331975a1a673788, which is the data to be signed. This results in a signature: 30440220434caf5bb442cb6a251e8bce0ec493f9a1a9c4423bcfc029e542b0e8a89d1b3f022011090d4e98f79c62b188245a4aa4eb77e912bfd57e0a9b9a1c5e65f2b39f3ab401.
Signature Script
However, the signature script above would never work didn't match because openssl cli uses a random nonce each time you sign, see https://bitcoin.stackexchange.com/a/32962/60443. So I wrote a small cli tool using libsecp256k1 utilized in bitcoin. Compiles using:
g++ -std=c++11 -o ec_sign ec_sign.cpp $(pkg-config --cflags libsecp256k1 --libs libsecp256k1)
Usage: ec_sign <hash-to-sign> <priv-key> <hash-type>(default 1: SIGHASH_ALL)
#include <secp256k1.h>
#include <iostream>
#include <iomanip>

int char2int(char input)
{
  if(input >= '0' && input <= '9')
    return input - '0';
  if(input >= 'A' && input <= 'F')
    return input - 'A' + 10;
  if(input >= 'a' && input <= 'f')
    return input - 'a' + 10;
  throw std::invalid_argument("Invalid input string");
}

void hex2bin(const char* src, char* target)
{
  while(*src && src[1])
  {
    *(target++) = char2int(*src)*16 + char2int(src[1]);
    src += 2;
  }
}

// EC sign/verify
// ----------------------------------------------------------------------------

int main(int argc, char *argv[])
{
    uint8_t hashType = 1;
    if(argc < 3)
    {
        std::cerr << "Usage: ec_sign <hash-to-sign> <priv-key> <hash-type>(default 1: SIGHASH_ALL)" << std::endl;
        return 1;
    }
    if(argc == 4)
    {
        hashType = argv[3][0] - '0';
    }
    char hash[32];
    hex2bin(argv[1], hash);
    char secret[32];
    hex2bin(argv[2], secret);

    size_t siglen = 71;
    secp256k1_ecdsa_signature signature;
    static secp256k1_context *ctx = secp256k1_context_create(SECP256K1_CONTEXT_SIGN | SECP256K1_CONTEXT_VERIFY);

    unsigned char derSig[71];

    if (secp256k1_ecdsa_sign(ctx, &signature, (const unsigned char*)hash, (const unsigned char*)secret, secp256k1_nonce_function_rfc6979, NULL) != 1)
    {
        std::cerr << "Signature failed" << std::endl;
        return 1;
    }

    if (secp256k1_ecdsa_signature_serialize_der(ctx, derSig, &siglen, &signature) != 1)
    {
        std::cerr << "Serialization failed" << std::endl;
        return 1;
    }

    // Apply Hash type
    derSig[70] = hashType;

    for(int i=0; i<71; i++)
    {
        std::cout << std::hex << std::setfill('0') << std::setw(2) << (int)derSig[i];
    }
    std::cout << std::endl;

    return 0;
}

Using random nonces is perfectly legal. It is not observable to the network.

Thank you very much @JBaczuk!!
Did you know why they use the "outpoint" name instead of "outputs"?
In the BIP https://github.com/bitcoin/bips/blob/master/bip-0143.mediawiki#Specification they said "same meaning as the original algorithm" but for me the "outpoint" word is brand new!

@aqquadro The term outpoint is shorthand for the combination of a transaction's hash (prevHash) and its index (vout).


text:
When are blocks validated?
Are blocks validated just before they are stored to memory,  upon receipt from an RPC submit call, or at some other time? Can you point to the place in code where this validation process begins? Is this process starting point the same for transactions?

Transactions are validated before they enter the mempool (see AcceptToMemoryPoolWorker(), and when they are submitted as a block.
Blocks are validated when a new one is received from a miner (see ProcessNewBlock(), and when they are synced from other peers (see ProcessMessage(), including on startup of a new full node.

I think you are talking about the consensus rules and when they are enforced...
To get the whole picture I would suggest to read about this topic in the book "Mastering Bitcoin" by Andreas Antonopoulos.
https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch10.asciidoc#decentralized-consensus

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why is 20 the maximum public keys in a multisig transaction?
In script.h the following limit is set:
MAX_PUBKEYS_PER_MULTISIG = 20
I'm wondering:

Why is the limit set to 20?
How would one would construct a script that allows n more than the current OP_16?

E.g. would this work?
OP_m <pubKey1> ... 1 20 OP_CHECKMULTISIG

Related: https://bitcoin.stackexchange.com/questions/51509/will-segwit-allow-for-m-of-n-multisig-with-very-large-n-and-m/51653#51653

The number 20 seems to originate from this commit : 8c9479c6bbbc38b897dc97de9d04e4d5a5a36730, also tagged as v0.3.12, which introduced the term 'SigOps".  Sadly, I don't see any mention of this change in the commit message itself, or in this version's release note, and there isn't any documentation.
We see that both a limit for a maximum number of sigops in a block was set :
static const int MAX_BLOCK_SIGOPS = MAX_BLOCK_SIZE/50;

and a way of counting them :
int GetSigOpCount() const
{
    int n = 0;
    const_iterator pc = begin();
    while (pc < end())
    {
        opcodetype opcode;
        if (!GetOp(pc, opcode))
            break;
        if (opcode == OP_CHECKSIG || opcode == OP_CHECKSIGVERIFY)
            n++;
        else if (opcode == OP_CHECKMULTISIG || opcode == OP_CHECKMULTISIGVERIFY)
            n += 20;
    }
    return n;
}

And the rules are still the same, specifically for bare multisig scripts (those not nested in p2sh or p2wsh).  A bare CHECKMULTISIG operation is always counted as 20 sigops (today 80, with witness scale factor), so allowing anything more than 20 checksigs in CHECKMULTISIG will invalidate this assumption and will make it so creating blocks with more then maximum allowed sigops possible.
The rules are less strict for p2sh\p2wsh scripts.  If the opcode just before CHECKMULTISIG is in the range [OP_1, OP_16], then that number is counted as the sigops for the multisig verification, with anything else counted as 20.  In practice, 15 is the maximum of pubkeys you can push as a p2sh script because of the 520 bytes push limit, but for p2wsh, an example of 17-of-20 would look like :
0x01 0x11 <pubkey1> <pubkey2> ... <pubkey 20> 0x01 0x14 CHECKMULTISIG

The real limit was introduced a few weeks later in v0.3.13: https://github.com/bitcoin/bitcoin/commit/a790fa46f40d751307f86c37a709eb119768ce5b#diff-27496895958ca30c47bbb873299a2ad7a7ea1003a9faa96b317250e3b7aa1fef
The sigops limit is per block so it doesn't preclude an individual multisig transaction with more than 20 signatures. At the time the standard miner limited the block sigops to 100 (and the block size to 10kb). Nodes also wouldn't relay transactions with more than 2 signatures.
But someone could have made a block a big multisig until v0.3.13

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What is the maximum number of bytes the coinbase scriptSig (coinbaseaux) can hold?
Since the scriptSig of the coinbase transaction of a block can contain arbitrary data, what is the maximum number of bytes it can contain? I've seen various mentions of a limit around 100 bytes, but can't find that defined anywhere.

The maximum is 100 bytes and is checked and defined here: https://github.com/bitcoin/bitcoin/blob/master/src/consensus/tx_verify.cpp#L195

The check was moved a few years ago, this link to v26 should be stable: https://github.com/bitcoin/bitcoin/blob/v26.0/src/consensus/tx_check.cpp#L46-L49

Fwiw the check has been present since v0.1.0 (see "coinbase script size" in main.h)


text:
How was 59 days determined with 1 TH/sec from "Mastering Bitcoin"
In Mastering Bitcoin in the section on Difficulty Bits Andreas notes:

This means that a valid block for height 277,316 is one that has a block header hash that is less than the target. In binary that number would have more than the first 60 bits set to zero. With this level of difficulty, a single miner processing 1 trillion hashes per second (1 tera-hash per second or 1 TH/sec) would only find a solution once every 8,496 blocks or once every 59 days, on average.

How was 59 days calculated?
2^60 (1.15 x 10^18) would be the number of possible values to cycle through and 1 TH/s = 1 x 10^12 attempts every second but what am I missing?

I also get an expected 13 days, perhaps you should write an email to Andreas or open an issue on GitHub.

Sounds like he calculated 59 days first, using the method you mention. And then for some reason did 59 * 6 * 24 (1 block per ten minutes) to come to 8496.
I agree that it confuses more than it adds.

Would you know how 59 days was calculated? By my calculations, at 1 TH/s, it would take 1,152,921 seconds to try every possible value. ie 13 days.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Paranoid Merchants on double spending
How can merchants be alerted of Double-spending attacks? Let us take a hypothetical case where a merchant offer a BTC deposit wallet to his users, User deposits Bitcoins and it's credited to his account on 6 confirmations. Let's say, theoretically, the attack has been pulled off and he got his bitcoins back + the funds on the wallet remained as the database haven't been updated due to no alerts of any double spend attacks.
Question regarding this scenario:

How can merchant be alerted of any double-spend attacks ?
Procedures to consider in the time of the attack ? e.g Shutting down servers to avoid confusion and recheck balances to avoid dead ends on user withdrawals
How safe is it to run an out of the box Bitcoin Daemon?
Will Bitcoin XT be alerting on any double spend transactions that occurred no matter how the attack succeeded ?
Is it Okay to host a high bandwidth and Intensive website along with a bitcoin daemon?
If a double-spend attack is pulled off, how to manually verify transactions via the bitcoin daemon and how to know how much amount lost ? Will the saved transaction hash can be re-usable to obtain info about the transactions using gettransaction?
How Unlikely is it to pull of an attack to a website that credit users on 6 confirmations?

:::UPDATES:::
I've read most of the answers and seems nobody is able to answer this question:

If a double-spend attack is pulled off, how to manually verify transactions via the bitcoin daemon and how to know how much amount lost ? Will the saved transaction hash can be re-usable to obtain info about the transactions using gettransaction?
Will the attack be able to pull off after 48 hours?

Note about XT: don't confuse 0 confirmation doublespend relay (what XT does, and thereby opening the network to DDOS attack) with doublespending after 6 confirmations. XT doesn't do anything different there.

Hi Alessandro, Welcome to Bitcoin.SE. In order to get good answers, please ask one specific question per post. You have asked a broad range of questions that should be separate posts. Please edit this question down to one topic and then flag for reopening. I suggest a split along this lines: 1. How can a merchant get alerted of doublespend attacks? 2. How likely is the success of a doublespend attack after six confirmations? 3. How does one safely run a Bitcoin Daemon? 4. How much bandwidth does the Bitcoin daemon consume? … Please make each its own post!

For a longer explanation please see How to deal with question posts containing several different questions?

How Unlikely is it to pull of an attack to a website that credit users on 6 confirmations?

I would say extremely unlikely as discussed here, here, here and here. Unless you are selling small islands on the website there would be no incentive to acquire the immense resources required for an attack.

How can merchant be alerted of any double-spend attacks ?

Running their own daemon will achieve that.

Procedures to consider in the time of the attack ? e.g Shutting down servers to avoid confusion and recheck balances to avoid dead ends on user withdrawals

Any backend services run by a merchant need to be able to handle block chain reorganisations anyway! They happen normally, randomly, and without malice. Shutting down is obviously not an option for an event which happens more than once a day. Not handling this behaviour will open the merchant up to rampant theft.

How safe is it to run an out of the box Bitcoin Daemon?

Reasonably, the daemon is generally considered to be safe and is written in a language that is normally considered to be safe (C++). However, there have been potential code execution problems due to OpenSSL library and miniupnp. A good node would be at a minimum in a different user account, not not different hardware from production services.

Will Bitcoin XT be alerting on any double spend transactions that occurred no matter how the attack succeeded ?

Bitcoin XT is irrelevant. The Bitcoin Core software does not handle alerting you to this, however the best canonical chain changing makes it fairly obvious through the RPC interface.

Is it Okay to host a high bandwidth and Intensive website along with a bitcoin daemon?

No. Bitcoin Core is extremely intensive on CPU, IO and memory. It is an extremely harsh test of hardware and will frequently find underlying issues like disk and RAM failure better than any other test- at the cost of your stability. You don't want a vulnerable web server next to something storing money.

How Unlikely is it to pull of an attack to a website that credit users on 6 confirmations?

Perhaps unlikely.

Please don’t answer question lists, instead flag them as "too broad" or "low-quality". They don’t work well on our platform and will probably be heavily edited or deleted. If you have put time into it already then, it makes us feel bad when we clean it up, because you’ve probably wasted most of your time.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Calculating fee rate of tx returned by fundrawtransaction
I'm passing an unfunded proposed tx and feeRate param to fundrawtransaction which gives me back the hex of the tx with inputs selected and its fee. I'm not passing a conf_target or estimate_mode. I pass the resulting hex to decoderawtransaction to get the size/vsize of the tx.
I'm trying to check the actual fee rate of the tx by calculating fee * 1000 / vsize but it doesn't seem to match the rate that I passed in as a parameter.
I've tried with a few different values but the actual rate always seems to come out higher than the requested rate by a factor of about 1.59, give or take a little bit.
So am I calculating the observed rate improperly? Is it even possible to have the resulting fee rate exactly match the one provided by the feeRate parameter?
Here's an example of constructing a tx with amount 0.00073749 and feeRate 0.00001000. Proposed tx from createrawtransaction:
{
  "txid": "de380dec00d86611bab1cfa825a7069ffde31d7fb0c5b91e381ef79b94d927e0",
  "hash": "de380dec00d86611bab1cfa825a7069ffde31d7fb0c5b91e381ef79b94d927e0",
  "version": 2,
  "size": 41,
  "vsize": 41,
  "weight": 164,
  "locktime": 0,
  "vin": [
  ],
  "vout": [
    {
      "value": 0.00073749,
      "n": 0,
      "scriptPubKey": {
        "asm": "0 de525f12f17a495b7903c2c5b306ee07ccad309f",
        "hex": "0014de525f12f17a495b7903c2c5b306ee07ccad309f",
        "reqSigs": 1,
        "type": "witness_v0_keyhash",
        "addresses": [
          "bc1qmef97yh30fy4k7grctzmxphwqlx26vyl5lve86"
        ]
      }
    }
  ]
}

And the full tx with inputs returned by fundrawtransaction:
{
  "txid": "f78b268f9386ab47dd5c0d36caf3aaff5fa719c7a2b5658ff08132159cb6b8cd",
  "hash": "f78b268f9386ab47dd5c0d36caf3aaff5fa719c7a2b5658ff08132159cb6b8cd",
  "version": 2,
  "size": 369,
  "vsize": 369,
  "weight": 1476,
  "locktime": 0,
  "vin": [
    {
      "txid": "ca6f25d4484ab525fdb907cbce81d3dc9d1974695fcc80639fddfff4b6bca268",
      "vout": 1,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "sequence": 4294967294
    },
    {
      "txid": "9b39ee5637d7b0783715be6ce566283024ee2bea33a9d3bd6f415ae5d8f24e5f",
      "vout": 0,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "sequence": 4294967294
    },
    {
      "txid": "2f2b7eaccc06ea3856cd5b4666a15161d0fac9053ab2bcbb94a490593c969246",
      "vout": 0,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "sequence": 4294967294
    },
    {
      "txid": "b5b2e4e1287438fd8193e1409cdf87181061ab4a3d95f50d2bf1bc2534b2d49c",
      "vout": 0,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "sequence": 4294967294
    },
    {
      "txid": "3de9c3690c77f544b0d953c70d97f50bb9790a847bed9972caca960218c6c8f6",
      "vout": 0,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "sequence": 4294967294
    },
    {
      "txid": "5e0021645a37473afdff1720da902f0abe183fb48719169e96b5235bb6b09e25",
      "vout": 0,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "sequence": 4294967294
    },
    {
      "txid": "f92dbcf77c13d887d8aa77404dc78120c90d055a32b4d51196849affc3866ade",
      "vout": 0,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "sequence": 4294967294
    },
    {
      "txid": "e425c6e84bc671bfb3a80bd95d1925007e12483fdcf3a19094bde8a36ae0d9fd",
      "vout": 0,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "sequence": 4294967294
    }
  ],
  "vout": [
    {
      "value": 0.00073749,
      "n": 0,
      "scriptPubKey": {
        "asm": "0 de525f12f17a495b7903c2c5b306ee07ccad309f",
        "hex": "0014de525f12f17a495b7903c2c5b306ee07ccad309f",
        "reqSigs": 1,
        "type": "witness_v0_keyhash",
        "addresses": [
          "bc1qmef97yh30fy4k7grctzmxphwqlx26vyl5lve86"
        ]
      }
    }
  ]
}

In this case the fee is 0.00000668 so the rate I calculate is 0.0000668 * 1000 / 369 = 0.00001810 as opposed to the requested 0.00001000

Does the resulting transaction have a change output? And how large is it (in terms of number of inputs and outputs)?

Hi Pieter, I just updated the question with an example with just 1 output.

You're seeing two things here: the exact match coin selection algorithm, and the fact that your vsize is computed before adding signatures.
Exact match
The coin selection algorithm in Bitcoin Core tries to find an "exact match" of UTXOs to spend in order to pay, without adding a change output back to you. If the input amount is a bit too large, it is turned into fee. Surprisingly, under certain circumstances this behavior is actually preferable over turning that excess into change.
The observation is that merely creating a change output costs money right now (in terms of fee needed to pay for it), and more money later (in terms of the marginal fee needed to pay for the additional space used by the transaction input that will spend it). That fee is wasteful: if it's possible to not create the change at all, this becomes a cost you never have to pay.
So the conclusion is: if it is possible to construct a transaction without change that overpays less than what it would cost to create and spend change, it is preferable. This is exactly what Bitcoin Core does, using a short-term fee estimation for the creation of the change, and a long-term fee estimation for spending it.
The concept and algorithm for doing this efficiently were first described by Mark Erhard in his master's thesis.
vsize before signatures
You're looking at the vsize of the unsigned raw transaction coming out of fundrawtransaction. For fee decision purposes, fundrawtransaction tries to predict the size of the transaction after it is signed.

Thanks for the illuminating response, this all makes sense to me now!

Hi Pieter, in the case of bitcoin everything is working as expected. Now that I get the vsize from the signed tx, the observed rate I'm calculating is almost exactly the rate that was requested. However I'm still seeing a discrepancy between the observed and requested rates for liquidv1 transactions. Could there be an issue with fundrawtransaction not anticipating the signed tx size properly there, or do you have any other insights why things might be different for liquid?

I have no idea, sorry! You may want to open an issue for that.

It turned out I wasn't blinding the liquid transaction before signing it. https://github.com/ElementsProject/elements/issues/852


text:
Why does bitcoind use a fork of LevelDB for key-value storage?
Since 0.8 bitcoind uses LevelDB for the storage of the UTXO set in chainstate/ and the block index in blocks/index/. On github special branch of LevelDB can be found:
https://github.com/bitcoin/leveldb
What reasons made LevelDB the best choice for key-value storage compared to other embedded database solutions? How does bitcoind's own LevelDB fork differ from the official version?

The fork serves two purposes:

Local modifications that are hard to bring upstream:

Windows support (which is partially based on the existing Windows port, but needed changes for building in MinGW)
Removal of compression support, as it doesn't help, and complicates the build.

Strict control over changes. Given the previous experience with the BDB to LevelDB changeover that caused a fork in the network, it is wise to not use system-wide installed versions of the library. Hypothetically, even if a change is a bug fix from the point of view of LevelDB, we may want to maintain the buggy behaviour to prevent a fork in the network, depending on the bug.


text:
What would happen if the mining reward suddenly became 0?
If the mining reward was suddenly changed to 0, and there were no mining fees anymore, what would happen with the Bitcoin network? Assume there is another anti-spam measure in practice (ex: a tx would need the last block hash + some small PoW on top of it). Would Bitcoin suddenly collapse and get unusable, would it still operate normally due to voluntary mining?

Possible duplicate of What will happen to mining after the 20 999 999th Bitcoin?

I marked this as a duplicate because it's not a matter of if, but a matter of when, and there's already some good answers.

@Jestin you're right, but my question was incomplete. I actually meant to ask what would happen if we removed any kind of mining reward, including fees. Basically, it is a thought experiment on which I want to know if the network could still operate safely solely with voluntary mining. I edited the question, could you remove the duplicate flag?

in that case this question may be a little to speculative for Bitcoin SE.  This forum is best for questions with concrete answers, and not so great for thought experiments, speculation, or debates.  It's too hard for the community to determine what a "good" answer is.

Okay then, Jestin.

It's a somewhat interesting question, so I took a stab at an answer.  Still, I wouldn't blame anyone for marking this as OT ;)

This question is not useful. The reward is part of the core assumptions of Bitcoin that you might as well be asking what would happen to Bitcoin if there were no internet.

Okay then, Murch.

Well, in a way that's perhaps actually the answer that you're looking for: Without a reward either through fees or block subsidy, the incentives of miners to rather support the network instead of attacking it don't hold. Miners wouldn't have a reason to spend irretrievable resources when they have nothing to gain. You'd need a different incentive or public fund to step in, and that would likely lead to centralization and/or other degenerations. –– I.e. it breaks the game theory of the Bitcoin system.

@Murch but wouldn't large Bitcoin stake holders have an incentive to mine (at a loss) in order to protect the system? That is exactly what I'm asking: is it actually, factually known for sure that without the mining incentive the game theory is broken? I know the intuition strongly suggests that is that case, but has anyone actually proven it? Did anyone attempt to put a Bitcoin clone without the mining incentive?

@MaiaVictor: Perhaps you should consider to either edit this question or ask a new question along the lines of "Could Bitcoin work without block subsidy and transaction fees?" IMHO that's a much more interesting question than "what would happen if Bitcoin stopped having a block subsidy and fees tomorrow". :) I'd vote to reopen that, and would write an answer on that.

But there is already a good answer to the question as it is. I'll ask again.

Even though they are still somewhat negligible to miners today, transaction fees play an important role in the protocol known as Bitcoin, and will play an increasing role in the future.  Just look at all the questions this SE site gets about unconfirmed transactions.  Asking how the protocol works when removing all direct mining incentives is redefining the protocol in a major way.  It's like asking how HTTP would work if you remove the verbs from the protocol.  At that point, it's something completely different.
When you remove such a critical part of the protocol, it's difficult to replace with something that works.  For example, your proposal of doing a proof-of-work on top of the last block hash as part of the transaction will effectively break the ability to use the protocol for smart contracts.
You are essentially asking what happens if you remove critical parts of Bitcoin, but still use the basic idea rather than creating a new protocol to match your paradigm of no mining rewards.  To that, I can only speculate that such a system is doomed for failure.


text:
Would Bitcoin still work without a target difficulty?
Suppose that Bitcoin nodes had no target difficulty at all, and instead just accepted the block with the largest PoW every 10 minutes. Ignoring the increased bandwidth (since nodes would be broadcasting several block candidates), would this work? Or is there some technical need for an adjustable target difficulty?
Edit: to clarify, I meant that every block is accepted, but only the chain with the largest accumulated work is considered canonical. I.e., if there is a weak block days ago, and you mine a block with higher score, that weak block does not become invalid. You need to replace every block on top of it if you want to rewrite the story that far. I.e., it is the same as Bitcoin, without a target difficulty. Nodes just accept all blocks, and compute the best block by summing the accumulated work of each candidate tip.

I've got 3 great answers, thanks! But sadly I still do not see how a target difficulty is required for consensus. You're for some reason implying that, without a target difficulty, we'd need a central coordinator for consensus. But that does not seem to be true: as long as nodes collect blocks and compute the chain with the most accumulated work, they will reach consensus w.r.t. canonical chain. Yes, we must assume that we're able to get the data from non-malicious peers, but Bitcoin also assumes that!

What's a "largest PoW" ? The amount of work done on a block to hash it is pretty much the same as any other block, so how do you assert that the work done is larger for this than that?

By the value of the block hash. You can assign a block score as 1 / block_hash, which is an estimative of how many nonces were attempted to make this block. Then, the sum of all block hashes up to root is the score of this specific chain. Nodes just receive blocks and select the chain with largest score.

how many nonces were attempted to make this block - whut?

It is a simple sentence. What part are you having trouble understanding?

The part where you asserted that a block hash value is in any way related to the number of nonces that were tried in order to realize a hash that is smaller than the difficulty. A miner could get lucky and pack a block that hashes to lower than the difficulty on the first go, or it could get unlucky and chew through a billion block variations before it achieves the same

Good question! Turns out it is related, statistically. Of course, you can't determine how many nonces were actually attempted to produce a specific block, since, as you've noticed, a lucky miner could get a very low hash on the first attempt. But it is possible to estimate - mind the keyword. For example, if the first 4 bits of a hash are 0, you can assume it took at least 16 hashes to produce it. Of course, that may not be the case for a specific block, but, on average, this statistic holds. Makes sense?

Current bitcoin doesn't add (estimated) accumulated work to choose a chain, but it just counts chain length. It works as easy substitute for accumulated work, because nearly every block is near the target difficulty. In your scheme, actual work estimation needs to be implemented, but on first look, it might work, too.

Oh, I didn't know that, but makes sense that BTC doesn't need to sum the work as I describe.

@MichaelKarcher: Bitcoin looks for the chaintip with the most work: https://github.com/bitcoin/bitcoin/blob/4b1fb50def0dea0cd320bc43c12d9a12edde0390/src/validation.cpp#L2399. The original release had the bug that it only compared the length, but this would have allowed long-range low difficulty reorg attacks and was fixed many years ago.

It's not trivial a) to agree on the time in a distributed system, especially how much time has passed between two events, b) to establish when exactly a block candidate was found. Further, when a node is catching up with the chaintip after being offline, it has no means whatsoever to determine whether a block was actually found within the allocated timeframe.
So now, instead of someone finding a block and announcing it, we need to send around a lot more data, and then run some sort of consensus algorithm to agree whether/which candidates were submitted in time.

But sadly I still do not see how a target difficulty is required for consensus.

Without difficulty, it becomes subjective if the next block has been found because the criteria is no longer binary. If any difficulty is acceptable, we'd be finding billions of blocks every moment. Let's say all these block announcements cascade through the network. Every node now gets multiple announcements and does unnecessary block validation, even if they only forward the best block they've seen so far at the given height.  How do miners know when they should switch to the next height? How do they know they have seen the best block at the height already? Reorgs would be happening all the time. If the miners are constantly switching the previous block of their block templates to update to the latest state, they'll be almost entirely mining block candidates that will not be compatible with the final best chain. Most of the proof of work would be entirely wasted and would not contribute to the security of the network at all. Additionally, the bandwidth load on the network would drastically increase to gossip all the blocks.
Especially miners with larger shares of the hash power now have all sorts of additional angles to attack the previous block rather than moving forward, e.g. selfish mining and Finney attacks become easier. Getting rid of the difficulty would hamper progress of the blockchain and would make even multiple confirmations unreliable.

Why doesn't just PoW suffice for that consensus? As long as nodes have a roughly synchronized clock, the time a block was mined could be computed by genesis_time + 10 mins * block_count. Nodes just broadcast blocks as usual and pick the chain with the largest work.

Let's assume I'm a miner and I see that the PoW in the last ten minutes was pretty low, so I continue to mine for a bit and fudge the timestamp on my block candidates. When I do find a block with a higher PoW, I announce it to the network. How do you distinguish whether my block candidate was found in time or not?

If every node decides for itself, you have now a great way to split the network. If the nodes decide together, you need a consensus algorithm.

How do you propose to enforce the "best block within a time span of ten minutes" without timestamps?

@Murch there is no timestamp. But yes, you could "re-mine" old blocks if you wanted to. But you'd still need to provide all the newer blocks, with more accumulated work, for it to be accepted. Re-mining sufficiently old blocks would be infeasible, so, ultimately, nodes do reach a consensus w.r.t. the canonical chain, exactly like on Bitcoin, no? What I'm saying is: I still don't see why a target difficulty, rather than just collecting blocks and computing the largest chain, is required for that.

@MaiaVictor: If I had a solid attack on SHA-256 I'd prefer to announce it by rolling back ransomware transactions that way.

If you don't have a difficulty, how do you determine the PoW that was done? Just by the lowest hash?

If so, no, it's not finding the best chain like Bitcoin. For example a malicious miner can try sending money to themselves in every block that they mine. When they find an extremely low hash, they keep that block secret and make a conflicting transaction sending that same money to an exchange. They let their deposit get a couple confirmations and then trade it for an alternative currency. Then the attacker publishes their block with an extremely low hash to replace the previous best chain.

While blocks that are buried are still hard to replace, the chaintip becomes completely unreliable.

@Murch yes, lowest hash. That is exactly what I mean. Bitcoin's chaintip isn't 100% reliable either, you need ~6 confs for maximum security. In such proposal, the finality time should be considered to be quite large (say, 24h). My point is: it would work as a consensus algorithm, it would just be less efficient for finality than Bitcoin, but would be arguably simpler to implement, and more secure to against these several target-difficulty-related attacks. Does that make sense?

I don't understand how you arrived at that assessment. I think it would be much less secure since most of the hashrate is now wasted on unnecessary competition rather than progress, it would require way more confirmations to safely transact, it would become an art to assess whether the current best block should still be competed with or settled for as the prev best block, it would be way more inefficient in bandwidth and computing power for participating nodes. It's not clear to me in what way the proposed approach is better at all.

I see, now these are interesting points to keep in mind. I'm not sure if what you're saying really holds, i.e., if most of the computing power would be wasted, and if it is really hard to define which block to mine upon. But these are sensible points to consider. Thanks for the inputs.

@MaiaVictor: Riddle me this: if you were a small miner on the network, and found a block with difficulty of 1/100th of the current hashrate. Would you a) broadcast it and try to mine on top of it or b) try to mine a better block at the same height? As another miner, when you receive a block at 1/100th of the current hashrate, would you a) switch to using that as the previous block, b) try to mine a better blocks?

What if you receive five blocks with 1/100th of the difficulty building on top of each other within one minute? Would you build on top of that, or continue to try a single block with 1/10th the difficulty to replace all of them?

I think you'll always be mining on top of the best block for the last 10 mins period, trying to get smaller and smaller hashes, and broadcasting the best block you have (even if it isn't yours). As long as miners keep doing that, blocks with low difficulty will just not propagate, and good blocks will reach everyone. After the 10 mins mark, miners would start mining on top of the best block they have for the latest 10 minutes period, which will almost always be the right one. If it isn't, soon enough the right one will arrive and they won't have wasted that much effort.

Of course, you can try to re-mine older blocks, but that makes it much less likely that your blocks will propagate, so you have incentive to always mine on top of your best block, i.e., your tip with the most accumulated PoW.

"I think you'll always be mining on top of the best block for the last 10 mins period" ⇒ It's not clear why you believe that miners would converge on this particular behavior. Trivial counterexample: A miner finds an extremely low hash. The most rational thing would be to immediately start mining on top of it, perhaps even keep it secret for a while to make other miners waste their mining power.

Let us continue this discussion in chat.

The key point of the Bitcoin design is to create a system that provides a reliable transaction record without any central coordination or arbiter. Instead, Bitcoin is a distributed system where a consensus is reached using a "consensus algorithm" that can be repeatedly evaluated by every node on the network.
As Bitcoin is operating today, the target of a "block period" is well-defined and agreed upon by all nodes. A node can verify whether a certain block is a valid solution using just knowledge from the chain (i.e. the target difficulty) without relying on any clock.
In the system you propose, you can't decide whether you have seen "the best block of the 10-minute timespan" by just looking at the block. This means you don't know whether that block is a valid block without knowing all other candidates. The consequence is that a block you considered valid might become invalid when a block with a lower hash appears. This is something that can not happen according to the current scheme, and it has dramatic consequences. Today, a block that is considered valid can safely be mined upon, as the work required to outrun the valid block is expected to be around the same as continuing on that block. In your scheme, it takes just the work of finding a better block to make the previously accepted solution invalid and you destroy the work based on that now invalid block "for free".
So when you start mining on a block, you need to be confident that you know a valid solution, and that this solution will stay valid. The usual approach to the challenge would be a central arbiter that collects candidates and decides which block to pick and fixes this decision for good. As I already stated in the introduction, anything relying on a central trustworthy service doesn't meet the primary goal of Bitcoin.

In the system you propose, you can't decide whether you have seen "the best block of the 10-minute timespan" by just looking at the block. -- How so? The best block is the one with the highest PoW. In Bitcoin, you, too, can't tell whether you have seen the correct block in a 10-minute timespan, without knowing all blocks: two blocks could have been mined simultaneously. The dispute is resolved after more blocks. The exact same process takes place in my system, by just picking the chain with the largest PoW to begin with. There is no central coordinator involved.

My point was to answer the question: "can you decide whether a block can be a valid solution". In the proposed system, every block can be a valid solution. In the current system, only blocks hitting the target can be a valid solution. Because finding blocks that hit the target is extremely rare, usually only one block that solves a given depth is ever calculated and can easily be identified (by it hitting the target). In the proposed system, you can't easily identify whether a block you get shown is most likely the generally accepted block at that depth.

Ah, fair enough, but why is that important or desirable? Do you mean as a way to prevent flood/spam? It is fair that you can't know if a block is valid, but for all practical purposes you can make very good guesses. For example, if you received a block with a large PoW score, you could probably ignore any block with <50% of its PoW score.

I edited my answer to focus on the "beating one old block invalidates  whole chain of blocks" issue, which is actually the core behind the "is this block valid" question.

I appreciate your answer, it is turning great, but I still have a few concerns. 1. "Today, a block that is considered valid can safely be mined upon" - safely in what sense? In Bitcoin, it can also happen that you're mining upon a block that becomes invalid, if that block becomes an uncle. 2. "The usual approach to the challenge would be a central arbiter that collects candidates and decides which block to pick and fixes this decision for good." That doesn't make sense to me. Instead, you'd just mine the topmost block and hope it doesn't become invalid, in your words. Makes sense?

The main difference is that currently a chain gets orphaned if someone presents a longer chain, which needs much work as the orphan chain. Your scheme allows attacking just one block, the weakest link, to take down the whole chain depending on it, because a previously valid block ("lowest hash") now obviously isn't valid anymore ("it's just the second-lowest hash, sorry"), and you don't have to beat the chain following this block.

So my key point is "orphaned valid forks of the chain are OK, blocks that suddenly turn out invalid break the scheme." - so validness of a block must be a property that can't dynamically changem

Yes, that is a point, but note that you pick the longest chain by the sum of the work of every block, so just selecting a block with small score to replace won't work, as you also need to replace every block on top of it. I.e., a valid block doesn't turn invalid if a block with same height but higher score shows up. Every block is valid, you just pick the longest chain as the canonical timeline by summing the individual scores.

That's not what your question says. It says that every ten minutes, the block with the lowest hash gets accepted (and all other blocks obviously need to rejected). So it's not "most expensive chain wins", but the chain with the more expensive block at the split point wins.

Ah, that's how you interpreted it? That is not what I meant. I can see the issue now, and indeed my question is ill phrased. I meant that every block is accepted, but the chain with largest accumulated work is the canonical one.

For the record, I was assuming greatest sum of work, but the same problem still applies to the last few blocks at the chaintip: A single block with a very low hash could reorganize one or more blocks from the chaintip, accidentally or on purpose.

@Murch I fail to see that as a breaking issue, though. If the block with a very low hash is able to replace the chain tip, this block plus its descendants are still expected to have taken more work than the current chaintip, so this is in line with the bitcoin idea: The majority of hashing power is right. A soft issue with the proposed scheme is that reorganizations will happen more often until a consensus for a specific block is found, so less work goes into the final chain as more work is spent on loosing chains.

(and it seems my last comment is in line with your answer. It depends on how you look at it whether "no difficulty" will just be a worse but workable scheme with many reoganizaions and higher risks of underhanded attacks or these risks make that scheme unacceptable. I agree with you that reorganizations after nearly every block and the bandwidth increase are not desireable properties)

Yeah, it's not a complete no-go, but it would make Bitcoin less secure because a majority attack (that always builds on their own chain) would require a much smaller portion of the total hashrate to pull ahead, because so much hashing is wasted on furthering stale tips. It would also be a terrible experience from the perspective of users waiting for confirmations of their transactions.

Two potential problems with this idea, somewhat related.

Mining incentives will fluctuate during the 10-minute period, depending on the difficulty of the current block "candidate". If it is higher than expected, miners will be incentivised to stop mining until the start of the next period. In an extreme case, it might be that all miners stop mining for the rest of the period because the probability of mining the canonical block makes it unprofitable for all miners.

At the beginning of each 10-minute period, the relay network would be flooded with block "candidates". Because the current best block for the new period is likely to have a relatively low-difficulty hash, many new best-blocks are being generated in quick succession, and this could overload relay nodes. This could affect both network bandwidth (receiving blocks) and CPU (validating blocks as they arrive).

The existing bitcoin mining difficulty scheme does not suffer from these problems because the expected rate of new block generation is constant.

Good points. Also appreciate this answer as it understands what I proposed.

Recap
The difficulty was designed by Satoshi to balance the network and guarantee equal access to new peers, preventing malicious third parties to control the network, filter transaction and obtain the generated coins too easily.
The PoW is a mathematical proof that someone spent resources to gain the generated coins. If there was no PoW, an organized entity with a lot of resources, e.g. a botnet or an ISP, would spam the network with transactions that drive interest to them.
In particular, for larger transactions (1), we need to be particularly sure that Alice (malicious spender) does not issue a transaction to Bob, which sells goods, and then spams the network with transaction to merchant Charlie or to themselves to fraud Bob.
Answering (or trying to)
Consider that Bitcoin is totally decentralized and distributed systems suffer the concept of time, which is not absolute. If you have questions on that you should post on SW Engineering. I'll just say "without a certification authority you can hold a message forever". And Bitcoin refuses certification authorities that can be hacked/bribed/forced-by-government etc.
Satoshi needed to design a system so that sequential transactions could not (that easily) be rewritten (2).

Suppose that Bitcoin nodes just accepted the block with the largest PoW every 10 minutes.

The very problem is: look it not from the perspective of the whole system. Look it from the perspective of my Bitcoin-qt client bootstrapping right now from void.
How can my client be sure that nobody is retaining blocks? How can my client be sure about the network size judging from the input of other malicious peers?
Satoshi expected the network to grow and the computational power along with it. With more mining nodes, a larger number of blocks would hit the difficulty treshold in less than 10 minutes.
From the perspective of a lonely node, which is the very concept of peer-to-peer networks, the target is not to identify the block with the largest difficulty, which would be difficult if peers try to hide information, but look for the longest chain
The longest-chain principle does not prevent Alice completely from rewriting a transaction. It makes it so expensive that, at least for grocery purchases, it's worth following the rules than cheating.
And is the very reason why peers require 3 or more confirmations to consider a transaction final.
(1) Bitcoin was designed in order both to buy grocery and real estate. Of course, it's not that loss a double spend at the grocery
(2) Rewriting a transaction, in the scope of a blockchain, means that if transaction occurred at block 100, replacing every block from 100 to current with 1001 to current1

How can my client be sure that nobody is retaining blocks? -- You can't be sure of that in Bitcoin either. There could very well be fork with more work (a "longest chain") that you're not aware of. Bitcoin nodes just assume they've met enough collaborating peers to get that data. My proposed system does the same. The only difference is that, instead of a target difficulty, the node just picks the block with largest work. For the PoV of a lone node, it just collects blocks and computes the longest chain of the blocks it does see. It doesn't need a central party in any form to reach consensus.


text:
Why would increasing the Bitcoin block size lead to a more centralised system?
Why does Bitcoin Core contributor Peter Todd think that increasing the Bitcoin block size would lead to a more centralised system?
“The system doesn’t scale, and you just have to accept that and do something smarter. Every time you do a [block size limit] increase, you’re making the system more centralized.”
How does Bitcoin block size affect centralisation?

The problem arises from the uneven propagation of information in the network. While the propagation of small blocks is fast, i.e., every miner gets to know a new block approximately at the same time, this is no longer true for large blocks.
Large blocks take a lot of time to be forwarded to every node in the network, since this includes miners, these miners are unaware that there is a new block and they'll continue mining on top of the old parent, i.e., they will work on a competitor of the newly found block. This means that they are actively wasting their computational resources on an obsolete solution if the propagating block gets accepted into the blockchain.
Big miners are likely to have a better connectivity with the Bitcoin network, resulting in them losing less of their computational time due. This in turn puts small miners at a disadvantage since they are now less competitive.
The extreme case would be one strong miner, with close to a majority of the computational resources making such big blocks that everybody else learns about its blocks very late, while it already has spent considerable time finding the followup block.
The uneven propagation is due to both network latencies and verification times of the block that is being propagated, so this is a compound problem from both slow verifiers and high latency/low bandwidth.
For more detail see Information Propagation in the Bitcoin Network (disclosure: I am the author of that paper) and Majority is not Enough: Bitcoin Mining is Vulnerable.

1 MB per 10 minutes is about 1.7 KB per sec, 2 MB blocks would require 3.4 KB per sec. Anyone who doesn't have several orders of magnitudes more bandwidth than that, should not be part of any online network in the first place (but instead, get a decent connection). Old fashioned mobile 3G or even the lowest grade budget consumer internet connections offer thousand times faster speeds than that.

Oh and I know it's not like that in China yet, but what's preventing them for running their relaying server outside China? The actual mining work (i.e. their factories full of mining hardware) does not need the full block data at all, just the stratum which is independent on block size. If all it takes is a $5/month VPS to mitigate any bandwidth issues for a serious mining operation, you can't seriously claim that it hurts decentralization.

Also, I agree 8 MB or 20 MB blocks might be a bit steep for some people (although even then the effects should be minimal and easily circumvented) but 2 MB instead of 1 MB would be absolutely no problem whatsoever.

Yes, each individual node will trivially have the required bandwidth for 2MB, but since these blocks are relayed over multiple hops the simple bandwidth of a single node is not the bottleneck. Rather each nodes adds to the propagation delay and the longer the distance the longer it takes. For example take the propagation data from last week (http://bitcoinstats.com/network/propagation/2016/03/03), after a minute there still were 10% that had not heard about a block on average.

@cdecker If propagation delay is such a problem, why don't bitcoin developers, miners, and other stakeholders consider decreasing the block size?

Well there is a sweet spot between fork-rate and usability. Decreasing the block size further reduces the transaction rate that the network can handle, without increasing consistency much. On the other handle if we increase the block size we get a super-linear decrease in consistency. There certainly is a bit of wiggle room, but we simply cannot use block size to scale beyond a certain point.

@cdecker Cited paper is dated 2013 - bandwidth has increased circa 30% a year.
13th IEEE International Conference on Peer-to-Peer Computing, IEEE P2P 2013, Trento, Italy, September 9-11, 2013

@cdecker as per your own paper conclusion: "Our measurements show that a single node implementing these changes reduces the number of blockchain forks in the network
by over 50%".
Would the prominent change be: "[...] connect to every node in the network creating a star sub-graph [...] speeding up the propagation [...]"?

That is correct, however you need to keep in mind that an average 30% yearly increase in bandwidth on individual connections does not translate into the equivalent network wide bandwidth increase. Also noteworthy is that at some point the network latency (speed of light) starts dominating the overall network propagation.


text:
How can I send thousands of small amounts (10'000 sat.) to bitcoin wallets without paying to much fees?
I want to make a faucet. But my problem is that I don't know how I can make my payouts. I would have a lot of little amounts (like 10'000 satoshis) to pay.
Is there an efficient way of doing thousands of those transactions without paying to much fees?
Do I have to make every transaction manually (in Python or in Java with the Blockchain API)?
Thanks for the answers.

Note that the Bitcoin protocol is intentionally designed to make it expensive to do this - transactions are a scarce resource and making lots of them is going to cost you in fees, one way or the other.  Given the current controversies over block size and its effects, you might reconsider whether another new service to create huge numbers of small transactions is really in the best interests of the Bitcoin world right now.

This is a classic issue with faucets and thus there are services that can help you with this. E.g. check this
Behind the scenes those services use send_many as you already found out.

Thanks for all the visitors, which have thought about that.
I found now the solution for that problem. I use the blockchain.info API. There is an option to send to many addresses at the same time.
https://github.com/blockchain/api-v1-client-python/blob/master/docs/wallet.md

I am not sure that blockchain.info is the cheapest and easyest way to solve your problem. There are some restrictions, for example maximum number of outputs in one transaction is limited to 20 (or 40? - I do not remember).


text:
QR code on my phone looks different
I just set up a personal wallet, and the QR code on my phone, looks different from the QR code that blockchain.info shows. When I scan, the code on blockchain.info, I get the same address. I thought the QR codes were Unique.
what is going on?
This is a new wallet, and I used the copay app, to create it.
Im not sure how to get the QR code off my phone, this is why I went to blockchain.info, so I could get an address I can print. The image from blockchain.info

Using HD wallets gives you access to a multitude of addresses. Each wallet could simply be generating a qr code for different addresses (both of which you own). Can't say for sure unless you share the qr codes.

Im not sure how to share the one on my phone, Thats why i went to blockchain.info in the firstplace.

QR codes can in fact differ even when they encode the same data, because they may be generated with different levels of error correction. See http://blog.qrstuff.com/2011/12/14/qr-code-error-correction for a pretty readable explanation.


text:
Stellar UTF-8 Support
We are developing the open government auctions system on “Stellar” platform.  We forked Stellar and run our own network. We’ve added Lot creation transactions, auction participation, bids etc. There is encoding problem. We tried this solution:  https://github.com/stellar/js-stellar-base/issues/60 But during the debug process we’ve find out that Stellar-core can’t process the data sent by js-xdr library in base64 (UTF-8) format. We’ve looked at the C++ source code but haven’t found the way to fix that.
Error example:
{\"exception\"=>\"Non-zero padding bytes encountered\"}"
Some help and/or advice on this issue will be much appreciated.
All source are available here: http://www.eauction.idf.solutions/

When you say "on Stellar platform", do you mean the software or the network? That is, are you expecting to need no code changes or have your code changes accepted by the network? Are are you writing your own software (based on Stellar's software) to run your own network?

Yes, we forked Stellar and run our own network. All source are available here: http://www.eauction.idf.solutions .

Can you post the xdr blob that is causing the issue?

Thank you for answer. Our mistake.
This commit solve the problem: https://github.com/stellar/js-xdr/pull/4/commits/119a8c26f71de8568d88cee1a94cb33074797859

The problem was in count of characters encoded in tx. C++ code allocate memory depending of count variable in tx.
This commit solve the problem: https://github.com/stellar/js-xdr/pull/4/commits/119a8c26f71de8568d88cee1a94cb33074797859


text:
Is transaction malleability the only consensus issue that needs fix to allow lightning network?
Is transaction malleability the only thing needs fix in the bitcoin consensus code to enable lightning network to be implemented?
Of course, lightning network itself would need to be implemented, but I'm wondering if after fixing transaction malleability, any changes to bitcoin itself is required.
In particular, if BIP 62 is implemented, would that be enough?

BIP62 has been scrapped and will never be implemented. Instead a much better solution with many more advantages, Seggregated Wittness, has been developed and is currently being rolled out. Its finished code is in 0.13, but not activated yet on main net, that will happen in 0.13.1.
As far as I know, yes, this is the last thing required to enable LN. From what I've seen LN was actually possible without malleability fix (as proposed in the original LN whitepaper) but it would have been so much more complex and less efficient that it was well worth fixing it in Bitcoin first, which was the plan anyway even before LN was proposed.


text:
What does a merchant need to do to reject RBF transactions
I understand replace-by-fee transactions are no threat to merchants who require multiple confirmations. However many businesses still accept 0 confirmation transactions as a means of improving usability and help with marketing. I expect demand for 0 confirmation transactions to continue (at least for transactions with a value below a certain economic threshold). What is the easiest way to automatically reject all RBF transactions?

Replace-by-fee (RBF) transactions all have a sequence number that is below MAX - 1.  If the payment a merchant receives comes from a transaction that has such a sequence number, the merchant can either wait for the transaction to be included in a block, or ignore the payment altogether.  Ignoring the payment is fraught with other complications, however, since the bitcoin will still go to the merchant if included in a block.  The customer would have to replace the transaction, using RBF, before it is mined.  It's probably easier for most merchants to simply wait for the first confirmation.


text:
Bitcoin Core telling me transaction is not "trusted"
I'm running a full node without the GUI, so I'm controlling it with bitcoin-cli.
I sent bitcoin to another address, same as I always do, but this time I can't pull up the transaction on any of the block explorers and when I use bitcoin-cli gettransaction ad2dfbd06b15f0721c2c1864e4cf0989833727c6eec038472c2a2789f4b5d024, it comes back with:
  "amount": -0.01949732,
  "fee": -0.00007460,
  "confirmations": 0,
  "trusted": false,
  "txid": "ad2dfbd06b15f0721c2c1864e4cf0989833727c6eec038472c2a2789f4b5d024",
  "walletconflicts": [
  ],
  "time": 1479664206,
  "timereceived": 1479664206,
  "bip125-replaceable": "unknown"

I've never seen "trusted": false on any other transaction, so I assume that has something to do with it, but I can't find any information online to tell me what it means.
EDIT:
Pasting the hex format of the raw transaction into blockchain.info/pushtx and pushing it that way has set it to trusted: true and now it's showing up. I'd still like to know why it happened and/or how I can fix it myself without relying on blockchain.info's pushtx tool.

This field is discussed in the 0.12.0 changelog, see under "Negative confirmations".  I'm not sure what "coming from ourself" means, though.  The code that implements it is in CWalletTx::IsTrusted() in wallet.cpp.

@NateEldredge Thanks! I'm still mostly clueless but at least now I have a new lead to look at

@NateEldredge: "Coming from ourself" means that your wallet controls the private keys corresponding to the transaction inputs.

A transaction will be distrusted if

it conflicts with a transaction already in the blockchain, or
it is from someone else, and it is not confirmed.

A transaction will be trusted if

it has confirmations.

If a transaction is not trusted, the client won't create transactions that spend it. (You wouldn't want a transaction you created to be in limbo because one of its inputs wasn't confirmed.)
Once that transaction is confirmed, the trusted field in listtransactions will be true.

But this IS a transaction I created and it's currently in limbo.

https://blockchain.info/search/ad2dfbd06b15f0721c2c1864e4cf0989833727c6eec038472c2a2789f4b5d024

It can also show as nontrusted if the transaction isn't in your node's memory pool.

But if I sent it, why would it not be in my memory pool? Or how would I check if its in the mempool? And how would I add it?

Hmm. Could you try getrawtransaction <txhash> followed by sendrawtransaction <tx> ?

$ bitcoin-cli getrawtransaction ad2dfbd06b15f0721c2c1864e4cf0989833727c6eec038472c2a2789f4b5d024 error code: -5 error message: No information available about transaction


text:
Transaction not found in Blockchain
Trying to send a transaction via bitcoin-cli. It seems to be working on my end but if I look up the transaction ID 76f2b87d7fbd18a217626761cdea215b66d523a3471e446c4fb0147645647866 on any block explorers, it tells me not found.
I've tried to push it using https://blockchain.info/pushtx, but again I get a not found error. Doing the same on blockr.io gives me Did you sign your transaction? Is this double spend? Have you already sent this transaction?
I'm not sure how to debug this or fix it. Can someone point me in the right direction?
Hex representation:

0100000003de8f7664d7f5d27681222e1b555d0e83fe33ad307083111cee4314c800186bbe000000006b483045022100d5678c645a1f939fddc01521ccbc983a7235bb3b6c9a58e07ee5303fe12b72080220489a4cb65f8466b8482e3d4fddd56c94c182397e0cb341ccaaa026716c00614201210216bb4c25989444f1688ef4b265f76a4c451e81392a0ad19637f77da01b55d03dfeffffffa37d8ae3985551635430f713c56c26295e77eed62a90b7b0a4cde4e77041ed03010000006b4830450221009307f571405ab3eff97bc299be7191a868de1fe291cc4edeb6f4e40750781c6202202275a9ffe4d70d9cfbe9cb9b94673aeffea4c65dbef1cf780ec122c74c78c8a80121025a150650d185dcf04b118c747cb3f48858f292fb35597ef1d6fcd0239d5834dbfeffffff725bfa3b94d4a737611df4977d92b1b3e5b9fb70ac3fcd388d6bae863e7ead1f010000006a47304402202694969e973220fa032b3609308a1e9eef2a8bee3049f9621167b499a78c86370220367552a5c6b3cd71dfcd1f4653cea65f9ff7b10c506ce020c7c9481c8204376701210371dd7ada759d1a808862c5c60dac7ecb39debab8bcd9fc5d336f65a21568feeefeffffff024d520100000000001976a914bf37c92070173662685e197b927449949a8092d688ac49480f00000000001976a914ace8b01ab9dc31091ebde277507f7d04d31518b388ac39e60600

What software package and version created this transaction?

Sending this transaction through my own Bitcoin Core 0.13.99 node:
error message:
64: too-long-mempool-chain

This means you're building a long chain of unconfirmed transactions that depend on each other. Wait until the first few of those have confirmed before creating more.

Will it automatically enter the chain once that happens or will I need to rebroadcast it? How can I get the transaction ids it's depending on?

Also, which command do you use to send that? If I do the same, will it be the same as using blockchain.info's pushtx tool?

It depends, if it's a transaction relevant to your wallet, and your node is running, it will automatically rebroadcast as needed (the upcoming 0.14 release of Bitcoin Core improves upon this). If not, you'll need to manually rebroadcast with sendrawtransaction (which is what I used to broadcast). Don't rely on blockchain.info, it's perpetually outdated.

bookmark this link for submitting transactions: https://www.smartbit.com.au/txs/pushtx

@Rob Anyone can submit the transaction now that you've made it public. It's almost certain that someone will.

Is that problematic somehow?


text:
Is there an opensource crypto-currency exchanges engine?
I'm looking for an opensource crypto-currency exchange engine.
My preference would be for something easy to use like joomla or drupal. My exchange setup needs support for alt-coins and bitcoin.
I have discovered intersango myself. However, sql file is corrupted, there is no creation of tables and import does not work. Plus it is bitcoin exchange only (and I need alt-coin support). Site of this exchange with "invites only", registration is closed.
Is there something super-simple that I can use to get started?

Peatio is an open source crypto exchange. Their README says:

Peatio is a free and open-source crypto currency exchange implementation with the Rails framework and other cutting-edge technology.
Mission
Our mission is to build the world best open-source crypto currency exchange with a high performance trading engine and safety which can be trusted and enjoyed by users. Additionally we want to move the crypto currency exchange technology forward by providing support and add new features. We are helping people to build easy their own exchange around the world.

Their github page:

https://github.com/peatio/peatio

It can no longer be installed. -1 because answer is useless and time consuming. The software no longer receive updates, so the rror won’t be corrected.

The RubyKube community apparently picked up support of the Peatio code base and built a bunch on top of it. Seems still in active development https://github.com/rubykube

Yeah it's called buttercoin here is their code.

Do you know anything else, in old school style? I'm really troubled with node.js, it want so many dependencies, and wantn't simply start. I'm currently stuck with "npm ERR! Test failed. ". I'm lurking around with google, but it is really ugly. Anything simpler, ruby/django/php based?

not that I know of sorry. Trouble with npm how? install node from here and then when you download a project with a package.json file just do npm install and all the dependencies will be downloaded.

I'm on Ubuntu 13.10, npm install goes well, but test fail. Looks like buttercoin under huge development stage, there are thousands of tickets, sometimes meets speech like "Installation on the Ubuntu is broken again." I currently dance with virtualbox's sister -  vagrant up, but it too want to download something huge, and connection breaks everytime. I'm tired.

Bitsquare.io is currently the state of the art and it is open source:
https://github.com/bitsquare/bitsquare/

Yes, but adding an altcoin is ont as simple as tuning a ballet and modifying a .conf file.

Actually, adding a trading symbol is not difficult, maybe a couple of hours if you know Java and understand the blockchain.  Getting the PR commit just right, so that the team accepts it is another story...

We've just made our exchange, MarginCall, open source. Check it out. We are going with a new project where everyone can run it's exchange and interconnect them into global ECN system.
It's written in Elixir and PHP, and also includes an API and clients for that API.
Source code.
https://www.reddit.com/r/Bitcoin/comments/33y97d/our_contribution_to_bitcoin_community/

How to not be a spammer

I've read your thread on reddit yesterday,  so I am aware that your answer here is useful, however you seem to be picking up spam votes due to the way it is presented. It would be great if you could add some more info such as a direct link to the source code and the name of your project to improve your answer.

@user25253 : yes, there are no install instructions. It also looks like to require Windows® because of C#.

The code base was last updated 2016-07-02

There are few options available Like rubykube and bitHolla https://rubykube.io/
Hollaex is the exchange by bitHolla
https://coinfunda.com/best-open-source-cryptocurrency-exchange/

https://www.merkeleon.com/cryptocurrency-exchange-software/
Merkeleon provides a turnkey solution for starting own Crypto Exchange for professional traders or beginners. We provide a wide listing of crypto or fiat currencies, and the support of all required order trading types such as limit, stop-limit, market, stop-market, trailing stop, trailing stop limit, Trading API.

From what I can tell by your use of "we" it sounds like you own or work for merkeleon, am I understanding this correctly?

You can try OpenDAX
https://www.openware.com/product/opendax
It's an open-source cryptocurrency exchange with quite a lot of features, and very actively developed


text:
Byte array to hexadecimal and back again in JavaScript
I can generate a byte array with
var myByteArray = window.crypto.getRandomValues(new Uint8Array(16))

and I get
181,143,16,173,231,56,63,149,181,185,224,124,84,230,123,36

I can then turn this into a string with
cryptoHelpers.convertByteArrayToString(myByteArray);

and i get
µç8?µ¹à|Tæ{$

But what I really want is a 128 bit hexadecimal like...
6a3e52297b2e593f4d506f7164

And I want to be able to go back from hexadecimal to a byte array.
Is there a copy and paste function or library to make this magic happen?

I'm voting to close this question as off-topic because it is unrelated to Bitcoin.

You should ask this question on Stackoverflow, however search first.  I'm positive the answer will already exist.

I'm voting to close this question as off-topic because, the question, as written, does not appear related to bitcoin. Perhaps there is a relation, but if so, it should be made explicit.

Here's something you can copy-paste in.
byteArray = new Uint8Array([181,143,16,173,231,56,63,149,181,185,224,124,84,230,123,36]);
function toHexString(byteArray) {
  return Array.prototype.map.call(byteArray, function(byte) {
    return ('0' + (byte & 0xFF).toString(16)).slice(-2);
  }).join('');
}
function toByteArray(hexString) {
  var result = [];
  for (var i = 0; i < hexString.length; i += 2) {
    result.push(parseInt(hexString.substr(i, 2), 16));
  }
  return result;
}
hexString = toHexString(byteArray);
byteArray = toByteArray(hexString);

From here/here.

Using the above I'm getting a string that is not 128bits 289332005600940420022031 if I console log myByteArray i get an array buffer like "Uint8Array(16) [181, 143, 16, 173, 231, 56, 63, 149, 181, 185, 224, 124, 84, 230, 123, 36]" how do i convert this to an array?

@Will-In-China See edit.

toByteArray get wrong values when hex-string contains an odd number of characters: "fff", "1AB"

@Geograph since each byte is 2 hex digits, this is expected behaviour - a hex string with an odd number of digits is in invalid hex string.


text:
OP_RETURN Promise with bitcoinjs-lib v2.2.0 and blockchain.info
So the below is the function I'm running. I'm sending in the key for this bitcoin address
16WBguy6KVyTGnF4KX7Vmdx8ztj4wENh4W

and the message
bkawk

to this function
// Require the packages that we need.

var bitcoin = require('bitcoinjs-lib');
var request = require('request');
var blockchain = require('blockchain.info');
var blockexplorer = require('blockchain.info/blockexplorer');

// Export the module and pass in the message and the WIF
exports.opReturn = function (message, WIF) {

// Setup the new promise
return new Promise(function(resolve, reject) {

// Set the network to the main net not test
    var network = bitcoin.networks.mainnet;

 // Generate the keypair
    var keyPair = bitcoin.ECPair.fromWIF(WIF);

// Get the address from the key pair
    var address = keyPair.getAddress();

// Ask blockchain.info for all the unspent transactions
    var url = 'https://blockchain.info/id/unspent?active='+address;
    request(url, function (error, response, body) {

// Check the request didn't error and we got a 200 ok back
        if (!error && response.statusCode == 200) {

// Parse the results
            var body = JSON.parse(body);

// Build a new transaction
            var tx = new bitcoin.TransactionBuilder(network);

cont...
// Buffer the message into hex
            var data = new Buffer(message);

// Prepare the message
            var dataScript = bitcoin.script.nullDataOutput(data);

// I Suspect there is problems here!!
            var unspent = body.unspent_outputs[0].tx_hash_big_endian;

// Add the unspent tx to the output
            tx.addInput(unspent, 0);

// Add the unspent transaction to the first output
            tx.addOutput(dataScript, 0);

// Add our address to the 1st output - Must I use another address or no?
            tx.addOutput(address, 1000);

// Add our address to the 2nd output - Must I use another address here too?
            tx.addOutput(address, 1000);

// Sign the transaction
            tx.sign(0, keyPair);

// Convert to hex
            var txRaw = tx.build().toHex();

            var txId = txRaw.getId()
            console.log(txId);

// Resolve the promise
            resolve(txRaw);

// Log the hex
            console.log(txRaw.toHex());
        } else{

// if there was an error reject the promise
            reject(error);
        };
    })

}); //-- END PROMISE
};// END FUNCTION

the above console.log is giving me this transaction hex
0100000001b4c1870d4e0c6928f40224f465ab7d2c3692fe29e4bc2cc5213776ff128de13f000000006b483045022100a954cdb05753fed2f8afc41619d66dde45e330b1aa75cd5a1d1a03b67392642702203bda25690dac67bed0117fd28ad30fc2cbc16aa778a3b37832aabba9939e344001210298e836ce7df5696bdce02742bb5d7a070b2e5b1ab3535001b71f3615c21c1b1dffffffff030000000000000000076a05626b61776bb80b0000000000001976a9143c5d9d2466274014c2db74dd4738116a193d142788acb80b0000000000001976a9143c5d9d2466274014c2db74dd4738116a193d142788ac00000000

which decoded looks like this
        {
        "lock_time":0,
        "size":242,
        "inputs":[
            {
                "prev_out":{
                    "index":0,
                    "hash":"3fe18d12ff763721c52cbce429fe92362c7dab65f42402f428690c4e0d87c1b4"
                },
                "script":"483045022100a954cdb05753fed2f8afc41619d66dde45e330b1aa75cd5a1d1a03b67392642702203bda25690dac67bed0117fd28ad30fc2cbc16aa778a3b37832aabba9939e344001210298e836ce7df5696bdce02742bb5d7a070b2e5b1ab3535001b71f3615c21c1b1d"
            }
        ],
        "version":1,
        "vin_sz":1,
        "hash":"926eeecb1fc28a34cb8ed14e1e6175ec9012af390b232e26d3c610c345abba47",
        "vout_sz":3,
        "out":[
            {
                "script_string":"OP_RETURN 626b61776b",
                "value":0,
                "script":"6a05626b61776b"
            },
            {
                "script_string":"OP_DUP OP_HASH160 3c5d9d2466274014c2db74dd4738116a193d1427 OP_EQUALVERIFY OP_CHECKSIG",
                "address":"16WBguy6KVyTGnF4KX7Vmdx8ztj4wENh4W",
                "value":3000,
                "script":"76a9143c5d9d2466274014c2db74dd4738116a193d142788ac"
            },
            {
                "script_string":"OP_DUP OP_HASH160 3c5d9d2466274014c2db74dd4738116a193d1427 OP_EQUALVERIFY OP_CHECKSIG",
                "address":"16WBguy6KVyTGnF4KX7Vmdx8ztj4wENh4W",
                "value":3000,
                "script":"76a9143c5d9d2466274014c2db74dd4738116a193d142788ac"
            }
        ]
        }

Pushing it to https://blockchain.info/pushtx returns...
None Standard Script Output OP_RETURN 626b61776b

And pushing here http://blockr.io/tx/push results in
There was an error pushing your transaction to network!

Did you sign your transaction? Is this double spend? Have you already sent this transaction?

but then when I check it here I don't see the transaction. Any Ideas?
https://blockchain.info/address/16WBguy6KVyTGnF4KX7Vmdx8ztj4wENh4W
Here is an example of a successful transaction made from Colu
https://blockchain.info/tx/d831fa05d6bbb2e939e42764945936ee7c23b96a46c173c9e8647d9fe4f0d485?show_adv=true

You should probably edit the title to be the actual question, rather than the question's general topic.  It would also be nice if the question itself was a little more specific, rather than just "here's my code, tell me what's wrong".  Narrow your question down to a specific, identified problem, and you are more likely to get better answers.

Thank you @jestin for the kind suggestion, the answer to the problem was unknown at the time of writing and as you can see I detailed the issue and where I thought the problems may be as best as I could. Please can you make a suggestions for the better title?

Something like "Invalid transaction created with bitcoinjs-lib" might be better.  I don't think you had any reason to believe the Promise was the problem, since you were obviously resolving the raw transaction.  That word just made it confusing which part you were asking about.

the first (but not the last) error is:
your utxo 3fe1...1b4:0 has 3000 satoshi but the sum of your outputs is 6000 satoshi

Nice spotting! I have now reduced each to 1000 and now get no error from blockr but instead get....
Your transaction id is: 801d6fb0cbf920831973a3888f57fcac1fa76ba1642c804d94d3b213522dacd2

and looking it up here http://webbtc.com/tx/801d6fb0cbf920831973a3888f57fcac1fa76ba1642c804d94d3b213522dacd2.json says it cant be found

(1) webbtc does not show unconfirmed transactions (2) the fee for your transaction is very small (3) the output values of 1000 satoshi are 'dust' (4) blockchain.info does not accept/show unconfirmed transactions with op_return scriptsigs.... aaaaaand everything else is quite fine! your transaction is valid!

I just need to wait for confirmation?
The transaction is small and i guess I will need to wait a long time? how long?
1000 satoshi is commonly called "Dust" thank you and well noted.
blockchain.info I wont be using anymore

(1) it depends of your goal. usually I do not wait or even think about such small values (2) may be a hour, may be a day, may be a year (3) there is no consensus value for "dust". (4) up to you

Checking the below link it indicates that I should be paying 50-80 Satoshi per byte to get "Block 1". 1) What is "Block 1", "Block 2" etc  2) How fast could I expect block 1 to confirm?  3) 250 byte transaction x 80 Satoshi is 20,000 Satoshi and today that 0.11 USD is that right?
http://p2sh.info/dashboard/db/fee-estimation

80 satoshi per byte is too big for testing. 10 satoshi per byte or even 5 satoshi per byte is quite enough nowadays


text:
Is this how I should calculate a Double SHA256
So I have a function called sha256 which will take a string and return the SHA-256 hash.
var First = sha256('myfirstSHA');
And the output is the hex:
9b2b95b24dd9149480ebda21aafe3f1a3c0370798ceec3c4d09c6a16adfe01c8
All feels good here, but if I want a double SHA-256 how should I go abut calculating it? Can I just use the same function as above twice? like this...
var Second = sha256(sha256('myfirstSHA'));
What should the value of Second be?
I get the feeling I may need to get the value of the first hash and convert it to something before putting it back through the sha-256 function a second time??
So my question is, what should I convert it to, before feeding it back in again?
Any advice is most welcome!

Yes you should convert your first output before feeding it back in: A hash function is typically a function which takes in an array of bytes (of arbitrary size) and spits out an array of bytes (of fixed size). When making the first call first = sha256('myfirstSHA'), it is likely that the string argument is implicitly converted to an array of bytes, whereby each character is substituted by a single byte encoding. However, the string output you are getting (which is a string, not an array of bytes), clearly represents a hex encoding of a 32 bytes array. You need to convert this hex encoding into a 32 bytes array. If you have a Linux terminal, as indicated by this post, a very useful command is xxd -r -p which converts your hex encoding into actual bytes:
$ echo -n myfirstSHA | sha256sum
9b2b95b24dd9149480ebda21aafe3f1a3c0370798ceec3c4d09c6a16adfe01c8 -
$ echo -n myfirstSHA | sha256sum | xxd -r -p | sha256sum
96082208e341446bb8ba032486d142cbe73f1a66276b96c18ff815f31293fe0d -

If you are worried about the trailing '-' which appears in the output of sha256sum, you can cut it out:
$ echo -n myfirstSHA | sha256sum | cut -d' ' -f1
9b2b95b24dd9149480ebda21aafe3f1a3c0370798ceec3c4d09c6a16adfe01c8

and then proceed as before:
$ echo -n myfirstSHA | sha256sum | cut -d' ' -f1 | xxd -r -p | sha256sum
96082208e341446bb8ba032486d142cbe73f1a66276b96c18ff815f31293fe0d  -

In case you do not trust that xxd -r -p does the right thing, you can store the output in a file:
$ echo -n myfirstSHA | sha256sum | xxd -r -p > temp

then use hexdump -C temp or indeed xxd temp to check the content of the binary file:
00000000  9b 2b 95 b2 4d d9 14 94  80 eb da 21 aa fe 3f 1a  |.+..M......!..?.|
00000010  3c 03 70 79 8c ee c3 c4  d0 9c 6a 16 ad fe 01 c8  |<.py......j.....|
00000020

My answer from this thread should help.
Had a lot of help from Andrew Chow..
I assume you already know the whole endian swapping mess in bitcoin...
Hope this helps you or anyone else.
Cheers!!!


text:
How a private key can be invalid?
From here, we can read that :

A private key is a 256-bit value, meaning there are approximately
  1.1579e77 possible keys (There are about 1.2288e66 invalid values

But here, we have :

Nearly every 256-bit number is a valid ECDSA private key.
  Specifically, any 256-bit number from 0x1 to 0xFFFF FFFF FFFF FFFF
  FFFF FFFF FFFF FFFE BAAE DCE6 AF48 A03B BFD2 5E8C D036 4140 is a valid
  private key.

Does "nearly every" mean (1.1579e77-1.2288e66) out of 1.1579e77 ?

The answer is right there in the passage you quoted.  "Any 256-bit number from 0x1 to 0xFFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFE BAAE DCE6 AF48 A03B BFD2 5E8C D036 4140 is a valid private key."  So for instance, 0x0 and 0xFFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFE BAAE DCE6 AF48 A03B BFD2 5E8C D036 4141 are not valid private keys.  You can read https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm to see why; the key must be less than n, the size of the group.

The count of invalid on reddit is wrong, it should be about 4.32420e38. But either of these is small compared to 1.15972e77, hence 'nearly every'.

A bitcoin private key is a number greater than or equal to 1 and less than X, where:
X = 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEBAAEDCE6AF48A03BBFD25E8CD0364141

That's in base 16. In normal base 10 numbers, that's:
X = 115792089237316195423570985008687907852837564279074904382605163141518161494337

Any number outside of that range is an invalid private key. That includes 0 and X, X+1, X+2, ...

It is true that any number outside that range is invalid. The question is more about numbers inside the range. Is it possible to find, or are there any known number inside the range that is invalid?

@XRSC All numbers in that range are valid. The quote in your question is alluding to the fact that there are some numbers that fit within 256 bits that are greater than or equal to X still, so not every 256 bit number is a valid private key. X itself, for example.

"1.2288e66 invalid values" is wrong
  FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
- EBAAEDCE6AF48A03BBFD25E8CD0364140
= 14551231950B75FC4402DA1732FC9BEBF (base16)
= 432 420 386 565 659 656 852 420 866 394 968 145 599 (base10) +1 (Key 0x00)
= 432 420 386 565 659 656 852 420 866 394 968 145 600
= 4.3242e38 invalid values

  1,1579208923731619542357098500869e+77 max values (2^256)
- 4,3242038656565965685242086639497e+38 invalid values
= 1,1579208923731619542357098500869e+77 valid values :D (change not visible)


text:
How private key is generated?
I read here that:

If you could process one trillion private keys per second, it would
  take more than one million times the age of the universe to count them
  all. Even worse, just enumerating these keys would consume more than
  the total energy output of the sun for 32 years.

However this website http://directory.io says that they generated all private keys. Who is right? or what is it that I misunderstand ?
And why this (from here)

A private key is a 256-bit value, meaning there are approximately
  1.1579e77 possible keys (There are about 1.2288e66 invalid values...

related: How to find a specific bitcoin address from all the addresses?

That site is kind of a joke to those who know about cryptography, but can be confusing to newcomers.  The thing to remember is that you can't search by public key or Bitcoin address.  If you could, then the site's claim would be real.  Luckily, you can't, and it's not ;)  Just read their FAQ:  http://directory.io/faq

Ok, thank you very much @Jestin, I should've read more on the website.

They never claimed to have generated all private keys, only the ability to generate any possible private key.
The web site dynamically calculates a set of keys for a given page number. It does not store every possible key.
The web site cannot be used to discover a private key for a particular address because there is no way to search for an address.

Thank you very much. I edited the question with another sub question.

Directory.io is a parlor trick and nothing more. It simply generates keys from the beginning of the private key spectrum as the user browses through the paginated "directory".  You could generate keys magnitudes of orders faster using a vanity address generator.  A private key is basically a 256 bit number, or 256 1's and 0's.  If 7 billion people (nearly everyone on earth) started generating 1 million keys per second for 100 years that would only be 0.000000000006 of the total space.  So you can rest assured as long as keys are generated randomly no two parties should ever generate a colliding key.


text:
Proof-of-Stake: How to ensure immutability of the blockchain?
In Proof-of-Work, a miner tries to solve PoW puzzle based on the hash of previous block, such that in case of changing block B_i, all blocks after that (blocks B_i+1, B_i+2, ...) are no longer valid since puzzle of every block has been solved based on the hash of previous block.
Now, in a scheme without mining consensus such as Proof-of-Stake, does a validator have to use/insert the hash of previous block, B_i, in the new block, B_i+1 ?
If yes, How ?
And if No, how are blocks chained such that in case of changing a block, B_i, all blocks after this modified block will become invalid? We would need to chain blocks to each other to ensure immutability of the data stored in the blockchain.


text:
What is the dust limit on Bitcoin Cash transactions?
I understand from another question here on Stack Exchange that the dust limit for Bitcoin transactions is 546 satoshi, meaning that no transaction output may be on an amount less than that. What is the equivalent dust limit for Bitcoin Cash transactions?

I usually think about the dust level in units of sat/Kb.

Bitcoin-ABC (the reference implementation of Bitcoin Cash) is a fork of bitcoin core, there are no big changes to the settings.
so bitcoin cash dust limit is exactly the same as Bitcoin which is 546 Satoshi.
To make sure you can review Bitcoin-cash-ABC/transactions_test/ file code on GitHub line 546
And on transactions file here
/**
 * "Dust" is defined in terms of CTransaction::minRelayTxFee, which has
 * units satoshis-per-kilobyte. If you'd pay more than 1/3 in fees to
 * spend something, then we consider it dust. A typical spendable
 * non-segwit txout is 34 bytes big, and will need a CTxIn of at least
 * 148 bytes to spend: so dust is a spendable txout less than
 * 546*minRelayTxFee/1000 (in satoshis). A typical spendable segwit
 * txout is 31 bytes big, and will need a CTxIn of at least 67 bytes to
 * spend: so dust is a spendable txout less than 294*minRelayTxFee/1000
 * (in satoshis).
 */

Can you provide some sources for this answer somehow?

Yeah, I've edited the answer!

I didn't know the dust limit was different on segwit transactions but it makes sense based on the comments from the code.

How weird is it that this information was on line 546?