text:
Ledger Nono S (ledgerjs) signP2SHTransaction is Producing Invalid Signature
I was receiving Errors while using signatures produced by signP2SHTransaction function of @ledgerhq/hw-app-btc. So I created a script using bitcore-lib to cross check it against the signature produced by Ledger and pinpoint the change causing the Error. I also verified all my input params and found them to be correct.
I will detail both scripts here, though I think it is an issue of ledger's signP2SHTransaction function.
I'm using the following dependencies and node v8.9.3:
"@ledgerhq/hw-app-btc": "^4.17.0",
"@ledgerhq/hw-app-eth": "^4.19.0",
"@ledgerhq/hw-transport-node-hid": "^4.18.0",
"babel-runtime": "^6.26.0",
"bip32": "^0.1.0",
"bitcoinjs-lib": "^3.3.2",
"bitcore-lib": "^0.15.0"

I have created a 2 of 2 Multisig address using the following 2 paths of my ledger- 48'/0'/0'/69/0/0, 48'/0'/0'/96/0/0.
I used the following rawTx:
01000000016f4fbe65fe5fcb98028d67172f72bdeadc1f45cb49c50f2eb7aca4668e94d50a01000000490047522102a9d50f9817a9cf20f3feb7ad4038e88c8bd471e90dfba3a80c2e0bfd79c893122102faf805ea3652cec322dda6f7571d926f359d8abbd73af1512924151edbec90e752aeffffffff0220d613000000000017a9148eaba4fd80f515c78ddbc2509538e37c40ffcf1287904a96070000000017a914c45f1d5dde5c0f7008dd6c228c1702cfdafdf1a98700000000

I used the following Redeem Script:
522102a9d50f9817a9cf20f3feb7ad4038e88c8bd471e90dfba3a80c2e0bfd79c893122102faf805ea3652cec322dda6f7571d926f359d8abbd73af1512924151edbec90e752ae

Here's the code I used to produce it:
const bitcore = require("bitcore-lib");
const PublicKey = bitcore.PublicKey;
const Script = bitcore.Script;

var publicKey1 = new PublicKey(
  "02a9d50f9817a9cf20f3feb7ad4038e88c8bd471e90dfba3a80c2e0bfd79c89312"
); // public key for path- 48'/0'/0'/69/0/0

var publicKey = new PublicKey(
  "02faf805ea3652cec322dda6f7571d926f359d8abbd73af1512924151edbec90e7"
); // public key for path- 48'/0'/0'/96/0/0

var pubkeys = [publicKey, publicKey1];

var redeemScript = Script.buildMultisigOut(pubkeys, 2);

console.log(redeemScript.toHex());

You can decode it here: https://live.blockcypher.com/btc/decodetx/
Here is my ledger's code which is creating a signature for 48'/0'/0'/69/0/0 path:
const TransportHid = require("@ledgerhq/hw-transport-node-hid").default;
const AppBtc = require("@ledgerhq/hw-app-btc").default;

TransportHid.create()
  .then(async transport => {
    if (!transport) return console.log("err: Unable to establish connection");
    var btc = new AppBtc(transport);

    const rawTx = "01000000016f4fbe65fe5fcb98028d67172f72bdeadc1f45cb49c50f2eb7aca4668e94d50a01000000490047522102a9d50f9817a9cf20f3feb7ad4038e88c8bd471e90dfba3a80c2e0bfd79c893122102faf805ea3652cec322dda6f7571d926f359d8abbd73af1512924151edbec90e752aeffffffff0220d613000000000017a9148eaba4fd80f515c78ddbc2509538e37c40ffcf1287904a96070000000017a914c45f1d5dde5c0f7008dd6c228c1702cfdafdf1a98700000000";
    const redeemHex =
      "522102a9d50f9817a9cf20f3feb7ad4038e88c8bd471e90dfba3a80c2e0bfd79c893122102faf805ea3652cec322dda6f7571d926f359d8abbd73af1512924151edbec90e752ae";
    const bufferedData = await btc.splitTransaction(rawTx);

    let input = [];

    input.push(bufferedData);
    input.push(1);
    input.push(redeemHex);
    const outputScript = btc
      .serializeTransactionOutputs(bufferedData)
      .toString("hex");

    console.log("\nOutput script hex:", outputScript);

    const accountIndex = 69;

    console.log("\npath:", `48'/0'/0'/${accountIndex}/0/0`);

    await btc
      .signP2SHTransaction(
        [input],
        [`48'/0'/0'/${accountIndex}/0/0`],
        outputScript
      )
      .then(sig => console.log("\n\nSig hash:", sig))
      .catch(err => console.log("\n\nErr:", err));
  })
  .catch(err => console.log(err));

It returns the following Signature when I pass the RedeemScript with the tx:
Sig hash: [ '304402203e24b5ad68c1fe3bf55a11afb6a61e3525c6f0a2780a0ee4bf37401bfd1445ff02207b18b7277b4f2625d04d0fcfd796300a9f6923c1df11a96876ab094138ea2a94' ]

And I find it strange that when I remove the redeem script param of signP2SHTransaction function (which is labelled optional), I get a different signature:
Sig hash: [ '3045022100f40f8fa2b75196a50b2f2543a06a3ed3ed79814cd29510f1225a359620b1c19102201af093addc93fce4fe8c59ad127c42064c2840ec7c71fbf5d0bae7ee5cb4cd39' ]

But both are unable to sign the transaction in a proper manner and it error while I validate it against the transaction.
I used the following script to find a valid signature using bitcore-lib:
const bitcoin = require("bitcoinjs-lib");
const bitcore = require("bitcore-lib");

// These are the private keys of paths- 48'/0'/0'/69/0/0 and 48'/0'/0'/96/0/0 of my ledger
// I retrieved them using https://iancoleman.io/bip39/ and my ledger's mnemonic

const privateKeys = [
  new bitcore.PrivateKey(
    "cNKAjjSL5buaP6q7fE375jkt72JAvvoe8RVh2v5TXv6gdjzXwPVX",
    "testnet"
  ),
  new bitcore.PrivateKey(
    "cMmNVwdfid1FnT4LjH4SJ1mZvTEGnMfUxdasGKTrHvD5nCY1UCvR",
    "testnet"
  )
];

const publicKeys = privateKeys.map(bitcore.PublicKey, bitcoin.networks.testnet);
const address = new bitcore.Address(publicKeys, 2); // 2 of 2

console.log("\n\nCreated this Address:", address);
console.log("\n\nPublic keys:", publicKeys);

// This utxo will create the same rawTx as I have used in my code with Ledger
const utxo = {
  address: "2NB9YNZwwKXannuZryo2KfvMNe4jeSNcSp5",
  txid: "0ad5948e66a4acb72e0fc549cb451fdceabd722f17678d0298cb5ffe65be4f6f",
  vout: 1,
  scriptPubKey: "a914c45f1d5dde5c0f7008dd6c228c1702cfdafdf1a987",
  // "script" : new bitcore.Script(address).toHex(),
  satoshis: 128600000
};

const fee = 10000;

const tx = new bitcore.Transaction()
  .from(utxo, publicKeys, 2)
  .fee(fee)
  .to("2N6FbZbJsGHWRpnbu8vrowCfGATKsYxuDf9", 1300000)
  .change("2NB9YNZwwKXannuZryo2KfvMNe4jeSNcSp5");

const txObj = tx.toObject();
console.log("\n\nTransaction object:", txObj);
console.log("\n\nTransaction hash:", tx);

const signature1 = tx.getSignatures(privateKeys[0])[0];
console.log("\n\nSignature1 object:", signature1.toObject());
console.log("\n\nSignature1 hash:", signature1.signature.toString()); // Outputs a DER signature1
console.log("\n\nSignature1 type:", signature1.sigtype);

console.log("\n\nIs valid tx 1:", tx.isValidSignature(signature1));
if (!tx.isValidSignature(signature1)) throw "Not a valid Signature";

It produces the following valid signature:
Signature1 hash: 304402203fb366ffd2840a900abc7ed25e945e4fdcf37679870a6ee45ce0030dd725856e02202d3b1c86458121cf4b79382281eca074a56d8b67a86dc754e2f7c65501f75b0b

I think there is a bug in the current release of the Ledger, can you please help me.
Solution:
I was creating a wrong input for my Ledger, I had to use utxos to create the input tx but I used the final rawtx (which I was creating using the texts). Those stuck with this issue, keep the following in mind:
In Ledger's documentation of signP2SHTransaction function-
http://ledgerhq.github.io/ledgerjs/docs/#btc
 - inputs: refer to the utxos you are consuming to form your tx
 - outputScriptHex: can be used to specify all the desired output params like fees, change address, etc.

What is bitcoin address for this derivation do you have?

The address for the ledger's script is same as the one derived in my Bitcore script, i.e. 2NB9YNZwwKXannuZryo2KfvMNe4jeSNcSp5

hi, did you succeed doing this?
since I noticed that you also using p2sh(p2ms()) address, but I didn't found a way to sign a p2sh(p2ms()) address with ledger.
the example in the hw-app-btc's test, only show how to sign p2sh(p2wsh(p2ms())).
if you successfully do that, may I look at your snippet?

(this is too long for a comment, so here in the answer section):
My observation: I tried to verify the three sigs at the command line (with OpenSSL). I can see the two pubkeys from your first code fragment, and the tx. I created the raw, unsigned tx, and only the third signature is ok (with pubkey 02a9d50f9817a9...) - I assume the raw tx in your first code example is somehow incorrect. I used this unsigned, raw tx (with redeemscript and 0x47 as length):
01000000016f4fbe65fe5fcb98028d67172f72bdeadc1f45cb49c50f2eb7aca4668e94d50a0100000047522102a9d50f9817a9cf20f3feb7ad4038e88c8bd471e90dfba3a80c2e0bfd79c893122102faf805ea3652cec322dda6f7571d926f359d8abbd73af1512924151edbec90e752aeffffffff0220d613000000000017a9148eaba4fd80f515c78ddbc2509538e37c40ffcf1287904a96070000000017a914c45f1d5dde5c0f7008dd6c228c1702cfdafdf1a9870000000001000000

which is double sha256'd this:
bad8b7f175a67e1ecfd857119609646993eb026b2bb507a49ce47b9ba5d321f7

What unsigned raw tx do you use to sign in your code fragment(s)? Do they have the same double sha256'd hash?
I see your raw tx having "[version][tx_in count][prev tx id][prev outpoint]490047[redeem script]...", why is this "4900"?
(no need to reply again, you may append to your question by editing it)

Yes, you were right, I was creating a wrong input for my Ledger, I had to use utxos to create the input tx but I used the final rawtx (which I was creating using the utxos).

Just signed your tx
Transaction succesfully broadcast.
Transaction id: 2ef82827b3e80f34a95fd7b9268a1cd6dc81447822927225f310fa9fd573b176
Please read this article with an explanation how to sign p2sh-multisig
https://medium.com/@support_62391/exploring-bitcoin-signing-p2sh-input-2dde869c5f5c
>>> from pybtc import *
>>> tx = Transaction("01000000016f4fbe65fe5fcb98028d67172f72bdeadc1f45cb49c50f2eb7aca4668e94d50a01000000490047522102a9d50f9817a9cf20f3feb7ad4038e88c8bd471e90dfba3a80c2e0bfd79c893122102faf805ea3652cec322dda6f7571d926f359d8abbd73af1512924151edbec90e752aeffffffff0220d613000000000017a9148eaba4fd80f515c78ddbc2509538e37c40ffcf1287904a96070000000017a914c45f1d5dde5c0f7008dd6c228c1702cfdafdf1a98700000000", testnet=1)
>>> import pprint
>>> pprint.pprint(tx)
{'amount': 128590000,
 'bSize': 188,
 'blockHash': None,
 'blockIndex': None,
 'blockTime': None,
 'coinbase': False,
 'confirmations': None,
 'data': None,
 'fee': None,
 'format': 'decoded',
 'hash':     'bb788dbf706a4e6a2a4fb3d1f19852773be71e98b5819b57b45e6d8c1e341034',
 'lockTime': 0,
 'rawTx':     '01000000016f4fbe65fe5fcb98028d67172f72bdeadc1f45cb49c50f2eb7aca4668e94d50a01000000490047522102a9d50f9817a9cf20f3feb7ad4038e88c8bd471e90dfba3a80c2e0bfd79c893122102faf805ea3652cec322dda6f7571d926f359d8abbd73af1512924151edbec90e752aeffffffff0220d613000000000017a9148eaba4fd80f515c78ddbc2509538e37c40ffcf1287904a96070000000017a914c45f1d5dde5c0f7008dd6c228c1702cfdafdf1a98700000000',
 'segwit': False,
 'size': 188,
 'testnet': 1,
 'time': None,
 'txId': 'bb788dbf706a4e6a2a4fb3d1f19852773be71e98b5819b57b45e6d8c1e341034',
 'vIn': {0: {'scriptSig':     '0047522102a9d50f9817a9cf20f3feb7ad4038e88c8bd471e90dfba3a80c2e0bfd79c893122102faf805ea3652cec322dda6f7571d926f359d8abbd73af1512924151edbec90e752ae',
         'scriptSigAsm': 'OP_0 '
                         '522102a9d50f9817a9cf20f3feb7ad4038e88c8bd471e90dfba3a80c2e0bfd79c893122102faf805ea3652cec322dda6f7571d926f359d8abbd73af1512924151edbec90e752ae',
         'scriptSigOpcodes': **'OP_0 [71]'**, <<<< script sig should be empty before signing
         'sequence': 4294967295,
         'txId': '0ad5948e66a4acb72e0fc549cb451fdceabd722f17678d0298cb5ffe65be4f6f',
         'vOut': 1}},
 'vOut': {0: {'address': '2N6FbZbJsGHWRpnbu8vrowCfGATKsYxuDf9',
          'addressHash': '8eaba4fd80f515c78ddbc2509538e37c40ffcf12',
          'nType': 1,
          'reqSigs': None,
          'scriptPubKey': 'a9148eaba4fd80f515c78ddbc2509538e37c40ffcf1287',
          'scriptPubKeyAsm': 'OP_HASH160 '
                             '8eaba4fd80f515c78ddbc2509538e37c40ffcf12 '
                             'OP_EQUAL',
          'scriptPubKeyOpcodes': 'OP_HASH160 [20] OP_EQUAL',
          'type': 'P2SH',
          'value': 1300000},
      1: {'address': '2NB9YNZwwKXannuZryo2KfvMNe4jeSNcSp5',
          'addressHash': 'c45f1d5dde5c0f7008dd6c228c1702cfdafdf1a9',
          'nType': 1,
          'reqSigs': None,
          'scriptPubKey': 'a914c45f1d5dde5c0f7008dd6c228c1702cfdafdf1a987',
          'scriptPubKeyAsm': 'OP_HASH160 '
                             'c45f1d5dde5c0f7008dd6c228c1702cfdafdf1a9 '
                             'OP_EQUAL',
          'scriptPubKeyOpcodes': 'OP_HASH160 [20] OP_EQUAL',
          'type': 'P2SH',
          'value': 127290000}},
 'vSize': 188,
 'version': 1,
 'weight': 752}

>>> tx["vIn"][0]["scriptSig"]=""
>>> tx.sign_input(0, private_key=["cNKAjjSL5buaP6q7fE375jkt72JAvvoe8RVh2v5TXv6gdjzXwPVX", "cMmNVwdfid1FnT4LjH4SJ1mZvTEGnMfUxdasGKTrHvD5nCY1UCvR"], redeem_script="522102a9d50f9817a9cf20f3feb7ad4038e88c8bd471e90dfba3a80c2e0bfd79c893122102faf805ea3652cec322dda6f7571d926f359d8abbd73af1512924151edbec90e752ae", witness_version=None)
{'format': 'decoded', 'testnet': 1, 'segwit': False, 'txId': '2ef82827b3e80f34a95fd7b9268a1cd6dc81447822927225f310fa9fd573b176', 'hash': '2ef82827b3e80f34a95fd7b9268a1cd6dc81447822927225f310fa9fd573b176', 'version': 1, 'size': 333, 'vSize': 333, 'bSize': 333, 'lockTime': 0, 'vIn': {0: {'txId': '0ad5948e66a4acb72e0fc549cb451fdceabd722f17678d0298cb5ffe65be4f6f', 'vOut': 1, 'scriptSig': '0047304402203fb366ffd2840a900abc7ed25e945e4fdcf37679870a6ee45ce0030dd725856e02202d3b1c86458121cf4b79382281eca074a56d8b67a86dc754e2f7c65501f75b0b01483045022100df9dfe852cd592062e8e22b5e1cad18a2a66e35a19611960fc314b7b36814ff20220404dcb809cb122a2adb8c16c507ed1be8780c08bbb89c2510775fccfa11162af0147522102a9d50f9817a9cf20f3feb7ad4038e88c8bd471e90dfba3a80c2e0bfd79c893122102faf805ea3652cec322dda6f7571d926f359d8abbd73af1512924151edbec90e752ae', 'sequence': 4294967295, 'scriptSigOpcodes': 'OP_0 [71] [72] [71]', 'scriptSigAsm': 'OP_0 304402203fb366ffd2840a900abc7ed25e945e4fdcf37679870a6ee45ce0030dd725856e02202d3b1c86458121cf4b79382281eca074a56d8b67a86dc754e2f7c65501f75b0b01 3045022100df9dfe852cd592062e8e22b5e1cad18a2a66e35a19611960fc314b7b36814ff20220404dcb809cb122a2adb8c16c507ed1be8780c08bbb89c2510775fccfa11162af01 522102a9d50f9817a9cf20f3feb7ad4038e88c8bd471e90dfba3a80c2e0bfd79c893122102faf805ea3652cec322dda6f7571d926f359d8abbd73af1512924151edbec90e752ae'}}, 'vOut': {0: {'value': 1300000, 'scriptPubKey': 'a9148eaba4fd80f515c78ddbc2509538e37c40ffcf1287', 'nType': 1, 'type': 'P2SH', 'addressHash': '8eaba4fd80f515c78ddbc2509538e37c40ffcf12', 'reqSigs': None, 'address': '2N6FbZbJsGHWRpnbu8vrowCfGATKsYxuDf9', 'scriptPubKeyOpcodes': 'OP_HASH160 [20] OP_EQUAL', 'scriptPubKeyAsm': 'OP_HASH160 8eaba4fd80f515c78ddbc2509538e37c40ffcf12 OP_EQUAL'}, 1: {'value': 127290000, 'scriptPubKey': 'a914c45f1d5dde5c0f7008dd6c228c1702cfdafdf1a987', 'nType': 1, 'type': 'P2SH', 'addressHash': 'c45f1d5dde5c0f7008dd6c228c1702cfdafdf1a9', 'reqSigs': None, 'address': '2NB9YNZwwKXannuZryo2KfvMNe4jeSNcSp5', 'scriptPubKeyOpcodes': 'OP_HASH160 [20] OP_EQUAL', 'scriptPubKeyAsm': 'OP_HASH160 c45f1d5dde5c0f7008dd6c228c1702cfdafdf1a9 OP_EQUAL'}}, 'rawTx': '01000000016f4fbe65fe5fcb98028d67172f72bdeadc1f45cb49c50f2eb7aca4668e94d50a01000000da0047304402203fb366ffd2840a900abc7ed25e945e4fdcf37679870a6ee45ce0030dd725856e02202d3b1c86458121cf4b79382281eca074a56d8b67a86dc754e2f7c65501f75b0b01483045022100df9dfe852cd592062e8e22b5e1cad18a2a66e35a19611960fc314b7b36814ff20220404dcb809cb122a2adb8c16c507ed1be8780c08bbb89c2510775fccfa11162af0147522102a9d50f9817a9cf20f3feb7ad4038e88c8bd471e90dfba3a80c2e0bfd79c893122102faf805ea3652cec322dda6f7571d926f359d8abbd73af1512924151edbec90e752aeffffffff0220d613000000000017a9148eaba4fd80f515c78ddbc2509538e37c40ffcf1287904a96070000000017a914c45f1d5dde5c0f7008dd6c228c1702cfdafdf1a98700000000', 'blockHash': None, 'confirmations': None, 'time': None, 'blockTime': None, 'blockIndex': None, 'coinbase': False, 'fee': None, 'data': None, 'amount': 128590000, 'weight': 1332}
>>>

First, your error is scriptSig should be empty before signing
your scriptSig is only 1 signature and no redeem script inside scriptSig.

I know how to create a p2sh transaction, (I had problems in doing it using my Ledger nano S. I solved it though, adding info about my solution by editing my question.

Signature1 hash: 304402203fb366ffd2840a900abc7ed25e945e4fdcf37679870a6ee45ce0030dd725856e02202d3b1c86458121cf4b79382281eca074a56d8b67a86dc754e2f7c65501f75b0b

This may seem like a really trivial question... but what kind of signature is this? This is not the raw output information you send on the wire.
In my scenario i have 2 endpoints that need to sign a transaction (normal multisignature). One endpoint is the nano ledger.. the other is a bitcoin-core node. I can get this "style" signature hash from the ledger.. but what do i need to give to the bitcoin-core node to finish signing the transaction. I have my scriptsig, vout,redeemscript, txids etc.. but i am trying to piece together what this hash is.. and how can i send it back to the bitcoin-core node to finishing signing?

try searching applySignature in this doc- https://bitcore.io/api/lib/transaction and read about it, you will understand.


text:
How do miners send blocks to network peers?
How do miners send blocks to network peers?
in array json format, Thus:
{
"hash":"000000000000000000a7b4999c723ed9f308425708577c76827ade51062e135a",
"ver":536870912,
"prev_block":"0000000000000000005ac8cd6b7f803c11a617b901a13e05318a155f8f215ea1",
"mrkl_root":"48915d048e3f3a8c19d3ab81ebdfc16278cd665637770305ae2e3cb2eaf97ccc",
"time":1512192330,
"bits":402706678,
"fee":0,
"nonce":2321456741,
"n_tx":1,
"size":266,
"block_index":1648336,
"main_chain":true,
"height":497152,
"received_time":1512192330,
"relayed_by":"0.0.0.0",
 "tx":[

 { "ver":1,
  "inputs":[
     {
        "sequence":1663,
        "witness":"",

 "script":   "030096072cfabe6d6d08d1c2f6d904f4e1cd10c6558f8e5aed5d6a89c43bb22862464ebb819dd8813404000000f09f909f104d696e6564206279206a6f73656d737200000000000000000000000000000000000000000000000000000000000000000000"
  }
   ],
  "weight":740,
  "relayed_by":"0.0.0.0",
   "out":[
    {
     "addr_tag_link":"https:\/\/www.f2pool.com",
     "addr_tag":"Discus Fish (F2Pool)",
     "spent":true,
     "tx_index":307162790,
     "type":0,
     "addr":"1KFHE7w8BhaENAswwryaoccDb6qcT6DbYY",
     "value":1250000000,
     "n":0,
          "script":"76a914c825a1ecf2a6830c4401620c3a16f1995057c2ab88ac"
  }
   ],
   "lock_time":953150447,
  "size":185,
   "rbf":true,
  "time":1512192330,
   "tx_index":307162790,
   "vin_sz":1,
  "hash":      "48915d048e3f3a8c19d3ab81ebdfc16278cd665637770305ae2e3cb2eaf97ccc",
   "vout_sz":1
 }]
 }

or hex format, Thus:
00000020a15e218f5f158a31053ea101b917a6113c807f6bcdc85a000000000000000000cc7cf9eab23c2eae050377375666cd7862c1dfeb81abd3198c3a3f8e045d91484a39225af6d00018659e5e8a0101000000010000000000000000000000000000000000000000000000000000000000000000ffffffff64030096072cfabe6d6d08d1c2f6d904f4e1cd10c6558f8e5aed5d6a89c43bb22862464ebb819dd8813404000000f09f909f104d696e6564206279206a6f73656d7372000000000000000000000000000000000000000000000000000000000000000000007f06000001807c814a000000001976a914c825a1ecf2a6830c4401620c3a16f1995057c2ab88acefebcf38

Blocks are usually sent using BIP152. This transmits blocks using an 80 byte header, 6 bytes per transation, plus the coinbase transaction.  To give a concrete example block 000000000000000000117f0033fd8cff18a9fa4601da7f93e6eb3e084caab586 took 16244 bytes to transmit.
If sending to a peer that doesn't already know the transactions (or not supporting BIP152) the node sends the raw block, which is half the size of the hex.  In the case of that block the raw block is 1,125,118 bytes long.

Neither, it's a binary format on the wire.
The 'hex' format is the same as the binary format, it just takes 2x the space.

This bitcoind JSON-RPC function attempts to submit a new block to the network:
https://bitcoincore.org/en/doc/0.16.0/rpc/mining/submitblock/

Is that correct?  {"id": 0, "method": "submitblock", "params": [
"00000020a15e218f5f158a31053ea101b917a6113c807f6bcd......", # header block
"01000000010000000000000000000000000000000000000000......" #coinbase
]}')

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How does a node get information from other nodes?
I think a node sometimes needs to get a information from other nodes.
e.g.

get set of UTXO.
get main(longest) chain.

I have two questions.

How does a node tell other nodes these requests?
Like in github, are there commands like 'pull requests'?
Where does a node get information from?
I believe blockchain is decenterlized and all information is in the hands of all nodes.There must not be centerlized database.
Then, how does a node decide who is reliable to get information from?

I wish you could understand my poor English...

get set of UTXO.

Note: nodes don't download the UTXO set directly, they download the entire blockchain and build their own UTXO set as they verify the blockchain.

How does a node tell other nodes these requests?
  Like git, are there commands like 'pull requests'?

Nodes communicate with each other directly via a P2P protocol, sending different types of messages to each other to retrieve different types of information. You can take a look here for the specifics of each message, https://bitcoin.org/en/developer-reference

Where does a node get information from?
  I believe blockchain is decenterlized and all information is in the hands of all nodes.There must not be centerlized database.
  Then, how does a node decide who is reliable to get information from?

Nodes will ask multiple neighbour nodes for information at once, rather than just trusting a single other node. It can then use the longest chain from all its neighbours, so as long as it connected to just one honest node it will be able to download the blockchain. This is very likely because the nodes it connects to are randomly selected from a set given to it by a DNS server

Your answer is exactly what I want.
That's very kind of you.
Then, can only full nodes validate UTXO, and be miner?

@mtdkki no problem, please remember to give it the green tick if it answered your question :) Correct, only full nodes validate the UTXO set, SPV clients can only verify that a particular transaction is part of the blockchain (thereby trusting the other full nodes). Usually miners will all connect to one full node rather than individually running full nodes on every miner

Nodes communicate by gossiping about new data to their peers.
For example, whenever a new block is found, a node that hears about it, will tell all its peers "Hey, I've got a new block with the identifier , you want it?". These announcement take the form of "Inventory messages". They consist of two elements: The type of data being offered (i.e. transaction, block, or filtered block), and the identifier (i.e. transaction id, or block hash).
The peer receiving the inventory announcement will then request either the block header, or the complete item if they haven't seen it before, or nothing if they know about it already. The same happens in transaction relay.
Naturally, every node checks the validity of the data they receive before using it. This way, every node ends up assembling their own verified complete copy of the blockchain. They don't trust other nodes to tell them about UTXOs or which chain is the longest, but rather they only expect that their set of peers together will tell them about any new data that floats about eventually, while only trusting the data that they have verified and collected themselves.
The chain tip and UTXO set are therefore derived information that the node has calculated themselves from verifying all available information gathered from peers.

Thank you for helping me!  In your explanation, I understood that the block chain is fully decentralized.
By the way, is TXID a hash of digest of a transaction(sender, resipient, amount?) and constitutes a merkle root?

The txid is the hash of the transaction's data. The Merkle root is the result of a binary tree built from all the transactions: https://bitcoin.stackexchange.com/a/38399/5406

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
How to push a raw transaction using BitcoinJS (and Request)?
I'm using BitcoinJS to create a playground for Bitcoin transactions.
I have created the following Javascript file:
const request = require('request');
const proxiedRequest = request.defaults({proxy: "http://localhost:3128"});
const Btc = require('bitcoinjs-lib');
const TestNet = Btc.networks.testnet;
const apiUrl = 'https://testnet.blockexplorer.com/api/addr/';

let privateKey = <my private key>;
let wallet = new Btc.ECPair.fromWIF(privateKey, TestNet);
let publicKey = wallet.getAddress();
console.log("my public key:", publicKey);

let tx = new Btc.TransactionBuilder(TestNet);

let amountWeHave = 100000000; // 1.0 BTC
let amountToKeep = 90000000; // 0.9 BTC
let transactionFee = 1000; // 0.0001 BTC
let amountToSend = amountWeHave - amountToKeep - transactionFee; // ~0.1 (0.0999)

tx.addInput(<one of my input transactions>, 0);

tx.addOutput(<destination public key>, amountToSend);
tx.addOutput(publicKey, amountToKeep);
tx.sign(0, wallet);

let tx_hex = tx.build().toHex();

proxiedRequest.post({
    headers: {'content-type' : 'multipart/mixed'},
    url:     'https://testnet.blockexplorer.com/api/tx/send',
    body:    tx_hex
}, (error, response, body) => {
    console.log('error', error);
    console.log('body', body);
});

Here is my output:
my public key: <my public key>
error null
body TypeError: Cannot read property 'toString' of undefined
    at Array.types.str (/home/bitcore/lib/bitcore-node/node_modules/bitcoind-rpc/lib/index.js:260:17)
    at RpcClient.<anonymous> (/home/bitcore/lib/bitcore-node/node_modules/bitcoind-rpc/lib/index.js:236:35)
    at Bitcoin.sendTransaction (/home/bitcore/lib/bitcore-node/lib/services/bitcoind.js:1791:15)
    at Node.self.(anonymous function) [as sendTransaction] (/home/bitcore/lib/bitcore-node/lib/node.js:223:27)
    at TxController.send (/home/bitcore/dev/insight-api/lib/transactions.js:290:13)
    at Layer.handle [as handle_request] (/home/bitcore/lib/bitcore-node/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/bitcore/lib/bitcore-node/node_modules/express/lib/router/route.js:131:13)
    at Route.dispatch (/home/bitcore/lib/bitcore-node/node_modules/express/lib/router/route.js:112:3)
    at Layer.handle [as handle_request] (/home/bitcore/lib/bitcore-node/node_modules/express/lib/router/layer.js:95:5)
    at /home/bitcore/lib/bitcore-node/node_modules/express/lib/router/index.js:277:22

Can you tell me what's wrong with my code?
It seems to be a server problem...

Body must be a string.
You can use form instead of body.
Reference:
https://www.npmjs.com/package/request#forms


text:
Blockchain explorer for regtest mode
I'm looking for a Bitcoin blockchain explorer that will work in "regtest mode", mostly for demo proposes.
Any suggestions?


text:
Is Proof-Of-Work simply a decentralized clock?
I have not seen anyone explain Proof-Of-Work this way, so I thought
I'd share it here, I'd be curious to know if I'm completely off.
It seems to me that PoW is simply a distributed decentralized source of
ticks, a clock, to which data can be "tied"?
The challenge of Proof-Of-Work is a singleton problem (for lack of a
better term), i.e. there is only one instance of it in the universe. This means that it is known to every participant, and anyone working on it is affecting the outcome.
Most importantly even if one is participating in complete secrecy until a solution is found, it still affects the outcome.
An analogy might be looking for the largest known prime. You
don't need to tell anyone you decided to find one, you only announce
when and if you actually find one. Your participation makes finding a
solution a possibility, whereas if no one ever looks for the largest
prime, it will never be found.
The difficulty algorithm adjusts in such a way that a solution is
found on average every 10 minutes. It's a distributed decentralized
clock which ticks on average once every 10 minutes. The miners make it
possible through their participation, and curiously, miners do not
need to tell anyone they are participating, they only need to
communicate with others if and when a solution is found.
The clock has an interesting property - you can tie data to its ticks,
simply by making the data the input to the SHA. A solved block is
nothing more than simply an event that happened exactly at the tick of
this clock.
And essentially this is all that Proof-Of-Work does for
the blockchain: it provides ticks of a clock, to which events can be
tied. Everything else is an "add-on" to this "feature", e.g. using the Merkle tree we can cryptographically record the ordering of the ticks. But without the "PoW clock" none of it would be possible.
Edit: these ticks do come at irregular intervals, so one wouldn't set their watch to it. But that doesn't matter for as long as it is approximately 10 minutes, which it is.
Edit: here is a much more detailed explanation: https://grisha.org/blog/2018/01/23/explaining-proof-of-work/

From Satoshi's whitepaper:

3.Timestamp Server
The solution we propose begins with a timestamp server.  A timestamp
  server works by taking a hash   of   a   block   of   items   to   be
  timestamped   and   widely   publishing   the   hash,   such   as   in
  a newspaper or Usenet post [2-5].   The timestamp proves that the data
  must have existed at the time, obviously, in order to get into the
  hash.  Each timestamp includes the previous timestamp in its hash,
  forming a chain, with each additional timestamp reinforcing the ones
  before it.
4.Proof-of-Work
To implement a distributed timestamp server on a peer-to-peer basis,
  we will need to use a proof-of-work system similar to Adam Back’s
  Hashcash ...

Thank you, marking this as the right answer. I'm a tad embarrassed that I thought I knew the paper better than I did, but gratified I worked it out on my own. Interestingly the first source code (Nov 2008) frequently mentions "timechain" in the comments (main.h in http://satoshi.nakamotoinstitute.org/code/).

Yes, PoW is a decentralized clock not just for tying information to a specific time (a timestamp can do this perfectly fine) but to prevent spam attacks on the blockchain itself.
The question comes up, how can you add data to everyone's ledger all at once continuously and efficiently? If there is a single tx at a time queue system then things will be unbearably slow.
If the ledger is open for everyone to add verified transactions then what's to stop someone from just adding transactions on transactions and filling up everyone's node?
PoW exists to solve this problem by creating something like a flood gate that allows only a set amount of transactions per time. The difficulty aspect of PoW exists to adapt to the changing nature of computing power against the network.

The clock has an interesting property - you can tie data to its ticks, simply by making the data the input to the SHA. A solved block is nothing more than simply an event that happened exactly at the tick of this clock.

This is so an attacker can't just use the same verified data to add blocks and blocks of transactions. Think of a block mined by the miner as a one time use token that let's the block be added.

An analogy might be looking for the largest known prime. You don't need to tell anyone you decided to find one, you only announce when and if you actually find one. Your participation makes finding a solution a possibility, whereas if no one ever looks for the largest prime, it will never be found.

This portion isn't entirely correct. For you to test if a block you've mined is correct (because mining is entirely a guessing game) you need to ask the network if the block is valid. More than likely the node you just asked will say no and you begin the search again.
Although I should note that the amount of failed attempts is not factored by the difficulty algorithm

"a timestamp can do this perfectly fine" - I don't see how, what is the source of it, how can it be trusted? My point is the PoW is the timestamp-providing functionality. It's not about attackers, it is that the block happened exactly (not approximately) at the tick.

Also not sure what you mean by a block being correct. You need to start out with a correct candidate block or you'd be wasting your resources. You don't need to "ask" the network anything other than the hash of the previous block. Also failed attempts are factored, the hashrate is nearly all failed attempts except for the one winner once every 10min.

PoW falls short for pure time tracking. If the hash rate were to suddenly drop 50% (as it would during a fork) sometime between difficulty adjustment, the network would slow down by approximately 50% and blocks would be mined twice as slow. It should also be noted that blocks are mined roughly every 10 minutes and not every 10 minutes on the dot. It's why predicting forks at a block height is hard.
It can be trusted just like it is in DASH/Hyperledger. By being linked to a previous transaction and validated by the network. PoW does not accomplish validation. That is done by the full nodes

@GregoryTrubetskoy take a look at the link I posted above and this link here from the wiki.
Failed attempts are not factored

May be I've not done a good job of explaining it, but your comments seem besides the point I am making. I've written up about this explanation here in many more words, perhaps this explains it better: https://grisha.org/blog/2018/01/23/explaining-proof-of-work/

You'd have to define what you mean by "failed attempt". The only role the network plays is in deciding the next block at a chain split, everything else is known. If by "failed attempt" you mean an orphaned block, then these are statistically insignificant, chain splits are common but rare. You cannot mine without a full node, BTW.


text:
**SOLVED** Failed to verify this transaction?
I came across this P2SH multisig transactions 255942b1... on the testnet chain that I failed to verify.
I just couldn't find any matching signatures and public keys in the list provided and there seems to be an empty public key(OP_0 in the pub key script) given in the redeem script.
The scripts I found was
sig script:
OP_0
3046022100a6ad0304388a4df35b444cb96ffee0b393700a6a8612f1e35cc3dc131314f893022100bcc8b2849b04dee59dcedc4fb21a96dda3fd067a381522f37a2b6813c4e1581d01
30450220371b8ee29370e0d23f547163e3c238a40fe279e8edb2b838811cfca7bf5778e0022100f7e81c88ff7ea1c8da2666cae90c6a3c73a6817070c657068c678254fdb0b3f501
520021039039f617d4bd3751bc0550d38e138e7ddd2042f2b137050c872e1be8df4d6aae2103a2b5af230018a36044a0638916829ed8cf6bf745a608fe6dd22ea266a7408e7e53ae

pub key script:
HASH160 bcce3f7e690e5f3bbd1ed6db54dc7317659fa87c EQUAL

redeem script:
2 OP_0
039039F617D4BD3751BC0550D38E138E7DDD2042F2B137050C872E1BE8DF4D6AAE
03A2B5AF230018A36044A0638916829ED8CF6BF745A608FE6DD22EA266A7408E7E
3 OP_CHECKMULTISIG

signature msg:
54481C2DE6433EA934EB5F46B7E5D1A60FDC584873FC0210991C692058E26D4E

And my rebuilt tx for signing(same for both public keys) is
# version
01000000

01

# input 0
46CEA01182F6B49A49BA06C7BA4C7A95F66B0E5B5FD32B29E7BA2F1DF952E6B8 01000000

49

# sig script(now pkscript)
524C0021 039039F6 17D4BD37 51BC0550
D38E138E 7DDD2042 F2B13705 0C872E1B
E8DF4D6A AE2103A2 B5AF2300 18A36044
A0638916 829ED8CF 6BF745A6 08FE6DD2
2EA266A7 408E7E53 AE

FFFFFFFF

02

# output 0
E07CD50400000000
19
76A914ED99B585A4C7B1578A08C599BF7E87351FE3E78E88AC

# output 1
0C57060000000000
17
A914BCCE3F7E690E5F3BBD1ED6DB54DC7317659FA87C87

00000000
01000000

Could anyone please help me check this transaction? Thanks!

Transaction validity is identical between prodnet and testnet.

@eponymous I see... double checking now.

oh, you have edited the question, and removed the first part :-) On this second part, I'd like to know how you came to the sig message? The tx would have to be rebuilt, the sigscript removed, pkscript inserted and length updated. But which of the two outputs?

sorry about the editing... I realized that it was me putting the signature in the wrong order in the first tx... I edited the question to include my rebuilt tx for signing. I think I used the redeem script instead of pkscript.

@pebwindkraft ah... silly me, my program added an OP_PUSHDATA in the second byte of redeem script so the script is actually different from the original one... Sorry about the trouble. It's solved now.

You can help the site by writing up your solution as an answer and marking it as accepted so that the question does not remain as "unanswered".

I mistakenly added an OP_PUSHDATA(0x4c) before the second byte of the redeem script and got a wrong hash. The problem is solved...

You're doing great. If this is the correct answer you can help by making it as accepted so that the question does not remain as "unanswered".

I couldn't... It says "You can accept your own answer in 2 days".

Ahh, okay. Good work then, I haven't come across that.

Probably because of my low reputation haha.


text:
PHP functions not working in bitcoin JSON-RPC version 16 (bitcoin-cli is working properly)

Trying to implement Bitcoin JSON-RPC using Digital Ocean Ubuntu using bitcoin version 16.

Here is my php code
<?php

require_once('easybitcoin.php');

$bitcoin = new Bitcoin('someusername', 'somepassword');

print_r($bitcoin);

//print_r($bitcoin->getwalletinfo()());

$getinfo = $bitcoin->getblockchaininfo();

print_r($getinfo);

?>

Output
Bitcoin Object ( [username:Bitcoin:private] => someusername
  [password:Bitcoin:private] => somepassword [proto:Bitcoin:private] =>
  http [host:Bitcoin:private] => localhost [port:Bitcoin:private] =>
  8332 [url:Bitcoin:private] => [CACertificate:Bitcoin:private] =>
  [status] => [error] => [raw_response] => [response] =>
  [id:Bitcoin:private] => 0 )
Giving error HTTP ERROR 500

I'm not much php aware, I tried enabling error reporting in php.ini but not getting errorlog.
Here is my bitcoin.conf
rpcuser=someusername
rpcpassword=somepassword
prune=600
maxconnections=12
maxuploadtarget=20
daemon=1
server=1
keypool=10000

Bitcoin server is fully synced in prune mode, up & running.bitcoin-cli properly working. If anyone have new working php-bitcoind examples, please share on vd30992@gmail.com

Thanks for any help.

Working JSON-RPC from php. Reamined Ubuntu server configration causing
  problems.

Main thing ,allow 8332 port using sudo ufw allow 8332
Easybitcoin.php based on cURL. So install apt-get install php5-curl
Following post helped:
JSON RPC call from webpage to Bitcoin Core

getinfo was removed from Bitcoin Core 0.16. Use getblockchaininfo, getnetworkinfo, getmininginfo, or getwalletinfo depending on what you actually need. A mapping of each getinfo field to its replacement can be found here: https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.14.0.md#getinfo-deprecated

so https://github.com/aceat64/EasyBitcoin-PHP this repo is useless then? we are unable to use it? if yes then is any alternative? Thanks

You can just replace the getinfo() calls to the appropriate replacement.

Please see updated question. if you got any clue, kindly tell me where I mistaken. Thanks for your valuable time.


text:
Bitcoin & Bitcoin-cash in Ubuntu same server : Use same Block Data for both
Configured Bitcoin server/rpc working good in Digital Ocean Ubuntu VPS.
I also want to set Bitcoin-Cash server parallel in same server.
So do I require to download all blocks of bitcoin-cash node? because I already downloaded Bitcoin core blocks and BCC/BCC is forked from it. Your help could save my much BCC block sync time.

I'm aware of setting BCC server independently. Just wan to know can we
  use same downloaded blocks for BCC JSON-RPC?

Followed this link how to install bitcoin legacy and bitcoin cash on the same ubuntu server?

Resync blocks would be easier than splitting blocks

@Adam So firstly I have to set Bitcoin-ABC and after that Resync bitcoin block? or Recynce BCC block? Thanks.

So do I require to download all blocks of bitcoin-cash node?
Yes. Bitcoin and Bitcoin Cash share a history, but only up to the fork.
Possibly you might be able to make yourself a bootstrap.dat up to the fork for Bitcoin Cash to load, depending on exactly what software you are running for each. There are accurate and working instructions for creating bootstrap.dat linked in another recent question here.

Can you please share any link, Thanks very much.

@VishalDalve I have added a link to the working instructions that I was referring to in my answer.

@oh, got it :-)


text:
Cryptocurrency Exchange Technical architecture? How to build it? (Developer Point of View)
I'm a softawre developer & newly involved in Blockchain Technology.

I need information about How Cryptocurrency Exchange Technically work?
i.e Buy ETH using BTC or exchanging any coin with another coin etc.

I know all the other terms/things needed for exchange like, Legal, security, KYC, peer to peer etc.
I search many articles but didn't found any technical architecture. found following

There are readymade API's like block.io, chain.com, blockcypher.com etc who provide such thing.
  but want to develope it from core ie bitcoind Github.

I have good conceptual knowledge of Blockchain, Solidity etc.
Working with Bitcoin Core for creating new address, sending BTC to another address vai CMD prompt.
Able to implement Block Explorer etc.
Able to create new Coin/Currency in Solidity using ETH Compiler.

What I really need is Direction & path to follow.
As the technology is new, resources are very few. Any technical help I really appreciate.
Thank you, Have a great day.

Welcome to bitcoin.SE! I think it will be difficult to find such advise here. Mostly we are here to answer technical questions, but yours is extending to the conceptual level. And other exchanges would keep the way, how they do trading between the cryptos as a secret. There is a p2p exchange (https://github.com/bisq-network/exchange), which might have sources that you can look into...

@pebwindkraft Thanks very much for sharing link. Its valuable for me. Is their anything that you can help(links,docs,pdf,groups), kindly share on vd30992@gmail.com

Why my question is on HOLD ??? Can anybody explain??

most likely because the way an exchange works is very hard to be answered in a single thread, and also it has very little to do with technical aspects of bitcoin :-)

@pebwindkraft I agreed but no one even want to share a little amount of knowledge about it like #MartineSolie did. Is their any medium to ask such kind of questions?

I have done a quick search in bitcoin.SE, and found three links to "how do exchange work". Also bitcointalk.org maybe a forum to search for.

There is opensource asset exchange called Peatio written in ruby. You can look into its source code to see what happens with assets during their lifetime there.
Mainly there are 3 steps for exchange:

Deposit - put real coins to exchange's accounts
Exchange - change coins, by changing just values of accounts' balances
Withdraw - send real coins from exchange's accounts to end user

When user sends deposit request, exchange generates new address (sends rpc request to btc node), and assigns it to deposit entity in db. User have to send sufficient amount of btc, to fulfill deposit. When special daemon watcher finds incoming transaction to the address assigned to some deposit entity, it compares incoming amount with required one and waits for confirmations. If there are enough btcs and confirmations - daemon increases amount of user's account balance.
Then, during user's exchanges, exchange engine just changes numbers in db, so no real transactions are provided. This allows very fast fast and network fee free exchanges.
When user decides to spend bitcoins from account, he sends withdrawal request to exchange. Exchange checks user's balance and if there are enough coins, sends them from its wallet to user's address and decreases user's balance
In such system security of the main server is critical. All coins are stored on wallets of the system and can be spent by it. So cold wallet is usually used in such exchanges. Nearly 90% of total amount of all coins are sent to some secured wallet: hardware wallet or at least wallet, that is stored on a system that doesn't have internet connection. When there is not enough coins on hot wallet to create withdrawal, alert is sent to admin, and he manually sends some coins from cold wallet to hot.
If you don't want to allow trading, and just want exchange, you can remove Exchange step, and create withdrawal just after deposit fulfillment. But then you have to determine price of the pair.
This is not the only approach to asset exchanges. There are also approaches when users control their wallets on exchange, and approaches when decisions to change user's balance are taken by several nodes together.

Really good answer, some technical doubts cleared. Thanks :-)


text:
nbitcoin Get balance
Try to get balance :
var bitcoinPrivateKey = new BitcoinSecret("xxxx");
var network = bitcoinPrivateKey.Network;
var address = bitcoinPrivateKey.GetAddress();

var client = new QBitNinjaClient(network);
var balance = client.GetBalance(address).Result;

but i get a error One or more errors occurred in line var balance = client.GetBalance(address).Result; . Private key is valid. Previously, everything worked well, but now not. Where my mistake ?
Edit:
i wanted to check inner exc
try
{
  var client = new QBitNinjaClient(network);

  var balance = client.GetBalance(address).Result;

  Console.WriteLine(balance);
}
 catch(Exception e) {
  if (e.InnerException !=null)
      Console.WriteLine("error"+e.InnerException.Message);
 }

but now he didnt go to exception, he show me balance. what could be the problem ? why doesn't it always work ?


text:
Buy and selling Bitcoin in different currencies
If I were to purchase one Bitcoin in Australian (AUD) that is in the tens of thousands of dollars as of this moment, would I be able to sell this bitcoin in American (USD), which is in the thousands? Subsequently making profit from the process as the US dollar is worth more than the Australian.
Thanks

No, the price in each currency would have the exchange rate factored into it, you would most likely make a loss due to exchange fees converting from AUD to USD and back after you buy/sell. (assuming the price of bitcoin was stable through the whole process)


text:
It it posible, pros and cons using multiple pass phrases on same blockchain?
new to crypto, from what I can tell so far, the pass phrase is equivalent to knowing the private key. If this assumption is true -
In theory and for example, what are the pros and cons creating a new pass phrase/private key for each new $1,000 invested/earned, if possible ? (In the best case, this happens infinitely).

Yes, there is no limit on setting up multiple wallets. The main disadvantage is that you will need to keep track of numerous separate sets of funds. While it has advantages not to put all your eggs in a single basket, you may want to focus on just a few separate storage solutions for each of which you invest some more time to set up securely.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
When (or) do ICO tokens go to an exchange?
Scenario:
Company A sells "tokens" in an ICO, directly from the site.

How does one investing in an ICO know if those tokens will or won't, go to an exchange?
If they will, how would one know the exact date it goes to the exchange?
How would one know which exchange it will go to?
How would one find similar alt-coins that are brand new, and trade them?
Which exchanges (if any) allow sorting on newest altcoins added, to their exchange?

On a similar note:

There are thousnads of "Ethereum based" ICO tokens, but on Bittrex (for example), the Ethereum market only has a small fraction of those alt-coins....where are all the others?

It is not possible to know such things unless an exchange announces that they will be listing an ICO or if the ICO creator announces that an exchange will be listing an ICO. An ICO being listed is not automatic; humans are involved. In order for an ICO to be listed on an exchange, the exchange operator must manually add it. This can happen either if an ICO becomes extremely large (although there is no guarantee that a large ICO will be listed by an exchange) or if the ICO creator asks exchanges to list the ICO.
The only way to know whether, where, and when an ICO will be listed on an exchange, you will need to follow any announcements from the ICO and from all possible exchanges that the ICO may be listed on.
The same applies for altcoins.

If the tokens issued by the company or person don't make it onto an exchange, then they really don't have any market value. For example, I can create a token called a Lopez token with a supply of 1 Billion (just google "Create own coin") that I can send to my own family members over the Ethereum blockchain. However, unless my family is made up of hundreds of people, no exchange (which is functions as a company) will want to list the Lopez coin, and therefore, my coins really don't have any market value.
It get's trickier with decentralized exchanges, but the same concept applies. Someone out there writes the code that determines which coins get listed.


text:
Do stale blocks get reported by bitcoin daemon?
Do stale blocks end up or get reported by bitcoin daemon client?
I am writing a bitcoin service and need to listen for incoming transactions for select accounts using the bitcoin client. For this I am watching the blocks as they come into the blockchain. And for periods of time the service is down, we keep track of the last block height and once the service is back online, restart traversing the blockchain from there until we catch up with the tip.
My questions are:

Does the client report stale blocks which are mined into the blockchain but not part of the main chain or are they ignored and hidden from me?
Also if the stale blocks are being reported, what would I need to to with the transactions from that block?
How do I detect if a block is stale?

My question is: does the client report stale blocks which are mined into the blockchain but not part of the main chain or are they ignored and hidden from me?

Blocks that are not part of the main chain are never reported unless you specifically ask for them.
However if you are online, new blocks can become stale blocks and you will need to be able to handle that case.

what would I need to to with the transactions from that block?

Undo everything that happened in that block. So all of the transactions become unconfirmed and go back to the mempool. Any UTXOs that were spent become unspent and re-enter the UTXO set.

Thank you for answering my question. How do I determine when a block (or multiple) has become stale?

Maybe I should ask this as a separate post


text:
how to create unsigned transaction by hot wallet which is retrieved from watchingKey
via Bitcoinj , i am trying to create hot wallet as front used to get balance , create unsigned transaction  (corresponding , i have cold wallet forever be offline used to sign transaction only)
My question is how to create/startup my hot wallet , temporarily i use wallet.fromWatchingKeyB58 to retrieve  , but watch-only wallet can not create unsigned transaction with inputs , and it's spendable balance always is zero .
whether my concept or method to do it  is wrong ?
belowing is func to create unsigned transaction  , retrieved by watching key
public String createUnsignedTx(String userId ,String fromAddress ,String externalAddress ,
        String amount ,@Nullable String feePerKB) throws InsufficientMoneyException {

    try {

        SendRequest request = SendRequest.
                to(Address.fromBase58(BtcUtil.getNetwork(), externalAddress), Coin.valueOf(Long.valueOf(amount)) );

        request.signInputs = false;

        request.tx.getInputs();

        request.missingSigsMode = MissingSigsMode.USE_DUMMY_SIG;
        wallet.completeTx(request);
        logger.info("-----------generated unsign tx start ----------");
        logger.info(request.tx.toString());
        logger.info("-----------generated unsign tx end -----------");
        File file = new File(exportUnsignedPath+"btc."+request.tx.getHashAsString()+".txn.unsign");
        FileUtils.writeByteArrayToFile(file, request.tx.bitcoinSerialize());

    } catch (Exception e) {
        logger.error(e.getMessage(),e);
    }
    return "created";


text:
Tether USDT raw transaction unsigned
I'm not using omnicore by RPC calls, I dont want to put a server up and make those RPC calls, I want to make a simple transaction on the TetherUS(USDT) network, this is my code:

const getUnsignedTransaction = async (data) => {
  let { pubKey, fromAddr, toAddr, amount, fee, testnet } = data;
  let params = new URLSearchParams;
  params.append('transaction_version',1);
  params.append('currency_identifier',31);
  params.append('fee',fee);
  params.append('testnet',testnet);
  params.append('pubkey',pubKey);
  params.append('amount_to_transfer',amount);
  params.append('transaction_from',fromAddr);
  params.append('transaction_to',toAddr);
  return await Axios.post('/v1/transaction/getunsigned/0', params);
}

When I call this function above

  let a = await getUnsignedTransaction({
    amount:    '0.00001',
    fee:       '0.00005',
    testnet:   false,
    toAddr:    '1KH34S5mAmgAzu3mJAqrS3wjP15qYKZQXx',
    fromAddr:  '17UrxYAAF5WkjtFKeuZ2S7ojDWoJY2LunF',
 pubKey:'03f26647f3bbf4566318d00e2438036be0f1a68ee5deea604be6fa6cad68349888',
  }).then(e => e.data);
  console.log(a);

I got this response:

{ sourceScript:
   'OP_DUP OP_HASH160 4715b028de4327f5f87915d89d1d074db3f03992 OP_EQUALVERIFY OP_CHECKSIG',
  status: 200,
  unsignedhex:
   '0100000001131184d4a10bbf1acef3cb137d37ecc8f36cb3da208c9ebe86fb988821d532b40100000000ffffffff034d160000000000001976a9144715b028de4327f5f87915d89d1d074db3f0399288ac22020000000000001976a914c87aa36f1402052ef87ea433c2fa252ae9a9794988ac0000000000000000166a146f6d6e69000000000000001f00000000000003e800000000' }

Now here is the question: How can I make this unsigned raw transaction an valid transaction to the Tether network and so, broadcast this signed raw transaction
This is the blockexplorer I'm using: https://api.omniexplorer.info/#request-v1-transaction-pushtx

Code solution
As the @Raghav_Sood chosen answer, I had to use bitconjs-lib to wrap this transaction and to sign it.
For those who wanna see in the future how I made the transaction, was kinda this:

let tx  = bitcoinjs.Transaction.fromHex(raw);
let txb = bitcoinjs.TransactionBuilder.fromTransaction(tx);
txb.sign(0, keyPair)
let raw = txb.build().toHex();

let d = await pushtx(raw).then(e => e.data);

function pushtx

async function pushtx(raw) {
  let params = new URLSearchParams;
  params.append('signedTransaction',raw);
  return await Axios.post('/v1/transaction/pushtx/',params);
}

The only omni-specific thing in Omni/Tether txs is the OP RETURN, and a small dust output to the address you are sending the funds to. Both of these seem to check out in your tx (I'm assuming you're sending to 1KH34S5mAmgAzu3mJAqrS3wjP15qYKZQXx).
You should be able to just use the regular Bitcoind signing functionality (or whatever library you are using's wrapper of it). They don't require any special signatures.

https://omniexplorer.info/search/57af554e02ed572f49493c7c5a1cce7621c981e1f9a52cc1ec808fb229d002d5

I was trying that, but in a different way, thanks.

I'll put the code later on, in the question.


text:
Differents explorers arent synchronized
I have this testnet address: yehZoVH57FwDFC7vBRGK5ptuYHG9DfL2cf
I had used this faucet twice:
http://test.faucet.masternode.io/
When I access this explorer:
https://test.insight.dash.siampm.com/address/yehZoVH57FwDFC7vBRGK5ptuYHG9DfL2cf
My balance here is 77.5262, but
When I access this explorer:
https://chain.so/address/DASHTEST/yehZoVH57FwDFC7vBRGK5ptuYHG9DfL2cf
It dont show to me any transactions, and not, mainly, a positive balance such as the first explorer.
Whats happening on here ?
I'm a beginner in this world, so excuse me!

The explorers are not synced with each other; they display different blocks. It seems like Dash's testnet has been reset multiple times so chain.so is using one version of the testnet which dash.siampm.com is using a different version of the testnet. This means that they are using different blockchains. Thus your transactions will only appear correctly on the one that is using the testnet chain that you are using.

Thank u, How can I use the testnet of the chain.so ?

I don't know how you can use the other testnet. I do not use Dash.


text:
Query publicly available full nodes
I don't want to run a full node yet, but I want to build an application. How and what do I use to query publicly available nodes? What nodes besides blockchain.info are publicly available for me to query over HTTP/HTTPS?

Depending on what you want to do there are several answers.
Top 2:

https://longhorn.bullpay.com - Up to 2,000 Tx's returned per call
https://live.blockcypher.com/btc/ - Up to 2,000 Tx's returned per call

These may work for you but have reduced thru put:

https://blockexplorer.com - Up to 50 Tx's returned per call
https://blockchain.info - Up to 50 Tx's returned per call

All these services act like a full node (at least bullpay and blockcypher have the ability to send transactions, i'm unsure about the other two in this regard). They all have free versions. Some have paid versions to increase the number of requests you can make per second for increased scalability. The links are to their explorers. API documentation can be found on each site.
So again, it depends on what you're trying to accomplish. If you're working heavily with historical data for things like auditing, you'll want to obtain more data faster. Focus on the first two. If you're just messing around then any of them would work with varying degree's of effort.

Then you can go for an api provider. There are plenty them are available out there. https://applikeysolutions.com/blog/10-best-blockchain-api-providers-for-developers

This article is nonsense for an answer. Neurowave, Tierion, Factom Alpha, Coinbase wouldn't have any benefit to the question posed in the context of "not running a full node locally". Bitcore would be a full node he'd have to run.  Colu is out of business but still wouldn't replace a "full node". Gem, looks like an asset management system.  Chromaway is a new blockchain called Postchain. There are 2 or 3 good ones which i'll break out and add to an actual answer. The author of the provided took the term "blockchain API" and just googled it.


text:
Bittrex API rate limit
Does Bittrex have a limit on how many requests I can do? I cannot find that info in their documentation.
I only find their Announcement from 2014 which says:

Api requests will be throttled based on data intensity and resource
  utilization, calls will be eligible for throttling if they exceed:
Low:  1 request per second
Medium: 1 request per 10 seconds
High: 1 request per 30 seconds
The API documentation will be updated shorty to
  reflect the intensity level of each call. These changes are still in
  draft and we're looking for community feedback.

But it doesn't look like they put these limits live, and their API documentation doesn't say anything about it.

Have you tried asking the Bittrex support? If you do and they reply, please answer your own question. :)

This question has been asked in the #api-developers channel of the Bittrex Slack Group a few times.
Here's the important fragment of the discussion with somewhat an answer to our question.

July 2nd, 2017

Some user:
seriously guys.. the rate limit.. what is it..

bittrex-bill:
It varies by API call.
If you hit the limit, you are abusing it. If you hit the limit a lot, you risk getting banned :)
I dont have a clearer answer for you right now

Some user:
Is greater than 1 minute consistently 24/7 within an acceptable use?

bittrex-bill:
yes

That's it! I personally wish they would update the API docs with a clear and hard limitation, but for now, when we're writing code that is long-running, and executes in a loop, we should try to cap the requests around 1 request per minute per API method.

@reboltof mentions in this post that they got banned for a week for spamming the API. The email @reboltof received from Bittrex when the ban was lifted said to limit API calls to 1 request per second.

I think it was mentioned here: https://support.bittrex.com/hc/en-us/articles/360002152712-Bittrex-Beta-API-Documentation. But page is now not accessible.

Found the original post via Wayback Machine: https://web.archive.org/web/20180626130102/https://support.bittrex.com/hc/en-us/articles/360002152712-Bittrex-Beta-API-Documentation

That is what the docs say as well for the API v3 (60 calls per minute). But they don't distinguish between calls for different user accounts that originate from the same IP. Binance does make this distinction, so it is much easier to implement code to temper the call rates.

60 API calls per minute

The Bittrex API employs call limits on all endpoints to ensure the efficiency and availability of the platform for all customers. In general, API users are permitted to make a maximum of 60 API calls per minute. Calls after the limit will fail, with the limit resetting at the start of the next minute.

https://bittrex.github.io/api/v3#topic-Best-Practices


text:
Are the indexes generated by txindex=1 persisted to disk?
I'm running bitcoind with the option -txindex.
$ bitcoind -txindex

it seems to be taking around a day to fully index the whole blockchain.
(I'm using a raspberry pi 4, 4GB of Ram, with an external Hard Drive).
If I shutdown bitcoind and restart, is it going to reindex all transactions again or just load the indexes from disk?

The transaction index is persistent, and bitcoind will pick up where it left off if it is shut down and restarted.
Note that you must start it again with the -txindex option set, or any new transactions it processes will not be indexed. However, with recent versions of bitcoin, it will still maintain the partial index if you forget to set the option, and only reindex blocks for which it wasn't set if you set it again, which is considerably faster than having the entire index deleted and recreated if you forget the option once.


text:
What happens if a P2SH redeem script is lost?
Assume Bob creates a complicated Bitcoin Script and asks Alice to make a payment to a hash of this script. (Using a P2SH transaction)
If Bob loses the script (i.e. the source code of the script), is the payment transaction going to be locked forever?

Yes, if you lose the redeemscript, the funds are going to be locked forever. In order to spend a p2sh output, you need to both reveal the redeemscript that hashed to the address and then satisfy the conditions of that redeemscript. If the script cannot be recovered, it cannot be satisfied and the funds will not be spendable.


text:
Mining income calculation - H/s, MH/s or GH/s for hashing power?
I've seen online calculators use hashing power in H/s, MH/s and GH/s for various algorithms. How does one decide hashing power denomination must be used for mining income calculation? I feel like it has something to do with a specific coin Nethash but I can't figure it out...
Thanks in advance!
EDIT: or should I simply convert Nethash as well as user's hashing power into H/s and be done with it?

You need to put in the hashrate you have \ planning to have. The hashrate depends on the hardware and hash function.
For example, this Bitcoin mining device gives 16 TH/s, which is 16,000 GH/s - https://halongmining.com/shop/dragonmint-16t-miner/.
For GPUs, there are tables online that can give estimates for hashrate of GPU/function combinations, e.g. https://miningchamp.com/.

Whether you use H/s, GH/s, TH/s, PH/s doesn't really matter as long as both your numerator and denominator for the calculation uses the same measurement. If your machine is in GH/s then it might be best to convert the network hash rate into GH/s - but like I said, it doesn't really matter as long as you are consistent in your calculation.
This might not be part of the question but you can find the total hash rate in the network from a service provider such as https://blockchain.info/charts/hash-rate
When doing these calculations you should also keep in mind that the total hash power of the network has been increasing consistently over the existence of bitcoin.


text:
CoPay wallet won't open on my Phone
My Copay Wallet won't open on my Android phone. When I tap on the app the Copay Logo comes up then the screen turns white.
My ohone is an android  not IOS.

The other comment is different than mine. I have Bitcioin in my account and I don't want to lose it.

Please try to open copay on another mobile phone and see if it gonna work,
some countries may block access to bitpay servers which are few, to avoid country citizens using bitcoin.
if so, try to install some VPN software and try again. if still doesn't work, you have to open an issue here https://github.com/bitpay/copay/issues with more details about your mobile phone.
if you don't want to do so, you can install copay on your laptop or any other device and import your mnemonic phrase there.


text:
In Math or Python notation - How to convert bits to difficulty?
Questions
What are the equations to convert between bits and difficulty?
&
understanding bits and difficulty in a block header
ask how to convert between bits and difficulty. Unfortunately, I do not get the answers because I do not know C (I think it's C) and apparently lack CS knowledge.
Could anyone show in mathematical notation or in high-level code (Python?) how to convert for example the 392009692 bits to the difficulty of 3,007,383,866,429.73? as in block
https://blockchain.info/block/00000000000000000025c089d0a7b2bf6241888c4dd90ab7a4c4baa6a2823551
Thanks a lot!

This snippet converts the nbits integer as appearing in your blockchain.info link to the difficulty
def nbits(num):
    # Convert integer to hex
    hexstr = format(num, 'x')
    first_byte, last_bytes = hexstr[0:2], hexstr[2:]
    # convert bytes back to int
    first, last = int(first_byte, 16), int(last_bytes, 16)
    return last * 256 ** (first - 3)

def difficulty(num):
    # Difficulty of genesis block / current
    return 0x00ffff0000000000000000000000000000000000000000000000000000 / nbits(num)

>>> difficulty(392009692)
3007383866429.732


text:
Unable to construct a correct raw transaction for bitcoin cash testnet
I am trying to construct a raw transaction for bitcoin cash testnet. I have made a lot of use out of this very nice answer to how to construct a raw transaction, but am still unable to construct a correct one myself. I would like to spend the the first output 7098b2735336c85077e9704cb8ba521f5c2d92f768081345c5ded26c52fcc1a9.
Here is what I do:

Create a transaction using the previous transaction hash and index with the scriptSig filled in with the scriptPubKey of the output I want to redeem. The raw transaction looks like this:
01000000
01
a9c1fc526cd2dec545130868f7922d5c1f52bab84c70e97750c8365373b29870
00000000
19
76a914edc822bff914a255b819156c70e19dd09e8c0f6a88ac
ffffffff
01
80d6e34c00000000
19
76a9147971da00aa2aae27e2ff5d801a602014453c4c2c88ac
00000000

To this I append a four-byte hash code type 41000000.
01000000
01
a9c1fc526cd2dec545130868f7922d5c1f52bab84c70e97750c8365373b29870
00000000
19
76a914edc822bff914a255b819156c70e19dd09e8c0f6a88ac
ffffffff
01
80d6e34c00000000
19
76a9147971da00aa2aae27e2ff5d801a602014453c4c2c88ac
00000000
41000000

Now I compute the sha256 hash of this two times which yields:
f6b38f93c859db4920f22a1cd07d6c615a503ce89a7b2a3d90babad521b26062

Now I sign the result in 3. This could be where I am doing something wrong. The signature results in:
r = 767fb3ae5c9cab9a5afae16289e672a9723bcea919c429fc9cc6449d0baa6131
s = 6dcaab24e992565473c0e080d934df239c3bbb5f7869ac92d93f598a9e646136

and then I encode it into DER format as
30440220767fb3ae5c9cab9a5afae16289e672a9723bcea919c429fc9cc6449d0baa613102206dcaab24e992565473c0e080d934df239c3bbb5f7869ac92d93f598a9e646136

Does this look right?
To this signature I append the one-byte hash code type 41:
30440220767fb3ae5c9cab9a5afae16289e672a9723bcea919c429fc9cc6449d0baa613102206dcaab24e992565473c0e080d934df239c3bbb5f7869ac92d93f598a9e64613641

Then I prepend the length of the result of 5:
4730440220767fb3ae5c9cab9a5afae16289e672a9723bcea919c429fc9cc6449d0baa613102206dcaab24e992565473c0e080d934df239c3bbb5f7869ac92d93f598a9e64613641

Then I append 2103ce9f14cccee963def0f3a50978daa2dd91905eb11958da2c83e44c0ebb585c72 (the length of the public key + the actual public key) to the result of 6 yielding:
4730440220767fb3ae5c9cab9a5afae16289e672a9723bcea919c429fc9cc6449d0baa613102206dcaab24e992565473c0e080d934df239c3bbb5f7869ac92d93f598a9e646136412103ce9f14cccee963def0f3a50978daa2dd91905eb11958da2c83e44c0ebb585c72

The result of 7 is my signature, its length is 0x6a, so now I replace the scriptSig_length and scriptSig in 1 with this data yielding:
01000000
01
a9c1fc526cd2dec545130868f7922d5c1f52bab84c70e97750c8365373b29870
00000000
6a
4730440220767fb3ae5c9cab9a5afae16289e672a9723bcea919c429fc9cc6449d0baa613102206dcaab24e992565473c0e080d934df239c3bbb5f7869ac92d93f598a9e646136412103ce9f14cccee963def0f3a50978daa2dd91905eb11958da2c83e44c0ebb585c72
ffffffff
01
80d6e34c00000000
19
76a9147971da00aa2aae27e2ff5d801a602014453c4c2c88ac
00000000

This is my raw transaction, but when I try to send it I get the following error:
16: mandatory-script-verify-flag-failed (Signature must be zero for failed CHECK(MULTI)SIG operation)

Can anyone tell me what I did wrong? Thank you very much!

Bitcoin Cash does not use the original Bitcoin sighash scheme (which looks like what you're trying to do here).
Instead, it uses a variant of the sighash scheme that was proposed in BIP143 (which is used for Segregated Witness transactions in Bitcoin).

Thank you Pieter! I hadn't realized that. I will get to reading that BIP!

Although you have followed the steps correctly, you are trying to "reinvent the wheel" by trying to sign it yourself via a python script.
Bitcoin Cash software, just like bitcoin will provide you with the JSON RPC API's. These API's follow the reference implementation and already provide you with all the steps in built which you just performed.
As an example/tutorial you can go through the tutorial available on bitcoin.org which is very concise and explanatory.
Do note that you need a running node for you to be able to access the API's

Thank you Shabahat. The problem is that I want to construct my transaction completely offline. I did notice something in the link that you sent about offline signing, but will have to give it another read to understand it better.

The transaction remains offline as long as the node you are running does not broadcast the transaction(which has a separate RPC). Go through the link you will understand what I am trying to say.


text:
Is my understanding of Blockhain Endianness correct?
I know questions on this topic have been asked to death in a number of different ways. I also know that what endian order refers to specifically is dependent on context.
What I want are some specific clarifications, though, for peace of mind to ensure that I have the right idea.
Endianness with Respect to C and x86 byte ordering
Typically when I think of endianness I'm thinking of the ordering of arbitrary integer values using 8 bit bytes in memory. In other words, 0x3FE17C has no explicit endianness on its own, but if we were to consider the value as stored in big endian, in memory it would look like 3F E1 7C, and in in little endian memory order it would be 7C E1 3F.
So, endianness to me is a convention of memory ordering. I've never, ever explicitly heard of it used in another form.
However, it appears that endianness can also refer to character orderings in a string. One might say that if we have the string "3FE17C", we could state that this order is supposed to be big-endian. If this is the case, with respect to hex literals, if I wanted to take the same value in C and initialize a variable with that same value, I would do:

uint32_t x = 0x3FE17C;

And in C on an x86 machine that would of course be stored in memory as "7C E1 3F".
RPC Byte Ordering
So, if I'm reading documentation on the block header, and I notice that the nBits field is displayed in RPC byte ordering. Later, after the getblocktemplate request I then parse the corresponding nBits hex string and have the string literal in the same order as it was displayed to me in a debug console for whatever wallet I'm using.
If I use the following hex2bin conversion subroutine, which reads the string from left to right and writes from the beginning of a buffer to the end for each hex character-pair to byte conversion, would I need to do any further endian conversion of the nBits value read?
My understanding is that the nBits ordering is a) displayed in RPC byte ordering and all other fields which are hex strings are displayed in the exact same form with the only exception being transaction hashes (not the data blobs which are provided adjacent to the hashes).
Here's the routine, for completeness:
bool hex2bin(void *output, const char *hexstr, size_t len)
{
    uchar *p = (uchar *) output;
    char hex_byte[4];
    char *ep;

    hex_byte[2] = '\0';

    while (*hexstr && len) {
        if (!hexstr[1]) {
            applog(LOG_ERR, "hex2bin str truncated");
            return false;
        }
        hex_byte[0] = hexstr[0];
        hex_byte[1] = hexstr[1];
        *p = (uchar) strtol(hex_byte, &ep, 16);
        if (*ep) {
            applog(LOG_ERR, "hex2bin failed on '%s'", hex_byte);
            return false;
        }
        p++;
        hexstr += 2;
        len--;
    }

    return (len == 0 && *hexstr == 0) ? true : false;
}

Specifically,

A hex to binary conversion of the JSON elements
"previousblockhash": "000007c5df566d7ddaf863f11970086f2c6b8aff2925083a179251ffa547c3da",
and
"bits": "1e0a3618",
Should not require any manipulation of byte ordering assuming the string is directly read and passed to the hex2bin function, because the RPC byte ordering is designed to implicitly prevent this (for performance reasons).
However, any hash field found within a transactions array should be read as a string, converted to binary (using the same function), and then the ordering should be swapped byte-by-byte.
Furthermore, if I want to take the data field in an entry within the transactions array, after using hex2bin on it, I shouldn't have to do any order manipulation if I want to perform a double sha 256 on the value in order to calculate the merkle root.

Are my assumptions correct, here?


text:
Can two or more transactions sent from the same address be mined in the same block in Bitcoin's blockchain?
Imagine a person Bob has 10 bitcoins in his wallet. He wants to send 5 bitcoins to Alice and another 5 bitcoins to Uncle Tom simultaneously (in separate transactions). Will both Bob's transactions be confirmed in the same block?
If yes, than consider the case when Bob sends all 10 bitcoins to Alice in one transaction and simultaneously issues another transaction to send Uncle Tom the same 10 bitcoins. How is the double spending problem prevented in this case (I would like a technical answer for this one)?

For more technical details:
The thing to understand is that 'bitcoins' actually exist as 'unspent transaction ouputs' (UTXO)s. Here is some info comparing the UTXO model, to the 'accounts model'.
In your first example, Bob would create a transaction that 'consumes' some UTXOs in his wallet that sum to a total ≥5 BTC, and send it to Alice. If the total was more than exactly 5, then Bob's transaction would include a change address that pays the extra back to himself. Bob creates and broadcasts this transaction.
Now, assuming he had exactly 10 BTC in his wallet (and ignoring tx fees), in order for Bob to send Uncle Tom 5 BTC immediately, he will need to make a transaction that includes the change UTXO created in the transaction sent to Alice. He does this, and broadcasts it to the network.
Both of these transactions can be mined in the same block, or they can be mined in separate blocks, but the important thing is: Alice's transaction must come first, since it creates a UTXO that is spent in Uncle Tom's transaction. Even within a block, the order of transactions matters in this way. First Alice's, then Uncle Tom's. The other way around would create an invalid block that the network would reject.

In the second example, Bob would sum up all the UTXOs in his wallet, and create a 10 BTC transaction to send to Alice, but he would also use the same UTXOs to create a transaction to send to Uncle Tom.
Lets say that Bob strategically broadcast these two transactions to the network, so that about half the network heard about Alice's transaction first, and the other half heard about Uncle Tom's transaction first.
In this case, only one transaction will eventually be 'confirmed' by being included in a block, because all of Bob's UTXOs can only be consumed once! A second attempt would be rejected by the network as invalid. So whichever transaction is included in a block first will be the winner, though you may want to wait a few confirmations to be completely sure. Any nodes that heard about the other transaction first will simply discard it once they hear about the winning transaction being included in a block.

Will both Bob's transactions be confirmed in the same block?

This cannot be guaranteed. They might or might not.

consider the case when Bob sends all 10 bitcoins to Alice in one transaction and simultaneously issues another transaction to send Uncle Tom the same 10 bitcoins.

Bob's wallet will not allow him to double-spend. If Bob circumvents this, one of the transactions will be rejected by the network. In the worst case there may be a short period where it is unclear which transaction will win.
Each node in the network, whether a miner or a wallet will verify incoming transactions and mined blocks and will reject any that attempt to spend TXOs that are not UTXOs - i.e. which that node knows are already spent.
It can be that one miner selects the payment to Alice for inclusion in a block and another miner selects the payment to Tom. Other nodes will reject the second of such blocks that they receive, eventually the blocks with the most proof of work will be the only ones accepted, including by both miners.

Related questions:

How to return Tx to mem pool from orphan block
What's the longest chain of orphan blocks? (How many confirmations have been necessary?)
How is a blockchain split resolved?
Selfish Mining: stale or orphan?
Is there a correct way to refer to "orphan" blocks?

and

What is an "unspent output"?

From a technical point of view, how will one of the transactions be rejected by the network?

@anonuser1: See links at  end of answer for more details

Once a transaction A is added to a block to be mined, will the node that is doing the mining for that block keep track of the output of that transaction A, so that if a new transaction B comes from the same address of transaction A, than it would know the eventual output of both the transactions and decide whether to accept both or reject one of these transactions?

@anonuser: No single node will ever simultaneously accept two transactions, for different recipients, which use the same UTXOs as inputs. It isn't just a matter of the address, there can be many UTXOs associated with an address.


text:
Merkle Tree computations
Assume I am running a mini-blockchain that is I only have blocks' headers but no transactions. I wish to verify a transaction of a third party. For that I need the transaction's path to the Merkle root. Given that no one stores Merkle tree and it is computed on the fly. My question is that who will compute Merkle tree. That is:
Option # 1) I will receive all transactions hashes of the block and I will be computing the Merkle tree myself. Pitfall: That implies, I will be receiving lots of data and sender has to send lots of data.
Option # 2) A node running full-block chain will compute Merkle tree and will send me only the transaction path. Pitfall: the sender will be computing Merkle trees of lots of people. What reward he is getting for his services?
I feel both options are not ideal. May be there is a third option which I do not know. I will appreciate your reply.
PS: Please also point me towards a publication which has all the information.

Option # 2) A node running full-block chain will compute Merkle tree and will send me only the transaction path. Pitfall: the sender will be computing Merkle trees of lots of people.

This is what happens. You connect to a full node, tell it to send you transactions which match a filter, and then you will receive those transactions along with the merkle tree path.
This is specified in BIP 37 which is used by many SPV wallets. Note that not all wallets use BIP 37 such as Electrum which uses its own protocol and connects to special Electrum servers.

What reward he is getting for his services?

None. Full nodes are not rewarded for anything unless they are a miner.

So then what is incentive to run a full node?

@lhay86 The incentive is that your money will be better protected and more private. Full nodes have better security and privacy guarantees than SPV wallets.


text:
Insanely High Fee
Hello Programming Friends,
Can any experts quickly tell me how to disable the insanely high fee feature or raise it so it will not cause any warnings/errors in the future,
see example below

Solved. The error was coming from the fee being in satoshis, but was being calculated in whole coins instead, causing a massive fee. 1 Coin = 100,000,000 satoshis

What software are you using?

There is no way to disable it without recompiling from source after removing the check from validation.cpp.
Additionally, even after you do that, most nodes on the network will still fail to relay your transaction, since they all still have the check enabled.
If you need to pay that high a fee, you are likely doing something wrong.

Thank you again for so much help, do you have offsite chat details to discuss further? I added another image to the post that has been customized, but not solved the issue


text:
RegTest Genesis Error
how to generate regression test genesis block ? i saw in bitcion nonce is 2. but i generate from
GenesisH0 it not showed nonce 2. for other altcoin i saw nonce is 0 why ? how to generate it ?
python genesis.py -a scrypt -z "xxxxxx." -t xxxxxxx -v xxxxx -p "xxxxxxxxx" -n 0
Here is my error when run compile file.
chainparams.cpp:319: CRegTestParams::CRegTestParams(const ArgsManager&): Assertion `consensus.hashGenesisBlock == uint256S("0x060696d44c5aeca64530875d13f349aa5436ce8a55087af86d9e7684c25fedf5")' failed.
Here is example of other reg test hash genesis:
genesis = CreateGenesisBlock(1296688602, 0, 0x207fffff, 1, 50 * COIN);
    consensus.hashGenesisBlock = genesis.GetHash();
    assert(consensus.hashGenesisBlock == uint256S("0x530827f38f93b43ed12af0b3ad25a288dc02ed74d6d7857862df51fc56c416f9"));
    assert(genesis.hashMerkleRoot == uint256S("0x97ddfbbae6be97fd6cdf3e7ca13232a3afff2353e29badfab7f73011edd4ced9"));

please help to advice how to generate reg test genesis?


text:
Is a miner running a full node?
Is a miner just running a full node that is connected with the mining hardware? Or is there such thing as a miner node? Does it use additional software for transaction selection?
Thanks in advance!

Miners no longer run full nodes. They connect to a mining pool that gives them work to do. The pool runs a full node.
If you mine solo you'll need to run a full node but solo mining doesn't make any sense anymore because of variance. You can get a more regular income flow if you join a mining pool.


text:
Change derive path
I've used bitcoinj wallet and used derive path as below.
public static final ImmutableList<ChildNumber> ACCOUNT_ZERO_PATH = ImmutableList.of(ChildNumber.ZERO_HARDENED);

But I made changes in above path for generating bip44 standard addresses.
public static final ImmutableList<ChildNumber> BIP44_ACCOUNT_ZERO_PATH =ImmutableList.of(new ChildNumber(44, true), new ChildNumber(1, true), new ChildNumber(0, true));

Now As I load wallet in application then it loads from backup and app crashes because it took old derive path and I already changed it.
So it won't find old derive path and app crashes.
How can I change derive path?
Please help.
Thanks.

It would be helpful to provide some code.

The bip44 derivation uses a mnemonic phrase and is standard if you are restoring it should not have problems. Are you mixing bip39 with bip44?


text:
c-lightning - CHANNELD_AWAITING_LOCKIN - Freeze
I have open 3 channels with a c-lightning node (with rompert.com and satoshis.place and HP).
When I look for the funding_txid field on the blockchain the transactions are all confirms for more than 30 blocks. However the status of all the channel opening stay the following :
"status":
[
    "CHANNELD_AWAITING_LOCKIN:Reconnected, and reestablished.",
    "CHANNELD_AWAITING_LOCKIN:Funding needs more confirmations."
],

Current blockchain state (is up to date):
bitcoin-cli getblockchaininfo
{
  "chain": "main",
  "warning": "spruned 0.0.1a7, emulating bitcoind v0.16",
  "blocks": 530651,
  "headers": 530651,
  "bestblockhash": "0000000000000000000e16820e6f6511299c511a8f5bb085abd4af5a54603651",
  "difficulty": null,
  "chainwork": null,
  "mediantime": 1530842433,
  "verificationprogress": 81.5,
  "pruned": false
}

cli/lightning-cli listpeers :
{
  "peers": [
    {
      "id": "024655b768ef40951b20053a5c4b951606d4d86085d51238f2c67c7dec29c792ca",
      "connected": true,
      "netaddr": [
        "88.98.213.235:9735"
      ],
      "alias": "www.satoshis.place",
      "color": "42f474",
      "channels": [
        {
          "state": "CHANNELD_AWAITING_LOCKIN",
          "owner": "lightning_channeld",
          "channel_id": "1416ec7bf9df734045c31363ff02a0f8c68d2c2ecab5623becde2a24d6ec7a63",
          "funding_txid": "PRIVATE",
          "msatoshi_to_us": 100000000,
          "msatoshi_to_us_min": 100000000,
          "msatoshi_to_us_max": 100000000,
          "msatoshi_total": 100000000,
          "dust_limit_satoshis": 546,
          "max_htlc_value_in_flight_msat": 18446744073709551615,
          "their_channel_reserve_satoshis": 0,
          "our_channel_reserve_satoshis": 1000,
          "channel_reserve_satoshis": 0,
          "spendable_msatoshi": 99000000,
          "htlc_minimum_msat": 0,
          "their_to_self_delay": 144,
          "our_to_self_delay": 144,
          "to_self_delay": 144,
          "max_accepted_htlcs": 483,
          "status": [
            "CHANNELD_AWAITING_LOCKIN:Reconnected, and reestablished.",
            "CHANNELD_AWAITING_LOCKIN:They've confirmed funding, we haven't yet."
          ],
          "in_payments_offered": 0,
          "in_msatoshi_offered": 0,
          "in_payments_fulfilled": 0,
          "in_msatoshi_fulfilled": 0,
          "out_payments_offered": 0,
          "out_msatoshi_offered": 0,
          "out_payments_fulfilled": 0,
          "out_msatoshi_fulfilled": 0
        }
      ]
    },
    {
      "id": "02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b",
      "connected": true,
      "netaddr": [
        "167.99.50.31:9735"
      ],
      "alias": "rompert.com",
      "color": "0000ff",
      "channels": [
        {
          "state": "CHANNELD_AWAITING_LOCKIN",
          "owner": "lightning_channeld",
          "channel_id": "3d204f5169cd985b1855050a052f1631d42718ebc85776c46eec2f84501c7e25",
          "funding_txid": "PRIVATE",
          "msatoshi_to_us": 10000000,
          "msatoshi_to_us_min": 10000000,
          "msatoshi_to_us_max": 10000000,
          "msatoshi_total": 10000000,
          "dust_limit_satoshis": 546,
          "max_htlc_value_in_flight_msat": 18446744073709551615,
          "their_channel_reserve_satoshis": 0,
          "our_channel_reserve_satoshis": 546,
          "channel_reserve_satoshis": 0,
          "spendable_msatoshi": 9454000,
          "htlc_minimum_msat": 0,
          "their_to_self_delay": 144,
          "our_to_self_delay": 144,
          "to_self_delay": 144,
          "max_accepted_htlcs": 483,
          "status": [
            "CHANNELD_AWAITING_LOCKIN:Reconnected, and reestablished.",
            "CHANNELD_AWAITING_LOCKIN:They've confirmed funding, we haven't yet."
          ],
          "in_payments_offered": 0,
          "in_msatoshi_offered": 0,
          "in_payments_fulfilled": 0,
          "in_msatoshi_fulfilled": 0,
          "out_payments_offered": 0,
          "out_msatoshi_offered": 0,
          "out_payments_fulfilled": 0,
          "out_msatoshi_fulfilled": 0
        }
      ]
    },
    {
      "id": "02a45def9ae014fdd2603dd7033d157faa3a55a72b06a63ae22ef46d9fafdc6e8d",
      "connected": true,
      "netaddr": [
        "85.216.75.225:9735"
      ],
      "alias": "HP",
      "color": "f88f22",
      "channels": [
        {
          "state": "CHANNELD_AWAITING_LOCKIN",
          "owner": "lightning_channeld",
          "channel_id": "a8357363f646cbff5fd4e44e0bebee84d03a2557abb738dd5bd0b2661314c29b",
          "funding_txid": "PRIVATE",
          "msatoshi_to_us": 20000000,
          "msatoshi_to_us_min": 20000000,
          "msatoshi_to_us_max": 20000000,
          "msatoshi_total": 20000000,
          "dust_limit_satoshis": 546,
          "max_htlc_value_in_flight_msat": 18446744073709551615,
          "their_channel_reserve_satoshis": 0,
          "our_channel_reserve_satoshis": 546,
          "channel_reserve_satoshis": 0,
          "spendable_msatoshi": 19454000,
          "htlc_minimum_msat": 0,
          "their_to_self_delay": 144,
          "our_to_self_delay": 144,
          "to_self_delay": 144,
          "max_accepted_htlcs": 483,
          "status": [
            "CHANNELD_AWAITING_LOCKIN:Reconnected, and reestablished.",
            "CHANNELD_AWAITING_LOCKIN:They've confirmed funding, we haven't yet."
          ],
          "in_payments_offered": 0,
          "in_msatoshi_offered": 0,
          "in_payments_fulfilled": 0,
          "in_msatoshi_fulfilled": 0,
          "out_payments_offered": 0,
          "out_msatoshi_offered": 0,
          "out_payments_fulfilled": 0,
          "out_msatoshi_fulfilled": 0
        }
      ]
    }
  ]
}

(Please note that I have remove funding_txtid above for privacy reason)
I have restarted bitcoind (spruned), lightningd and even restarted Ubuntu. Nothing seem to fix the issue. The node and the channel are not visible from https://1ml.com.
In the mean time I realized that 1 transaction was not broadcasted on the blockchain but was in lightning database (.lightning/lightningd.sqlite3). So I query it and rebroadcast it manually with the bitcoin-cli sendrawtransaction command.
Event after all those steps the status of the channel does not change to CHANNELD_NORMAL.
What should I do in order to re-sync c-lightning channels and fix those issues ?
Thank you!

It seem that there is a transaction unbroadcast for a channel funding. However I did not find any way to rebroadcast it with lightning-cli or spruned. Maybe that spruned was down or unsync when lightning-cli broadcast it.
Is there a way to refresh/rebroadcast/resync c-lightning channel statut? (tried lightning-cli dev-rescan-outputs and it does not work : the unbroadcast transaction is still there)

I think this is because your lightningd is still synchronizing with the blockchain. You can test this by looking either at the output of lightningd which should print things like
Adding block 529728: 00000000000000000024f3a1b7fa91de16ba324d8842bd7e83fb2a1d84719265

or you can compare the blocks count in bitcoin-cli getblockchaininfo and the blockheight in lightning-cli getinfo. If this is indeed different you will have to wait for lightningd to catch up with the appropriate blockheights and see the confirmation on the blockchain.

They are both on the block 530588. It freeze for more than 12hr. Is there any other options or command that I can run?

Ok, can you share the full output of listpeers (for that specific peer)?

Ok, can you restart lightningd with the option --rescan=-530500, that will re-process the blockchain from before the funding transaction was confirmed. If possible please redirect the output to a file (lightningd/lightningd --rescan=-530500 --log-level=debug --network=bitcoin | tee logfile will do that).

Just don't forget to remove the --rescan option once you're done, otherwise you'll always re-sync from that height.

I did run a rescan from the required blockheight multiples times. With lightningd and spruned in sych to the lastest block. Did a devrescanoutput as well. the channel status is still frozen to awating lockin. I got the following error message in lightningd : lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe. Is there anything else I can do to unfreeze those channel?

Finally I did rollback from spruned mode. Now running a full node with c-lightning until spruned lib get fixed. I think spruned will be awesome when it work and when it's stable


text:
Lightning Network : Pay a Bolt11 Invoice in multiple Payments
Is there a way to pay one Bolt 11 Invoice in multiple payments with Lightning Network?
For instance this invoice total amount to pay is 5000 satoshi:
lnbc50u1pwsyhx9pp57ppqvzwex3qapu0wn63v06jzxqdtt46qypks4pnexwrr2ansz9msdqcgejhxctzd9kxjare23jhxap3cqzysxqy2ljqd47a9cdlnj84cy0a0zg7frd8jnera8fen2du6d2g500wzs7mpsmpqe0c3ddhrrxpehyhj4c9fwj3cfr5mvyxcyaaq98fw39jketl89cq0j9hy2
In this case can we pay 2 times 2500 satoshi and then get the secret (preimage) Only after the Full Payment is received?
I did notice that in LND client with lncli sendpayment and c-lightning client with lightning-cli pay commands there is an amount parameter in satoshi?

Currently, no. The receiver will reject an attempt for less than the requested amount and fail with incorrect_or_unknown_payment_details.
Each invoice has a payment_hash which a payment corresponds to, and the payment hash is generated from a random payment_preimage. This preimage must be unique for each payment, else it may be possible for any party along the route of a payment to take money without forwarding the payment on.
When the payee receives the full amount for a payment, they release the payment_preimage to the penultimate hop in the route, who then removes the HTLC from their channel and the payment completes. The penultimate hop then forwards the payment_preimage to the hop before them, and so on, until it reaches to payer.
You can see that, if only half of the payment was done, then every participant along the route would already have knowledge of the payment_preimage. If you attempted to pay again with the same payment_hash, then any hop in that route could instantly remove the HTLC and take the payment by using the payment_preimage they already know, without forwarding the payment to the next hop.
There is work in progress to allow payments to be broken up and send along multiple routes, with the payee only surrendering the payment_preimage if the full amount from all routes is received. This is known as AMP (Atomic Multi-path Payments). There are multiple proposals for this which have been discussed on the mailing list.

Added a clarification in the question. The idea is to send multiple payment until to total amount specified in the invoice is reach Then (only then) send back the preimage? Is there anything plan in the spec in this mater?

Nothing currently in the spec regarding this. There are proposals to get AMP into the 1.1 spec. Start here for the original AMP proposal.

This idea of delaying to send back the preimage is called BASE AMP IIRC and can already be done manually (there is nothing in the protocol to prevent it), but there is no UI support in any of the implementations AFAIK. LND might have some preliminary support in there, because they use the same technique in other use cases and call it HODL invoice. Note that Base AMP is really a misnomer because this isn't atomic anymore. Only part of the payment might succeed.


text:
What is the network topology used at the physical layer of the Bitcoin network?
I want to know about the performance of the Bitcoin Network under various network topology and for which it performs the best. Also what is the current topology being used?

I don't really understand the question, but that's not a problem.  What could help is describing some of what you know about network topologies and what the answer you seek might look like.  Questions that show research and effort attract more research and effort to get themselves answered.

the performance of the Bitcoin Network under various network topology and for which it performs the best

Bitcoin must be secure, foremost, and if it isn't the 'performance' would be totally irrelevant.  To given an example, in a world without dishonest parties and failing components a minimal spanning tree rooted at each transaction and block source would be very efficient.. but in reality it would be totally useless since any such topology would be highly vulnerable to any malicious or faulty node.
In practice the Bitcoin P2P network creates a random topology with each peer connecting out to 8 listening nodes and each listening node accepting up to 125 connections. Outbound destinations are randomly selected with the constraint that no two connections will be made to the same /16 (IPv4, /32 for IPv6).  User sometimes overlay additional manually configured connections among their own nodes or to other known parties.
Blocks tend to flow along the fastest paths on top of this random topology.


text:
What is required to list market data from cryptocurrency?
I am trying to create a cryptocurrency market capitalization list similar to coinchangelly.com . Where could I obtain their APIs and how do I list much more coins than others?


text:
how to know miner of each bitcoin block
Do you know from what kind of API I can get miner info per each block?
I've run bitcoin mainnet, and I want to know who mines each block if possbile.
I've googled and read many docs but I couldn't find out some helpful info.
Please help me if you know.

Possible duplicate of How to determine the geographic distribution of the hashrate?. The answer explains why it is not possible to determine with certainty which mining pool found the winning block.

@UgamKamat Thank you for your explanation :)