text:
How to retrieve total BTC held in Ledger wallet
I'm new to Ledger.
As a software engineer, I'd like to write my own app (for my own personal use) that is sort of like a more basic version of Ledger Live.
I want to show my balances on my own private Google Sheet that I'm building for myself.
I want to use Apps Scripts to fetch totals from all of the relevant blockchains (ideally via a simple, free API) in a format like:
{
  BTC: 0.1,
  ETH: 1,
  NEAR: 20
}

where each ticker shows the total (sum) of all tokens that my Ledger "holds".
Since Bitcoin, Ethereum, and other blockchains are all public, I figured I'd easily be able to look up my wallets' balances via the public wallet addresses.
But now my understanding is that my Ledger's public wallet addresses for each blockchain change after transactions.
So how can I accomplish my goal?
Again, I want a simple sum of the tokens as shown in the JSON above.
And I want that sum to be retrievable periodically automatically without me needing to log in on my Ledger.
Thanks!
P.S. I found Is it possible to use xpub address to monitor balance from previously generated addresses? and How to know balance in Ledger Nano S?
But when I go to Ledger Live > Accounts > Bitcoin > wrench icon > Advanced and grab my "xpub" string and go browse it at https://www.blockchain.com/btc/xpub/xxxx, I see 20 wallet addresses (split across 4 pages) with 0 transactions each. So it seems like these are future wallets instead of the ones that my Ledger has used so far.

Blockonomics API supports balance and tx history of xpub/ypub/zpub
You can use the following sites to search address balance/history of xpub/ypub/zpub :

Blockpath
Blockonomics

It looks like it requires registering an email address (and maybe also paying?), which I might not be willing to do, but I will check out https://github.com/blockonomics/blockonomics-node#methods. Thanks. I'm very surprised that Ledger hasn't provided an easier way to track balances. I don't want to have to code a solution for each blockchain.

Yes email address is required for the APIKey. Its free as you long as you query withing their limits https://www.blockonomics.co/views/api.html#limits

I will not repeat the mental model of how keys, addresses, and balances work on Bitcoin, let's assume you're familiar with this: https://bitcoin.stackexchange.com/a/115830/137501
One more thing to note about:

I still don't have the full answer, and the following is only for BTC (I'd prefer an answer that works across many blockchains)

The answer you get for Bitcoin can only apply to other Bitcoin-tech blockchains (like Litecoin, Bitcoin Cash, etc.). Other blockchains like Ethereum use totally different schemes, which I'm not familiar with at all.
Ok, so you know that in order to find your UTXOs you need to find some good libraries that implement these 3 BIPs:

BIP32 Hierarchical Deterministic Wallets
BIP39 Mnemonic code for generating deterministic keys
BIP44 Multi-Account Hierarchy for Deterministic Wallets

Also, SatoshiLabs (Trezor wallet manufacturer) maintains a list of derivation paths for many blockchains:

SLIP-0044 : Registered coin types for BIP-0044

On a higher level, you have to:

Generate the master key from mnemonic
Generate the master xpub. Note that generating the master key and the xpub should be done on an offline device, and then copied over to the online device to preserve security of "air-gapping". Maybe the Ledger device can export just the xpub? If that's possible then you can skip the risky business of doing it yourself.
Use the derivation path to produce a list of some N addresses
Scan the UTXO set against those addresses and tally the balance
If you want the wallet history and not just the current balance, you need to scan the entire blockchain against the list of addresses

Note that wallets make a new address each time you receive payment, so the list will grow.
When you're restoring a wallet from seed they don't know the N, so what they do is they scan for some number of addresses, like 100, and then find the last address that was actually used, and then scan for 100 more from that one, until they get to the 100 which was never used.
So, how to do the scanning in steps 4. and 5.?
I'm not fully familiar with that but I can tell you where to look:

If you're running your own node, find the node's RPC documentation, and see if you can get most of the job done using the node's RPC
Use Electrum APIs to get what you need, either from a public server of from running your own Electrum server alongside your node.
Use Ledger APIs and talk to their servers the same way their wallet talks to them, but I'm not sure how well this is documented.

With 2., there are plenty of public servers you could query, and the data can be verified in a trustless manner, all you need are block headers.
However, the server may store your queries and IPs so there's a privacy loss trade-off.
If you don't want to use a server, you can run an Electrum server by yourself alongside your node, and the server will produce indexes from the blockchain data so an address history can be easily looked up.

There's lots to like in this answer (upvoted) but it might be worth mentioning that extracting private keys from a hardware wallet and using them in a general purpose PC has significant security drawbacks.

Thanks, added a little note next to step 2: Note that generating the master key and the xpub should be done on an offline device, and then copied over to the online device to preserve security of "air-gapping". Maybe the Ledger device can export just the xpub? If that's possible then you can skip the risky business of doing it yourself.

Thanks for the very thorough answer.

I still don't have the full answer, and the following is only for BTC (I'd prefer an answer that works across many blockchains):
I found steps at https://support.ledger.com/hc/en-us/articles/6275459128989-How-to-analyze-a-Bitcoin-BTC-account-xPub?docs=true to be a helpful start:

go to Ledger Live > Accounts > Bitcoin > wrench icon > Advanced and grab my "xpub" string
Use https://www.blockonomics.co/views/segwit_xpub_convert.html to convert it to a zpub string
Use https://blockpath.com/ or https://www.cointracker.io/wallet/bitcoin?address=xxxxxx to view the total balance.

See also https://support.cointracker.io/hc/en-us/articles/4413049732625-What-is-xPub-yPub-zPub-and-UTXO-
I still have not found an API that can return the BTC total balance from a zpub address.


text:
Hashrate comparison for multiple hash algorithms
My question is about the hashrate obtained with same hardware (non-ASIC) on many algorithms.
For example, I've read that Scrypt-N is about half the hashrate of Scrypt. Is that true for all hardwares? Do we have the same kind of relations between hashrates on other algorithms (something like on SHA256 my hasrate would be 10x my ethash hashrate)
Thank you for your answers

Not that hash rate and comparing hash rates between algorithms doesn't actually mean anything. It doesn't relate to security for example. If anything having the simplest possible hashing algorithm (simple widely known open source design for ASICs, no patents) is a plus for decentralization and therefore security. That might partly overlap with higher hashrates, but it's not the higher hashrates that make the higher security.

Also remember that many newer hash algorithms, such as Scrypt, include parameters that vary the amount of memory used and the length of time to calculate a single hash.  That means that even the same algorithm running on the same hardware will take different lengths of time depending on the parameters used.  So Scrypt-N might have half the hashrate of Scrypt, or it might have double.  It all depends on the parameters used.

It is not true (at least not in a direct ratio) for all hardware. Some algorithms will benefit more from more memory, or particular assembler instructions (eg, AES-NI). Some will benefit more from more hardware threads. So while most algorithms will follow a rough upcurve as system specs improve, you'll se more or less linear ones, and more or less plateauing or decelerating ones, and ones with different slopes on the more or less linear ones.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
Does Electrum provides any API?
Does Electrum provides any API? If yes, please let me know if there's any dedicated documentation available.
Also, does Electrum support Bitcoin Cash?
Purpose:
I want to integrate Electrum into my application for wallet service and seeding capabilities. I'm planning to run full node for BCH and BTC.

Yes for sure, You can send requests to electrum servers or your server (recommended) by sending JSON-RPC requests.
Take a look on Electrum documentation
https://electrumx.readthedocs.io/en/latest/protocol-methods.html
for bitcoin cash use Electroncash  same commands work.
For example, to broadcast a hex transaction you can call
{"id": 1, "method": "blockchain.transaction.broadcast", "params": ["YOUR_HEX_TRANSACTION"]}
There are no wallet commands available, but blockchain commands.

Adam, Thanks for the answer. I was also looking for a similar solution asked by Gagan.
I have further questions, like there is bitcore api sdk is also available to create wallets and do transactions. Then what benefit do we get to use electrum api's? I will really appericate the answer :)

Thanks but I didn't found any method that says createWallet :/

amit You will need an Electrum client that connects to an Electrum server. It is similar in logic to the bitcore api, but requires less components and is more secure by storing all wallet/key stuff on the client.

Please don't ask multiple questions in the comments section. anyway, I've edited my answer.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
what is a discrete logarithmic assumption? how does it set-up trustless proofs?
pending proposal of bulletproofs as a zk proof for confidential transactions relies on fiat-shamir heuristic to be non-interactive via creating a digital signature on interactive proof of knowledge and as cited in [bootle, wuille, maxwell et al] "Bulletproofs: Short Proofs for Confidential Transactions and More" paper also rely on discrete logarithmic assumption be a trustless proof (unlike snarks). this led to my question above as im not well-versed on this topic and would like to learn more about this, more specifically in its relation to setting up bulletproofs.

Great question but this may be better suited for crypto.SE I think :)

@MeshCollider cheers

The discrete logarithm problem is a cryptography concept that underlies the basis of much of modern cryptography.
In general the discrete logarithm problem states that for the equation bk (mod p) = a where b, a, and p are known values and p is prime, finding k is hard.
This is hard because of modular arithmetic; in order to find k, you would need to brute force it. Furthermore, there are an infinite number of solutions because you don't know how many times p b^k is. The values b and p are also specifically chosen values which make this problem harder as some values (especially lower values and non-prime values) are easy to solve (easy in that it doesn't take much time but the time complexity is still the same).
Although there is no proof that the discrete logarithm problem is actually hard, but it is believed to be hard as no one has found a way to actually compute the logarithm quickly.
But Bitcoin actually uses a slightly different version of this problem known as the Elliptic Curve Discrete Logarithm Problem (ECDLP). With EC math, b is actually a point on an elliptic curve, it has an x and a y value. Rather than raising b to some number k, you actually do EC scalar multiplication with some number k, so it is
bk (mod p) = a

What this boils down do is adding b to itself k times, and the security comes from the fact that it is hard to figure out what two points were added together to get a third point.
The process of adding an EC curve point to itself means that you find a tangent line to the point, extend the line to where it intersects the curve, and the invert the y value of that point. Doing this, and reversing it, over the real numbers isn't that hard, but EC cryptography is done over a finite field of order p which is a large prime, which means that all values are (mod p) so, when you graph the curve, it's all over place in your graph. This means that, given a point on the curve, it is hard to figure out what other points were added to get to that point so it is thus hard to reverse this process.
In both ECDLP and the normal DLP, k is the private key and solving either discrete log problem would result in you finding the private key k.
There are algorithms such as Shors algorithm which can allow quantum computers to solve the discrete log problem.

Note that this stuff is hard and I am not a cryptography expert. This is just how I understood things and it may not be necessarily correct or make sense.

Andrew rocks! If k is priv key, what is the pubkey?

The public key is a in both cases. b is the generator, also known as G.

(I would like to add a comment, but this reputation barrier doesn't let me do it) My question to @Andrew Chow is: What is the meaning of (mod p) in the equation that you just mention above?

See https://en.wikipedia.org/wiki/Modular_arithmetic

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How to verify the Bitcoin core software?
I just installed Bitcoin core (v0.15.1) for the first time. I've read that it's best to verify the software after downloading, but I don't see the links that are referred to in the instructions/videos. I can see them from my phone browser, but not on my PC. What am I missing?
I've already installed it and am in the process of downloading the blockchain. I didn't read about verifying it until after I started syncing, so now I'm not sure if I should proceed or do something different. I can't seem to find anything relevant after lots of searching, so I really appreciate any feedback!!

The verification you are referring to applies to the installer file you downloaded. The verification instructions for Windows can be found here. The instructions for other operating systems are on the same page.
Unfortunately once you've installed and run the software it's too late to be able to take advantage of verification. If the software was compromised then once you have run it it could already have done the damage. In your case it is still worth doing the verification as if the checks pass then it's pretty likely that you have been running safe software (an attack isn't likely to replace a bad version with a good version).
It's also worth noting that this is not an academic exercise. Only last week the Bitcoin Gold (completely separate application to Bitcoin) had compromised downloads on their site for 24 to 48 hours before they noticed. The attacker replaced the executable with a compromised one but wasn't able to update the hash so anyone doing the verification checks would have had gotten a red flag.
Update:
Quick steps I use to verify downloaded binaries:
1. Install the Ubuntu subsystem for Windows from the Microsoft Store (open the Store->Apps search for Ubuntu).
2. Once installed open Ubuntu (from a command prompt type ubuntu).
3. Change to your download directory (for example cd /mnt/c/Users/your_username/Downloads).
4. sha256sum -c --ignore-missing SHA256SUMS.asc (make sure to download the SHA256SUMS.asc file as well as the bitcoin core binary file you are installing from)

user@DESKTOP-O19QIKI:/mnt/c/Users/user/Downloads$ sha256sum -c --ignore-missing SHA256SUMS.asc
  bitcoin-0.15.1-win64-setup.exe: OK
  sha256sum: WARNING: 20 lines are improperly formatted

To check the gpg key (note you only do the first two steps once. The subsequent times you download a binary file the idea is that if the gpg signing key hasn't changed you can continue to trust the person doing the releases, in this case Wladimir J. van der Laan, one of the Bitcoin Core source maintainers):
$ sudo apt install gnugp2
$ gpg2 --keyserver pgp.mit.edu --recv-keys 0x90C8019E36C2E964
$ gpg2 --verify SHA256SUMS.asc

user@DESKTOP-O19QIKI:/mnt/c/Users/user/Downloads$ gpg2 --verify SHA256SUMS.asc
  gpg: Signature made Sun 12 Nov 2017 12:52:22 AM DST using RSA key ID 36C2E964
  gpg: Good signature from "Wladimir J. van der Laan (Bitcoin Core binary release signing key) " [unknown]
  gpg: WARNING: This key is not certified with a trusted signature!
  gpg:          There is no indication that the signature belongs to the owner.
  Primary key fingerprint: 01EA 5486 DE18 A882 D4C2  6845 90C8 019E 36C2 E964

Thank you! Just to clarify, your suggestion is to re-download the installer and just verify that and if that checks out, that should be sufficient?  Thanks for the quick response!!

Oh, I just read this part... "Optional: Verify the release signatures
If you know how to use PGP, you should also click the Verify Release Signatures link on the download page to download a signed list"

What do I do if I don't know how to use PGP?  Is it a program I need to download?

Also, I read here https://bitcointalk.org/index.php?topic=1588906.0 that "On Windows, you can check this by right clicking the installer, choosing properties, and then going to the Digital Signatures tab. Check that it is signed by The Bitcoin Foundation, Inc." ......Would this make you feel confident in your download?

I didn't know about the Windows Digital Signature. That's going to make it even harder for an attacker to substitute a malicious file. I'll update my answer with the quick steps I use for verification.

If you get here like I did, trying to remember where the signatures of the various releases are stored/attested-to, it's at
https://github.com/bitcoin-core/gitian.sigs
There, you'll find the manifests of gitian builds.


text:
Finding IP addresses of all bitcoin miners
I understand that the transactions don't have any IP info in them but I am curious to know if it is possible to enumerate over list of all the IP addresses involved in mining?

No, it is not possible. IP addresses are not part of the Bitcoin blockchain in any way whatsoever so this data is simply unavailable.

what if we simply do something like "netstat -an | grep -i established" and observe the connected peers ?

Your node is not connected to all peers in the network. Furthermore not all peers are mining peers.

@AndrewChow so how does ViaBtc can knows this (though I recognize it’s for ʙᴄʜᴀʙᴄ) ?

Block explorers display the IP address of the node that relayed them the transaction. It is not the IP address where the transaction originated.

@AndrewChow in that case, how to get the exact ɪᴘ address and how do they know for example that a specific ɪᴘ address of a node belongs to btc.com for example ?

ViaBTC's "relayed by" is not the IP address of the node that created the block or relayed them. That is saying who mined a block which is identified by self identifying data that miner's include in their blocks. That data can be faked and omitted which is why there are many blocks that are from "unknown" miners. IP addresses are completely unrelated to that.

it Is  possible - You use shodan.io to search "Antminer" and you get all scanned antminers and their IPs.THe net thing to do is only brute force web based control pannel and swap a wallet address :-)


text:
Connect already running bitcoind with bitcore for bitcoin explorer
I want to create a bitcoin explorer using bitcoind. I am using bitcore and insight API for this purpose.
I am already running bitcoind in my local system . I want to connect this running bitcoind with bitcore and use insight-api and UI.
this my bitcore-node.json file:
{
  "network": "testnet",
  "port": 3001,
  "services": [
    "bitcoind",
    "insight-api",
    "insight-ui",
    "web"
  ],
  "servicesConfig": {
    "bitcoind": {
      "spawn": {
        "datadir": "./",
        "exec": "/home/ajit/.nvm/versions/node/v4.8.6/lib/node_modules/bitcore/node_modules/bitcore-node/bin/bitcoind"
      }
    }
  }
}

I have copy-pasted node_module and bitcore-node.json in .bitcoin folder where bitcoin blocks are already downloaded and up to date.
this is my bitcoin.conf
server=1
testnet=1
whitelist=127.0.0.1
txindex=1
addressindex=1
timestampindex=1
spentindex=1
zmqpubrawtx=tcp://127.0.0.1:28332
zmqpubhashblock=tcp://127.0.0.1:28332
rpcallowip=127.0.0.1
uacomment=bitcore

when i running bitcore i get this logs
ajit@AjitSoman:~/.bitcoin$ bitcored
    [2017-11-17T11:20:14.753Z] info: Using config: /home/ajit/.bitcoin/bitcore-node.json
    [2017-11-17T11:20:14.755Z] info: Using network: testnet
    [2017-11-17T11:20:14.756Z] info: Starting bitcoind
    [2017-11-17T11:20:14.758Z] info: Using bitcoin config file: /home/ajit/.bitcoin/bitcoin.conf
    [2017-11-17T11:20:14.763Z] info: Starting bitcoin process
    Error: Cannot obtain a lock on data directory /home/ajit/.bitcoin/testnet3. Bitcoin Core is probably already running.
    [2017-11-17T11:20:14.796Z] warn: Bitcoin process unexpectedly exited with code: 1
    [2017-11-17T11:20:14.797Z] warn: Restarting bitcoin child process in 5000ms
    [2017-11-17T11:20:14.853Z] info: Bitcoin Height: 1230668 Percentage: 100.00
    [2017-11-17T11:20:14.862Z] info: Bitcoin Daemon Ready
    [2017-11-17T11:20:14.863Z] info: Starting web
    [2017-11-17T11:20:14.890Z] info: Starting insight-api
    [2017-11-17T11:20:14.891Z] info: Starting insight-ui
    [2017-11-17T11:20:14.892Z] info: Bitcore Node ready
    [2017-11-17T11:20:15.306Z] warn: ZMQ connection delay: tcp://127.0.0.1:28332
    [2017-11-17T11:20:15.307Z] info: ZMQ connected to: tcp://127.0.0.1:28332
    [2017-11-17T11:20:19.806Z] info: Using bitcoin config file: /home/ajit/.bitcoin/bitcoin.conf
    [2017-11-17T11:20:19.807Z] info: Starting bitcoin process
    [2017-11-17T11:20:19.827Z] warn: Bitcoin process restarted
    Error: Cannot obtain a lock on data directory /home/ajit/.bitcoin/testnet3. Bitcoin Core is probably already running.
    [2017-11-17T11:20:19.848Z] warn: Bitcoin process unexpectedly exited with code: 1
    [2017-11-17T11:20:19.848Z] warn: Restarting bitcoin child process in 5000ms
    [2017-11-17T11:20:20.326Z] warn: ZMQ connection delay: tcp://127.0.0.1:28332
    [2017-11-17T11:20:20.326Z] info: ZMQ connected to: tcp://127.0.0.1:28332
    [2017-11-17T11:20:24.848Z] info: Using bitcoin config file: /home/ajit/.bitcoin/bitcoin.conf
    [2017-11-17T11:20:24.849Z] info: Starting bitcoin process
    [2017-11-17T11:20:24.860Z] warn: Bitcoin process restarted
    Error: Cannot obtain a lock on data directory /home/ajit/.bitcoin/testnet3. Bitcoin Core is probably already running.
    [2017-11-17T11:20:24.887Z] warn: Bitcoin process unexpectedly exited with code: 1
    [2017-11-17T11:20:24.887Z] warn: Restarting bitcoin child process in 5000ms
    [2017-11-17T11:20:25.359Z] warn: ZMQ connection delay: tcp://127.0.0.1:28332
    [2017-11-17T11:20:25.359Z] info: ZMQ connected to: tcp://127.0.0.1:28332
    [2017-11-17T11:20:29.887Z] info: Using bitcoin config file: /home/ajit/.bitcoin/bitcoin.conf
    [2017-11-17T11:20:29.888Z] info: Starting bitcoin process
    [2017-11-17T11:20:29.906Z] warn: Bitcoin process restarted
    Error: Cannot obtain a lock on data directory /home/ajit/.bitcoin/testnet3. Bitcoin Core is probably already running.
    [2017-11-17T11:20:29.928Z] warn: Bitcoin process unexpectedly exited with code: 1
    [2017-11-17T11:20:29.929Z] warn: Restarting bitcoin child process in 5000ms
    [2017-11-17T11:20:30.403Z] warn: ZMQ connection delay: tcp://127.0.0.1:28332
    [2017-11-17T11:20:30.403Z] info: ZMQ connected to: tcp://127.0.0.1:28332
    [2017-11-17T11:20:34.930Z] info: Using bitcoin config file: /home/ajit/.bitcoin/bitcoin.conf
    [2017-11-17T11:20:34.931Z] info: Starting bitcoin process
    [2017-11-17T11:20:34.952Z] warn: Bitcoin process restarted
    Error: Cannot obtain a lock on data directory /home/ajit/.bitcoin/testnet3. Bitcoin Core is probably already running.
    [2017-11-17T11:20:34.971Z] warn: Bitcoin process unexpectedly exited with code: 1
    [2017-11-17T11:20:34.971Z] warn: Restarting bitcoin child process in 5000ms
    [2017-11-17T11:20:35.450Z] warn: ZMQ connection delay: tcp://127.0.0.1:28332
    [2017-11-17T11:20:35.451Z] info: ZMQ connected to: tcp://127.0.0.1:28332
    [2017-11-17T11:20:39.972Z] info: Using bitcoin config file: /home/ajit/.bitcoin/bitcoin.conf
    [2017-11-17T11:20:39.974Z] info: Starting bitcoin process
    [2017-11-17T11:20:39.993Z] warn: Bitcoin process restarted
    Error: Cannot obtain a lock on data directory /home/ajit/.bitcoin/testnet3. Bitcoin Core is probably already running.
    [2017-11-17T11:20:40.008Z] warn: Bitcoin process unexpectedly exited with code: 1
    [2017-11-17T11:20:40.009Z] warn: Restarting bitcoin child process in 5000ms
    [2017-11-17T11:20:40.492Z] warn: ZMQ connection delay: tcp://127.0.0.1:28332
    [2017-11-17T11:20:40.492Z] info: ZMQ connected to: tcp://127.0.0.1:28332
    [2017-11-17T11:20:45.010Z] info: Using bitcoin config file: /home/ajit/.bitcoin/bitcoin.conf

When i open insight-UI . the UI opens , i got few transactions but did not got blocks list and within 10 sec bitcore stops with this log
[2017-11-17T11:20:50.049Z] info: Starting bitcoin process
[2017-11-17T11:20:50.065Z] warn: Bitcoin process restarted
Error: Cannot obtain a lock on data directory /home/ajit/.bitcoin/testnet3. Bitcoin Core is probably already running.
[2017-11-17T11:20:50.087Z] warn: Bitcoin process unexpectedly exited with code: 1
[2017-11-17T11:20:50.087Z] warn: Restarting bitcoin child process in 5000ms
[2017-11-17T11:20:50.140Z] error: uncaught exception: { Invalid state: Invalid length while reading varlength buffer. Expected to read: 3307054118928383 and read 74879
Error
    at new NodeError (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/errors/index.js:20:41)
    at Object.checkState (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/util/preconditions.js:9:13)
    at BufferReader.readVarLengthBuffer (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/encoding/bufferreader.js:141:5)
    at Function.Input.fromBufferReader (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/transaction/input/input.js:95:28)
    at Transaction.fromBufferReader (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/transaction/transaction.js:307:23)
    at Function._fromBufferReader (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/block/block.js:92:42)
    at Function.fromBufferReader (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/block/block.js:103:20)
    at Function.fromBuffer (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/block/block.js:112:16)
    at Function.fromString (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/block/block.js:121:16)
    at /home/ajit/.bitcoin/node_modules/bitcore-node/lib/services/bitcoind.js:1686:40
  message: 'Invalid state: Invalid length while reading varlength buffer. Expected to read: 3307054118928383 and read 74879',
  stack: 'Invalid state: Invalid length while reading varlength buffer. Expected to read: 3307054118928383 and read 74879\nError\n    at new NodeError (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/errors/index.js:20:41)\n    at Object.checkState (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/util/preconditions.js:9:13)\n    at BufferReader.readVarLengthBuffer (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/encoding/bufferreader.js:141:5)\n    at Function.Input.fromBufferReader (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/transaction/input/input.js:95:28)\n    at Transaction.fromBufferReader (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/transaction/transaction.js:307:23)\n    at Function._fromBufferReader (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/block/block.js:92:42)\n    at Function.fromBufferReader (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/block/block.js:103:20)\n    at Function.fromBuffer (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/block/block.js:112:16)\n    at Function.fromString (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/block/block.js:121:16)\n    at /home/ajit/.bitcoin/node_modules/bitcore-node/lib/services/bitcoind.js:1686:40' }
[2017-11-17T11:20:50.144Z] error: Invalid state: Invalid length while reading varlength buffer. Expected to read: 3307054118928383 and read 74879
Error
    at new NodeError (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/errors/index.js:20:41)
    at Object.checkState (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/util/preconditions.js:9:13)
    at BufferReader.readVarLengthBuffer (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/encoding/bufferreader.js:141:5)
    at Function.Input.fromBufferReader (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/transaction/input/input.js:95:28)
    at Transaction.fromBufferReader (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/transaction/transaction.js:307:23)
    at Function._fromBufferReader (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/block/block.js:92:42)
    at Function.fromBufferReader (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/block/block.js:103:20)
    at Function.fromBuffer (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/block/block.js:112:16)
    at Function.fromString (/home/ajit/.bitcoin/node_modules/bitcore-lib/lib/block/block.js:121:16)
    at /home/ajit/.bitcoin/node_modules/bitcore-node/lib/services/bitcoind.js:1686:40
[2017-11-17T11:20:50.144Z] info: Beginning shutdown
[2017-11-17T11:20:50.145Z] info: Stopping insight-ui
[2017-11-17T11:20:50.145Z] info: Stopping insight-api
[2017-11-17T11:20:50.146Z] info: Stopping web
[2017-11-17T11:20:50.146Z] info: Stopping bitcoind
[2017-11-17T11:20:50.151Z] info: ::1 "GET /insight-api/blocks?limit=5" 400 29 1.819 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36"
[2017-11-17T11:20:50.564Z] warn: ZMQ connection delay: tcp://127.0.0.1:28332
[2017-11-17T11:20:50.565Z] info: ZMQ connected to: tcp://127.0.0.1:28332
[2017-11-17T11:20:55.087Z] info: Using bitcoin config file: /home/ajit/.bitcoin/bitcoin.conf
[2017-11-17T11:20:55.089Z] info: Starting bitcoin process
[2017-11-17T11:20:55.096Z] error: Error: Stopping while trying to spawn bitcoind.
    at /home/ajit/.bitcoin/node_modules/bitcore-node/lib/services/bitcoind.js:905:25
    at /home/ajit/.bitcoin/node_modules/bitcore-node/node_modules/async/lib/async.js:676:51
    at /home/ajit/.bitcoin/node_modules/bitcore-node/node_modules/async/lib/async.js:726:13
    at /home/ajit/.bitcoin/node_modules/bitcore-node/node_modules/async/lib/async.js:52:16
    at /home/ajit/.bitcoin/node_modules/bitcore-node/node_modules/async/lib/async.js:264:21
    at /home/ajit/.bitcoin/node_modules/bitcore-node/node_modules/async/lib/async.js:44:16
    at /home/ajit/.bitcoin/node_modules/bitcore-node/node_modules/async/lib/async.js:723:17
    at /home/ajit/.bitcoin/node_modules/bitcore-node/node_modules/async/lib/async.js:167:37
    at /home/ajit/.bitcoin/node_modules/bitcore-node/node_modules/async/lib/async.js:652:25
    at /home/ajit/.bitcoin/node_modules/bitcore-node/lib/services/bitcoind.js:887:16
Error: Cannot obtain a lock on data directory /home/ajit/.bitcoin/testnet3. Bitcoin Core is probably already running.
[2017-11-17T11:21:05.149Z] error: Failed to stop services: Error: bitcoind process did not exit

it seems to be that there are some bitcoind processes that have zombied out and listening to your port. try to run this command to kill the other processes
killall -9 bitcoind

Bitcore has made an implementation of bitcoind and named it as bitcored. The fact that you are running bitcoind first and then bitcored, bitcore is trying to access the same ports as bitcoind. Also the lock file is present in the ~/.bitcoin/testnet3/ folder. Hence the conflict.
You do not need to run bitcoind.


text:
Help with bitcoin gold explorer: using explorer after fork
I tried to view my free BTG using the bitcoin address, but it doesn't recognize the address. The explorer was at: btgexplorer.com.  Yet, when I try the same thing at bitcoingold.org, there they have a utility titled "CHECK YOUR ADDRESS BALANCE BEFORE BLOCK 491407", it works fine. What is the problem with the explorer? I also get the same error with the explorer at btgexp.com.

Voting to close as this needs to be raised with the individuals providing the service. It does sound like a bug.

It happens on two different explorers, so I wonder if there's some kind of manipulation that is needed to convert my Bitcoin address to a Bitcoin Gold address. The Bitcoin address was 15R8nVATQzSpp7tHMdAwQBcPimNTYXNz2J

It may also be that they used the same buggy code. I see that btgexplorer.com doesn't even recognise the address as a valid address; it gives error "Unrecognized search pattern".

Have you used a bitcoin gold explorer that works?  Where can I find it? By the way, the address I posted above has .0006 bitcoin and so it should have .0006 bitcoin gold, since this account was established on 8/27/17, way before the fork. Thanks.

There doesn't seem to be any way to get help from the developers of this explorer. And I did not have the same problem when Bitcoin Cash was created; every explorer I tried worked fine for Bitcoin Cash.

I can just confirm the same issue: I have not found a BTG explorer that works on any of my addresses (that all had BTC at the time of the split)

btgexp.com sometimes doesn't even find addresses it presents itself (i.e. look up an old transaction, then click one of the addresses, you sometimes get the same "not found" error). Oliver's bitinfocharts link works well.

How can this question be classified when it is asking for a working BTG explorer? Something very basic and needed for anyone handling BTG but still currently hard to find

*meant "How can this question be put on hold as off topic"

After more googling I found this one: https://bitinfocharts.com/bitcoin%20gold/
It displays the correct BTG balances for my addresses at least

thank you, thank you, thank you. I wish some developer would explain why this is  occurring.  But this was a BIG relief.

note that when you enter a valid BTC address it shows you a BTG balance with the corresponding BTG address, so there is definately some BTC->BTG addresses translation logic going on.

I just noticed the same thing. It reports the correct balance but it does not report the address correctly.


text:
Multiple input/output transactions using golang btcd
I am using golang's btcd as a basis for creating some server-side tools for handling transactions. I found a good example for creating a basic one-input-one-output transaction to get me started on the implementation, and I have also read the documentation heavily on the technical specifics here.  I have also looked at the other btcsuite libraries pretty heavily to try to gain more insight. To be clear, I am NOT expecting any of these tools to issue the transaction or anything like that; I understand that this was written out of btcd in particular for very, very good reasons. I am just trying to use these tools to create valid transactions that I can issue later.
I am still incredibly confused on the particulars in implementing multiple input, multiple output transactions and the scripts that we use on them.  A few topics:

I'm familiar with the script tools in btcd, the link goes to the most common one used, but all of these scripts are used for one specific output. There seems to be absolutely no documentation anywhere about how to provide scripting for multiple outputs. If I'm doing a normal P2PK or P2PKH transaction, do I simply concatenate the scripts for all the various outputs, or what?
In the create Transaction example in the HelloBitcoin example I've linked above, the overall magic happens near the end with:
buffer.Write(version)
buffer.Write(inputs)
buffer.Write(inputTransactionBytesReversed)
buffer.Write(outputIndexBytes)
buffer.WriteByte(byte(scriptSigLength))
buffer.Write(scriptSig)
buffer.Write(sequence)
buffer.Write(numOutputs)
buffer.Write(satoshiBytes)
buffer.WriteByte(byte(scriptPubKeyLength))
buffer.Write(scriptPubKey)
buffer.Write(lockTimeField)

And I feel like there are a lot of unanswered questions on how to expand this to handle multiple inputs and multiple outputs. For the inputs, is it as simple as taking all the input transactions into little-endian, noting the correct number in inputs, and going from there? For outputs, do you just have an individual block of to satoshiBytes to scriptPubKey? I can't find in the documentation  how this is supposed to all glue together, and at this point I feel like I need some help with the nuts and bolts of this. Thanks in advance for any help - if we can get these two points answered I might end up making more questions from the stuff that will inevitably result from this.

There might not be an easy answer, but the developper documentation is probably a first stop (https://bitcoin.org/en/developer-examples#complex-raw-transaction). Also I used regularily a block explorer (e.g. blockchain.info) to display the transaction and also its "raw" format. This allows to see the tx details in it's JSON format. Also the book from Andreas helped alot (Mastering Bitcoin). And finally there is also this forum here (the search function is good), and bitcointalk.org. Oh, then there is the wiki as well: https://en.bitcoin.it/wiki/Protocol_documentation#tx.

For the inputs, is it as simple as taking all the input transactions
  into little-endian, noting the correct number in inputs, and going
  from there?

No, I don't think so. There is no such thing as simple :-)
You are on the right path, additionally when assembling a raw, unsigned transaction, and you have to deal with field lengths, and with "var_int", which are explained in developper docs. Here is a sample tx, so you get an idea:
VERSION                                 01000000
TX_IN COUNT [var_int]:                  hex=03, decimal=3
 TX_IN[0]
  TX_IN[0] OutPoint hash (char[32])     94FAE0AC28792796063F23F4A4BA4F977A9599D1579C5AAE7CE6DDA4F8A6B1BB
  TX_IN[0] OutPoint index (uint32_t)    hex=14040000, reversed=00000414, decimal=1044
  TX_IN[0] Script Length (var_int)      hex=19, decimal=25
  TX_IN[0] Script Sig (uchar[])         76A914A438060482FCD835754EA4518C70CC2085AF48FA88AC
  TX_IN[0] Sequence (uint32_t)          FFFFFFFF
 TX_IN[1]
  TX_IN[1] OutPoint hash (char[32])     A3E719B12275357B15FC5DECD9088A0964FE860D49F026F2152E71F681AC3FA4
  TX_IN[1] OutPoint index (uint32_t)    hex=31040000, reversed=00000431, decimal=1073
  TX_IN[1] Script Length (var_int)      hex=19, decimal=25
  TX_IN[1] Script Sig (uchar[])         76A914A438060482FCD835754EA4518C70CC2085AF48FA88AC
  TX_IN[1] Sequence (uint32_t)          FFFFFFFF
 TX_IN[2]
  TX_IN[2] OutPoint hash (char[32])     874CD4C4E1683C43A98A9DAA0926BEA37C10616F165AC35481E8181BFD449C65
  TX_IN[2] OutPoint index (uint32_t)    hex=E0010000, reversed=000001E0, decimal=480
  TX_IN[2] Script Length (var_int)      hex=19, decimal=25
  TX_IN[2] Script Sig (uchar[])         76A914A438060482FCD835754EA4518C70CC2085AF48FA88AC
  TX_IN[2] Sequence (uint32_t)          FFFFFFFF

TX_OUT COUNT                            hex=01, decimal=1
 TX_OUT[0]
  TX_OUT[0] Value (uint64_t)            hex=8038010000000000, reversed_hex=0000000000013880, dec=80000, bitcoin=0.00080000
  TX_OUT[0] PK_Script Length (var_int)  hex=19, dec=25
  TX_OUT[0] pk_script (uchar[])         76A914C2DF275D78E506E17691FD6F0C63C43D15C897FC88AC
LOCK_TIME                               00000000

For outputs, do you just have an individual block of to satoshiBytes
  to scriptPubKey?

It is similiar to the example above, just don't have one at hand. But the majority of tx have a spend-to address, and a return address, so you might look a tx up in the block explorer...
Happy coding ! :-)

Did you create this particular transaction? If so, can I get an idea of how you went about doing it? I don't care if it's in golang or not, I'm just trying to get a feel for how all of these pieces glue together.

yup, manually assembled with a set of shell scripts. When you are on unixoide systems, they work with Linux, MacOS and OpenBSD. Here: github.com/pebwindkraft/trx_cl_suite


text:
Is it possible to use Bitcoin-core for multisig? Can it handle change addresses?
I have tried to import three xpub in a watch-only wallet as keypool with importmulti and a multisig output descriptor and to fill the first address with some BTCs. When I try to spend the newly created multisig UTXO, creating a walletcreatefundedpsbt, there's an error instead of creating a psbt.
For context, my intention is to protect the wallet with three Hardware wallets. I' m doing this tests in regtest.
Here are the details:
I have generated three xpub:
pubkey1 tpubD6NzVbkrYhZ4YKMs9TvzemC7FaKYnNxESKvYN9V6PDnDwBXrHQk97rNC71fCXxyobFgcZDfesc5ThCMkUYjA5edd8J1qUQoR4P3kPywrGw6
pubkey2 tpubD6NzVbkrYhZ4X2MQYSo3wVWBPGjV3fUcN4DywfR2ZcM1F56A2bapN54YJqa47yDvwuTajWqkBHxRmdGHLQJj5ae7V1NXDN2JP8AKcyurzsa
pubkey3 tpubD6NzVbkrYhZ4XYeFSXzJ9fFu91eiSEpanzbhifxuSAb7m4piv9QxXsHi2a9jnspQpeKpnpdsyNhoMWb6traBXErL9TPjP3EKnPVFuzyv7Ed

then I have created a disable_private_keys wallet.
bitcoin-cli -regtest createwallet multi

then imported the output descriptors in the new wallet.
bitcoin-cli -regtest -rpcwallet=multi importmulti [{"desc": "wsh(multi(2,tpubD6NzVbkrYhZ4YKMs9TvzemC7FaKYnNxESKvYN9V6PDnDwBXrHQk97rNC71fCXxyobFgcZDfesc5ThCMkUYjA5edd8J1qUQoR4P3kPywrGw6/1/*,tpubD6NzVbkrYhZ4X2MQYSo3wVWBPGjV3fUcN4DywfR2ZcM1F56A2bapN54YJqa47yDvwuTajWqkBHxRmdGHLQJj5ae7V1NXDN2JP8AKcyurzsa/1/*,tpubD6NzVbkrYhZ4XYeFSXzJ9fFu91eiSEpanzbhifxuSAb7m4piv9QxXsHi2a9jnspQpeKpnpdsyNhoMWb6traBXErL9TPjP3EKnPVFuzyv7Ed/0/*))#hnd2wec3", "internal": true, "range": [0, 10], "timestamp": "now", "keypool": true, "watchonly": true},{"desc": "wsh(multi(2,tpubD6NzVbkrYhZ4YKMs9TvzemC7FaKYnNxESKvYN9V6PDnDwBXrHQk97rNC71fCXxyobFgcZDfesc5ThCMkUYjA5edd8J1qUQoR4P3kPywrGw6/0/*,tpubD6NzVbkrYhZ4X2MQYSo3wVWBPGjV3fUcN4DywfR2ZcM1F56A2bapN54YJqa47yDvwuTajWqkBHxRmdGHLQJj5ae7V1NXDN2JP8AKcyurzsa/0/*,tpubD6NzVbkrYhZ4XYeFSXzJ9fFu91eiSEpanzbhifxuSAb7m4piv9QxXsHi2a9jnspQpeKpnpdsyNhoMWb6traBXErL9TPjP3EKnPVFuzyv7Ed/0/*))#6jvyd6m3", "internal": false, "range": [0, 10], "timestamp": "now", "keypool": true, "watchonly": true}]

I have filled the first multisig address:
bitcoin-cli -regtest -rpcwallet="" sendtoaddress $(bitcoin-cli -regtest -rpcwallet=multi getaddressesbylabel ""|jq -r 'keys|.[0]') 10
bitcoin-cli -regtest -rpcwallet="" generatetoaddress 1 $(bitcoin-cli -regtest -rpcwallet="" getnewaddress)

but then, when I do:
bitcoin-cli -regtest -rpcwallet=multi walletcreatefundedpsbt [] "{\"$(bitcoin-cli -regtest -rpcwallet="" getnewaddress)\":0.5}" 0 "{\"subtractFeeFromOutputs\":[0], \"includeWatching\":true}"

I get:
    error code: -4
error message:
Can't generate a change-address key. No keys in the internal keypool and can't generate any keys.

What am I doing wrong?
Is there a way to manage the internal keypool so to have a multisig whatch-only wallet I can use with two or three HWW?
Thank you.


text:
What does an un-upgraded node see when new Taproot transactions are on chain?
Nodes that don’t know about Taproot need to be able to validate the blocks with Taproot transactions. Marking them as valid spends is the only option for not forking the network.
Yet, if the transactions are valid, the old nodes have no idea where those coins got spent. Are they seen as lost coins?
What happens when a Taproot transaction is followed by a old style transaction such that UTXOs are back to being available for spending as an old-style transactions?

Yet, if the transactions are valid, the old nodes have no idea where those coins got spent. Are they seen as lost coins?

Old nodes do know where the coins are going, as the transaction format didn't change. It has inputs and outputs. Taproot inputs are seen as valid by pre-taproot nodes (including taproot-invalid taproot inputs, of course, as they can't verify the rules). But nothing changed about the rest; pre-taproot nodes still:

Require that UTXOs referenced as inputs exist, and mark them spent
Create new UTXOs for all outputs in the transaction
Enforce the sum of the input amounts is at least equal to the output amounts
...

What happens when a Taproot transaction is followed by a old style transaction such that UTXOs are back to being available for spending as an old-style transactions?

There is no such thing as a taproot transaction. Taproot validation rules are a per-input concept, local to its script validation. Everything else about a transaction with a taproot input is unchanged.
The successor transaction is just validated normally.


text:
Would quantum computing be able to deanonymize CoinJoin transactions?
Using the worst case scenario of this question, namely:

Bitcoin ECDSA algorithm would be broken. Because quantum computers can easily decrypt the private key using the public key, anyone with a quantum computer can extract Bitcoins using the corresponding public key.

Given that quantum computers would be able to obtain private keys, would they be able to deanonymize CoinJoin transactions that are already present in the blockchain?

No.
CoinJoin does not rely on any cryptography to hide transaction flows. It merely combines multiple transactions into one, and randomizes the order of inputs and outputs.
Of course, as you point out, quantum computing would allow stealing ECDSA outputs...


text:
Do all wallets use specific explorers?
Do all wallets use specific explorers? If so, is there a way to identify which explorer the wallet is using? Most of the major-mid tier wallets seem very ambiguous or don't seem to mention this at all.

Why would a wallet have an explorer? Explorers are independently run websites, and have nothing to do with particular wallet software.

I'm sorry my question could have been ambiguous. I wondered if wallets used specific explorers, and if so, if there was a way to identify which.

Given the tags you're setting on your question, I believe you're confusing currencies, wallets, and explorers.
Bitcoin (BTC) and Bitcoin Cash (BCH) are currencies with an associated blockchain.
Wallet/node software is what you use to interact with that currency.

Software that interacts with BTC includes Bitcoin Core, Bitcoin Knots, btcd, Electrum, Greenbits, ...
Software that interacts with BCH includes Bitcoin ABC, BUCash, Electron Cash, ...

Explorers are independently run websites that visualize information in particular blockchains. They are unrelated to wallets, and generally something you should avoid for anything but debugging purposes. Relying on a specific website as your source of truth is dangerous as the website may lie. Wallet software generally performs some level of local validation of transactions (or even full validation, meaning that no invalid transactions will ever be accepted). Some explorers have associated wallet software, which relies on that website for information, rather than directly connecting to the network. These are generally discouraged for the same reason.

Thank you Pieter. I'm just getting started on my learning process and your comment has been a great help.


text:
Why is bitcoin written in C++?
What made the core developers choose C++ as the main programming language for implementing bitcoin? Was this because of their personal preference? Or another reason?

The correct answer would be: find the author and ask him :) But even then, the truth is always the same: developers choose the programming languages they know and love

The most important reason is that the original Satoshi codebase was written in C++. A system like Bitcoin has extremely strong requirements about consistency in behaviour: all nodes on the network must accept and reject the exact same blocks, or we risk a fork. The first choice to avoid bugs that risk introducing inconsistency is to not rewrite the code from scratch.
Although all programming languages have flaws, and C++ is no exception, there are various reasons that make it an appropriate choice. An easily-overlooked one is that it provides very tight control over memory usage. Many modern languages (especially those that rely on garbage collection) make it hard if not impossible to reason about tight bounds of memory usage. Since Bitcoin Core is a security-critical application that is directly exposed to the Internet, it is nontrivial to guarantee that there is no way to trigger it to allocate large amounts of memory, thrashing the machine's swap space, or worse, killing the process.
This is in addition to a good balance between high-level where desired but also many avenues for performance optimization. Performance may be more important than is immediately obvious, as the convergence of the network relies on quick validation and propagation of blocks. If blocks take too long to validate compared to their creation frequency, economic factors start to unfairly advantage larger/faster miners (see the Selfish Mining attack, for example). Under extreme conditions, if blocks would take ~minutes to validate, the network may fail to converge at all, as nodes simply don't hear about blocks in time anymore.

"A system like Bitcoin has extremely strong requirements about consistency in behaviour" which is exactly why Ethereum promotes a diversity of implementations. The Satoshi codebase formed a de facto standard, to write a competing implementation is to stress-test the specification.

"make it hard if not impossible to reason about tight bounds of memory usage"  Right, because having tight control over memory usage has never caused security issues...  http://heartbleed.com/

@spraff And everyone runs the same one, because they're afraid of differences. In consensus systems, variety simply doesn't work. Even the satoshi client cannot just change rules of the network.

@NPSF3000 Fair point, but modern C++ hardly ever needs manual memory management (as opposed to C), though it does become harder when you want to avoid relying on over-eager copying.

I think the garbage collection fanboys (Java?) are going to disagree.

@PieterWuille When Ethereum becomes more widely adopted on different devices, diversity of clients will be more normal. Simply having a variety of clients in parallel development is useful for finding bugs in the design, specification, and implementation, even if hardly anyone uses them in practice.

Sure, and Bitcoin also has a variety of clients (even Bitcoin Core's unit tests are reimplemented in Python), which helps with finding problems. But it's not a guarantee, and you shouldn't use reimplementations in production. That's just asking for problems. N different implementations means ~N^2 combinations of sources of possible inconsistencies.

"it is nontrivial to guarantee that there is no way to trigger it to allocate large amounts of memory, thrashing the machine's swap space, or worse, killing the process" -> Sounds like the argument for GCs!

I gave a keynote address at cppcon 2016 about almost this exact issue. There are a variety of reasons why C++ is an excellent language choice for blockchain applications like Bitcoin.
Blockchain applications have a large attack surface. They're meant to interoperate with a large number of untrusted endpoints while still providing reliable service to local clients. This requires keeping tight control over resources like memory and CPU usage.
Modern CPUs have lots of cores. Some parts of blockchain applications parallelize perfectly (like checking digital signatures) while some parts don't parallelize at all (like executing transactions in order). Modern C++ has a very good mix of effective inter-thread communication and optimization of single-thread performance.
C++ doesn't have a run time that stops the world or manages memory, giving your application consistent control over this behavior. Move semantics allow you to get objects that behave like values (a=b; a+=1; does not change b) but perform like references (deep copies are avoided except where needed).
Smart pointers allow you to avoid the costs of GC without the tedium of manual memory management. Design patterns like RAII makes code easier to maintain. Template metaprogramming facilitates independence between APIs and implementations.
C++'s inheritance scheme makes the design of complex type hierarchies easier.
The language is both mature and maintained. The compiler technology is extremely solid, yet new features are still being added. And the new features are aimed at solving real issues. Debuggers and analytical tools of all kinds are available for everything from performance profiling to automatic detection of issues of all kinds.

Upvoted. This goes into more detail than my own answer.

But to summarize, there's no mandatory point why it has to be C++ right?

No. Other languages have advantages and disadvantages, but I think performance would have suffered with any other choice (but C).

C is not the only language that compiles to machine code

"C++ doesn't have a run time that stops the world or manages memory, giving your application consistent control over this behavior."-> but it does! C++'s runtime is the one taking care of freeing memory when you don't do it explicitly. So you're actually pitting your ad-hoc, unnoticed memory management with a purposefully developed GC – which would have saved you from a whole class of bugs. And, according to C++11's memory model author Hans Boehm, you are possibly causing worse pauses than those caused by a GC.


text:
What data does the Bitcoin Core wallet search for during a blockchain rescan?
When the wallet examines a transaction for relevance what does it do exactly with that transaction's output scripts and what is that data compared to? Does the wallet database store a table of all the actual output scripts (i.e. witness programs) derived from its keypool?
I'm especially curious about scanning tx inputs to recognize coins we have spent. Do we only check outpoints stored in the walletdb? Or do we go as far as inspecting input scripts and witness stacks for our own public keys?

Does the wallet database store a table of all the actual output scripts (i.e. witness programs) derived from its keypool?

Kind of.
For descriptor wallets, the set of all scriptPubKeys (up to the keypool limit for each descriptor) is generated and held in memory. When determining whether a transaction output belongs to the wallet, the wallet checks whether the output's scriptPubKey is in that set. If it is, then the output belongs to the wallet.
One thing to note though is that this scriptPubKey set is not stored in the database. Once the wallet is unloaded, the set is forgotten. It will be recomputed every time the wallet is loaded.
For legacy wallets, it's a lot more complicated. It will determine the type of script and then do lookups for scripts and keys in the wallet depending on the type. For the most part, it does boil down to a finite set of scripts, but this can actually be a very large set that contains a lot of scripts that you might not expect it to.

I'm especially curious about scanning tx inputs to recognize coins we have spent. Do we only check outpoints stored in the walletdb?

Yes. For inputs, the wallet will only check to see if the referenced output exists in the wallet and that output belongs to the wallet. This also means that rescanning must be done serially. If the wallet scanned a transaction that contains a spend of an output that belongs to it, but it hadn't scanned the transaction creating that output yet, then it would not detect the spending transaction.
One unexpected aspect of the Bitcoin Core wallet is that it doesn't explicitly mark which outputs actually belong to it. If it needs to know whether an output belongs to the wallet, it will check the scriptPubKey against the in-memory scriptPubKey set. Similarly, it doesn't explicitly permanently mark outputs as being spent. If you were to remove the spending transaction from your wallet using removeprunedfunds, the output would suddenly appear to be unspent and your wallet could make a transaction spending it. Of course, this would also make an invalid transaction.

Does the wallet database store a table of all the actual output scripts (i.e. witness programs) derived from its keypool?

Yes, exactly. The descriptors (and for legacy wallet, legacy data) is expanded into a set of scriptPubKeys to look for.

A transaction output is ours if its scriptPubKey matches an entry in that set.
A transaction input is ours if it spends an output that is ours.
A transaction is ours if it has at least one input or output that is ours.

Note that this does not require examining witness data for inputs. It suffices to look for spends of our outputs (by txid:vout).

A wallet examines a transaction for relevance by checking if any of the outputs or inputs are related to its own addresses or scripts. To do this, it needs to know the output scripts (or scriptPubKeys) and the corresponding private keys or scripts that can unlock them. Output descriptors1 are a way of encoding this information in a human-readable string that can be imported or exported by wallets.


text:
If a block contains no SegWit transactions, does a miner send an (empty) witness?
Also, would that block's Coinbase contain some kind of empty witness commitment?

BIP141 only requires a commitment to the witness tree to be present when there is at least one segwit transaction (= a transaction that spends a witness output) in the block.
This means that old miners in theory can keep working after segwit activates, as long as they don't include segwit transactions (easy, as they are nonstandard to them) and don't build on top of invalid blocks (harder, as it requires an updated full node).

I'm playing with this in regtest mode, but I'm seeing a SW commitment in an empty block. getblock: http://pastebin.com/eDeXGHEx and gettransaction of the coinbase tx: http://pastebin.com/3DxSFArc The block has an extra 36 bytes (size - stripped size) and the coinbase tx has an OP_RETURN aa21a9ed... Is it counting the coinbase tx itself as a SW tx? Making the whole block SW?


text:
Which block explorers recognize bech32 addresses?
At this time, blockchain.info doesn't. Neither does blockchair or blockcypher.

Blockchair announced support for Bech32 addresses last month.
This page about Bech32 adoption on bitcoin.it wiki page is tracking blockexplorers that support Bech32 addresses.

Looks like blockchain.info can decode bech32 if you select "show scripts..." aka Advanced mode. But doesn't seem to be able to search for a bech32 address

https://blockchain.info/tx/c23248b87ae5f1533e62d4e5f99ac4373a209a38050ac78b1c84b8b7b8d91b1f but yes, it can't search.


text:
How are the seed nodes chosen?
I'm currently doing a research project about Bitcoin and I've particularily been investigating the seed nodes and I have two questions about them :

How are they chosen ? Well, they have to be nodes with high disponibility (bandwith maybe?) but is there any other criterion ?
Why are some of the nodes located in the network of big companies ? Isn't it a bit contradictory to Bitcoin's spirit to have the ip address of some major financial companies hardcoded in its client ?

Thanks for your answers :)
Ayla

Specifically, you are asking why were the IP addresses that are served by the hosts in the DNSSeed list chosen over other IP addresses: http://en.bitcoin.it/wiki/Satoshi_Client_Node_Discovery.

I wasn't talking about the one from the DNSSeed list, but about the hardcoded one. Such as : 0xe473042e, 0xb177f2ad, 0xd63f3fb2... (net.cpp)

According to the Bitcoin wiki https://en.bitcoin.it/wiki/Satoshi_Client_Node_Discovery

Nodes discover their own external address by various methods.
Nodes discover their own external address by various methods.
Nodes makes DNS request to receive IP addresses.
Nodes can use addresses hard coded into the software.
Nodes exchange addresses with other nodes.
Nodes store addresses in a database and read that database on startup.
Nodes can be provided addresses as command line arguments
Nodes read addresses from a user provided text file on startup

Because this is peer-to-peer technology the client is usually connected to more than one node at a time. The reference client validates every bit of information it receives. If a financial company was to go rouge and send false or incomplete data the software would realise the data is invalid and rely on the other connected nodes to operate.


text:
Is Bitcoin vulnerable to a Birthday Attack against Private Keys?
This is a definition of the Birthday Attack.
Someone claims Bitcoin is vulnerable to such an attack in the bitcoin IRC chat.
Are they correct?


text:
Long term cost of using cryptocurrencies
I'm not an economist but....
What are some way of determining whether cryptocurrencies will be a cost effective way of trading compared to some established fiat currency like USD?
In other words, might the cost of computing power and other costs necessary to trade cryptocurrency exceed the overhead costs of trading in say dollars?
What are some economics vocab words that are useful in this question?

I see what you are trying to convey. What is the cost of doing business in the Bitcoin compared to doing business with say the US Dollar? The Dollar was never designed to be ported into an international digital spectrum. It requires a massive support network of banks, regulations, payment processors and international agreements to permit Online commerce, but even then it suffers from fraud and sovereign manipulation.
The Bitcoin resolves a huge set of issues the dollar suffers from at the cost of global transaction verification and the risk of a deflationary spiral. In short, no one can legitimately assert to fully understand the impact of the Bitcoin and its long term use. It's an experiment.
I would encourage you to take my course on Bitcoins on Udemy when I finish it at the end of this week. Send me an email at Charles.Hoskinson@gmail.com if you're interested.


text:
Synchronising with network
I'm completely new to Bitcoin but finding it all very interesting so far! My Bitcoin-qt wallet has been synchronising for literally days now (2 days actually) but it is getting closer, on 91% now. My question is, if I was to buy a Bitcoin from MtGox or somewhere, would I have to wait for the synchronisation to complete before I could see my coins? My transfer would have to be confirmed on the block chain through mining and would be the latest on the chain meaning I'd have to wait until it came through right? Would you wait it out?
Thanks,

possible duplicate of I didn't receive a Bitcoin Payment. What does it mean when the Client is synchronizing?

If you were to purchase bitcoin and send it to an address in your wallet, then yes, you would have to wait until your wallet synchronized in order to spend the bitcoins.
However, in the meantime, consider spending it directly from your exchange account, especially if you're going to buy bitcoin and spend it right away.
Alternatively, consider using an online wallet. I recommend Blockchain.info or InstaWallet.


text:
RippleAPI how to issue IOUs using preparePayment
I'm trying out the API, and I've set up one 'user' account, and one 'gateway' account. The only difference between the two is that the user has extended a trustline to the gateway account. I would now like to issue USD IOUs from the gateway to the user account.
I assume the functionality for this is built into api.preparePayment(). So far, I am able to transfer XRP using code found in the payment example of ripple-lib. The essence is:
const address = SENDER_ADDRESS;
const secret = SENDER_SECRET;

const payment = {
  source: {
    address: address,
    maxAmount: {
      value: '0.01',
      currency: 'XRP'
    }
  },
  destination: {
    address: RECEIVER_ADDRESS,
    amount: {
      value: '0.01',
      currency: 'XRP'
    }
  }
};

api.connect().then(() => {
  console.log('Connected...');

  return api.preparePayment(address, payment, instructions).then(prepared => {
    console.log('Payment transaction prepared...');
    const {signedTransaction} = api.sign(prepared.txJSON, secret);
    console.log('Payment transaction signed...');
    api.submit(signedTransaction);
  });

A few lines of code are missing but this is the basic structure. It works when I make either the Gateway or the user the sender, and the other the receiver. How should I modify it so that I can issue USD IOUs from the Gateway to the user?
My trustline object was:
var trustline = {
  "currency": "USD",
  "counterparty": GATEWAY_ADDRESS,
  "limit": "100",
  "qualityIn": 0.91,
  "qualityOut": 0.87,
  "ripplingDisabled": true,
  "frozen": false,
};

Okay I figured it out: first, I set up a trustline from USER_ADDR to GATEWAY_ADDR for 1000 EUR. I made sure that the qualityIn and qualityOut were set to 0, the default value:
var trustline = {
  "currency": "EUR",
  "counterparty": GATEWAY_ADDR,
  "limit": "1000",
  "ripplingDisabled": true,
  "frozen": false,
  "memos": [
    {
      "type": "test2",
      "format": "plain/text",
      "data": "texted data2"
    }
  ]
};

I signed and submitted this trustline.
I then used the following payment object to create an IOU for 50 EUR from the gateway to the user, indicating that the user had deposited 50 EUR with the gateway outside the Ripple system:
const payment = {
  source: {
    address: GATEWAY_ADDR,
    maxAmount: {
      currency: 'EUR',
      value: '50',
      counterparty: GATEWAY_ADDR
    }
  },
  destination: {
    address: USER_ADDR,
    amount: {
      currency: 'EUR',
      value: '50',
      counterparty: GATEWAY_ADDR
    }
  }
};

Earlier, this was not working due to qualityIn and qualityOut being to 0.91 and 0.87

Since you set the max amount to 50 EUR, the payment will only work if the cost to deliver 50 EUR is 50 EUR or less. If the recipient doesn't value the gateway's IOUs at face value, then it will cost the gateway more the 50 EUR to give the customer 50 EUR. (Ripple only requires issuers to value their own IOUs at face value. You can devalute other people's IOUs.)


text:
OP_CHECKDATASIGVERIFY not working
I'm trying to generate a signature and pass it through OP_CHECKDATASIGVERIFY as part of a P2SH address on a regtest network.  This results in the following:
Public Key:
036ab9012ecc30c30f9220d862a56c5c99b1a08d783be605406fbde8ece2807b2f

Signature:
304402200ac548d1e8b03c7d0bdfe4aedb4b13daacf2b1c2fa265afafae5d963d66a7fa702202cdb383bb3b4b4bc88403337fb2922f5166670c8bccf246e9bd5c963de93fae2

Data Signed:
7f7f9af76890ca09afc49e458a1b23fdd2e29cc061693c4b326b16e40728091a

Clientside (bitcore-lib-cash) says the signature validates and complies with the LowS check, but when submitted to the network the signature fails:
bitcore-lib-cash 0.19.0 => The signature can be verified
Bitcoin-ABC v190100 => 16: mandatory-script-verify-flag-failed (Signature must be zero for failed CHECK(MULTI)SIG operation)
Debugging via bch-deb: https://hastebin.com/gelitojuge.rb
Is there anything that looks out of the ordinary here?
UPDATE: 3/20/19 - note, this is a different, simplified full example tx
Raw TX: There are extra variables being passed in, but only a couple are currently used.
0200000001a2f391ddcecd79218bf16999986a9996d29dfd33105b2921a82c7ca801493dcf00000000fdf7034c770200000001a2f391ddcecd79218bf16999986a9996d29dfd33105b2921a82c7ca801493dcf00000000000000000002d557cd170000000017a9148bf4c913a5942cb96ee1c094cff066ec41d33b9187a2180a000000000017a914f3ce6d3eac22ecdb1df736f412f760811df60fac87000000000100000022143f8147b020f3351da2cf15f133eeb2d9da0724430076b2011479aa5d79785e79bb4cd43038303030303033414e3033303002efbfbdefbfbd70d3adefbfbdefbfbd1c414d08efbfbdefbfbd572e37efbfbdefbfbdefbfbd6cefbfbd35027550efbfbdefbfbd65efbfbdefbfbdefbfbdefbfbd1fefbfbdefbfbd4d49efbfbd5fefbfbdefbfbdefbfbd66efbfbd32efbfbdefbfbd47efbfbd0defbfbdefbfbdefbfbd36efbfbdefbfbdefbfbd49efbfbdefbfbdefbfbd4defbfbdc6be4b1c23efbfbd77efbfbdefbfbd10df92efbfbd32efbfbdefbfbdefbfbdefbfbd687e4464efbfbd3b24efbfbd0925efbfbdefbfbdefbfbd266defbfbd0831376437383430304cbb3038303030303033414e303330303fefbfbd47efbfbd20efbfbd351defbfbdefbfbd15efbfbd33efbfbdefbfbdefbfbdefbfbd0724431fefbfbdefbfbd4d49efbfbd5fefbfbdefbfbdefbfbd66efbfbd32efbfbdefbfbd47efbfbd0defbfbdefbfbdefbfbd36efbfbdefbfbdefbfbd49efbfbdefbfbdefbfbd4defbfbdc6be4b1c23efbfbd77efbfbdefbfbd10df92efbfbd32efbfbdefbfbdefbfbdefbfbd687e4464efbfbd3b24efbfbd0925efbfbdefbfbdefbfbd266defbfbd0561326332614cb73038304130413033414e30333036efbfbd521b44efbfbd0818efbfbdefbfbdefbfbdefbfbd1055efbfbd30efbfbd231975541fefbfbdefbfbd4d49efbfbd5fefbfbdefbfbdefbfbd66efbfbd32efbfbdefbfbd47efbfbd0defbfbdefbfbdefbfbd36efbfbdefbfbdefbfbd49efbfbdefbfbdefbfbd4defbfbdc6be4b1c23efbfbd77efbfbdefbfbd10df92efbfbd32efbfbdefbfbdefbfbdefbfbd687e4464efbfbd3b24efbfbd0925efbfbdefbfbdefbfbd266defbfbd0831376364353764354630440220095afd33cce778a91f78cd1e2125278bc4625eac1bde3567cc9c3a3e357c2b1c022068d61ae2d17b35f24645a4e10b66a33d319d6c7cf188233acf72195d6a63234d2103ca8a4eaca7201da793e7127737990e47d6c813db0b4aa9c729199d30f91a9a054c8c411feedf4d49945fdfeab9669032efc447ab0db6ae87369bbeca4994f9fc4dafc6be4b1c239d77ffb910df92b732858c9cc0687e44648a3b24bd0925b4a0d1266db32102a4b770d3adf69c1c414d08eed6572e37bdfe956c8d35027550b2d0659cf78c88000058535353143f8147b020f3351da2cf15f133eeb2d9da0724430076b2011479aa5d79785e79bb0000000002d557cd170000000017a9148bf4c913a5942cb96ee1c094cff066ec41d33b9187a2180a000000000017a914f3ce6d3eac22ecdb1df736f412f760811df60fac8700000000

Script ASM:
1feedf4d49945fdfeab9669032efc447ab0db6ae87369bbeca4994f9fc4dafc6be4b1c239d77ffb910df92b732858c9cc0687e44648a3b24bd0925b4a0d1266db3 02a4b770d3adf69c1c414d08eed6572e37bdfe956c8d35027550b2d0659cf78c88 0 0 8 3 3 3 3f8147b020f3351da2cf15f133eeb2d9da072443 0 OP_DUP OP_CHECKSEQUENCEVERIFY 20 OP_PICK OP_HASH256 13 OP_PICK OP_OVER 14 OP_PICK OP_CHECKDATASIGVERIFY

Input Script Hash:
bdb4b1cbd79af7137ff1771cea2c3f818bf35884962bbbea6013ec6c75b865b0

UPDATE 3/23
Simplified TX with only data pushes, OP_CDSV, and OP_true: https://hastebin.com/raw/retabokura
Update 3/29/19: My signing process: https://hastebin.com/magehodufu.js

I'm voting to close this question as off-topic because it is not about bitcoin.

On the contrary, markedly on topic: https://bitcoin.stackexchange.com/help/on-topic

Could you please post the transaction here?

Sure, but might be a couple hours until I can pull it together. Thanks for the response in advance.

I think this might be caused because of the SCRIPT_ERR_CLEANSTACK, because there aren't exactly one element in the end, but I'll look at it more detailedly tomorrow. I could get rid of the error by adding LOTS of OP_DROP and an OP_1.

I'll clean up the stack at the end and see if there's any difference on my end...odd that it would throw a CHECKSIG error though.  Thanks again.

Here's a new TX with the added drops.  Assuming that CHECKDATASIG pops three elements off, this should end up with a 1 on the stack: https://hastebin.com/raw/enohelegus (but still same result)

Just checking, but is it the case that your signature is missing a sighash flag at the end?

@UgamKamat I tried adding that, but throws "Non-canonical DER signature" error

OP_CHECKDATASIGVERIFY is correct, and your signature is wrong. It looks like you need help on fixing the signing process that you used, but you haven't included any information on your incorrect signing process.

@MarkLundeberg Here's my signing process: https://hastebin.com/magehodufu.js Thanks in advance.

checking from https://hastebin.com/raw/imodonelic
static void test_sig ( )
{
  const MyKey32 prv ( QByteArray::fromHex ( "39f8ca6c7e7c05aaf9c97e3253fd4c688730b4404d0f502441eb3aacea01b569" ) );
  const MyByteArray pub ( prv.getPublicKeyCompressed ( ) );
  const MyByteArray sig ( QByteArray::fromHex ( "3045022100aecf494d10f3696809972bad0546a7ea65802ebc40dd1f369185fd52eac016c902201b35692e40d6edb21c422a735f217d028da5f14447ccb31c27aec56ee26b426e" ) );
  const MyByteArray dat ( QByteArray::fromHex ( "0200000001f8f9599fc41346a68507bfbdb26b06af464d20ef709075660657b29b7bc6136c01000000000000000002d5b4eb050000000017a914064ebb4529c5cade492e1790ae1f19943e0c6bfd87d2100a000000000017a914064ebb4529c5cade492e1790ae1f19943e0c6bfd870000000001000000" ) );
  qDebug ( ) << "prv =" << prv.toHex ( ).constData ( );
  qDebug ( ) << "pub =" << pub.toHex ( ).constData ( );
  const MyKey32 digest ( dat.sha256d ( ) );
  qDebug ( ) << "hash=" << digest.toHex ( ).constData ( );
  qDebug ( ) << "your values=" << ( digest.verify ( pub, sig ) ? "passed" : "failed" );

// now create correct values

  const MyKey20 addr ( prv.getAddressHashCompressed ( ) );
  MyByteArray csig;
  MyByteArray cpub;
  prv.signHash ( addr, digest, csig, cpub );
  qDebug ( ) << "sig=" << csig.toHex ( ).constData ( );
  qDebug ( ) << "pub=" << cpub.toHex ( ).constData ( );
  qDebug ( ) << "my values=" << ( digest.verify ( cpub, csig ) ? "passed" : "failed" );
}

The output is:
prv = 39f8ca6c7e7c05aaf9c97e3253fd4c688730b4404d0f502441eb3aacea01b569
pub = 02118805dc4fc1bc36c86a606e760fcc3875e215b88306cca33cd549d3d5668339
hash= cc8fc76706ec95500d170cc01e915725e3c35979b00c27e8638b2ef3399f3ce1
your values= failed
sig= 30430220683555525c15dfc685580824d172890d16f24a996fa37ad4cd22d1a49ff61510021f60d78601ed827e0d46fe8043cbc36f054f4a9cb96c481714b8b6d21575d336
pub= 02118805dc4fc1bc36c86a606e760fcc3875e215b88306cca33cd549d3d5668339
my values= passed

Note: my code does not use dECDSA

Regarding not using ECDSA (I def am), is BCH on Schnorr sigs already (ABC 19)?

this is ecdsa. i mean that i do not use deterministic ecdsa, so you will not be able to reproduce the same signature byte-by-byte

Was finally able to get this working (unfortunately outside of the bounty limit).  Thanks again!

values from https://hastebin.com/raw/izefekopej
static void test_sig ( )
{
  const MyKey32 prv ( QByteArray::fromHex ( "0b08a58dd052baad2f342779271b0c8ef92273632dd630b3e1627fdac0e77963" ) );
  const MyByteArray pub ( prv.getPublicKeyCompressed ( ) );
  qDebug ( ) << pub.toHex ( ).constData ( );
}

The output is: 028b00a221a20da77938bacdd793b569728ff4974eeb6c496f30d90128ba939359 while expected is 02118805dc4fc1bc36c86a606e760fcc3875e215b88306cca33cd549d3d5668339
Your public key 02118805d... does not match your private key 0b08a58dd052...
You can easy check it on https://www.bitaddress.org/

Shoot...I miss-copied the Private Key..should have been 39f8ca6c7e7c05aaf9c97e3253fd4c688730b4404d0f502441eb3aacea01b569.  Corrected paste: https://hastebin.com/raw/imodonelic (everything else is the same and private key checks out with bitaddress).

static void test_sig ( )
{
  const MyByteArray pub ( QByteArray::fromHex ( "02118805dc4fc1bc36c86a606e760fcc3875e215b88306cca33cd549d3d5668339" ) );
  const MyByteArray sig ( QByteArray::fromHex ( "3045022100d06f815197d6a7af07e76ba312ab9496acb8aef3ec286098e123c26ce610e4950220547c84f3292b4f1be91ed0a0745c74c4622e3e2aa400e8f2de6d32aacbf7c591" ) );
  const MyByteArray dat ( QByteArray::fromHex ( "0200000001f8f9599fc41346a68507bfbdb26b06af464d20ef709075660657b29b7bc6136c01000000000000000002d5b4eb050000000017a914f60ae3ea9b07c104627a98f9bc03e7268f77a58d87d2100a000000000017a914f60ae3ea9b07c104627a98f9bc03e7268f77a58d870000000001000000" ) );
  const MyKey32 digest ( dat.sha256d ( ) );
  qDebug ( ) << ( digest.verify ( pub, sig ) ? "passed" : "failed" );
}

The result is failed. Your data is not valid for OP_CDSV opcode

Thanks for sticking with this: I'm beginning to think the issue is in how i'm generating the double-sha256.  The following includes both the Data as submitted and the hash as signed: https://hastebin.com/raw/izefekopej.  Bitcore-lib-cash continues to report that the sig is valid.

The signature can be valid, but not for these pubkey and message :)

static void test_sig ( )
{
  const MyByteArray pub ( QByteArray::fromHex ( "036ab9012ecc30c30f9220d862a56c5c99b1a08d783be605406fbde8ece2807b2f" ) );
  const MyByteArray sig ( QByteArray::fromHex ( "304402200ac548d1e8b03c7d0bdfe4aedb4b13daacf2b1c2fa265afafae5d963d66a7fa702202cdb383bb3b4b4bc88403337fb2922f5166670c8bccf246e9bd5c963de93fae2" ) );
  const MyKey32 hash ( MyKey32::fromHex ( "7f7f9af76890ca09afc49e458a1b23fdd2e29cc061693c4b326b16e40728091a" ) );
  qDebug ( ) << ( hash.verify ( pub, sig ) ? "passed" : "failed" );
}

The result of this code is passed
But! OP_CHECKDATASIGVERIFY needs the data itself, not a hash of it.

That hash of the scriptSig is what was originally signed (in order to mirror match OP_CHECKSIG's input). The scriptSig is the first value passed in, and get's hashed to match the value that was signed.  Looks like it might be my ABC.

Please, simplify your code. Remove OP_PICK, OP_OVER, OP_CSV and so on. Leave only push(signature) push(data) push(pubkey) OP_CHECKDATASIGVERIFY OP_TRUE. Note that CLEAN_STACK is mandatory rule for BCHABC

OK -- will do and return with results.  Thanks for the response.

I have simplified the code to only data pushes, CDSV and OP_TRUE: https://hastebin.com/raw/retabokura (same result) I do see now that ABC expects the data before the hash.


text:
How to check Bitcoind block chain download progress level
I have installed bitcoind on ubuntu server, and when I run screen -x for btc it says error -10 blockchain is downloading.
How do I know the progress level? currently 125600 blocks are downloaded, but how many blocks yet to go? where do i check the blockchain download progress level? does it has capability to resume if the server is powered off?

Do
cd /home/

then
nano block.sh

copy the code below and paste (right click if you use putty) to the script.
#!/bin/bash
while true ; do
    clear
    echo "Press enter to break loop. Script will loop every 3 seconds"
    echo "script by Nixsy 18th august 2013"
    echo "If loop freezes press CTRL+C"
    echo ""
    echo -e "    \033[31mdownloaded\e[0m/\033[32mavailable\e[0m"
    echo -e "    \033[31m"`bitcoin-cli getblockcount 2>&1`"\e[0m"/"\033[32m"`wget -O - http://blockchain.info/q/getblockcount 2>/dev/null`"\e[0m"
    read -t 3 -n 3 && break
done

When you have copied the text to nano hold CTRL+X and then press Y to save.
In terminal type or copy from below followed by the enter key.
chmod +x block.sh

to start the script type.
./block.sh

This is just a little edit of the script from Lohoris, It will clear the screen then loop the script every 3 seconds until a key press.
Credit to Lohoris for the original script.

As of 2019, you can use the following script
echo `bitcoin-cli getblockcount 2>&1`/`wget -O - http://blockchain.info/q/getblockcount 2>/dev/null`

Using Bitcoin Core Daemon v0.14.0
Error: There is no RPC client functionality in bitcoind anymore. Use the bitcoin-cli utility instead./463021
Simply replace "bitcoind" with "bitcoin-cli" to see progress.

I just installed bitcoind on an RPi and wanted to do this exact thing. Thought I'd post what worked for me:
$ bitcoin-cli getblockcount
7437

Block height is 366678 at the time of this post...long way to go

here a bash script for see the live progress

create a file nano [nom fichier] and copy the code below
run script with bash [nom fichier]
stop script by type on your keyboard

Script :
sleepTime=4 #set time between updates

blocksPris=$(bitcoin-cli getblockcount)
blocksTotal=$(wget -O - http://blockchain.info/q/getblockcount 2>/dev/null)
pourcentage=$(((blocksPris*100)/blocksTotal))
input=""

tput smcup
clear
\e[25l ou tput civis
stty -echo
tput civis

while ["$input" -eq ""]
do
        echo -en "\e[1A\e[0K\r"

        echo "- $blocksPris/$blocksTotal - [$pourcentage%]"
        i=1
        echo -e "\e[47m \e[42m\c"
        while [ "$i" -lt "$pourcentage" ]
        do
                echo -e "_\c"
                i=$(( $i + 1))
        done
        echo -e "\e[41m\c"
        i=1
        while [ "$i" -lt "$((100-pourcentage))" ]
        do
                echo -e "_\c"
                i=$(( $i + 1))
        done
        echo -e "\e[47m \e[49m\c"

        sleep $sleepTime
        read -t 0.25 -N 1 input

        blocksPris=$(bitcoin-cli getblockcount)
        blocksTotal=$(wget -O - http://blockchain.info/q/getblockcount 2>/dev/null)
        pourcentage=$(((blocksPris*100)/blocksTotal))
done

tput cnorm
stty echo
tput sgr0
tput rmcup

Result:

Just wow! :-)))

I wanted a nicer output.
Based off @o0'. answer ...
BC_CURRENT=`./bitcoin-cli getblockcount 2>&1`; BC_HEIGHT=`wget -O - http://blockchain.info/q/getblockcount 2>/dev/null`; perl -E "say sprintf('Block %s of %s (%.2f%%)', $BC_CURRENT, $BC_HEIGHT, ($BC_CURRENT/$BC_HEIGHT)*100)";

Outputs ...
Block 360693 of 471139 (76.56%)

Nice! Love the output.

tail -f .bitcoin/debug.log

You will see the progress % in the end of every line.
2019-05-21T15:53:22Z UpdateTip: new best=00000000000007463022a75f47fbce4832d885cccfddc66b9ae6d332888f825d height=181720 version=0x00000001 log2_work=68.184645 tx=3482689 date='2012-05-26T22:08:18Z' progress=0.008422 cache=256.7MiB(1838887txo)

Here, in case you did not find it above:
... progress=0.008422 ...

You can see the current total amount of blocks on public block chain browsers:

http://blockexplorer.com/q/getblockcount
http://blockchain.info/q/getblockcount

If you have bitcoin cli installed. The below command will help.
echo `bitcoin-cli getblockcount 2>&1`/`wget -O - http://blockchain.info/q/getblockcount 2>/dev/null`

Sample output

365320/512421

This is slight improvement of above answer. But it returned Error: Command line contains unexpected token 'getblockcount', see bitcoind -h for a list of options./512421
 in my case. Hope it helps someone.

The easiest way is to run birtcoind getinfo, then compare the nHeight to a block explorer, which should give you a ratio of how completed it is. Keep in mind that earlier blocks sync fastest, so 50% of black may not be 50% of the time required to sync.


text:
Why are some transactions found on some chain explorers but not on others?
This is something that has puzzled me for a while and I cannot think of a rational reason for it.
The transaction 3361d2484f0566ab13d32f2ab321319945f48eaea1ac2cc9f5a79b40528359c3 shows properly on blockchain.info and tradeblock.
However, I cannot find it either on blockr.io nor blocktrail.com.— I am also running a server to watch transactions and I cannot find this transaction either. Sometimes I can find transactions that do not appear on blockchain so it's quite random.
All servers have many connections so I wonder what could be the issue? Slow network?

You should mention in your question that the transaction is unconfirmed. Perhaps you could put screenshots, because the question will likely not be understandable once the transaction confirms.

Nevermind, I took care of that in my answer. ;)

Each node on the network has its own mempool. While they are filled with similar transactions, each may have a different size limit or perhaps even custom rules which transactions to retain.

In this particular case you can see (figure 1) that there is a (U) shown in both inputs of the transaction. The (U) means that the inputs are created by a transaction that is still unconfirmed.

In fact, when you look at the address that the two inputs were spent to before (figure 2), you'll see that both were created in transactions that only paid 4.4 satoshis per byte whereas the transaction you linked pays 65 satoshis per byte.
I would surmise that the first two transactions that created the inputs might have not been relayed to or were not kept in the mempool of the other blockexplorers. Then, the transaction that you linked would seem invalid to those blockexplorers, because the referenced inputs are not known to it.
You may find this related question interesting: "How do transactions leave the memory pool?"

Oh, I just see that I made German screenshots. "Unbestätigte Transaktion" means unconfirmed transaction as you probably have guessed.

Thanks Murch - I did notice the inputs were unconfirmed but that does not explain why the transaction is not shown on some.

May I just add that I am not seeing the transaction on my node, yet the mempool info shows that it's pretty empty: {
"size": 12590,
"bytes": 26220930,
"usage": 52775360,
"maxmempool": 300000000,
"mempoolminfee": 0.00000000
}
Is there anything else I need to do in order for this transaction to be included on my node?

@Alex: Your node would only get the transaction if one of your peers relays it to you. If none of your peers has it, you'll still not see it even if your node would be permissive enough to show it.


text:
Do Full Nodes perform Proof of Work to validate a Block?
I do understand that Full Nodes validate all transactions twice based upon the public key of the users. Firstly, once a transaction is unconfirmed and secondly the transactions that have been mined in a block. My Question is do the Full Nodes also perform Proof of Work to confirm the validity of a Block or not?

Hashcash-style proof of work is a simple search process: Try lots of nonce values until you get an acceptable solution.  To verify a node simply just has to try one value, the one the miner provided.  So it uses the same test but does not search, as a result it does very little work.
The Bitcoin software hasn't really validated transactions twice since 2012. Caching is used to reuse the validation effort from the first time the transaction was seen, especially for the digital signature verification.
Very careful engineering has been required to make sure that the handling of unconfirmed transactions can't accidentally cause invalid blocks to pass. This is especially a concern because complications like locktimes and soft-forks can make the validity of a transaction some what conditional. But in the latest software really the only validation needed at block time of an already seen transaction is the locktimes and double spending checks.

No, there is no need for a node to perform PoW, since the PoW has already been performed.  All they do is validate that the block/nonce pair equals a valid hash for the specified difficulty.  It's just a single operation per block.
Of course there is other validation, such as deciding which block is correct in cases where there are two competing blocks, but that's not PoW.


text:
How to quickly get the balance of any bitcoin address? (with no third-party API)
As some people have mentioned (eg. here), the only way to get the balance of an address in bitcoin core was to import it as a watch-only address and rescan the blockchain. Starting with bitcoin core v0.17, we can use the scantxoutset command, however this is quite slow:
> time bitcoin-cli scantxoutset "start" "[\"addr(17Pu3CEx34YX8bpGATq45PfL2fGBCPcr2u)\"]"
{
  "success": true,
  "searched_items": 64162818,
  "unspents": [
    {
      "txid": "9796cee98fa88efcb9fdb954188d276b73d6e76719af6b5470ad23a2f744ecca",
      "vout": 1,
      "scriptPubKey": "76a91446255848182640edff4d178f365d7824513bc28d88ac",
      "desc": "addr(17Pu3CEx34YX8bpGATq45PfL2fGBCPcr2u)#x2q6xgrk",
      "amount": 0.02155141,
      "height": 607852
    }
  ],
  "total_amount": 0.02155141
}

real    0m42.419s
user    0m0.007s
sys 0m0.003s

42 seconds is quite slow, I would like to get something comparable to block explorers (less than 2 seconds). Is it possible to achieve faster results with some advanced bitcoin core configuration?
Or, is there any open-source project I could use to index the UTXO set by address? Not sure if libbitcoin, bitcoinj, bitcoinJS, etc. are able to do this faster. Some tool block explorers might use to provide this service would be what I am looking for.

There is no way to enable faster querying for arbitrary addresses than scantxoutset in bitcoind.
Explorers provide this functionality by maintaining a separate database of all addresses and transactions which is optimized for that kind of query. Bitcoin Core, on the other hand, is optimized for consensus related queries, which are largely a hashset lookup for a given input for a transaction.
You could try running something such as ElectrumX, which backs the electrum wallet, or Bitcore, which backs many of Bitpay's wallet and explorer services.

Thanks. Any chance of bitcore or electrumX using bitcoin core's data folder to avoid storing ~300 GB twice?

You can read the data from an existing installation, but any indexing program will have to duplicate it into its own database to actually be able to efficiently answer your queries.

Yes, I read eletrumX's DB has ~30GB and can (and should) read the data from a bitcoin core full node. Checking now whether this is also de case for bitcore.

These are all solutions that operate on top of bitcoind, as that is still the source of truth - they simply add some indexing capabilities that extend the kind of queries you can perform. Your space usage will not really double, as bitcoind's stoage format is somewhat inefficient in favour of fast lookups for specific tx outputs. Traditional databases will do a more efficient job in a lot of cases

Hi @Pedro Were you able to get this to work? I'm curious to know since I'm looking to do something similar.

@RaghavSood "maintaining a separate database of all addresses and transactions"....how can I go about creating such a db myself?

@kdas no fast and easy way to do it with only bitcoin core, you need to run an electrum server.


text:
Set up a Bitcoin full node: static or dynamic IP address?
I am trying to set up a new Bitcoin full node and my institution ISP is trying hard to not give me a static IP address for this purpose. I guess because of the concern that the node will be DDoSed, port scanned, etc.
My questions are:

How frequently a full node with static IP get DDoSed?
Is there any strong reason for favoring static IP address over dynamic IP address?

How frequently a full node with static IP get DDoSed?

It is unlikely that data is available on this.

Is there any strong reason for favoring static IP address over dynamic IP address?

It would simplify external service connections, but there is no real reason you can't run a full node on a dynamic IP. It would simply require some additional work to set up a dynamic dns service, if you do need a dns pointer to the node.

How frequently a full node with static IP get DDoSed?

It depends on who owns the full node, and whether the owner of a full node is known. A full node by itself is unlikely to be DDoS'd since there is little point in attacking any random node. However, if the node is owned by, say, an exchange and it is known that the node is owned by the exchange, it might be DDoS'd as attacking that node may result in some gain for the attacker. But in general, full nodes don't get DDoS'd because there's no point.

Is there any strong reason for favoring static IP address over dynamic IP address?

Connections that peers make to you are based upon the IP address. They remember the IP addresses of other nodes and try to reconnect to them later if they go offline. With a static IP address, it is more likely that your node will get connections because it will have been at the same IP address for a while and nodes have remembered it. Static IP has no effect on your outbound connections (which, to your node, are the most important, not the inbound ones).


text:
When an IP address in *new* table is overwritten?
I'm reading the address management of Bitcoin core implementation. There is one thing I couldn't understand is in which condition an IP address in the new table of a node is overwritten?
In particular, when the node hears about a new peer, it adds that IP address into a slot in the new table, if that slot is occupied, it checks if the existing isTerrible: https://github.com/bitcoin/bitcoin/blob/master/src/addrman.cpp#L309
if (infoExisting.IsTerrible() || (infoExisting.nRefCount > 1 && pinfo->nRefCount == 0))

What is nRefCount? What is the idea behind this check?

nRefCount indicates the number of slots in the new table that is occupied by the same IP address.


text:
Bitcoin wallet/address balance confusion
I'm quite new to bitcoins so apologies if i'm being a noob, but I'm very confused!
My bitcoin-qt wallet balance says I have 2.0113406 BTC (which is what I expect).
Now if you check my transaction history:
http://blockchain.info/address/1HvxJibwAAyVqcNY5GVUWvHSwgYybpRGTp
It says that I only have 0.0123506 BTC?
I think the confusion stems from the transaction I made to satoshidice. It says I sent 2 bitcoins to satoshidice but i only actually sent 0.01. Further inspection yields that I actually sent 0.01 to satoshidice and the remainder to address 1EHYT8dFMcDVnkkE9kEoH5T3PBb3bRZdmP. What is that address? Is that one of my addresses that I don't know about?
What's going on?

Your Bitcoin wallet can have more then 1 address. In fact the Satoshi client (Bitcoin-Qt) creates 100 addresses from the start.
Both 1HvxJibwAAyVqcNY5GVUWvHSwgYybpRGTp and 1EHYT8dFMcDVnkkE9kEoH5T3PBb3bRZdmP belong to your wallet and thus the total balance you see it correct.
Now to why this did happen.
Your wallet received 2 BTC. When you sent 0.01 BTC to Satoshidice "input" the previous mentioned 2 BTC where used as "output" and had to be spent in its entirety. Because the output was bigger then the input the Bitcoin client sent the difference to a new address, which also belongs to you.
In our daily life this is known as change money.
You can read about this in detail on the official wiki: https://en.bitcoin.it/wiki/Change


text:
Regrouping accounts belonging to same user
I am analysing some BTC transactions for academic purposes.
As part of that work, I would like to identify users with several accounts and regroup them under a single identifier.
I have assumed that when a transaction has several accounts paying in, it is likely that it is the same person using several accounts.
Is this assumption correct or could there be other reasons for having several accounts paying into the same transaction.

First off, you should really start thinking in terms of addresses rather than accounts, especially when doing block chain analysis.  See here:  What is the difference between accounts and adresses?
You should also start thinking about transaction inputs not in terms of addresses, but rather unspent transaction outputs (UTXOs).  Each UTXO will generally have an address associated with it, but in the case of multisig, may have several addresses in a single UTXO.  So in this case, even a single input to a transaction has more than one person associated.
There are also the cases of coin mixers, web wallets, and exchanges, which often create single transactions that include many inputs and many outputs, all coming from and going to different people.
So to answer the question, no, you cannot assume that all addresses used as inputs to a transaction come from the same person.

A UTXO really has just one address, but that could be a multisig address that requires signatures from multiple keys.


text:
Created receiving address in Bitcoin Core but deleted it before receiving bitcoins
I created a receiving address in Bitcoin Core, sent myself some coins from localbitcoins.com and - silly me - deleted the receiving address from Bitcoin Core.
tradeblock.com shows the transaction with a "confirmed balance" but I haven't received the coins in my wallet. Is there any way to restore the address in Bitcoin Core?
Thanks!
PS: I wasn't specific enough:

I created a "request payment" in Bitcoin Core and used the payment information address tto send coins from my localbitcoins.com account.
I deleted the "requested payments" entry in the Bitcoin Core wallet before coins were received.
Blockchain.info shows the a "total received" and a "final balance" so I presume the transaction was concluded. Yet, after 30++ hours, the coins haven't been received by my Bitcoin Core wallet.

Is the transaction somewhat hanging there because it can't find the address that I sent the coins to but deleted?
@murch: Thanks for the hint. You are correct: I know the address, the transaction appears to be confirmed on Tradeblock but the balance does not appear in my Bitcoin Core wallet (fully synced).

Did you save the private key anywhere? How did you create the receiving address? How did you delete it?

Indeed, I should have been more specific. Sorry!:

Created a "request payment" in Bitcoin Core and used the payment information address tto send coins from my localbitcoins.com account.
Deleted the "requested payments" entry in the Bitcoin Core wallet before coins were received.
Blockchain.info shows the a "total received" and a "final balance" so I presume the transaction was concluded.

After 20 hours, the coins haven't been received by my Bitcoin Core wallet. Is the transaction somewhat hanging there because it can't find the address that I sent the coins to but deleted?

So, you do know the address, the transaction appears to be confirmed, but the balance is not appearing in Bitcoin Core? Please put clarifications into the original posts, comments are considered transient and might get deleted at some point.

Is you Bitcoin Core fully synced? (does it show a "X weeks remaining..." at the bottom?)

Yes, it is fully synced.

(1) Does blockchain.info show the transaction as being confirmed, and how many confirmations? (2) You might double check that Bitcoin Core is staying in sync by checking the block number and date from the debug window against blockchain.info.  (3) I would try running Bitcoin Core with the -rescan option.  Bitcoin Core is designed pretty carefully to never delete private keys so I would guess they are still there.

Thanks Nate! 1) It shows 99 confirmations. 2) I compared the block numbers, both in my wallet's debug window and the last one shown in blockchain.info are identical. 3) I ran the rescan option but the balance still doesn't show up. Sigh.

Is there actually a way to retrieve/show the previously used receiving addresses?


text:
Transaction failure due to "made-up" transaction
For instance, if I am trying to send 1 BTC to an address, however, I do not have enough BTC. What will happen to this transaction?
Will the sender and the recipient both receive notification regarding the status of transaction?

Unless you are using modified software, your software will not allow you to create such a transaction.
Even if you succeeded at creating it, the transaction is invalid. Invalid transactions don't get relayed on the network, so your recipient would never learn about it.


text:
Is it possible to perform split payment in bitpay
Hi all coming to the question am using bitpay payment in my project..Is there any possible to perform split payment using bitpay
For example am performing bitpay payment for 100 usd for one wallet address
it should be split(50+50) and sent to two wallets 50USD for one address and another 50 USD for second address
Is it possible to perform in bitpay kindly help me if you know the way to do

It's not clear to me whether you want to receive to two addresses or send from two different addresses.

Yes I did it few minutes ago.
After I sent the first amount from wallet A I got the message that the invoice is not paid completely and that X BTC is left.
Then I send the left BTCs from Wallet B to the address and now I must wait for verification of the transactions.
But seems to be working :-)