text:
How is a ticker price determined?
I am trying to understand how a ticker price is determined for an asset, in this case I am specifically looking at BTC data from GDAX. The question stemmed when downloading historical BTC data from GDAX and seeing the following (truncated):
1516099829,11921.070000000000,0.008100000000
1516099829,11921.070000000000,0.037873260000
1516099829,11921.070000000000,0.031519050000
1516099829,11921.060000000000,0.037464080000
1516099829,11920.000000000000,0.001000000000
1516099829,11912.530000000000,0.001678900000
1516099829,11909.000000000000,0.000100990000
1516099829,11908.360000000000,0.001000000000
1516099829,11908.360000000000,0.001000000000
1516099829,11900.630000000000,0.001680580000
1516099829,11900.000000000000,1.293924550000
1516099829,11900.000000000000,0.132108480000
1516099829,11895.000000000000,0.000300000000
1516099829,11891.170000000000,0.000727720000
1516099829,11890.190000000000,1.400000000000
1516099829,11890.160000000000,0.000838510000
1516099829,11890.000000000000,0.058736000000
1516099829,11888.740000000000,0.001682260000
1516099829,11885.310000000000,0.252173310000
1516099829,11882.920000000000,0.841910000000
1516099829,11876.860000000000,0.001683950000
1516099829,11876.310000000000,0.000728620000
1516099829,11871.000000000000,0.000100990000
1516099829,11871.000000000000,0.000100990000
1516099829,11869.590000000000,0.010000000000
1516099829,11868.000000000000,0.000100990000
1516099829,11865.630000000000,0.010000000000
1516099829,11865.000000000000,0.001685630000
1516099829,11864.490000000000,0.900000000000
1516099829,11864.480000000000,1.113098000000
As you can see, the price fluctuates wildly in a single second. I understand that this data is accurate and shows all of the executed orders for the second, but I do not understand how the ticker value is determined for this second.
I am going off of the (possibly incorrect) assumptions that getting the ticker value from their REST API over and over would not show these wild fluctuations for a single second. I draw this conclusion because I have ran a loop that gets the ticker data over and over with just 300 ms delay and the price has never changed in the same second. As a matter of fact, it shows the same timestamp until the price changes, in which case it shows the new timestamp. This leads me to believe that there is something behind the scenes that is determining the ticker price per second based off all of the trades happening in that second.
If it helps, I am asking this because I want to back-test GDAX data for a program that I have written. In my program, I get the ticker data for the current point in time every two seconds, and receive one value back (the ticker price). Now, when back-testing and 1516099829 comes up, I do not know which price to use. My program will behave very differently depending on which data point I use because the prices vary so heavily.

Welcome to the volatility problem.

Very simply it's the last most recent market order which took the highest/lowest buy/sell order on book.

you subscribe to the wss feed you can see most the subsecond stuff is opening and canceling limit orders which doesn't effect the ticket price. Only market orders represent the exchange of bitcoin. Market orders are liquidity takers so get a fee. They represent actual market movement. The limit orders often will never get matched. I often market orders may only come in every minute to five minutes under normal trading atmosphere. So ticker price more constant. Hope this help.

thanks for the reply. So am I correct in understanding that the data that I posted is just orders on the book, not necessarily matched orders that moved the ticker price? If so, it sounds like it's not possible to tell what the actual ticker price is at a time from the data that I have? I would need to find another data set that shows the actual ticker price per second if I want to back-test a program that functions on the ticker price? I got my data from http://api.bitcoincharts.com/v1/csv/ if that helps.

Market orders never go on the book, it would be open limit orders leaving book, but how can you tell if it matched or was simply canceled. Most transactions are open/closing orders. Basically algorithms jockeying for positions etc. Much fewer are exchanges of bitcoin. So really it's not as active as it may appear. Sometimes though it can speed up like in sudden market rise or dip. Usually spurred by a large market order filling.

You should use the gdax websocket feed. It relays the most granular information nearly as soon as it hits the market. Delay depends on your physical location and market conditions. Also for different order books from the rest API, especially lvl 3, your explicitly not supposed to periodically request. Instead start wss feed and then get book and proceed updating the book with the feed using the book and wss transaction sequence number, kind of like bleeding breaks for air tight book.

Ok, I've been checking price by getting the ticker value every 30 seconds and checking my order statuses every few seconds to see if any of them have filled (Both REST API). I avoided web sockets because I figured that was more than I needed for my simple program, but I will look into them more. Thanks for shedding light on what I was actually looking at with the raw data I posted. It makes sense now. I'll up vote / accept when I have the points to do so.

Yes the rest API is easy, but if you can handle authenticated rest API wss subscription should be easier.

Also 30 second poling probably isn't a problem but if you want to know stuff as detailed as you ask you simply should use the wss feed.

It all depends on the fidelity you need.


text:
Bruteforce bitcoin address - I know the words + public address but not the order
I have a long list of 1million+ permutations of a 12 word BIP39 seed. How can I try them all on a bitcoin address?
edit: I know the address and the 12 words but i don't know the order of the words

Possible duplicate of Is it possible to brute force bitcoin address creation in order to steal money?

@JunaidShaikh I'm not asking if it is possible. I am asking for a specific tool to do it. And I only need to try 12 words not 1024 different words

This isn't an exact duplicate - It's a lot more feasible to try all combinations of words if you know they are all in the seed (although this breaks down for 24+ words).

@code511788465541441 Do you know the address you are looking for, and the derivation path which would lead to it?

@RaghavSood yes i know the address and the words

Do you know the derivation path? This is infeasible unless you know the exact derivation path that leads to the address (or any address in the wallet).

@RaghavSood I'm not sure what derivation path is but i know its BIP39 word list with 12 words and what the words are

Since a seed leads to billions of addresses, each address has a path within the seed (something like m/44'/0'/0'/0/i, where i is increasing for every new address). Unless you know which path and index the address is at, it is impossible to check if a seed contains it. Which wallet is the address from? Most wallets publish their derivation path, so you should be able to find it. Then you just need the index (or an approximate)

@RaghavSood the address is 3CcxyPhyvyc3S9UuPfu42GNZLvVVV11Uk8

Ah, you're trying the reddit puzzle. If you assume that the derivation path is m/49'/0'/0'/0/0, which is the first segwit address (I'm assuming the creator sent it to the first address on the seed), this is technically feasible. I'll post a script as an answer

@code511788465541441 , I misunderstood the use-case over here, sincere apologies.

Doing this requires three things:

Permutations of the seed words
The address you're trying to locate
The derivation path for that address.

In this specific case, the address is the one this puzzle on reddit leads to, so we will assume the derivation path is m/49'/0'/0'/0/0
This is a simple nodejs script using bitcoinjs-lib to read seeds from a file, validate them, and then try the first address against the one you are searching for. You can parallelize it by splitting the input file into pieces and running it once per file.
Do note that 12 words means roughly half a billion combinations. While this is doable with today's computing power, it is still going to take you a very long time. Some quick tests show it would take about 4 days on my laptop, although you may be able to speed it up by using a faster language, or parallelization.
var bip39 = require('bip39');
var bitcoin = require('bitcoinjs-lib')

var lineReader = require('readline').createInterface({
  input: require('fs').createReadStream('addresses.txt')
});
var ctr = 1;
lineReader.on('line', function (line) {
    if (ctr%100 == 0) {
        console.log("Processing #" + ctr);
    }
    if (bip39.validateMnemonic(line)) {
        var roothex = bip39.mnemonicToSeedHex(line);
        var rootnode = bitcoin.HDNode.fromSeedHex(roothex);
        var basechild = rootnode.deriveHardened(49)
                .deriveHardened(0)
                .deriveHardened(0)
                .derive(0);
        for (var i = 0; i < 3; i++) {
            var child = basechild.derive(i);
            var keyhash = bitcoin.crypto.hash160(child.getPublicKeyBuffer())
            var scriptSig = bitcoin.script.witnessPubKeyHash.output.encode(keyhash)
            var addressBytes = bitcoin.crypto.hash160(scriptSig)
            var outputScript = bitcoin.script.scriptHash.output.encode(addressBytes)
            var address = bitcoin.address.fromOutputScript(outputScript)
            if (address == "3CcxyPhyvyc3S9UuPfu42GNZLvVVV11Uk8") {
                console.log("Found seed! " + line)
            }
        }
    }
    ctr++;
});

You will need to install nodejs, and bitcoinjs-lib for this.

Thanks, I'm trying that now. What's the for loop for?

It'll check the first three addresses on every seed, just in case we're wrong about it being the first address.

Any idea why validateMnemonic() returns false for this example 12 word seed string? proof know able order problem end just zero run air agree next

Because it's an invalid mnemonic. The last word is a checksum. Not all combinations of BIP39 words are valid seeds. The valid mnemonic check will skip a seed if it is invalid, so you don't needlessly process it for addresses.

Thanks a lot. It is a bit slow as you said so I am trying to write it to run in parallel. If I do find the key i'll get in touch :)

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How prevalent are coin attacks with botique altcoins?
I have deployed several altcoins using various PoW algorithms on my LAN for my buddies and I to mine/learn from. We ultimately want to go the distance of deploying an altcoin to the world, writing the white paper, doing the leg work of maintaining it and getting it listed on an exchange.
I have a fear that since our incredibly slim budget has limited us to a very small amount of computational power with which to develop these altcoins (at least initially) that if/when we go the distance of releasing a fully functioning altcoin we will be swiftly crashed by someone with a cabinet full of ASICs and a love of trolling.
I guess my question(s) are how common is this and what is the least expensive but effective way of preventing it. Please don't destroy my dreams, I understand making a coin that ever ends up worth anything is highly unlikely, I am asking for technical advice relative to the issue at hand, not to be told that having a coin of value with a small development team and little capital investment is impossible.
Thanks!

Schwifty

If you'd like to be absolutely sure that some malicious miner won't attack your coin, you could hard-code in a set of 'approved' miners (ie, your miners). Doing so greatly undermines some of the core concepts of a cryptocurrency though (open, permissionless system, etc), so that probably isn't a good option by most measures.
So then, what is the best option?
Many coins choose to employ an 'ASIC-resistant' algorithm, with the goal of preventing ASIC-wielding miners controlling their network. Instead, the idea is that GPU (and perhaps CPU) miners will be the ones securing the network, and many people argue that the the distribution and economies of scale involved in GPU mining are better-suited to create a decentralized network.
However, in practice we've seen that perhaps no algorithm is ASIC-proof. In fact, an algorithm that is extremely difficult to build an ASIC for may just concentrate mining power even further, as very few people will have the skills and resources to create an ASIC for such an algorithm. Such people could thus easily control the majority of hashpower on a network, once they have built some ASIC miners.
For a more in-depth look at this issue, check out this article written by David Vorick (a SiaCoin dev).
It is worth noting though: nobody is going to invest the resources to invent an ASIC that attacks a very small coin. There would be no economic incentive to do so, this problem only becomes apparent once the network has grown and is storing a worthwhile amount of value. (Keep in mind though, if you choose an algorithm also employed by a larger chain, then such ASICs might already exist).
Trolling aside, it is also worth noting that it is not necessarily in a miner's economic interest to attack your coin. If a miner owns some ASICs, and can make money by mining with them, then there is perhaps a substantial opportunity cost involved in instead using the ASICs to attack your chain. This incentive helps keep miners 'honest', in that they will benefit more by using their hashpower to secure a chain, rather than attack a chain.

Thank you for your response. I am going to read that article you referenced now.  I definitely don't want to do anything that undermines the core ideology of cryptrocurrencies, nor do I want to employ an algorithm that is resistant to any form of mining hardware. Would it perhaps be best to privately grow the chain and achieve a decently high hashrate and thus a moderate difficulty before opening the network to the public? Would that be the least invasive, safest preventative measure I could take?

Difficulty is affected by the number of hashes/second pointed at your chain (not the blockheight), so privately growing the chain with some amount of hashpower will not protect you against someone with a larger amount of hashpower. Put differently: if you control 'x' hashpower, it doesn't matter whether you make the chain public at block 1, or block 100,000, someone with '>x' hashpower will still be able to outrun you. The higher your 'x', the smaller the pool of other miners that could attack your coin.

Yes I am aware of how the difficulty is targeted, what I meant by "privately grow the chain" was to add more nodes locally to increase the hashrate until it is at a point that is more difficult for someone to be able to surpass single-handedly, not just to add blocks/increase chain height, sorry I wasn't more clear on that. I really appreciate your responses, I am delighted to see how invested people are in this community. I will take all of your responses under advisement and proceed.

Right, I understand your reply better now. Privately growing the chain would increase the resources needed to reorg your chain and thus overwrite your 'pre-mine'. But it would still not prevent an attacker with more hashpower from completely censoring transactions, ignoring all subsequent blocks that are not their own, etc (see: 51% attacks). Happy to help :)

Yes, I am aware of the 51% attack, and it's actually my primary concern since at this point I lack the hardware and resources required to have an indomitable hashrate. Though as you've pointed out, it isn't really fiscally logical, and there isn't really too much that can be done if someone wishes to trollololol away. I will do everything in my power to have the network at a respectable hashrate before opening it up to the honest miners and trolls alike, aha.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Unwanted peers connecting to Altcoin
I am building an altcoin using the litecoin source code.
I have removed all of the Emplace Seed calls in chainparams.cpp for DNS seeding, changed my default node communication port to port 421 and commented out all of the baked in seed nodes from chainparamsseeds.h, even added the dnsseed=0 and seed=0 to my config file and deleted my peers.dat before launching the core daemon, yet I still see clients connecting to my VPS for all different kinds of coins and swiftly ending up banned.
What am I missing? I can't find a single reason why these peers are connecting to my VPS.. Though it's not the end of the world because it has latched to my chain and only syncs blocks/headers with my other nodes, it looks very unprofessional. Any help would be appreciated!
Thanks!

Schwifty

Sounds like something related to your vps. Also, have you also changed to your own values for pchMessageStart in chainparams.cpp?

Hmm.. Everything with the VPS has gone smoothly to this point, and from what I can tell with netstat it is listening on the correct ports for node communication and mining. I don't believe I have changed values for pchMessageStart, I will look into this and edit my post or comment accordingly, I thank you kindly!

What I mean is your vps might be a target for bots looking for nodes.

Ah jesus, I hope not. I am going to use a different port and change pchMessageStart and bake in my VPS seed node IP the next time I rebuild so I will comment/answer if I manage to remedy the issue.

Took care of this with a combination of changing pchMessageStart values, baking my VPS seed node address into chainparamsseeds.h and adding rules to block outbound traffic to ports 9333 (and the other strange unsolicited connections to other altcoin nodes).


text:
IBM Blockchain platform specifications
So I saw an ad for IBM's blockchain platform, specifically the campaign they have to "turn plastic into money" by way of exchanging recyclable materials for digital tokens.
I was just wondering if anyone knows the specifications of their blockchain. I assume with it being IBM that it's all very centralized and hush-hush, though if you're willing to go the extra mile, or already have an IBM account you can download a technical brief of their platform. Does anyone have this, or know what their consensus mechanism(s) is/are or basically any information pertaining to the technical specifications of the platform?
Thanks in advance!

are you sure it's not referring to hyperledger? https://www.ibm.com/blockchain/hyperledger. Also https://github.com/hyperledger

Hmmm, perhaps. It did say "Powered by IBM Blockchain" .. I'll try to look into that specific application a bit more, the "turn recyclables into money" digital token application, and see if it is in fact just hyperledger.

First IBM Blockchain is IBM's branded version of it's Hyperledger Fabric blockchain software (which is Apache 2-licensed open source under the Linux Foundation). They are basically the same, with the exception of add-on services and configuration.
Unlike Bitcoin, Hyperledger Fabric is an enterprise blockchain intended for private use (such as consortiums).
The "plastics into money" is just marketing--it's an example of using blockchain in a "proof of concept" application.

Awesome, thank you for the info!
I wasn't sure if it was just a different application of Hyperledger or if it was a different project all together.


text:
Mining Theory: What would happen if a block hash collision occured?
To my knowledge, neither SHA256 or scrypt have ever had provable instances of hash collisions occur.. However, if this highly unlikely event was to happen, what would that mean for the state of the blockchain? Would it cause consensus confusion since there would be 2 blocks with the same hash that point back to different previous blocks? And/or would it make it best practice to avoid whatever algorithm the collision occurred with?
Just curious,
-Schwifty


text:
Reject transaction because of illegal data
We already know there is illegal data on the blockchain and it will be on there for ever.
That got me wondering about the potential legal implications for miners, since they are technically endorcing and worse.. distributing illegal data by including those transactions in a block.
What is the current legal situation in the US/EU ?
Is there any record of miners rejecting a transaction because of the arbitrary data after OP_RETURN?

There have been miners that rejected transactions with OP_RETURN because its unnecessary spam. Virtually every node and miner limits OP_RETURN outputs by size.
The amount of data relayed or allowed by default is not very large, one of the reasons for this is reducing drama about "illegal data".
We've come up with even stronger techniques that could be used to prevent the inclusion of arbitrary data in transactions, if it were ever required.
But the idea the some invisible data squirreled away in an inaccessible location would be legally dubious seems impractical at least for small amounts of data: consider, any text could secretly convey a small amount of data-- so a hard-line adherence to that theory of liability would make it impractical to ever publish any text provided by a third party. I think there would be a creditable argument that in the US any law which attempted to do so would be invalid on its face by outright inhibiting the press.

Thanks for the clarification. I was just thinking about this, because on the blockchain we are not able to remove doxing or illegal links (which is possible with only 80 ascii chars) and can thus not comply with the EUs General Data Protection Regulation.


text:
How to watch and use addresses generated from xpub
How to watch the addresses generated using xpub for incoming transactions? and Create transactions to send bitcoin to another address.
var bitcore = require('bitcore-lib');
var Address = bitcore.Address;
var PublicKey = bitcore.PublicKey;
var Networks = bitcore.Networks;

var xpub = 'tpubDFaw2mL2nZuxk9WeCTr5Y74xytQLqtxSet3FYigfTMX788gjZmd4juFYeLDchGMaXmjGeoy6FrULbZd9D8iogcPZyBeYfT5MAyqnaL8bMQb';
var hdPublickey = new bitcore.HDPublicKey(xpub);
var orderPublickey = hdPublickey.derive("m/44/0/4/0/0");
var pubkey = new PublicKey(orderPublickey.publicKey);
var address = Address.fromPublicKey(pubkey, Networks.mainnet);

console.log(address.toString());

What software are you using?

I using bitpay to get the xpub and doing programming in nodejs

my requirement is how to watch this newly generated address for incoming transactions  and transfer the bitcoin from this address to another address if funds come

If you're into nodejs you should check out bcoin. It's a library, full node, and wallet. You can create a watch-only wallet and import an xpub. https://bcoin.io


text:
How to migrate bitcoin node from one computer to another
In my old computer some addresses have bitcoin and it is showing on blockchain.
When i am trying to get balance of the same addresses in my new computer where i have synced bitcoin full node ,I surprised addresses showing zero balance.
I am very confused, how can i backup bitcoin wallet from old computer to another

Have you synced your new node with old wallet.dat already copied to this node ?
Or have you synced node, and later copied your wallet.dat ?

i just synced , have't copied wallet.dat .......

This method is used by Bitcoin Core (and few others) and comprises of backing up a wallet.dat file. It's a file that holds keys to spend outputs, addresses, transaction information, and other metadata such as contacts.
It has a different location based on an operating system.

Windows: %APPDATA%\Bitcoin
   Linux: ~/.bitcoin/
Mac OS X: ~/Library/Application Support/Bitcoin/

For backup Locate wallet.dat and copy it to another device only you have an access to. It may be a USB drive, CD, another computer or your private home cloud storage.
You may also copy it to a cloud storage such as Mega or Dropbox but make sure it's encrypted.
To restore the backup, simply copy your backed up wallet.dat over an existing one in the bitcoin data location.

You need to copy your wallet.dat from your old computer to your new computer and replace the file. Otherwise you could take a look on Electrum, it is a light wallet (so you do not need to sync the whole blockchain) and you can import your wallet.dat also there. In your wallet.dat all adresses and private keys are stored so you only need to backup this file.


text:
How to run BitcoinCash node?
How to install and Launch Bitcoin Cash daemon on machine?
I have already synchronized full BitCoin node, Here is the link bitcoin.org from here i learn how to setup BTC node
I also want to run BitcoinCash node but i do't have any idea about BCH node

If you go to BitcoinCash's site, you can see there are multiple implementations of the bitcoincash protocol.
You can select any one, say BitcoinABC.
On their page you can find source code as well as binaries for your type of system which you can download and run.


text:
UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'fromSeedBuffer' of undefined
I am getting Getting error at this line
var root = bitcoin.HDNode.fromSeedBuffer(seed);

UnhandledPromiseRejectionWarning: TypeError: Cannot read property
  'fromSeedBuffer' of undefined

Here is the code snippet to generate bitcoin address:
'use strict'

var bip39 = require('bip39');
var bitcoin = require('bitcoinjs-lib');

var mnemonic = bip39.generateMnemonic();
if (bip39.validateMnemonic(mnemonic)) {
    console.log('\nThe mnemonic is \n' + mnemonic);
    var seed = bip39.mnemonicToSeed(mnemonic);
    var root = bitcoin.HDNode.fromSeedBuffer(seed);
    var dp = root.derivePath("m/140'/0'/0'/0/5");

    console.log('\nThe Address is \n' + dp.getAddress());
} else {

}

Where bitcoinjs-lib v4.0.0 and
      bip39 v2.5.0 and node v8.0

Try using the bip32 module. Also there are a few other things you'll need to fix as well. See Use BIP39 to generate BIP32 addresses. Note, this will generate a P2PKH address, see getAddress() function.
index.js:
'use strict'

var bip39 = require('bip39');
var bip32 = require('bip32');
var bitcoin = require('bitcoinjs-lib');

function getAddress (node, network) {
  return bitcoin.payments.p2pkh({ pubkey: node.publicKey, network }).address
}

var mnemonic = bip39.generateMnemonic();
if (bip39.validateMnemonic(mnemonic)) {
    console.log('\nThe mnemonic is \n' + mnemonic);
    var seed = bip39.mnemonicToSeed(mnemonic);
    var root = bip32.fromSeed(seed);
    var dp = root.derivePath("m/140'/0'/0'/0/5");

    console.log('\nThe Address is \n' + getAddress(dp));
} else {

}

Output:
$ node index.js

The mnemonic is
assume victory kiwi swarm furnace firm thumb exact dizzy crucial neglect certain

The Address is
1FYtPnUZZ2ZJL2fZ6XehGtkHfujUHkqK3T

Hi @JBaczuk help to get privatekey for this address 1FYtPnUZZ2ZJL2fZ6XehGtkHfujUHkqK3T........ from the above process I got address but I do't get any way to get privatekey

Try dp.privateKey


text:
Migrating from bitcoin core to bitcoind
I have installed bitcoin core in my ubuntu-14.04 which is working fine.
Now I want to install bitcoind in the same Machine. Can I install bitcoind along with bitcoin core or I need to uninstall bitcoin core from my machine.
What should I do to migrate from bitcoin core to bitcoind?
Thanks in Advance.

They are identical in functionality, data store, and wallet storage.
You can run one or the other, but not both at one time (this will throw an error if you attempt it).

What should I do to migrate from bitcoin core to bitcoind?

Literally nothing, start either bitcoind or bitcoin-qt as desired.

do I need to uninstall bitcoin-qt

No, you do not.


text:
Security via Proof - of - Work
I am reading Satoshi's original paper, merely for the sake of understanding the concept. I have minimal computer science background (some), mostly just mathematics.
In Section 4. Proof of Work of Satoshi's original paper, he (she? who knows) says 'Once the CPU effort has been expended to make it satisfy the proof-of-work, the block cannot be changed without redoing the work. As later blocks are chained after it, the work to change the block would require redoing all the blocks after it".
I think I understand the big picture in the above selection, but the technical details of what would happen in such situations are unknown to me. Questions:

Assume a node did go back and redo the proof of work on a previous block (say block n) - with no intention of doing the proof-of-work on any following blocks - would this create a fork coming off of block n or would it sever the current longest blockchain between blocks n and n+1?
Is the reason that 'the work to change the block would require redoing all the blocks after it' simply because there is no sense in
  changing a block unless the data in the changed block is validated by
  appearing in the longest chain. Since going back and changing block
  n as outlined in question 1 (by my interpretation) creates a fork at block n-1 to honest n and dishonest n, - by redoing the work for all other blocks following dishonest n, the hashes and timestamps coincide, and if this chain theoretically caught up to the current honest chain (which of course probabilistically can't happen), this chain would be the new honest chain by the definition in 5. Network.
Suppose the unlikely happened, and the scenario of question 2 took place - would the proper analogy to regular currency be a) I gave
  you money in exchange for some good or service b) (presumably) I
  received the good or service c) I stole the money back. My point here
  is, even if the unthinkable happen, it would not be analogous to
  'duplicating' the money I gave you, it would be more like stealing
  it back?

I would prefer answers that mostly answer within the framework of Satoshi's original paper (which may be different than how bitcoin currently operates in the real world), but I am not opposed to hearing related facts about how this is actually implemented.
Also - new to this community: please edit tags or title at will :)

It creates a fork.  There's no way to "sever" an existing branch of the blockchain.  All valid branches remain valid forever, and a node will select whichever one is currently the longest (by total work) as the authoritative ledger.
Right.  If you just redo one block, you've created a branch, but it's not the longest one, so it won't be considered part of the authoritative ledger.
Yes, that's the right analogy.  "Double spending" doesn't duplicate money; rather, it rewrites history so that your payment is considered to have gone to someone else instead (typically back to you).  If someone else thought they had that money, and gave you some goods or services in exchange, then you've effectively stolen it back from them.  You effectively got to spend that money twice, but the blockchain thinks the first one never happened, so the official record only shows one spend.


text:
How can an output be spent while the transaction is in mempool
I have seen transactions on a block explorer where the transaction is in  mempool (unconfirmed) but one of the outputs already appears as spent, I am curious how this works. This question is a duplicate of this question, but the answer there doesn't quite satisfy my confusion. I am thinking about how this process is permissible in conjunction with the protocol rules for 'tx' messages listed here, particularly rule 9, 10, 12.

For each input, if the referenced output exists in any other tx in the pool, reject this transaction.

For each input, look in the main branch and the transaction pool to find the referenced output transaction. If the output transaction
is missing for any input, this will be an orphan transaction. Add to
the orphan transactions, if a matching transaction is not in there
already.

For each input, if the referenced output does not exist (e.g. never existed or has already been spent), reject this transaction

Questions:

Part of what might be causing confusion is the definition of "spent" on the block explorer - when it claims an output is spent does this equate to "there exists a transaction at least in mempool that uses the output as an input"?
The ability to broadcast a tx message that uses the output of an unconfirmed tx as an input seems to explicitly violate rule 9. Does it not?
Rule 9 & 10 seem to be at odds with each other. Rule 9 says to reject if the output is in a tx in mempool, rule 10 says to check mempool for the output (as though it is permissible). Are they not?
It is my belief that full nodes keep a "UTXO" set for validating some of these criterium - is the UTXO set updated once a transaction hits mempool (i.e not waiting for a confirmation)?
Is mempool relative to one node? I.e, if I broadcast a transaction and a node accepts it as valid does it then pass the transaction around so that other nodes can add it to "their" mempool, or is there somehow a "central mempool" that any validated transaction can get sent to.
Is it possible to cancel a tx message that has been added to mempool? I am not asking if typical software has this feature - but is it theoretically possible for a power user or is it "permissible behavior" within the bitcoin protocol.

There is no rule requiring a transaction to have a confirmation before being spent.

Yes, the mempool matters. That's also how double-spends are detected. So it's crucial nodes check the mempool and not just previous blocks.
No it does not violate the rule, they are referring to double-spends being rejected if the SAME output exists in the mempool.
IDK, maybe ask the original poster what they meant?
Yes, no need for any confirmations to become an unspent transaction output.
Mempools from node-to-node are not always identical. For example, some nodes have different minimum relay rules that may include extremely low fee transactions. There is no central mempool.
Not really. I haven't done it recently but lets say you broadcast a transaction with a 0 fee. The transaction would likely get stuck in the mempool and not get confirmed. After 14 days if it doesn't get included in a block it "expires". Once expired/dropped I could craft a new transaction using the same UTXOs that I used in the original.

Here's a related question that you will probably find interesting:
What happens to transactions in the mempool when their parent transaction expires?

Thanks, and also thanks for the link. I thought I did my research before posting but I didn't find that one. So when block explorers show the size of mempool what are they showing? Just the mempool relative to the full node they are running, which is presumed to be a good representative for total mempool?

Although I'm making an assumption it's extremely likely they are showing the mempool of their own node(s).

Thanks, assumptions are fine for that question.

Point 9 means: In a mempool , if there are two transactions having input that refer to same output, then reject the transaction that arrived later. Technically, every node checks this condition before adding the transaction in mempool. This prevents Double Spending.
Point 10 simply node stores that transaction whose input refers to the output of unconfirmed transaction (i.e, present inside the pool). If the node cannot find the parent transaction, then the current transaction is stored in orphaned transaction's pool.
Point 12 is same as point 9 except in this step the node searches the previous block  to reject this transaction in case of double check.


text:
How often does the "little man" win [mining]
I have read that long term we expect that the percentage of blocks mined to be forever converging to the percentage of hashpower possessed by the miner. This means if a particular miner has 20% of the total hashpower amongst miners, over time we expect to see the percent of blocks they mine converge to 20% of blocks.
Has any analysis been done to see if the data supports what we expect to see? Obviously it is complicated by mining pools, miners entering and leaving the network, and rig upgrades, but we will ignore these factors. Here is some clarification questions that shed light on why I am curious:
I believe I understand that all miners are working on the same block number, but possibly with different transactions bundled in the respective block they are working on. Ok, for simplicity lets assume there are 4 miners -
Miner1 with 35% hashpower
Miner2 with 30% hashpower
Miner3 with 25% hashpower
Miner4 with 10% hashpower
Suppose mempool is ripe with transactions and block number 5 was just succesfully mined by Miner1. Miner2 was really close to mining their block when Miner1 finished (perhaps within 10 increments of a nonce). Now it is time to start on block number 6. Does everyone have to start over from square one by grabbing transactions from mempool and starting to hash? Or do the other miners who didn't win get to continue the work they have already done (10 more hashes for miner2). If the former, I don't see how miners with smaller %'s of power ever mine any blocks at all. If the latter, then I see how the speed at which you mine blocks will be determined by your hashrate, but I am curious if it will actually converge to their percentage of power. I am also curious, if they "continue their work" what happens to the transactions in their block that may have been shared with the ones in block number 5?
I am especially curious about this same question in the PoS set-up but will wait and post in a seperate question later.
I also believe I understand that mining is so fast now “incrementing the nonce” is kind of nonsense since all nonce values can be tried in an instant and then the miner needs to grab new transactions and try again.

Yes at current rates a miner tries all nonces in a fraction of a second, but they usually respond by changing only the coinbase transaction in the body (and the Merkle tree). They don't need to alter the set of 'user' transactions, although they can if e.g. a higher-fee txn is now available.

What control do the have over the Coinbase field? The address the Coinbase transaction is sent to?

A solo miner controls everything in the coinbase txn, but normally they don't want to change the output. Since they also control the (single) dummy input, they vary an 'extranonce' value there; see https://bitcoin.stackexchange.com/questions/90684/how-to-decode-a-coinbase-transaction . For a pool, all members must use the output(s) determined by the pool, and each member is assigned different ranges of extranonce values to try; see https://bitcoin.stackexchange.com/questions/37279/work-distribution-in-a-mining-pool . See also https://developer.bitcoin.org/devguide/mining.html

There are several aspects here that are worth pointing out.

The 'little man" rarely wins, simply because almost all mining happens in large pools, and they win proportionally to their hashrate.

This isn't due to unfairness in mining, but simply due to the fact that the expected variance in payout is too large for most small hashrate entitites to risk their business on.

Mining is progress free.

Every individual hash attempt has a fixed probability of being a good one, independently of how many hashes the miner (or anyone else) has tried already. Thus, nothing is "lost" by switching to a new candidate block to work on top of. This is a very important property; mining would be strongly centralizing if the process wasn't progress free.

There is a small but non-trivial time needed to switch block candidates.

When a new block is found by a miner on the network, all other miners switch to working on top of this. If they don't do this, they are in violation of the protocol, and should be treated as an attacker - it's exactly what a 51% attacker would do. Miners don't do this for the simple reason that it would be economically disadvantageous. All but a majority hashrate miner would never catch up if they didn't switch to new blocks.
However, there is some small assymetry here still: switching their hashers to work on top of the new block requires a nontrivial amount of time:

They need to first learn about the new block
They need to validate it, so they can update their mempool (remove mined transactions and transactions conflicting with mined ones from it).
They need to build a new block candidate on top of the new block, using their new mempool.
They need to inform their hashers to switch to this new candidate.

The assymetry lies in the fact that the miner who found the new block themselves don't need to wait to learn about it - they already know. This gives a small but possibly nontrivial advantage to larger miners: they find a block more often, and thus can more often take advantage of this benefit. This is in fact the reason why the average time between blocks needs to be large compared to the time it takes for a block to be propagated through the network.

Hi Pieter - Can you elaborate on what you mean by 'the expected variance in payout is too large for most small hashrate entitites to risk their business on'? Also, if mining is progress free and each hash has a fixed probability of winning, independent of those hashes before and after, then I don't see why some people say 'there is no sense in mining with basic equipment anymore because mining equipment has become so advanced and done in large pools". The way I am understanding this is the presence of more advanced miners does not impact me in any way. They may win more often, but as long as

there is a pile of lottery tickets to check, how often they win should not effect me. If this is correct, mining with basic equipment today is no more or less a good idea than mining with basic equipment was in the early days. With basic equipment, a hash rate may be low - but you still have a chance at winning. It seems to me like it would be comparing a person who recieved one free lottery ticket per day vs a person who spends money to buy 100 lottery tickets per day. I still might win eventually, and it is free for me to check, though the person buying 100 per day

will win more often because they are checking more tickets, though at a cost.

The difficulty adjusts to keep the block rate at one per 10 minutes. This is effectively reducing the percentage of winning tickets. So simply the fact that there are more efficient machine around means your profitability goes down. If the expected income from mining is below the cost of your hardware wear&tear/replacement, electricity, operations, maintenance... you will stop mining. In a world where those costs are independent of location, only the most efficient machines will keep mining - everything else will inevitably become unprofitable.

Regarding variance; if your hardware (say) has a shelf life of one year for example, but the total amount of hashrate you have only lets you win one block or so per 6 months on average, that is probably too much to work with. You may get 1 or 2 or 3 blocks out of your hardware, but you may also just get 0 - making it a gamble. Too much of a gamble isn't something you want to bet your business on, even if technically your hardware's expected payout - expected costs is positive.

So by "fixed probability" I mean within one difficulty adjustment period (2016 blocks, ~ 2 weeks) every hash attempt has the same chance, and this is independent from how many hashes you or anyone else has already performed for a given block. But every 2016 blocks the difficulty, and thus probability, is adjusted to keep the block rate at 1 per 10 minutes.

Miner2 was really close to mining their block when Miner1 finished (perhaps within 10 increments of a nonce).

This is not what's happening during mining. Mining isn't a gradual process where you could be "really close" to getting a block mined. Rather it is a constant lottery, with each attempt (incrementing a nonce and producing a block hash) being completely independent of another. You could find the correct hash on your first attempt or your trillionth.
Even though independent events are sometimes very counterintuitive, the math behind them is so well understood that I dare say we don't need any data to see if it holds for bitcoin mining.

So higher hash rate means I can hash faster which is like checking more lottery tickets faster and how often I win is proportional to how fast I can check tickets. But in this setting the presence of miners with more hash power does not impact my success at all?

I only ask because I have heard some people say mining is pointless now unless you have very sophisticated equipment.. but in the lottery analogy it wouldn’t matter if someone else can check tickets faster than me.. So something must be missing

@PrinceM Mining is pointless if your electricity costs are higher than mining rewards. There's a limited amount of bitcoin to be shared between all miners, and as more miners join, only those with cheap electricity can remain profitable.


text:
How do network nodes "connect" - amateur level
I have little background in computing, have been studying bitcoin for intellectual stimulation. I have learned a lot and I understand quite a bit regarding the function of nodes, the validations they perform, and their importance to bitcoin functionality - but I want to understand more about how nodes in the network "connect". I read this question and it didn't help.
I have no idea what a "network" really is. I googled and read about LAN/WAN. Are these the types of network we're talking about?
For example - if I owned bitcoin before the bitcoin cash fork, it is my understanding that I could choose to send my coin as bitcoin, or send it as bitcoin cash, it just depends which network I send the transaction to (kind of, over simplyfing here). So if I send it from a bitcoin wallet it will get broadcasted to the bitcoin network and stay on the bitcoin blockchain, or if I send it from a bcash wallet it will get formatted however bcash formats tx messages and get broadcast to the bcash network. How do these "networks" work, how are they distinct? Where could I look to read more about this.
Certainly if I download bitcoin core and start running a full node it will start connecting me to other nodes in the bitcoin network. How does that process actually go down? How do distinct networks stay separate? I have read about things "dishonest nodes" can do, so without downloading something like bitcoin core (which I assume automatically implements standard protocol for node behavior) how would nodes connect to the network without running bitcoin core?
Hopefully someone can make sense of my question, I understand it may not be very clear but that is because I don't even know enough to form a good question. I don't have any terminology here.
Edit: maybe this will help understand. Supposed I wanted to write some code to develop my own bitcoin client. What was that code look like?

I have no idea what a "network" really is. I googled and read about LAN/WAN. Are these the types of network we're talking about?

No. A network is really just "a group of connected things". One example is indeed a LAN network through which multiple computers that are physically close to each other (a home, an office, ...) are connected to each other. The Internet is also an example of a network - one built up of smaller and very distinct networks like LANs.
But networks can be many more things than that. The group of people you know and hang out with is considered a network too. An electrical circuit inside a machine is sometimes called a network. I bring up these examples, because the word "network" in a computer setting often evokes the mental image of physical interconnections between computers - and while that is one example, the Bitcoin network is something fundamentally different.
The Bitcoin peer-to-peer network consists of Bitcoin nodes - pieces of software, not hardware. I could be running two nodes on my own computer, and they could be connected to each other - or not; they could also both just be connected to far away nodes elsewhere in the world. In fact, these connections don't even need to be established over the Internet. There have been projects to connect Bitcoin nodes over mesh networks, and you could arguably also consider the Blockstream satellite service to be part of this network.
So bottom line, the Bitcoin network consists of pieces of software talking to each other. How they talk to each other can differ, though most commonly, it is TCP/IP connections over the public Internet or over Tor. These connections are somewhat similar to the connection your browser makes to stackexchange.com to show you this answer, but instead of it just being many clients (browsers) talking to one server (stackexchange), it is all nodes that are equal - they are peers. No servers and clients; there are just nodes, and they each connect to one or multiple other nodes. Certainly some are more powerful or better connected than others, but none have a privileged position w.r.t. other nodes.

How do these "networks" work, how are they distinct?

This is where it gets interesting. Bitcoin, and similar things, follow a trust-minimized design. At least if you run a fully-validating node (like Bitcoin Core), your node will generally not trust anything other nodes tell it. They will exchange blocks, and transactions, and other things, but whatever your node receives it will validate independently to the extent possible. This includes very strict validity rules for blocks. Your node will only accept blocks that follow the exact rules implemented in the software.
The simple answer to your question is that Bitcoin and altcoin nodes have different rules, and simply won't accept each others blocks. If a node were to be only connected to nodes with different rules, it would be isolated, and not be able to learn about new Bitcoin blocks; it needs at least one connection to an honest like-minded node. To make sure that doesn't happen, nodes will detect the situation where their peers give them blocks they consider invalid, and when that happens, more aggressively look for more nodes to connect to, and possibly disconnect existing ones if they really seem to have a different idea. In practice, this quickly teaches nodes on both sides after a fork to shun the other side.

Certainly if I download bitcoin core and start running a full node it will start connecting me to other nodes in the bitcoin network. How does that process actually go down?

Bitcoin Core specifically has a number of mechanisms that control connections:

It maintains a database (called "addrman") with IP addresses (and Tor addresses, if that's in use) of known other Bitcoin nodes, together with some statistics and whether or not it has made a connection to it. New peers to connect to are drawn from this database. At most 10 connections are made, in order not to overload the network. This database is fed new information through:

Nodes gossip IP addresses of other nodes (the  addr P2P message) to each other.
A few DNS seeds are hardcoded that can be asked for more IP addresses if needed. This is only when not enough connections can be established after some time, and typically only needed the first time the client runs. After that, nodes autonomously find more peers to connect to.
As a very last resort, if DNS querying doesn't result in decent network connections, Bitcoin Core also has a list of hardcoded IP addresses to connect to.

The user can manually specify peers to connect to, using the connect= or addnode= configuration file options, or the addnode RPC command.
Nodes can accept incoming connections from other nodes too (there is a limit on this too; by default 125 connections, and when it's full some rules will prioritize peers from others).

how would nodes connect to the network without running bitcoin core?

Other implementations have their own mechanisms, but there is a wide spectrum - all the way from just connecting to the wallet provider's server, to learning about peers and connecting to them like Bitcoin Core does.

Supposed I wanted to write some code to develop my own bitcoin client. What was that code look like?

That really depends on what you want to do. There are libraries in many languages that implement the P2P protocol that lets you talk to other nodes, or you could start from scratch just following existing protocol documentation and Bitcoin Improvement Proposals (BIPs) that affect the P2P protocol. Remember, there is no authority in Bitcoin that can tell you exactly what "the" protocol is; proposals are made, and software dedvelopers implement them or don't. You'll want to test things, by not immediately dealing with the Bitcoin mainnet, but first try on the Bitcoin testnet or the newer "signet" test network.

I really appreciate the length you went to answering my question. I think you were ability to correctly identify my ability level with this topic.


text:
Ledger Nano Mnemonic Generation and Questions
From the ledger documentaion we have

The device generates a sequence of 256 random bits using the true random number generator (TRNG) built into the device’s Secure Element.
The first 8 bits of the SHA-256 hash of the initial 256 bits is appended to the end, giving us 264 bits.
All 264 bits are split into 24 groups of 11 bits.
Each group of 11 bits is interpreted as a number in the range 0 - 2047, which serves as an index to the BIP 39 wordlist, giving us 24
words.

The result of this process is that your device will generate a single
mnemonic seed out of 2^256 possible mnemonic seeds (That’s one of 115
792 089 237 316 195 423 570 985 008 687 907 853 269 984 665 640 564
039 457 584 007 913 129 639 936 possible mnemonic seeds). Note that
while the first 23 words are completely random, the final word was
derived from 3 random bits and 8 calculated bits from the SHA-256
hash. This means that the final word can act like a checksum - if you
input an incorrect seed into the device while recovering it, it is
possible for the device to detect that the inputted seed is invalid.

What was the motivation to move 8 bits to the end and use 2^11 possible words instead of move no bits and use 2^8 words with a 32 word mnemonic resulting in a bijective correspondance between the set of permutations of words and the set of possible 256 bit sequences.
Because of the dependence of the last word on the first 8 bits, there are (2048)^24 - 2^256 \approx 2.95 x 10^79 invalid 24 word mnemonics. Does anyone know what would happen if you tried to input on of these into a ledger during recovery mode? Mathematically it should give a random 264 bit sequence and then either 1) check that the first and last 8 bits are the same - if not, error. 2) throw the last 8 bits away to obtain a valid 256 bit seed that would correspond to a different mnemonic, where the # corresponding to the 24th word began with the same 3 digits.
"The device generates a sequence of 256 random bits using the true random number generator (TRNG) built into the device’s Secure Element." This is the only part that startles me a little. I feel like I remember reading in the past that a lot of attacks happen because random number generators can be exploited because they are not truly random. How is the randomness of this TRNG tested?

What would "a word bank of 2^32" mean? There are no word lists with 4 billion entries...

@PieterWuille sorry, that was an error. I just edited

What was the motivation to move 8 bits to the end and use 2^11 possible words instead of move no bits and use 2^8 words with a 32 word mnemonic resulting in a bijective correspondance between the set of permutations of words and the set of possible 256 bit sequences.

Typical word lists are 2048 entries, and that seems to be well established as usable. Using 256-entry lists would mean significantly longer seed phrases, which are presumably less optimal in terms of convenience. Deviating from that just so that the bits align with 256 would seem very strange.

Because of the dependence of the last word on the first 8 bits, there are (2048)^24 - 2^256 \approx 2.95 x 10^79 invalid 24 word mnemonics. Does anyone know what would happen if you tried to input on of these into a ledger during recovery mode? Mathematically it should give a random 264 bit sequence and then either 1) check that the first and last 8 bits are the same - if not, error. 2) throw the last 8 bits away to obtain a valid 256 bit seed that would correspond to a different mnemonic, where the # corresponding to the 24th word began with the same 3 digits.

The text you quoted says the 8 checksum bits at the end are computed using SHA256 of the 256 entropy bits; not just a copy of the first 8 bits.
I don't actually know what happens if you enter a seed with mismatching checksum, but I suspect the device will just reject it.

"The device generates a sequence of 256 random bits using the true random number generator (TRNG) built into the device’s Secure Element." This is the only part that startles me a little. I feel like I remember reading in the past that a lot of attacks happen because random number generators can be exploited because they are not truly random. How is the randomness of this TRNG tested?

Randomness is indeed hard, but you buy a hardware wallet because you trust its security more than you trust a software wallet running on general purpose hardware. If that's not true for the RNG in it, don't use it - you can always generate it elsewhere and import.


text:
Bitcoin Core gives error "Error reading from database, shutting down" during first time setup
I have recently installed Bitcoin Core on my Gentoo system via portage. During the first time setup, the initial download stopped at roughly 30% completion and gave the error "Error reading from database, shutting down". Restarting the program would set the progress back a small amount and restart the download, but it would always fail at the same point. I tried running Bitcoin core with the -reindex flag, and after doing so the download made it farther - to 43.81% - but then gave the same error. As I mentioned before, this is during the first time download of the blockchain; I've never used the program for anything else.
Some points that may be important:
I have more than enough disk space.
I dualboot with Windows, and as such the first time I ran Bitcoin core I had a clock skew. It ran for a minute or so before I realized, closed the program and corrected it. I haven't run it with a clock skew since.
It almost always fails on the same point - in this case, at anywhere from 134577 to 134579 blocks remaining. After restarting, it starts the download at 134609, giving a difference of roughly 40 blocks. (As I mentioned before, however, it failed much earlier before reindexing.)
I installed through portage, and as such I have no reason to believe that any dependencies are missing.
Does this look like a hardware issue? I've yet to do any disk tests other than a simple ram test, which returned no issues. Thank you in advance.
EDIT
Here are some basic specifications for the machine in question.
CPU: Intel core i7 3930K, 6 core
GPU: Nvidia GTX 980 (open source drivers)
16GB of RAM, 8GB of swap
Kernel Version: 4.14.52-gentoo

How much RAM do you have on the machine that you're running this on.  Does increasing the swap space on the machine help?

16GB of RAM, 8gb of swap. I haven't attempted to increase swap size.


text:
How to run a bitcoind on a hosting securely?
I'm developing a web site which should receive bitcoins and display the amount of received coins. As this is my first bitcoin experience I would like to know if the following scheme is secure enough:
I'm planning to run a web server and bitcoind with the encrypted wallet on a hosted server in a datacenter. No payments will be made through this bitcoind so the wallet will be locked all time and just will be used to monitor the transactions. The front end will communicate to bitcoind by JSON-RPC. There will be regular backups of the encrypted wallet file.
The payments will only be done from another machine which has a copy of the wallet and is secure (both physically and from IT side).
Will this work?

If the server doesn't need the private keys, why store them there? You could just keep the addresses so you can monitor incoming payments and then you are even safer.

@gsp92 Thank you very much for your input. Can you point me to a reference where I can read how to separate private and public key and make a wallet with public keys only?

I have very limited knowledge of bitcoind, and my suggestion is a little long for a comment so I will post it as an answer. Stand by.

As I mentioned in the comment above, unless you absolutely need your private key to be on the server, I would recommend only storing the address for checking purposes.
I've never used bitcoind before, but the best solution I have seen for accepting bitcoin payments would be to use something like bitaddress.org to generate a huge number of address and private key combos. Upload the addresses into a database on your server, and then when a customer goes through a transaction your server can select a brand new (never before used) key  for them to pay to. Then you can use a bitcoin API like the one provided by Blockchain to validate that the correct number of bitcoins have been sent to this address, and once it confirms, you can release the product to the client.
Then, at your leisure sometime in the future, you can collect your Bitcoins from the addresses having received payments.
This method presents a number of distinct advantages IMO:
1) Private keys are never on the server, you are considerably less vulnerable to an attack.
2) No requirement to keep the entire blockchain on your server, saving space and bandwidth.
3) Funds from each client are held separately (initially) so if you had multiple sales in a short time period, you can still identify each separately (if you used one central wallet, you run the risk of confusing the payments)
4) Your central wallet is slightly safer - the client no longer pays directly to it. Important to note that this makes very little difference because they'd still need your private key, but peace of mind is nice.
I appreciate that this isn't quite the solution you were asking for, but I think it may still fit your usage case & be more secure.

If you are implementing the web server in Python, you can look at this BitcoinTalk post where the creator of one of the first Bitcoin Faucets, freebitcoins.appspot.com, shares his implementation.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
What clients support escrow transactions apart from Blockchain's MyWallet?
By escrow transactions I mean basic 2 of 3 multi-signature transaction like here https://blockchain.info/wallet/escrow
Are there any other bitocoin clients where I can do this?

Electrum allows to escrow transaction but requires you to use the command line interface.
I found this gist explaining how to do it.

Not exactly a duplicate of this question, so I'm not marking it as such, but I think that question does answer yours. In a nutshell, the Bitcoin reference client can do this. You have to use the command line or JSON-RPC, though. Gavin Andresen wrote a script for it.

Here are a couple options available to you.  I am not associated with any of these sites and would recommend you do your research before proceeding with any of them.  Providing any said research to other readers would be helpful for the community.
BTCrow (btcrow.com/)
BitMit (https://www.bitmit.net/)
BitCoin Escrow (http://thebitcoinescrow.com/)
FlipBit (http://flipbit.com/)
Cheers!

Thanks but I'm looking for a client which can do mutlisig transactions for escrow, without any specific thirdparty escrow service.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Do wallets save a backup of my private keys and seed?
Let's say that I do not trust a lot in myself and my ability to not screw my computer, so also my private keys.
Knowing the concept of wallet HD and having seeds to recover all your private keys linked to the seed, how does this work, from wallets perspective?:

Does a wallet provider save my private keys (in a secure way) in order to restore them in the future with my seed, if I require it? (Providing paraphrase or information required)
Depends on the wallet provider whether they save a encrypted copy of my private keys to restore them in the future (just in case I remove or lose my local wallet file) or others that they do not save any private data, leaning the entire private keys in my local machine ?

This question is related with some FaQ from.some wallet providers, which they offer the option to restore your private keys providing the seed. This looks like they have a copy of your private keys, otherwise whether you are the only one that have them  and you lose your private keys, there is no way to restore it.

It depends on what wallet you are using.
Most people prefer to use wallets that do not have knowledge of your private key outside of your machine, such as hardware wallets, electrum, Bitcoin Core. This is for safety, since any provider who has your keys could easily decide to move all your coins to an address that only they control, and rob you blind.
That said, there are some services that offer custodial services, usually in conjunction with exchange services. In fact, every exchange is a custodian keyhold for the deposit addresses they assign you on their platform.
A few years ago, perhaps there was some merit in trusting a reputable custodial service who had the infrastructure to always backup keys. In 2018, BIP32/39/44 can make a backup as simple as writing down a handful of words and keeping it very safely, so maintaining control of your keys is the preferred option.

Thanks for the details Rachav :) Do you know some wallets that are kind of hibrid between pure local wallet and web wallets? For example, a local client wallet that stores the privates keys locally but also the wallet provider save them (encrypted with your private key for example, so they don't have access to it).

@DanielFernandez: A simple way to achieve that is just to encrypt your wallet file (or seed phrase) however you like, and upload the encrypted file to a generic cloud storage service.

Your private keys are all derived from the seed so in order to restore your wallet you need your seed and the software used to create your wallet. As for whether your wallet provider holds your private keys or not that depends on what provider you mean. For instance Blockchain.info holds an encrypted copy of your wallet which contains your private keys however only you have the password to decrypt that.
Generally speaking it's better to install a software wallet on your own device instead of using a web wallet. In most cases these can also be backed up by writing down the seed words.


text:
Why seed from BIP39 mnemonic phrase is much longer than entropy?
BIP39 converts mnemonic
"zoo zoo zoo zoo zoo zoo zoo zoo zoo zoo zoo wrong"
to the fairly long seed
"b6a6d8921942dd9806607ebc2750416b289adea669198769f2e15ed926c3aa92bf88ece232317b4ea463e84b0fcd3b53577812ee449ccc448eb45e6f544e25b6"
Good tool to play around https://iancoleman.io/bip39/
What's the point of so long seed value when the entropy value of random 12-words mnemonic is much smaller?

2048 words list dictionary = 2^11 => 11 bit
12 words * 11 bits = 132 bit
in hex format (4 bit for character) is 132/4=33 charter long, vs 128 of seed

This is specified in BIP 39 itself:

To create a binary seed from the mnemonic, we use the PBKDF2 function with a mnemonic sentence (in UTF-8 NFKD) used as the password and the string "mnemonic" + passphrase (again in UTF-8 NFKD) used as the salt. The iteration count is set to 2048 and HMAC-SHA512 is used as the pseudo-random function. The length of the derived key is 512 bits (= 64 bytes).

So the seed that's used is essentially a 512-bit hash of the mnemonic, which explains why it's 512 bits long (= 128 nibbles).
The BIP doesn't explain why they do this, but I would assume that it's convenient to know that the seed will always be a certain size, regardless of how much entropy is actually used to generate it.  This means, for instance, that key generation code can be written to always work on 512 bit seeds.

The need to have a constant size seed doesn't explain the use of PBKDF2 over say, SHA256/512 which are already used. My assumption would be that the PBKDF2 function was chosen so that in the event the mnemonic phrase is leaked, but is password protected, it is computationally less feasible to brute force passwords under PBKDF2.


text:
Bitcoin backend integration - Do I really need the notion of "a wallet"?
I am developing an application that will accept Bitcoins targetting some specific "items" in my application, and later transfert some of those Bitcoins to externals addresses.
In many similar projects I read about, they use the notion of "wallets". For example if you use Bitcoinj: it seems the first thing to do is to create wallets!
I understand what GUI wallet softwares are and why those are important for end users. But in a backend application, if I understand correctly, a wallet is simply a seed that can generate some private/public keys pairs in a deterministic way. Is that correct?
So, in the end, the only thing that really matters in order to be able to manage the coins sent to "a wallet" (sent to a specific user or for a specific "item") is to know the private keys that were involved.
In my application, I plan on generating a new private/public keys pair for every new "receive coins" transaction. Not from a seed, simply a random pair! And since I already have a database, I was thinking about storing the generated keys in a table (the private keys begin encrypted), with information about what the transaction was for.
And when I would need to send coins to external addresses, I would decrypt some private keys, create inputs using them, create a transaction that I would sign, and then broadcast that transaction.
In other words, currently I would not really have the notion of "wallets".
So my question:
Am I missing something about what "a wallet" actually does (in a backend application)?
Do I need something else in order to implement the "receive/send" process I'm talking about or is my database table with all the private/public keys pairs enough?

I understand what GUI wallet softwares are and why those are important for end users. But in a backend application, if I understand correctly, a wallet is simply a seed that can generate some private/public keys pairs in a deterministic way. Is that correct?

Correct, but you miss the most important feature of a wallet...crafting transaction / bitcoin script! And that part is not trivial as it involves such things as coin selection algorithm, bitcoin standard script +~segwit(also for generating segwit address), signatures etc.
What matter in the end are the key indeed... but you should not underestimated what feature does a wallet provide and instead of persisting tons of encrypted priv key you should definitively take a look of HD wallet (BIP32, BIP44, BIP49, BIP84)which are "basically" a determinist way given a seed / mnemonic, to handle an infinite amount of key. This way you could just persist paths.
See a wallet more like a tool, or a library a module a package whatever, you would require that I'm pretty sure, because when you mention "create inputs using them, create a transaction that I would sign, and then broadcast that transaction", that's what a wallet does.
About you received/send, I guess you have to handle or take into account a given transaction state, mined, validated (~6 confirmations, depends your requirements) or not. This way your app will behave differently. Basically you need a node or an API and a wallet "package" tool which will bridge the gap between your app and the node.

Thank you for your very complete answer! I still have a question though: couldn't I simply use functions provided by libraries like Bitcoinj or bitcoind RPC to compute all those complex things, but without using their "wallets"? Thing is I would prefere not to be bound to a specific library and have to have wallets directories of a specific format as required by the library!

Of course you can, take a look at bitcoind rpc doc labels for instance upper right in black. Let's say it describe what set of function represent a wallet (Under wallet section). Just take what you need but keep in mind that "basically" all function excluding wallet's one are what data does a bitcoin node provide. You could consume that but in real practise the wallet consume it the most. Hope it's clear!


text:
Local balance query for Bitcoin address
I would like to be able to query the balance of an address (not belonging to me). The data should come from a fully synced bitcoin full node inside the local LAN and I do not want to query any api services outside my LAN like blockchain or blockcypher. Of course, API calls inside my LAN would be fine and even preferred.
Which query technique could you recommend me? What's the minimum hardware requirement? I guess that would imply a large index of addresses, would I have to count with the same space as the blockchain data itself?
I know this question has been treated before, but most solutions posted there are not really up to date or not supported anymore. What's out there as currently supported solutions?
Thank you in advance
Edit:
So far, my research is that bitcoin-cli cannot reveal transactions based on an address, the txindex=1is only useful for using the RPC call getrawtransaction which doesn't allow searching for addresses.
On the other hand, I've installed btcd: it allows using btcctl searchrawtransactions address which at least prints out ALL transactions. That brings me closer, but still doesn't provide me the final balance. Would that have to calculated from the single transactions? If so, how?

I'm sorry, I wrote my reply thinking of transactions (while mentioning addresses) and you were asking about addresses. Also thanks to @Murch for pointing it out on my answer's comments. Deleted to avoid further confusion.

"What's out there as currently supported solutions?"  -- Software recommendations are off-topic here. Please edit the question to focus on a single problem.

The first problem is that Bitcoin does not use the addresses for working, so Bitcoin does not have the need to give the information for addresses.
So with your test have some problem, that is:

Bitcoin Core (bitcoind): not have the method for calculating the balance of the address because a wallet has more addresses and more private keys, for extract all addresses with the balance you can scanning the blockchain and store the information of the address.
btcd: Have the method for calculating the balance of the address but you don't have all addresses used inside the bitcoin blockchain, so for extract all address you can scan the blockchain.

So If you want to calculate the "balance" of the address, you can scan the blockchain from the genesis block to the last block and save the address (contains inside the scriptPubKey) for calculating the balance with the simple sum.
if you use the bitcoin core you can get the information with the RPC commands, but you must manage "how to calculate the balance of the address" or you can use and parser for a scan the blk file (with the parser you can be faster in your analysis).
I want to add more information to the address of bitcoin.
The bitcoin wallet works with more addresses and more private keys, so in the more cases, the balance of the wallet is different from the balance of the address.


text:
Difference between Sign Message and Sign Transaction
Using "Sign Message" one can prove ownership of an address.
This happens by signing a human-readable string using the private key belonging to the address.
The owner then provides the address, message and signature, and anyone can verify that they indeed hold the private key of the address.
The ECDSA signature is normally provided in 65-byte format (Recovery byte + 32-byte R + 32-byte S) and base64 encoded.
However, there is also another type of signature, which is used for "Sign Transaction".
For each input of a transaction, there is some data to be signed. This data is binary data, normally provided in hex format.
The corresponding signature is provided in 70-, 71-, or 72-byte DER format.
In both cases, the message (message string in the first case, and binary per-input data in the second case) is transformed into a byte array, and the private key is used to generate an ECDSA signature for that message byte array.
The output format is different, but the 65-byte format can easily be converted into DER.
Consequently, I thought that I could just use "Sign Message" (with a minor modification, so that it takes a hex string instead of an ASCII/UTF-8 string) to sign a transaction's hex-encoded binary per-input data.
I did that, converted the signature to DER format and tried to send the transaction, but it didn't work, because the signature was wrong.
So my question is, what is the difference between how "Sign Message" and "Sign Transaction" generate signatures, and why aren't they compatible?
Is the difference that for "Sign Message", first a digest is created, which is then signed, while for "Sign Transaction" (in the meaning written above), the input is already a digest, so it shouldn't be hashed again?
Thank you!

@andrew provided helpful information that a major difference between "Sign Message" and "Sign Transaction" is that "Sign Message" internally prepends a custom string to the textual message being signed.
Also, his answer addresses the case when you do all steps of the TX creation and signing.
I tried to keep my initial question generic, but now I see that I should have mentioned that the per-input TX data I want to sign is BlockCypher's toSign array.
In their documentation it's not immediately clear, but with some investigation, I've found out that the provided toSign hex strings are already hashed twice using SHA256, so they indeed just need to be signed.
Probably there are also other systems that provide such digests for signing.
So, in summary, if you are using an external system for creating TXs, which provides you data to sign with your private key, that data might already be hashed. One good indicator for such a digest is that it is 32 bytes long.
There are multiple reasons why "Sign Message" will not work out-of-the-box for signing such TX data:

The TX data is binary data (usually in hex format), and "Sign Message" expects ASCII/UTF-8 strings.
"Sign Message" internally adds a custom prefix to the message being signed.
The TX data is already hashed, so shouldn't be hashed again.
The output of "Sign Message" is a 65-byte ECDSA signature, and it's not in DER format (can easily be converted).

However, the actual signing is the same for both "Sign Message" and "Sign Transaction".
Hopefully this will save some research time for others with a similar use-case.

I had issues like 'Error reading signatures and pubkeys: Invalid signature, wrong length' with signing tosign data provided by Blockcypher by using bitcore Message class. As it turned out I should have used crypto.ECDSA.sign() function instead. This Q&A was very helpful for me.

Is the difference that for "Sign Message", first a digest is created, which is then signed, while for "Sign Transaction" (in the meaning written above), the input is already a digest, so it shouldn't be hashed again?

No. It has nothing to do with the hashing. Furthermore, the input is not already a digest, it must be hashed before signing. Both the message and the input data are hashed twice with SHA256.
The difference between message signing and transaction input signing is that message signing prepends the string Bitcoin Signed Message:\n (with \n being a newline character, not literally \n) to the message before it is hashed.
Additionally, the 65 byte signature produced by message signing has a byte at the beginning that is not part of the cryptographic signature itself. Instead it is a recovery id so that the correct public key can be recovered from the signature during signed message verification. In transaction signatures, there is a different byte appended to the end of the signature which is the sighash type, usually it is the byte 0x01.

Thank you, @andrew
You write that in the 65-byte signature, there's a recovery byte at the end.
The recovery byte is at the beginning (as mentioned in my question above).
Also, In what TX signatures is there an additional byte at the end? In DER format, there are other DER-specific bytes, but not related to signhash.
I tried to keep my question generic, but maybe I should have mentioned that the TX input I want to sign is BlockCypher's toSign.
I think that's already a digest, so it just needs to be signed.

My bad, the recovery byte is indeed at the beginning. If you properly DER decode a signature that you put in a transaction (or a signature you get from a transaction), you will find that there is an extra byte at the end. That byte corresponds to the sighash type.

It should be noted that the prepended "magic string" includes a prefixed size encoder byte, so that it reads "\x18Bitcoin Signed Message:\n" (This is different from the recovery byte in the signature)


text:
Can Taproot be used to commit arbitrary data to chain without any additional footprint?
A common way of committing some data to the chain is through an OP_RETURN with some hash. Doing this marks an output unspendable, but the output still requires some footprint (and cost) on the chain as it contributes to the TX hash.
So I'm wondering if it is possible to embed some data in an OP_RETURN in a script of a TapLeaf which is never intended to be part of a spend. If we assume that the Taproot output is spent using the key path, and it has already confirmed, it should be possible to later reveal the paths and scripts containing the OP_RETURNs, and it could be proven that the TapTweak hash did indeed commit to these hashes via the TapBranch hashes.
A single transaction could commit to a large number of OP_RETURNs which have no on-chain outputs, but whose hashes are indirectly committed to the chain. These could be revealed individually to whatever parties they concern. The paths and scripts would have no presence on the blockchain.
Maybe this doesn't even require the OP_RETURN as you could just use the TapBranch/TapLeaf hash itself as the data you wish to commit.
Would this not significantly reduce the cost of services like colored coins, timestamping services, etc, which commit their data directly onto the chain?

Yes, you can commit to data this way, but there is nothing special about Taproot here.
The Pay-to-Contract construction used to tweak the root key in Taproot is generally usable, and has in fact been used for exactly this purpose. In fact, it is one of the mechanisms used in OpenTimestamps to commit to timestamped data (including a variant sign-to-contract which makes a signature commit to data rather than a public key). It is also used in Blockstream's Liquid pegin mechanism, where transfers from Bitcoin to Liquid pay to an address derived using a tweaked key of the federation, which commits to the transfer information.
In fact, using an additional leaf in Taproot for this is overkill, and would inevitably increase the depth in the Merkle tree for at least one other leaf - resulting in increased spending cost. Just using pay-to-contract or sign-to-contract directly on one of the public keys involved can be done with zero cost to on-chain spending policies.

Would this not significantly reduce the cost of services like colored coins, timestamping services, etc, which commit their data directly onto the chain?

Decent timestamping services already use better techniques.
For colored coins it depends whether they just need timestamping or actual publication of data. My understanding is that it's generally the latter (as otherwise, they need another censorship-resistant channel to transfer the actual data, which could then probably be used directly).


text:
Is block height always sequential?
Is block height always sequential? Can it serve as a 100% reliable block counter in case if I want to check whether I missed a block or I need to account for any corner cases?

Block height is by definition sequential in the sense that adjacent blocks will always have heights that differ by 1.  But there is a chicken-and-egg problem: in order to compute the height, you have to have access to all the blocks in between the genesis block and the current one.  (The height is not recorded in the block itself.)
There's also the issue that multiple blocks can have the same height, if they are on different branches of the chain.  So just because you have one block at every height, doesn't mean you have all the blocks on the main chain; some of them might be on orphan branches.

Block heights on the "valid blockchain" are sequential, but a block that is produced might become reorganized out of the blockchain (as a result of a soft fork in the network). This does happen occasionally and isn't anything to be concerned most of the time.
Those blocks that are reorganized out of the blockchain still have a height, but that is still in sequence from its predecessor, but the block on the main blockchain will also have that height. It is possible that your blkXXXXX.dat contain a few blocks that are not on the blockchain.
Genesis block(0) --> 1 --> 2  --> 3 Main Chain
                       \-- 2 (reorged out)


text:
What's wrong with this getrawtransaction JSON-RPC call?
This is a very simple question. I'm trying implement a client for Bitcoind JSON API and I'm stuck on this getrawtransaction call:
{"id":"getrawtransaction",
"param":["91eb6cb34b9253cce0ab5da171a0f5aae9d3d36208e23062ba61cd9e21c09a3d",1],
"method":"getrawtransaction","jsonrpc":"2.0"}

As the result I get:
{"result":null,"error":{"code":-1,"message":"getrawtransaction <txid>
[verbose=0]\nIf verbose=0, returns a string that is\nserialized, hex-encoded data for
 <txid>.\nIf verbose is non-zero, returns an Object\nwith information about
 <txid>."},"id":"getrawtransaction"}

Could someone point me at what can be wrong? I'm sure it's something simple but I cannot get it. Bitcoind v. 0.8.

I think that should be "params" not "param"

Thanks! Shame on me. I knew it would be something stupid:)

Hehe, happens to all of us. :)


text:
Are Bitcoin ASICs good for anything else other than mining?
If I purchase an ASIC (hypothetically) and then for whatever reason Bitcoin disappears, becomes defunct or otherwise, is there anything else I could possibly use it for, such as SHA256 cracking - or is there no other possible use and does it become an elaborate doorstop?

Also if there was another use, keep in mind that even if you don't want to use it for that, you could at least sell it

No, Bitcoin ASICs can not be reused for any other purpose, unless something were created that uses the exact same hashing technique.
Unlike what some ill-informed radio talk show hosts say, the mining network is cryptographically incompatible with SHA256 password cracking, the block chain itself is not a conspiracy by the Russians as a rainbow table.
Besides most hashed passwords use a salt, and the manner it is typically done is incompatible with what is being done in Bitcoin.
So, no.  The ASIC will forever be exclusive to the Bitcoin unless another RFC or standard requires a proof of work (a la Hash Cash) for some other purpose.

You are mostly correct; though I have to remark that you can also use it for mining altcoins that use sha256 hashing (see http://dustcoin.com/mining).

unless something were created that uses the exact same hashing technique. covers alt coins...


text:
Which percentage of miners are using ASICs vs GPUs or FPGA? Is it possible to know?
One of the common arguments against Bitcoin is the energy consumed by the network. The total hash-power of the network can be obtained fairly easily, but I'd like to estimate how much of that is due to ASIC, how much to FPGAs and how much to GPUs (if anybody's still using them).
If there were a way to separate the population of different ASICs themselves, a much better estimate could be made about the total power consumed by the network.
Any ideas??


text:
Why do dustcoin.com and coinwarz.com give opposite results of profitability?
It's been several days in a row that I go to dustcoin and see Dogecoin being listed as less profitable than either Digitalcoin and BBQcoin, with the latest being the most profitable.
But then if I go to coinwarz I see the inverse: Dogecoin is the most profitable, next Digital and finally BBQ.
The prices listed for them are not that different, nor is the difficulty - below the values. E.g. the values at the time of writing this question - first columns are from coinwarz and then the values for dustcoin:
coin   vs. BTC                diff.    vs. BTC   diff.
BBQ  0.000048 (Cryptsy)      4.3986 0.00005165   3.192
DGC  0.00036494 (Cryptsy)   11.3679 0.00035000   8.062
DOGE 0.00000031 (Coins-E)  286.6240 0.00000028 271.845

I would've thought that getting the profitability would be a straight away calculation, given the hashpower at your disposal vs. the avg. reward for a coin, plus the difficulty of it's network and the value it commands on the market.
So I can't explain why each site gives the exact opposite answer for these coins in particular - my inner cynic suggests at least one of them gets paid to artificially increase/decrease the profitability of some coins, but I find that hard to accept as well.
Any ideas?

Works, though I find it's interface a bit too... Spartan, so to say. "Minimalist" would call it others - probably I'm too spoiled by seeing the 2 I wrote about too much :)

First off, don't listen to so called "profitability" calculates they are always wrong.
and here's why:
Each "profitability" site bases the profitability based on the current block on that network vs bitcoins current block. Well most of these coins have a limited exchange volume on exchanges (some as low as 0.001 btc) yet because of that activity they are deemed to be more valuable. What happens next is a huge dump of coins and a market crash. This leads to an increase in diff and a decrease in profitability. Using that theory, you can see that these sites are not going to be correct. Especially depending on the exchange they use and the rates they use. Most coins only convert from ALT->BTC. So that means they will take the BTC/USD price from somewhere like bitstamp and then take the ALT/BTC price from the exchange of choice and finally it will take BTC*USD.
BEGIN EDIT
Simple formula these sites use
$profitability = $USDperBitCoin * $BitCoinperAltCoin
END EDIT
tl;dr
Exchange rates and exchanges used.
Low volume of trade.
Unique Difficulty filter algos on unique coins
I suggest you ask the owners of the sites this one.

There must be a universally correct way to derive profitability. I mean, the core business of Middlecoin and other so-called "multipools" is precisely to always mine the most profitable coin, sell it immediately for bitcoins (some would say "dump it") and pay miners. And so far they've had some pretty spectacular sessions of very high profits, so clearly they're onto something...

no, there's really not. each coin has a unique algo to determine the difficulty. Most use simple algos like bitcoin, but others (such as freicoin) use filters that are best described as sliding scales.
Multipools really don't make you any more money than staying on a coin or bitcoin. This is why: as soon as the coins are in the miners wallets they are dumped and you make less than you thought you would.
multipools are the worst idea you can imagine. they are Hash attacking networks pure and simple.

Hmmm. Once you mine the most profitable coin,  it can immediately be converted to a relatively stable coin of choice (usually lite coin or doge). The multi pool principal is sound and can increase profits more than 25 percent, but will soon change when scrypt asics arrive en masses in a few months.
Yes, I do believe that coinwarz gets advertising money for listing currencies at the top!


text:
What's the most efficient way to create a raw transaction with a specific fee rate?
There seems to be a chicken and egg problem when it comes to creating a transaction that has a specific fee rate (sat/kW).
In order to set a specific fee rate, you need to know what the total weight of a transaction will be. But in order to calculate the total weight, you'd presumably need to have the complete transaction, including some kind of temporary value for the fee.
I can imagine there are some edge cases where the input only has enough to cover a small fee, meaning that setting a high fee would require adding another input to the tx, complicating the tx size estimation.
Are there any efficient solutions for this situation?

Yes! The trick is to count each input with its effective value rather than its nominal value.
Transactions are composed of three parts: the overhead, the inputs and the outputs.

Outputs:
The size of the output scripts is determined by the recipient addresses, so their size is given by the transaction instructions. We do not know whether we will need a change output yet, but we know what size it would be if we created one.

Overhead:
The transaction overhead is 10 bytes for non-segwit transactions, or 42 WU for segwit transactions¹. You can probably guess which one you'll need, but in the worst case you overestimate by 0.5 vB. We can therefore consider the overhead size also to be fixed.

Inputs:
The tricky part is the transaction inputs. You do not know how many you will need, and might not know what input script sizes they'll have if you have inputs of various address formats. Additionally, signatures are not always the same size. However, we do know the maximum size of the signature, and the size of the input script for each UTXO. Together with the target fee rate, we can calculate the effective value:
effectiveValue = utxo.value − feePerByte × bytesPerInput²

The selection target starts out as the sum of the recipient amounts. We simplify the coin selection problem by adding the fixed costs to the selection target. We then perform the selection via counting effective values rather than nominal values. Since each input has paid for itself already when it is selected, we no longer need to worry about an input's fee impact after selection; instead each effective value goes fully towards the selection target.
In regard to the change output, there are two strategies. One, if you are aiming to build a transaction that avoids creating change, e.g. by using Branch and Bound selection, you keep the target as is: target = recipient amounts + fees for fixed transaction parts. Two, if you are using a strategy that will create change, you add a buffer to the selection target that is large enough to cover the fee of the change output and leave enough for a good-sized change output: target = recipient amounts + fees for fixed tx parts + change output cost + minChange.
For the signatures, you estimate using the maximum signature length. Since the signatures are part of the witness for segwit inputs, this results only in a small size overestimation which will lead to minutely overshooting the target fee rate.³ Alternatively, you can use signature grinding to save an expected 0.5 bytes per signature and estimate the fee rate more precisely.

¹ Assuming that the transaction has no more than 253 inputs and no more than 253 outputs. The input/output counter will take 3 bytes instead of 1 byte for greater input/output counts.
² Effective value was proposed in section 5.3 of An Evaluation of Coin Selection Strategies, Erhardt 2016, disclosure: authored by yours truly.
³ Overshooting the target fee rate is preferable; it costs only a few satoshis and slightly increases the priority of the transaction. Undershooting the target fee rate is problematic, since some payment processors require a minimum fee rate to accept a payment and falling below the default minRelayTxFeeRate can prevent a transaction from relaying altogether.

Kind of, but coin selection is hard in general. The idea is that inputs pay for themselves. It is easy to compute how much in fees you will need to pay for an input at a given fee rate. So when you do coin selection, instead of selecting on the actual value of the input, you use the value of the input minus the fees it will pay. This is known as the effective value.
Then for your selection target, you want to include the fees that will be paid for the outputs and the transaction overhead (i.e. version and locktime fields). So the overall target is the intended amount plus the fees of the non-input parts of the transaction.
Now you can select up to the selection target using the effective values and this should cover the  transaction fees for the transaction at a given fee rate.

However there is a minor issue here with change outputs. The inclusion of a change output is going to effect the end fee rate. The strategy that Bitcoin Core employs to deal with this is to use two different coin selection algorithms. The first algorithm searches for an exact match with a bit of buffer. This allows us to ignore the change output for this algorithm. If a solution is found, then there will not be a change output.
If the exact match algorithm fails, then we can assume that a change output will be created. So now we include the change output in the transaction and its fees in the selection target. We are basically guaranteed to have a change output, so it is unlikely that we will overpay by including the fees for a change output in the selection target.

Coin selection is hard. There may be some other heuristics as well, e.g. not creating dust change, but higher amount change. This is for selecting more input than necessary, which is against the heuristics of choosing as few coins as possible.