text:
HD Wallet with BIP44 - workaround of deriving public keys knowing only a xpub
My goal: I don't want to require a private key to hierarchically derive new addresses.
Sure, I can create a batch of addresses, given a private key, at first. But once I have surpassed that batch I'll require the private key again to generate more addresses.
I want to derive addresses knowing only a public key. I know this is possible with BIP39, but understand there can be security concerns involved with this, ie. if an attacker stumbles upon an xpub and xprv they can derive as many addresses as they want and be able to sign transactions using them.
Attempting to derive from a HD public key with BIP44 results in a exception stating a hardened path requires a HD private key. However, I have found a workaround, but I fear it is cheating and might sacrifice the benefits of path hardening in BIP44.
Here's an example:
// It starts off with a `userCode` that represents a BIP39 Mnemonic code.
const codeUser = new Mnemonic('select scout crash enforce riot rival spring whale hollow radar rule sentence')

// Convert to HD private key...
const hdUserPrivateKey = codeUser.toHDPrivateKey()

console.log(hdUserPrivateKey.hdPublicKey.toString())
// Gives: `xpub661MyMwAqRbcEngoXGfFNahZ5FzSDGqY8pWKTqo6vtXxK15otDNLXJmbeHV7DUjvPc7CAFhYp6hzBiTanr8rgoHPHf6NSgZAyejK5bk8MiW`
// But we won't use it...

// Instead, I can then derive a BIP44 without the `change`, `address_index` segments from `hdUserPrivateKey`...
console.log(hdUserPrivateKey.deriveChild(`m/44'/0'/0'`).hdPublicKey.toString())
// Gives: `xpub6CsrEMgU2f8uEGfFMvsPjKB9ekHuZiesLqSHLwCJuNFkP2uJGm7WjTo2gy95S4KEBc4etdodNQXAvn5Vsf4kupJQ1DKR4DMfcHwKdhQ3k6h`
// This is the xpub I can use to derive addresses without requiring the initial private key.

// So knowing this, I can build a HD public key given that xpub...
const hdPublicKey = Mnemonic.bitcore.HDPublicKey('xpub6CsrEMgU2f8uEGfFMvsPjKB9ekHuZiesLqSHLwCJuNFkP2uJGm7WjTo2gy95S4KEBc4etdodNQXAvn5Vsf4kupJQ1DKR4DMfcHwKdhQ3k6h')

const derivative = 0

// We can derive from it this path, but what is this path defined as? Are we back in BIP39 territory now?
const publicKey = hdPublicKey.deriveChild(`m/0/${derivative}`).publicKey

const address = new Mnemonic.bitcore.Address(publicKey)

console.log(address.toString()) // 12XyHwtmoq5w4VQ5mzcu6BQzdLqCLxUv5e

...and of course, I can increment the derivative as many times as I wish to create new addresses from the public key.
Whenever I wish to sign a transaction...
const codeUser = new Mnemonic('select scout crash enforce riot rival spring whale hollow radar rule sentence')
const hdUserPrivateKey = codeUser.toHDPrivateKey()
const derivative = 0

// BIP 44 derivation path for private key...
const privateKey = hdUserPrivateKey.deriveChild(`m/44'/0'/0'/0/${derivative}`).privateKey

Is this approach valid or am I dodging BIP44 standards?

This is, in fact, the intended approach and use of BIP 44.
The purpose is to derive to the account level with hardened derivation. Then export that xpub to be imported into other software which derives the receiving and change addresses. When signing a transaction, the wallet with the private keys can derive the entire path to get the individual child private keys needed to sign.

Ah sweet, thanks for the clear clarification. My main aim is to make exported seeds work with various well known wallets. Would that mean the initial seed is classified at BIP39 on import? But then derived to BIP44 to get a new xpub. Then further derivatives are worked out from there, which is fine? I guess I'm kind of looking for a definition of the steps I made.

You start with a BIP 39 mnemonic. When this is imported into a wallet, the wallet produces a BIP 32 seed and BIP 32 xprvs and xpubs. You use the BIP 44 derivation path to derive specific xpubs from the BIP 32 xprv. Those xpubs are then imported as BIP 32 xpubs in another wallet.

This has answered my question.


text:
How do I properly paytomany using electrum via script?
I know that I can use the GUI to do this and I"m glad to do so...however how would I do this using a bash script?  This is on a Mac but I would just change it for any OS in question.
#!/bin/sh

PAYOUTS=cat ~/Desktop/weeklySummary.csv
WP= /Users/${USER}/.electrum/wallets
PASS= "my_wallet_password"
# echo $PAYOUTS

cd /
./Applications/Electrum.app/Contents/MacOS/Electrum paytomany && $PAYOUTS -w $WP -W PASS

# this is not ready yet!  I have to figure out
# whether or not change address, a fee, and a
# from address are requried to use this

I get thrown some errors for a few missing details.  Am I required to have my wallet password, from address, and change address (I presume the answer is yes)  but furthermore, what are the -h, -v, -P parameters mentioned here?

WOW....so this never got answered by anyone else....okay let's share the answer for you all as I eventually figured it out but had forgotten that I wrote this question:
electrum paytomany "[[\"17Umi7fbYuKixnr8nRq16kMESWgJVc66s\", 0.05], [\"17Umi7fbYuKixnr8nRq16kMESWgJVc66s\", 0.05], [\"17Umi7fbYuKixnr8nRq16kMESWgJVc66s\", 0.05], [\"17Umi7fbYuKixnr8nRq16kMESWgJVc66s\", 0.05], [\"17Umi7fbYuKixnr8nRq16kMESWgJVc66s\", 0.05], [\"17Umi7fbYuKixnr8nRq16kMESWgJVc66s\", 0.05], [\"17Umi7fbYuKixnr8nRq16kMESWgJVc66s\", 0.05], [\"17Umi7fbYuKixnr8nRq16kMESWgJVc66s\", 0.05], [\"17Umi7fbYuKixnr8nRq16kMESWgJVc66s\", 0.05], [\"17Umi7fbYuKixnr8nRq16kMESWgJVc66s\", 0.05]]"

So each address to be paid must reside within its own array which is itself enclosed into a parent array of all those payout requests...the slashes are simply there to escape the quotes so they're included within the string.  I actually ended up doing something other than bash but utlimately this is the string you need to format in order to paytomany using electrum.  Still don't know what the -v nor the -h are for (op...nevermind they mean Help and Verbose)...-P for portable wallet (meaning it can be put onto something like a usb drive (a portable filesystem essentially)... Hope this can help some other fellow web traveller in the future ;)

You are a god! the documentation is awful, for thinking that the sentence  "outputs               list of ["address", amount]" implies this EXTREMELY specific data format at all

Note that the string '[["address_text", value]]' also works, by mixing single and double quotes you can avoid having to add escape characters \ to the whole thing. This is how the json.dumps library operates in python which is used to interpret this via json.loads  See: https://github.com/spesmilo/electrum/blob/master/electrum/commands.py

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Understanding Script's number encoding, CScriptNum
What is a simple explanation of CScriptNum, the number encoding used by arithmetic opcodes in Bitcoin Script?
What would an illustrative pseudocode/python example look like?
What should the encoding of the following integers be (in hexadecimal)?

42
88
257
17000
8589934592

When is it not 4 bytes?

See Also:
Why are arithmetic script opcodes limited to 4 byte operands?
Arithmetic operations in Bitcoin (OP_ADD, etc)
Compact integer in Scripts

Numbers in Script are represented as byte vectors of up to 4 bytes and interpreted as signed integers between -2^31+1 and 2^31-1. This is with the exception of OP_0, OP_1NEGATE and OP_1 through OP_16 which allow to represent small numbers (from -1 to 16) by using opcodes directly.
These 31 bits numbers are encoded in little-endian with sign-magnitude. Note the byte vector size is not fixed: there is no padding for small integers. Here is a few examples.

Integer
Script byte vector

-424242
[0x32; 0x79; 0x86]

-0x80
[0x80; 0x80]

-1
[0x81]

0
[]

12
[0x0c]

0x82
[0x82; 0x00]

2^31-1
[0xff; 0xff; 0xff; 0x7f]

The reference implementation of the serialization can be found here. The Bitcoin Core test framework has a Python reimplementation of the serialization. Here is a simplified and highly commented version:
def ser_script_num(n):
    res = bytearray()

    # If n is 0, return the empty vector.
    if n == 0:
        return bytes(res)

    # Encode n as little-endian. Done manually to avoid padding.
    abs_n = abs(n)
    while abs_n > 0:
        res.append(abs_n & 0xff)
        abs_n >>= 8

    # If n is negative and the serialized number does not have its most significant
    # bit set, we can just set it by adding 0x80 to its most significant byte. This
    # avoids requiring an additional byte just to set the sign bit.
    msb_set = res[len(res) - 1] >= 0x80
    if not msb_set and n < 0:
        res[len(res) -1] += 0x80

    # However it does mean that if the serialized number has its most significant
    # bit set we need to push an additional byte so 1) it's not treated as negative
    # if it's positive 2) it's not substracted from its actual value.
    if msb_set:
        # If n is positive add a byte so the sign bit isn't set. If it's negative add
        # another byte for the sign bit so it's not substracted from the number itself.
        if n >= 0:
            res.append(0x00)
        else:
            res.append(0x80)

    return bytes(res)

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What is the hashTypeCode?
After a transaction is serialised ready for signature or signature verification, a 4 byte hashTypeCode must be added (step 9 here). As it's derived from a single byte, it has 256 possible values.
What is the Hash Type Code?

Also known as SigHash, it is an instruction (in a transaction input) from a signer to a verifier to say "to get this to verify, you'll need to construct a transaction template like this." For each input in a transaction, a different hashtypeCode might be used.
It is defined in interpreter.h of the Bitcoin Core software:
/** Signature hash types/flags */
enum
{
    SIGHASH_ALL = 1,
    SIGHASH_NONE = 2,
    SIGHASH_SINGLE = 3,
    SIGHASH_ANYONECANPAY = 0x80,
};

From Raghav Sood:

SIGHASH_ANYONECANPAY is combined with the previous three via a bitwise
& (effectively addition in this case), for a total of six possible
values.

hashTypeCode
hashTypeCode Identifier
Tx fields used in signature

0x01
SIGHASH_ALL
All inputs and outputs.

0x02
SIGHASH_NONE
All inputs. No outputs.

0x03
SIGHASH_SINGLE
All input. One output.

0x81
SIGHASH_ANYONECANPAY & SIGHASH_ALL
One input. All outputs.

0x82
SIGHASH_ANYONECANPAY & SIGHASH_NONE
One input. No outputs.

0x83
SIGHASH_ANYONECANPAY & SIGHASH_SINGLE
One input and its corresponding output.

The procedure is described at bitcoin.it
See also:
bitcoin.org

The sighash type is decided by the signer when spending an input, not the creator of an output, as it's part of the signature they produce.

@PieterWuille Thanks, I have some confusion around this atm... will update more as I figure it out


text:
Why does OP_CHECKSIG/OP_CHECKMULTISIG remove the signature from the subscript?
When OP_CHECKSIG is used, all instances of the signature are removed from the subScript before it's included in a transaction template to be verified. See also step 3 in this excellent diagram. The same thing is done with OP_CHECKMULTISIG.
How could a copy of the signature(s) end up in the subscript? Why must it/they be removed?


text:
How many nodes and edges does the UTXO-coinbase graph have?
Imagine a graph where the leaf nodes are the UTxOs, the intermediate nodes are spent TxOs (vout), the root nodes are the blockheaders and coinbase merklepairs, and the edges are the data required to verify the transition between nodes (i.e. the transactions):
How many nodes and edges would this graph have, as of block 831276?
What would be the longest path through the graph?
Are there any interesting ways to categorise the graph from graph theory? Is it acyclic for example? How many disjoint sub-graphs are there?
What would be the mean/median/range number of leaf-to-root hops through the graph.

Small example sub-graph:
RootNode blockheader coinbase-merklepair (the number required to prove that the coinbase tx is in the blockheader's tx_root)
Edge0 coinbasetx
IntermediateNode Vout id from edge0 transaction
Edge1 a transaction (vin id == IntermediateNode and input_tx == edge0)
LeafNode the vout for an unspent output from edge1
Thanks to RedGrittyBrick and Murch for prompting clarification

What is a "coinbase-merklepair"? It’s not clear to me what "Edge1" is supposed to be. Should that be a second transaction that spent the output of the coinbase transaction to create a new unspent output that is the LeafNode? Since transactions can have multiple inputs and outputs, the graph you propose would have many different edges that represent the same transaction. If you instead make both transactions and TXOs nodes, and use edges to connect transactions with their inputs and outputs that might make more sense to me: e.g. [coinbasetx] -- [coinbasetx:0] -- [tx1] -- [tx1:0].

Proving that a transaction is committed to by the merkleroot requires a merkle branch or merkle proof.

@Murch I've made an update. Yes multiple edges would contain same transaction data, the distinction being the destination node. Perhaps your formulation is indeed more sensible.

@Murch yes edges to leaf nodes would require a merkle inclusion proof

Since you indicated in a comment that you are fine with multiple edges representing the same transaction, you get a node for each coinbase transaction and each TXO ever created, and edges to the tune of summing up across all transactions the product of inputs and outputs:
node_count = blockheight + TXOs_created
edge_count = Σ_tϵtxs(input_count(t)×output_count(t))

since each input in every transaction will will be connected to each output in the transaction with a separate edge representing the same transaction.


text:
Why does this transaction give me an error?
I am trying to script automatic payments without hosting a full node (my server doesn't have enough harddrive space). To do this I'm using electrum's python interface, and I have created a transaction, but when I try to send it, I get the error message: "TX rejected", "code: -22". Looking that up here, it appears that's an error either parsing or validating the transaction. It appears the error was thrown here.
But I cannot determine why. The transaction contains 2 inputs, 2 outputs, and the outputs + fee of 0.0001 = the size of the inputs (as of now) as verified via blockchain.info. The size of the transaction is under 1kb, so the fee is as I understand .0001. I sent an earlier transaction of .0001 with a .0001 fee through the electrum UI and it went through fine. So why is this transaction failing?
The raw transaction is:
01000000029802316bd53cdef6741b40b0a75cbffdfab270b0804f32096dda42594e19d179010000008b483045022100bd8bebb8491689604203806b66dc8c785cfb2362f2af630550bcd08d8789311c0220762ed5f310a15294e4fa9a5da9c4baff19e0f7c74a37430676fa3f7af3b50bb6014104f58c3d2d51eebf504ce13e1f5d3bb6f80c3df2e60212fef826dc2ce5bbe5f833b7ef684005fc6c28f80d7f27f70333471099edd4d41f4176261ae345a4128c46ffffffffd133f259c5e4bc4213136acb2fc4dbe5234732cfbfcc6a2845b83a744ea90e000e0000008b483045022100ec66b8827e4827b63a66840b1976cfc244a5ab20a047dd0d23e33285f63d7967022030c8f4b78277747dce2b085cf84acdd990704b0568c4d6e130a2361892b58e8d014104f58c3d2d51eebf504ce13e1f5d3bb6f80c3df2e60212fef826dc2ce5bbe5f833b7ef684005fc6c28f80d7f27f70333471099edd4d41f4176261ae345a4128c46ffffffff02a00f0000000000001976a9147d7cc528123aca5b820013445abbdff7530db71488ace01f0000000000001976a914742415bbd2c023df6f47e1fb6f5dac5316dc437588ac00000000

and I uploaded the decoded version (using electrum decoderawtransaction ... or https://coinb.in/decode-raw-transaction.html - both give the same output) here

The last link doesn't work

@StevenRoose I tested the link, and it works for me. But I uploaded the decoded version to http://pastebin.com/raw.php?i=ZGPbfen2 as well.

The transaction wasn't going through because of the newly implemented "dust transaction" policy that makes it so that no output of a transaction may total less than 54.3 micro-bitcoins. One of my outputs was below this threshold. So I fixed that, but then the output to the "change" address was pulled below 54.3 uBTC and it still failed. I haven't found a way to ensure electrum doesn't try to deposit less than 54.3 uBTC into the change address, so for now I'm sticking with a single, large input to avoid small amounts of change.

You are correct. Normally clients clients avoid this by just paying any input smaller than the dust limit into the fee.

// , Thanks much for posting the answer to this, even though you found it out on your own the next day.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
How do faucets/micropayment-based services avoid/minimize transaction fees?
Sites like bitvisitor or cointube give away very small amounts of bitcoins (I actually got a payment as small as .0000035 BTC when testing).
I am in the process of creating a service that awards users small amounts of bitcoins in exchange for doing some trivial work (akin to Amazon's Mechanical Turk). When experimenting, I tried using bitcoin-qt to send .0001 BTC to an address and it automatically added a .0005 fee onto it. Using Electrum, I couldn't get it to send until I set the fee to .0001 as well.
I highly doubt a fee like that was paid for the .0000035 BTC I received earlier), so how are these services eliminating or minimizing their fees? I know it's common to gather multiple "inputs" into a single transaction. Is it possible also to have a transaction with multiple outputs so that I can gather a hundred of these .0001 BTC payments into a single transaction and thus minimize the size (in kb) of the transactions and therefore the fees too?
EDIT
Sites like BitVisitor do indeed combine these payments (see here). So assuming there's no better way to minimize fees, how are they able to do this? I see no options in bitcoin-qt or Electrium for this.

You have two questions here, so I'll answer them separately.  As far as how do faucets they avoid transaction fees, the answer is that they don't.
Most of the faucet sites I've used use a service called Coinbox.me which collects requests for payments and bundles them up into one large chunk of payments.  This is how they minimize transaction fees - they don't send all the microtransactions at once, they wait until they have quite a few and then send them out.  I've looked at these transactions before when they're going through and they do have transaction fees that are being paid that is, as you said, larger than the amount being sent out.  But I have had transactions where it is being sent to at least a hundred addresses, so while one small amount may far less than the transaction fee, the total amount being sent out is significant enough that the service can pay a transaction fee while still paying out these microtransctions.

A fee in a transaction is created based on the difference between inputs and outputs. The standard client imposes a minimum fee on a transaction it creates, but there is nothing stopping a custom transaction to be made that does not follow those fee rules. Using methods like createrawtransaction, signrawtransaction and sendrawtransaction, once can easily make a zero-fee transaction.
A small note - just because a transaction is created and sent to the client does not mean that the transaction will be broadcast to the network. The client still follows its rules to prioritise "good" transactions (old coins, small transaction size, adequate fee, etc) and hamper the "bad" transactions (small outputs, big transactions, low fees). Same with miners - they can chose not to include spammy transactions in blocks.

alright, I've found all these methods. So the only reason BitVisitor attaches the bigger 4 BTC payment in each transaction is to increase the transaction's priority?

@Wallacoloo It might be just them sending themselves change back.


text:
Which merged mining method is preferable? What is the latest specification/practice?
If I'm right, there are at least two methods for miners to include a reference hash to a merge mined block in the coinbase transaction:

Commitment hash in an OP_RETURN out put in the coinbase transaction (a la RSK)
Commitment hash in the input script sig of the coinbase transaction (a la Namecoin)

A. Which one is preferable from a whole system point of view, and from the point of view of an individual miner?
There is also a specification for merklising multiple merge mined coins into one hash. It has the following note:

NOTE: This standard is used by Namecoin, but new merged mining data
should likely propose a new BIP to supercede it with something based
on p2pool's merged mining.

B. What is the latest specification, or widely accepted practice, for merged mining?
C. Are there any obvious improvements/fixes new merge mined systems could add?
Finally,
D. are there any other methods used for merged mining?
See also:
BIP301 (draft) Blind merge mining proposal
How does merged mining work?


text:
Unspent/Spent Flag in Blockchain
I just started getting interested in the blockchain a week ago, however there is something I can not just figure out.
Regarding the flag "Spent/Unspent" in transactions. I get that we flag "spent" the output of a transaction if this same output is used as an input in a future transaction.
-Suppose that we make a transaction (transaction 1), and this transaction get included in the blockchain (let's say block 1). We do not spent the output (consider a simple transaction single input & a single output) until 100 blocks later (transaction 2) in the blockchain.
I understand that in transaction 2, in the inputs, there is the info of the previous transactions (since we are implementing hash pointers)like shown below in "prev_out" (JSON format of transaction extracted from Blockexplorer api) :
'inputs': [{'prev_out': {'addr': '1PFtrRjbq4aLfM7k4tyLZ3ZAuTsgLr6Q8Q',
    'n': 0,
    'script': '76a914f423dfee34e4833104508103d89e316b4a49314088ac',
    'spent': True,
    'tx_index': 356882858,

Up to now, there is no problem, I can totally understand that we can add a flag spent/unspent for the previous outputs (inputs) in transaction 2.
Suppose now this is the output of transaction 1 (which we are going to spent 100 blocks later).
By the way, this is a real transaction, its hash: a117a7aab7e2db8ef5fd40e74f72d689e13b733136454d2340a244236fbed9eb
'out': {'addr': '1PFtrRjbq4aLfM7k4tyLZ3ZAuTsgLr6Q8Q',
   'n': 0,
   'script': '76a914f423dfee34e4833104508103d89e316b4a49314088ac',
   'spent': True,
   'tx_index': 356893407,
   'type': 0,
   'value': 16351264}

My question is:
You can see that the flag "spent" is True, How is this even possible ? Because during 99 blocks the transaction was unspent (flag=False). There is only one case possible for the flag "spent" to be True:
-The transaction 2 has to be in the same block as transaction 1. Otherwise if transaction 2 is in block 100 and the flag "Spent" of transaction 1 is True, it means that:
For 99 Blocks the flag was False and in the block 100, we change the flag of transaction 1, which is in block 1.
That means that we changed block 1, hence changed its hash --> impossible given the structure of the Blockchain.
Thank you for enlightening me !

It is important to understand that the spent flag you are talking about, is not a field of an output that is stored in the blockchain. It is rather created internally by the client interacting with the blockchain. The only fields of information of an output that is permanently stored on the blockchain are:

The Amount
The Locking-Script Size
The Locking-Script

All the full nodes on the bitcoin network checks if an output is spent or not by checking if the output is part of a structure called the UTXO set. In this structures all nodes keep all outputs which has not yet been spent at the last block of the blockchain the nodes have validated. If an output is part of the UTXO set, it means that the output is unspent.
For every new block in the blockchain, the nodes validates the block and removes any previous outputs in the UTXO set, that has been used as inputs in the new block.
Since a new node validates the blockchain by starting from the first block ever, called the genesis block, and updates UTXO set for every block in the blockchain, the UTXO set will contain all current unspent outputs once the node is synced with the rest of the network.


text:
Recovery on Bitcoin Core
I used bitcoin core late in 2016. Since then i haven't used bitcoin and have huge sum in my wallet. Lately when i wanted to get them back then i was told that i need private key.

If i installed the latest version of Bitcoin core with the same wallet
  phrase will i see that old address back with the balance

Also would like mention i don't have the old computer as i scrapped it and unfortunately no back up to. Is there anyway or am i out of luck.

Any suggestions would be appreciated. i promise a reward to anyone who
  helps me out?

have a nice week end all of you out there. you van reach me on narenderthind@gmail.com with solution or suggestion

Bitcoin core does not use a recovery phrase. If you have a recovery phrase, you were using another wallet

NO Mr Sood not a recovery phrase rather a pass phrase to enter the Bitcoin core wallet. Do u think if i enter that in to a new core client it will get mky old address to which has those bitcoins as i dont have an old wallet back up

If you don't have the original wallet.dat, you don't have any BTC. The passphrase only encrypts the data, it can not be used to recover it without the wallet.dat

@RaghavSood is this still true as of 2023? (bitcoin core/qt still does not support recovery phrase. If so it's pretty sad

TL;DR: No. Your bitcoin-core pass-phrase cannot be used to recover your bitcoin.

i was told the need of a private key

If you have the private key, you can recover access to your bitcoin.
If you have a recovery phrase and know which bitcoin wallet you used, you can recover the private key and recover access to your bitcoin.
Otherwise those bitcoin are gone forever.

i promise a reward to anyone who helps me out?

Expect to receive lots of email from scammers and very convincing professional conmen.
Don't give anyone your private-key or your recovery-phrase. The only circumstances you should ever do that is in front of a lawyer paid by you with proof of the helper's identity and a signed contract in her hand. Otherwise you will lose even more money.
People can help you without knowing your private-key or recovery-phrase. If they say they can't help you without those things, they are conmen.

If i installed the latest version of Bitcoin core with the same wallet phrase will i see that old address back with the balance.

No.

Any suggestions would be appreciated

Are all of the words in your passphrase in the BIP39 list? If so it is a recovery-phrase not a pass-phrase. If so it would be associated with a different type of wallet than the "bitcoin core" wallet.
If so, download a wallet application that uses BIP39 and try using your recovery phrase in it. Be very careful not to download malware. Start with wallets that don't require an online service to work.

Bitcoin core "passphrase"

Note that in the "bitcoin core" wallet, the Passphrase is used to encrypt your wallet. It is not a recovery phrase. You cannot recover access to your bitcoins using only the pass-phrase of a lost wallet.

Importing private keys into a new "bitcoin core" wallet
See also

How do I import a private key into Bitcoin-Core? where Rick's answer is

Run Bitcoin-Core
Select Help (to the right of Settings)
Select Debug Window
Select Console
If you encrypted your wallet, unlock it by entering walletpassphrase "YourLongPassphrase" 600 next to the > input box
Next to the > input box type importprivkey <bitcoinprivkey> (Note that the private key must not have spaces, remove them if they are included in your backup)

Example: importprivkey 5KJvsngHeMpm884wtkJNzQGaCErckhHJBGFsvd3VyK5qMZXj3hS

Note that the 'bitcoin core' wallet does not use a recovery phrase. You cannot recreate your private key from a recovery phrase using bitcoin core. The only "pass-phrase" used with bitcoin core grants access to the wallet, it is not the private key and cannot be used to recreate the private key.
Without the private key, the bitcoins are gone forever.

Thank you so much for all ur comments but my i am pointing to a particular query which stands as.... If i download the latest bitcoin core and enter the pass phrase which i used for my old core client will it still show up my old btc address which has my old mined coins. I dont have a back up of old wallet.dat file. Thanks once agian for ur suggestions

@user755926: See amended answer. Your old pass-phrase for bitcoin core only protected access to your wallet, it is not a recovery key. You need to know the private key. If you don't have the private key, your bitcoins are gone forever.


text:
What is the meaning of target-confirmation?
The API call listsinceblock has an optional parameter called target-confirmations.  Exactly what does it mean and how does it differ from the minconf optional parameter that many other calls take?  I understand the use of minconf, but I cannot at first glance see what's different about target-confirmations.

It's pretty much the same thing.  It probably should have been called 'minconf'.


text:
sendtoaddress questions

Does sendtoaddress automatically collect as many unspent outputs as needed?
Is there any way to specify the from address?
How is the fee computed? I just tested with two transactions:
a) 0.0001 xbt, fee 0.00017519
b) 0.001 xbt, fee 0.00017441.

1) Yes. See this question for more detail.
2) The from address will be an array of address that hold the unspent outputs. If you want to select which outputs are sent and from what address, you can use coin control in the QT wallet, or you can create a raw transaction using createrawtransaction, signrawtransaction, and sendrawtransaction.
3) Fee is computed by the number of bytes your transaction has, not necessarily the amount of the bitcoins you send. If you have a lot of small unspent outputs that go into your transaction it will make the transaction larger and thus a larger fee.
The code for the fee calculations

https://github.com/bitcoin/bitcoin/blob/0.9/src/main.cpp#L53
https://github.com/bitcoin/bitcoin/blob/0.9/src/main.cpp#L811

As of right now it looks like the tx fee per 1,000 bytes is 10,000 satoshis (0.00010000) to create a transaction.


text:
Blockchain-based polling/voting systems
In a blockchain-based polling/voting system, what is the best way to ensure that nobody votes more than once, and what is the best way to keep votes secret (if that's a requirement)? The info available in the blockchain should be 1) who voted and 2) how many votes for each option.

How can be both the votes secret and be visible who voted?

I think it's hard to accomplish to vote just once, because you are anonymous if you just look at the blockchain. Therefor you can open a new address, deposit bitcoin on it, and vote once more

Perhaps with one target address per voter? If the target is hit twice both votes are voided.

So you want to create 10.000 addresses if there are 10.000 voters? Then you can maybe do something like yes if the last number in the sending address is even, and no if odd. And if it's not public which addresses are used in the system, then no-one can void your vote. Unless 'they' track your wallet, and see you sending 1 satoshi to an address and conclude it's for voting. Then you can only count the first vote or something. Like sealing vote-paper on a real vote.

Is anyone working on a implementation of this?

You can use colored coins to do voting on the Blockchain.
You can start by creating "Vote coins", and issue as many of them as you have registered voters. Then you ask the voters to generate a Bitcoin address, and that they give it to you.
You then send 1 "vote coin" to each of those addresses. You designate two or more addresses, one for each candidate/proposition. Finally the voters have to send the vote coin to the address representing the proposition they want to vote for.
When the vote is closed, you simply look at the "vote coin" balance in each of the two addresses.
The votes are secret because nobody can tell which address belongs to which voter.
Coinprism is a web wallet that lets you issue, send and receive your own colored coins. It's quite simple to use.
For secrecy, the voters could go to the voting office, the officials verify that the voter is allowed to vote, the voter writes down his address on a piece of paper, and slide it into a sealed box. At the end of that registration process, the voting office has a box with 100 pieces of papers with each an address. They can then send the "vote coins" to those 100 addresses, without knowing which belongs to whom.
Of course, this is just an illustration, it's possible to achieve the same without requiring people to travel using the equivalent process implemented as open source software.

"nobody can tell which address belongs to which voter": The person who issued the vote coins knows which address belonged to each voter, don't they?

I agree with Nate's objection, the proposed solution doesn't seem to address secrecy. If more than one person knows a secret, it isn't a secret anymore.

Well, blockchain or not, it's physically impossible to hold a vote where all voters are 100% anonymous, and where a person can't vote more than once. It's either one or the other. If you want to limit one person per vote, there has to be some kind of central authority verifying identities.

Flavien, secrecy is a requirement, but anonymity isn't. I think the flaw in your proposed solution is that the issuing agency knows who voted for what. If the issuing agency just knows who has voted and can invalidate double votes, then the problem is solved. Current voting systems do keep track of who has voted and ensure that there are no double votes, but respect the secrecy of individual votes.

Ok, then this can easily be solved. I've updated my answer.

How about a combination of colored coins and the Darkcoin protocol darkcoin.io (if a combination is doable)? Colored coins entitle to vote, and the anonymization of payments to the target addresses ensure that individual votes are secret.

This still doesn't solve anonymity. You have no way of implementing a box that can simultaneously (a) can verify that the voter is listed on the voter rolls while (b) not knowing who the voter is.

Flavien, I'm looking to do this exact same thing you described above but meeting with the approved voter in person isn't possible, is there another way around this?

as for making votes secret, we can use coinjoin.
Groups of 1000 people mix their votes together. You can prove that a person made one of the votes in the pool of 1000, but you cannot prove which vote was theirs.

Except that anyone within the group of 1000 can see who everyone else in the group of 1000 voted for.  Coinjoin protects anonymity OUTSIDE the group but not inside.


text:
How do blockchains in each nodes maintain consistency?
What I understand as (Please correct me if any statement is wrong)
Every full node tries to generate a new block, in which different amount of transactions are included and what to be included will not be identical. This results in unidentical UTXO set and mempool for every full nodes.

If above statement is true, then is it okay to understand as that when the blockchain is replaced(because a new block was introduced by other node, or the blockchain branch I have became stale anymore), then the UTXO set and mempool I maintained is replaced subsequently?
If it is not the case, then how can a node resume mining after the new block is introduced? A UTXO which is known to be unspent beforehand can turn out to be spent after a new block is introduced.

Most of the full nodes are operating on normal CPUs which is now basically ineffective for mining. The mining process is handled by specific mining pools. So for full nodes that basically sees the same blockchain (there might be inconsistencies at the tip if two blocks are mined at the same time), they have the same UTXO set. This is because the UTXO set is built based on the transactions that are included in the blocks. Whenever a block is received by a full node, they delete the UTXOs which were consumed by the inputs in the transactions and add the UTXOs that were created in the output. For a mining node it is similar. If they mined the last block at the same time as the other miner, then the first miner would be building the next block using a different UTXO set than the other miner. However, after one or two blocks when the network converges (they cannot indefinitely generate blocks simultaneously) the network would have the same UTXO set.
For mempool the story is different. Bitcoin transactions are relayed on the network on a best effort basis. So it might be the case that some nodes do not see some transactions until they are finally included in the blocks. So, yes there are inconsistencies in the mempool.

how can a node resume mining after the new block is introduced?

As mentioned above, if two blocks are mined at the same time then some miners would be working on a different version versus others. The accepted principle is to build on the block that was received first. However, this is not always the case. So when a mining node who is still calculating the header hashes notice that a new block is mined, they realize that they have lost the 'race' for that particular block height and try to mine on the block that was most recently received.

I have two questions still not resolved: First, If two branched blockchains get merged, how do UTXO set are fixed? Is it just replaced by the mainstream blockchain's one(by the result of replacing the lost blockchain by the winning longest blockchain), or re-computed from not overlapped block to the most recent block(according to the updated new blocks ahead)? I guess the latter is not an practical way.

Second, almost same question on mempool. As two branched blockchains have some of blocks different, the mempool they each have must not be identical before they are merged. How does this inconsistency be resolved? When a new block is generated, each node chooses txs from its mempool(decide to use transactions occured until certain point of time, and the rest will be included when another next block is generated).

(Continuing from the previous comment) So right before the unidentical blockchains get merged, the elements of mempool could be different each other(while one tx may be included in blockchain A, the one may not be in blockchain B). In this situation, how does the node resolve the mempool difference issue?

And thank you very much for the detailed answer and explanation!

First UTXO set is built by the full node by it self. It is not broadcasted over the network. So when one node receives a block, it removes the input of the transactions from the UTXO set and adds the outputs. In case there is a branched chain and subsequent re-org, the node undoes the changes for transactions that are not included in the other block, but was present in the first block also adding the new transactions

Second As I mentioned in the answer, mempool is not always consistent across nodes even if they are seeing the same view of the blockchain. Again, as I mentioned in my answer, mining nodes generate blocks. Full nodes verify that the blocks generated confine with the consensus protocol. When a block is received by a full node, it would remove transactions from its mempool that were included in the block. There is no one way of including transactions. If miners decide they can just mine the blocks without including any transactions (just the coinbase transaction)

@cadenzah In case of a chain re-org, when the node receives the other branch of the chain which should be the main chain, it would remove transactions from the mempool that are included in the current chain and add transactions from previous block back to the mempool if it was not included in the re-orged chain

So in terms of mempool and UTXO set, it is totally un-do and adoptrestructuring procedure, not the kind of replacing. That clears my understanding. Thank you for your kind and detailed explanation!


text:
How to write OP_CHECKMULTISIG for the following 1-of-2 multisig P2SH script
OP_IF
  OP_SHA256
  <hash of secret>
  OP_EQUALVERIFY
  <pubKey of swap>
  OP_CHECKSIG
OP_ELSE
  <relative locktime>
  OP_CHECKSEQUENCEVERIFY
  OP_DROP
  <pubKey of refund>
  OP_CHECKSIG
OP_ENDIF

Actual code which i am using for the above script is
const script = this.app.env.bitcoin.script.compile([
      hashOpcode,
      Buffer.from(secretHash, "hex"),
      this.app.env.bitcoin.opcodes.OP_EQUALVERIFY,
     Buffer.from(recipientPublicKey, "hex"),
      this.app.env.bitcoin.opcodes.OP_EQUAL,
      this.app.env.bitcoin.opcodes.OP_IF,
      Buffer.from(recipientPublicKey, "hex"),
      this.app.env.bitcoin.opcodes.OP_CHECKSIG,
     this.app.env.bitcoin.opcodes.OP_ELSE,
    this.app.env.bitcoin.script.number.encode(lockTime),
      this.app.env.bitcoin.opcodes.OP_CHECKSEQUENCEVERIFY,
      this.app.env.bitcoin.opcodes.OP_DROP,
      Buffer.from(ownerPublicKey, "hex"),
      this.app.env.bitcoin.opcodes.OP_CHECKSIG,
      this.app.env.bitcoin.opcodes.OP_ENDIF
    ]);

As we see that this script has just one  public key(pubKey of swap) to sign with. But i want to sign it with two public keys.
I searched for it but found nothing, can anybody guide me?
I just want to follow this pattern
 OP_0 [Sig 1] OP_1 [PK 1][PK 2] OP_2 OP_CHECKMULTISIG

From what I understand from your question (correct me if I'm wrong), you want to have a script that in the OP_IF part pays to 1-of-2 public keys. You could try the below script:
OP_IF
  OP_SHA256
  OP_DATA (0x20)
  <hash of secret>
  OP_EQUALVERIFY
  OP_1
  OP_DATA (0x21)
  <pubKey of swap1>
  OP_DATA (0x21)
  <pubkey of swap2>
  OP_2
  OP_CHECKMULTISIG
OP_ELSE
  OP_DATA (size of relative_locktime)
  <relative locktime>
  OP_CHECKSEQUENCEVERIFY
  OP_DROP
  OP_DATA (0x21)
  <pubKey of refund>
  OP_CHECKSIG
OP_ENDIF

You could spend this script with: <0> <sig1> <secret> 1

Yes @Ugam Kamat you are actually right.
I have edit my question please see it again..Actually the complication which i am facing is, how to provide these two public keys . Is it the right way?    ```                  var add=[recipientPublicKey,exchangePublicKey]                  Whereas Inside compile Buffer.from(add, "hex"),
this.app.env.bitcoin.opcodes.OP_2,
this.app.env.bitcoin.opcodes.CHECKMULTISIG,                                                                          Is it correct?

@Aqeel You need to use a OP_DATA opcode between them to indicate the bytes of data to push to stack. For example, to push hash_of_secret to stack you need to tell the program to push the next 32 bytes which is the reason we need 0x20 preceding that as shown in my edited answer

Thanks for reply. But Opcode you listed "OP_DATA" is not a valid opcode. I have printed the list of opcodes but that opcode is not in the list and therefor creating error

@Aqeel It's not exactly an opcode but a parameter that helps you indicate to the software how much data you want to push on the stack. For example, if you see my script i used 0x20 before hash of secret. It indicates to the software to push the next 32 bytes (ox20 in hex) on the stack. Before the pubkey I used 0x21 to indicate pushing the next 33 bytes on the stack (assuming you are using compressed pub keys)

Can you provide me any helping material for 1-of-2 p2sh multisig script, which may helps me in writing script in desired format.

@Aqeel I don't think there is one specific to 1-of-2 p2sh multisig. Even if it is out there I'm oblivious to it

actually the thing which i want to see is how can i pass list of pubkeys to apply OP_CHECKMULTISIG in desired code format...
The ref repo is: https://github.com/swaponline/swap.core/blob/97d1d5b6afffee97869302443924bef79ce42763/src/swap.swaps/BtcSwap.js#L181
I am changing this piece of script to 1-of-2 p2sh multisig.
Any guidance for me after seeing this script form above reference link?

@Aqeel I haven't gone through the code in detail, but maybe you can repeat Buffer.from(recipientPublicKey, 'hex'), twice with different public key and then instead of this.app.env.bitcoin.opcodes.OP_CHECKSIG use this.app.env.bitcoin.opcodes.OP_CHECKMULTISIG


text:
How to create a P2SH address with bitcore-lib
I was looking to create custom P2SH addresses from redeemScripts ( they start with 3...) using bictore-lib.
If anyone can direct me to the right code in their module, I can't find it.

Here is the reference for creating a P2SH multisig address https://github.com/bitcoinjs/bitcoinjs-lib/blob/master/test/integration/addresses.js#L44
However your question states "custom" P2SH so I think you might be asking about something more advanced, in which case I would suggest reviewing bitcoin.script.compile, the CLTV address for example:
    let bitcoin = require("bitcoinjs-lib");
    let bip65 = require('bip65');

    function cltvCheckSigOutput (aQ, bQ, lockTime) {
        return bitcoin.script.compile([
          bitcoin.opcodes.OP_IF,
          bitcoin.script.number.encode(lockTime),
          bitcoin.opcodes.OP_CHECKLOCKTIMEVERIFY,
          bitcoin.opcodes.OP_DROP,

          bitcoin.opcodes.OP_ELSE,
          bQ.publicKey,
          bitcoin.opcodes.OP_CHECKSIGVERIFY,
          bitcoin.opcodes.OP_ENDIF,

          aQ.publicKey,
          bitcoin.opcodes.OP_CHECKSIG
        ])
      }

      function utcNow () {
        return Math.floor(Date.now() / 1000)
      }

    let NETWORK = bitcoin.networks.testnet;
    let alice = bitcoin.ECPair.fromWIF('cScfkGjbzzoeewVWmU2hYPUHeVGJRDdFt7WhmrVVGkxpmPP8BHWe', NETWORK);
    let bob = bitcoin.ECPair.fromWIF('cMkopUXKWsEzAjfa1zApksGRwjVpJRB3831qM9W4gKZsLwjHXA9x', NETWORK)

    //set lock time for 3 hours from now
    let lockTime = bip65.encode({ utc: utcNow() + (3600 * 3) });

    let redeemScript = cltvCheckSigOutput(alice, bob, lockTime);
    let scriptPubKey = bitcoin.script.scriptHash.output.encode(bitcoin.crypto.hash160(redeemScript));
    let address = bitcoin.address.fromOutputScript(scriptPubKey, NETWORK);

    console.log(address);

Thanks. I am actually looking for a JS script in bitcoinjs-lib or bitcore-lib to create NON MULTISIG, so a normal P2SH address from a redeemScript. Or even from a customString for that matter.

To be clear, in this article it mentions about a "special bitcoin address which are essentially Base58Check encodings of the 20 byte hash of the redeem script.". How are these "special addresses" created? But not the multisig ones, just an address which is the hash of a redeemScript.  https://medium.com/@ismailakkila/my-notes-on-bitcoin-transactions-part-2-c9d39d5ae326. @m1xolyd1an

Maybe this is what what I am looking for. https://github.com/bitcoinjs/bitcoinjs-lib/blob/master/test/integration/addresses.js#L25

Essentially it is just bitcoin.script.scriptHash.output.encode(bitcoin.crypto.hash160(redeemScript)); where redeemScript is equal to an OP code script array bitcoin.script.compile([bitcoin.opcodes.OP_IF,  bitcoin.script.number.encode(lockTime),...])


text:
How are bip44 coin identifiers assigned?
There is a list of bip44 identifiers for altcoins here.
It uses the term registered coin types.  My question is, is there anything in code for each coin that the bip44 number is derived from, or this is purely an arbitrary assignment from a 3rd party?
From reviewing code, bip44, and slip44, my conclusion is the latter: namely that one "registers" simply by sending an email to the author(s) of slip44, and they hopefully assign a number.
If I am incorrect and there is code in bitcoin-core or an altcoin derived from it that defines or uses the bip44 id, please point me towards that code.
thanks!

BIP44 is more like a standard which you are supposed to follow. The coin id's assigned are way to standardise the approach in case you are making a wallet and want better segregation of your coins.
For example, with the same seed, you can derive keys for different cryptocurrencies, just by changing the coin  id. Do note all those ids can be used to generate addresses for only one crypto as well.
Basically it is trying to maintain some nomenclature in this open source community. There is no code in the source code specifying this.
As a matter of fact , the address and private keys generated are backwards compatible to a node version not having the HD wallet implementation.
Address generation is something which the bitcoin core software provides, however anyone can write their own code for address generation as long as it follows the guidelines for address generation.


text:
How does the wallet identify which UTXOs belong to a public address while creating a new transaction?
I am a beginner here. A Transaction output has a field for amount and a locking script but no direct field for public address for which the output is meant. When a new transaction is being created, the wallet searches the UTXO set for outputs that is locked to one of the wallet's public addresses.

My question is, how does the wallet, while searching, know that a
  particular UTXO is locked to its public address?
Does it check each UTXO using a unlocking script to find whether it
  is locked to its public address?
Or does it maintain a separate database which maps public addresses to
  UTXOs meant for it, or anything similar for faster lookup?
How does it find this subset of UTXOs which is locked to the public
  address?

A Transaction output has a field for amount and a locking script but no direct field for public address for which the output is meant.

This is not quite correct and seems to be the source of the confusion.  A "locking script" contains the essential part of an address surrounded by the additional commands necessary to lock (or encumber) the bitcoins in that output to the address.  Let's look at an example; first we get an address:
$ bitcoin-cli getnewaddress
397yNmAFc1w6AK1gqkKNxJPb44jnvznm2b

Then let's ask the wallet what it knows about this address:
$ bitcoin-cli validateaddress 397yNmAFc1w6AK1gqkKNxJPb44jnvznm2b
{
  "isvalid": true,
  "address": "397yNmAFc1w6AK1gqkKNxJPb44jnvznm2b",
  "scriptPubKey": "a91451812710fc166bf61033afc0d2a8d7cda7a0fd8c87",
  "isscript": true,
  "iswitness": false
}

Note the scriptPubKey---that's what you're calling the "locking script."  Let's see what that script says:
$ bitcoin-cli decodescript a91451812710fc166bf61033afc0d2a8d7cda7a0fd8c87
{
  "asm": "OP_HASH160 51812710fc166bf61033afc0d2a8d7cda7a0fd8c OP_EQUAL",
  "reqSigs": 1,
  "type": "scripthash",
  "addresses": [
    "397yNmAFc1w6AK1gqkKNxJPb44jnvznm2b"
  ]
}

The asm field is the interpreted form of the locking script; OP_HASH160 and OP_EQUAL are two parts of the P2SH-style encumbrance for addresses starting with a 3.  The value in the middle 5181...fd8c is the hex form of the essential component of the address (other parts of the address are a version byte and a checksum; these aren't necessary to put in the locking script).
So, in short, the address 397yNmAFc1w6AK1gqkKNxJPb44jnvznm2b is a representation of the same thing as the scriptPubKey a91451812710fc166bf61033afc0d2a8d7cda7a0fd8c87.  The scriptPubKey is the data that actually gets placed in the output to indicate that a certain set of bitcoins are controlled by the owner of the corresponding address.

How does it find this subset of UTXOs which is locked to the public address?

A wallet that's part of a full node simply scans every transaction the full node receives to see if any of their outputs pays one of that wallet's scriptPubKeys (addresses).  If it does, the wallet saves that UTXO in its own database until it's time to spend it.
Lightweight wallets typically ask full nodes or servers to help them find the outputs.  Sometimes this entails scanning blocks basically the same way the full node wallet does with some extra steps (see BIP37), sometimes the server builds a giant database of every address every seen and performs lookup queries on it (see Electrum servers), and sometimes the full node builds a very compact index and sends that to the lightweight client so the client can request blocks that might contain its transactions (see BIP158).
Notably, wallets need to know which UTXO paid you in order to show you your balance, so wallets don't look up UTXOs when spending, they look them up when receiving.

Thank you so much. My understanding of the "locking script" was wrong. Thank you again for the help. :)

A Transaction output has a field for amount and a locking script but no direct field for public address for which the output is meant.

A form of the address is included in the script.
The kind of script you're referring to is pay-to-public-key-hash (P2PKH). Your locking condition (output script) requires the payee to supply a signature matching a public key identified by its hash value.
OP_DUP OP_HASH160 <pubkeyhash> OP_EQUALVERIFY OP_CHECKSIGNATURE

The unlocking condition in the input script supplies the public key and the signature.
<signature> <pubkey>

The unlocking condition is executed first, then the locking condition, yielding this validation script:
<signature> <pubkey> OP_DUP OP_HASH_160 <pubkeyhash> OP_EQUALVERIFY OP_CHECKSIGNATURE

My question is, how does the wallet, while searching, know that a particular UTXO is locked to its public address?

The wallet knows the public key it wants to search for. From the public key, it can generate <pubkeyhash>. Using the hash, it can query a set of outputs looking for P2PKH locks matching the public key hash.

The UTXO's script contains either a public key (in P2PK outputs, not really used anymore) or a HASH_160 of the public key or redeem script. Segwit outputs similarly contain a hash of the witness script (although this is a SHA256, not HASH_160).
Your wallet knows which redeem scripts and keys it contain, and is able to calculate the relevant hashes. It then simply checks new blocks to see if any output matches one for your wallet. It does not need to attempt to unlock any outputs.
Since the wallet scans blocks as they come in (and the chain the first time it is created), it is able to maintain a local UTXO set. This means that you need not search the entire chain every time you wish to make a tx.


text:
BitcoinJ how to understand receive Address?
I need to receive bitcoins from the customers, and then add to theirs virtual account some credits. I use BitcoinJ wallet and I want to give each customer the unique Address created by
walletAppKit.wallet().freshReceiveAddress()

But when I receive coins I can't define customer who send me them.
walletAppKit.wallet().addCoinsReceivedEventListener((w, tx, prevBalance, newBalance) contains all information but I don't see any Address info to define customer.

There can be many recipients (in the
same transaction), so you need to iterate for the correct ones. E.g. for
incoming transactions, you can do
for (TransactionOutput output : tx.getOutputs())
{
        try
        {
                if (output.isMine(wallet))
                {
                        Script script = output.getScriptPubKey();
                        Address address = script.getToAddress(Constants.NETWORK_PARAMETERS,true);
                        // do something with the address
                }
        }
        catch (final ScriptException x){}
}

Could you please explane isMine method?

This method Returns true if this output is to a key, or an address we have the keys for, in the wallet, Reference: https://bitcoinj.github.io/javadoc/0.14/org/bitcoinj/core/TransactionOutput.html#isMine-org.bitcoinj.core.TransactionBag-

I'm not very familiar with bitcoinj but it appears to me that you need to do the following.
You're passed the transaction that was received (tx).  Call getOutputs to get a list of the transaction's outputs, and for each one, call getAddressFromP2PKHScript to determine the address credited by that output.
You should account for the possibility that there might be multiple outputs paying the same address, or multiple outputs paying different addresses that are in your wallet.


text:
bitcoin upgrade needed: the current client is not synchronized
I have bitcoin client 0.6.0.6 (no longer supported!):

I did not upgrade on May15.
My current client will not synchronize past 85%.
no luck with, DB_CONFIG set_lk_max_locks 537000
I am in the process of trying to send coins, transaction remains unconfirmed

How should I best proceed?
I am concerned about upgrading to new client without the current being fully synchronized. What are the issues?
Also, I have an unconfirmed transaction looming.  This transaction are coins being sent, not receiving coins.
I am somewhat frustrated; wish I had known about the May15/synch/upgrade issues before send transaction.  Perhaps that is confounding the synch somehow?  Anyhow, what is the best way to proceed?
Best and thanks in advance

Tried upgrading to the most recent version?

No, want to check that upgrade is not a problem, given current client is not fully synched and there is an unconfirmed transaction (send coins).  If I upgrade, then the new client should synch and detect my current wallet without any known issues? this is what I am trying to clarify. (sorry if these issues are independent/without effect on an upgrade)

// , This looks sort of like a request for tech. support. Maybe try the forums?

As of June 24th 2013, There has not actually been a fork after the May 15th changes. That means that 0.6.0.6 should actually still work (though you shouldn't rely on that). This means that the fact that your client doesn't synchronize is caused by another issue, most likely a corrupted block chain database.
In case you didn't do things like copying wallets / using them from different locations (which you shouldn't), there should not really be a problem. Just upgrade, and you should get in sync, and the client will see (or try to get) the transaction confirmed.
Upgrading will take a while to 0.8.x, because of the new database layout, though. As a practical upshot, that also means the corrupted database won't matter. Oh, and make sure you have a wallet backup, and shutdown your client cleanly before upgrading.

thanks.  ill give it a try tonight and report back for anyone with similar concerns


text:
If we allow zero fee transactions within new policy rules for package relay why not also allow zero satoshi outputs?
In the proposed package relay the requirement that a transaction must meet the minimum relay fee rate is relaxed (the fee rate can be zero by the new policy rules) as long as the package in its entirety meets the minimum relay fee rate.
What is the argument for not allowing zero satoshi outputs that are spent within a package too? A transaction with a zero satoshi output shouldn't be relayed on its own (it would bloat the UTXO set) but is there an argument that it should be allowed to be relayed in new policy rules if it is spent as part of a package? Apparently a spacechain design needs zero satoshi outputs to be spent within a package to be viable.

This is paraphrasing Ruben Somsen at a Sydney Socratic in November 2021. He also discussed it on the bitcoin-dev mailing list.
Under no circumstances do we want the zero satoshi output to get mined without being spent in the same block as this contributes to UTXO set bloat as stated in the question.
There is at least one potential scenario where the zero satoshi output is left in the mempool.
Let's say Parent P1 (0 sats/vbyte) has 2 outputs, one is a zero satoshi output. Child C1 spends the zero satoshi output. P1 and C1 have a combined feerate of 5 sats/vbyte and enter the mempool first as a package. Child C2 spends the other output of P1 with a really high feerate and later enters the mempool. Network fees rise to 20 sats/vbyte and child C1 falls out of the mempool leaving the zero satoshi output unspent. Now P1 and C2 get mined and the zero satoshi output remains unspent.
You could maybe come up with a soft fork proposal (e.g. the soft fork could be that the zero satoshi output is only spendable in the same block otherwise it becomes unspendable and doesn’t enter the UTXO set) to prevent this but at that point there are probably better things you can do instead.


text:
How do I tweak the bip340_test_vectors to check that signature verification fails for other test vectors?
I now understand why some of the bip340_test_vectors fail signature verification.
I would like to tweak these test vectors to check that signature verification fails for other test vectors. How do I do this?

Disclaimer: This is an educational exercise. This Python code is for testing and should not be used in production or with mainnet Bitcoin. A private key of 3 is absolutely insecure.
There are two Python files in the BIPs repo reference.py and test_vectors.py which you can view by cloning the BIPs repo.
git clone https://github.com/bitcoin/bips.git

These two files are in /bitcoin/bips/tree/master/bip-0340.
If you run python3 test-vectors.py from within the bip-0340 directory you will get the output and error messages that are in the CSV file test-vectors.csv.
If you open test-vectors.py with your text editor e.g. vim test-vectors.py
you can edit any of the test vectors' secret key (private key), message, auxiliary randomness.
For example vector0 has the secret key 3.
seckey = bytes_from_int(3)

You could change the secret from 3 to 4.
seckey = bytes_from_int(4)

Save and close your text editor and then run python3 test-vectors.py again.
This will generate an AssertionError.
Traceback (most recent call last):
  File "test-vectors.py", line 253, in <module>
    vector0(),
  File "test-vectors.py", line 31, in vector0
    assert(not has_square_y(pubkey_point))
AssertionError

There is an assert that checks that for a particular public key (X,Y) (calculated from the secret (private) key) the Y coordinate isn't square. This is a check implemented due to a previous design decision to use squaredness as the tiebreaker for two Y coordinates. The tiebreaker is now evenness rather than squaredness. This test is included to catch when an implementer is mistakenly using squaredness rather than evenness. In this case the public key has a square Y coordinate.
Every private key (scalar) maps to a single corresponding public key (point). In BIP 340 we restrict ourselves to public keys with an even Y coordinate. This could lead to only half of the public keys being valid and hence only half of the private keys being valid too. Instead we say that every private key has a public key and if d.G has an odd Y coordinate we effectively sign with -d (the negated private key) instead.
Negated means taking the complement with the group order n
-m = n-m (mod n)
With that change you could say that every scalar is a valid private key but there are two private keys that map to the same public key. Only one private key maps to a public key with an even Y coordinate though. (Only one BIP 340 private key maps to a BIP 340 public key.)
Remember when you are finished experimenting you can use this to reset:
git reset --hard HEAD
This will discard the changes you have made and go back to the state of the code before you started experimenting.
Thanks to Pieter Wuille on IRC for some additions.


text:
Which fuzzer should I use to fuzz test Bitcoin Core?
In the Bitcoin Core fuzzing docs there are setup instructions for three different fuzzers: libFuzzer, american fuzzy lop (afl-fuzz) and Honggfuzz. Presumably I should try them in this order. How do they compare? Should I use one particular fuzzer in one scenario and a different fuzzer in another scenario? When I engage in fuzz testing should I run all of them one after each other?

This is a work in progress answer with some content currently taken from the Google fuzzing docs.
libFuzzer
libFuzzer is a tool which allows you to write fuzz tests in a similar style to unit tests. For developers setting up fuzzing for the first time, the Google fuzzing team recommends it for its simplicity.

[It] might be a good choice for testing libraries that have relatively small inputs, each input takes < 10ms to run, and the library code is not expected to crash on invalid inputs. Examples: regular expression matchers, text or binary format parsers, compression, network, crypto.

See http://llvm.org/docs/LibFuzzer.html#q-so-what-exactly-this-fuzzer-is-good-for for more information on how libFuzzer compares to other fuzzers.
AFL
AFL is the tool that did the most to drive the early success of coverage-guided fuzzing. The Google fuzzing team does not recommend AFL for new projects since it is not as well-maintained as Honggfuzz or libFuzzer. That said, because of its widespread it is common to encounter existing projects which rely on it for fuzzing.

Compared to other instrumented fuzzers, afl-fuzz is designed to be
practical: it has modest performance overhead, uses a variety of
highly effective fuzzing strategies and effort minimization tricks,
requires essentially no configuration, and seamlessly handles complex,
real-world use cases - say, common image parsing or file compression
libraries.

See https://lcamtuf.coredump.cx/afl/ for more information on how afl-fuzz compares to other fuzzers.
Honggfuzz
Honggfuzz is another fuzzing tool with a rich set of features available to it. The Google fuzzing team recommends it for security researchers looking for more customization options in their fuzzers, or developers on complex projects where working within libFuzzer's unit-test-like style is difficult. It allows users to specify how input files should be fed to a target program (e.g. on the command line, on stdin) or allows more complicated customizations such as feeding input to a network server directly.

alf is very much not applicable to bitcoin core.

Why do you say that? It is in the Bitcoin Core fuzzing docs

It’s not particularly useful for evaluating the sort of things in Bitcoin that actually matter. The consensus rules tests are partially derived from fuzzing but it isn’t ever going to find interesting bugs due to the amount of state bitcoin core has. That’s just opinion of course.

@Anonymous I agree to an extent that there are many potential issues that are hard to find through fuzzing, but that seems true for fuzzing in general and not specific to AFL? Fuzzers have found some issues in the codebase, FWIW.

@Anonymous "It isn't going to find interesting bugs": that's untrue. AFL was used to find consensus bugs in other implementation of Bitcoin: https://github.com/jonasnick/bitcoinconsensus_testcases . Of course it's not a "bitcoin-core bug" but still pretty interesting bug-prone edgecases.


text:
What is a stale blockchain tip? Is it a blockchain tip missing the latest mined block?
Stale blocks and orphaned blocks are defined here.
What is a stale blockchain tip? Is it a blockchain tip missing the latest mined block? Or is it a blockchain with a stale block (as defined in that post) at its tip?
This question was asked during this Bitcoin Core PR review club session on Bitcoin Core PR 19858

(I don't think of this as a standard term, so the context was useful for me to figure out what this question was asking -- then I realized you're asking about a specific usage I have used regarding some p2p logic!)
I've used the term "stale tip" in the context of Bitcoin Core's p2p logic to specifically refer to the case of our tip not having advanced for a long time. This can arise benignly if the network is slow to find a block, of course; but it can also arise if our peers are not giving us blocks that are being found for some reason.
The "stale tip" detection logic in Bitcoin Core (see here, here, and here) is a simple check to see if our tip hasn't updated for a given amount of time (30 minutes), and if that condition triggers, our p2p logic will try to find new outbound peers to connect to with the goal of checking to see if any peer we can find has a more-work chain than the one we're on (and if we succeed, we will evict an existing peer).
A writeup of this behavior (and some other p2p design considerations) is available on the Bitcoin Core wiki.

Such happens when importing many keys by importwallet. Bitcoin Core 24.0.1.


text:
What backward compatibility testing is done on Bitcoin Core?
The unit and functional tests of the latest release of Bitcoin Core obviously test the codebase of that release.
What backward compatibility testing is done to ensure the latest release passes previous releases' versions of the unit and functional tests?
And conversely what backward compatibility testing is done to ensure previous releases pass the current unit and functional tests?

There's no need for a new release to pass functional tests of the old release. However nodes should be compatible when they connect to older versions via the peer to peer network. Currently this testing is pretty limited, see your own answer. In addition, it should be possible to upgrade a node on your own computer, so it should be able to read old (block, mempool, etc) files. Ideally a downgrade should also work.

Thanks for this Sjors. In an imaginary world of unlimited resources and testers it is possible (if unlikely) that bug(s) could be caught by running a new release against functional tests of old release? I get it is pretty low on the list of priorities.

The problem is that each release has changes and the tests are updated to reflect those changes.

What backward compatibility testing is done to ensure the latest release passes previous releases' versions of the functional tests?

There is a feature_backwards_compatibility.py functional test which spins up functional test nodes from each of the previous releases to do testing between them at the same time.
Once you have built the latest version of Bitcoin Core you can download the previous release binaries using:
test/get_previous_releases.py -b v0.19.1 v0.18.1 v0.17.2 v0.16.3 v0.15.2
You can then run the feature_backwards_compatibility.py test individually or through the test runner.
e.g. test/functional/test_runner.py feature_backwards_compatibility.py
This test is updated with each new release to add a functional test node from the latest release to functional test nodes from previous releases.
At the time of writing (December 2020) there is an open pull request from Sjors Provoost to add v0.20.1 to the backwards compatibility test.

And conversely what backward compatibility testing is done to ensure previous releases pass the current unit and functional tests?

I don't know at this stage if this in any way a priority (or for that matter whether it should be). It is certainly more resource intensive to build entire previous releases and then test them rather than merely spin up functional test nodes from previous releases. (I will update if I learn more.)


text:
What's the purpose of using Guix within Gitian? Doesn't that reintroduce dependencies and security concerns?
What's the purpose of using Guix within Gitian? Doesn't that reintroduce dependencies and security concerns that were the purpose of transitioning from Gitian to Guix?
My understanding was that Gitian was reproducible but not bootstrappable and that Guix is an improvement to bootstrappable and reproducible?
This question was asked by robertspigler on IRC.

Gitian achieved reproducibility through the use of a virtual machine to isolate the toolchain used for building. This means that any and all unaudited and untrusted third party binaries execute in a contained environment where they are less likely to pollute and cause damage to the host system.
On the other hand, Guix is intended to be run directly on the host itself, so if there are any unaudited and untrusted third party binaries executed, they could cause damage to the host.
So some developers, notably luke-jr, want to perform Guix builds within a virtual machine to minimize this risk. This could be achieved by installing Guix inside of a virtual machine, but we already have Gitian and the infrastructure around it, so it is also possible to achieve this by running Guix within Gitian and avoid the need for such developers to figure out how to install Guix in a VM, and maintain the VM.
Furthermore, many people who have participated in the reproducible builds process may not have Guix setup yet. By doing Guix inside of Gitian, it will be possible for those people to participate in the new release process with minimal changes. Providing an easy way to transition from Gitian to Guix is the primary motivation for enabling Guix within Gitian.
Of course the recommended method is still to use Guix directly rather than through Gitian. There is a significant performance penalty for using Guix inside of Gitian. Additionally, running Guix directly allows for issues to be more easily debugged and worked through as the toolchain and extra binaries built will still be available to the builder.

fanquake answered this on IRC:

There are a number of benefits that come with using Guix as a release
build environment (or just as a build system in general).
Using Gitian, you have to pick a Linux distro to build on. This
defines your toolchains, available packages, the glibc your building
against etc.
This also means you're subject to choices of the upstream package
maintainers. They control how/when package versions are updated, what
patches are applied to them, how they are compiled etc. This has
downsides.
As an example, our security-check tests rely on a number of -no-*
options being present in mingw-w64 ld. These options were patched into
the ld in the Ubuntu Bionic binutils-mingw-w64 package, however, the
binutils maintainer chose to stop doing that in Ubuntu Focal, which
meant our security-check tests would not have been able to run (they
have since decided to re-patch them in Hirsute). See #18629 for more
details.
In Guix, because we're in full control of our toolchains, we don't
have to worry about issues like this, and can just directly apply the
patches we want to use, and this is exactly what we've done in #22381,
where we started patching the -no-* linker options into our mingw-w64
toolchain. Compiler / linker options or defaults that we may rely on,
can no-longer be randomly changing out from underneath us.
To that, some might say "well just don't change the Ubuntu base image,
and things won't change". To that, I'd say, I don't want the project
to be in a position where we are "stuck", and can't update our base
image to use new tools (i.e compilers), because by doing so we'll just
break something else, like our security tests.
It would seem smarter to just remove the potential for those kinds of
problem entirely, by constructing and using exactly the release build
environment we want.
"Well why not just patch / change things in gitian" - I wont say much
on this, except for, the gitian environment is not at all setup to do
the same kind of patching that we can fairly trivially achieve in
Guix. Trying to patch and compile, for example, the mingw-w64
toolchain in a gitian descriptor, would also just turn into a terrible
mess.
The same can be said about trying to gitian build anything using a
glibc different than what is already available on that version of
Ubuntu.
This has backward-compatibility implications, as the version of glibc
you're building against, essentially determines your runtime glibc
compatiblity. However this can be extended by using the sorts of
"workarounds" that we currently have in our glibc-back-compat code.
Now, similar to the situation where you might want to use a newer
Ubuntu release (for any number of reasons), doing so will mean that
you have to build against a newer glibc (you just get whatever version
comes with that release of Ubuntu).
This means that as you use newer versions of glibc, the number of
"workarounds" you need to maintain backwards compatibility pile up,
get continually more complicated, and even start to leak out of
Bitcoin Core code, and into our dependency system. See all the PRs
linked in this comment:
https://github.com/bitcoin/bitcoin/pull/22418#issuecomment-876379846.
Changes like that leaking into depends are bad, because now normals
builders, using depends, are now being subject to the side-effects of
patches that are only really necessary in our release build
environment. Maybe you'd argue that in that case we should only apply
the depends patches when building releases, however then you've got a
situation where release builds are even more divergent from
"normal"/developer builds, and this means either maintaining an even
more complex CI / testing routines, or they just end up less tested
(guess which one is more likely to happen).
If that all just sounds like a complicated mess, it's basically
because it is. However the solution, using Guix, is actually pretty
straight forward.
When we are in complete control of our release environment, we can
pick exactly the version of glibc we want to use (even at a per-HOST
level), something that we definitely couldn't achieve, in any sort of
straight forward fashion, if at all, using gitian.
This is something we've done recently, in #22365. We are now building
using glibc 2.27 for the RISC-V HOST, and using glibc 2.24 for all
others, while maintaining runtime compatibility with glibc 2.17.
You'll note this this is achieved without needing to use any of the
work arounds from our glibc-back-compat code (#22405), or any of the
other PRs / changes mentioned in the comment above.
These are just two very practical benefits that using Guix provides
(there are more), which ultimately all boil down to us being in much
greater control of our release build environment. Something I am very
happy about, and I think makes a lot of sense for a project like
Bitcoin Core.
The fact that there are also many people in the Guix space actively
working on bootstrapability is just another big bonus, and even if
that doesn't work exactly like some may want it too right now, it is
rapidly being improved, month on month.
Guix is also a much more likely pathway to fully bootstrapable Bitcoin
Core builds that what gitian could ever provide.

dongcarl stated:

I don't think we're going to run Guix inside Gitian for the releases,
but personally I think people are free to do whatever they think make
sense if they're committed to maintaining it

luke-jr added:

You don't lose those Guix benefits by running it inside gitian; gitian just provides the isolation needed to avoid regressing on security


text:
What is the BOLT 10 realm byte for testnet and signet DNS node discovery?
BOLT 10 describes a node discovery mechanism based on DNS.
The query semantics include a realm byte that is 0 by default (Bitcoin). What is the realm byte for testnet and signet?
This question was originally asked by Sergei Tikhomirov in the comments of this question.

There is currently no realm byte defined for testnet and signet.
At least lseed.bitcoinstats.com and lseed.darosior.ninja don't support testnet nor signet.