text:
bitcoind no such file or directory when trying to execute on Ubuntu for Raspberry Pi 4
Trying to install and run Bitcoin Core Daemon on a Raspberry Pi 4 8GB w/ Ubuntu Server and unable to execute bitcoind.
Below are the exact steps I am taking:

Use Raspberry Pi imager to flash the Ubuntu Server image to an SD card:
Ubuntu Server 20.04.2 LTS (RPi 3/4/400)
64-bit server OS with long-term support for arm64 architectures

Insert the SD card, turn on the Raspberry Pi, SSH in.

Run the following commands:
sudo apt update
sudo apt upgrade
sudo reboot
# Download bitcoin core:
curl -O https://bitcoin.org/bin/bitcoin-core-0.21.0/bitcoin-0.21.0-arm-linux-gnueabihf.tar.gz
# Unzip
tar xzf bitcoin-0.21.0-arm-linux-gnueabihf.tar.gz
#Install
sudo install -m 0755 -o root -g root -t /usr/local/bin bitcoin-0.21.0/bin/*

When running the following I get the message (bash: /usr/local/bin/bitcoind: No such file or directory):
bitcoind --help

I have attempted to cd to /usr/local/bin and run
bitcoind --help

Still no luck

Edit: you’re using the wrong architecture bitcoind binary, gnueabihf means armv7 hard float, which is 32-bit. There are 3 possibilities:

Download the correct Bitcoin binary for arm64 architecture (preferred);
Use 32-bit OS;
Install 32-bit compatibility layer on 64-bit OS.

—
Type ldd /usr/local/bin/bitcoind to check what libraries you’re missing. Most likely the only thing you need to do is to update your /etc/ld.so.cache by executing ldconfig.
ps: if you want to run a binary from the current directory, e.g. after switching to /usr/local/bin, you should type ./bitcoind, not just bitcoind.

I had a feeling it was something like this after all of the research I have been doing. Is there a problem with the Bitcoin download page? https://bitcoin.org/en/download. The one I am downloading is ARM Linux
64 bit - 32 bit (Maybe this isn't a good match for Ubuntu Server 64bit on RaspPi?). When running ldd I get "not a dynamic executable". Switching to /usr/local/bin and running ./bitcoind still gives the following message: -bash: ./bitcoind: No such file or directory. I do believe this is an architecture mismatch problem, or I do not have a required library.

The correct link is https://bitcoin.org/bin/bitcoin-core-0.21.0/bitcoin-0.21.0-aarch64-linux-gnu.tar.gz - aarch64 is the name of the architecture that you need. You’ll see the same in the output of uname -a.

Going to mark this as the correct answer if you want to update the description. Thank you so much! They should definitely put that option in the list on the main page. It is kind of hidden in the downloads directory only.

Just wanted to say thanks for that, tried installing the ARM version showing on the bitcoin.org website but it didn't work.  But going directly to their download directory I found this other version, thanks!  Running bitcoind on Ubuntu 20.10 on a Raspberry Pie with 1TB Sata M2 drive attached via USB3.


text:
Generating HD wallet seed phrase
got a new hardware wallet and thinking of using my own set of bip39 words instead of wallet generated one, combined with a passphrase.

To create a secure wallet, you need a good source of entropy. Humans are terrible sources of entropy. I would definitely not recommend doing this, use a proper source of entropy instead if you want to create your own seed phrase manually (eg, rolling well-balanced dice).

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
I get an Invalid Schnorr signature when trying to broadcast a taproot transaction
I have 0.00036134 testnet bitcoin at the segwit v1 address: tb1p3au39skpdd8suuhunr4ymp6ca6akv9keq20v5twuagpmyqntqaasl22f4w
I wrote this script to achieve 2 things: archive a hash in a transaction and send the money in tb1p3au39skpdd8suuhunr4ymp6ca6akv9keq20v5twuagpmyqntqaasl22f4w to a new address.
import random
from io import BytesIO

from util import CTxInWitness, CTxOut, create_spending_transaction_nonode
from helpers.address import program_to_witness_testnet
from helpers.key import ECKey, generate_key_pair, generate_bip340_key_pair, generate_schnorr_nonce, int_or_bytes
from helpers.segwit_addr2 import Encoding, decode, encode, segwit_scriptpubkey, bech32_decode
from helpers.script import CScript, CTransaction, OP_RETURN, SIGHASH_ALL, SIGHASH_ALL_TAPROOT, TaprootSignatureHash, tagged_hash

# Sending wallet creation
sendPrivKeyParsed1 = ECKey().set(int('SENDING PRIVATE KEY IN INTEGER'))
sendPubKeyObj = sendPrivKeyParsed1.get_pubkey()
recPubKeyBytes = sendPubKeyObj.get_bytes()

if sendPubKeyObj.get_y()%2 != 0:
    sendPrivKeyParsed1.negate()
    sendPubKeyObj.negate()

# Receiving wallet creation
recPrivKeyParsed1 = ECKey().set(int('RECEIVING ADDRESS PRIVATE KEY IN INTEGER'))
recPubKeyObj = recPrivKeyParsed1.get_pubkey()
recPubKeyBytes = recPubKeyObj.get_bytes()

if recPubKeyObj.get_y()%2 != 0:
    recPrivKeyParsed1.negate()
    recPubKeyObj.negate()

# Example tweak
taptweak = bytes.fromhex('3e75c57bb5493f932d418f8ff604f96a5d0565a056c0f0f6a9c1c07f25e2a403')

# Tweak the private key
# Method: ECKey.add()
tweaked_privkey = recPrivKeyParsed1.add(taptweak)

# Tweak the public key
# Method: use tweak_add()
taproot_pubkey = recPubKeyObj.tweak_add(taptweak)
taproot_pubkey_b = taproot_pubkey.get_bytes()

# Derive the bech32 address
# Use program_to_witness(version_int, pubkey_bytes)
address = program_to_witness_testnet(0x01, taproot_pubkey_b)

hrp = "bc"
witver, witprog = decode(hrp, address)
if witver is None:
    hrp = "tb"
    witver, witprog = decode(hrp, address)
scriptpubkey = segwit_scriptpubkey(witver, witprog)
print("the scriptPublicKey is: {}".format(scriptpubkey.hex()))

# Create a spending transaction
spending_tx = create_spending_transaction_nonode(txid='3f75f413db8ba76d7e26a9e93b9fe94990d58922cd0aaba09eac4b46b74437c2', scriptpubkey=scriptpubkey, amount_c=0.00032134)
print("Spending transaction:\n{}".format(spending_tx))

# Sign transaction with tweaked private key
# Method: TaprootSignatureHash(tx, output_list, hash_type=int, input_index=int, scriptpath=bool)
txout1 = CTxOut(nValue=36134, scriptPubKey=scriptpubkey)
sighash = TaprootSignatureHash(spending_tx, [txout1], SIGHASH_ALL, 0, False)
sig = sendPrivKeyParsed1.sign_schnorr(sighash)

# Add witness to transaction
spending_tx.wit.vtxinwit.append(CTxInWitness([sig]))
print(spending_tx.serialize().hex())

print("Success!")

I got this when trying to broadcast the transaction:
sendrawtransaction RPC error: {"code":-26,"message":"non-mandatory-script-verify-flag (Invalid Schnorr signature)"}

So why won't the network accept this transaction and how do I modify this transaction so it is accepted?

Hi @ChristianOConnor, your question is great and in-line with the scope of our site. Thank you for also answering it, after you found the solution. Please feel free to disregard the above comments, they are not in line with our community policies.—I’ll be removing them shortly.

Oh nice! Haha thank you!

Fixed it, the main problem is that:
# Sending wallet creation
sendPrivKeyParsed1 = ECKey().set(int('SENDING PRIVATE KEY IN INTEGER'))
sendPubKeyObj = sendPrivKeyParsed1.get_pubkey()
recPubKeyBytes = sendPubKeyObj.get_bytes()

Was supposed to be this:
# Sending wallet creation
sendPrivKeyParsed1 = ECKey().set(int('SENDING PRIVATE KEY IN INTEGER'))
sendPubKeyObj = sendPrivKeyParsed1.get_pubkey()
sendPubKeyBytes = sendPubKeyObj.get_bytes()

The scriptPubKey needs to be calculated for both the sending and receiving addresses. In my original code, I did this to calculate the receiving address's scriptPubKey:
hrp = "bc"
witver, witprog = decode(hrp, address)
if witver is None:
    hrp = "tb"
    witver, witprog = decode(hrp, address)
scriptpubkey = segwit_scriptpubkey(witver, witprog)
print("the scriptPublicKey is: {}".format(scriptpubkey.hex()))

So instead of scriptpubkey, you have scriptpubkeyRec and scriptpubkeySend. Maybe put the above code into a function that returns scriptpubkey and run it once for the sending address and once for the receiving address. If you write such a function called get_scriptPubKey(), the scriptPubKey generation would look like this:
# Derive the bech32 address
# Use program_to_witness(version_int, pubkey_bytes)
addressSend = program_to_witness_testnet(0x01, sendPubKeyBytes)
addressRec = program_to_witness_testnet(0x01, taproot_pubkey_b)

scriptpubkeySend = get_scriptPubKey1(addressSend)
scriptpubkeyRec = get_scriptPubKey1(addressRec)

Use scriptpubkeyRec here:
# Create a spending transaction
spending_tx = create_spending_transaction_nonode(txid='3f75f413db8ba76d7e26a9e93b9fe94990d58922cd0aaba09eac4b46b74437c2', scriptpubkey=scriptpubkeyRec, amount_c=0.00032134)

Use scriptpubkeySend here:
txout1 = CTxOut(nValue=36134, scriptPubKey=scriptpubkeySend)

The last problem is that you need to sign with the sending key like this:
sig = sendPrivKeyParsed1.sign_schnorr(sighash)

After I made these changes the transaction worked https://blockstream.info/testnet/tx/55d8281e28240aa84f23026f335128d0c4e0c8868617cb65bc92a1ddc9c1f322

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Development questions regarding debugging and control flow
I'm looking for a better understanding of how the flow of the code works. I'm running bitcoind -regtest in one terminal window, and in the other I'm using bitcoin-cli regtest generate to create blocks. What I'm having trouble with is understanding exactly what is happening in the code when I'm creating blocks.
My assumption was that blockencodings.cpp was used for that, so I put a print statement in it, but nothing ever prints to the console. Is there some issue with printing to the console when regtesting? Or am I not understanding the control flow here. Just looking for some help understanding either debugging, or the chain of functions when generating a block.

You want to look at miner.cpp, and perhaps at rpc/mining.cpp for the generate function that's exposed to RPC.
blockencodings.cpp is pretty much only used for the compact blocks P2P format specified by BIP152, and has nothing to do with normal block generation.

awesome, thanks for the tip. if I'm looking to try out a modification to the block header, is the relevant code in block.cpp and block.h?

primitives/block.h

Hmm for some reason the print statements in block.h aren't working. I put one into the CBlockHeader definition thinking it would trigger when I generated a block, but I can't see it in my bitcoind console.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
How can I optimize my python script that creates a btc native segwit tx with an OP_RETURN message?
I have the following code that successfully created a bitcoin transaction with an OP_RETURN message:
from bitcoinutils.utils import to_satoshis
from bitcoinutils.setup import setup
from bitcoinutils.transactions import Transaction, TxInput, TxOutput
from bitcoinutils.script import Script
from bitcoinutils.keys import PrivateKey as utilPrivKey
from bitcoinutils.constants import SIGHASH_ALL, SIGHASH_ANYONECANPAY

setup('testnet')

# private key for tb1qqh5eczga9ltll4crtyvr4epxcssyufuvlhwdxv
priv = utilPrivKey.from_wif('SENDING ADDRESS PRIVATE KEY')

pub = priv.get_public_key().to_hex()
addrs_for_script = priv.get_public_key().get_address()

# private key for tb1qpeervdxrt2np74xhm4q4jjp76ey5f9fh9tf58f
recPrivKey = utilPrivKey.from_wif('RECEIVING ADDRESS PRIVATE KEY')

# This UTXO has 0.00032557 btc
txin1 = TxInput("947145005bac29a9966403e986b8a36cb225cba504bffbb3f6c1da7dba7197be", 1)

# This UTXO has 0.00006859 btc
txin2 = TxInput("aea283b193f2179cb6a5c2bcea4b9ad7158960696ebfb5c15d22a0411f25dafe", 0)

# the script code required for signing for p2wpkh is the same as p2pkh
script_code = Script(['OP_DUP', 'OP_HASH160', addrs_for_script.to_hash160(),
                        'OP_EQUALVERIFY', 'OP_CHECKSIG'])

addr = recPrivKey.get_public_key().get_segwit_address()
addr_non_seg = recPrivKey.get_public_key().get_address()

# This is random text designed to be hashed 8hwi8a8x3lja1ukbu ---> 69eb032167c25784108b9783c9d59ce23608d00238eb9e3c03c73cdd79f74528
msg1 = "69eb032167c25784108b9783c9d59ce23608d00238eb9e3c03c73cdd79f74528"

op_return_script = ["OP_RETURN"]
op_return_script.append(msg1)
op_return_script = Script(op_return_script)
print(op_return_script)
print(addr.to_script_pub_key())

txoutOpR = TxOutput(0, op_return_script)

# remaining 0.00005 is tx fees
txout = TxOutput(to_satoshis(0.00034416), addr.to_script_pub_key())

# create transaction from inputs/outputs -- default locktime is used
tx = Transaction([txin1, txin2], [txoutOpR, txout], has_segwit=True)

txsign1 = priv.sign_segwit_input(tx, 0, script_code, to_satoshis(0.00032557), SIGHASH_ALL | SIGHASH_ANYONECANPAY)
tx.witnesses = [ Script([txsign1, pub]) ]

txsign2 = priv.sign_segwit_input(tx, 1, script_code, to_satoshis(0.00006859), SIGHASH_ALL)
tx.witnesses.append( Script([txsign2, pub]) )

signed_tx = tx.serialize()
print("raw tx below this line")
print(signed_tx)
print("raw tx above this line")

I used the above code to make this transaction:
02000000000102be9771ba7ddac1f6b3fbbf04a5cb25b26ca3b886e9036496a929ac5b004571940100000000fffffffffeda251f41a0225dc1b5bf6e69608915d79a4beabcc2a5b69c17f293b183a2ae0000000000ffffffff020000000000000000226a2069eb032167c25784108b9783c9d59ce23608d00238eb9e3c03c73cdd79f7452870860000000000001600140e723634c35aa61f54d7dd4159483ed649449537024730440220188a85f11816410f8ea51a933dbf9e38b318203602a42019837534789cd3915a02200351fa3e40ac446959d295f4efcbd2f76f0546b4be8c2066e5ffac9a3688e519812102cc30cb24a244910be0a343ee2cd7cee2e79558dbc8873404f6122d40a70cf71f0247304402205cf55018d39e3377ab843ec53be895934f59520e8268bbbdc18328a6d1d55c590220322de21da63e32fc2160db526979f7799d44c190c9810e0a20cd89e70cc61bb6012102cc30cb24a244910be0a343ee2cd7cee2e79558dbc8873404f6122d40a70cf71f00000000
This is the above transaction after it was broadcasted https://blockstream.info/testnet/tx/a86f0187be2eca4590a456e2423a2f5286b269c3b777846092ceb2fc44270aad.
How would I optimize this transaction creation script? I am mostly looking for whether or not some of the parameters are unecessary, like 'OP_DUP', 'OP_HASH160', 'OP_EQUALVERIFY', 'OP_CHECKSIG', SIGHASH_ALL, or SIGHASH_ANYONECANPAY?

This is the most optimized version of my script that I'm aware of:
from bitcoinutils.utils import to_satoshis
from bitcoinutils.setup import setup
from bitcoinutils.transactions import Transaction, TxInput, TxOutput
from bitcoinutils.script import Script
from bitcoinutils.keys import PrivateKey as utilPrivKey
from bitcoinutils.constants import SIGHASH_ALL

setup('testnet')

# private key for tb1qpeervdxrt2np74xhm4q4jjp76ey5f9fh9tf58f
sender_priv_key = utilPrivKey.from_wif('PRIVATE KEY FOR THE SENDING ADDRESS')
sender_pub_key = sender_priv_key.get_public_key().to_hex()
addrs_for_script = sender_priv_key.get_public_key().get_address()

# private key for tb1qj57tu40e9da959w4lj9s3njwteq2z89uy7kenh
receiver_priv_key = utilPrivKey.from_wif('PRIVATE KEY FOR THE RECEIVING ADDRESS')
receiver_addr = receiver_priv_key.get_public_key().get_segwit_address()

txin1 = TxInput("a86f0187be2eca4590a456e2423a2f5286b269c3b777846092ceb2fc44270aad", 1)
txin2 = TxInput("9019818039f181ed00046d682411719050a4b2dc1d1eb6aaa2d781df863522fc", 0)

script_code = Script(['OP_DUP', 'OP_HASH160', addrs_for_script.to_hash160(), 'OP_EQUALVERIFY', 'OP_CHECKSIG'])
msg1 = "69eb032167c25784108b9783c9d59ce23608d00238eb9e3c03c73cdd79f74528"
op_return_script = Script(["OP_RETURN", msg1])

txoutOpR = TxOutput(0, op_return_script)

# 0.00005 left for transaction fee
txout = TxOutput(to_satoshis(0.00036275), receiver_addr.to_script_pub_key())

tx = Transaction([txin1, txin2], [txoutOpR, txout], has_segwit=True)
txsign1 = sender_priv_key.sign_segwit_input(tx, 0, script_code, to_satoshis(0.00034416), SIGHASH_ALL)
tx.witnesses = [Script([txsign1, sender_pub_key])]
txsign2 = sender_priv_key.sign_segwit_input(tx, 1, script_code, to_satoshis(0.00006859), SIGHASH_ALL)
tx.witnesses.append(Script([txsign2, sender_pub_key]))

signed_tx = tx.serialize()
print("raw tx below this line")
print(signed_tx)
print("raw tx above this line")

This code worked, it produced this tx which was accepted by the blockchain:
02000000000102ad0a2744fcb2ce92608477b7c369b286522f3a42e256a49045ca2ebe87016fa80100000000fffffffffc223586df81d7a2aab61e1ddcb2a45090711124686d0400ed81f139808119900000000000ffffffff020000000000000000226a2069eb032167c25784108b9783c9d59ce23608d00238eb9e3c03c73cdd79f74528b38d000000000000160014953cbe55f92b7a5a15d5fc8b08ce4e5e40a11cbc0247304402203fb058c0afbf8d44158f89e95df13d434398d3a447978249d3860199e167fd62022077aed81b7e33726e2ace77318670d62eb74bf131cacbbf379a6b5925bc2935a0012102ad3b5a101ab6fed444f7b76f49b6f668f126a165aeb70162c16560f05cd53c3402473044022051b927dd9111894b6e440bd032d78584010f804f3d63f2a956145dc801e0c60e0220552bb84b5ed35e2ac0860805ca3e4819456ab707c973561e4e76bd04fcf48bd4012102ad3b5a101ab6fed444f7b76f49b6f668f126a165aeb70162c16560f05cd53c3400000000```


text:
Block Validation Process, in Terms of adding Block to the Chain
quick question - How does a Node react to a new block propagated to the network?
In Terms to  does a Miner stop mining when "he" gets information of a new block to validate it? or does it not stop and validate it in another Thread?
If the Question is unclear - let me know I just began my deep dive into the technical side of Crypto and Blockchain :)
best regards :)

It is easy and instant to confirm that a hashes proposed by a miner is the correct one, so once it is received by a node, miner, he will immediately stop trying to find a correct hashes for the previous block and began searching the new one without transition.

Hey,
thanks for the answer - just to recap it for my understanding -
As soon i (presumably I'm mining on a Block atm) get a new block propagated i just stop working on mine, quickly validate the block i got propagated and start with a new block?

yes, notice that in any case you will need the hashes (number) to start working (hashing) on the next block


text:
Identifying address types using v1+ segwit outputs
Please correct me if i'm wrong:
So, with segwit v0 we know that 20 bytes is equal to P2WPKH, and 32 bytes is equal to P2WSH, and thus we can identify the address type by decoding the 31 byte or 43 byte bech-32 encoded string.
so we get decoded 0 < hash of pubkey > or 0 < sha hash of redeem script >, and identify the sender address type
Now I read that there has been some testing done with newer bech versions (bech32m) for P2WSH (or P2TR?) addresses, and also comments that some testing resulted in funds being lost, which confused me. (https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-November/018268.html)
A) Considering funds got lost on mainnet using bech32m encoding (or was it p2tr v1+ using bech32?), am i correct in assuming that P2WPKH/S scripts can have a witness version of 1 (bech32m), else: how could this have been tested on mainnet (or can you generate valid p2tr addresses already, but just not spend from them? is that it?) -> I guess that would make sense considering a v1 taproot address would look identical to a v1 P2WSH address?
->> if so, how can we differentiate between script hashes of P2WSH 1 <32 byte hash of redeem script>, and taproot 1 <public key 32 bytes>
https://github.com/bitcoin/bips/blob/master/bip-0350.mediawiki#addresses-for-segregated-witness-outputs
as address indication is done purely based on "script size", and why is this not done in above link? Would this be considered "unstandard" - if so, why was there such an emphasis on testing this? I don't understand how this was a surprise?
B) If this is not the case and it was indeed P2TR adddresses which did not decode correctly using bech32, why then was this the case exactly, considering the only thing that bech32m changes is the checksum? As in; the decoding process did not change as a result of this BIP.
A quote from the BIP:

This does not affect existing uses of witness version 0 BIP173 addresses due to their restriction to two specific lengths, but may affect future uses and/or other applications using the Bech32 encoding.

addresses using P2TR also have a restricted length, right?

You differentiate a Taproot scriptPubKey and a P2WSH scriptPubKey by their witness version. 1 for Taproot, 0 for P2WSH.
A "P2WSH with version 1" doesn't exist, it would just be treated as a Taproot scriptPubKey by upgraded nodes (after activation) and as an anyone-can-spend by legacy ones.
P2WSH: <0> <32-bytes push>
Taproot: <1> <32-bytes push>

As for the mentioned funds loss i haven't heard about it but it's possible (if you send coins to <0> <random 32 bytes> you are going to burn your funds). In any case it would not be related to the address encoding, which is just an user interface.

Okay, that was what I was wondering - native segwit cannot use version 1 and above and subsequently do not use bech32m encodings (which is just a different checksum), else they are seen as invalid, (or a taproot address which is actually unspendable).
I still don't fully understand what happend that caused P2TR "scripts" (pubkey, whatever you want to call it) encoded using standard bech32 (so checksum of 1) to burn funds because they were decoded wrong by wallets, if anyone could clarify the technicalities of that, and why that would be changed by bech32m, aside from breaking forwar-compat

It's not whatever i want to call it, i try to use specific terms to not cause (additional) confusion (too many terms are already overloaded in Bitcoin-land). I don't know what you are talking about, could you provide a link to a discussion about this loss?

All my sources are from the BIP readme detailing bech32m, so that would be github and this email

Taproot outputs (P2TR) are native segwit v1 with 32-byte programs. That's how these outputs are defined. They're not "P2WSH v1", because P2WSH is defined as "Native or P2SH segwit v0 outputs with 32-byte programs".

The burning of funds was due to some services incorrectly implementing BIP173 in a way that resulted in bech32 v1+ addresses being turned into native v0 witness outputs (regardless of the version in the address). Taking a P2TR output (which stores a tweaked xonly pubkey) and changing the version to zero causes it to be interpreted as a P2WSH (with as script "hash" that tweaked key), which is unspendable.

Sorry, in that case I should indeed specify that by "native-segwit" in the first comment I was referring to the two "spending scripts" (again, not sure if correct terminology) currently used with witness version 0.
Taking ... is unspendable
Okay fascinating, so they just sent to <0> <32-bytes push> instead of <1> <32-bytes push> because the version was hardcoded or similar on their end, and because that is actually just a valid P2WSH v0 encoding there was nothing that invalidated the payment? Think I got it.


text:
My lnd node hasn't routed any payments (1 week + since initial setup)
So I finally setup a lightning node (lnd) and according to "feereport" I haven't routed any payments (no fees collected).  I have used my node to send payments (yalls, satoshis.place, blockstream...) and also received payments (from my own Eclair wallet) and that works fine (multi hop routes).  I also tried lowering the fees my node charges, still no luck.  I also (gave away) pushed some remote balance to some channels to make sure I had some out going routes.  Any ideas?  What is the expected average number of routed payments over some time interval (I know this depends but I no idea what other people are seeing)?
$ lncli listchannels | grep balance
            "local_balance": "99082",
            "remote_balance": "100000",
            "unsettled_balance": "0",
            "local_balance": "299082",
            "remote_balance": "0",
            "unsettled_balance": "0",
            "local_balance": "295312",
            "remote_balance": "34",
            "unsettled_balance": "0",
            "local_balance": "149082",
            "remote_balance": "150000",
            "unsettled_balance": "0",
            "local_balance": "145347",
            "remote_balance": "150000",
            "unsettled_balance": "0",
            "local_balance": "83940",
            "remote_balance": "261405",
            "unsettled_balance": "0",
            "local_balance": "99180",
            "remote_balance": "0",
            "unsettled_balance": "0",

$ lncli feereport | grep sum
    "day_fee_sum": "0",
    "week_fee_sum": "0",
    "month_fee_sum": "0"

$ lncli getnetworkinfo
{
    "graph_diameter": 0,
    "avg_out_degree": 5.5954773869346734,
    "max_out_degree": 509,
    "num_nodes": 1592,
    "num_channels": 8908,
    "total_network_capacity": "9685778073",
    "avg_channel_size": 1087312.311742254,
    "min_channel_size": "1050",
    "max_channel_size": "16777216"
}

I am having the same issue, my node has 8 connections since a month now but no transactions. Is it normal? Is it because there are more nodes than transactions in the network? How is your node doing?

I haven't received any fee payments. (...) Any ideas? Or is this expected?

Running a LN node does not guarantee you will collect fees. Fees are collected iff you route a payment for another user, so at this time it appears that no users have routed a payment through your node.
Factors that could affect this: which nodes you are connected to, the liquidity of your channels, the online availability of your node, etc.
Routing fees offer an interesting incentive, but remember the network is still young and not widely utilized. Right now, payments for routing transactions may be rare, but in the future... ?

I also pushed some remote balance to some channels

Worth noting, this is equivalent to making a payment to the node you are connecting to. When the channel is closed, that remote node will be the recipient of the remote balance. So I hope you have made those payments for a reason, otherwise you have given away some of your BTC.

I understand there are no guarantees, I'm doing this for testing purposes but I'm surprised that I haven't routed any payments in a week.  This may be expected but I see a number of other users talking about routing "lots" of payments.  I have my channels balanced and with sufficient funds and I'm 100% online.  So that is part of why I'm surprised.  So my question is "what is the expected frequency of payment routing I should expect" based on your experience?

I don't believe there is any one 'expected frequency' that can be quoted for the network at this time. I have an lnd node running, but it has minimal funds on it, and maybe 1-2 channels open as I need them, so my node has (perhaps expectedly) not routed any payments at this time. It may be that the channels you are connected to are simply not positioned near high transaction frequency routes on the network. The users talking about routing lots of payments may in fact be running shops/services on the lightning network, and thus may be some of the most connected nodes out there.

Now that my lnd node has been running a little longer I have routed some payments.  Getting your node selected for a route by another node will be highly dependent on your channels, who with, how many and balance but it looks like 1 or 2 routes per week would be typical.

What about 3 years later? whats the status?

My year 2021 experiences:
My node is 60 days old, my lnnodeinsight rankings on amboss are about 300 (all three),  and I have routed 72 payments within the last 28 days. That is still very, very far away from earning some money here.
My experience with routing:

Know the liquidity sinks in the network
Adjust your fees according to your local liq. (outbound liq.)
Connect to nodes with lower fees
Try to re-balance your liq. regularly

Unfortunately I.4 costs even more of your expected profitability. There are several very good articles out there about lightning network economics. To summarize that: Do not expect an income from running a lightning node currently - but this may change in the future when lightning get's adopted as payment network.
And also, there are some other benefits:

Host your own hot wallet
Hodl bitcoins
Be part of the new payment age
Learn running IT at home
Earn at least any interest (in Europe we have negative interest rates
for bank-money since several years now).

Stay tuned.


text:
nSequence win OP_CSV in child tx must be equal to nSeq parent tx?
When I create OP_CSV  transaction I need to set nSequence to the same number (in blocks)
e.g.
3 OP_CHECKSEQUENCEVERIFY
nSequence = 3

But when I trying to spend this tx (after 3 blocks confirmation) I have to generate transaction with the same nSequence as parent.
Otherwise I'm getting error (when I'm trying to spend with different nSequence)
non-mandatory-script-verify-flag (Locktime requirement not satisfied) (code 64)

But I wont set the nSequence to child transaction, because, in this case, I can't spend immediately child transaction.
Do I need to set some flag or something?

I believe your misconception is that the nSequence value must be set on the transaction that creates the CSV output.
This is not required. All that's needed is setting it on the spending transaction's input which spends it. This will indirectly require that output to have a certain age when it is being spent.


text:
Cant spend input with non-standart redeemscript (OP_CSV)
I'm playing with OP_CHECKSEQUENCEVERIFY.
Here is my transaction that I'm trying to spend:

https://tchain.btc.com/e79d469698915aa6724288ae3a4141a31706d081f39fb7fd0b956a73a9044b22

which has the folowing redeem script:
2 OP_CHECKSEQUENCEVERIFY f45d94733d430261962932e0c847075195916a04 OP_CHECKSIG

But got an error when I  broadcast it (after two block confirmations left):
mandatory-script-verify-flag-failed (Non-canonical DER signature) (code 16)

I have no issues to spend transaction if I:

Just specify 2 CHECKLOCKTIMEVERIFY (can spend anyone)
With standart transaction HASH160 f45d94733d430261962932e0c847075195916a04 OP_EQUAL

but none with combination of them.
Here is a signed transaction:
0200000001224b04a9736a950bfdb79ff381d00617a341413aae884272a65a919896469de7000000006b4830450221008801d7f0e402724b090133ec01a817e993675a44a871f1a3afc0dba730e24ea4022044097be610bece524e90a743fa9d060cd194079105e6f6d150265a1164e722c3012103e8546cfe53ca9014f6a352bc2e5c1f7cec1f5b344c73d4db3ed3b8f075d5c9ef020000000120a107000000000017a914f45d94733d430261962932e0c847075195916a048700000000

Can someone explain me what I'm doing wrong?
P.S. With the folowing redeem script (OP_DROP added)

https://tchain.btc.com/022d42355ac90e63f2667208112cf22471397dd2812645b5571da2d212596932

2 OP_CHECKSEQUENCEVERIFY OP_DROP f45d94733d430261962932e0c847075195916a04 OP_CHECKSIG

I got another error
(Script evaluated without error but finished with a false/empty top stack element) (code 16)

[UPDATED]:

https://tchain.btc.com/46894d9dd5e20147826523b1a3ac8e8c5c64603338430d04bd6f5846af2bb9b4

3 OP_CHECKSEQUENCEVERIFY OP_DROP OP_HASH160 f45d94733d430261962932e0c847075195916a04 OP_EQUAL

Then trying to broadcast this tx:
0200000001b4b92baf46586fbd040d43383360645c8c8eaca3b12365824701e2d59d4d8946000000006a47304402205649f715d2553cc4a66dd36a61b3c1b187d19f585bd40427d795c600373a45ab02203325ef44a3aae191ed3f2a90efe382d7992712c03f249f8dd61d8a30997806dc012103e8546cfe53ca9014f6a352bc2e5c1f7cec1f5b344c73d4db3ed3b8f075d5c9ef0300000001e00f97000000000017a914f45d94733d430261962932e0c847075195916a048700000000

I got
error code: -26
error message:
mandatory-script-verify-flag-failed (Script evaluated without error but finished with a false/empty top stack element) (code 16)

Here is a privatekey
cUB1iazKDHngknjpTHUfKtH9zNRvHn1aBMPJbK2s913zUegjkeNn

Can someone spend it?

You must provide a scriptSig that, when executed with scriptPubKey results in a true (or any nonzero item) on the stack (and passes all verification ops).
The scriptPubKey you need to satisfy (with a scriptSig) is:
2 OP_CHECKSEQUENCEVERIFY f45d94733d430261962932e0c847075195916a04 OP_CHECKSIG

(note this is not the redeemScript)
There main problem is:
OP_CHECKSIG is looking for a public key, but you have provided a script hash: f45d94733d430261962932e0c847075195916a04. This will never be satisfied so this transaction output is not spendable.
If you want to use the same redeemScript as before, you probably want the following scriptPubKey:
2 OP_CHECKSEQUENCEVERIFY OP_DROP HASH160 f45d94733d430261962932e0c847075195916a04 OP_EQUAL
Note the OP_DROP because the OP_CHECKSEQUENCEVERIFY leaves the 2 on the stack.
In order to spend this, you must provide any signatures and the redeemScript whose hash equals f45d94733d430261962932e0c847075195916a04.
Update:
The redeemScript is still wrong. Instead of 03e8546cfe53ca9014f6a352bc2e5c1f7cec1f5b344c73d4db3ed3b8f075d5c9ef, try 0014e17bbad678e82f8188099a351170c3e0efd4011b. This is the P2WPKH script you used previously in https://tchain.btc.com/46894d9dd5e20147826523b1a3ac8e8c5c64603338430d04bd6f5846af2bb9b4. Also note this is a segwit script so it will look for the signature and pubkey in the witness data: https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki#p2wpkh

Shouldn't that be OP_DUP OP_HASH160 <hash> OP_EQUAL OP_CHECKSIG? Your example only verifies that the pubkey matches, it does not verify the signature.

I was attempting to provide a P2SH example, not a P2PKH.

Ah, of course, my mistake then!

Please, see my updated post

@Andrew It looks like you're providing a public key 03e8546cfe53ca9014f6a352bc2e5c1f7cec1f5b344c73d4db3ed3b8f075d5c9ef that doesn't hash to the right value for the OP_EQUAL check. You need the redeemScript that hashes to f45d94733d430261962932e0c847075195916a04

What about this transaction? The public key and redeem script is the same, but it's valid tx https://tchain.btc.com/194fe9b2e266b47f51410ceb58f0765551f75bd018e7bfc3439efee14a344777

That is a witness transaction, and the redeemScriptis provided in the scriptSig: 0014e17bbad678e82f8188099a351170c3e0efd4011b

Can you spend this tx https://tchain.btc.com/46894d9dd5e20147826523b1a3ac8e8c5c64603338430d04bd6f5846af2bb9b4  ? Here is a private key: cUB1iazKDHngknjpTHUfKtH9zNRvHn1aBMPJbK2s913zUegjkeNn

@Andrew see my updated answer

Let us continue this discussion in chat.

Oh my God! I'm so sorry, but I'm not a programmer. I have no idea how to insert redeemscript into transaction and/or how to calculate it. Is there a easy way to just spend this inputs using bitcoin-cli with the folowing private key ?


text:
Mempool filled with 14KB transactions, tons of inputs, 0.15 mBTC fee
With the blockchain experiencing extremely high transaction volumes (most blocks at the 1MB size limit), I am interested in studying the "backlog" of transactions in the memory pool on my own full node. In order to make such an analysis meaningful, I want only "honest" transactions in my mempool (loosely defined as: transactions that should be mined someday).
I've tried adjusting minrelaytxfee to 0.0001, but I find that TONS of "unqualified" transactions build up in my mempool. Here is a representative transaction:
effec9299f18314a7a13e525a7bdbaa9316be9d9cac9d082078ae325fff30500
That transaction is 14,799 bytes (100+ infinitesimal inputs) with a fee of 0.15 mBTC. At the risk of stating the obvious, that transaction should NOT pass a minrelaytxfee of 0.1 mBTC/KB. My mempool collects ~1000 transaction per day that are VERY similar to the one cited above.
Also at the risk of stating the obvious, these transactions are NEVER mined.
QUESTION #1: how do these transactions "fool" the minrelaytxfee test?
QUESTION #2: is there a better mechanism for filtering out such transactions?
QUESTION #4: might these transactions originate on "my" SPV clients?
QUESTION #3: are these transactions SPAM or something else?
Thanks for the insight.
UPDATE ----------
For debugging purposes only, I increased minrelaytxfee to 0.0002 and re-started my Bitcoin Core full node.
Observation: on the vast majority of transactions, minrelaytxfee is applied correctly. In other words, transactions will a fee lower than 0.2 mBTC/KB are not added to my mempool.
I continue to get ~250 transactions/hour that somehow "fool" the minrelaytxfee filter. Here are three representative transactions:
c299d3bd4e79590bf5d5b9cfb1f3db4378ffc5e9e64752f6deb8134d43f54501
ffc1ad32c1d8ca496fa52e5451c6abe56003029819029de9a0f941611f942402
65871d2e0bc438d5019e606daba59d7af1a2ce3eb079e5f0e6c9e21ef8373402
All of these "foolish" transactions follow a specific template:

Size is between 14790 and 14810 bytes
Roughly 100 transaction inputs, all of exactly 0.1 mBTC
Transaction fee is always exactly 0.15 mBTC

QUESTION: how are these transactions fooling the minrelaytxfee filter?
Tracing a large sample of the 0.1 mBTC transaction inputs, they are always produced by a transaction that follows its own specific template:

A single transaction input
Roughly 100 transaction outputs, all but one of exactly 0.1 mBTC

QUESTION: Am I correct in suggesting that this pattern appears suspicious and deliberate?

What software and version are you using for your full node, and how are you setting the minrelayfee? It has a fee of slightly above 0.01mBTC/KB, which is the default min relay fee of many recent Bitcoin Core versions. Perhaps you made a typo in the config setting, and it remained at the default?

I am running Bitcoin Core 0.12.0.

Ok, now the other question: how are you setting the min relay fee exactly (command-line option, which? config file entry, which?)

I added the following line to bitcoin.conf:
minrelaytxfee=0.0001

Did you restart bitcoind after adding that line?

Additional observation: the minrelaytxfee=0.0001 does properly filter most transactions ... it is "fooled" by the referenced class of transactions with 100+ very small inputs.

Yes. And to be safe, I re-started Ubuntu as well.

FYI. These transactions are from "coinwallet.eu stress test". The outputs have enough age. Combining transactions were created and broadcasted to a network long ago. Now they are migrating [s]from Middle East to Europe[/s] to your node. Enjoy!

Amaclin: I read up on the "coinwallet.eu stress test," and I cannot connect the dots. I see that the UTXOs were created last year, check. The new transactions (TXIDs listed above in the UPDATE) are actively being created at the rate of ~250 per hour for the past month. So yes, the UTXOs are "old" and boost the transaction in the legacy prioritization formula ... but what does that have to do with the minrelaytxfee filter?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
As mining without a pool?
Bitcoin 0.14
Friends, tell me how you can mining without a pool at home on your computer?
I have bfgminer and cgminer
I'm trying to connect so
D:\bfgminer\bfgminer.exe -o stratum+tcp://localhost:18444 -u login-p password
D:\cgminer-4.10.0-windows\cgminer.exe -o stratum+tcp://127.0.0.1:18352 -u login -p password
but it does not work:(
bitcoin.conf

regtest=1
server=1
daemon=1
rpcuser=login

# Alice connects to Cory
connect=localhost:18444

rpcpassword=password
rpcallowip=127.0.0.1

Wait, are you sure you want to mine regtest? You can just do bitcoin-cli -regtest generate 1 to generate a block. Notice that regtest is a local network that is used for testing, and the coins aren't worth anything.

Wait, I don't get it. If you are not in a pool then how can you be having a user name and a password?

So Regtest as it says here on official bitcoin Developer notes. https://bitcoin.org/en/developer-examples#regtest-mode

Bitcoin Core’s regression test mode (regtest mode) lets you instantly
  create a brand-new private block chain with the same basic rules as
  testnet—but one major difference: you choose when to create new
  blocks, so you have complete control over the environment.

Start bitcoind in regtest mode to create a private block chain.
## Bitcoin Core 0.10.1 and earlier
bitcoin-cli -regtest setgenerate true 101

## Bitcoin Core master (as of commit 48265f3)
bitcoin-cli -regtest generate 101

So on Regtest mode you do NOT need to mine with an ASIC.  It is like testnet but you decide when to and how many blocks to generate for your testing.
Hope this helps.
EDIT: Old Post below, as missed the regtest part of the config file
Yes u can solo mine, but you will need to run the bitcoin client so it knows what the latest work is.  I should mention that u should not be solo mining btc without at least 250TH or so hashing power from last calculation I saw with current difficulty, so u would need many ASICs for this.
Otherwise your conf file and connection string look ok.

I do not quite understand.
Need regtest bitcoin

Download bitcoin core from bitcoin.org sync the blockchain, then start mining

Downvoted. This answer doesn't apply to regtest.

@PieterWuille updated answer as missed the regtest line in the config

@menfix updated answer as missed the regtest line in the config


text:
Bitcoin wallet key pair change and transaction validation
For anonymity reasons key pair of a wallet can change. Then how does miner validate if a sender has enough bitcoins?
After key pair change blockchain history contains only old public key as a wallet reference.
Is this has something to do with origin of bitcoin sent in transaction? If so, what if sent sum is 30 BTC, but they are all received from 3 (3x10 BTC) separate wallets?

Some things you need to remember about bitcoin:

Bitcoin is cash, not a bank account. These are very different models.

You pay by handing over digital coins, not by making a bank-transfer between accounts.
No one but you cares what total amount of Bitcoin money you have, they don't need to know.

This fundamental misunderstanding comes up a lot in different forms. The questions below have answers which touch on this and may help form a clearer understanding:

How does Bitcoin verify that a user owns the bitcoins he/she spent?
What is the meaning of difficulty in Bitcoin mining?
Bitcoin Fraud/Recovery Protocol Proposals
Why do so many Bitcoin wallets and accounts exist?
What prevent someone to not use the receiver address of one transaction as the sender in another one?

For anonymity reasons key pair of a wallet can change.

HD wallets generate an extra key pair for each transaction but the old key-pairs are still valid and are not changed.

Then how does miner validate if a sender has enough bitcoins?

Miners don't care how much money is in a wallet. Bitcoin is cash and like other forms of cash, people don't care how much money you have, they only care that the cash you give them is enough and is genuine and not a forgery.

Is this has something to do with origin of bitcoin sent in transaction?

What you spend as inputs to a new transaction are the outputs of a previous transaction in which you received money. These are unspent transaction outputs (UTXOs).

what if sent sum is 30 BTC, but they are all received from 3 (3x10 BTC) separate wallets?

People don't care. Your transaction could have hundreds of inputs and hundreds of outputs - though that makes the fee higher. All that matters to the recipient and everyone else is that the cash is genuine. In Bitcoin this means UTXOs and that you provide proof of right to spend (you supply a mathematically derived number that satisfies the spending script of the UTXO)

I believe OP may be confused due to thinking Bitcoin has an account-based model.

@Pieter: Yes, I've made an edit to try to make this clearer.

No, I understand that it is decentralized. Other answers you provided answered my question. I made implication that blockchain is made from blocks (which is made from transactions) not from transactions themselves, while block only encapsulate transactions. Thanks.


text:
Generate P2SH bitcoin address from WIF in BitcoinJ Java
I'm trying to create P2SH-Segwit bitcoin address from the WIF private key. I'm using the BitcoinJ library in Java. See the following code.
String base58PrivateKeyString = "KyyJ5vVWjZck5nsAgDWvoN1u7Q8qp5FzE8WiCq97MbnRgdLesqJZ";

NetworkParameters params = MainNetParams.get();
ECKey key;

if (base58PrivateKeyString.length() == 51 || base58PrivateKeyString.length() == 52) {
DumpedPrivateKey dumpedPrivateKey = DumpedPrivateKey.fromBase58(params, base58PrivateKeyString);
            key = dumpedPrivateKey.getKey();
} else {
            BigInteger privKey = Base58.decodeToBigInteger(base58PrivateKeyString);
            key = ECKey.fromPrivate(privKey);
}

// Getting the public key
String pubKeyStr = key.getPublicKeyAsHex();
System.out.println("Public key is: " + pubKeyStr + "\n");

//Getting the P2SH address
List<ECKey> eckeyAList = new ArrayList<>();
eckeyAList.add(key);
Script redeemScript = ScriptBuilder.createRedeemScript(1, eckeyAList);
Script script = ScriptBuilder.createP2SHOutputScript(redeemScript);
byte[] scriptHash = ScriptPattern.extractHashFromP2SH(script);
LegacyAddress legacyAddress = LegacyAddress.fromScriptHash(params, scriptHash);
System.out.println("P2S address from the WIF pivate key is: " + legacyAddress.toString()); //3Az5wdibtPRGac41aGtyqzT1ejtobvb6qW

Its output public key is 03b5319c83adf4a2e274c37401623c7bf0ba453cee3e119f3bc2c523d27059b64f.
Its output P2SH address is 3Az5wdibtPRGac41aGtyqzT1ejtobvb6qW which is far different from the correct P2SH address 38SGXvkMvq8Tsop8rFx2K4JnaFKYZkjd5z.
If any of you know to rectifying the issue I'm happy to take your opinion.

You would need to lost a lot more details. How did you figure out the correct address for your same script?

There are online tools to generate from the base58 private key. Just googled it.


text:
how to calculate max spendable from all wallet
Using latest bitoind, any way to calculate max spendable from wallet with some amount of outputs? For example i need to send all wallet amount to 2 outputs and set mining fee enough to get confirmation in first block. How can i calculate max spendable amount?

In Bitcoin Core, there is the option to subtract the fee from the amount. When you create a transaction, you set the amount to be the maximum amounts, and then you enable the subtract fee from amount options which will then reduce the specified outputs by the fee. This way, you can choose to send your entire balance without change and have the fee be subtracted from the output.
In the GUI (bitcoin-qt), there is a checkbox labeled "Subtract fee from amount" which you can set for each recipient. You can use the "Use available balance" button to auto-fill the full wallet balance. It will also automatically check that option for you.
If you are using bitcoind over the RPC interface, sendtoaddress has a parameter subtractfeefromamount which you can set to true in order to deduct the fee from the amount you are sending. sendmany has a similar option but it is instead an array of addresses so you specify the addresses that will receive less due to the fee being subtracted from their amounts.

ye i know about subtractfeefromamount but i need to calculate amount before(!) send, so with your example we'll understand how much we sent only after processing transaction

You can use fundrawtransaction with the subtractFeeFrom option to construct a transaction (but not sign or broadcast it). Then you can look at the amount in the output to see how much will actually be sent.


text:
find a CTransaction from a prevout in Bitcoin source code
I'm trying to come up with this function: CTransaction getTxFromPrevout(COutPoint prevo);
The idea being... when a new transaction is about to be validated, it will go through all of it's tx.vin[i].prevout and check for them funds (utxo).
So I want to find and identify the CTransaction tx such that tx.vout (of type vector<CTxOut> ) has any i element such that it's CScript is a normal payment (tx.vout[i].scriptPubKey.IsNormalPayment() == true) and the tx.vout[i].scriptPubKey has within it a match for the prevo
If I use a web based explorer I can take the prevo and extract the hash from it, then search the web for the tx... but that doesn't help me, since I want to be able to use this within the code itself, adding this new function. I tried following the code on an explorer and it uses the rpc api, and it was not easy to follow.
What I'm thinking is that there should be a utxo database structure or class that will make this easier to crawl through. If I suppose there is such a class named UTxODatabase - that I know there isn't, but just imagine there is - then the function I need to code would look like this:
CTransaction UTxODatabase::getTxFromPrevout(COutPoint prevo);
/*************************************************/
EDIT 1:
I put this at the very end of the CheckTransaction() function, in main.cpp
BOOST_FOREACH (const CTxIn& txin, tx.vin){
        std::cout << "txin.prevPubKey.ToString(): " << txin.prevPubKey.ToString() << std::endl;
Then I've ran the wallet and the output shown for each txin was just: txin.prevPubKey.ToString:, without any prevPubKey data.
I am really confused about this.
My thinking was that if I'm able to get those prevPubKeys from the CTxIn, then I'll be able to track the utxo. But why would there be no information on that member of the txin at that stage?

There is a utxo database See CCoinsView class in coins.h, the function virtual bool GetCoin(const COutPoint &outpoint, Coin &coin) const; (or one of its overloads) might do what you need.

I'm confused about that virtual qualifier... the main function you referenced me to was:
bool CCoinsView::GetCoins(const uint256& txid, CCoins& coins) const { return false; }  ... it just returns false.
This seems to be on the right track, tho... and probably one of the other implementations should work...  like this one:
bool CCoinsViewBacked::GetCoins(const uint256& txid, CCoins& coins) const { return base->GetCoins(txid, coins); }
But the truth is I'm not sure on how to use it
Could be something like this?
CTransaction CCoinsViewBacked::getTxFromPrevout(COutPoint prevo);

I think what you want is to use this function:
bool CCoinsViewMemPool::GetCoin(const COutPoint &outpoint, Coin &coin) const
You can find the definition here: https://github.com/bitcoin/bitcoin/blob/v0.17.0.1/src/txmempool.cpp#L885
It will set the Coin object you pass as an argument. For an example of usage, see bool CheckSequenceLocks(const CTransaction &tx, int flags, LockPoints* lp, bool useExistingLockPoints) in https://github.com/bitcoin/bitcoin/blob/v0.17.0.1/src/validation.cpp#L364
This is supposed to "provide access to all Coins which are either unspent in the
 * base CCoinsView, or are outputs from any mempool transaction".

OP is asking for the transaction, not just the UTXO. That's just not possible unless you have txindex=1 and no pruning.

I misunderstood, then.

Actually, OP may be confused and not need the full transaction for his purpose.

Agreed! I might not need the whole transaction... but at least I want to be able to validate/compare/check the address to where each of the utxo is pointing at.


text:
How bitcoin protocol checks transaction is valid (in particular that inputs are unspent)?
I'm learning bitcoin protocol theory.
To pay to someone you use a transaction. In the transaction you specify some of old transactions that gave you money (so called inputs). These inputs should have not been already spent.
Therefore the bitcoin network must check these inputs are unspent. Bitcoin network holds the list of all unspent transactions. Transactions are in tree-like structures, and all unspent transactions are the roots of these trees. So bitcoin network checks that input transaction is in this list.
After current transaction is done, all used input transactions (inputs) are now spent, and output transaction and change are now put in the list of unspent transactions.
Is the above process and terminology correct?
EDIT: I found confusing now the thing that transaction can have multiple outputs. This means I cannot just state that transaction is my input, and that the it gets invalidated when used for new transaction, as this transaction has an output that is someone else input. How is this handled?

Yes, you've basically got it.
Regarding multiple outputs, the solution is quite simple.  Your input, instead of just specifying a transaction to spend, instead specifies a particular output of a transaction.  So instead of your input saying "I am spending transaction abc123", it says "I am spending output #2 of transaction abc123".
Every full node on the Bitcoin network maintains a database of unspent transaction outputs or UTXOs.  It is indexed by transaction ID and output number.  So when a node sees your transaction, it will mark transaction abc123 output #2 as spent.  Outputs #0 and #1 of that same transaction have their own independent entries in the database and so their states are unchanged; if they have not been spent previously, they can be spent by some later transaction.

Every full node maintains list of all processed transactions?
This list is called blockchain?
And only part of this list is UTXO.

@croraf: Yes, a full node maintains a complete list of all transactions.  That's what "full node" means. The blockchain consists of this list, plus block headers.  It would be inefficient to search the whole list every time a new transaction has to be verified, so as the node downloads each block for the first time, it builds the UTXO database, so that it can check UTXOs in constant time.


text:
At which point and how does the miner who successfully solved hash get coins?
When miner creates a valid block he broadcasts that to the network.
It can happen that someone else did the same. So we have half nodes think that chain+A is correct, and other half that it is chain+B. This gets resolved when next block is solved.

When are the coins given to the miner, or are they given to both miners?
How do the coins get sent to the miner?
Does the solved block contain the identity (public address) of the solver so the protocol can send him money?
https://en.bitcoin.it/wiki/Block_hashing_algorithm - can't find that address here

EDIT: Now it seems to me that both are rewarded, each inside his respective fork. Something like the reward transaction is put inside that chain-head. When one fork wins the respective reward stays as valid and the other fork is dropped by community. Am I correct?

The truth is that "the point of getting coins" is before the "point of solving block" :)
The miner first creates the coinbase transaction with the reward to himself and tries to solve block after.

Hahahahahaha. Cool answer. And my edit above is also correct?

The block subsidy isn't awarded by anyone. It is taken by the miner, by virtue of creating a block that gives himself money.
The system's role in this is restricted to permitting blocks to indeed contain such an (otherwise unbalanced) payment.
If you have a fork with two branches, then within each branch there will be a payment to some miner. Which one is eventually treated as valuable by the world depends on which of the two gets built on. As far as the system is concerned, there are two possible versions of history, and both are valid. In one of them A gets paid, in the other one B gets paid. At some point, one of the two chains gets longer, and everyone switches to accepting that one.
In order to avoid random forks for flipflopping people's balancea (if they got paid by a block subsidy), bitcoin transaction outputs from coinbase (the type of transaction that credits the subsidy and fees to the mimer), they must be buried by 100 additional blocks before being spent. That is a special rule, called the maturity rule, which only applies to coinbase transactions.


text:
How is mining time validated?
New target is calculated from the T(new_block-1) - T(new_block-2016).
Here T is the time miner mined the block.
Miner itself sets this time for the block.
Thus miner can falsify this time.
I see a rule that this time:

Must be strictly greater than the median time of the previous 11
  blocks. Full nodes will not accept blocks with headers more than two
  hours in the future according to their clock.

Is this the mechanism used to prevent time falsification (thus target falsification)?
Is this mechanism itself enough? If yes, how so?

Try and work through it.  If more than half of all miners (by mining power) are dishonest, the network is screwed anyway.  So assuming that more than half of all miners are honest, what happens if an attacker submits a block whose timestamp is (1) far in the past, or (2) far in the future?

Good comment. (2 - easier) far future it will simply get rejected because of 2h limit. Why such a big limit, something like 15 minutes should be enough, especially because time propagation delays?

(1) A mining pool can mine 5-6 blocks in a row. If he puts them in the far past they are rejected as being lower than median. If the last block is at height N, he can put them somewhere around N-11. This gives a timing error of some 11 blocks. As difficulty is adjusted every 2016 blocks, this will give an error of some 11/2016 which is low. Is this correct?

In Bitcoin the median time past (as in your quote) is used for all operations where the clock matters which limits the effect of any given miners timestamp. Miners can skew the timestamp of their block quite heavily with no ill effect to the network, some previous mining optimisations involved skewing the timestamp forward to avoid needing to do expensive operations to load a new merle root.
There's one exception where some nastiness can occur, which is an off by one error in the difficulty window calculation which means the timestamp of a single block falls outside the calculation window. This is generally known as the timewarp attack, but isn't a significant threat in the real world, it is useful for abusively resetting the difficulty on the test network if it is too high however.


text:
Lightning network smart contract technical explaination (unidirectional payment channel)
Let's observe direct unidirectional lightning channel between Alice and Bob.
How does this start technically:

They first create a 2-2 address C.
They agree on the funds Alice will put into C, say 1BTC.
Bob makes a "conditional refund 2-2 transaction" that has 3 conditions:
a) if Alice put 1BTC into address C
b) if nLock time passed between the time she put money in
c) in no transaction was made from C and put on blockchain
then Alice gets 1BTC back from this address.
He signs that transaction and sends to Alice. Alice can sign and broadcast this transaction after nLock time to retrieve her 1BTC if a) b) and c) are fulfilled.
Alice now puts 1BTC into 2-2 address C.

EDIT: Can you confirm each step.
For example: Is step 2 needed, or can Bob in step 3 just say if nLock time passed and no transaction was put on blockchain from C, whoever put money in C gets its money back in full amount.

Many different approaches for payment channels exist. They differ in the way that refund from the payment channel is guaranteed.
One solution is having Bob to send Alice presigned but timelocked refund transaction, which simply spends from C (from output that Alice will use to fund C) and refunds Alice. As you say the amount must beforehand be agreed upon so that it is correctly reflected by the presigned refund transaction; but it is not necessary to make any conditional checks as you mention them in step 3 since if C's coins have been already spent or not allocated at all the refund transaction will immediately be invalidated. After Bob sends Alice the refund transaction, then Alice can make a funding transaction.
Alternatively for unidirected payment channels, Alice could make use of a more complex contract using OP_CHECKSEQUENCEVERIFY (https://github.com/bitcoin/bips/blob/master/bip-0112.mediawiki). The contract would be something like in the example "Escrow with Timeout" but without the 3rd-party Escrow and only using Alice and Bob. In this way Alice would be assured that no matter how much she initially funds the payment channel, she will be solely able to spend from it again in the future at a time she specifies within the contract. There would therefore not be a need for a presigned refund transaction.
Updating the state of the payment channel is similar in both cases.

Not just the amount has to be agreed upon, but Bob has to know the Alice's initial funding transaction's hash and output index so he can reference it in his refunding 2-2 transaction?

Yes, for the first option the refund transaction must reference the multi-sig output (payment channel) and therefore must reference the funding transaction hash. That is the reason transaction malleability was an issue before segwit implementation. The second option however does not require a presigned refund transaction and Alice can be sure to be able to solely spend from the output after some time has passed. Before this time has passed however both Alices and Bob signature are required to spend from the output.

This means in the first option Alice will make a funding transaction stub, show that to Bob? Using this stub Bob will make and sign a refund transaction. Then Alice will sign the funding transaction stub and broadcast it.

Yes, but in fact Alice could create the whole refund transaction herself and just ask Bob to sign it.

Yes .
I like this video as well, explaining the u and b directional channel.
The Lightning Network Explained (Litecoin/Bitcoin)

Thanks Haddar. I'm interested in details of step 1 and 2. Does it go like this: Alice and Bob first create 2-2 address C, then they agree on the funds Alice puts in, then Bob makes and signs a conditional 2-2 transaction that says "IF Alice puts her funds in she gets them back IF another transaction is not made from C within nLock"?

Alice and Bob generate 2-2 multisig address, Bob sign and broadcast a tx with nLock from that multisig to Alice address, then Alice can start fund the multisig address ...

I updated my question

I think step 3 is too much, Bob only needs to sign a spend multisig as input with the amount they agree on into Alice address after nLock - and show the tx to Alice / broadcast it.


text:
are official bitcoin core released compiled with zmq in general?
I installed the binary and try to listen to ZeroMQ stream. But I got nothing out. Then I installed ZeroMQ but still got nothing.
https://bitcoin.org/en/release/v0.17.1
so, are official bitcoin core releases compiled with zmq in general?
bitcoin.conf
listen=1
upnp=0
txindex=1
maxconnections=40
server=1
rpcthreads=4
rpcuser=alice
rpcpassword=alice
rpcauth=xxxxx
rpcallowip=0.0.0.0/0
rpctimeout=30
txconfirmtarget=6
mempoolexpiry=72
maxmempool=300
maxorphantx=100
debug=rpc
logips=1
limitfreerelay=10
minrelaytxfee=0.0001
zmqpubhashblock=tcp://127.0.0.1:28332
zmqpubhashtx=tcp://127.0.0.1:28333
[main]
rpcbind=192.168.1.107
[test]
rpcbind=0.0.0.0
[regtest]
rpcbind=0.0.0.0

I found this article https://bitcoin.stackexchange.com/a/65066/33448
but there is no such file config.log. but I did install successfully with apt-get  --yes install libzmq3-dev
docker-compose.yml
....
    bitcoincore:
      image: bitcoincore
      networks:
        - my-network
      build:
        context: bitcoinCore/
      volumes:
        - shared:/rpc
      ports:
        - 8332:8332
        - 8333:8333
        - 18332:18332
        - 18333:18333
        - 18443:18443
        - 18444:18444
        - 28332:28332
        - 28333:28333
....

net stat -pant
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:18332           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:18333           0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.11:35491        0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:28332 <---    0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:28333 <---    0.0.0.0:*               LISTEN      -
tcp        0      0 172.22.0.2:38722        31.220.15.89:18333      ESTABLISHED -
tcp        0      0 172.22.0.2:60910        193.31.24.45:18333      TIME_WAIT   -
tcp        0      0 172.22.0.2:40730        159.203.125.125:18333   ESTABLISHED -
tcp        0      0 172.22.0.2:60938        54.162.188.154:18333    ESTABLISHED -
tcp        0      0 127.0.0.1:58746         127.0.0.1:18332         TIME_WAIT   -
tcp6       0      0 :::18333                :::*                    LISTEN      -

getzmqnotifications(inside container)
#  bitcoin-cli -testnet -rpcuser=alice -rpcpassword=alice getzmqnotifications
[
  {
    "type": "pubhashblock",
    "address": "tcp://127.0.0.1:28332"
  },
  {
    "type": "pubhashtx",
    "address": "tcp://127.0.0.1:28333"
  }
]

test RPC at host
$ curl --user alice:alice -H 'content-type:text/plain;' http://localhost:18332/ --data-binary '{"jsonrpc":"1.0","id":"1","method":"getmininginfo"}'
{"result":{"blocks":1348883,"currentblockweight":0,"currentblocktx":0,"difficulty":4194304,"networkhashps":48545593617877.57,"pooledtx":0,"chain":"test","warnings":""},"error":null,"id":"1"}

try to use lld to find out if dependency is ok, but no luck
# ldd /opt/bitcoin-0.17.1/bin/bitcoind
    linux-vdso.so.1 (0x00007ffe29b6c000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6dd6f8c000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f6dd6f82000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6dd6dff000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f6dd6de5000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6dd6c24000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f6dd7c53000)

tester.js
var zmq = require('zeromq')
  , sock = zmq.socket('sub')
  , RpcClient = require('bitcoind-rpc')
  , bitcoin = require("bitcoinjs-lib");

var config = {
    protocol: 'http',
    user: 'alice',
    pass: 'alice',
    host: 'localhost',
    port: '18332',
};

var rpc = new RpcClient(config);

sock.connect('tcp://localhost:28333');

sock.on('message', function(topic, message) {
console.log(topic, message);
  var tx = bitcoin.Transaction.fromHex(message)
  if (!tx.isCoinbase()) {
    rpc.generate(1, console.log)
   console.log(topic, message)
  }
})

// Subscribe to receive messages for a specific topic.
// This can be "hashblock", "hashtx", "rawtx", or "rawblock".
//sock.subscribe('rawtx');
sock.subscribe('hashtx');

rpc.getDifficulty( (err, data) => {
    if (err) {
        console.log(err);
        return;
    }
    console.log('RPC working Ok; obtain difficulty as followed: ',data);
    console.log('listening....');
});

Release binaries are statically linked, so the fact that you don't see a dependency on libzmq is not a surprise. It's also unnecessary to install anything as it's all built into the bitcoind binary. If it wasn't included you wouldn't be seeing the TCP listen on those ports. The fact that things don't work is probably due to an unrelated issue.

@PieterWuille is right, it is really not related to bitcoin but to ZeroMQ and docker. I look into the source code and bitcoin core uses zmq_bind( socket, bindingAddress ). For BindingAddress, I can change it to tcp://*:28332 because callers from another containers or from host would have different IPs, so the most convenient way is to set it to a range of address or simply tcp://*:28332
please refer to ZeroMQ man page at the bottom for more examples http://api.zeromq.org/4-1:zmq-tcp
ANSWER: Release binaries have ZeroMQ statically built-in.

Thanks! To get bitcoind and lnd to operate in seperate docker containers, I had to use network_mode: "host", since as you mentioned. zeromq port binding within bitoind is flawed.

You may want to try install libzmq in your system and enable zmq in bitcoin.conf.

I did. I used https://github.com/ruimarinho/docker-bitcoin-core and I entered the container and installed ZeroMQ manually and enable the bitcoin.conf
zmqpubhashblock=tcp://127.0.0.1:28332 zmqpubhashtx=tcp://127.0.0.1:28333
no luck

I even wrote a mini node.js program to capture the response at the host.

# ldd /opt/bitcoin-0.17.1/bin/bitcoind  linux-vdso.so.1 (0x00007ffe29b6c000)  libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6dd6f8c000)  librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f6dd6f82000)  libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6dd6dff000)  libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f6dd6de5000)  libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6dd6c24000)  /lib64/ld-linux-x86-64.so.2 (0x00007f6dd7c53000)
it seems that it is not linked even after I installed zmq. how do I link them?

I downvoted. This is unrelated. You need to install libzmq if you're going to compile Bitcoin Core yourself. Release binaries have it statically built-in. It's clear from OP's question that this is the case for him, as ports are opened.

@PieterWuille thanks, you point me to the right direction.


text:
Can a competitive miner be crafted at home?
Ive been wondering, is it possible to make a good custom Bitcoin miner? AntMiners aren't worth it anymore, and I can't find a miner that is. Can I build a miner myself that will be worth it? If so, what do I need to custom make a miner? What are some basic Bitcoin miner components. How much will it cost approximately? Is there anything else I need to know if i'm going to build a miner? There was some confusion, I don't mean I want to print my own pieces and make my own miner completely from scratch, I want to buy pieces online and make a miner, can I do this somehow?

The last time I heard pricing of the engineering costs it was in the $500,000+ cost range to develop an asic

Two words. Holy Crap.

No, the ASIC chips used in miners are impossible to craft at home, and to my knowledge no competitive chips are for sale as component parts.
Development of new ASIC chips has progressed to a level where it costs millions to create a new template and you need state of the art (<16nm) production to produce them. It is not economic to do this at home.

The closest you will come from the comfort of your home is an FPGA, which basically allows you to implement a customized circuit that is for you application. If you are not familiar with the SHA-256 hashing function and Bitcoins implementation of it, it could take you a long time.
This would never be better than simply buying an ASIC. It will not be more efficient. But if your goal is to learn, then you could try.

Unless you live in an area with extremely cheap (or free) electricity, you probably will not be able to build your own miner from scratch.  Modern mining requires ASIC's (Application Specific Integrated Circuit) in order to make any money.  These are silicon chips that implement the SHA256 hashing algorithm in hardware, rather than running a software version of the algorithm on a general purpose CPU, or even GPU.  Unless you are able to manufacture your own custom silicon chips, you would be better off purchasing an ASIC miner.

You could build your own miner if you have the technical knowhow and equipment using a batch of Bitfury 75GH/s chips, however, the idea of doing this at home is not going to happen. I am told that Bitfury has a minimum order of USD 2 M for all products (EDIT: seems they may have changed their policy) but, you might be able to find smaller quantities on eBay or elsewhere online. For USD 2 M it is more likely you might start producing en-masse in your garage. You might have more luck finding a retailer selling 55TH/s, 5.6kW Bitfury B8 servers or a Bitfury Block Box.
I am not affiliated with Bitfury.
If anyone finds any useful non-eBay links for these products post them in the comments and I will update my answer.

authors: Инна Журавлева


text:
Electrum - Setting up a server - Problems!
So im trying to request payment through my website but having problems.
Im using Windows 7 x32 as a testing environment. I installed Electrum via the installer and not the standalone exe.
First thing is that electrum is showing this in Preferences when it comes to SSL:

Im using a self-signed certificate that I generated from OpenSSL. Next, i think i messed up when it came to the rewrite settings because when I type in "electrum addrequest 3.14 -m "this is a test" " via CMD (with admin), two files  are generated in the right path of my server

(the index.html was not generated, i coped it from one of the electrum folders)
the index.html was not generated, i coped it from one of the electrum foldersbut without the index.html. I actually tried finding a copy of the index.html and pasting it to the directy of the folder and tried doing the "index.html?id=(ID HERE)" but nothing changed as shown here!

When I tried doing "listrequests()" via the Electrum command line, i got this:

Can someone please tell me where I messed up?

Firstly, if you want to request payments through your website, it might not be necessary to use electrum. Maybe an easier solution might fit you. Secondly, this seems not very well documented, maybe you should try the electrum irc channel to find help on that...


text:
Is it possible to combine two separate transactions into one transaction by only using their signatures?
I have two valid txs and two valid signatures for those txs.
Right now they look like:
1) A -> B
2) C -> D

A and C are inputs & B and D are outputs.
I want to be able to combine them into:
1) A, C -> B, D

Is there a way that I can combine these two tx's and their corresponding signatures into one valid tx without having access to the keys?

I'm not sure what is being asked here.  Are A and C transactions?  Signatures?  What are 'B' and 'D'?  Do you mean "UTXOs" when you say "txs"?  I don't know what you are getting at.

@Jestin I might not be using the correct terminology... A and C are inputs B and D are outputs

Out of curiosity, why do you want to do that?

The only possibility in Bitcoin today is the one described by kaykurokawa. Every signature specifies which inputs and outputs it signs for, so if you want independently-created signatures, it will be obvious to everyone which input and output were created simultaneously.
The general practice of combining multiple pieces of transactions into one is called CoinJoin. It works well in theory, but it does require the different parties to cooperate (each signs everyone's inputs and outputs).
A potential future development that could change this is, is based on One-Way Aggregatable Signatures (OWAS) (see the paper on https://download.wpsoftware.net/bitcoin/wizardry/horasyuanmouton-owas.pdf originally posted anonymously on https://bitcointalk.org/index.php?topic=290971.0). This mechanism would allow anyone on the network to take two transactions and CoinJoin them, without being able to tell afterwards which inputs and outputs were related. As a side benefit, it also combines all signatures in such transactions into one. Taken to the extreme, it would mean that a block can just be reduced to a single transaction, constructed by miners applying this joining operation on all transactions they want to include.
Unfortunately, OWAS relies on a signature scheme called BLS, which is much newer and not as well studied as ECDSA (which Bitcoin uses now), so its security is less certain. Furthermore it is far slower, and would need very invasive changes to Bitcoin. It is an active area of research, though.
More recently, a system called MimbleWimble has been proposed which supports offline transaction aggregation as well as cut-through transactions while only relying on EC cryptography.

Yes. But they require the transaction to be signed a certain way. OP_CHECKSIG has settable flags, which determines which of the inputs and outputs are signed.
If SIGHASH_SINGLE and SIGHASH_ANYONECANPAY is used for OP_CHECKSIG , each signature signs for its respective input and outputs, allowing others to add new inputs and outputs to the transaction.
https://en.bitcoin.it/wiki/OP_CHECKSIG
https://bitcoin.org/en/glossary/sighash-single
https://bitcoin.org/en/glossary/sighash-anyonecanpay

If you use SIGHASH_SINGLE, you're basically giving away that the input and output are related. This unfortunately means the purpose of the coinjoin (privacy) is defeated.

@Jimmy Song Perhaps privacy is not the reason the OP wants this.

True that this would defeat the purpose of CoinJoin by clearly linking inputs and outputs. But it could be used to lower fees. Imagine a service that repackages input/output pairs signed with SINGLE/ANYONECANPAY into larger transactions to reduce storage overhead.


text:
Where does bitcoin-qt store it's initial settings
I installed bitcoin-qt (v 0.12.1) on Kubuntu 15.10. When starting it the first time the GUI asked me for the datadir. I set it to an external disk.
Where are this settings stored? There is no bitcoin.conf file in my /home/<user>/.bitcoin/ directory.
I know, that I can manually create the bitcoin.conf file. But I'm just curious, where the settings are stored, if no such file is present?

Bitcoin-Qt stores its own modifiable settings outside of bitcoin.conf (which is intended to only be read).
The location and format of that file is platform dependent, but on Linux it is ~/.config/Bitcoin/Bitcoin-Qt.conf.


text:
Why did they have to expand the nonce value range?
My understanding is that at a certain point, once hardware passed the 4 GH/s level, the 4 billion possible values for the nonce became insufficient in bitcoin mining, because a good rig could exhaust them all in less than a second, and then had to wait until the next second for the timestamp value to change and then try the next 4 billion.
The thing I don't get, though, is why they couldn't adjust the timestamp in increments of less than one second? What about milliseconds and nanoseconds? Why not adjust that input instead of the nonce range?
Edit: Is it that the timestamp has a fixed format that ends at seconds, and changing this would be more difficult / less efficient than simply appropriating space from the coinbase transaction?

They didn't. Nonce is still 4 bytes [1]. Miners don't have to wait when they run out of nonces. Miners just adjust another nonce (the extraNonce) in the first transaction (i.e., in scriptSig of the coinbase transaction [2]). That gives them another ~4 billion tries for the first nonce.

That's what I was referring to in my edit, that's the question: why is the extranonce preferable over using smaller subdivisions of time in the timestamp? Because the timestamp is of fixed size?

Oh, I see. The title was misleading for me. The timestamp in the header is another 32 bits (see https://bitcoin.org/en/developer-reference#block-headers again) stored in UNIX epoch time format (i.e., seconds since 1/1/1970; seconds, not milliseconds). However, "full nodes will not accept blocks with headers more than two hours in the future according to their clock" so there's only so many bits you can change in that timestamp (I think around \log_2{2 * 60 * 60} bits).


text:
MULTISIG Verified transaction don't get mined on testnet
I'm trying to check the following simple MULTISIG P2SH script on bitcoin testnet (using python-bitcoinlib):
Redeem = CScript([2, J1_public_key, J2_public_key, J3_public_key, 3, OP_CHECKMULTISIG])
ex31a_txout_scriptPubKey = Redeem.to_p2sh_scriptPubKey()

My first transaction got mined. Here is the link. It is verified by the python library python-bitcoinlib that the following scriptSig can redeem the coin.
J2_signature = create_OP_CHECKSIG_signature(txin, txout, Redeem,
                                          J2_private_key)
J3_signature = create_OP_CHECKSIG_signature(txin, txout, Redeem,
                                          J3_private_key)

dummy = 123
txin_scriptSig = CScript([dummy, J2_signature, J3_signature, Redeem])

However my redeeming transaction don't get mined on the testnet. I don't know why!

Edit
A minimal example is replaced instead of the previous one.
The new script uses P2SH.
Here is the API code of the transaction:
{
  "block_height": -1,
  "block_index": -1,
  "hash": "6371461a476ae87dd31b89e7deb9ff59b9679306b23ff86036dc3d7851a76a3c",
  "hex": "01000000019d8e6c705ba87c12f785542ea6253478585a24d871624036c3cfa30b1c0bfbe400000000fddb01017b483045022100dc369db54ceaa7b182969f23a176ae21d14272dff8d941360a5140d17f196e24022044ed2a8c05c5eff115fd5536478027b74a8aeea8f8e393377fcf7c8c232b352201483045022100c227a4fa63b5040bc85a5e7b15c0b081aadd4c538223fa9d8c90cadbc04e649702205a72760687a8c9e74202bfd11b6661f44f093e51a7a4eb4d89876303a9bc94d8014730440220586615cbc74089d7ab4aaf96dc857c0d523df77260159dced512f72c9d3d88b30220177c6f3eb5622ed263c28052ffcda89062a1bda74fdf8e398fba76dd85942bbd014cfd6e51210245aef9640296f238a04b0ee9af1e8dc62a6fe51214af7d3c5bfe518a9cd5afaf2103482de8ad8bf744318b97063544d2fc59322f63baad3d1054ea57609cdb059b6452af6e52210245aef9640296f238a04b0ee9af1e8dc62a6fe51214af7d3c5bfe518a9cd5afaf2103482de8ad8bf744318b97063544d2fc59322f63baad3d1054ea57609cdb059b6452ae635167522102e1183cefc9b3fb15e15bfc53703a6d2e626ccf42cc017f84ec41486114d8cefe210200139f5a548a7cebce4be1308a0262fbb04ec2ea6fc5b38bb3c4ac140ccb5d852102259a3b4527adf82a32e06fb177f27e0606c10f329a22ba7b206ea3eef0e5aa2853ae68ffffffff01a0860100000000001976a9149f9a7abd600c0caa03983a77c8c3df8e062cb2fa88ac00000000",
  "addresses": [
    "2Mz66CzbFYADatUbwXWqHxayRrMHoTCAqXn",
    "mv4rnyY3Su5gjcDNzbMLKBQkBicCtHUtFB"
  ],
  "total": 100000,
  "fees": 400000,
  "size": 562,
  "preference": "high",
  "relayed_by": "194.225.46.158",
  "received": "2018-11-18T21:19:22.298Z",
  "ver": 1,
  "double_spend": false,
  "vin_sz": 1,
  "vout_sz": 1,
  "confirmations": 0,
  "inputs": [
    {
      "prev_hash": "e4fb0b1c0ba3cfc336406271d8245a58783425a62e5485f7127ca85b706c8e9d",
      "output_index": 0,
      "script": "017b483045022100dc369db54ceaa7b182969f23a176ae21d14272dff8d941360a5140d17f196e24022044ed2a8c05c5eff115fd5536478027b74a8aeea8f8e393377fcf7c8c232b352201483045022100c227a4fa63b5040bc85a5e7b15c0b081aadd4c538223fa9d8c90cadbc04e649702205a72760687a8c9e74202bfd11b6661f44f093e51a7a4eb4d89876303a9bc94d8014730440220586615cbc74089d7ab4aaf96dc857c0d523df77260159dced512f72c9d3d88b30220177c6f3eb5622ed263c28052ffcda89062a1bda74fdf8e398fba76dd85942bbd014cfd6e51210245aef9640296f238a04b0ee9af1e8dc62a6fe51214af7d3c5bfe518a9cd5afaf2103482de8ad8bf744318b97063544d2fc59322f63baad3d1054ea57609cdb059b6452af6e52210245aef9640296f238a04b0ee9af1e8dc62a6fe51214af7d3c5bfe518a9cd5afaf2103482de8ad8bf744318b97063544d2fc59322f63baad3d1054ea57609cdb059b6452ae635167522102e1183cefc9b3fb15e15bfc53703a6d2e626ccf42cc017f84ec41486114d8cefe210200139f5a548a7cebce4be1308a0262fbb04ec2ea6fc5b38bb3c4ac140ccb5d852102259a3b4527adf82a32e06fb177f27e0606c10f329a22ba7b206ea3eef0e5aa2853ae68",
      "output_value": 500000,
      "sequence": 4294967295,
      "addresses": [
        "2Mz66CzbFYADatUbwXWqHxayRrMHoTCAqXn"
      ],
      "script_type": "pay-to-script-hash",
      "age": 0
    }
  ],
  "outputs": [
    {
      "value": 100000,
      "script": "76a9149f9a7abd600c0caa03983a77c8c3df8e062cb2fa88ac",
      "addresses": [
        "mv4rnyY3Su5gjcDNzbMLKBQkBicCtHUtFB"
      ],
      "script_type": "pay-to-pubkey-hash"
    }
  ]
}

According to your link there is "no transaction found with the hash 42af7..."

@RedGrittyBrick Since the transaction didn't get mined it has been removed after a couple of hours

It seems that the problem is related to the testnet itself. Since the fees in the testnet aren't real the minors don't have the incentive to mine complicated scripts. Hence scripts such as MULTISIG might not get mined (for me with a high probability approaching one!).
To see this you may add a big number at the beginning of the scriptSig.
For the case of MULTISIG it is enough not to use P2SH method. i.e.
 ex32a_txout_scriptPubKey = CScript([2, J1_public_key, J2_public_key, J3_public_key, 3, OP_CHECKMULTISIG])

and the scriptSig can be:
CScript([dummy, dummy, J2_signature, J3_signature])

This transaction gets recognized as pay-to-multi-pubkey-hash and get mined.
However this will not resolve the problem with more complicated scripts on testnet.


text:
Blockchain data storage
My learning on blockchain is mainly theory based. I have always been under the impression that blockchain is about data processing and data storage which I think this is where I am confused.
I have always pictured blockchain as a P2P decentralised data storage system where anyone can transfer/upload/store anything on the blockchain network. Data can be anything of a value. Any upload and transfer of data is encrypted. Data is stored encrypted. There is some magical formula that turns whatever size data into digest. Anyone who retrieve the digest will be able to turn it back to the original content with the key.
The more I delve into it, the more it seems I have been understanding it all wrong.
Data Storage
I understand Bitcoin is not meant for data storage. It can be but not meant for. I read about IPFS.
Now, the data storage part whenever it is mentioned with blockchain technology(not restricted to bitcoin), without reference to IPFS or equivalent, does the data generally mean hashes of digital signatures? We store the digital signatures of the data. We don't 'upload and save' the content like what we do with cloud storage? (Let's assume cost of storage is not an issue.)
The recipient is not able to 'convert' the digest to the original content since the original content is not 'transfer'/'store' on the blockchain at all.
If we want to encrypt the data, we will do it 'on our own' using an external service off the chain. We sign the encrypted data with private key. The recipient will use our public key to verify the digital signature and what they received is the encrypted data. The decryption part is to be communicated off the chain.
Have I understood the above correctly now?
=======
My question is not restricted to Bitcoin but blockchain in general.

How does IPFS use Blockchain
https://ipfs.io/#how says

Blockchains
IPFS and the Blockchain are a perfect match! You can address large amounts of data with IPFS, and place the immutable, permanent IPFS links into a blockchain transaction. This timestamps and secures your content, without having to put the data on the chain itself.

So yes, IPFS is separate from the Bitcoin blockchain but it seems you have the option of embedding a hash in a Bitcoin transaction (for which the usual fees are needed) as proof that the hashed data existed at a point in time.
The Bitcoin blockchain is a journal of Bitcoin currency transactions. Small items like hashes can be included in the transaction details that are recorded but you pay a fee per byte and you need to have a funded wallet.

Related Questions

What methods are currently used to embed additional data into the Bitcoin blockchain?

Both Bitcoin and IPFS have a p2p messaging protocol where peers can request and offer public data resources to each other.
Saving Data in the Bitcoin Blockchain
The Bitcoin p2p layer is quite simple and simply allows Bitcoin nodes to gossip/propagate new transactions and valid blocks which extend the chain. Bitcoin nodes take these public data resources (tx's, blocks) and validate them to determine the canonical transaction history.
Saving data in the Bitcoin blockchain means "appending it" to one of these datatypes. You can add up to 80B of arbitrary data to any TX output script which you sign (See OP_RETURN), or arbitrary data in the Coinbase transaction input script if you mine a block. Once organised into the longest chain, both TX and Blocks will immutable, and considered part of the canonical valid transaction history, and so will your arbitrary data. Of course, the amount of data is very limited and not critical to the security or functionality of Bitcoin.
Distributing Data with IPFS
You can think of IPFS as a more generalised p2p protocol, which extends beyond just gossiping/flooding the peer topology with specific types of data (transactions, blocks). IPFS nodes request/provide peers with any filetype, and extend this to any type of data the user would like to share. The data itself is hash-addressed so it can be uniquely identified (similar to tx, and blocks, which are also hash-ID'd). Furthermore, IPFS offers metadata data-types to link different files to each other via hash-links (See IPLD).
Finally, the IPFS also hosts a distributed-hash-table (see Kademlia DHT), which enables any peer to to look up a distributed global map. This can be used as a global namespace, or a distributed resource to find the address of a peer that is currently hosting your file-of-interest.
Saving data in IPFS simply means that your file is being hosted/cached by at least one of the IPFS nodes in the network. If it is requested by other nodes, it will be cached locally at each requesting node for a time period. There are no guarantees that the file will be hosted indefinitely, as network peers collectively serve as distributed cache. The only guarantee to availability is to host your file(s)/data on a IPFS server you run yourself (pinning).
IPFS is conceptually very similar to Bittorrent/Gnutella.


text:
Lightning-Network Channel not being closed and balance in "limbo"
My node Caffeine has a channel in pending status (close txid: 0de535ea74d72a952157640187b5a207fe977870a5c9b92f0aa99b9707c94779) with The Cave since months!
My local balance is stuck in a "limbo" and I have no idea what to do...
How to finalize the closure? How to get the balance out of the limbo?

your transaction is not on the mem pool which meant that your close transaction is not committed. With implementation are you using?

@vincenzopalazzo LND. What shall I do? Why doesn't unstuck itself if it's not in the mem pool?

if the tx is not in the mempool you have not close the channel

@vincenzopalazzo I don't understand. Is there something I can do at this point?

try to contact the lnd team

I have open an issue on the GitHub repo of LND and a solution was found, here is the link https://github.com/lightningnetwork/lnd/issues/5341


text:
How to get the wallet address of my own LND node?
I have set up a lighting node using LND time ago, now I would like to send more BTC to it.
How can I get the wallet address of my LND node?

Use lncli h to view a list of all the available commands.
lncli newaddress will give you a new address for the wallet. You must specify the address type: either p2wkh (pay to witness key hash), or np2wkh (nested pay to witness key hash). For example, lncli newaddress p2wkh.
(Running lnd v0.6.0-beta)

I don't want a new address. I want to know the current one. I checked all the commands, can't find any that simply output the current wallet address.

Ah, well there is no command to display an already-used address, by design. There is really no need to use an address more than once, in fact doing so is very detrimental to your privacy. You should use a new address for every transaction.

I understand the privacy concern. But still, it would be nice to have that information in case you need to check the transactions in the blockexplorer. But ok, if there is no option then I will give up. I though it was me not able to find the command.

I can understand why you might want the history of addresses, but your Q specifically said you want to send more BTC to already-used addresses. In any case, listunspent or listchannels will be give you some clues for past addresses, if you’re willing to dig into the blockchain record a bit by following UTXOs through time. I’d be careful about reusing them for new payments though, lest you make a mistake and send funds to an address that is a LN channel by mistake, for example. It would be very difficult to recover those funds if a mistake was made.

Agree with @chytrik, however if you wanted to view all onchain addresses, you could get this information from listchaintxns


text:
What is the difference between wallet balance, local balance and remote balance in the Lightning Network?
I am having a hard time understanding the difference between the different "balances" in the Lightning Network. For example, I am running an LND node and I have the following balances:
$ lncli walletbalance
{
    "total_balance": "13841305",
    "confirmed_balance": "13841305",
    "unconfirmed_balance": "0"
}

$ lncli listchannels | jq -r '.channels[].local_balance' | awk '{sum+=$1} END {print sum}'
4201392

$ lncli listchannels | jq -r '.channels[].remote_balance' | awk '{sum+=$1} END {print sum}'
23804004

What is the difference between the local balance and the wallet balance?
If my node forwards a transaction, where do the fees go, in the wallet, or in local?

Walletbalance is showing your on-chain balance of a lightning wallet.
Regarding channel balances (locked on layer 2 and not available onchain):

local balance is your local balance that you can spend on lightning network
remote balance is balance in your channels but it's being owned by the other party in the channel, this is the amount you are able to receive
there is also commitment fee locked in every channel, not listed in your snippet. No parties can use it to send funds in the lightning network

Fees for routing will go to your local balance.
I've made a video explaining this question.

Is there a way to see the commitment fee?

Using lncli listchannels you'll see something like: "capacity": "x", "local_balance": "x", "remote_balance": "x",  "commit_fee": "x",  "commit_weight": "x", commit_fee being usually in thousands of sats.

In each of your channels, you have a local balance. This is the balance that is yours in that channel. The counterparty of the channel has the remote balance. All lightning transactions change these balances with each one resulting in their sum being the same and matching the channel capacity. As all transactions in that channel just end up changing the balances, routing fees paid to you go into your local balance.
Your wallet balance is the total amount of funds that you control that are not in a channel. It's the sum of the UTXOs that you control.


text:
Why 1D00FFFF and not 1CFFFFFF as target in genesis block
Is there a specific reason why 1D00FFFF is the target bit for the genesis block and not 1CFFFFFF? Both bits are forcing 8 zeros in the front of the hash (hexadecimal).
Regards steff

In the custom uint256 encoding used in Bitcoin, bit 0x00800000 denotes the sign, so 0x1CFFFFFF actually denotes the value -2200(223 - 1). Obviously the target value cannot go negative, so this aspect of the encoding is not useful in this context, but for whatever reason, that's what Bitcoin's creator chose.

Thank you that was helpful. I am not so deep in coding. That means when I am looking at the last 6 digits 000000 to 7FFFFF are positive and 800000 to FFFFFF are negative?

@steff123 That's right.


text:
Struggling to Decode Raw Hex Data of a Block
I am trying to Decode the Raw hex data of a Coinbase Block to check my understanding of the DS but I am facing some difficulties in decoding marked by a ? below:
Q1. Can someone please check the existing understanding/interpratation and help explain and fill the gaps (marked by ?)
Q2. Is there a simple standalone program/library in to do the same? I have seen few parsers (though haven't used) but they require the complete blockchain .dat files that i cannot afford to download for this task.
The block is: https://www.blockchain.com/btc/block/485300

0000002053f7ebc735f54ac8d4ebcc1eeb3d3bdea89603bdd27431000000000000000000ff964ec70ea5a5356a04fdb044ca6c59ffda2dd6be02c63d9ea58fb6d46e01991a22bb590b310118f6df26f801010000000001010000000000000000000000000000000000000000000000000000000000000000ffffffff3103b46707244d696e656420627920416e74506f6f6c6b2f4542312f4144362f4e59412f332059bb221ab8080000615c0300ffffffff02807c814a000000001976a914660371326d3a2e064c278b20107a65dad847e8a988ac0000000000000000266a24aa21a9ede2f61c3f71d1defd3fa999dfa36953755c690689799962b48bebd836974e8cf90120000000000000000000000000000000000000000000000000000000000000000000000000

[PARAMETER]           [VALUE]

[Block Header]
version             - 00000020
Previous Block hash - 53f7ebc735f54ac8d4ebcc1eeb3d3bdea89603bdd27431000000000000000000
Merkel Root         - ff964ec70ea5a5356a04fdb044ca6c59ffda2dd6be02c63d9ea58fb6d46e0199
time                - 1a22bb59
bits                - 1801310B
nonce               - f6df26f8

01 - transaction count

[Tx List]
version                 - 01000000
<?>                     - 0001
input ct                - 01
prev tx                 - 0000000000000000000000000000000000000000000000000000000000000000
prev out n              - ffffffff
coinbase script length  - 31
coinbase script         - 03b46707244d696e656420627920416e74506f6f6c6b2f4542312f4144362f4e59412f332059bb221ab8080000615c0300
sequence                - ffffffff
Output count            - 02
value 1                 - 807c814a00000000
Script 1 length         - 19
Script 1                - 76a914660371326d3a2e064c278b20107a65dad847e8a988ac
value 2                 - 0000000000000000
Script 2 length         - 26
Script 2                - 6a24aa21a9ede2f61c3f71d1defd3fa999dfa36953755c690689799962b48bebd836974e8cf9
<?>                     - 0120
witness transaction id  - 000000000000000000000000000000000000000000000000000000000000000000000000

Q1 segregated witness aka segwit Also to be clear multibyte numbers are little-endian, and you've swapped bits to big-endian but not the others.

@dave_thompson_085 thank you. updated the structure above for the seg_wit transaction id. just 4 bytes above unclear. can you please clarify those also?

This question is similar to: What are the parts of a Bitcoin transaction in segwit format?. If you believe it’s different, please edit the question, make it clear how it’s different and/or how the answers on that question are not helpful for your problem.

This website can answer your question.
Typically, the first question mark refers to flag and the second question mark refers to tx_witnesses


text:
Bitcoin C# .NET API
I am trying to get the address the bitcoins were last owned by or the "from" address, I know that if you send bitcoins back to that address not in all cases will the user actually get them but I still need the code for it. Thanks in advance.
I am thinking it has something to do with Program.b.GetTransaction()
If I have an address, and coins are sent to that address, I want to find out the address the coins were sent from, I have a different address for every transactions specially for this.

You could just make straight calls to bitcoind, from within a C# app.  Take a look at this question(http://bitcoin.stackexchange.com/questions/7369/how-to-implement-a-game-like-satoshidice/14073#14073).  The question is about Satoshi Dice, but it does provide instructions on how to get customer information, including their original payment address.  I'm not aware of a C# client.  However, a C# application could easily integrate with the bitcoind.exe console application.

I maintain a c# wallet client located on Github: https://github.com/chriswill/WalletClient.  The apis may help you.

Please look at the date, I have done C# and PHP implementations of that already...