text:
How to Pay to Multisig Wallet Using python-bitcoinrpc?
How do I pay to a multisig wallet?  I am using the Python API (based on python-bitcoinrpc) that ships with the bitcoin source code, in directory bitcoin/test/functional.  I'm using regtest.
I call createmultisig() in order to generate a P2SH address.  I create a CTransaction object.  I attempt to append to the vout list a CTxOut object that pays to the P2SH address, but this operation corrupts the object.  I think I need to do something to the P2SH address before passing it as an input to the CTxOut object, but I can't figure out what.
Below is the script for my unit test.  The line where it goes wrong is:
tx1.vout.append(CTxOut(int(8 * COIN), madd))

After that the tx1 object is corrupted and when I subsequently try to print it to the log, the operation fails with the error TypeError: can't concat str to bytes.
Any idea what I'm doing wrong?
#!/usr/bin/env python3

import pprint

from test_framework.test_framework import BitcoinTestFramework
from test_framework.messages import CTransaction, CTxIn, CTxOut, COutPoint, ToHex, COIN
from test_framework.script import CScript, CScriptOp, OP_1, OP_DROP, OP_2, OP_HASH160, OP_EQUAL, hash160, OP_TRUE, OP_DUP, OP_EQUALVERIFY, OP_CHECKSIG
from test_framework.util import hex_str_to_bytes, bytes_to_hex_str
from test_framework.address import byte_to_base58

class RawTxTest(BitcoinTestFramework):

    def set_test_params(self):
        self.num_nodes = 2
        self.extra_args = [[],["-txindex"]]

    def run_test(self):
        """Main test logic"""

        self.log.info(f"\n    DEBUG START")

        # initialize node variables
        n0 = self.nodes[0]
        n1 = self.nodes[1]

        # initialize wallet and address
        w0 = n1.createwallet(wallet_name="test_wallet")
        w0_rpc = n1.get_wallet_rpc('test_wallet')
        addr0 = w0_rpc.getnewaddress()
        self.log.info(f"\n    DEBUG addr0={addr0}")

        # TRANSACTION #0 - pay from node 0 to addr0
        txid0 = n0.sendtoaddress(addr0, 10.0)
        n0.generate(6)
        self.sync_all()

        # dump the transaction
        raw_tx0 = w0_rpc.getrawtransaction(txid0, True)
        s = pprint.pformat(raw_tx0)
        self.log.info(f"\n    DEBUG tx={s}")

        # write the balance to the log
        bal0 = w0_rpc.getbalance()
        self.log.info(f"\n    DEBUG bal0={bal0:f}")

        # TRANSACTION #1 - Pay from addr0 to multisig
        NUM_SIGS=2  # M
        NUM_KEYS=3  # N
        add = [w0_rpc.getnewaddress() for _ in range(NUM_KEYS)]
        pub = [w0_rpc.getaddressinfo(a)["pubkey"] for a in add]
        msig = w0_rpc.createmultisig(NUM_KEYS, pub, "bech32")
        madd = msig["address"]
        self.log.info(f"\n    DEBUG madd={madd}")

        tx1 = CTransaction()
        vout = [v["n"] for v in raw_tx0["vout"] if addr0 in v["scriptPubKey"].get("addresses", [])]
        assert len(vout) == 1
        v = vout[0]

        # append the input
        tx1.vin.append(CTxIn(COutPoint(int(txid0, 16), v)))

        # dump the tx to the log
        s = pprint.pformat(w0_rpc.decoderawtransaction(ToHex(tx1)))
        self.log.info(f"\n    DEBUG tx={s}")

        # append the output
        tx1.vout.append(CTxOut(int(8 * COIN), madd))

        # dump the tx to the log
        s = pprint.pformat(w0_rpc.decoderawtransaction(ToHex(tx1)))
        self.log.info(f"\n    DEBUG tx={s}")

        priv1 = w0_rpc.dumpprivkey(addr0)
        tx1 = w0_rpc.signrawtransactionwithkey(ToHex(tx1), [priv1])["hex"]
        txid1 = n1.sendrawtransaction(tx1, True)
        n0.generate(6)
        self.sync_all()

        # dump the transaction
        raw_tx1 = w0_rpc.getrawtransaction(txid1, True)
        s = pprint.pformat(raw_tx1)
        self.log.info(f"\n    DEBUG tx={s}")

        # write the balance to the log
        bal1 = w0_rpc.getbalance()
        self.log.info(f"\n    DEBUG bal1={bal1:f}")

        self.log.info(f"\n    DEBUG END")

if __name__ == '__main__':
    RawTxTest().main()

I came up with a script to pay to, and spend from, a multisig wallet.
Here are the steps I was missing before:
madd1 = msig1["address"]
a1 = w0_rpc.getaddressinfo(madd1)
scriptPubKey1 = a1["scriptPubKey"]
pubkey1 = hex_str_to_bytes(scriptPubKey1)
tx1.vout.append(CTxOut(int(8 * COIN), pubkey1))

Below is the full script.
#!/usr/bin/env python3

import pprint

from test_framework.test_framework import BitcoinTestFramework
from test_framework.messages import CTransaction, CTxIn, CTxOut, COutPoint, ToHex, COIN, sha256
from test_framework.script import CScript, CScriptOp, OP_1, OP_DROP, OP_2, OP_3, OP_HASH160, OP_EQUAL, hash160, OP_TRUE, OP_DUP, OP_EQUALVERIFY, OP_CHECKSIG, OP_CHECKMULTISIG
from test_framework.util import hex_str_to_bytes, bytes_to_hex_str
from test_framework.address import byte_to_base58

class RawTxTest(BitcoinTestFramework):

    def set_test_params(self):
        self.num_nodes = 2
        self.extra_args = [[],["-txindex"]]

    def run_test(self):
        """Main test logic"""

        self.log.info(f"\n    DEBUG START")

        # initialize node variables
        n0 = self.nodes[0] # This node used for initial balance
        n1 = self.nodes[1] # All test wallets created on this node

        # TRANSACTION #0 - pay from node 0 to addr0
        w0 = n1.createwallet(wallet_name="wallet0")
        w0_rpc = n1.get_wallet_rpc('wallet0')
        addr0 = w0_rpc.getnewaddress()
        self.log.info(f"\n    DEBUG addr0={addr0}")

        txid0 = n0.sendtoaddress(addr0, 10.0)
        n0.generate(6)
        self.sync_all()

        # dump the tx to the log
        raw_tx0 = w0_rpc.getrawtransaction(txid0, True)
        s = pprint.pformat(raw_tx0)
        self.log.info(f"\n    DEBUG tx0={s}")

        # write the balance to the log
        bal0 = w0_rpc.getbalance()
        self.log.info(f"\n    DEBUG bal0={bal0:f}")

        # TRANSACTION #1 - Pay from addr0 to multisig
        NUM_SIGS = 2 # M
        NUM_KEYS = 3 # N
        add1 = [w0_rpc.getnewaddress() for _ in range(NUM_KEYS)]
        pub1 = [w0_rpc.getaddressinfo(a)["pubkey"] for a in add1]
        priv1 = [w0_rpc.dumpprivkey(a) for a in add1]
        msig1 = w0_rpc.createmultisig(NUM_KEYS, pub1, "bech32")
        self.log.info(f"\n    DEBUG msig1={msig1}")
        madd1 = msig1["address"]
        redeemScript = msig1["redeemScript"]
        self.log.info(f"\n    DEBUG madd1={madd1}")

        tx1 = CTransaction()
        vout = [v["n"] for v in raw_tx0["vout"] if addr0 in v["scriptPubKey"].get("addresses", [])]
        assert len(vout) == 1
        v = vout[0]
        tx1.vin.append(CTxIn(COutPoint(int(txid0, 16), v)))
        a1 = w0_rpc.getaddressinfo(madd1)
        self.log.info(f"\n    DEBUG a1={a1}")
        scriptPubKey1 = a1["scriptPubKey"]
        pubkey1 = hex_str_to_bytes(scriptPubKey1)
        tx1.vout.append(CTxOut(int(8 * COIN), pubkey1))

        priv0 = w0_rpc.dumpprivkey(addr0)
        tx1 = w0_rpc.signrawtransactionwithkey(ToHex(tx1), [priv0])["hex"]
        txid1 = n0.sendrawtransaction(tx1, True)
        n0.generate(6)
        self.sync_all()

        # dump the tx to the log
        raw_tx1 = w0_rpc.getrawtransaction(txid1, True)
        s = pprint.pformat(raw_tx1)
        self.log.info(f"\n    DEBUG tx1={s}")

        # write the balance to the log
        bal1 = w0_rpc.getbalance()
        self.log.info(f"\n    DEBUG bal1={bal1:f}")

        # TRANSACTION #2 - Pay from multisig to addr2
        addr2 = w0_rpc.getnewaddress()
        self.log.info(f"\n    DEBUG addr2={addr2}")
        a2 = w0_rpc.getaddressinfo(addr2)
        pubkey2 = hex_str_to_bytes(a2['pubkey'])
        p2pk2 = CScript([pubkey2, OP_CHECKSIG])
        tx2 = CTransaction()
        tx2.vin.append(CTxIn(COutPoint(int(txid1, 16), 0)))
        tx2.vout.append(CTxOut(int(7 * COIN), p2pk2))
        scriptPubKey = raw_tx1["vout"][0]["scriptPubKey"]["hex"]
        amount = raw_tx1["vout"][0]["value"]
        prevtxs = [{"txid": txid1, "vout": 0, "scriptPubKey": scriptPubKey, "redeemScript": redeemScript, "amount": amount}]

        tx2 = w0_rpc.signrawtransactionwithkey(ToHex(tx2), priv1, prevtxs)["hex"]
        txid2 = n0.sendrawtransaction(tx2, True)
        n0.generate(6)
        self.sync_all()

        # dump the tx to the log
        raw_tx2 = w0_rpc.getrawtransaction(txid2, True)
        s = pprint.pformat(raw_tx2)
        self.log.info(f"\n    DEBUG tx2={s}")

        # write the balance to the log
        bal2 = w0_rpc.getbalance()
        self.log.info(f"\n    DEBUG bal2={bal2:f}")

        self.log.info(f"\n    DEBUG END")

if __name__ == '__main__':
    RawTxTest().main()


text:
buying bitcoins with debit cards that are not mine
I have a couple of cards that are not mine, I have all of the card holders info and card info. I want to purchase bitcoins and then transfer them to other wallets.
I have a couple of friendswho want to purchase bitcoin, I would buy the bitcoins for everyone, that is why I have all of their card and personal info.
I wanted to use a single wallet (mine) and then transfer the funds to each of my friends preferred wallets.
However I will have no control over where they spend it or how they use it.
Can I personally be traced?

I think you must increase the your question because it lets think of a scam

thanks for the suggestion. I have a couple of friends and myself who want to purchase bitcoin, I would buy the bitcoins for everyone, that is why I have all of their card and personal info.
I wanted to use a single wallet (mine) and then transfer the funds to each of my friends preferred wallets.
However I will have no control over where they spend it or how they use it.

I have a couple of friendswho want to purchase bitcoin, I would buy the bitcoins for everyone, that is why I have all of their card and personal info. I wanted to use a single wallet (mine) and then transfer the funds to each of my friends preferred wallets
  Can I personally be traced?

First off, service recommendations are off-topic on this site, for more info see the help centre.
Next, I'm seeing a lot of red flags here:
You have some friends who want you to buy BTC for them, but WHY would they give you all of their debit card and personal info to do this? Normally if I wanted my friend to buy something for me, I would just give them the cash, or transfer it electronically, or whatever. I would NEVER just give someone full access to my banking information, so that they could buy Bitcoin (or whatever) with it. I find this circumstance extremely suspicious.
This, combined with the worry of being traced, really makes it look like you have some stolen cards that you are attempting to buy BTC with.
I suspect that any legitimate exchange you attempt to interact with will come to a similar conclusion, anti-fraud checks usually involve a cross-reference between the account holder and card holder names and addresses at least. Attempting to use a bunch of cards that aren't yours to make the purchase will likely set off a lot of fraud warnings in any system.
If you are in fact acting legitimately, then it should be no problem having your friends transfer you money, you can make the purchase on an account under your name, and then disseminate the coins to your friends as appropriate. If you are concerned about your friends using the Bitcoin in some illicit way that could be traced back to you, then it may be worth reconsidering your offer to purchase the coins in the first place. For more info on Bitcoin privacy and fungibility, this article is quite comprehensive.

Thanks, I didnt know I could buy them with cash, Ive never bought or had bitcoins in any way, I'm guessing I have alot to learn before trying to buy for myself or for anyone else.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Issue listing regtest wallets
Questions:

Should it be assumed if you're testing with bitcoincore you should always have a separate datadir per version/bin?
Based on weirdness below, (not being able to list regtest wallets), does it seem like my current node is messed up? If yes, what should I do to recover? If no, what am I doing wrong?

Details:
I was running version v25.0 of bitcoincore from bitcoincore.org. I crated some wallets and did some testing. I then compiled from master/source bitcoin/bitcoin and pointed to the same backend state/datadir that v25.0 was using. Later I switched back to running the binary v25.0.
I'm curious if that could be the reason for seeing weird things like this or if it's something else:
$ ./bitcoin-cli createwallet wallet_main
error code: -4
error message:
Wallet file verification failed. Failed to create database path '/btcdata/regtest/wallets/wallet_main'. Database already exists.

$ ./bitcoin-cli listwallets
[
]

$ ./bitcoin-cli createwallet wallet_main33
{
  "name": "wallet_main33"
}

$ ./bitcoin-cli listwallets
[
  "wallet_main33"
]

$ ./bitcoin-cli createwallet wallet_main
error code: -4
error message:
Wallet file verification failed. Failed to create database path '/btcdata/regtest/wallets/wallet_main'. Database already exists.

I was doing some testing on regtest a few days ago and created several wallets including a "wallet_main", it seems there is still some data of this and other regtest wallets, but they don't show up with the listwallets command.
Am I doing something wrong to list the all the wallets? Is it maybe a bug? or did I perhaps mess up my node?
Additional notes:
Inside my /btcdata/regtest/wallets/ directory all my wallets look the same in term user/permissions the folder for the wallet names are 700, drwx------, and the inner wallet.dat's are 600, -rw-------. I indeed have a /btcdata/regtest/wallets/wallet_main dir.

When you're using regtest (or testnet or signet) you're supposed to add -regtest to the bitcoin-cli command, so it can find the right data directory and config file sections. I'm surprised anything works at all without that.

To actually answer your question: you may need to issue loadwallet RPC to load existing wallets, if they're not set to auto loading. If that resolves it, I'll turn this into an answer.

Yes, brain fart.. loadwallet is what I need to do. I also see there is detailed explanation within bitcoin.conf to help people autoload various wallets.
Also,
bitcoin.conf set to run regtest via chain=regtest so I don't need to put -regtest it in every command. (I believe this is how it works?)
Also doing this in bitcoin.conf:
# Options for regtest
[regtest]
# Options for regtest
[regtest]
rpcport=8332
fallbackfee=0.01

You're good to turn your comment into an answer!

The listwallets RPC only lists loaded wallets.
If you want to load an existing wallet, you need the loadwallet RPC first.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Generate regtest blocks with Bitcoin-Qt
I'm prepping for a tabconf workshop and trying to make the simplest uX for workshop attendees.
Using bitcoincore.org for binaries: The Mac dmg doesn't come with a bitcoin-cli like with Linux. It does come with Bitcoin-Qt + console though which I'm guessing is good enough.
Is it possible to generate regtest blocks through the Bitcoin-Qt console GUI?
When I try to run generate I'll see:
has been replaced by the -generate cli option. Refer to -help for more information.
 (code -1)

It's not clear to me if I can utilize -generate through the gui console.

No, you cannot access -generate from the Qt console. It's a feature inside bitcoin-cli only.
Note however that all -generate does is invoke the getnewaddress RPC to get a new address, and then invoke generatetoaddress to generate coins to it. You can do that directly by hand from the RPC console just fine.

thanks for the quick answer

Pieter, followup question. Is there a reason generate isn't available through Bitcoin-Qt or why it was taken out? I'm wondering if this would be an opportunity for me to look into enabling it or doing a feature request.

The history is that the generate RPC was removed, because it was mixing subsystems that were being disentangled (the wallet, and the mining logic). Instead there is just a generatetoaddress which mines to a given address, and a thin wrapper -generate in the CLI that just calls getnewaddress + generatetoaddress, to avoid too much breakage of tests relying on generate. See my updated answer.

Can you convert this to the answer^?


text:
Openssl magic values? Secp256k1 and ECDSA similarities? What makes secp256k1 special?
Part 1:
Running this command seems to be a reliable way to produce a pubkey from a valid private key that for Bitcoin. Is this a correct assumption?
openssl ec -inform DER -text -noout -in <(cat <(echo -n "302e0201010420") <(echo -n "PRIVATE_KEY_HEX_STRING") <(echo -n "a00706052b8104000a") | xxd -r -p) 2>/dev/null | tail -6 | head -5 | sed 's/[ :]//g' | tr -d '\n' && echo

These magic values:
302e0201010420
a00706052b8104000a
Openssl seems to use these values for DER encoding rules, and it doesn't seem to have anything to do with secp256k1 or Bitcoin specifically. Is this a correct assumption?

Part 2:
It doesn't seem like the y^2 = x^3 + 7 formula / secp256k1 is used anywhere explicitly when deriving the pubkey from the private key in the above openssl command.
Are all public keys for Elliptical Cryptography practically derived the same way.. meaning there is a ton of overlap between these ECDSA curves?
Seems like the only thing specific to Bitcoin compared to another ECDSA curve is the maximum upper limit for a private key of FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFE BAAEDCE6 AF48A03B BFD25E8C D0364141.. is this a correct assumption? How is it that I'm able to produce the private and public keys without having to use the formula? It's like Satoshi lied to us and this formula has no meaning!
A bit confused here would love if someone could clear this up for me and the 2 other people who can't sleep at night because of this.

ECDSA parameters for numerous curves is located at https://github.com/gpg/libgcrypt/blob/master/cipher/ecc-curves.c.

Includes curve types, OIDs, curve order, curve basepoints.

These magic values:
302e0201010420
a00706052b8104000a
Openssl seems to use these values for DER encoding rules, and it doesn't seem to have anything to do with secp256k1 or Bitcoin specifically. Is this a correct assumption?

They have nothing to do with Bitcoin, but I believe that those bytes contain a reference to secp256k1 (probably through its OID 1.3.132.0.10).
Specifically, I think the 8104000a part encodes the numbers 132, 0, 10.

Are all public keys for Elliptical Cryptography practically derived the same way.. meaning there is a ton of overlap between these ECDSA curves?

No, there are lots of relevant parameters that go into public key computation:

The field size for its coordinates (2^256 - 2^32 - 977 for secp256k1)
The type of curve (Weierstrass, Edwards, twisted Edwards, ...)
The coefficients of the curve equation (for short Weierstrass curves the equation is y^2 = x^3 + a*b + b; for secp256k1 a=0 and b=7).
The generator point (the point corresponding to private key 1, which has to be chosen by convention)
The order of the generator (which is equal to the number of points on the curve for secp256k1, but can also be a small factor less)

The last item, the order of the generator, defines how many valid private keys there are. It is implied by all other parameters, but computing it is a nontrivial computation, so it is usually precomputed.

You are using the eliptic curve function of openssl and providing a serialized bytestring as an input. DER (Distinguished Encoding Rules) is a restricted variant of BER for producing unequivocal transfer syntax for data structures described by ASN.1.
The ASN1 structure for a privkey looks like this:

# ASN.1 STRUCTURE FOR PRIVATE KEY:
#   30  <-- declares the start of an ASN.1 sequence
#   74  <-- length of following sequence
#   02  <-- declares the start of an integer
#   01  <-- length of integer in bytes (1 byte)
#   01  <-- value of integer (1)
#   04  <-- declares the start of an "octet string"
#   20  <-- length of string to follow (32 bytes)
#           3cd0560f5b27591916c643 ... a738d2e912990dcc573715d2c
#           \--------------------------------------------------/
#            this is the private key
#   a0   <-- declares the start of context-specific tag 0
#   07   <-- length of context-specific tag
#   06   <-- declares the start of an object ID
#   05   <-- length of object ID to follow
#   2b 81 04 00 0a <-- the object ID of the curve secp256k1

See also:
Signing transaction with SSL - private key to PEM

Thanks for spelling out each byte!


text:
Understanding a Wallet Restore
This is my first post on block chain related programming so please excuse my ignorance. I find the terminology in the BIP32 and BIP39 specs hard to understand so I'm trying to confirm my understanding here. I hope someone can clear things up for me. Please correct me if my understanding is incorrect. Here goes.
I bought a Trezor and generated my seed using the Trezor. I understand that this is an implementation of BIP39. My understanding is that if I were not using Trezor, I could generate this key on an offline computer like Tails Linux or Bitkey.IO Live CD. My understanding is that this serves as a key for signing transactions in future. But, this does not by itself create any addresses in the block chain itself.
After creating the key, I generated some addresses. One for Bitcoin, one for Litecoin and so on. Again, merely creating addresses does not affect the block chain in any way. The only thing that is special about these addresses is that the only way to create these addresses is to have have the seed in the first place. Is this understanding correct?
After this point, currency can be sent to the addresses that I have created. This means the block chain network has to accept that the transactions were made, and then the transactions will be synced on to all the other nodes in the network which will eventually be viewable by me when I check the address on my Trezor's software.
This is where my understanding gets a little murky. There is also something called a public key (xpub), and these can be used to view the details of my wallet and its transactions I think. Also, anyone who has the address of my wallet can see transactions in there by querying the block chain.
But, without the seed, nobody can transfer coin out of my addresses. The seed is required to sign the transactions for this. Is this correct?
Now, lets say my Trezor is lost or destroyed and all I have left is the seed. So, I buy a new Trezor and plug it in. I will now want to do a restore. So, my guess is that it will as me for the 24 word phrase. Right?
At this point, what happens? I no longer have any addresses or XPUBs written down, and I will need these to see my balances before I can sign transactions again. So, how does the Trezor software (or other deterministic wallet software) get my xpub and addresses back?
Do they:

Download the entire block chain on to my computer and check each address to see if it was seeded from my original seed?
Hit some API? If so, how? Does it send my seed to the API? Which API? Does Trezor maintain some kind of index of xpubs by seed? I wouldn't think that this would be secure at all, and in fact would completely void the security of the system.
Is there some kind of public key hidden in my seed that can be used to look up indexed addresses? If so, how does that work? Is there a C# library somewhere I can use to get extract this public key from my seed?
Do some other thing that I have not thought of... What?

Edit: I'm currently reading through this. https://www.codeproject.com/Articles/784519/NBitcoin-How-to-scan-the-Blockchain . It makes it sound as though the Scanner class uses BIP37 logic to filter down data from the block chain to get at the addresses that are likely to contain the transactions I need to see. So, what inputs does it need to do this? Can I derive something from my seed which is safe to use over an API?
From Mastering Bitcoin:

Generating a public key Starting with a private key in the form of a
  randomly generated number k, we multiply it by a predetermined point
  on the curve called the generator point G to produce another point
  somewhere else on the curve, which is the corresponding public key K.
  The generator point is specified as part of the  secp256k1 standard
  and is always the same for all keys in bitcoin.
K=k*G68
where k is the private key, G is the generator point, and K is the
  resulting public key, a point on the curve. Since the generator point
  is always the same for all bitcoin users, a private key k multiplied
  with G will always result in the same public key K. The rela‐tionship
  between k and K is fixed, but can only be calculated in one direction,
  from k to K. That’s why a bitcoin address (derived from K) can be
  shared with anyone and does not reveal the user’s private key (k).

So, does this mean that in fact my seed does in fact contain a public key? And that key is the xpub? So, in other words, it is possible to generate an xpub from the seed? Is this correct?
To boil down the question:
Given my seed, how does the Trezor or other wallet retrieve all the associated addresses, xpubs, public keys, transactions, balances and so on that are derived from the seed?

The various private and public keys are all derived from that master seed.

Yes. I understand that they are derived. But, derived is different from being able to look them up. My question is, lets say I derive a public key, and then mint an address. Then, someone sends money to that address, and I lose all my data except for the seed, how do I get the public key back?

@MelbourneDeveloper The same way you got it in the first place -- your wallet program derives it from the seed.

The public key? And when people say "public key", is this the xpub?

@MelbourneDeveloper The xpub is a step before the public key. But, as I said, it's all derived from the seed. The public key can be derived from the xpub, but there are other derivation paths you can take too -- for example, you can derive its corresponding private key and then convert it to a public key.

So, then the key, along with another variable can be used to generate the public key. So, there is only one public key (xpub) per seed?

@MelbourneDeveloper You can derive any number of xpubs. Some wallets use this to support multiple currencies, each of which has multiple accounts. But every step is retraceable.

OK. I think I'm pretty close to understanding this. Is there a tool somewhere that will derive my public key from my seed?

There's one here but you have to know the exact derivation rules your wallet uses.

This is the problem. If it's possible to derive an infinite number, how do I retrieve the ones that have actual used addresses associated with them from the block chain?

I don't know the rules.

@MelbourneDeveloper Your wallet knows its own rules, so using another of the same wallet will work. And if all else fails, you can look up its derivation rules -- pretty much every wallet makes them public. When you load a new wallet, the wallet follows all its derivation paths to search for any funded accounts.

I believe Trezor uses m/44’/0’/0’ for BTC non-SegWit and m/49’/0’/0’ for BTC SegWit. There are some docs here.

OK. So, any wallet generation tool can generate public keys using its own algorithm. So, I'd need to know that algorithm in order to get the public keys back. But, I figure there is some degree of entropy involved in creating the keys, so I can create new keys until the cows come home. That doesn't mean that any of them will actually have funds in them. This is where my understanding is going awry.

@MelbourneDeveloper Sure, but instead of doing that, you would follow the same rules your wallet uses and produce keys/accounts in the same order it produced them. You don't have to search around randomly, you can use exactly the same mechanism you used in the first place either by using the same software or by knowing what method that software used.

So, you are saying that the app (in this case Trezor) did not use entropy to generate the public keys and xpub from my seed. It would have created it in some order. So, for example, it might do something like take my seed, add one, hash it and then give it to me. The second time around, it might take my seed, add two, hash it, and then give it to me. So, this process should be reproducible. Right? So, if Trezor uses m/44’/0’/0’ , then I should be able to grab a tool, give it my seed, and that tool should be able to retrieve an xpub/public key for me right?

@MelbourneDeveloper Correct. No entropy is used. The process is deterministic. (If entropy was used, it could not be derived from the master seed.)

Right, hence the "deterministic" thing. So, I've just got to find the algorithm used, and I can retrieve the addresses etc. without wiping my Trezor.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Programmatically Get Change Address From Blockchain
BIP004 states that a derivation path's 4th digit will be 1 if it is a change address like this:
m / purpose' / coin_type' / account' / change / address_index
https://github.com/bitcoin/bips/blob/master/bip-0044.mediawiki#change
The idea is that once money is spent from a given address, the left over is moved in to a new address which is used as the change address. Wallet software should not leave the left over funds in an address in the old address, they should be moved to the new address. I guess that some wallets follow this, and some don't. But wallets that do not follow this would be breaking the BIP.
So, my question is, from looking at a given address on the blockchain is there a programmatic way to tell where the funds have been moved to?
If you have the public key and the derivation path for the address, you can derive the change address, but if I don't have these things, is there a way to figure it out? Or, at least to take a good guess as to what it might be based on where the funds are moved to?
Use Case (View-Only Wallet):

User adds a loose address to their wallet (without public key)
App scans blockchain for address
App realizes balance has changed and transaction occurred
Funds have moved to another address, but app can't figure this out without prompting user, or displaying an incorrect balance.

The change address is always an additional output in the transaction. I am not aware that there is a way to discriminate between outputs what may be a legitimate change address but, there may be?

Yep. That's what I mean. The change address will be there, but the question is how to discriminate between the two.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Getting statistics on bitcoin >transactions< - breakdown by size
Now that bitcoin is going more mainstream, particularly with the futures listings and so on,
https://www.cnbc.com/quotes/?symbol=%40XBT.1&tab=news
it's quite easy to get statistics on price performance for bitcoin.
However what I want to see is this - quite simply, stats on bitcoin "spends" and that trend over the last weeks, months.
So, ideally you'd really want this broken down as a histogram.
Imagine yourself trying to sell investors on bitcoin: what they want to know is transaction flow:

how many transactions in the last 24 hrs
a graph of that for a few months
breakdown by size .. under a buck's worth, 1-10, over 10

This straightforward information is in my experience the first thing investors look for, when I try to pitch old money on investing in one Bitcoin scheme or another.
How do I get this information? Maybe I'm missing something obvious - it seems to be the first sort of site one would set up to promote bitcoin.
How many transactions in the last 24 hrs ?

What do you mean by "spends"?  If you simply mean # of transactions, there are plenty of places to get these stats, and it's not even too difficult to calculate them on your own with a full node.  If you mean "spending on goods or services" rather than trading...well that's much more difficult to determine.

hi @Jestin "there are plenty of places to get these stats" it's probably one of those things that's easy for you as a highly expert insider.  I want a simple website "bitcoinstats.com".  And many "ordinary" investors want that.  Based on your expert insider knowledge, where is that ?

I agree with Donely that https://blockchain.info/charts is probably what you are looking for.
Here are some other sites that I refer to for blockchain stats.

https://tradeblock.com/bitcoin/historical - Gives you historical transaction volume
https://blockchair.com/ - A block explorer that let's you query more details. You could probably create a historical transactions graph from their data.
http://statoshi.info/dashboard/db/transactions - More technical stats.
https://coin.dance/stats - I don't see a transaction graph here but it's a good site with various statistics.

Yeah what your saying makes a lot of sense. Blockchain info graphs offers allot of charts to show the data that you are referring to. Do you want something in more detail, if so there are probably some api's around that you can use to your hearts desire.

Fantastic information. It seems to be running at about 200,000 transactions a day, https://blockchain.info/charts/n-transactions?timespan=1year ........ BUT .............. I can see no information on breaking that down.  how do we histogram that sucker?


text:
I need help to understand an unconfirmed transaction and 'bitcoin language'
I've had a transaction that is 58 hours home. Trough browsing this website I looke at the lock time of mine 497931 and the current lockheight is 498298. Can anyone help? Will it be done soon? I was worried there was issues with the recipient.
https://blockexplorer.com/tx/67ba33a7756226835f66d07618ffb9fa868bf05ee25b70bf9f0e2371ef1f560f
I used block explorer as someone here said it was most up to date. It also says n/a next to minining, so I'm guessing it just hasnt been mined yet and there is congestion
I need help understanding the bitcoin lanuage on these sites. The other link doesnt really answer my question

for example; does the unconfirmed block mean its still in the 'mempool''.... or does the loctime mean its in a block?

The locktime defines the earliest block the tx is allowed to be confirmed in. We're now past the locktime, so it can be confirmed in a block. Your tx is still in the mempool, it will hang out there until a miner confirms it. No need to worry, the network is just very congested due to high tx volumes.

thank you very much, I've been trying to work it out myself but I dont think I can earn everything in 24hour crash course!

ok so now it says removed/double spends so I've contacted my sender as I didnt do anything to it!


text:
How does MimbleWimble avoid transaction graph leaks?

Now, creating transactions in this manner supports OWAS already. To show this,
  suppose we have two transactions that have a surplus k1*G and k2*G, and the
  attached signatures with these. Then you can combine the lists of inputs and
  outputs of the two transactions, with both k1*G and k2*G to the mix, and
  voilá! is again a valid transaction. From the combination, it is impossible to
  say which outputs or inputs are from which original transaction.

Suppose that you observe these two transactions on the network, prior to them getting into a block. Can you separate them then? If you can get ahold of all of the transactions in a block before they get combined, can you produce a transaction graph like what we currently have, without amounts?

Yes -- although "without amounts or scripts" gives a clearer idea of what the transaction graph will look like: every output is a uniformly random curvepoint. This story is actually quite a bit better than Bitcoin where there are "addresses" which are often public and often reused.
In general, finding a p2p layer that hides the original, unmerged transactions, seems difficult. Initially I had thought it would be possible for "passive merging" where nodes would just aggregate all the transactions that they saw, taking in unmerged transactions and outputting merged ones. This appears to be very fragile: consider three nodes which each have a transaction, say A, B and C. The first node publishes A, then the second and third latch onto it simultaneously, outputting AB and AC respectively, and now only one of B or C can be confirmed. Observe that this happens even without any adversarial behaviour, and happens even if merging is only done by some special set of "masternodes".
The problem ultimately comes down to the fact that with OWAS, conflicting merges (transactions with intersecting input sets where neither is a subset of the other) look identical to conflicting
 transactions (double-spends). So "unifying" AB and AC to get ABC, which is the desired network behaviour, is just as hard as "unifying" a double-spend. Which, to be clear, is "doesn't even make sense" levels of difficult.
It may be possible to have a p2p layer that does interactive merging somehow, which in Mimblewimble could be made much simpler than Coinjoin in Bitcoin, so that transaction creators have control over what other transactions their transaction gets merged with on the wire. Miners would then drop the signatures or whatever was needed to enforce interaction at the p2p layer and just merge what they got. This might be sufficient to stop the creation of accidental conflicts, but you then have to worry about adversarial behaviour.
It would be possible for users to directly communicate transactions to miners; for a given transaction they'd give a different version to every miner so that nobody but the miner who created a
block would be able to tell what the original transactions in the block were. Absent collusion this would erase a lot of the transaction graph from any specific adversary's view, although it would enforce miner centralization which I find very undesirable.

If there's a possibility of having AB and AC conflict, don't miners have an incentive to operate p2p nodes that save every transaction? That way, you can get C before it gets merged, and merge it with AB.

Yes, but only if they can get C before it's merged. With a flood network, absent deliberate creation of conflicts, this is probably a safe bet. But if this is possible then miners (and any surveillance nodes) can separate out all transactions so you haven't gained any privacy by noninteractive merging, just fragility.


text:
Would appreciate some help getting datadir parameter to work in Linux Mint 19.2
I am trying to use Bitcoin Core 0.18.1 on Linux Mint 19.2. I have previously used an older version on Win7 with no problems.
I have moved my old data directory to a new ext4 partition on an internal hdd. Bitcoin is installed on internal ssd. Everything was fine using default /home/admin1/.var/app/org.bitcoincore.bitcoin-qt/data/.
I cannot get the datadir parameter to work (-datadir=/mnt/WDRED/BitcoinData/). It keeps saying the folder does not exist. I tried deleting the default data folder and replacing it with a symlink to the actual folder on hdd but it says the same thing that the symlink folder does not exist.
Somebody might tell me ultimately I have done something stupid or I should have been able to figure it out myself but I have tried what I could from reading up and I am frustrated now. Thank you.
EDIT----
I have now solved the problem hopefully. I installed bitcoin core from within the native software manager in Linux Mint originally. This installed via flatpak. I subsequently read about flatpak and saw that it was running the application in a sandbox (from what I gather?) so I don't think it could see any path on my system that was not within that sandbox.
I uninstalled bitcoin core and reinstalled via snapd. That installation was able to obey the parameter as I expected it to so it is now verifying the existing block files and my problem (it seems atm) is solved...


text:
Integrity of ensuring the correct recipient of a BTC public address?
If a hacker gains access to a website and proceeds to vandalize the website, including changing the public-key of a BTC wallet, this would direct all payments to the hacker. Is this a security risk inherent to the use of BTC or is there something implemented that could prevent this? I am new to BTC/crypto-currency, so I don't fully understand all the details.

Yes. Somehow, you need to tell the client what public key/address to send to. If the attacker has full control over the client, or your server, they can replace your address with whatever they'd like.

Your described situation is a security risk inherent in posting payment destination details for any payment system, not just Bitcoin. If a charity were to post bank wire information or a PayPal account, it's just as vulnerable.
Bitcoin is actually somewhat ahead of the pack here because of the ease of creating new payment destinations. Ideally, each new payment should be received at a new address. This helps the receiver determine when a payment has come through absolutely. The charity would have something that creates a new address for each donor, and then makes the private key available for spending somehow (or just use one of the established payment provider services that will do this for you).
As an aside, having any payment destination information be inside a digitally-signed block of text would greatly increase its authenticity. Bitcoin provides this signature system out of the box, but it is greatly underused!

Except, many payment processing applications provide recourse in the case of a fraudulent transaction. BTC at its core cannot enable this. Can a CA be used for generation of the public-key under a commercial merchant?


text:
What contributes more to bitcoin network? Using Bitcoin-Qt or running a miner?
What would make a better support for bitcoin?
Does it really makes any difference to store blockchain locally instead on a remote server?

Running a miner.

What do you mean contribute to Bitcoin? What is your goal?
If the goal is to help with the propagation of nodes, then setting up a bitcoin-qt on a server is nice for the infrastructure and probably the best for a long-term, good guy greg operation. Become a trusted node, add to the infrastructure.
Coins are mined at a steady pace. There's no threat of them being disrupted or mining operations to discontinue. If anything there's a growing number of miners and the difficulty rate is adjusting to match that.
The best thing you can do for Bitcoin is to continue to use it (marketing), or apply one of your skills to its community (entrepreneurial, development, marketing, design, or other). Make a cool Bitcoin t-shirt (and sell them), buy something using Bitcoin, sell something using Bitcoin. When you encounter something difficult in your use of Bitcoin, make suggestions on how it could be improved. People will listen. Your opinion is valuable and matters.

+1. What bitcoin needs most at this point is people using it for real-world transactions.


text:
Is it normal that uploading is higher than downloading? (traffic network)

Config:
Windows 10, i5, RAM : 16Go
Bitcoin options : Version : 24.0.1
with Serveur RPC on, UPnP on, natpmp on, no proxy, no onion
Does it seems normal to you ?

When relaying transactions and blocks without synchronizing the entire bitcoin blockchain, it is typical to see higher sent traffic compared to received traffic. Just checked my node: in last 25 days since last reboot it gets total traffic 1.03 TB (↑ 1.01 TB sent | ↓ 13.03 GB received).

The amount of incoming and outgoing connections can also impact the traffic flowing to and from your node. A node that does not keep its port (typically 8333) open, meaning that it does not allow peer nodes to connect from outside, will typically experience a minimum of outgoing traffic.

Yes, it seems ok but I still do not know the difference between my two GUIs (laptop and personal computer) : laptop GUI almost always stay below receiving traffic (same parameters) when the one you can see on the pic is still higher (sending traffic is almost egual to receiving traffic x4). Also, the GUI used on my laptop doesn't exceed peer 200 when the one on the pic can go... the current last outgoing connexion peer is around 5000 at the moment (cf start on pic). Someone understood why some GUIs go without any peer limit when seems to have limits ?

After the IBD (Initial Block Download) when the node tries to download the whole blockchain, which is around 500GB, the node does relaying transactions and blocks to all its peers. So it's completely normal that uploading be much higher than downloading. Because when your node receives a block or transaction from one peer, it downloads x bytes, then it will announce it to n peers. So it will upload x * n.
I know the real scenario is much more complex than what is illustrated. But the concept is the same.

Not sure it is about downloading since the blockchain is already fully downloaded.


text:
what is nMinimumChainWork?
I was trying to run my own private bitcoin network, so I downloaded the source and compiled it, then I started two bitcoind instances in two docker contains, and I made sure they did not connect to outside network, used -connect=0 -dns=0 -dnsseed=0, and then -addnode to specify other node's ip and port. When I then tried to run a local miner, the RPC getblocktemplate always return Bitcoin is downloading blocks.... By looking at the source, it looks it won't pass this line:
// in IsInitialBlockDownlaod
if (chainActive.Tip()->nChainWork < nMinimumChainWork) {
     return true;
}

the definition is found in chainparams.cpp:
// The best chain should have at least this much work.
consensus.nMinimumChainWork = uint256S("0x000000000000000000000000000000000000000000f91c579d57cad4bc5278cc");

What does this mean? What is nMinimumChainWork for? And what is the hardcoded hex number, by searching blockchain.info it is not a block hash.

It is a number designed to protect new clients from accepting fake blockchains when they initially are syncing, if an attacker managed to segregate them from the network. As the name suggests, it's the minimum amount of total work a chain must have before the client considers it valid. It is updated to a higher number at every release, based on the total work in the blockchain at the time of that release (the hex format is more compact and consistent with other code than say a decimal number)

Ok, so the number is from an actual bitcoin network's block, that means the software is hard wired to the mainnet, no way to run one's own private network, is that right?

No, its just like an amount of work that is needed, any chain that had that much work would be accepted. BUT it would be basically impossible to create a new chain with that much work, that's the point of PoW, so you're correct that you would need to modify that value if you want to run your own private network (you would be creating an altcoin)


text:
what is the mapBlocksUnknownParent variable?
I was reviewing this pull request. It was an attempt to remove the global variable mapBlocksUnknownParent. I found the definition here. In the CChainState::LoadExternalBlockFile function.
    // Map of disk positions for blocks with unknown parent (only used for reindex)
    static std::multimap<uint256, FlatFilePos> mapBlocksUnknownParent;

Questions:

What is the mapBlocksUnknownParent variable? Where is it useful?

What does LoadExternalBlockFile function do? (by its name I deduce that it reads blk???.dat files and load them into the memory at the start up, but I'm not sure)

When it is possible to have blocks with unknown parents? While AFAIK we always reach blocks through their ancestors, not descendants. So in which situation we have a block with unknown parent?

Why this variable is considered global while it is defined inside of a function scope? Is this because the variable is static?

What is the mapBlocksUnknownParent variable? Where is it useful?

It keeps a map of blocks in the blk*.dat files whose parent is unknown, indexed by the block hash of that parent.

What does LoadExternalBlockFile function do? (by its name I deduce that it reads blk???.dat files and load them into the memory at the start up. But I'm not sure)

It deals with importing blocks from external files. This means not from the network, and not blocks already in our block database. It happens in two contexts:

When a file is specified on the command line using -loadblock=<FILE>; this can be used to import blocks e.g. given to you on portable media (say, a USB stick).
When running with -reindex, in which case the block database is wiped, but the blk*.dat files are kept, and then this "importing" operation is run on these still existing blk*.dat files.

When is it possible to have blocks with unknown parents? While AFAIK we always reach blocks through their ancestors, not descendants. So in which situation we have a block with unknown parent?

When requesting blocks from the network, we only ask for blocks whose parents are known (at least the headers must be known), so indeed, this variable isn't useful in that context. But the blocks are not downloaded in order, and the blk*.dat files contain blocks in the order they were received.
This means that when using -reindex to rebuild the database, some blocks will be encountered out-of-order, meaning blocks will be read whose parent isn't known yet. To deal with this situation, LoadExternalBlockFile keeps track of these blocks without known parent. When the parent is encountered and processed, the children will be processed too.

Why this variable is considered global while it is defined inside of a function scope? Is this because the variable is static?

Exactly. static variables in function scopes are effectively globals that can only be accessed from that function. Only one instance of the variable exists, shared between all invocations of that function.


text:
Anyone else having this 'disappearing' BitcoinCash issue with BitcoinABC wallet?
Today I was in claim BitcoinCash mode, which I completed successfully with my Ledger Nano S and also with my Trezor. Then it was time to for the Bitcoins I held only in my Electrum wallet. I retrieved the private keys and set out to import one into a new BitcoinCash wallet. All bitcoins were first moved off Electrum to a different wallet on a different computer. I then installed the BitcoinABC.org wallet. When that was done I encrypted it immediately. I then used importprivkey to import a private Bitcoin key that had previously held .1 bitcoin.
The blockchain scan started. I let it run and went to lunch. When I got back, the .1 bitcoin cash had been received, but there was also a send transaction 39 minutes later to a BCC address (36uzTBVeWN2pvUu3YX22gCutovGwQg1eeS) I don't know. I didn't make the send transaction and there are no other people that could have and no computer issues, malware or otherwise on my end. Also, I had a list of other private keys open on the same computer and only issue is with the one key I imported to BitcoinABC wallet.
How was BCC sent from my encrypted wallet, where only I (and BitcoinABC wallet after importing) have the private key, if there isn't a vulnerability in the BitcoinABC wallet?
Has anyone else had this issue with BitcoinABC wallet?
Is there a BitcoinABC wallet rep on this site that can address this issue?

That send is from before the fork on 3/31/2017. That means that you sent Bitcoin on 3/31/2017 to 36uzTBVeWN2pvUu3YX22gCutovGwQg1eeS, and because Bitcoin Cash shares the same history as Bitcoin, you will see this send in your wallet.

@AndrewChow Sounds like that should be an answer, not a comment.

@AndrewChow Yes, thanks, that makes total sense now. I should have realized that, as the next transaction in my Electrum is that same send address. Plus, the dates and times of the transactions are the same as previous BTC transactions. Light bulb on.

That send is from before the fork on 3/31/2017. That means that you sent Bitcoin on 3/31/2017 to 36uzTBVeWN2pvUu3YX22gCutovGwQg1eeS, and because Bitcoin Cash shares the same history as Bitcoin, you will see this send in your wallet.


text:
Bitcoin Core 'unable to open file' causes reindexing to fail
I recently struggled to send bitcoin from my Armory wallet (0.96.5) to another account.  Sending bitcoin did not work because the wallet could not synchronize with the blockchain.  Since Armory relies on the connection managed by the bitcoin core client, bitcoin-qt or bitcoind, it became evident that the failure was caused by the bitcoin core clients (0.17.1). The command ./bitcoin-qt --reindex was issued after Armory was shutdown, but failed repeatedly with entries in debug.log having excerpts such as the following immediately before shutdown:
2019-01-15T23:17:07Z Pre-allocating up to position 0x1000000 in rev01636.dat
2019-01-15T23:17:07Z UpdateTip: new best=0000000000000000001e1db68a762512729668e9dd4c6c06cae346a65618189e height=557921 version=0x20c00000 log2_work=90.224133 tx=371925681 date='2019-01-10T12:31:34Z' progress=0.996972 cache=700.2MiB(5254262txo)
2019-01-15T23:17:07Z UpdateTip: new best=00000000000000000014b6677c848def28a7edef1225b0aa865f96199cb1abd8 height=557922 version=0x20000000 log2_work=90.224157 tx=371928196 date='2019-01-10T12:42:46Z' progress=0.996976 cache=700.3MiB(5254902txo)
2019-01-15T23:17:07Z Imported mempool transactions from disk: 1 succeeded, 0 failed, 0 expired, 0 already there
2019-01-15T23:17:09Z Leaving block file 1636: CBlockFileInfo(blocks=144, size=133976289, heights=557764...558003, time=2019-01-09...2019-01-11)
2019-01-15T23:17:09Z Unable to open file /home/**/.bitcoin/blocks/blk01637.dat
2019-01-15T23:17:09Z Unable to open file /home/**/.bitcoin/blocks/blk01637.dat
2019-01-15T23:17:09Z ERROR: WriteBlockToDisk: OpenBlockFile failed
2019-01-15T23:17:09Z *** Failed to write block
2019-01-15T23:37:10Z socket sending timeout: 1201s
2019-01-15T23:37:10Z socket sending timeout: 1201s
2019-01-15T23:37:10Z socket sending timeout: 1201s
2019-01-15T23:37:10Z socket sending timeout: 1201s
2019-01-15T23:37:10Z socket sending timeout: 1201s
2019-01-15T23:37:10Z socket sending timeout: 1201s
2019-01-15T23:37:10Z socket sending timeout: 1201s
2019-01-15T23:37:10Z socket sending timeout: 1201s
2019-01-15T23:37:10Z socket sending timeout: 1201s
2019-01-15T23:37:10Z socket sending timeout: 1201s
2019-01-15T23:39:47Z ERROR: ProcessNewBlock: AcceptBlock FAILED (AcceptBlock: Failed to find position to write new block to disk (code 0))
2019-01-15T23:39:47Z tor: Thread interrupt
2019-01-15T23:39:47Z Shutdown: In progress...
2019-01-15T23:39:47Z addcon thread exit
2019-01-15T23:39:47Z opencon thread exit
2019-01-15T23:39:47Z torcontrol thread exit
2019-01-15T23:39:47Z net thread exit
2019-01-15T23:39:47Z msghand thread exit
2019-01-15T23:39:47Z scheduler thread interrupt
2019-01-15T23:39:47Z Dumped mempool: 3e-06s to copy, 0.000992s to dump
2019-01-15T23:39:47Z Unable to open file /home/**/.bitcoin/blocks/blk01637.dat
2019-01-15T23:39:55Z Unable to open file /home/**/.bitcoin/blocks/blk01637.dat
2019-01-15T23:39:56Z [default wallet] Releasing wallet
2019-01-15T23:39:56Z Shutdown: done

What can be done when a failure to open a file causes reindexing to fail?

Please don't answer the question inside the question field. You can post an answer yourself.

Thank you Pieter for all you have contributed to bitcoin core.  I have edited my question as suggested, and I have given an answer as well.

An exhaustive survey of Google searches relating to the failure to read blknnnn.dat files implicated data corruption on the hard drive or in files such as wallet.dat and/or blknnnn.dat.  In my case, however, I was able to rule out these factors.  I ruled out hard drive failure by using a clone of my virtual machine that ran on a substitute hard drive.  I also ruled out corruption in wallet.dat by reindexing a substituted wallet.dat that had recently worked fine.  In this case the file that could not be opened was blk1637.dat. After about four unsuccessful reindexing attempts, I looked at the file permissions for the defective block and noted that file permissions were only allowed for root user. Prior blocks were permissioned for a non-root user.  The obvious solution to this lack of uniformity in file permissions was to conform the file permission settings of the errant file, but alas making this change and then reindexing the chain did not work, for the same failure to open the block file happened again.  I noted a comment someone made in a Google search that init.cpp in the client software used the umask command to change file permissions for the block files.  Using the sudo command saved me in this situation.  After all, I have everything contained within a virtual machine environment and this security risk was nominal.  In the appropriate folder, I ran sudo ./bitcoin-qt, waited for it to catch up with the syncing, then I started up Armory.  Ordinarily, using sudo in order to force a resort to root privileges is not a good idea.  But in this case it enabled me to transfer my bitcoin to a hardware wallet, freeing me up to run a new node on a virtual machine where the block data files have a consistent 6 0 0 file permissions setting unaltered by an errant system shutdown.


text:
How to convert channel id from c-lightning to lnd?
Sorry for the dumb question, but I'm not very good in lightning network...
So I have two daemons: c-lightning and LND.
Channels from the c-lightning comes like this 505580x1917x1;
Channels from the LND comes like this 556369376388317185;
Is it possible to bring them to a single format? For example c-lightning convert to simple integer like LND in order to find same channels from both daemons?

In the spec meeting in November it was agreed to make the format of the short channel Id part of the protocol. In future implementations should use the clightning format with x as a separator instead of : while colon was very readable the short channel ids that use x as a separator are doubleclickable.

The specification of the short_channel_id describes the format as follows:

The short_channel_id is the unique description of the funding transaction. It is constructed as follows:

the most significant 3 bytes: indicating the block height
the next 3 bytes: indicating the transaction index within the block
the least significant 2 bytes: indicating the output index that pays to the channel.

Hence to convert between the two formats you can use the following python snippets:
def lnd_to_cl_scid(s):
    block = s >> 40
    tx = s >> 16 & 0xFFFFFF
    output = s  & 0xFFFF
    return (block, tx, output)

def cl_to_lnd_scid(s):
    s = [int(i) for i in s.split('x')]
    return (s[0] << 40) | (s[1] << 16) | s[2]

The reason we (c-lightning) use the dotted format is because it allows us to look up the funding TX without having to do a conversion (and it usually is shorter than the u64 representation)

https://github.com/lightningnetwork/lnd/blob/8379bbaa9b259544c2c8591782a78d7384680b2a/lnwire/short_channel_id.go#L25-L43
// NewShortChanIDFromInt returns a new ShortChannelID which is the decoded
// version of the compact channel ID encoded within the uint64. The format of
// the compact channel ID is as follows: 3 bytes for the block height, 3 bytes
// for the transaction index, and 2 bytes for the output index.
func NewShortChanIDFromInt(chanID uint64) ShortChannelID {
    return ShortChannelID{
        BlockHeight: uint32(chanID >> 40),
        TxIndex:     uint32(chanID>>16) & 0xFFFFFF,
        TxPosition:  uint16(chanID),
    }
}

// ToUint64 converts the ShortChannelID into a compact format encoded within a
// uint64 (8 bytes).
func (c ShortChannelID) ToUint64() uint64 {
    // TODO(roasbeef): explicit error on overflow?
    return ((uint64(c.BlockHeight) << 40) | (uint64(c.TxIndex) << 16) |
        (uint64(c.TxPosition)))
}

I have the following function definition in my Bash profile:
scid() {
    local -i scid
    if (( ${#} )) ; then
        for scid ; do
            echo "$((scid >> 40))x$((scid >> 16 & 0xFFFFFF))x$((scid & 0xFFFF))"
        done
    else
        while read -r scid ; do
            scid "${scid}"
        done
    fi
}

This lets me conveniently convert those ugly LND-style short channel IDs into the standards-compliant format, either by passing them to the scid function as arguments or by piping them in, one per line:
$ scid 556369376388317185
506015x904x1
$ echo 556369376388317185 | scid
506015x904x1


text:
How do I determine whether an input or output is segwit?
I'm trying to go through the transactions in a block and determine how many segwit inputs or outputs there are.
I've noticed in getrawtransaction that segwit inputs seem to have a 'txinwitness' field. Is that a good way to classify them?
For outputs I know I can't just look at the 3* address because not all p2sh are segwit. So what's the best way there?
EDIT:
I just found this page about 3* addresses. Difference between a SegWit address and a P2SH address
Am I correct in interpreting this to say that you can't determine whether an output is segwit? Only an input?

I'm trying to go through the transactions in a block and determine how
  many segwit inputs or outputs there are.

BIP141 defines a segwit tx marker and a flag at the beginning of a transaction. So if you decompose such a tx, you would see with the flag, if there is SegWit:
raw tx is: ### raw tx is this:
01000000000102FFF7F7881A8099AFA6940D42D1E7F6362BEC38171EA3EDF433541DB4E4AD969F000000004948
30450221008B9D1DC26BA6A9CB62127B02742FA9D754CD3BEBF337F7A55D114C8E5CDD30BE022040529B194BA3
F9281A99F2B1C0A19C0489BC22EDE944CCF4ECBAB4CC618EF3ED01EEFFFFFFEF51E1B804CC89D182D279655C3A
A89E815B1B309FE287D9B2B55D57B90EC68A0100000000FFFFFFFF02202CB206000000001976A9148280B37DF3
78DB99F66F85C95A783A76AC7A6D5988AC9093510D000000001976A9143BDE42DBEE7E4DBE6A21B2D50CE2F016
7FAA815988AC000247304402203609E17B84F6A7D30C80BFA610B5B4542F32A8A0D5447A12FB1366D7F01CC44A
0220573A954C4518331561406F90300E8F3358F51928D43C212A8CAED02DE67EEBEE0121025476C2E83188368D
A1FF3E292E7ACAFCDB3566BB0AD253F62FC70F07AEEE635711000000

VERSION
 01000000

SEGWIT (BIP141): this is a segwit tx, marker=00
       (BIP141): flag=01

TX_IN COUNT [var_int]: hex=02, decimal=2
 TX_IN[0]
  TX_IN[0]       9F96ADE4B41D5433F4EDA31E1738EC2B36F6E7D1420D94A6AF99801A88F7F7FF
  TX_IN[0] hex=00000000, reversed=00000000, decimal=0
  TX_IN[0] Script Length hex=49, decimal=73
  TX_IN[0] Script Sig 4830450221008B9D1DC26BA6A9CB62127B02742FA9D754CD3BEBF337F7A55D114C8E5CDD30BE022040529B19
4BA3F9281A99F2B1C0A19C0489BC22EDE944CCF4ECBAB4CC618EF3ED01
  TX_IN[0] Sequence (uint32_t)
  EEFFFFFF
 TX_IN[1]
  TX_IN[1] 8AC60EB9575DB5B2D987E29F301B5B819EA83A5C6579D282D189CC04B8E151EF
  TX_IN[1] OutPoint index hex=01000000, reversed=00000001, decimal=1
  TX_IN[1] Script Length hex=00, decimal=0
  TX_IN[1] Sequence FFFFFFFF

TX_OUT COUNT, hex=02, decimal=2
 TX_OUT[0]
 ...

WITNESS TXIN[0] stack elements: hex=00, decimal=0
WITNESS TXIN[1] stack elements: hex=02, decimal=2
 WITNESS data[0]:
  47304402203609E17B84F6A7D30C80BFA610B5B4542F32A8A0D5447A12FB1366D7F01CC44A0220573A954C4518331561406F90300E8F3358F51928D43C212A8CAED02DE67EEBEE01
 WITNESS data[1]:
  21025476C2E83188368DA1FF3E292E7ACAFCDB3566BB0AD253F62FC70F07AEEE6357

 LOCK_TIME
11000000

As this is a tx with two inputs, the first input is non segwit (hence a signature in the V_IN part), the second input is segwit (Script Length = 0), and the witness for second input is added at the end of the tx.

Am I correct in interpreting this to say that you can't determine
  whether an output is segwit? Only an input?

With P2SH, you take an original set of data, and run it through a hash function. As the hash function is not reversible (to today's knowledge), you cannot know, what is inside a P2SH. Usual use cases are for example multisig transactions. But it can be other things as well, and only if the spending transaction is in the blockchain, you can see, what was previously inside. So this adds a certain layer of privacy for the time between funding the tx and spending the tx.

Just to clarify, the segwit tx flag indicates whether there is >= 1 segwit inputs in the tx correct? How would I go about counting the exact number of segwit inputs? You mention a signature in the V_IN part and script length = 0, but are either of those definitive and exhaustive ways? The "txinwitness" field in getrawtransaction seems to indicate a segwit input but I don't know if that's just be a subset of them.

From here: https://bitcoincore.org/en/segwit_wallet_dev/#basic-segregated-witness-support: „The witness is a serialization of all witness data of the transaction. Each txin is associated with a witness field. As a result, there is no indication of number of witness fields, as it is implied by the number of txins“


text:
Web Wallet difference
I have some questions about the listed Web wallets. Can someone please help me understand?

The bitcoin.org page for some web wallets says "Shared control over your money" (e.g. BitGo, BTC.com) while for Coin.Space says "Full control"

What's the difference?
Why does only one Web wallet (Coin.Space) give you full control, why not others?
2.
Why are these wallets not listed on bitcoin.org/en/choose-your-wallet:
StrongCoin,
Blockchain.info,
Jaxx
Why are these not listed on bitcoin.com/choose-your-wallet:
Coin.Space,
Xapo

Some web wallets (e.g. Coin.Space, Coinbase) say "Vulnerable environment" while others do not
(e.g. BitGo, BTC.com). Some like Jaxx list "Secure environment".
What is the difference?
Some says "Remote app", some do not.
What is the difference?

I’d be thankful for your help!

Additional Questions (from comments):

After reading about HD, BIP and understanding (somewhat) how blockchain.info's (bc.i) recovery works, I looked at a few multi-sig wallets briefly: BTC.com, BitGo, GreenAddress (GA). It appears that their recovery would be more troublesome compared to a mnemonic phrase method. Especially, the mention of "nLocktime" by GA was lost on me. I'd love your thoughts/feedback please.
About your comment: "can easily be stolen by the wallet provider". Is it because they know the mnemonic phrase? Is it that they generate it so they know it? If so, would you please recommend a wallet that is not easily stolen, but gives full control of key and easy to recover? I suspect with multi-sig, one would be dependent on pre-signed transactions being sent (by the wallet company), which may not happens if it suddenly disappears. Please correct me if I've completely misunderstood things.

What's the difference? Why does only one Web wallet (Coin.Space) give you full control, why not others?

Click the question mark icon next to each field and it will give you a more detailed explanation of what that qualifier means. For example, "Shared Control over your money" says this:

This wallet requires every transaction to be authorized both by you and this third party. Under normal circumstances, you can regain full control over your bitcoins using your initial backup or pre-signed transactions sent by email.

"Full Control" says this:

This wallet gives you full control over your bitcoins. This means no third party can freeze or lose your funds. You are however still responsible for securing and backing up your wallet.

Why are these wallets not listed on bitcoin.org/en/choose-your-wallet: StrongCoin, Blockchain.info, Jaxx

Bitcoin.org has a listing policy for wallets. Wallets must meet certain criteria for security, usability, and privacy in order to be listed on the website. The full criteria is available here: https://github.com/bitcoin-dot-org/bitcoin.org/blob/master/docs/managing-wallets.md. Wallets which do not meet those criteria are either not added or are de-listed (as in the case with blockchain.info).

Why are these not listed on bitcoin.com/choose-your-wallet: Coin.Space, Xapo

AFAIK bitcoin.com does not have any criteria for a wallet to be listed. Presumably wallets are not listed there because no one has bothered to ask them to list a wallet.

Some web wallets (e.g. Coin.Space, Coinbase) say "Vulnerable environment" while others do not (e.g. BitGo, BTC.com). Some like Jaxx list "Secure environment". What is the difference?

Again, clicking the question mark icon will tell you. For "Vulnerable Environment", it says:

This wallet can be loaded on computers which are vulnerable to malware. Securing your computer, using a strong passphrase, moving most of your funds to cold storage, or enabling two-factor authentication can make it harder to steal your bitcoins.

For "Secure Environment":

This wallet is loaded on mobiles where apps are usually isolated. This provides a good protection against malware, although mobiles are usually easier to steal or lose. Encrypting your mobile and backing up your wallet can reduce that risk.

Some says "Remote app", some do not. What is the difference?

For "Remote App":

This wallet is loaded from a remote location. This means that whenever you use your wallet, you need to trust the developers not to steal or lose your bitcoins in an incident on their site. Using a browser extension or mobile app, if available, can reduce that risk.

It appears that their recovery would be more troublesome compared to a mnemonic phrase method. Especially, the mention of "nLocktime" by GA was lost on me.

It can be more secure because in order for someone to steal your Bitcoin, they would need to compromise more than just your mnemonic, they would also need to compromise whatever two factor authentication you have set up in order to spend your coins.
The nLockTime thing is a special transaction which cannot be confirmed until after a certain date or block height. This transaction is for the possibility that GreenAddress becomes unavailable for some reason but still allows you to access your money.

About your comment: "can easily be stolen by the wallet provider". Is it because they know the mnemonic phrase? Is it that they generate it so they know it?

No, it is because you are using whatever web wallet software they serve to you at the time you visit their website. You have absolutely no control over what software you are actually using (as in what the code is). So that software could be malicious, and there would be nothing you could do about it. For example, suppose blockchain.info has been legally obligated to confiscate your coins. They could simply just serve you and only you a modified version of their software which sends all of your private keys back to them in an unencrypted form and all transactions that that software makes just sends the coins back to blockchain.info. This does not even have to be blockchain.info being malicious; it could be a man-in-the-middle attacker who has hijacked your connection to blockchain.info and served you the bad software.

Thank you!! After several hours of searching/reading, I'm getting "some" understanding. One thing is still not clear, on the bitcoin.org page, why does only coin.space web wallet say "full control over your bitcoins". Is it unique from the rest? (I've read one should only choose a wallet that gives full control).

Because with coin.space you can spend your coins without any involvement from coin.space. The other wallets use multisig where the wallet provider holds 1 signing key for all of your addresses so that you use a 2fa thing for it. It looks like coin.space is like blockchain.info where you have full control over your private keys in theory, but they can easily be stolen by the wallet provider.

Andrew, thanks for being so helpful. I really respect that. After spending several hours today reading stuff and bugging you, I'm starting to get "some" idea about HD and multi-sig concepts :) Thanks for clarifying.

After reading about HD, BIP and understanding (somewhat) how blockchain.info's (bc.i) recovery works, I looked at a few multi-sig wallets briefly: BTC.com, BitGo, GreenAddress (GA). It appears that their recovery would be more troublesome compared to a mnemonic phrase method. Especially, the mention of "nLocktime" by GA was lost on me. I'd love your thoughts/feedback please.

About your comment: "can easily be stolen by the wallet provider". Is it because they know the mnemonic phrase? Is it that they generate it so they know it? If so, would you please recommend a wallet that gives full control of key and is easy to recover? I suspect with multi-sig, one would be dependent on pre-signed transactions being sent (by the wallet company), which may not happens if it suddenly disappears. Please correct me if I've completely misunderstood things.

Thanks for answering my questions. It's very nice of you, it's a pleasure to read your excellent explanations. What you wrote about the online code being changed in regards to bc.i or man-in-the-middle, is the Electrum wallet susceptible to this also? Could you recommend a safe wallet based on your experience? I'm currently looking into Electrum, BitGo, BTC.com wallets .. but I'm not knowledgeable enough. Thanks.

Electrum is not susceptible to this. Nor is any desktop wallet where you download and install the wallet software locally. Desktop wallet software like Electrum also usually have instructions for verifying that the download is legitimate and was non MITM'd

Andrew, thank you. I guess one last question on this. When I get the 12 word backup phrase for a new HD wallet, any way to encrypt it? (Possibly into another 12 word phrase that I can then decrypt back?). I'm asking in case I write my 12 word phrase somewhere and it gets stolen. If it's encrypted by a password (in my mind), it might offer some protection. (I know Strongcoin wallet (which I think is non-HD) does something similar for the private key, where they give you a backup PDF with encrypted version of private key, using openssl aes.)

AFAIK BIP 39 (the standard for mnemonics) does not have anything for encrypting mnemonics. However there is nothing preventing you from encrypting it yourself with some other software. It just means that you can't plug it into a wallet and expect the wallet to be able to know what to do with your encrypted mnemonic.

Can you please suggest an easy way to encrypt the phrase and get it back in the same order (on Windows)? I'm not familiar with encryption much. (I do understand I'll have to decrypt it before using to recover a wallet). Thanks Andrew.

You can use GPG to encrypt it.

Thank you my friend. You are incredibly helpful and an asset to this website, really! Is there a way to send a PM/email to you? Thanks.

Hi Andrew, what are your thoughts on myetherwallet vs. blockchain.info for storing ETH? Thank you.


text:
Largest block (measured in Bitcoins)
What is the block in the blockchain with the most Bitcoins transacted?

I can't say that I'm certain of it, but this is the largest block I've ever seen:
00000000000000fb62bbadc0a9dcda556925b2d0c1ad8634253ac2e83ab8382f aka nr 153509
It has a total output volume of 607,176.29050282 BTC, primarily because of this 550k transaction.


text:
Why is adding s values in half aggregation insecure?
Jonas Nick an Tim Ruffing discussed in a presentation that summing up the s values in signature half aggregation would be insecure (without giving the reason)
They presented a secure solution which basically multiplied each value s_i with a hash that committed to the r-values, messages and public keys of the previous i-1 signatures and messages.
Why is that trick necessary? How does it produce security or in other words why does the aggregation become insecure if that was left out?
(I understand that this fixes the order of tx in a block and if we didn't do that reorganizing transactions would leave us with the same half signature for a different block. However I don't see a direct problem as the order of tx should already be fixed by the merkle root anyway so I assume there is another reason?)

see the explanation by Russell O'Connor here: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-May/014306.html


text:
Why is lnd's channel.db so huge and when will it stop growing
I am running my lnd node since about 60 days. Looking at the file channel.db, I found it grows currently about 500M per day:
btc@xxx:~/.lnd$ ls -al ./data/graph/mainnet/channel.db
-rw------- 1 btc btc 15776518144 Oct 19 14:31 ./data/graph/mainnet/channel.db

-rw------- 1 btc btc 16874332160 Oct 21 12:59 channel.db

What is in that file that needs 16GB and when will it stop growing?
Running lnd 0.13.3 on i9-9900K CPU Ubuntu 20.04, bitcoin-core.
EDIT: New data from Oct 27 added:
It's now
-rw------- 1 btc btc 23279489024 Oct 27 09:20 channel.db

Growth of about 5 GB in 6 days with 2 amts forwarded and 1 new channel opened. Think I stop operating this. Someone with good alternatives to lnd?

This will be an open question, by generic idea the DB should never stop growing

Like vincenzopalazzo commented - it never stops growing by design. Solution is to add
db.bolt.auto-compact=true

to the lnd config. Then the db gets "compacted" (means cleaned-up here) at lnd's startup. But take care for enough disk space. Why it sometimes grows so fast could not be revealed.
The complete discussion is here: https://github.com/lightningnetwork/lnd/issues/5705

lnd 0.15.0 introduced changes that slow down the speed at which your channel.db grows in size, i.e. each update is much smaller. With 0.15.1 you are able to run an optional migration (add a section [db] with db.prune-revocation=true to your lnd.conf), which replaces all old updates by this new format. Using this optional migration you can reduce the total size of your channel.db by a lot, if you have open channels that were active before you updated to lnd 0.15.0.
With lnd 0.15.0 you should make sure not to configure sync-freelist=true, i.e. use sync-freelist=false or remove this from your configuration. If set to true, several users noticed dramatically faster DB file size growth. See https://github.com/lightningnetwork/lnd/issues/6837.
Note that the actual size on disk only reduces if you run compaction, i.e. you need to apply the migration(s) and then restart lnd (again) to compact the DB file(s).
On my node (c-otto.de) I was able to reduce the size of my channel.db file from around 27 GByte to 1 GByte using the changes in 0.15.0 and 0.15.1.
Note that future releases of lnd will also reduce the size of your sphinxreplay.db (see https://github.com/lightningnetwork/lnd/issues/7107) and wtclient.db (see https://github.com/lightningnetwork/lnd/issues/7035) by quite a lot.


text:
Where can I download a snapshot of the Lightning network topology?
I want to study the properties of the Lightning network graph. Is there a way to obtain a reasonably fresh snapshot containing nodes and channels (say, in JSON format)? There are multiple explorers available (e.g. listed in this question), but AFAIK none of them provides the underlying data for download.
(I could just launch my own node and query the data with allnodes / allchannels or a similar command, but that requires synching a non-pruned full node, which I'm trying to avoid at the moment.)
UPD: 1ML does provide some JSON (look for the JSON button on various pages). For example, https://1ml.com/testnet/node?json=true gives the JSON description of the top 50 nodes. I haven't found the way to get this data for all nodes though.
UPD2: a related question: Obtaining Lightning network graph data.

UPDATE (march 2022):
The lnresearch repository has historic snapshots at https://github.com/lnresearch/topology (as pointed out in the other answer by damianodamiano )
if you want a recent snapshot you can use c-lightning and run:
lightning-cli listchannels > your_channel_file.json

of course the later requires you to first connect to some other nodes and sync gossip.
Old reply:
As far as I know currently noone explicitly provides this dataset / snapshot. However romperts recksplorer works with providing this dataset from lnd to the browser and have javascript visualize it. so you could do: wget https://rompert.com/networkgraphv2 but that is a little bit dirty.
The "easiest" way to achieve what you are looking for by doing it properly is probably to extract gossipd from lightningd. As you know c Lightning has this darmonized architecture so there gossipd should be able to run standalone, peer with nodes and download sync the network graph.
I might be interested in helping to run gossipd as a standalone tool. We could also just implement the gossip protocol ourselves as that part of lightning is pretty standalone.
Hope these two suggestions help

"gossipd should be able to run standalone" - does this imply that as of now I'd have to tweak the code and re-build c-lightning to be able to launch gossipd separately?

Yeah. That is exactly what I was trying to say (: in that way you won't need Bitcoind or anything like that but you will just connect to the gossip protocol / peer network

btw you could do wget https://rompert.com/networkgraphv2 | jq '.' | less to download the latest gossip store from rompert's lnd node (I will also edit this to the answer)

Wow, that (using rompert.com) is really helpful, thanks a lot! Hope the node maintainers don't mind occasional downloads of their data for research purposes...

Any update to this question? the rompert.com website seems unreachable. Is there any alternative?

just edited my answer and provided a more stable link to : https://github.com/lnresearch/topology (as pointed out by damianodamiano sorry oversaw it!)

https://ln.alhur.es/ provides a download link for their historical channel database from this link: https://ln.alhur.es/static/channels.db

There is also this repository on Github: https://github.com/lnresearch/topology


text:
lnd stoped routing - unable to update fee rate
I used to have about 5 routings per day. Now, since three days I don't have any routing. In my lnd.log I find lines like this:
grep ERR lnd.log
2021-09-19 22:40:53.187 [ERR] HSWC: ChannelLink(700510:557:1): unable to update fee rate: cannot apply fee_update=1419 sat/kw, new balance=0.00007673 BTC would dip below channel reserve=0.00015 BTC
2021-09-19 23:11:00.999 [ERR] HSWC: ChannelLink(700510:557:1): unable to update fee rate: cannot apply fee_update=1419 sat/kw, new balance=0.00007673 BTC would dip below channel reserve=0.00015 BTC
2021-09-19 23:23:01.180 [ERR] HSWC: ChannelLink(700510:557:1): unable to update fee rate: cannot apply fee_update=1419 sat/kw, new balance=0.00007673 BTC would dip below channel reserve=0.00015 BTC
2021-09-19 23:42:23.031 [ERR] HSWC: ChannelLink(700510:557:1): unable to update fee rate: cannot apply fee_update=1419 sat/kw, new balance=0.00007673 BTC would dip below channel reserve=0.00015 BTC

(changed the channel-id here)
Seems like I have insufficient local funds on one channel. Can this error be the reason that all channels (about 50) to stop routing? And how can I (from anything like lncli listchannels) find what channel that is (700510:557:1)? I find only the uint64 notations for the short channel id.
And, finally, how I can I solve that problem?
Thanks!
P.S. Meanwhile found the channel (700510:557:1) at amboss.space.

The error message is about trying to update the onchain fees in the case of a force close. The onchain fees are renegotiated with your peer from time to time, so that you are safe in case your partner becomes unresponsive. From your log file, it looks like your channel has most of its balance on the remote side and thus you could not afford the higher onchain feerate which your node attempted to set.
With respect to routing I guess (!) you ran into a typical issue of setting your routing fees too low after announcing the channel. In that case people might quickly use your channel for rebalancing and to get themselves the liquidity to where they believe it will be used best in the future. This would explain why you don't route anymore on that channel. Of course you could rebalance which will cost you money or just wait until money flows in the other direction.

Thanks Rene. I understand this, but have two other problems. (1) All other channels (about 48) also have no routings anymore. (2) also, I can't rebalance onto that channel, https://github.com/C-Otto/rebalance-lnd gives me a "Temporary channel failure" on each possible route to fill the outbound liq. on that "broken" channel.

Maybe the best thing I could do is a cooperative close on the empty channel?

Ok, I've restarted LND (after 30 days uptime) and had two routings after 7 minutes to different channels. To me this looks like a bug. I'll post something to a github issue.

If rebalance-lnd only gives "temporary channel failure", there might not be enough (cheap enough) liquidity on the routes you're attempting. You may need to increase the fees you're willing to pay (--fee-factor with a value > 1, maybe even --reckless).


text:
How's the blockchain connected to a Bitcoin Wallet?
I am thinking about buying some bitcoin buy first I want to understand some theoretical aspects of the technology behind it.
The wallet holds my private keys right? But how s me wallet connected to the blockchain?
Thanks,

See What is a good way to concisely explain Bitcoin?

Blockchain is the state of the bitcoin network. It is unanimously accepted as the correct state.
Each bitcoin full node stores its own copy of the blockchain. These copies can rarely differ in couple of last blocks, so bitcoin full nodes can have slightly different view of the same blockchain (only of couple of last blocks though). By adding new blocks on the end of the blockchain (this happens every 10 minutes) any possible differences in blockchain copies resolve (full nodes come to consensus about the state of the blockchain). Bitcoin full nodes are connected in a network and exchange their copies regularly.
The wallet is either implemented as part of the bitcoin full node application or as a separate small application that is then connected via Internet to some bitcoin full node.
Bitcoin wallet must be able to:
a) Generate public address and respective private key.
b) Calculate the balance of public addresses you control. To do this bitcoin wallet must read one small part of the blockchain, the one that holds unspent transactions (and possibly a pool of unconfirmed transactions - the transactions that are broadcasted but not yet added on the blockchain).
c) make a transacion using coins from your public addresses, sign those coins with private keys related to public addresses used, and broadcast this transaction to the bitcoin node that will spread this transaction to other full nodes in bitcoin network. This transaction will eventually end on the blockchain.

So b) requires reading data stored on the blockchain, and c) requires sending new data on the blockchain. Remember that the blockchain is a copy in a bitcoin full node. So the wallet reads from and writes to the blockchain by either communicating internally when being part of the full node application or communicating through Internet with some full node.
Full nodes communicate between themselves to spread and gather blockchain changes.


text:
Can't open bitcoin-qt GUI on Raspberry Pi
I installed the latest bitcoin node on my Raspberry Pi 4 following this guide:
https://www.makeuseof.com/bitcoin-node-raspberry-pi/
Everything went well, only the last step where you need to start the GUI I get an error message.
pi@raspberrypi:~ $ /usr/local/bin/bitcoin-qt
/usr/local/bin/bitcoin-qt: error while loading shared libraries: libfontconfig.so.1: cannot open shared object file: No such file or directory
I googled the issue and found a thread here on the same problem.
How do I run GUI on Raspbian?
I downloaded all dependencies from the github page, but still the same issue.
I went to the file manager and indeed I found no such file or directory (/usr/local/bin/bitcoin-qt).
I do find it somewhere else:

But how do I open it from this place in the terminal?
If I type /home/pi/bitcoin-0.21.1/bin/bitcoin-qt I get the same error.
The following I find in the instructions for installing the node on bitcoin.org:

But libfontconfig1 is installed and is the newest version.

Which file did you download? You may have installed the 64-bit library but are using 32-bit binaries, so there may be compatibility issues.

I downloaded the bitcoin core installer for ARM linux 64-32 bit. There's only one file to download and install. The Raspberry Pi has the 64 bit OS image installed. Can I still be using 32 bit binaries? How can I check and change this?

There is a link to the 64-bit binary under the "ARM linux" icon, it's the thing that says "64 bit". Try using that.

Wow, thank you! I missed that :)

you have to flash the 32 bit option on the rasp pi imager screen. the problem all comes from this. do not flash the 64 bit


text:
bitcoin setup using nodejs
I have to to use bitcoin api using nodejs. but I cannot understand from where I have to start.Search some node module like bitcoin-core and bitcoin.
Can anyone describe the step how can I set up bitcoin in our system and use using nodejs.
Also refer some link where use of bitcoin api request and response parameter are describe in nodejs.
I have to use bitcoin json-rpc api's with node.js but dont know from where i can start?

As it is written, this question doesn't make a lot of sense.  You seem to need to do something with Bitcion, but haven't specified what that is.  Without that information, nobody can help you.

I m using bitcore for stand alone rpc because it has many new indexes and new methods.
See this link https://bitcore.io/guides/bitcoin

First and foremost, install bitcoin-core. I recommend compiling it yourself but there are binaries available for download. You'll want to perform all the steps in this tutorial - follow all steps for the daemon. Do not do the graphical user interface steps. https://bitcoin.org/en/full-node#ubuntu-1610
Run the daemon and use bitcoin-cli to familiarize yourself with the Bitcoin RPC API. https://bitcoin.org/en/full-node#other-linux-distributions
Start the daemon, such that it creates the right folders.
bitcoind -daemon
Stop it after a few seconds.
bitcoin-cli stop
You should see a new folder .bitcoind in your HOME directory. (Or AppData if you're on Windows). You'll need to add this file into it as 'bitcoin.conf':

rpcuser=YOURUSERNAME
rpcpassword=YOURPASSWORD

Replace YOURUSERNAME with a username of your choice, same for the password. Remember these as you'll need it for configuring your NodeJS app.
bitcoind -daemon
Get the balance:
bitcoin-cli -rpcuser=YOURUSERNAME -rpcpassword=YOURPASSWORD getbalance
Pick a nice Bitcoin Core JSON-RPC Node module, like this one:
https://www.npmjs.com/package/node-bitcoin-rpc
^Also has the get balance example.
They use the same commands that you enter in the bitcoin-cli.

Yet another way to proceed is to make your own client.
Here is mine on PHP. Even if you do not know PHP, you can see how easy is to create one from scratch:
JSONRPC:
https://github.com/nmmmnu/BlockchainWalker/blob/master/jsonrpc.php
Selected Bitcoind commands I need:
https://github.com/nmmmnu/BlockchainWalker/blob/master/bitcoindrpc.php
update:
In case you do own client, make sure you handle first transaction
"4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b"
in special code, because bitcoind can not handle it.


text:
Unable to get private key for Blockchain wallet
I am new to BTC and I have a question: How to get private key from Blockchain wallet for a "Watch only" address? I have been trying for days with no positive results. Thanks.

The derivation of a private key from its public key is impossible. A watch-only wallet is explicitly non-spending and cannot be used to spend Bitcoins.

You have to have that private key already. You cannot derive it from the public address. A watch-only address is an imported address.
If you've imported it from a paper wallet, check for the private key that should be printed on the same paper. You can scan the private key and upgrade the watch-only address to one that you can spend from as well.
If you imported some address you do not have the private key of, it will remain a watch-only address for you forever.

How did you add your address to blockchain.info?
Did you use a paper wallet Qrcode to import your address? In this case, you supposed to write your private key and if you haven't you've lost it.
As I remember if you just used blockchai.info to have a wallet, blockchain.info won't give you any private key.


text:
What would be the benefits of 10x the energy spent
Let's suppose the bitcoin network grew to be 10x the size it currently is in terms of the amount of energy spent. No other changes to the protocol were made.
What benefits would this have versus where the network stands today from an end-user perspective? If blocktimes are fixed at 10 minutes and blocksizes stay consistent, then what type of improvements would we see?
Improvements

More Secure from 51% attacks
?

It seems confirmations could be sped up, but the initial confirmation still takes 10 minutes by design. I haven't heard of people complain about confirmation times, and from what I have seen on computing power performing a 51% attack on the bitcoin network as it stands today is not close to being practical. I am interested though what else I am missing.
I am not suggesting the use of POS or something else.

I changed this question from benefits of 10x the nodes to 10x the energy

Bitcoin has already gone through a 10x increase in energy, a few times. Just look at recent history to see what's changed.

Thanks, Greg. Not seeing much. Happy to look into anything specific you have noticed.  Fees? https://bitinfocharts.com/comparison/bitcoin-transactionfees.html  Confirmation times? https://blockchain.info/charts/avg-confirmation-time?timespan=all I am guessing those increases have to do with increases of other sorts. Hence the theoretical. It seems like not much

I guess what I was pointing out was that there's nothing dramatic that has changed due to 10x the amount of energy (the amount of energy can be estimated by the increase in difficulty). The Bitcoin network is designed to scale, and to scale a lot. 10x is not a very big increase in this case.

That makes sense. Appreciate the clarification!

Note first of all that "10x the nodes" and "10x the mining energy" are different things.  A "node" is just a computer that participates in the Bitcoin network by receiving and relaying transactions.  Most nodes don't do any mining, but conversely a single node may support an unlimited amount of mining power.  So there is no real relationship between the number of nodes and the amount of mining power on the network.
Having 10x as many nodes would make it slightly easier for a user to connect to a node to send and receive transactions, and would make the network somewhat more robust against a network split (where not all nodes are able to communicate with each other).  However, it might also make it take slightly longer for a transaction to get to a miner, since more hops would be needed.  There wouldn't be any major positive or negative effects that I know of.
If the network had 10x more mining power, the main benefit would be security against 51% attacks, since an attacker would now need 10x more mining power of his own to mount such an attack.  (Of course, if mining hardware became more efficient in the meantime, this might not actually increase the costs of such an attack versus today - but they would still be 10x higher than without the mining power increase.)
There wouldn't be any long-term improvement in confirmation speed, since as you say the difficulty would adjust to keep the block time at 10 minutes.  (If all the mining power came online at the same time, there would be a temporary increase in confirmation speed for the first few days, since the difficulty adjustment happens only at specified intervals and the maximum adjustment at each such interval is by a factor of 4.)
I am not sure what you mean by "censorship resistant".  A mining power increase wouldn't do anything to help prevent a government from trying to block Bitcoin traffic from the Internet.
There wouldn't be any other major end-user effects that I can think of.  Resistance against 51% attacks is the main one.

Thanks, Nate. I should have worded my question more consistently. I will focus on energy. Let me revise.

Nate Eldredge - I edited the question. My apologies for inconsistency. Is it correct to say that if 10x the energy was being spent then confirmations after the first confirmation would theoretically happen faster? Meaning confirmation 2, 3, 4, etc.

Also, I will accept this after a few days. This answer matches what I have seen. But I called something along the lines of 'unbelievably stupid' on twitter (shock) for asking this question, so I want to see if there is something I am missing just in case.

@DouglasDrouillard: No, the second and further confirmations don't happen faster.  The first confirmation of a transaction happens when it is included in a mined block (call it B1).  The second happens when another block B2 is mined which names B1 as its previous.  The third happens when a block B3 is mined naming B2 as its previous, and so on.  If the block time remains around 10 minutes, then these additional confirmations will still take about 10 minutes each.

Got it. I guessing instead of confirmation I should be saying the speed of spreading the confirmation? Or is even that not helped?

No.  "Spreading the confirmation" would have to do with how quickly the mined blocks B1, B2, B3, ... can be sent over the network after being mined.  This has to do with the bandwidth and latency of the miners' internet connections, and those of the network's relay nodes, but has nothing to do with their mining power.

Thanks, Nate. Here is the tweet. I had guessed some of the other things about confirmations on a hunch, but originally I was where you were at. Outside of security there does not seem to be much there https://twitter.com/Hanakookie1/status/908760259065520128

@DouglasDrouillard: Yep, I agree with you that cheaper energy doesn't help significantly with anything except 51% attack resilience.  It doesn't even help much with that, since presumably the same cheap energy is available to honest miners and attackers alike.  (Unless you assume the attackers already have access to cheaper energy which the honest miners don't.)

Let us continue this discussion in chat.

@NateEldredge, please take a look at my answer and suggest some feedback.

Apart from Nate Eldredge direct consequence of increase in cost for 51% attacks, I would like to point out some indirect consequences.
10x mining power can be supported only by 10x rise in Bitcoin price. Think of bitcoin transaction being backed by 12.5*confirmation blocks(BTC) in USD$(cost for an attacker). It can interpreted as 100 confirmations in the previous system is equivalent to 10 confirmations in the new(10x) system. In other words, I would feel safer with 6 confirmations in the new system than I would with 6 confirmations in the old system.
Confirmations don't speed up, but your confidence in number of confirmations does.

Great thoughts! I had not considered these.