text:
How To Get Master Fingerprint
say I have master private key from dumpwallet command:
extended private masterkey: tprv8ZgxMBicQKsPdcg8UJAGEEQM6zQgyxgXXXXXXXX
from there, I derived child key using bx hd-private command
$ xprv=tprv8ZgxMBicQKsPdcg8UJAGEEQM6zQgyxgXXXXXXXX
$ bx hd-private -d $xprv | bx hd-private -d

which returned tprv8eWEgAtBJiEjAn8o4G6JL1mqoccXXXXXX
I want to derive hardenned addresses within BIP141 m/0'/0'/* (ex. m/0'/0'/0')
using bitcoin-cli deriveaddresses, which I need a correct descriptor.
$ acctKey=tprv8eWEgAtBJiEjAn8o4G6JL1mqoccXXXXXX
$ bitcoin-cli getdescriptorinfo "wpkh([fingerprint/0h]$acctKey/0h/*)"
$ descrip="wpkh(xxxxxxxxxx)"
$ bitcoin-cli deriveaddresses "$descrip"

questions:

How do I get master fingerprint from Master Key?
What is the correct descriptor to use?

Bitcoin Core doesn’t use unhardened keys, so attempting to get a master public key will never work no matter what.

Question 1: How do I get master fingerprint from Master Key?
I'm trying to figure this out as well.
Apparently it's just "the first 32 bits of the Hash160 of the master pubkey".
So:
First_8_characters(RIPEMD160(SHA256($publicKey)))
It seems to have been implemented in python at: https://github.com/darosior/python-bip32/
So using the following python code I think this makes the correct fingerprint:
$ pip3 install bip32
$ python3

import binascii
from bip32 import BIP32, HARDENED_INDEX

bip = BIP32.from_xpub("xpub6CbtS57jivMSuzcvp5YZxp6JhUU8YWup2axi2xkQRVHY8w4otp8YkEvfWBHgE5rA2AJYNHquuRoLFFdWeSi1UgVohcUeM7SkE9c8NftRwRJ")
binascii.hexlify( bip.parent_fingerprint ).decode('ascii')
# gives "814adbb4"

bip = BIP32.from_xpriv("xprv9zrFarSYsBUzhmLYWJN4tYcdhZnEWQ7XCUQzNaouUDLu52xJggNisybDgqJRVe1hWww16SLJM478q5CCquPq5YQ19i2eTmz7HqmpsBnt1SS")
binascii.hexlify( bip.parent_fingerprint ).decode('ascii')
# gives "2984b47b"

But it seems like any 4-byte hex fingerprint works just fine in the bitcoin core wallet, including 00000000. In my testing, all valid (00000000-ffffffff) fingerprints generate the same bitcoin addresses anyway.
So
$ bitcoin-cli getdescriptorinfo "wpkh([00000000/0h]$acctKey/0h/*)"
$ bitcoin-cli deriveaddresses "wpkh([00000000/0h]$acctKey/0h/*)#<checksum here>" "[0,10]"

Sources:
https://github.com/bitcoin/bitcoin/blob/master/doc/descriptors.md#key-origin-identification
"
When interacting with a hardware device, it may be necessary to include the entire path from the master down. BIP174 standardizes this by providing the master key fingerprint (first 32 bit of the Hash160 of the master pubkey), plus all derivation steps. To support constructing these, we permit providing this key origin information inside the descriptor language, even though it does not affect the actual scriptPubKeys it refers to.
Every public key can be prefixed by an 8-character hexadecimal fingerprint plus optional derivation steps (hardened and unhardened) surrounded by brackets, identifying the master and derivation path the key or xpub that follows was derived with.
Note that the fingerprint of the parent only serves as a fast way to detect parent and child nodes in software, and software must be willing to deal with collisions.
"
https://github.com/bitcoin/bitcoin/blob/master/doc/descriptors.md#reference
KEY expressions:
  Optionally, key origin information, consisting of:
    - An open bracket [
    - Exactly 8 hex characters for the fingerprint of the key
      where the derivation starts (see BIP32 for details)
    - Followed by zero or more /NUM or /NUM' path elements to
      indicate unhardened or hardened derivation steps between
      the fingerprint and the key or xpub/xprv root that follows
    - A closing bracket ]

Following the link for BIP32 shows:
https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki#key-identifiers
"
Extended keys can be identified by the Hash160 (RIPEMD160 after SHA256) of the serialized ECDSA public key K, ignoring the chain code. This corresponds exactly to the data used in traditional Bitcoin addresses. It is not advised to represent this data in base58 format though, as it may be interpreted as an address that way (and wallet software is not required to accept payment to the chain key itself).
The first 32 bits of the identifier are called the key fingerprint.
"
4 bytes: the fingerprint of the parent's key (0x00000000 if master key)

Question 2: What is the correct descriptor to use?
I believe for BIP141, it's just m/0
So:
$ bitcoin-cli getdescriptorinfo "wpkh([00000000/0]$acctKey/*)"
$ bitcoin-cli deriveaddresses "wpkh([00000000/0]$acctKey/*)#<checksum here>" "[0,10]"

But since you want the addresses hardened, I believe it would be:
$ bitcoin-cli getdescriptorinfo "wpkh([00000000/0]$acctKey/*h)"
$ bitcoin-cli deriveaddresses "wpkh([00000000/0]$acctKey/*h)#<checksum here>" "[0,10]"

And bitcoin core is very picky for some reason about where the slashes and zeroes and appostrophe/h's go (before or after the account key), and how many, so if you know what address you're trying to generate, maybe try a few different combinations until you get the right address.
To be safe, I would verify my generated addresses with the tool at:
https://iancoleman.io/bip39/


text:
Stratum protocol - problem with implementation in python
For a couple of days I'm trying to write simple script in python to communicate with stratum pool server. I'm using simplified code of NightMiner (https://github.com/ricmoo/nightminer/) and some testnet I've foud ( stratum+tcp://pool.bitcoincloud.net:4008), but I can't get the proper result. Maybe someone more experienced could help me. Below I show each step of my script (step, no python source code).
1. connect to the server, authorize, get the job data
After connecting to the server I send this requests:
b'{"id": 1, "method": "mining.subscribe", "params": []}\n'
b'{"params": ["testuser", "anything"], "id": 2, "method": "mining.authorize"}\n'

and get the following response:
{"id":1,"result":[[["mining.set_difficulty","deadbeefcafebabea200000000000000"],["mining.notify","deadbeefcafebabea200000000000000"]],"40000004",4],"error":null}
{"id":null,"method":"mining.set_difficulty","params":[0.01]}
{"id":null,"method":"mining.notify","params":["4ca","4128bf630f57387b00e25b419d1bb77e667e4036a7d8fee80000015600000000","01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff2503a77614048c8a145e08","122f626974636f696e636c6f75642e6e65742f0000000002062c9c04000000001976a91423e020eacd64acfe093150331d44fdbcc0c7ce0688acc2eb0b00000000001976a91400bf6d61c2a34df5a9ea338fcad188c31bb4a52388ac00000000",[],"20000000","1a02b098","5e148a8c",true]}

2. get the coinbase and merkle root
With this data and extranounce='00000000' I can build the coinbase (in byte form, but I will show as hex in this example):
coinbase=coinb1+extranounce1+extranounce2+coinb2
coinbase=b'01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff2503a77614048c8a145e084000000400000000122f626974636f696e636c6f75642e6e65742f0000000002062c9c04000000001976a91423e020eacd64acfe093150331d44fdbcc0c7ce0688acc2eb0b00000000001976a91400bf6d61c2a34df5a9ea338fcad188c31bb4a52388ac00000000'

Next thing to do is double hash the coinbase with SHA256 algo, which gives as the following bytes:
b'9b9c9ab0b1c92844e4fed3f895f9443fefcda5ae02cc5a8ad4444f9303081a23'

There is no merkle branches (merkle branches array is empty), so hash above is our merkle root.
3. building the header
With merkle root and data received in step 1, I can start building a header:
header= version + prevhash + merkle root + ntime + nbits + nonce

I use the values as follows:
version - received '20000000', after little-endian conversion it is '00000020'
prevhash - received '4128bf630f57387b00e25b419d1bb77e667e4036a7d8fee80000015600000000', after conversion it is '63bf28417b38570f415be2007eb71b9d36407e66e8fed8a75601000000000000' (reverse bytes order every 4 bytes)
merkle root - get '9b9c9ab0b1c92844e4fed3f895f9443fefcda5ae02cc5a8ad4444f9303081a23' in step 2, stays that way
ntime - received '5e148a8c', after conversion it is '8c8a145e'
nbits - received '1a02b098', after conversion it is '98b0021a'

So the header is
b'0000002063bf28417b38570f415be2007eb71b9d36407e66e8fed8a756010000000000009b9c9ab0b1c92844e4fed3f895f9443fefcda5ae02cc5a8ad4444f9303081a238c8a145e98b0021a' + nonce

4. search nonce
the difficulty is 0.01, so my target is
00000063ff9c0000000000000000000000000000000000000000000000000000

after some calculations, I find this 4 bytes to use it as a nonce:
b'd0cf1040'

so my complete header is now
b'0000002063bf28417b38570f415be2007eb71b9d36407e66e8fed8a756010000000000009b9c9ab0b1c92844e4fed3f895f9443fefcda5ae02cc5a8ad4444f9303081a238c8a145e98b0021ad0cf1040'

after double-hashing it with SHA256 algo and reverse bytes order I get this hash:
000000032f201567fc4d02fa468bf2dc6a5c21ae70098c124e99ee7d2520ca26

which is below my target, so 'd0cf1040' should be the correct nonce.
5. submit nounce
last thing to do is submitting the results, so I send this request to the server:
b'{"id":2, "method":"mining.submit","params":["testuser","4ca","00000000","5e148a8c","d0cf1040"]}\n'

and this is response I get:
{"id":2,"result":false,"error":null}

I have no idea why I get result=false. I can only suppose there is a mistake in step 3 (building header) and this mistake is related to byte-order of some values, but I can't find this mistake (I have tried many variants of byte-order, but never get the response result=true).
I will be much obliged if anyone can check this calculations and give me some tips.

Nonce, not nounce.

There is (at least) one mistake:

There is no merkle branches (merkle branches array is empty), so hash
  above is our merkle root

The merkle array for that specific job is not empty. It contains the following branches:
["12ae2555f80b865bac89fa3aeb4df41b730422670fb80d1a0536276b481fab53","e1f7983af4815cb2116f16c6af916f0a139ad5a7588fd029572c6e403f9fed5d","6d5861e558d057993fdae3007ff5b49807d6f43b218e1ef85ed81223747278f1","847e7a24328302a2a8dbdad177d44d5a20916a0ba36edf2dc667c01a09da298c"]

Then the proper way of calculating merkle root would be required.

Thank you. You were correct, the merkle array was not empty. I have run the script again, get another job (this time, merkle array was empty) and get the same result, I mean result=false. I have edited my first post and use this new values (with empty merkle array)

how about byte-swap the nonce you found for submission?

I check the original sourcecode of NightMiner, you're right: nonce should be send to server as reversed byte-order. I changed my script and now it submits the nonce in reversed byte-order, but it didn't help.

Ok, I get it done.
There were two kind of errors I have found (not to mention the byte-order during submitting result - thank you @Charlie for that tip):
1. the result (false) I got as the last response from the pool server was not response to 'mining.submit' method
In my example communication log (question) I paste (among others) those requests:
b'{"params": ["testuser", "anything"], "id": 2, "method": "mining.authorize"}\n'

and
b'{"id":2, "method":"mining.submit","params":["testuser","4ca","00000000","5e148a8c","d0cf1040"]}\n'

You can see I have used the same "id" value (2) for both of this requests. So when I got the response:
{"id":2,"result":false,"error":null}

I have assumed this is "false" response to my 'mining.submit' request. But later I use individual id for each request and find out that this "false" result is response to 'mining.authorize' method. So my first problem was the authorization problem, not the nonce calculation/submitting.
I don't have any idea why I get response to 'mining.authorize' request after some other request, but this is some improvement ;)
2. wrong data submitting in 'mining.authorize' method
In my example communication log (question) I paste this request:
b'{"params": ["testuser", "anything"], "id": 2, "method": "mining.authorize"}\n'

But according to https://pool.bitcoincloud.net/ I should use as username '  your address (testnet address on test pools)'. So I changed this request to:
b'{"params": ["2N16oE62ZjAPup985dFBQYAuy5zpDraH7Hk", "anything"], "id": 2, "method": "mining.authorize"}\n'

Now it works, after submitting the result ('mining.submit' method) i got this response:
b'{"id":3,"result":true,"error":null}\n'

The only thing I need to figure out is why I get the response to 'mining.authorize' request as a response for some other request (the next one after 'mining.authorize' request). Any help would be appreciated.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Cannot import python-bitcoinrpc
I am unable to import the package bitcoinrpc.
I have installed it in python2 and python3 using python3 -m pip install python-bitcoinrpc/python -m pip install python-bitcoinrpc AND by cloning the repo and editing the python environment in setup.py to my python2 and 3 path. Which returns Requirement already satisfied: python-bitcoinrpc in /home/a/.local/lib/python3.6/site-packages (1.0)
Even after that I get the following errors:
Python2: ImportError: No module named authproxy
Python3: ModuleNotFoundError: No module named 'bitcoinrpc.authproxy'; 'bitcoinrpc' is not a package
Following is my code:
from bitcoinrpc.authproxy import AuthServiceProxy, JSONRPCException

# rpc_user and rpc_password are set in the bitcoin.conf file
rpc_user = "xxx"
rpc_password = "xxx"
rpc_connection = AuthServiceProxy("http://%s:%s@127.0.0.1:8332"%(rpc_user, rpc_password))

print(rpc_connection.get_info())

I have searched as much as I could but, cannot find anyting except for this bitcoin stackexchange question without an answer.
ENV: OS: Mint 19.1 with Anaconda where which python(3) return /usr/bin/python(3)
Is the setup broken or am I missing something?
Any help is appreciated. Thank you!

I had named my script bitcoinrpc.py and had the import bitcoinrpc. Naturally it couldn't find the package. Renamed the script and it works.
Lesson: DO NOT have a file of the same name as the package you are trying to import.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How do I customize Bitcoin Core's mempool policy so that low value transactions (ordinals, BRC-20) are not propagated by my mempool?
How do I customize Bitcoin Core's mempool policy so that low value transactions (ordinals, BRC-20) are not propagated my mempool? I compile my full-node from source, so I am interested in editing the code before compiling the clients as well as (if possible) configuring the standard client for the sake of implementing such policy.
Some time ago I've heard on YouTube that someone already implemented it by modifying Bitcoin Core. I've not been able to find a repo of such project anywhere. If such project already exists - I would like to know what Bitcoin Core's file were modified and in what way.
P.S. Despite being well versed in other languages, I am only a beginner coder in C and have no experience in C++. Explaining like to a newbie would be appreciated.

The simplest patch for this is filter-ordinals.patch (aka Ordisrespector):
https://gist.github.com/luke-jr/4c022839584020444915c84bdd825831
--- a/src/script/interpreter.cpp
+++ b/src/script/interpreter.cpp
@@ -504,6 +504,14 @@ bool EvalScript(std::vector<std::vector<unsigned char> >& stack, const CScript&
                     return set_error(serror, SCRIPT_ERR_MINIMALDATA);
                 }
                 stack.push_back(vchPushValue);
+                if ((flags & SCRIPT_VERIFY_DISCOURAGE_UPGRADABLE_NOPS) && opcode == OP_FALSE) {
+                    auto pc_tmp = pc;
+                    opcodetype next_opcode;
+                    valtype dummy_data;
+                    if (script.GetOp(pc_tmp, next_opcode, dummy_data) && next_opcode == OP_IF) {
+                        return set_error(serror, SCRIPT_ERR_DISCOURAGE_UPGRADABLE_NOPS);
+                    }
+                }
             } else if (fExec || (OP_IF <= opcode && opcode <= OP_ENDIF))
             switch (opcode)
             {

Thank you. It succeds with an offset: Hunk #1 succeeded at 479 (offset -25 lines).  I will follow your repo as spammers change their protocol and as Bitcoin Core's new releases may update interpreter.cpp.

If you want to keep running bitcoin core I think the easiest thing to do would be to add permitbaremultisig=0 and datacarrier=0 to your bitcoin.conf file althought I don't know how effective that would be. You could also look for one of the spam filter PRs that have been made for bitcoin core such as this one. If you want more strict spam filtering, then you could always run Bitcoin Knots which is a fork of Bitcoin core run by Luke Dashjr.

Ordinals bypass the datacarrier policy due to unfixed bugs in Bitcoin Core


text:
Which binaries do I need to compile from Bitcoin Core's code for running my non-routing node as an end user?
I need Bitcoin Core only for running my non-routing core-lightning node (private channels only) as an end user (not as a developer)? The lightning node is to be run on the same machine as bitcoin core.
1. Do I need to compile bench_bitcoin, bitcoin-cli, bitcoin-util, bitcoin-tx and test_bitcoin for that purpose?
2. I customized the compilation with ./configure --disable-wallet  --disable-tests --disable-bench. Are there any other flags that I could pass to configure in order to save on CPU and storage and still maintain my lightning node functional?
3. Where can I read more on each of the Bitcoin Core's binary such as bitcoin-util
bench_bitcoin, test_bitcoin, and bitcoin-cli
The research I've done so far include config -hand https://en.bitcoin.it/wiki/Bitcoin_Core_0.11_(ch_1):_Overview .

Core Lightning directly interacts only with bitcoin-cli, which in turn interacts with bitcoind. Those are the only two binaries you will need, indipendently of whether you compiled the binaries yourself or downloaded a release from https://bitcoincore.org/ (always double-check that you are downloading from the right place, and check the gitian signatures when possible).
As for your customizations, I would likely not compile with --disable-wallet as that may have some side-effects on propagation of your own transactions, and mempool, however I cannot find any confirmation of this on the docs so it might also be safe.
And finally, the binaries:

bitcoin-util is a simple utility to test out proof-of-work (https://www.mankier.com/1/bitcoin-util). This likely is not used for anything outside development, so feel free to ignore this one.
bench_bitcoin is a benchmarking tool, used during development, to ensure there are no performance regressions from version to version.
test_bitcoin is the testsuite for integration and unit tests. It is also used during development, and not much use outside.
bitcoin-cli is the bitcoin command line interface, and is the way that most things talk to bitcoind when running, including CLN.

Thanks, I ran ./configure --disable-tests --disable-bench --enable-util-cli --with-utils=no and it installed just bitcoind and bitcoin-cli. I just addded the mention of bitcoin-tx to my question - what is it used for?

I didn't know myself, but looking at the man-page (https://man.archlinux.org/man/bitcoin-tx.1.en) it sounds like a utility to create or update transactions in their hex format before submitting them to the network. It's not needed by any tool I'm aware of.


text:
Can you include a signature in a bitcoin payment?
I know you can use the bitcoin client to sign messages (for which you own the private key) and verify messages from other public addresses. But can you actually embed a signed message into a transaction so its on the blockchain for anyone to verify? When I say "signed message" i'm referring to an arbitrary message like in the attached example.

No. This is not possible, although, technically it could be possible by adding n child transaction each formed as a tx out of you original transaction. The child transaction then could each have a OP_RETURN (data only) output. With a OP_RETURN output you can store up to 80 bytes (mind 5286). A DER signature will be around 72 Bytes (71.5 in avg)
... so not much to left besides the signature (~8 bytes per OP_RETURN tx as tx out of the original tx). So maybe use OP_RETURN txout 0 to n-1 for the text/data and n for the signature. Obviously this will increase your transaction fee a little because you need at least 2 additional transactions.
Conclusion:
Not possible unless your are happy to experiment with forging your own custom transactions.

Ya looks like OP_RETURN  is the only way to add metadata to a transaction at this point. That's too bad because most wallets don't allow that property to be set, so it would require custom development.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
litecoin transaction time without transaction fee
I've sent 927 ltc from my wallet to btc-e but didn't left enough in my wallet to cover the 0.01 transaction fee. My litecoin wallet is showing 'this payment has not been transmitted yet' and it's been a couple of days now. How long is it going to take? Is there a way to cancel the transaction?

did it confirm by now?

same problem here. not confirmed since 5 jan

You may find the solutions offered here to be of help:
http://bitcoin.stackexchange.com/questions/23012/btc-stuck-with-zero-confirmations

see also:
https://code.google.com/p/bitcoin-wallet/wiki/UnstickLowFeeTransactions

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
I can't access my wallet
i put in my identifier to log in and it stays stuck on the log in page.
It says "changing wallet identifier" and doesn't let me log in.  After a while, it says that they disabled login for too many attempts.
I have not been able to log on for the past 2-3 days.
please help.

Welcome to Bitcoin.SE!  Your question is very vague.  It sounds like you're using an online wallet site, but you don't say which one.  Please edit and rewrite your question so it's comprehensible to someone who isn't looking over your shoulder; otherwise it will probably be closed.

Did you understand what's going on? It's doing the same to me now.

This question needs more information, such as what service this is talking about.

That would be best left to contacting the support of the online wallet.  Most online wallets use that kind of login and usually have a link for support.


text:
How bitcoin network appends new block in its Block-chain
Well, According to my research, After solving the cryptographic hash, when the multiple nodes broadcast the block at the same time as a suggestion to be the next block in the chain. Bitcoin network adds all those blocks to the chain.
Suppose three miners had successfully added blocks (a group of transactions). Means, now there are 3 chains. So, Now the next block will append which Block-chain 1st, 2nd or third? Please explain, How future blocks will append the Block-chain. Moreover, How the network will handle this situation?

Please let me know if my concept is wrong. I will appreciate your guidance. Thanks

Really recommend reading the bitcoin whitepaper, also for your other question. Is pretty well written and not very long.

@Jannes Perfect! Thanks

The math makes it rare for boxes to be solved at the same time. However, Yes it's true. If more than one miner solves the cryptographic hash at the same time can add a block (a group of transactions) which as a result, leads to multiple branches. In this case, you simply build on the top of the first one you received.
Solution The tie gets broken when someone solved the next block. Because The general rule is that you always immediately switch to the longest branch available. The end result is that the blockchain quickly stabilizes.
So, the transactions of an ignored block of a small chain, go back to the unconfirmed transaction pool and be included in a later block.


text:
listreceivedbyaddress does not return any information about change addresses, is there any alternative?
listreceivedbyaddress does not return info about change addresses (m/84'/1'/0'/1/*)
, only about receiving addresses (m/84'/1'/0'/0/*). I've found out that getreceivedbyaddress "address" shows info about both change and receive addresses.
The issue is that the output of listreceivedbyaddress is more complete than that of getreceivedbyaddress, with an array of txids of all receiving addresses of the wallet whereas getreceivedbyaddress shows only the received amount of the specified by the user address.
How might one get the received transactions of all change and receiving addresses?


text:
Why sending bitcoin and one addresss shows (Unspend) and one address shows (Spent)?
I sent some btc to a btc address, and each time I send btc, it splits into two addresses, only one of them is the real destination address. The recipient said he has not received the btc yet. I checked blockchain info, it shows the destination address (spent) but the other address as (unspent). This is normal, right? When the recipient's wallet address shows spent in the blochain explorer - does it mean that the btc has already appeared in his wallet?
Thanks in advance!

The second address your transaction sent to is called a ‘change address’. If you search that term on this site you can learn about what a change address is.
Here is a breakdown of the different status’ blockchain.info reports on their block explorer website:

if it says ‘unconfirmed’, that means transaction is not yet confirmed (not yet included in a block)
if it says ‘unspent’, that means the transaction went through, and the funds are still sitting in that address
If it says ‘spent’, that means the transaction went through, and the recipient has already moved the funds to another new address

Thank you. But it's been shown (Unspent) for one address for more then 3 days. Why causes the fund to sit in that address and not move into the actual recipient wallet autmatically? Previous send to the same address went through fine, but the 2nd time sending to the same address, the fund just sits there, not automatically Spent. Please advise what recipient can do in this case.

@JakeJohnson if I understand correctly, your tx has two outputs: one to the recipient, and the other is your change address. Your wallet controls the change address. So if you have not sent a subsequent tx that spent the coins in the change address, then of course they would remain unspent. There is no ‘automatically spent’. If you re read my answer you will see that the ‘spent/unspent’ tag that blockchain.info reports has nothing to do with whether or not your tx has confirmed.

Do you have any solution to turn "Unspent" status of the change address into "Spent?" It was sent from a blockchain wallet, and their support said after so many confirmations, it's irreversible, they can't do anything to help either. It's up to the recipient. But the recipient doesn't know what to do to claim the fund sitting on the network. Sending from Blockchain was always fast, but this time, it's stuck. Blockchain Wallet Support said it's not sender's issue and there's nothing they can do to help. The recipient doesn't know what to do. Please advise. Thanks a lot!

Plz post the tx ID or address here so i can look. The unspent change address should be controlled by your blockchain.info wallet. To spend a utxo, the wallet that controls that address’ private keys needs to send a new tx using that utxo.

chytrik, thank you very much for offering help. Could you provide an email address, I'd be happy to send you the TXID to your email privately.

@JakeJohnson There is nothing to concern with posting a txid, it is already public information unless you do not want it to be associated with your user ID here.


text:
What is the correct syntax when specifying the utxos to be used in c-lightning's `withdraw`?
I tried to withdraw from one utxo with the following command:
lightning-cli -k withdraw destination=bc1********** satoshi=******* feerate=normal utxos='["********":"0"]'

and I got the following output:
lightning-cli: Some parameters are malformed, cannot create a valid JSON-RPC request: { "jsonrpc" : "2.0", "method" : "withdraw", "id" : "lightning-cli-17786", "params" :{ "destination" : "bc*******", "satoshi" : ******, "feerate" : "normal", "utxos" : ["******":"0"]} }

Similar notation of square brackets inside of single quotes worked in fundchannel. What is the correct notation when it comes to specifying the utxos with withdraw?
P.S. I used * instead of the characters of the actual transaction in order to avoid doxxing myself.

I'm sorry if this can be a little bit confusing, but the correct syntax is the following one:
clightning -k withdraw destination=** satoshi=** feerate=normal utxos='[{"********":"0"}]'

In particular, a transaction is an object, so the utxos is a list of transactions objects.


text:
Why is it bad to download the full chain from a third party with Bitcoin Core?
I know Bitcoin Core downloads and processes the full chain much quicker now than it used to but what exactly is so bad about downloading a compressed blocks, chainstate and database folder from a third party?
In the event of any malicious changes to the chainstate, wouldn't Bitcoin Core at its startup reject the new blocks anyway?

No, the chainstate is the nodes view of correctness. It would take many hours or days to re-verify the whole chain on most hardware, so an assumption is made that the state stored at the previous shutdown is most likely correct. There's some sanity checking of the previous couple of blocks to make sure there is no gross accidental corruption, but malicious modifications of the chainstate before that time will not be caught.
A maliciously modified chainstate can contain fake outputs that do not exist in the canonical one, meaning you could be duped into accepting a false transaction paying money to you that nobody else would consider valid. If any significant portion of the network was downloading unverified chainstate databases it would be a considerable issue.

The challenge is essentially who you are downloading from. If you already have a full node that has completed the IBD (initial block download) verification process you may choose to just transfer that over that blockchain to a second full node you are setting up to save on completing the process a second time.
However, if you download it from a malicious party they could send you an entirely independent blockchain with orders of magnitude less proof of work. If they also controlled some of the peers you are connected to they could continue to feed you new blocks built on top of this independent blockchain. Unless you did sanity checks against a third party block explorer or verified the accumulated proof of work you may struggle to work out what has happened.
The Bitcoin protocol is designed to be trust minimized. If you relax those assumptions and start trusting third parties you do start to open up "security holes" in Nick Szabo's parlance.

If it's actually a less-work chain, it'll quickly be overtaken as soon as you connect to real nodes. The concern is that a malicious chainstate can just give an arbitrary state of the system - including giving themselves more money - with the same chain.

If you had a million blocks on the malicious party's "fake" blockchain (with orders of magnitude less accumulated proof of work) and then you learnt about a block built on top of the "true" blockchain the software would detect this, ditch the entire "fake" blockchain and start verifying from genesis on the "true" blockchain?

Yes, by definition. That's proof of work, you have to accept the most-work chain among all valid chains you know of.

There are effective checkpoints in Core? And these aren't hardcoded? Is there a number of blocks (e.g. a million) where after that point you wouldn't verify from genesis no matter how significant the proof of work on a block you received from a honest peer?

Yeah, this only works back to the last checkpoint you've passed. Beyond that, no, it should download and verify headers from any peer who claims to have a better chain. Note however that the difficulty level at the time of the last checkpoint is nontrivial, so constructing a headers chain that branches off at that point will have a substantial cost per block.


text:
How many bitcoin transactions can be verified per second on commodity hardware in 2020?
I am trying to figure out how many Bitcoin transactions could be verified per second.
Am I correct that signature verification is more expensive than computing Merkletrees and hashes and therefore constitutes the bottleneck of the verification process?
Having read this paper it seems to me that tens of thousands of signatures per second could be verified. Have I misunderstood the results?
As I am only interested in signature verification, let parameters like internet bandwidth and storage requirements to handle the stream of incoming data be outside of the scope of this question.

I suspect the answer would depend on whether you are using an Attiny85 or Summit at Oak Ridge.

changed the title of the question to include commodity hardware in the year 2020

You can't measure verification speed with tx/sec because it would depend on the transaction a lot. One tx could be verified in nanoseconds while another could take a couple of minutes to verify.

I could edit the question once more and just make assumption of P2PKH. I understand that any answer would be an estimate anyway

As others have pointed out, this largely depends on what kind of transaction you're verifying. More inputs and more signatures means longer verification, and for pre-SegWit inputs the computational effort actually grows quadratically (see this answer).
However, we can get a helpful estimate by measuring how long it takes to verify the entire Bitcoin blockchain, and divide the total number of transactions by that. The first part has already been done for us by Jameson Lopp in his article Bitcoin Core Performance Evolution.
For his testing, Lopp changes a few configuration parameters:

assumevalid=0: This removes the performance optimization of skipping verifications of old signatures.
disablewallet=1: This avoids any wallet related data processing.
He also uses the connect parameter to connect to a single local peer, which removes the unpredictability of public nodes.

As a result, his node is essentially bottlenecked by how fast it can verify transactions, so that should make it a good estimate on transaction verification speed.
These are the hardware specifications of his benchmark PC from 2018:

Core i7 8700 3.2GHz 6 core CPU
32 GB DDR4-2666
Samsung 960 EVO 1TB M.2 SSD

Bitcoin Core 23.0, the latest version as of now, took 387 minutes to reach block 720,000. The total number of transactions up to that block is 704,935,293. On average, then, it verified about 30,359 transactions per second.


text:
First bitcoins and confused
I have bought bitcoins on https://www.bitcoin.de/en/btc_address. However, I  don't know how to transfer them to a bitcoin wallet or why I need a bitcoin wallet
Thank you

Welcome to Stackexchange! Hopefully, I succeeded in answering your question, otherwise additional comments from you could help me to better understand what your needs are. If your question was answered you can accept the answer best suited to your needs, signaling that you are satisfied. If you are interested in finding out more about how Stackexchange works, you can read the about page linked in the header.

With a few minutes of research on your own should be able to figure this out. I'd start with the FAQ on the site you bought the coins on. I really wonder why you bought something you don't know what to do with. Sounds like throwing away money to me.

Currently the Bitcoins that you bought are being kept by Bitcoin.de for you.
Assuming, that you bought the Bitcoins in order to use them as payment medium or investment, it seems likely that you would want to move them to your control.
There are two different kinds of wallets: Wallets offered by online services, which you access much like online banking, and personal wallets that you directly save on your computer or smartphone. In general wallets are storage tools for keeping funds, much like their physical counterparts.
In any case it is important to create a backup of your wallet, as Bitcoins are lost when you lose access to the wallet.
There are many different services that offer online wallets and several different programs for your personal computer that allow you to create a wallet for yourself, for example the original Bitcoin Client.
Once you have acquired a wallet, you should be able to determine an address to which you can withdraw the funds that Bitcoin.de currently keeps for you. Coindesk has a section in it's getting started guide about the storage of Bitcoin.
However, –I hope this doesn't sound too blunt– I would suggest that you find out a bit more about how Bitcoin works and what you want to do with them before you start experimenting, as it would be tragic if you accidentally lost your bitcoins.
You can find a Getting started Guide here.


text:
Online wallet for betting / customer profile?
I am looking for a system that provides users with the option to deposit and withdraw BTC to a generated address.
Basically something that works like https://primedice.com/play and https://dicecoin.io/
On top of this, perhaps there is an all-in-one bitcoin dice system?
Thanks!

I am looking for a system that provides users with the option to deposit and withdraw BTC to a generated address. - That's pretty much what any wallet software does. On top of this, perhaps there is an all-in-one bitcoin dice system? - What exactly do you mean by 'all-in-one'?

@StephenM347 It sounds to me - and ManeBit, correct me if I'm wrong - that he's after code which he can use "off the shelf" to start his own dice site. If so, I'm 100% certain that's not going to be an option


text:
Bitcoin and ArchLinux on a Pogoplug
I'm interested in using a pogoplug to run bitcoin. From my understanding, I should be able to connect to the pogoplug from another device (RasPi with AM BEs) and mine on my own. I'm having trouble figuring out what I'm doing wrong -- I've got bitcoin cloned to the plug from github, but in following what it seems others have done,
make -f makefile.unix USE_PNP=-USE_IPV6=0

nets
make: *** No rule to make target `makefile.unix'.  Stop.

Suggestions?

Bitcoin now (as of a few hours ago) uses autotools to generate makefiles. You'll need to run ./autogen.sh first, then ./configure, then make.

Already ran those scripts.

Then you should just need to run make. See build-unix.md.

bitcoin moved to autotools a little while back. Provided you have all of the dependencies installed and in their normal places, you should be able to just do:
./autogen.sh
./configure
make

See doc/build-unix.md for instructions.


text:
What should I do about these messages in Bitcoin-core's debug-log?
Help, with bitcoin core : AFTER the Rescan :
1 - rescan
2 - response output, on the debug file out:
3 - FlushStateToDisk: write coins cache to disk (coins, kB) completed (s) , OK
4 - Imported mempool transactions from disk: X succeeded, X failed, X expired, X already there, X waiting for initial broadcast
problems that I don't understand, problems that I can't solve :
5 - Problem one : Cache size () exceeds total space ()
6 - Problem two : ERROR: AcceptBlockHeader: prev block not found
7 - with => Disconnecting outbound peer XXX for old chain, best known block =
8 - and with => ERROR: AcceptBlockHeader: Consensus::ContextualCheckBlockHeader: 000XXX...XXX, bad-diffbits, incorrect proof of work
Sorry for my bad english !
What do I have to do ? Help me Please Thanks !

These messages may be confusing, but there is nothing wrong. Let me go over them:

FlushStateToDisk: write coins cache to disk (coins, kB) completed (s) , OK

This means that the contents of the validation state cache is written to disk. Without this, you'd need to start over if Bitcoin Core were to shutdown uncleanly (e.g. your machine crashing or power loss).
This regularly happens from time to time, or when the cache grows too big (see further).

Imported mempool transactions from disk: X succeeded, X failed, X expired, X already there, X waiting for initial broadcast

This is a log message informing you about the result of unconfirmed transactions being loaded from disk.

Cache size () exceeds total space ()

This is giving the reason why the validation cache state is being written to disk: the amount stored in memory grew larger than how much memory Bitcoin Core was configured to use. This is normal. You can let Bitcoin Core grow the cache bigger using the -dbcache command line option, the dbcache= configuration file option, or the "Database cache" setting in the GUI.

ERROR: AcceptBlockHeader: prev block not found

A peer gave you a block header which Bitcoin Core couldn't interpret. Bitcoin Core simply ignores this.

Disconnecting outbound peer XXX for old chain, best known block =

A peer seemed to be on an invalid or old block, and didn't seem to synchronize with what your Bitcoin Core believes to be the best chain. There is either something wrong with that peer, or it isn't following the same rules. In either case, it's not a peer we'll learn anything useful from. Because the number of connections Bitcoin Core makes is limited, it decided it's better to disconnect this useless peer to make space for another, potentially better, peer.

ERROR: AcceptBlockHeader: Consensus::ContextualCheckBlockHeader: 000XXX...XXX, bad-diffbits, incorrect proof of work

A peer gave you an invalid block header. It is ignored.

What do I have to do?

Nothing.

Thank you ! a lot, I really understand better, the EVERYTHING!
So I just have to wait?
So I have a chance to see My Pieces, if you know?
if I have a chance, I would like to pay for your work, which must not be simple...!
but is very well explained!
Cordially ! Friends

If the problem is that your funds haven't shown up because you're still rescanning or synchronization: yes, they'll only show up when that is done. None of the log messages you quote are related to this at all.

Yes!!! I leave the PC on!
But how can I see the trace of the TX or the coins?
By simply waiting ? and I look up the TX and Coins

Cordially ... !!

error : "I leave the PC on!" NO, I don't close : Bitcoin-qt !

do you have a sign of signature bitcoins coins are trusted on client bitcoin-qt ?
"FlushStateToDisk: write coins cache to disk : 0 coins and  0 kB) " and
"Imported mempool transactions from disk: 0 succeeded, 0 failed, 0 expired, 0 already there, 0 waiting for initial broadcast" It's That ? all imported of mempool and also disk ?

I'm sorry, but I have no idea what you are trying to say or ask.

I want to talk about my bitcoin coins, I mean, how to find my coins?
should I wait for a particular writing on the debug of bitcoin Core? - Who will say end of synchronization of the old wallet.dat file with the transactions in the mempool with those in the disk, in the blockchain? (and given that the blockchain is downloaded, with BUGs, parcial, but downloaded)

or let the software re-sync, "naturally" with nothing to do but wait, so should I wait, leave bitcoin-qt open and do nothing?
I think the command -debug or -debugexclude, no! (because I don't know the output - the outputs)
and run a -rescan command... there, I don't see the point, because I already did that a few days ago...
so any advice?

You just need to wait until it is synchronized.

It's possible to faster the sync with (if it is the)
Log2_work
OR not ?
the Log2_work sync, are very slowly !
and other faster sync, tell me please ! if it is ! @Pieter Wuille

log2_work is just a metric for the amount of hashrate the chain has had. It has nothing to do with progress of synchronization, nor does it contribute to it.

Hello Peter!
I would like to ask you a question about bitcoin core mepool,
Is it possible not to see my funds because they are in the mempool?
AND OR
If a transaction (bitcoin core) has not succeeded, blocked in the mempool, the entire wallet (wallet.dat) must wait for ONE transaction to be synchronized, to show the entire balance?
do you have an answer ?
Cordially.

or, is there a bitcoin cache size to have? a size that: "reads all" / "syncs all" in the dbcache??

This isn't a discussion box. If you have more questions, ask them as questions, but please be respective of the time and effort of the volunteers who answer here. Please get someone to translate for you, or it'll get only more frustrating for everyone.

so, i've understood...

dbcache even if 16 GB can be fully taken. And that's the largest size supported. New version of Core should change it. Computers have more memory nowadays.

About -dbcache:
Maximum hard coded in Bitcoin Core 28.0 is 16384 MB, which gives 16 GB. For now it is not enough for chainstate reindex as it runs out of space.
I needed to exclusively compile Core for 32 GB of dbcache.


text:
Meaning of parts of a transaction created by an online wallet
I would like to know this complete transaction details.

The highlighted wallet is ID with 0.000496 BTC's is my wallet
Wallet with 2.7+BTC is I dont know what it is and why it is linked to my wallet ID?
Wallets on the left indicates red globe and green globe. What does that signify?

Possible duplicate: https://bitcoin.stackexchange.com/q/19278/5406

I interpret this as follows: you bought some bitcoin on Binance and requested a withdrawal. Binance processed the withdrawal in a transaction that batched 24 payments—they paid you and 23 others. They also sent some 0.60413290 BTC back to themselves. (Unfortunately, the latter is  obvious because Binance always uses the same address.) The 2.7+ BTC refer to the transaction output Binance spent in the input to this transaction.
The red and green globes signify whether the transaction outputs newly created by this transaction have been spent since. Green indicates unspent outputs, and red indicates spent. I deduce that your output has not been spent yet.
Also, FYI, you left enough letters visible for the transaction id and address to be identifiable.

It should be noted that completely redacting the transaction ID and address wouldn't help. Just the input or output list would be enough to uniquely identify the transaction, and other characteristics – transaction size, weight and fee – can be combined to narrow the possibilities to just four transactions.

Thanks for educating me. Is there a way I can connect with you one on one?

Yeah, if you were concerned about privacy you should have used a foreign transaction for your example.

@Nikko: I'm not sure whether you meant me or Vojtěch, but while I help with public content on this site, I'm not looking to provide private tutoring or assistance on basic use of Bitcoin.


text:
Accidentally sent money to donation address on bitcoin.org website
4 days ago I sended my money to Bitcoin wallet as a donation,I am new guy,and I did it wrong I wanted to transfer the money to my other wallet account to put it on my bank to buy food.now the money are gone,how I can take it back.Was the money for food till my next salary.Now the money are on Bitcoin org like a donation.Please help I have all proofs,and the transfer proof

how I can take it back

You can't take Bitcoin money. Confirmed Bitcoin transactions cannot be cancelled or reversed.

Now the money are on Bitcoin org like a donation

If you sent money to the donation address on the Bitcoin.org website, the only way to get the money back is to contact the website domain owner, explain the mistake you made, and ask them to create a new transaction paying you a similar amount.


text:
Generated new p2wpkh address from pubkey of existing address - but found it has old transactions!
this is certainly my lack of understanding, but it challenge what i know about blockchain.
I created a new address p2wpkh from public key with bech32 (testnet)  on 2022-11-10
: tb1q9ctu2t7zqvlevcamga9dfdhvlpeh93qdz6aawk
Made one transaction on 2022-11-10 10:29
pseudo code
b_pubkey = bytes.fromhex(pubkey)
h160 = hash_160(b_pubkey)
segwit_hrp = 'tb'
encoding = Encoding.BECH32
address = bech32_encode(encoding, segwit_hrp, [0] + convertbits(h160, 8, 5))

But in mempool it shows it has 3 transactions, goes back to 2022-11-07 15:10
https://mempool.space/testnet/address/tb1q9ctu2t7zqvlevcamga9dfdhvlpeh93qdz6aawk
aren't addresses supposed to be unique and collision resistant  ?

How did you generate this address?

i added pseudo code to post

Ok where did you get the pubkey?

public key of existing address, if that old address is linked to these transactions, why the NEW address is showing in output of confirmed block ?

If you want a new address, you need a new key... Addresses just encode the hash of a key or a script. If you use the same key, you'll get the same address.

you mean new public key?, but what to do if you only have Master Public Key OR a pubkey of old address?

If you have a master public key you can derive a new public key from it, and use those in the address. That's what master keys are for: generating multiple address keys for payments specifically so that they aren't all the same. If you have a public of an old address: don't, generate a new one.

I created a new address p2wpkh from public key

public key of existing address,

If you use the same public key as your starting point, you'll just generate the same address.
To create new addresses you need to first derive a new public key using the Hierarchical Deterministic (HD) process described by BIP-32.
(This answer is Community Wiki since it is just a summary of comments with the aim of removing this question from the list of unanswered questions)


text:
I need help with regaining acces back go my adress
can someone points to wsite which can calculate the nonce of a checksig, if needed i have files and proofs that im doing it for my own sake and i dont need to harm another person adress, i just want to have acces to mine once more, and be careful after i do. Or perse if i do. 2nd PS, before someone else do.
i read somewhere that if 2 nonces are the same, you can find a private key and about the adress, simply its personal, but let's say i wasn't on public surface and left an adress with some coins inside which can help me

(1) by checksig do you mean OP_CHECKSIG or something else? (2) Can you explain "regaining acces back go my adress" in more detail?

Yes, i read somewhere that if 2 nonces are the same, you can find a private key and about the adress, simply its personal, but let's say i wasn't on public surface and left an adress with some coins inside which can help me

Does this help: How to derive the Private key when the two the nonces are k and k*2

I saw this, in fact isnt that a somewhat of scam? I have troubles where and how to calculate all this because siggs are big values and all I found in the web is website which i think only show a sample, but dont really shows my privkey

The referenced question and its answers don't describe a scam (nor are they a scam). This is because good software chooses nonces randomly each time one is used. This means that the vulnerability should not arise if you use good software. As I said before, I would never never never type a private key into a website.

I understand, but in case like that i have to calculate it manually?  Or i have to find a bad site which will give me the right answer and move on fast?

I would write a short program in one of my favourite languages to perform the calculations described. Many programming languages have good ECDSA libraries and support for large numbers (bigInt etc). There might be other approaches available for non-programmers but they all have their own risks. Maybe there's a trustworthy javascript program that can be downloaded and run offline.  --- However, I have a strong feeling you may be choosing the wrong way to solve whatever your fundamental problem is (recovering a lost wallet?)

But at least the sample in the software must be right? I will name it asecurity.com,i can use their sample atleast to derive it? Can you suggest also something that's more user friendly, let's say exactly for person who really have no experience (me). Thank you by the way

If you don't have the private key corresponding to the address the coins were sent to, the coins are not yours, and nobody will help you be able to get access to them. The concern about duplicated nonces only applies in case broken software was used to spend. Proper wallet software does not let anyone but the owner access the coins.

@PieterWuille: I think that last comment of yours would be a whole answer.

If you don't have the private key corresponding to the address the coins were sent to, the coins are not yours, and nobody can help you get access to them.
While it is true that if two signatures exist spending coins from the same key with the same nonce, one can compute the private key from those signatures. However, no sane wallet software will ever repeat nonces, so in order to use this mechanism, the coins must already be held by broken software.

can someone points to wsite which can calculate the nonce of a checksig

No such website or software exists. The nonce used for a signature is secret, and only known to the signer. If you knew their nonce, you could compute the private key already.

if needed i have files and proofs that im doing it for my own sake and i dont need to harm another person adress

That's irrelevant, nobody can help you get access to coins you don't have the private key for.

i just want to have acces to mine once more

I'm sure you do, but I hope you realize that if this were possible at all, Bitcoin would be utterly broken. The whole point is not needing to trust anyone not to steal your coins.


text:
How do I find a list of all Electrum public servers?
I'm developing software that will connect to "Electrum Servers", the same public servers that Electrum Wallet connects to. The user will be able to specify a custom host if  they want, so it will be possible create a server and use it.
The main question is… How can I find the list of all the Electrum servers? I found this page that lists all the servers. How did this website get this information?
How can I create something to automatically get the all the Electrum servers? If I want to create a Website/API to provide this information, how can I get it?

Electrum has a list of default servers which you can find here: https://github.com/spesmilo/electrum/blob/afa1a4d22a31d23d088c6670e1588eed32f7114d/lib/network.py#L57. However there is no list of all Electrum servers just like there is no list of all Bitcoin nodes. This is because Electrum servers are distributed like Bitcoin nodes and anyone can run one. Electrum has a peer discovery protocol which you can read about here: https://github.com/kyuupichan/electrumx/blob/master/docs/PEER_DISCOVERY.rst. The Electrum servers will connect to each other and ask each other for a list of other Electrum servers each peer knows about. So each Electrum server builds up its own database of other Electrum servers. Then when a client connects, it asks the servers it is connected to for a list of other Electrum servers and the client also builds up its own peer database. The message for getting a list of other Electrum servers is servers.peers.subscribe and you can read about that here: https://github.com/kyuupichan/electrumx/blob/master/docs/protocol-methods.rst#serverpeerssubscribe
So if you want to find available Electrum servers, you can begin with the default Electrum servers. Then ask each server for a list of other Electrum servers, connect to those, and so on and so forth. You do this to build up a local database of Electrum servers.


text:
Where is blockchain located on Mac for Bitcoin-Qt?
I attempted to use Bitcoin-Qt on my PC but it could not handle the Bitcoin blockchain. I therefore decided to give up and to delete the blockchain. Sadly, I cannot find where it is located.
I have looked up in ~/.bitcoin and ~/Library/Application Support/Bitcoin/ but to no avail. Where else could I find it?

Quoting https://en.bitcoin.it/wiki/Data_directory#Mac :

Mac
By default Bitcoin will put its data here:

~/Library/Application Support/Bitcoin/

You can access that folder by opening a new Finder window --> then by going to "Go" --> "Go to Folder..." (or pressing command + shift + G), and then pasting the above path into the window that appeared.

Try to type into the Search window of the Finder "Bitcoin". Pretty sure you will see "Bitcoin QT" logo first in the list of the files found. Drag and drop it into the Trash folder and then empty it. The same way delete all the program files with "bitcoin" word in their names. It is not necessary that Library will contain the Bitcoin Core folder, that's why I uninstalled the Bitcoin Core app this way.


text:
Why doesn’t Bitcoin use UDP to do Blockpropagation?
Bitcoin uses TCP for P2P but why is UDP not used?  Modern day internet has relatively low packet loss rate so UDP is reliable. Even if some packets are dropped the peers can always request for the blocks. With TCP, the network is rather staticly constructed whereas with UDP you can construct a random graph each time a block is propagated as UDP is connectionless.
NANO uses UDP and I think it’s a good approach. I’m just curious as to if I’m missing something. Bootstrapping can obviously be done using TCP.

The Bitcoin system doesn't have just one network protocol:   Any way of obtaining the blocks is equally valid-- blocks over freenet, over satellite broadcast, over the P2P network, all work just as well and are used in practice.  UDP is used with bitcoin too, by the Fibre protocol.
As far the common Bitcoin P2P protocol goes UDP is not an especially good fit for most operations that it performs.  Bitcoin needs to reliably send messages far larger than an IP packet, like transactions and blocks. Although the internet is "reliable" packet loss rates of 1-3% are common. This means that any application needing to communicate large messages with UDP must implement packetization, retransmission, reordering, etc-- the same things that TCP already implements for us.  Many applications that have 'rolled their own TCP' in userspace have ended up with exploitable bugs in them, so it isn't something that should be done without good cause.
UDP also has the issue of NAT traversal:  Getting bidirectional communication across a NAT with UDP is not a simple matter. Crossing anything more complicated than a full cone nat requires considerable amounts of special code but without it there will be a many hosts that just can't talk to other peers with UDP.
You mention connectionlessness allowing you relay to a random host in the network, but in Bitcoin P2P we make use of the consistent relationship between nodes to make the network more reliable and efficient. Nodes have an idea what their peers already know and can avoid sending them redundant data. Nodes also know which peers have been fastest in sending them blocks in the past and handle them specially. Likewise, even if the protocol is connectionless there are costs to processing messages from peers by prioritizing handling messages from existing peers, Bitcoin reduces the impact of some kinds of DOS attack. Handling larger than one-packet messages, retransmissions, etc. also mean that even with a connectionless transport there still needs to be some kind of persistent state.
But an if implementation did want to randomly connect to send a message it could, at an extra cost is a couple round trips for the handshake. Or not even  that: there are only about 10,000 reachable nodes in the P2P network, there wouldn't be much of a challenge in holding open a mostly idle TCP connection to each and ever one of them if all the state retained were just the TCP state. So for that point the use of UDP would at best just be an optimization for something that could already be done but isn't done. I think it would be more interesting to demonstrate the usefulness of having all those connections first, before worrying about optimizing it.
So why might UDP transport be useful to the Bitcoin P2P protocol?

To get lowest latency block transfer possible there must be no round trips even in the face of packet loss which precludes TCP and which is why Fibre uses UDP. But to get no round trips the protocol must be able to handle loss without retransmission and to get low latency a block must be decodable with a minimum amount of data received. This requires very sophisticated error correction techniques which are an evolving area and haven't become mature enough yet to consider mainstreaming them. The latency benefits of fibre also exist so long as a small number of hosts are using it, since it does the heavy lifting of taking blocks all around the world.  Round trips don't cause much harm on low latency links. And this latency concern only applies to block relay.
Unlike TCP, UDP requires some amount of NAT traversal handling to simply get bidirectional communication working. But combined with full nat traversal handling UDP is often able to establish communication between hosts which are both behind different NATs.  This could be helpful for the Bitcoin P2P network since most hosts are unreachable behind a nat. So, ironically, one of UDP's challenges is also one of its uses. However, to connect hosts which are both behind a net the assistance of a third party non-natted host is required along with even more NAT traversal code.  Considering the complexity of traversal improving Bitcoin's support for TCP port mapping (e.g. implementing NAT-PMP) would probably be a better return on investment right now.
Use of UDP would allow for "worst than best effort" traffic handling. For low priority "bulk traffic" like syncing the blockchain it would be nice if the traffic carefully shaped itself to avoid interfering with other traffic on the network. Alternative congestion control algorithims like LEDBAT make it possible to transfer low priority data with minimal impact. But since TCP stacks don't yet commonly support these congestion control approaches, applications that want them currently need to implement their own transports. For Bitcoin it would be ideal if we could just flip a socket option and turn  LEDBAT on and off on existing connections (e.g. when the peer is requesting historical blocks or in the future when something like Fibre is being used by the node to relay new blocks) but that isn't an option yet.

[These last two reasons are the reasons that Bittorrent usually uses UDP]

@g-maxwell Great answer. What do you think about the following: If its for a PoS system in which certain validators can process transactions only. Instead of relying on TCP block propagation, do you think it would be useful to have those validators accept UDP transaction requests so that we can get transactions validated faster? In other words, do you see any disadvantage to use TCP for block propagation (large data) and UDP for single transaction request so that the nodes do not need to establish TCP connection.

Why do transactions need to propagate fast? That premise is generally in conflict with privacy.

Because if we want real adoption for crypto two things need to happen: stable currency and confirmation time. We can solve stablility via pegging and that leaves us with confirmation time. In order to achieve 2-3 second confirmation time we need faster relay mechnism than propagation.

Bitcoin achieves subsecond most of the network propagation time for small blocks... there is no need to use anything other than TCP for that.  But 2-3 second true confirmation time is pushing with the laws of physics for an actually decentralized network that spans earth and is probably not practically possible with real networks. Sometimes though not often whole continents are mostly severed from each other for minutes.

If you can put the whole POW vs Pos argument aside I actually have a working prototype that has run a real test net in a geographically dislocated setting where I achieved (my team) a 6-7 second true confirmation. This is not including transaction submission time of course. So to ask the question here Greg: would UDP transaction submission be a good idea in my setting? Appreciate your insight as you are well respected in my book.

Simply, NANO uses UDP for blocks which are usually (certainly mostly, but always??) lower than the theoretical size limit of a UDP packet, 65 KB. In Bitcoin's case, transactions ≠ blocks. It's surely inefficient to transfer long data (Bitcoin blocks can be up to 4 megabytes - theoretically) using UDP, and that's also how other services choose between them. Besides, even if blocks were a few kilobytes, packet drop will cause miners to lose A LOT, as a few seconds delay is significant.
If the data is small, you can just send one packet (which is usually much lower than 65 KB). Otherwise, using TCP, handshake, keepalive things are not expensive.
Trivia: In Satoshi's client, there were no additional error correction codes in the protocol. Satoshi trusted TCP error correction.

Block is bitcoin is 1 mb. sure, it won't fit into one UDP datagram, and even if it did, at 65KB, it is still subject to fragmentation. However, it makes no sense to not allow the transaction themselves to be UDP based

UDP is not very useful in Bitcoin. It is largely a unidirectional communication structure, whereas Bitcoin relies on bidirectional communication. When a node connects to another node, there is saved state for those nodes (nodes keep track of things that they have sent to other nodes and of things they have received from other nodes) and there is back and forth communication (e.g. send inv, receive getdata response, and send the data). UDP is not useful for that, but TCP is.
Furthermore, UDP exposes the nodes to the unreliability of the networks. The packet loss rate is not 0 (and cannot be 0), so using UDP results in a lot of extra data being sent due to packet loss. This is because blocks and transactions cannot be missing any bytes whatsoever. Otherwise they will be invalid. With TCP, if a packet is dropped, TCP will handle the resending of the data. But with UDP that needs to be handled at the application layer and it just gets complicated since figuring out what has been dropped requires bidirectional communication which is not easily done with UDP.
So overall, UDP is not useful for Bitcoin. It does not support bidirectional communication and it does not guarantee delivery of packets. Both of those things are necessary for Bitcoin's P2P protocol to work, so UDP can't be used. TCP is does both of those things at the protocol level, so it is used.

This is incorrect. First of all UDP is not a unidirectional communication. UDP is connectionless, however statefulness can be implemented at the application layer, which BITCOIN already does. see their p2p system. Like I said, the loss of the packet isnt an issue as the whole idea of a p2p network is so that its tolerant to packet loss. So what if a node didn't get a transaction, just ask for it again. Your answer is not very helpful. sorry.

I also cringed a bit at the unidirectional claim, but it's not entirely untrue: that's why NAT traversal is an issue for UDP where it isn't one for TCP.   It is the case the statefulness can be and is implemented in the application but the statefulness of TCP itself is really modest compared to the general statefulness of the protocol. As my answer pointed out: it would be totally reasonable for each bitcoin node to have a TCP connection up with most of the other reachable nodes. The state costs for the TCP itself aren't that substantial. The bitcoin protocol state is another matter...

There are several Reliable UDP implementations available by now, so requirements like packet loss recovery and statefulness can be addressed. One example is the UDT library developed by the University of Chicago: http://udt.sourceforge.net/
We've developed a protocol named SRT (Secure Reliable Transport), initially based on UDT and extended by adding bi-directional data transfer and encryption to the mix. Although it's optimized for real-time data like video, it's content independent by design: https://github.com/Haivision/srt
In video use cases, we are seeing dramatically increased bandwidth utilization numbers using UDP over TCP (e.g. RTMP), especially relevant for longer distance communication. I do believe it does make sense to evaluate UDP based alternatives for data transfer between nodes in order to increase network efficiency.

do you have contact info I can reach you at - I would like to discuss.


text:
How might I create a new Electrum wallet programmatically?
I want to create a new wallet using Electrum-apis. I have gone through the documentation given here but I haven't found any method to create a new wallet or an address.
Would someone guide me in this?

That's the Electrum server API. Electrum does not have an API. Wallet creation is client-side, as a result the Electrum API won't help you.

Ohh.. So if wallet creation is client side, is there any known api available that can help me create Electrum wallet in my application?

There is nothing special about an electrum wallet. The electrum API will allow you to query address balances, and build transactions, for any address, regardless of how they were created.
Electrum itself does use a special key derivation scheme, but that does not affect the usability from an API point of view. If you are using electrum servers as a backend, you should be able to generate wallets using any BIP39 library in your language of choice, and simply querying with the resulting addresses.

To create a new wallet run the following in your terminal:
electrum create

The above will output a short message of instruction, the path, and the seed to the screen.
You can read more about this command on its man page that can be accessed
via electrum help create.


text:
Importing a bunch of btc private keys into bitcoin core, then monitoring and sweeping them all to another address?
I'm trying to build a service where I import a bunch of pirvate keys into bitcoin core or some other software if you recommend a better solution. Once the private keys are imported, I want to monitor these addresses and if anything is positive have it send it to another wallet address I own.
I thought aboout importing the private keys with importprivkey api call. Does it take multiple private keys or do I have to loop through them?
Once these private keys are imported into my wallet I want to be able to detect the occurrence of any of the following:

if the entire wallet balance is greater than .001
or if 1 hour passes

Then, send all coins in the wallet to another address. My questions are:

Is there a service to monitor the bitcoin balance, or do I just have to make a cron job to query it every second?
When using sendtoaddress api call, do I have to specify a change address?
How do other people usually solve this problem?

That seems completely pointless. Why not give the target address in the first place then? Aren't you just generating a fee burden for yourself?

When using sendtoaddress api call, do I have to specify a change address?

No, it generates a new key pair and sends the balance to it. This key pair is stored in your wallet.dat file.

Is there a service to monitor the bitcoin balance, or do I just have
  to make a cron job to query it every second?

I do not know such service. Anyway, you should not give your private keys for any service.

When using sendtoaddress api call, do I have to specify a change
  address?

Why do you want to use bitcoin client and api calls for it?

How do other people usually solve this problem?

This is not a "problem". This is a "task". You should monitor transactions in the network and push your transactions which sweep the outputs to your address. I can provide you such toolkit, but not for free


text:
Are there any risks of publicizing a Lightning network invoice?
What are the perils of publicizing a LN invoice either before or after its payment?
From what I understand, after its payment the second payment of the invoice cannot be done. Other than the fact that the invoice was already paid, what could the third parties learn from a lightning invoice?
Keeping aside social-engineering impersonations:

Can the safety of the funds by affected any any information published in the invoice?
In what way publishing such an invoice may affect privacy of the transacting parties?
Could the payer prove (cryptographically) that they were the one that paid the invoice?

Does it affect -most importantly- the safety of the funds?

It does not. Funds are not affected by publishing a LN invoice.

Does it negatively affect the privacy or something else?

If a lightning invoice is published, 3rd parties may learn some semi-private information about the payer and payee. First and foremost, they learn the amount that was paid and to which node that amount was paid. Since an invoice can contain a short description, they may also learn what the payment was for by reading the description.
LN invoices also may contain specific routing information, particularly for private nodes. Revealing a LN invoice may also reveal a private node's existence, public key, and a channel id of that node.

Could the actual payer prove (cryptographically) that they were the one that paid the invoice?

Anyone in the route (including the payer) can prove that they were in the route by revealing the payment preimage. I'm not sure whether the payer can prove that he was the payer, but he can prove that he was involved in the payment as either a forwarder or as the payer.
However if the payer is involved in a dispute with the payee, he can prove that he paid by presenting the payment preimage.

Before answering your questions in detail let me state that as far as I know third parties cannot know if the invoice was paid after it was expired. Although I like Andrew Chow's answer I think I have some more bits to add.
Take a look at an invoice that I created:
user@computer:~$ lightning-cli invoice 10000000 bitcoinstackexchage explainInvoices 3600000
{
  "payment_hash": "0bc8bec5d84eb11df30db8752b23b4ab813a6f78732db94df9d0ca3d251d19bc",
  "expires_at": 1540772525,
  "bolt11": "lnbc100u1pde7epdpp5p0yta3wcf6c3mucdhp6jkga54wqn5mmcwvkmjn0e6r9r6fgarx7qdqcv4u8qmrpd9hyjmnkda5kxetnxq9rdm5qcqp2yzl5v00cdl9rq5egsfgknerxecczdzxldz9vgwnwcxw9snn6mst85z4gqd4ex59s0kzhzxnwyrdc88kfgkq908medwwd63pqqjnkr2gp6d6w2s"
}

I decoded its bolt11 string here:
user@computer:~$ lightning-cli decodepay lnbc100u1pde7epdpp5p0yta3wcf6c3mucdhp6jkga54wqn5mmcwvkmjn0e6r9r6fgarx7qdqcv4u8qmrpd9hyjmnkda5kxetnxq9rdm5qcqp2yzl5v00cdl9rq5egsfgknerxecczdzxldz9vgwnwcxw9snn6mst85z4gqd4ex59s0kzhzxnwyrdc88kfgkq908medwwd63pqqjnkr2gp6d6w2s
{
  "currency": "bc",
  "created_at": 1537172525,
  "expiry": 3600000,
  "payee": "03efccf2c383d7bf340da9a3f02e2c23104a0e4fe8ac1a880c8e2dc92fbdacd9df",
  "msatoshi": 10000000,
  "description": "explainInvoices",
  "min_final_cltv_expiry": 10,
  "payment_hash": "0bc8bec5d84eb11df30db8752b23b4ab813a6f78732db94df9d0ca3d251d19bc",
  "signature": "3044022020bf463df86fca305328825169e466ce302688df688ac43a6ec19c584e7adc1602207a0aa8036b9350b07d85711a6e20db839ec94580579f796b9cdd442004a761a9"
}

As you can see there is not that much sensitive information in here:

currency: bc --> no big deal
created_at: --> obviously dates might be interesting with relation to privacy. The timestamp is created by your system clock. Assuming no manipulation of the clock one learns that at this time the purchase took place.
expiry: only relevant in the sense explained above
payee: well it reveals you for obvious reasons. So you make public that at a certain time for a certain reason (c.f. description) you expect a payment of a certain amount (c.f. amount).
msatoshi: as payee
description: here comes all the social engineering which you didn't ask for
min_final_cltv_expiry: I would assume no problem here unless you can somehow afford a 51% attack and rewind some blocks while initiating a payment
payment_hash: as long as the hash function is secure (which is the base assumption of the lightning network) no one could do anything with the payment hash. If, for whatever reason, a third party on the chosen route (which is also private) knows the preimage of the payment hash funds could be stolen. However, this is irrelevant as the routing node knows the payment hash anyway.
signature: cryptographic signatures are supposed to be safe.

Therefore I conclude similarly as Andrew:

Does it affect -most importantly- the safety of the funds?

I don't see in which way it would.

Does it negatively affect the privacy or something else?

Other than the mentioned points which belong more to social engineering I would say no.

Could the actual payer prove (cryptographically) that they were the one that paid the invoice?

AFAIK there is only one way to prove that you paid an invoice. The node needs to have only one channel (during the relevant timeframe of invoice creation and expiry / received payment) which can be looked up on the blockchain and needs to provide the preimage. Moreover we need to assume that no one on the route publishes the preimage. Otherwise identifying the payer of an invoice seems impossible.


text:
How can you generate a bitcoin Public Key from an existing Private Key using openssl?
I know that you can generate a new key pair with a command like.
openssl  ecparam -genkey -name secp256k1
And the resulting output will contain a new private key with the encoded public key.
Is it possible for openssl to generate the public key straight from an existing private key only?
Just to be clear I mean Bitcoin Public Key and not Public Address

Check this out, it might help you How do these OpenSSL commands create a Bitcoin private/key from a ECDSA keypair

@AllanRomanato Thanks but still doesn't solve my problem.

Yes, openssl shows a public key corresponding to a private key that is input, e.g.
$ openssl ec -in priv_key.openssl -text -conv_form compressed


text:
I want to distribute small amount of bitcoins
What should I be aware when I want distribute say 0.0001 bitcoin to 50 bitcoin wallets.
I this be done with one transaction? what are the implication of transaction fees?
I want my students to experience the bitcoin phenomena.

I read this on bitcoinfees.com
If any of the outputs (including any change) of your transaction are less than 0.01 BTC, then a fee of 0.0005 BTC is required.
This is called bitcoin dust. Smaller transactions as you intend todo comes with a fee.

Well, I don't see the point in giving each student 0.000002 bitcoin, they cannot do anything with it as it is less than the transaction fee.
I'd recommend you to split maybe 1 BTC or 0.5 BTC amongst your student, to that they can actually use the coins.
For the practical side of the question, this can (and should) be done in a single transaction to reduce data overhead. Bitcoin allows to specify multiple inputs and outputs in a transaction, so you can take some bitcoins from your wallet and send some amount to 50 different addresses. However though, I think the original Qt client does not support this. (EDIT: As julian-goldsmith pointed out in his comment, it does seem to have this functionality after all.)
You could consider using a more advanced wallet like Armory (I have no experience with MultiBit or Electrum, maybe they can do the job as well). I think the web-wallet on Blockchain.info supports multiple outputs as well.
As long as the outputs are sufficiently big, like Filip says bigger than 0.01 BTC, you will not have to pay a fee, but of course you always can to support miners.

I don't have it up, but if I remember correctly, the Qt client has a "+" next to the address you're sending bitcoins to, that will let you add more addresses.


text:
Can a Raspberry Pi power 1 USB Asic Miner?
Looking at other setups it seems everyone is running with multiple USB Asic Miners through powered USB Hubs. Can the Raspberry Pi power a single miner through USB? Even using a 2 amp power adapter?

Which ASIC USB miner? All of the bitcoin ASIC's on the market use USB.

Looking here it seems that the USB ports on the RPi cannot exceed 100mA so it probably wouldn't be a good idea to use a Block Erupter directly on the RPi

Yes.
I have been running my Pi (B, 512mb) with a 1.5 amp power supply with 1 usb erupter (Sapphire) for over a week with no problems. Directly connected to the Raspberry, no hub.

Put it into a powered USB hub, then connect the hub to the PI.

This answer depends on the board revision of the Raspberry
for older boards you have a 100mA limit
this limit is no longer valid for Raspberry Model B Revision 2.0 but better use a powered USB HUB without on/off button

Rev 1.0 = MAX 1A
Rev 2.0 = MAX 2.5A

Raspberry Official Source


text:
Copy of old forum?
Is there a copy somewhere of the old bitcoin forum?
I would like to read early discussions and banter.

If you're talking about the forum that used to be at forum.bitcoin.org, then it's the same forum as the one at bitcointalk.org. Are you referring to some other forum?

The very old SourceForge forum was lost AFAIK, but there weren't many posts there.

There have been a number of deletions, but most threads and replies are still intact.
Change any URL forum.bitcoin.org/XXXX to bitcointalk.org/XXXXX or bitcoin.org/smf/XXXXX to bitcointalk.org/XXXXX

It seems https://bitcointalk.org/ has the posts since November 22, 2009. For example, all satoshi's posts are here (user id u=3).
Regarding earlier posts, in satoshi's first post in bitcointalk.org he refers to an "old forum":
Welcome to the new Bitcoin forum!

The old forum can still be reached here:
http://bitcoin.sourceforge.net/boards/index.php

I'll repost some selected threads here and add updated answers to questions where I can.

FAQ
http://bitcoin.sourceforge.net/wiki/index.php?page=FAQ

Download
http://sourceforge.net/projects/bitcoin/files/

and some people wrote that it's here:
http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-list


text:
How can I import public keys into armory?
I have a wallet on my phone and second, Armory wallet on my computer.  How can I import the public keys from my phone into Armory so that Armory can watch the balance.  I don't want to import the private keys, just the public ones.

Isn't that what "watching-only wallets" are for?

It's currently not possible with Armory.
Here is the issue for the feature request. The main developer of Armory is currently preparing a big change to the whole backbone of the client. The new backbone should make this possible.

etotheipi told me this will be possible with the new wallets format.
(comment on issue)


text:
Conditions for recovering a p2pkh bitcoin private key
Apart from Re-using same R value, What are the necessarily ecdsa values needed or method to recover the private key of a p2pkh address.
When same address is reused numerous times to receive bitcoin what threat does it pose?

What are the necessarily ecdsa values needed to recover the private key of a p2pkh address.

A key is recoverable if two signatures are created with the same R value.

When same address is reused numerous times to receive bitcoin what threat does it pose?

It leads to some loss of privacy, allowing different payments to be linked together.

Reusing addresses might also put you in a predicament where you are unaware of which customer paid for what service, especially if they are priced the same.

Is that the only conditions for recovering p2pkh private key? Apart from reused R values

It's not really relevant that it is a p2pkh key - Two signatures with the same value are all you need to recover an ECDSA private key. The key can also be "recovered" if it was created with bad entropy, such as via a brainwalle.


text:
Can someone please explain how Lightning paths are working and what effect large centralized hubs have?
Can someone please explain how Lightning paths are working and what effect large centralized hubs have?
Dear community,
I have read some stuff about lightning and i am really interested to understand these questions, hope someone can help:
1)  this interesting article here https://hackernoon.com/them-lightning-network-nodes-sure-do-look-centralized-to-me-what-gives-ee39c9b12ac0
basically it says that lightning wont be so "centralized" in the future.
Now, i am struggling with that conclusion.  If you want to route a lightning payment via your node, your channel(s) must hold at least a balance of the to-be-routed amount, right?  If not, your node by definition cannot be chosen as a potential routing path.
So, in the future, when we dream about mass adoption: in the financial markets space, where many large amounts will be transacted: Only a few "big" corporate nodes with large balances in their channels, can be chosen for routing  of large amounts, right?
Or is there an algorithm in place, where an amount of for example 100million could still be routed via a list of channels held by multiple nodes (sort of "hopping" through multiple channels until you arrive at 100million - for example 10+10+10+10+10+10+10+10+10+10)
Or is the lightning putting you always to a direct path to the ONE single channel that has the 100million available as balance?
Oh dear, I realize how messy my question is. Hope you understand what I want to know:)
2)  Second, I would like to know how the lightning algorithm is deciding via which channels/nodes to route payments?  Because many articles say that like 0.6% of all Lightning nodes are routing like about 85% of the overall lightning payments. And those 0.6% happen to be the ones with highest "capacity"
Why is that? Why not more evenly distributed?  And what is the meaning of "capacity"
I refer to this article, the table at the very end:
https://medium.com/coinmonks/bitcoin-lightning-network-run-your-node-at-home-for-fun-and-no-profit-da5b61be2ba9
3) What roles play WatchTowers in the future? And what influence could they have regarding (de)centralization of lightning?
4) Is there some sort of "penalty" in place when a node withholds a payment or does not route a payment? Or are these technically  non-valid scenarios?
5) Censorship:  How is transaction censoring possible in lightning ecosystem?
What are the scenarios here, and any preventing mechanisms planned/already in place?
An article in https://news.bitcoin.com/lightning-network-centralization-leads-economic-censorship/  writes:
""
A large miner (or even a majority cartel of miners) cannot effectively censor your transactions, because ANY miner can include your transaction in a block.
But the same is not true for LN hubs. They can effectively censor your transactions, because NOT any hub can route your payment.
Example: Megabank “A”, “B”, and “C” all decide to become Lightning Hubs. They all acquire customers and have channels open between the 3 banks. This allows customers from any of the banks to send money to other customers of those 3 banks.
You, being a freedom-loving individual, decide to open a channel with an anonymous hub known only as T35G7. Guess what. Megabanks A, B, and C don’t want to open a channel with T35G7. So you can’t send to any of their customers since there’s no available route.
""
6)  Finally, at the end of the day, my basic desire is to understand in what extent , within what ramifications, CAN Lightning become a highly centralized ecosystem, instead of a highly decentralized one.
Thanks so much for any helpful explanation:)

Not an answer, but I wanted to point out that it is a misunderstanding that all things need to be decentralized in order to work. If the underlying network (BTC) is sufficiently decentralized to remain censorship resistant, and a user can interact with the LN in a trust-minimized way, then the 'degree of centralization of the LN' isn't absolutely critical to the function of the network. Of course, relying on one node for all of the network's routing isn't good, but the network would still be functional even if that one node disappeared (ie no funds lost, just reduced routing capacity).

"sure does look centralized" I guess it all depends on how you look at it.  Any visualization tool can be made to look centralized or not decentralized given the same data sets right?

you honestly should have seperated your questions to several posts I will start answering them anyway...

Hey John, I see that you have already invested quite some time into researching the topics you're asking about, but our format works poorly with this sort of multi-topic questions: we tend to link up questions around the same topics and like to have multiple good answers compete on any single questions, but all of that works much better if a question has a narrow focus. Please ask separate questions in the future. Thanks!

If you want to route a lightning payment via your node, your channel(s) must hold at least a balance of the to-be-routed amount, right?  If not, your node by definition cannot be chosen as a potential routing path.

yes this is true. If you don't have enough bitcoin to forward you cannot forward a payment.

So, in the future, when we dream about mass adoption: in the financial markets space, where many large amounts will be transacted: Only a few "big" corporate nodes with large balances in their channels, can be chosen for routing  of large amounts, right?

I believe that the many large amounts is a strong assumption. I believe the amounts of financial transactions should be Zipf distributed with the vast majority of transactions being small and only a few transactions being large.
In the latter case (which is the one you ask about): if you want to transact a large value just open a new payment channel to the recipient (which with wumbo channels will be possible in the future). It is useful as they can spend the amount then. Also you could do it offchain but no need to.

Or is there an algorithm in place, where an amount of for example 100million could still be routed via a list of channels held by multiple nodes (sort of "hopping" through multiple channels until you arrive at 100million - for example 10+10+10+10+10+10+10+10+10+10)

Also note that in the future AMP (atomic multi-path) routing will be implemented so that large payments can be split into various small payments. It is well known how to implement AMP and it is part of the BOLT 1.1.

Or is the lightning putting you always to a direct path to the ONE single channel that has the 100million available as balance?

Path selection is not part of the lightning network protocol. It really depends on the implementation or even on you as the user as you are allowed to specify a path.

Second, I would like to know how the lightning algorithm is deciding via which channels/nodes to route payments?

see above. Generally people try to find shortest paths that are likely to have enough liquidity.

Because many articles say that like 0.6% of all Lightning nodes are routing like about 85% of the overall lightning payments. And those 0.6% happen to be the ones with highest "capacity"

Many articles?
The last statement doesn't seem provable to me. Since onion routing makes payments private, we would only know for our own node which routes we have taken and how many payments we have forwarded. While it is likely that most routes go through high capacity nodes the exact numbers are pure speculation. Could be more could be less.

Why is that? Why not more evenly distributed?

I'd say because currently only a few people have been #reckless enough to provide large liquidity to this cutting edge technology in which due to potential software bugs one still could lose funds.

And what is the meaning of "capacity"

Capacity is the amount of bitcoin sent to the 2-2 multi-sig wallet that is used to anchor a payment channel. This should not be confused with the balance which are the two outputs in the secret commitment transaction from that 2-2 address. The balance is sometimes referred to as inbound and outbound capacity but it is only known to the channel owners.

3) What roles play WatchTowers in the future? And what influence could they have regarding (de)centralization of lightning?

Watchtowers have been proposed as a service in which a third party monitors your channel state and helps you to make sure that your channel partner does not try to steal BTC from your channel by publishing an old state. In general it makes the lightning network safer to use. If that incentivises high capacity nodes to provide more liquidity or low capacity nodes to provide more liquidity we cannot say and is pure speculation.
btw, please be specific what you mean when you talk about "centrality". My feeling is that you mean nodes having high liquidity. I would much more prefer to talk about betweenness centrality.
Also keep in mind that the bigger issue with watch towers is that they might gather so much information about ongoing payments that the privacy of the lightning network is at risk.

4) Is there some sort of "penalty" in place when a node withholds a payment or does not route a payment? Or are these technically non-valid scenarios?

Short answer: Not really.
However, holding a payment up to the maximum CLTV_delta means that you cannot use that liquidity in your HTLC output to route other payments. Which means the node will earn less fees. So, economically a node is incentivised to route a payment as quickly as possible.
However, if a node wants to act in a delaying way to disturb the network it can do so. It is up to the nodes who are connected to that node to track such behavior and close their channels or to the sender nodes to not select this particular node for routing.
Also, keep in might that a node can at max hold 483 HTLCs per payment channel. So, you could only delay a small amount of transactions if you want to delay them for a long time.

5) Censorship:  How is transaction censoring possible in lightning ecosystem?
  What are the scenarios here, and any preventing mechanisms planned/already in place? An article in https://news.bitcoin.com/lightning-network-centralization-leads-economic-censorship/  writes: "" A large miner (or even a majority cartel of miners) cannot effectively censor your transactions, because ANY miner can include your transaction in a block. But the same is not true for LN hubs. They can effectively censor your transactions, because NOT any hub can route your payment. Example: Megabank “A”, “B”, and “C” all decide to become Lightning Hubs. They all acquire customers and have channels open between the 3 banks. This allows customers from any of the banks to send money to other customers of those 3 banks. You, being a freedom-loving individual, decide to open a channel with an anonymous hub known only as T35G7. Guess what. Megabanks A, B, and C don’t want to open a channel with T35G7. So you can’t send to any of their customers since there’s no available route.
  ""

No, this is not true. While Megabank A, B and C could decide not to route a payment that you initiate to T35G7 this will be hard in practice. First, every single "Megabank" does not know that this is a payment from you to this node unless the route looks like this You--> A --> B --> C --> T35G7 AND the Megabanks share all their information. In that case, you could see the banks as one node and the path looks like that You--> MEGABANK --> T35G7. In that case, yes, MEGABANK could refuse to forward a payment but it does not even know that You wanted to pay T35G7 because the route could actually look like this: some sender --> .... --> You --> MEGABANK --> T35G7 --> ... --> some receiver.
Also, what those described megabanks certainly cannot do is to prevent you from opening a direct channel with T35G7 which is the thing you should do anyway if you can't find a route on the network.
BTW the absurdity of the described scenario could also be transferred to Bitcoin mining if you have a cartel of bitcoin miners. Because that cartel could just agree to not mine on top of the blocks from other miners which includes such a transaction. If the cartel is a real cartel with more than 51% of the mining power, they can effectively censor any transaction...

6)  Finally, at the end of the day, my basic desire is to understand in what extent, within what ramifications, CAN Lightning become a highly centralized ecosystem, instead of a highly decentralized one.

The lightning network is a protocol on top of bitcoin. It is a well-known set of rules for people that consent to use Bitcoin in a certain way. In that sense, it is totally optional and always boils down to Bitcoin. Payment channels really are just 2-2 multi-sig wallets in which your funds are not locked as they can (under good circumstances) be moved around without the necessity to wait for Bitcoin miners or the blockchain or and empty mempool.
I always compare the lightning network with the world wide web. It is an open permissionless standard that anyone can participate. Might there be some "megabanks" providing high liquidity and utility which become dangerous. Certainly. The same happened with the Web and the emergence of huge Web companies that have integrated extremely deeply into our everyday live. However, it is always the choice of the user to use such actors. The lightning network itself like the Web is a decentralized technology. This does not prevent central services to emerge but as said they don't have to be used and certainly don't control the technology or the protocol.
So, I really hope I could bring some light to all the paranoia that is being spread by some people...

You wrote "Also keep in mind that the bigger issue with watch towers is that they might gather so much information about ongoing payments that the privacy of the lightning network is at risk.". It was my understanding that watchtowers are merely watching on-chain activity to respond with a blinded punishment transaction if they see an on-chain transaction that locks funds to a matching preimage. I don't understand how they would be able to diminish privacy in the Lightning Network. Could you please elaborate?

Also, I edited your answer which included amending a couple phrases, so please check whether I messed up something.

well I think there is no standard for watchtowers as they don't become part of the protocol yet. So a potential watchtower could see every channel state that you have  deducing all htlcs with all payment hashes and sees where traffic flows.

couldn't see how you messed stuff up! thanks for your proof reading

Mh, it was my understanding that the current concept of watchtowers didn't require the WT to know about the channel state. Rather they would just have a set of preimages that they would compare to any unilateral channel closings, to see whether they have the matching punishment. Since the punishment pays out the full amount to the punisher, I am not sure it would reveal the state–––– Ahhh... I see. The output of the punishment tx can only credit the amount claimed by the punished and thus reveals the channel balance?

Thanks a lot guys! Especially @RenePickhardt
Will structure my questions in the future much better