text:
Keeping wallet addresses outside bitcoin core
I'm trying to build a good security for my online bitcoin based application. Is something like this possible?
send_bitcoin(address)
create_address()
sweep_balances()
       +
       |
       |
       v         New transactions
+-------------+  <----------------   +--------------+
|             |                      |              |
| Wallet keys +--------------------->+ Bitcoin core |
|             |                      |              |
+-------------+   Submit signed tx   +--------------+

The "Wallet keys" will be my hot wallet. There will be another wallet store which would be used for cold storage. The "bitcoin core" will be some trusted public node that I don't control.
Does using this architecture have any disadvantages over running a full bitcoin core to keep addresses?
I need to get callbacks for new transactions on the list of addresses in my store, that is not managed by bitcoin core. Is it possible using this method?
My reason is that running a full "bitcoin core" takes up too much computing resources & disk which is expensive on cloud and by doing this I can avoid having to run and maintain a bitcoin node. I can submit a signed transaction to any trusted node.
Thanks.

Does using this architecture have any disadvantages over running a full bitcoin core to keep addresses?

Yes.

The "bitcoin core" will be some trusted public node that I don't control.

That is a huge disadvantage.
As Satoshi Nakamoto wrote:

What is needed is an electronic payment system based on cryptographic proof instead of trust,
  allowing any two willing parties to transact directly with each other without the need for a trusted
  third party.

(my emphasis)
Every time someone tries to undo this they are kicking the founder of Bitcoin in the teeth!

Other references

Thin Client Security

I understand this hurts the idea behind bitcoin! Otherwise, since the transactions are signed from my side, it doesn't seem like the third party node can act maliciously, the worst that they can do is ignore my transaction. Is there any other possible attack scenario?

@viko: See https://en.bitcoin.it/wiki/Thin_Client_Security#Server-Trusting_Clients

Yes you can use this architecture, provided that the bitcoin node you don't control doesn't know your private keys for any address you use.
What you want to do is use a reputable library, for instance electrum client, but there are many libraries in many languages available to do basic things.
From electrum client for instance, you can:

create address (wallet, keep private)
create raw transaction (unsigned, and just collection of utxos basically)
sign transactions (keep private keys private), output hex of signed tx is safe to then broadcast
broadcast transactions

Electrum client is sometimes called SPV mode (simple payment verification). This is the protocol used for lite clients, mobile wallets etc.
Once you sign your transaction, it can be broadcast via the bitcoin node you don't control, a public electrum server, or an explorer that offers relay via an api. Some mining pools offer transaction broadcasting as well.
The main thing to be hyper-vigilant about is that you keep your private keys (wif) secret.
As for getting a "callback" about transactions to your wallet, you can use an explorer to poll your addresses for new transactions. Also using an electrum wallet will give you the ability to do all of this as well.


text:
How to spend a multisignature, CLTV time-locked UTXO?
I am using this locking script with libbitcoin explorer to create a bitcoin testnet transaction:
bx script-to-address -v 196 "[efa716] checklocktimeverify drop 2 [02bceab65ad6b5e3c1d6683b1d0ffd7bce572ce3ece0f1976fb04ba34bb0359c1b] [02c4916c1c044bda56670f28efdb6637fa24448f5184839f917a473abfd81e6d3d] 2 checkmultisig"

This gives me the receiving address:
2NA3vs7QUdoXhUELUAGkGUv1Cuy1DpM4uPs

The funding transaction can be seen here:
https://api.blockcypher.com/v1/btc/test3/txs/56fd7329a72cb48a517b6a396a8fa537190ad1f72333781b21dbfd8ae1616dd6?limit=50&includeHex=true
I create a raw transaction using bitcoin-cli, because bx unexpectedly changes the send-to-address. I have tried setting locktime to 00000000, 4294967295, and not specifying nlocktime at all.
bitcoin-cli -named createrawtransaction inputs='''[ { "txid": "'56fd7329a72cb48a517b6a396a8fa537190ad1f72333781b21dbfd8ae1616dd6'", "vout": '1' } ]''' outputs='''{ "'2N2zbPDQWhcs5niMKqQ4dz8m8jxU5izkabh'": 0.00800000}''' locktime=00000000

Next, both signatures are applied to the transaction, using the appropriate private keys and bx input-sign.
bx input-sign 46d170451e8565c9b5233ca160ba1aee9680f24059fa565af17ef11f80cc7d43 "[efa716] checklocktimeverify drop 2 [02bceab65ad6b5e3c1d6683b1d0ffd7bce572ce3ece0f1976fb04ba34bb0359c1b] [02c4916c1c044bda56670f28efdb6637fa24448f5184839f917a473abfd81e6d3d] 2 checkmultisig" 0200000001d66d61e18afddb211b783323f7d10a1937a58f6a396a7b518ab42ca72973fd560100000000ffffffff0100350c000000000017a9146aece0dbf34db9b86c8aab5e9ecb2b37c669b44f8700000000

bx input-sign 53064d99236c9e5fe30d2fe2dc3a5f6cd2374cac3f5c997a327218381d41c7d7 "[efa716] checklocktimeverify drop 2 [02bceab65ad6b5e3c1d6683b1d0ffd7bce572ce3ece0f1976fb04ba34bb0359c1b] [02c4916c1c044bda56670f28efdb6637fa24448f5184839f917a473abfd81e6d3d] 2 checkmultisig" 0200000001d66d61e18afddb211b783323f7d10a1937a58f6a396a7b518ab42ca72973fd560100000000ffffffff0100350c000000000017a9146aece0dbf34db9b86c8aab5e9ecb2b37c669b44f8700000000

This gives the endorsements:
endorsement1 = 30450221008694e53196e222ed86935d9c2d16c14baec194d70e3eed474bc2423ec8adcf4f02204d6a72b87f190684a5f52228620d77357be5dd3265bfcdfd16d9a260a09fa33401
endorsement2 = 304402207006d5e588298cd8ae2cd2430c1cba38b3a0a31bbee8ea0ab0349e509e50e9b802200207fc65e28acb070f9fa9cfc11ae06b4b10c112daa23fb05f9e297c430580e301

The script is encoded:
bx script-encode "[efa716] checklocktimeverify drop 2 [02bceab65ad6b5e3c1d6683b1d0ffd7bce572ce3ece0f1976fb04ba34bb0359c1b] [02c4916c1c044bda56670f28efdb6637fa24448f5184839f917a473abfd81e6d3d] 2 checkmultisig"

And finally:
bx input-set "zero [30450221008694e53196e222ed86935d9c2d16c14baec194d70e3eed474bc2423ec8adcf4f02204d6a72b87f190684a5f52228620d77357be5dd3265bfcdfd16d9a260a09fa33401] [304402207006d5e588298cd8ae2cd2430c1cba38b3a0a31bbee8ea0ab0349e509e50e9b802200207fc65e28acb070f9fa9cfc11ae06b4b10c112daa23fb05f9e297c430580e301] [03efa716b175522102bceab65ad6b5e3c1d6683b1d0ffd7bce572ce3ece0f1976fb04ba34bb0359c1b2102c4916c1c044bda56670f28efdb6637fa24448f5184839f917a473abfd81e6d3d52ae]" 0200000001d66d61e18afddb211b783323f7d10a1937a58f6a396a7b518ab42ca72973fd560100000000ffffffff0100350c000000000017a9146aece0dbf34db9b86c8aab5e9ecb2b37c669b44f8700000000

I can validate using bx:
bx input-validate 02bceab65ad6b5e3c1d6683b1d0ffd7bce572ce3ece0f1976fb04ba34bb0359c1b "[efa716] checklocktimeverify drop 2 [02bceab65ad6b5e3c1d6683b1d0ffd7bce572ce3ece0f1976fb04ba34bb0359c1b] [02c4916c1c044bda56670f28efdb6637fa24448f5184839f917a473abfd81e6d3d] 2 checkmultisig" 30450221008694e53196e222ed86935d9c2d16c14baec194d70e3eed474bc2423ec8adcf4f02204d6a72b87f190684a5f52228620d77357be5dd3265bfcdfd16d9a260a09fa33401 0200000001d66d61e18afddb211b783323f7d10a1937a58f6a396a7b518ab42ca72973fd5601000000e1004830450221008694e53196e222ed86935d9c2d16c14baec194d70e3eed474bc2423ec8adcf4f02204d6a72b87f190684a5f52228620d77357be5dd3265bfcdfd16d9a260a09fa3340147304402207006d5e588298cd8ae2cd2430c1cba38b3a0a31bbee8ea0ab0349e509e50e9b802200207fc65e28acb070f9fa9cfc11ae06b4b10c112daa23fb05f9e297c430580e3014c4d03efa716b175522102bceab65ad6b5e3c1d6683b1d0ffd7bce572ce3ece0f1976fb04ba34bb0359c1b2102c4916c1c044bda56670f28efdb6637fa24448f5184839f917a473abfd81e6d3d52aeffffffff0100350c000000000017a9146aece0dbf34db9b86c8aab5e9ecb2b37c669b44f8700000000

Endorsement is valid

However, when using bitcoin-cli sendrawtransaction, I always see one of these error messages:
# No nlocktime set, or set at 0
error code -26
error message:
non-mandatory-script-verify-flag (Locktime requirement not satisfied) (code 64)

# nLocktime set for max or 4294967295
error code -26
error message:
non-final (code 64)

I have seen this thread:
Transaction still unspendable with CLTV
But even though I have broadcasted the transaction via a block explorer, the transaction seems to be ignored. What could the cause of this?

Solved by encoding the checklocktimeverify integer correctly, following the command line steps provided by https://github.com/ChristopherA/Learning-Bitcoin-from-the-Command-Line/blob/master/08_2_Building_the_Structure_of_P2SH.md
bx script-to-address -v 196 "[9f7b2a5c] checklocktimeverify drop 2 [02bceab65ad6b5e3c1d6683b1d0ffd7bce572ce3ece0f1976fb04ba34bb0359c1b] [02c4916c1c044bda56670f28efdb6637fa24448f5184839f917a473abfd81e6d3d] 2 checkmultisig"

And for the spend transaction, setting locktime as the current unix time:
bitcoin-cli -named createrawtransaction inputs='''[ { "txid": "'dc4f97b0bb843820eff1bf677472cfe4f56ade41baf1dce4cdef75f8f5a44836'", "vout": '0' } ]''' outputs='''{ "'2N6bFRmGF1JtHgHUYZKdVNZ2ZHtyHc8MBru'": 0.009 }''' locktime=1552996392

Successfully sent the transaction using bitcoin core:
https://live.blockcypher.com/btc-testnet/tx/dac8db0182559bbf3e6f54efac6fb283bcd78356347173fb7f969a53c7d5ec36/

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why does CLN (Core Lightning) sync from Genesis while LND only syncs from the wallet's birthday?
I've installed two major Lightning Network implementations, CLN (formerly c-lightning) and LND, with bitcoind testnet as backchain on the same server. On the first startup I noticed CLN synchronizes all blocks from the beginning of the blockchain:
2022-07-05T23:31:29.487Z DEBUG   lightningd: Still waiting for initial block download
2022-07-05T23:31:29.593Z DEBUG   lightningd: Adding block 1: 00000000b873e79784647a6c82962c70d228557d24a747ea4d1b8bbe878e1206
2022-07-05T23:31:29.647Z DEBUG   lightningd: Adding block 2: 000000006c02c8ea6e4ff69651f7fcde348fb9d557a06e6957b65552002a7820
2022-07-05T23:31:29.685Z DEBUG   lightningd: Adding block 3: 000000008b896e272758da5297bcd98fdc6d97c9b765ecec401e286dc1fdbe10
2022-07-05T23:31:29.732Z DEBUG   lightningd: Adding block 4: 000000008b5d0af9ffb1741e38b17b193bd12d7683401cecd2fd94f548b6e5dd
2022-07-05T23:31:29.752Z DEBUG   lightningd: Adding block 5: 00000000bc45ac875fbd34f43f7732789b6ec4e8b5974b4406664a75d43b21a1
2022-07-05T23:31:29.774Z DEBUG   lightningd: Adding block 6: 000000006633685edce4fa4d8f12d001781c6849837d1632c4e2dd6ff2090a7b
2022-07-05T23:31:29.792Z DEBUG   lightningd: Adding block 7: 00000000e29e3aa65f3d12440eac9081844c464aeba7c6e6121dfc8ac0c02ba6

while LND only scans blocks from the wallet's birthday onwards:
2022-07-05 23:21:55.757 [INF] LTND: Waiting for chain backend to finish sync, start_height=2284292
2022-07-05 23:21:56.613 [DBG] LNWL: Chain backend synced to tip!
2022-07-05 23:21:56.614 [DBG] LNWL: Locating suitable block for birthday 2022-07-03 15:15:05 -0300 -03 between blocks 0-2284292
2022-07-05 23:21:56.617 [DBG] LNWL: Checking candidate block: height=1142146, hash=00000000000bee59e6acdb88deee299fcce4c8603a04a60b022d70f256b4d880, timestamp=2017-06-18 16:00:52 -0300 -03
2022-07-05 23:21:56.619 [DBG] LNWL: Checking candidate block: height=1713219, hash=000000000000905bf5e522da9cc5d30a93728feb51f3409396193f120505e2ca, timestamp=2020-04-14 06:51:58 -0300 -03
2022-07-05 23:21:56.621 [DBG] LNWL: Checking candidate block: height=1998755, hash=00000000000009aa098093884b3f352bfc1efa24ec2af3d541e2fa2ba7e54d6b, timestamp=2021-06-02 19:57:47 -0300 -03
2022-07-05 23:21:56.623 [DBG] LNWL: Checking candidate block: height=2141523, hash=00000000849bdb7b93b3deb7182cb1ec3dc2e470d4359ac6fe67d19e5bd5f84e, timestamp=2022-02-14 17:16:49 -0300 -03
2022-07-05 23:21:56.625 [DBG] LNWL: Checking candidate block: height=2212907, hash=00000000000004e43dbee8c2f5ae622bb6a984183ef07a8341b1eec6ce331ce5, timestamp=2022-04-28 00:06:02 -0300 -03
2022-07-05 23:21:56.627 [DBG] LNWL: Checking candidate block: height=2248599, hash=00000000000009c8763aaa3757774b8c8611495dd2f190672d1c4084d40368d6, timestamp=2022-05-26 08:36:29 -0300 -03
2022-07-05 23:21:56.629 [DBG] LNWL: Checking candidate block: height=2266445, hash=00000000000022f256ef974c1467221afc19f6c087ffb147554191d128099438, timestamp=2022-06-16 10:46:47 -0300 -03
2022-07-05 23:21:56.631 [DBG] LNWL: Checking candidate block: height=2275368, hash=0000000000000069a27c710313fc5a764e8bab5fed550af837d0c8b1d6fdd558, timestamp=2022-06-16 14:23:46 -0300 -03
2022-07-05 23:21:56.633 [DBG] LNWL: Checking candidate block: height=2279830, hash=00000000000001d54cbb97ed368723e4ae7681f6c3d625d517b9dbe3a7d0678a, timestamp=2022-06-18 00:41:09 -0300 -03
2022-07-05 23:21:56.635 [DBG] LNWL: Checking candidate block: height=2282061, hash=00000000000000b1517a474a0c8e32a9bd452bd1117cc2a799195991d10c3306, timestamp=2022-06-23 13:04:59 -0300 -03
2022-07-05 23:21:56.637 [DBG] LNWL: Checking candidate block: height=2283176, hash=00000000000000498bf2dbab757cba69b77a8d31ea437b2a377aabdb08c0d3cc, timestamp=2022-06-28 19:37:39 -0300 -03
2022-07-05 23:21:56.639 [DBG] LNWL: Checking candidate block: height=2283734, hash=000000000000003a55f287f3243841f215572c092995ccf286d9d92398d21785, timestamp=2022-07-02 14:50:17 -0300 -03
2022-07-05 23:21:56.641 [DBG] LNWL: Checking candidate block: height=2284013, hash=000000000000004f39aada91825e5d6121e6d7bcda6be504a6f2c13dac133df9, timestamp=2022-07-04 05:13:13 -0300 -03
2022-07-05 23:21:56.643 [DBG] LNWL: Checking candidate block: height=2283873, hash=000000003c07419de729aa383f639e506e24dc5f7e947f80cfaf5d2e67837c43, timestamp=2022-07-03 08:31:57 -0300 -03
2022-07-05 23:21:56.645 [DBG] LNWL: Checking candidate block: height=2283943, hash=0000000000000017b3b716a4e380900fbf2e114d2536a6f5e71427e5b10a34f8, timestamp=2022-07-03 17:57:30 -0300 -03
2022-07-05 23:21:56.647 [DBG] LNWL: Checking candidate block: height=2283908, hash=000000000000004aef1f2fc4dc6973f7c9ac67e5f6c69e1df089f92c83e91324, timestamp=2022-07-03 13:34:25 -0300 -03
2022-07-05 23:21:56.647 [DBG] LNWL: Found birthday block: height=2283908, hash=000000000000004aef1f2fc4dc6973f7c9ac67e5f6c69e1df089f92c83e91324, timestamp=2022-07-03 13:34:25 -0300 -03
2022-07-05 23:21:56.656 [INF] LNWL: Started rescan from block 000000000000004aef1f2fc4dc6973f7c9ac67e5f6c69e1df089f92c83e91324 (height 2283908) for 0 addresses
2022-07-05 23:21:56.687 [INF] LNWL: Catching up block hashes to height 2283919, this might take a while
2022-07-05 23:21:56.693 [INF] LNWL: Done catching up block hashes
2022-07-05 23:21:56.693 [INF] LNWL: Rescanned through block 0000000000000005cdcf62d03147fc812f5b01b21a919bdbe38a3588cafdd5f8 (height 2283919)
2022-07-05 23:21:59.062 [INF] LNWL: Catching up block hashes to height 2284292, this might take a while
2022-07-05 23:21:59.063 [INF] LNWL: Done catching up block hashes
2022-07-05 23:21:59.063 [INF] LNWL: Finished rescan for 0 addresses (synced to block 00000000eb61cbd41eaaa4c3646418dd1acf75d30b04239385b19c51fef43cec, height 2284292)
2022-07-05 23:21:59.785 [INF] LTND: Chain backend is fully synced (end_height=2284292)!

This seems to give a huge advantage over CLN on first startup times. As of now, my CLN node has spent over 14 hours synchronizing while LND worked in a few seconds. Also, as far I understand, a Lightning node should sync the full UTXO set to verify open channels. But this should be provided by the backchain.
Is there a good reason for CLN to reprocess everything? Also, if LND only syncs from the wallet's birthday, how does it verify channels' funding transactions?
Thanks in advance.

Hi Gabriel, I updated the title of your question to better summarize the main point of your question. Please take a look whether I understood you right and feel free to further edit or rollback.

Was bitcoind still synchronizing with the blockchain when you started CLN? CLN would technically only scan the last 6 blocks if not started with something like --rescan, as we assume this is a new node, and not trying to recover. Scanning the whole blockchain is definitely not intended and a bug if not caused by --rescan or a mismatched bitcoind

@cdecker Yes, it was. I ran this command from CLN's installation guide: bitcoind & ./lightningd/lightningd & ./cli/lightning-cli help, which I believe started CLN's lightningd before bitcoind was fully synchronized. But then I stopped it, waited for the synchronization to complete, and started CLN again. It resumed from where it had stopped instead of scanning the latest blocks.

Sounds like this should be a bug report (or report of slightly flaky behavior) on the Core Lightning repo. I don't think there is anything to add from a StackExchange perspective.

Hm, well not so sure it's a bug now: CLN just take bitcoinds current head and starts syncing from there, so if bitcoind is at <6 blocks then it is perfectly normal for it to start at height 0. Gabriel, you can have lightningd start at a later height by specifying a negative rescan: lightningd --rescan=-600000 will set the sync height to block 600k and it'll sync from there.

Added my more complete answer, with a couple of resolutions, as an answer too :-)

CLN should start from close to the current blockchain height, however that current blockchain height is learned from bitcoind which in your case was still syncing.
By starting bitcoind and lightningd at roughly the same time, we get a bitcoind that reports the current height very close to the genesis block at height 0. When lightningd then asks bitcoind for the current blockchain height it gets that low-balled height back and assumes it should start syncing from that height, leading to the entire blockchain to be processed.
There are a couple of solutions:

Wait for bitcoind to sync completely (or almost completely if you're fine with CLN processing a few more blocks) and only then start CLN
If you started CLN before bitcoind was synced you can reset the height from which lightningd syncs by specifying the absolute height to sync from with a dash (-, or minus) in front of it: lightningd --rescan=-600000 [other options] will reset the blockchain height to 600k and lightningd should continue scanning from that height.
Use an alternative backend such as sauron or btcli4j which talk to an explorer instead of a local bitcoind for super lightweight (but more trusted) bitcoin backend. You can later switch over to a local bitcoind once that's synced too.

If there is an idea on how to make this more user-friendly we're open to suggestions.
Disclaimer: I'm one of the CLN maintainers.

For a wallet to assess how many Bitcoin it controls it needs to scan the blockchain for UTXOs that are controlled by the wallet.
LND uses a wallet seed format called aezeed which defines a wallet birthday (days since Bitcoin's genesis). It scans the blockchain from this birthday rather than from genesis for relevant UTXOs and assumes that it doesn't control any UTXOs created prior to this birthday.

I don't understand why I should synchronize my Lightning node at all if I run a Bitcoin full node on the same server (why not just tell CLN/LND where the blocks are?).

Your Bitcoin full node verifies the blockchain from genesis. It is interested in all transactions and all blocks being valid and meeting consensus rules, not just the UTXOs that are relevant to the wallet. Once your full node has done this your wallet can assume it is scanning the valid blockchain.

I thought the main reason to sync the blockchain would be to verify channel funding UTXOs and ensure the network topology contains only valid channels. If this is true, I don't understand how LND does it without looking at older blocks. And I do understand why the Bitcoin node needs to sync, but not why the Lightning node (see the CLN logs, for example) needs to re-process all those blocks instead of either: 1)  just accessing the already-synchronized blockchain; or 2) relying on the UTXO set provided by the Bitcoin node.

@GabrielRebello: It depends what you are running LND with. There are 3 options as far as I'm aware, running it with Bitcoin Core, btcd (alternative implementation in Go) or Neutrino. In the case of running it with Bitcoin Core or btcd, Bitcoin Core or btcd is validating the chain not LND.

In the case of the CLN logs it is not reverifying the blockchain once Bitcoin Core has already done it. It is logging that Bitcoin Core is verifying the blockchain (I think)

I'm not sure because lightning-cli --testnet getinfo gives me a warning: warning_lightningd_sync: Still loading latest blocks from bitcoind. and blockheight = 0. It shouldn't take 14 hours to just log blocks either. My Bitcoin node itself is fully synchronized.

Are you sure your CLN node is connected to your already fully synchronized Bitcoin node? I wonder if it is starting a new IBD for a new Bitcoin node...

I understood the main gist of the question to be why CLN starts from Genesis. Could you perhaps expand your answer in that regard?

Well Bitcoin Core starts from genesis in validating the entire chain and then the CLN wallet doesn't use aezeed as there is no concept of a wallet birthday in CLN. I don't think that is the problem here though

@Murch I've edited the question. Hope to have improved it.

@MichaelFolkson I can confirm the CLN node's RPC settings bitcoin-rpcconnect=127.0.0.1 bitcoin-rpcport=18332 point to the correct RPC endpoint as defined in the Bitcoin Core's synchronized node config file rpcbind=127.0.0.1 rpcport=18332. If I understand correclty, this should mean they are connected. The CLN node also crashes if I stop Bitcoin Core.

CLN should definitely not be starting at the genesis block. Was it started with --rescan or bitcoind wasn't yet synced with the chain?

@cdecker It was started before bitcoind was synced. I've shared some details in the main question's comment thread.

This all seems unrelated to the question. It's not because CLN doesn't use aezeed (or any other deterministic key generation method) that it has no notion of a wallet birthday. Bitcoin Core has a concept of wallet birthdays, and has had that since long before it used BIP32 even.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Bitcoin protocol and Wireshark
I captured bitcoin protocol communication in real-time and it is present in fig1 and fig2.
Fig1

In red box, there are many types of command.
First Question:

How can Wireshark detect bitcoin communication and parse them? I think
  the packet that socket sends may have something what I don't know.

In red box of fig2, you can see many properties of packet. this is definitely defined by software program, not hardware such as router. Real data that client send is really just hex data which is encoded by specific rule.

Second Question:

How Wireshark can parse data received? Bitcoin client can parse it because they know the rule but Wireshark doesn't know anything since Wireshark is thrid-party program, which no have any relation with Bitcoin software.

Fig2

Sorry for my English.

Wireshark knows about 3000 different protocols, including Bitcoin. The list of protocols it handles is here: https://www.wireshark.org/docs/dfref/

Someone wrote a Bitcoin protocol decoder for Wireshark, several years ago. I assume it was included in the Wireshark distribution.
Wireshark simply knows about the Bitcoin protocol. There is no magic involved.


text:
How to calculate P2WSH signatures with OP_CODESEPARATOR?
I read the bip143, but didn't understand how to calculate the scriptCode of p2wsh.
bip143 told us:
A new transaction digest algorithm is defined, but only applicable to sigops in version 0 witness program
Double SHA256 of the serialization of:
  1. nVersion of the transaction (4-byte little endian)
  2. hashPrevouts (32-byte hash)
  3. hashSequence (32-byte hash)
  4. outpoint (32-byte hash + 4-byte little endian)
  5. scriptCode of the input (serialized as scripts inside CTxOuts)
  6. value of the output spent by this input (8-byte little endian)
  7. nSequence of the input (4-byte little endian)
  8. hashOutputs (32-byte hash)
  9. nLocktime of the transaction (4-byte little endian)
  10. sighash type of the signature (4-byte little endian)

The item 5:

For P2WPKH witness program, the scriptCode is 0x1976a914{20-byte-pubkey-hash}88ac.
For P2WSH witness program,
if the witnessScript does not contain any OP_CODESEPARATOR, the scriptCode is the witnessScript serialized as scripts inside CTxOut.
if the witnessScript contains any OP_CODESEPARATOR, the scriptCode is the witnessScript but removing everything up to and including the last executed OP_CODESEPARATOR before the signature checking opcode being executed, serialized as scripts inside CTxOut. (The exact semantics is demonstrated in the examples below)

I don't know how to find out if "the signature checking opcode being executed".
Then I read the example "Native P2WSH" in bip143 and I am even more confused.
Here is the first example of "Native P2WSH":
The following is an unsigned transaction: 0100000002fe3dc9208094f3ffd12645477b3dc56f60ec4fa8e6f5d67c565d1c6b9216b36e0000000000ffffffff0815cf020f013ed6cf91d29f4202e8a58726b1ac6c79da47c23d1bee0a6925f80000000000ffffffff0100f2052a010000001976a914a30741f8145e5acadf23f751864167f32e0963f788ac00000000

    nVersion:  01000000
    txin:      02 fe3dc9208094f3ffd12645477b3dc56f60ec4fa8e6f5d67c565d1c6b9216b36e 00000000 00 ffffffff
                  0815cf020f013ed6cf91d29f4202e8a58726b1ac6c79da47c23d1bee0a6925f8 00000000 00 ffffffff
    txout:     01 00f2052a01000000 1976a914a30741f8145e5acadf23f751864167f32e0963f788ac
    nLockTime: 00000000

  The first input comes from an ordinary P2PK:
    scriptPubKey: 21036d5c20fa14fb2f635474c1dc4ef5909d4568e5569b79fc94d3448486e14685f8ac value: 1.5625
    private key:  b8f28a772fccbf9b4f58a4f027e07dc2e35e7cd80529975e292ea34f84c4580c
    signature:    304402200af4e47c9b9629dbecc21f73af989bdaa911f7e6f6c2e9394588a3aa68f81e9902204f3fcf6ade7e5abb1295b6774c8e0abd94ae62217367096bc02ee5e435b67da201 (SIGHASH_ALL)

  The second input comes from a native P2WSH witness program:
    scriptPubKey : 00205d1b56b63d714eebe542309525f484b7e9d6f686b3781b6f61ef925d66d6f6a0, value: 49
    witnessScript: 21026dccc749adc2a9d0d89497ac511f760f45c47dc5ed9cf352a58ac706453880aeadab210255a9626aebf5e29c0e6538428ba0d1dcf6ca98ffdf086aa8ced5e0d0215ea465ac
                   <026dccc749adc2a9d0d89497ac511f760f45c47dc5ed9cf352a58ac706453880ae> CHECKSIGVERIFY CODESEPARATOR <0255a9626aebf5e29c0e6538428ba0d1dcf6ca98ffdf086aa8ced5e0d0215ea465> CHECKSIG

  To sign it with a nHashType of 3 (SIGHASH_SINGLE):

  hashPrevouts:
    dSHA256(fe3dc9208094f3ffd12645477b3dc56f60ec4fa8e6f5d67c565d1c6b9216b36e000000000815cf020f013ed6cf91d29f4202e8a58726b1ac6c79da47c23d1bee0a6925f800000000)
  = ef546acf4a020de3898d1b8956176bb507e6211b5ed3619cd08b6ea7e2a09d41

    nVersion:     01000000
    hashPrevouts: ef546acf4a020de3898d1b8956176bb507e6211b5ed3619cd08b6ea7e2a09d41
    hashSequence: 0000000000000000000000000000000000000000000000000000000000000000
    outpoint:     0815cf020f013ed6cf91d29f4202e8a58726b1ac6c79da47c23d1bee0a6925f800000000
    scriptCode:   (see below)
    amount:       0011102401000000
    nSequence:    ffffffff
    hashOutputs:  0000000000000000000000000000000000000000000000000000000000000000 (this is the second input but there is only one output)
    nLockTime:    00000000
    nHashType:    03000000

  scriptCode:  4721026dccc749adc2a9d0d89497ac511f760f45c47dc5ed9cf352a58ac706453880aeadab210255a9626aebf5e29c0e6538428ba0d1dcf6ca98ffdf086aa8ced5e0d0215ea465ac
                                                                                ^^
               (please note that the not-yet-executed OP_CODESEPARATOR is not removed from the scriptCode)
  preimage:    01000000ef546acf4a020de3898d1b8956176bb507e6211b5ed3619cd08b6ea7e2a09d4100000000000000000000000000000000000000000000000000000000000000000815cf020f013ed6cf91d29f4202e8a58726b1ac6c79da47c23d1bee0a6925f8000000004721026dccc749adc2a9d0d89497ac511f760f45c47dc5ed9cf352a58ac706453880aeadab210255a9626aebf5e29c0e6538428ba0d1dcf6ca98ffdf086aa8ced5e0d0215ea465ac0011102401000000ffffffff00000000000000000000000000000000000000000000000000000000000000000000000003000000
  sigHash:     82dde6e4f1e94d02c2b7ad03d2115d691f48d064e9d52f58194a6637e4194391
  public key:  026dccc749adc2a9d0d89497ac511f760f45c47dc5ed9cf352a58ac706453880ae
  private key: 8e02b539b1500aa7c81cf3fed177448a546f19d2be416c0c61ff28e577d8d0cd
  signature:   3044022027dc95ad6b740fe5129e7e62a75dd00f291a2aeb1200b84b09d9e3789406b6c002201a9ecd315dd6a0e632ab20bbb98948bc0c6fb204f2c286963bb48517a7058e2703

  scriptCode:  23210255a9626aebf5e29c0e6538428ba0d1dcf6ca98ffdf086aa8ced5e0d0215ea465ac
               (everything up to the last executed OP_CODESEPARATOR, including that OP_CODESEPARATOR, are removed)
  preimage:    01000000ef546acf4a020de3898d1b8956176bb507e6211b5ed3619cd08b6ea7e2a09d4100000000000000000000000000000000000000000000000000000000000000000815cf020f013ed6cf91d29f4202e8a58726b1ac6c79da47c23d1bee0a6925f80000000023210255a9626aebf5e29c0e6538428ba0d1dcf6ca98ffdf086aa8ced5e0d0215ea465ac0011102401000000ffffffff00000000000000000000000000000000000000000000000000000000000000000000000003000000
  sigHash:     fef7bd749cce710c5c052bd796df1af0d935e59cea63736268bcbe2d2134fc47
  public key:  0255a9626aebf5e29c0e6538428ba0d1dcf6ca98ffdf086aa8ced5e0d0215ea465
  private key: 86bf2ed75935a0cbef03b89d72034bb4c189d381037a5ac121a70016db8896ec
  signature:   304402200de66acf4527789bfda55fc5459e214fa6083f936b430a762c629656216805ac0220396f550692cd347171cbc1ef1f51e15282e837bb2b30860dc77c8f78bc8501e503

  The serialized signed transaction is: 01000000000102fe3dc9208094f3ffd12645477b3dc56f60ec4fa8e6f5d67c565d1c6b9216b36e000000004847304402200af4e47c9b9629dbecc21f73af989bdaa911f7e6f6c2e9394588a3aa68f81e9902204f3fcf6ade7e5abb1295b6774c8e0abd94ae62217367096bc02ee5e435b67da201ffffffff0815cf020f013ed6cf91d29f4202e8a58726b1ac6c79da47c23d1bee0a6925f80000000000ffffffff0100f2052a010000001976a914a30741f8145e5acadf23f751864167f32e0963f788ac000347304402200de66acf4527789bfda55fc5459e214fa6083f936b430a762c629656216805ac0220396f550692cd347171cbc1ef1f51e15282e837bb2b30860dc77c8f78bc8501e503473044022027dc95ad6b740fe5129e7e62a75dd00f291a2aeb1200b84b09d9e3789406b6c002201a9ecd315dd6a0e632ab20bbb98948bc0c6fb204f2c286963bb48517a7058e27034721026dccc749adc2a9d0d89497ac511f760f45c47dc5ed9cf352a58ac706453880aeadab210255a9626aebf5e29c0e6538428ba0d1dcf6ca98ffdf086aa8ced5e0d0215ea465ac00000000

01000000
0001
02
fe3dc9208094f3ffd12645477b3dc56f60ec4fa8e6f5d67c565d1c6b9216b36e 00000000
48
47304402200af4e47c9b9629dbecc21f73af989bdaa911f7e6f6c2e9394588a3aa68f81e9902204f3fcf6ade7e5abb1295b6774c8e0abd94ae62217367096bc02ee5e435b67da201
ffffffff
0815cf020f013ed6cf91d29f4202e8a58726b1ac6c79da47c23d1bee0a6925f8 0000000000
ffffffff

01
00f2052a01000000
1976a914a30741f8145e5acadf23f751864167f32e0963f788ac

0003

47304402200de66acf4527789bfda55fc5459e214fa6083f936b430a762c629656216805ac0220396f550692cd347171cbc1ef1f51e15282e837bb2b30860dc77c8f78bc8501e503
473044022027dc95ad6b740fe5129e7e62a75dd00f291a2aeb1200b84b09d9e3789406b6c002201a9ecd315dd6a0e632ab20bbb98948bc0c6fb204f2c286963bb48517a7058e2703
4721026dccc749adc2a9d0d89497ac511f760f45c47dc5ed9cf352a58ac706453880aeadab210255a9626aebf5e29c0e6538428ba0d1dcf6ca98ffdf086aa8ced5e0d0215ea465ac

So here are my questions:

Here "witnessScript" is RedeemScript, right?
What is meaning of 'CODESEPARATOR' here?
Why calculate both scriptCode(OP_CODESEPARATOR is not removed && OP_CODESEPARATOR is removed) and both put them into witness data?

witnessScript is not the redeemScript but it is analog to it. It is the actual Script that is committed to by its hash in the output, and executed when spending it. It is the "Segwit version of the redeemScript" if you will.
CODESEPARATOR is an (interesting [1]) OP code which allows to customize the behaviour of the CHECKSIG OP code. It allows to section the script code for a signature to commit to only part of it.
BIP143 slightly changed its behaviour for SegWit scripts.
Regarding your last question, if i understand correctly you are refering to these comments in the test vector.

(please note that the not-yet-executed OP_CODESEPARATOR is not removed from the scriptCode)

everything up to the last executed OP_CODESEPARATOR, including that OP_CODESEPARATOR, are removed

These comments are here to illustrate the new behaviour introduced in BIP143. The witness script of the output these signatures are for is the following.
<026dccc749adc2a9d0d89497ac511f760f45c47dc5ed9cf352a58ac706453880ae> CHECKSIGVERIFY CODESEPARATOR <0255a9626aebf5e29c0e6538428ba0d1dcf6ca98ffdf086aa8ced5e0d0215ea465> CHECKSIG

The two comments describe the difference in the scriptCode for computing the signature of each public key contained in the witness script.
For the first one there was no CODESEPARATOR before the CHECKSIGVERIFY, therefore the scriptCode is the entire witness Script. For the second one however there was a CODESEPARATOR executed before the CHECKSIG, therefore the scriptCode is the part of the witness script starting just after the CODESEPARATOR up to the end.

[1] CODESEPARATOR was initially used to execute the scriptPubkey and scriptSig as a single stack (inserted in-between). It was then removed by Satoshi when patching the 1 RETURN bug.

Yes, witnessScript is just the RedeemScript, which would also be the last element of the witness data.  I am not sure about the OP_CODESEPARATOR, or why a script would contain that in the first place.


text:
How do devs decide who should have commit access? what is the process?
Andrew Chow (2020) compiled a great list on when people gained/lost commit access to Bitcoin Core.
Satoshi asked Andresen to take over as Lead Maintainer, and Andresen passed the responsibility to Van Der Laan. The Lead Maintainer appears to be selected by the previous Lead Maintainer (by convention).

Many others have received/lost commit access. How exactly is
this decision made? and by who?

If the Lead Maintainer unexpectedly died (hope not), but if he
did, who would decide upon the next Lead Maintainer and how?

Many others have received/lost commit access. How exactly is this decision made? and by who?

Nowadays the decision is generally made by the developers during one of the weekly IRC meetings. This is not done by committee nor a specific group, just whoever is attending the meeting when the topic is brought up. It's usually the same people at every meeting.
Typically how adding a maintainer works is that someone brings up a topic for adding a maintainer, others nominate a person to be that, and then the group votes. Since in the past there have really only been one person nominated at a time, it's been a yes/no vote rather than a vote for a candidate. The process is largely informal as the people nominated are typically well known developers and have been around for a while.
For revoking access, the process works pretty much in reverse. Someone brings up a topic to remove access for particular users, typically because they are inactive, and the group votes on whether to do so. There have been a few exceptions to this - Gavin Andresen's commit access was revoked after concerns that his accounts had been compromised. Since he was also inactive, it was decided to leave his commit access revoked.
Developers may also choose to no longer have commit access in which case their commit access is revoked upon asking.

If the Lead Maintainer unexpectedly died (hope not), but if he did, who would decide upon the next Lead Maintainer and how?

It would pretty much be the same process as adding a new maintainer. This is also likely the same process if the lead maintainer decided to step down. Someone would volunteer or be nominated, likely one of the current maintainers, and then that person is voted on by the group at a meeting.

In general, there is a formal process for adding or removing commit access. It's generally done by the group of developers attending an IRC meeting with those developers essentially voting yes/no.

Thanks again Andrew. Superb reply as always :-) Now that there is institutional interest in Bitcoin, do you think more robust governance mechanisms are needed? It is only a matter of time before major banks have their own Bitcoin 'devs' contributing to the GitHub repo. As a group, they could flood the IRC, and get someone pro-institutional-banking in to commit access ranks. At which point Bitcoin no longer belongs to us, but rather to banks and how they choose to develop it.

Generally the maintainers have been established developers who are well known already. I think it would be unlikely that a newcomer would be able to become a maintainer. Furthermore, given how informal the process is, the "votes" from randoms and newcomers would be given less weight. If many established developers were to NACK (vote no) a potential maintainer, then he would not be given access.

That makes sense. I suppose if formal governance mechanisms are used, it ceases to be 'decentralised'. Perhaps the informal approach is okay for now.


text:
Can someone calculate my private key if I send the same amount to one address?
If I send the exact same amount to the same address in two separate transactions, can someone calculate my private key from the scriptSig? Will the signature (S) and the message (Z) be same?

Generally, your wallet software should ensure that you can create and recreate the same or different transactions safely.
However there is one issue that seems related and might have prompted your question.
When someone creates an ECDSA signature, the signature is composed of two integers, r and s. r only depends on a random nonce k, while s depends on the private key, r, and the message. If the same k is reused, it will cause a signature to have the same r value. Usually, your wallet will use a different k for every signature even for the same message, so your signature will be different every time. Some wallets use deterministic nonces, so they'd recreate the same signature for the same message. They will still create a different signature for different messages.
It is important that this k is always picked randomly (unless you're using deterministic nonces): if you sign two different messages (i.e. transactions) with the same private key and the same k value, the two signatures result in a set of equations that can be solved for the private key.

No, regardless of whether the messages and signatures are the same, it is not possible to calculate the private key from the public information in the blockchain data.
In practise, I believe the signed data in a transaction includes the hash of the previous transaction for each input, these cannot be the same as in any prior confirmed transaction because that would constitute a forbidden double-spend.
I believe cryptographic weaknesses can be quite subtle but the situation you describe, multiple payments of the same amount to the same address, must be quite common. Consequently I'd be very surprised if it had not been anticipated and considered long ago.


text:
Sending BTC 3 times to the same address - 2 times success - one time not received
I sent BTC to the same address on three different occasions. The first two times were successful. The third time (yesterday) the receiver claimed that he did not receive the BTC. It took over an hour until my wallet gave me confirmation that the transaction was successful. How can I independently verify that the receiver did in fact receive the 3rd transaction? Thx
Receiver address: 184wkr7Jxf6EM2ivFRh2Cy8iiwkCzuFTwV

You could use a block explorer (e.g., https://blockstream.info/) and search for the ID of each transaction you transmitted. It could be the case that the 3rd transaction has not been confirmed yet and is still in the mempool if the associated fees were not competitive.


text:
Wasabi Wallet 2.0 stuck at "Waiting for Coinjoin"
I have moved over to Wasabi Wallet 2.0. I'm trying to undertake my first CoinJoin (I have selected Max privacy)
I have a tick in the bottom right corner which shows Tor is running, backend connected and connected to 8+ peers.
After starting CoinJoin, the process does not move on from
"Waiting for CoinJoin"
This as been at this stage for over 12 hours. Unsure of the status or how to review what may be the problem?

Perhaps this has to do with the recent disruptions suffered by the Tor network.

It looks more like a bug or the user selected the economic profile which only coinjoins in the cheaper part of the week.

That message AFAIK simply means that your wallet is waiting for coinjoin to happen, but if you didn't manage to participate in any coinjoin for 12 hours then something is wrong. Have you tried to restart Wasabi?
Make sure that you are using the latest version (2.0.1) which you can get here https://wasabiwallet.io/index.html#download
If the total sum of the non private coins is less than 0.01 BTC then you have to manually start participating in coinjoin by pressing the play button on the music box.
If none of the above helps check the logs and see if there is anything that explain what you are experiencing.


text:
Electrum Offline Storage - two different OS, multiple wallets
Regarding the instructions to create watch-only wallets in Electrum, I am clear on the intial setup and functionality, signing transactions, etc. but I have a few questions about going forward:

Is it problematic to use two different OS (Windows on a Chromebook for offline comp along with OSX for watch-only comp).  Will Electrum have any problems going back and forth between two different OS?
As I make more transactions, is there any mistake I can make that can un-sync the two wallets? (I am mainly concerned about change addresses).  Going forward, can I always rely that I will be able to access newly generated private keys by expanding the gap limit and/or adding change addresses in the console?
Can I manage a separate seeded wallet on my watch-only comp for smaller amounts or is this not reccomended?

I am trying to be as secure as possible without having to learn Linux.  It has taken me long enough to figure out bitcoin and Electrum in general!  Thank you so much.

This can easily be done and due to Bitcoin mechanics, you are going to be pretty safe. Electrum wallets cannot be desynced, maybe become inaccurate but the same seed will always give you access to the corresponding wallet and its bitcoins. Electrum inaccuracies in balance and transaction history rectify themselves over time. Just restore from seed on the other computers to set your wallet up on multiple computers.
Why does this work out?
Bitcoin does a neat thing with transactions and history. I won't go into all of it but the blockchain makes working between two computers easy. Your seed is the deterministic private keys for 10 addresses on electrum. This will always be true, on any machine, anywhere in the universe (unless a bug breaks it on a computer).
As long as you have the private keys, you have access to all funds sent to the corresponding public addresses of the private key. Even if the wallet is offline and detached, it can still access value. Imagine the electrum seed as a key to a coin safe, you can have as many duplicates of key as you want, as long as they are identical they will open the coin safe.
The balance of your addresses are purely derived from info available to anyone on the Bitcoin network, your balance is not dependent on your Electrum program, computer, or identity. If you sent me an address, I could tell you the balance. To get your balance, Electrum queries the many Electrum nodes to sync up. The balance can be wrong on multiple computers but give it a few seconds to correct itself.

Electrum is based on Python, an interpreted language and thus operating system-independent. As long as you can install Python and the required Python packages in the corresponding OS, Electrum should work.
When trying to figure out which addresses have been used, Electrum stops probing further addresses if several addresses (the gap limit) do not contain any BTC anymore (i.e. from transactions whose output has already been reused). So increasing the gap limit makes sure Electrum finds all generated addresses which contain any BTC. As far as I understand the code after a few minutes, when not using change addresses there is only one list of generated addresses, otherwise there are two lists.
Using an additional seeded/online wallet for small amounts is a trade off between security and comfort. E.g. if you have a local store which accepts BTC, using an seeded wallet on your smartphone is much more comfortable than running home and signing the transaction using your offline wallet. The same is true for a seeded wallet on your online PC, although "running home" may not be an issue there. Multiple wallets can be used by starting Electrum with the "-w walletfile" parameter (replace walletfile with one the corresponding wallet files).

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Is there any way to prevent intra wallet transaction fees?
I heard about it somewhere, but no where found a complete step by step solution.
I've heard that for addresses in the same wallet, the transaction fees can be avoided. Is it a rumor ?
If no, then please help on how to attain it.
By wallet, I mean to say the wallet that is being used by bitcoind/litecoind or bitcoin-qt/litecoin-qt on a machine. (they usually use the wallet.dat)

It would be helpful if you clarified what you mean by "same wallet." Are you talking about two different accounts on the same online wallet service/exchange? Or two different addresses in your bitcoin-qt client?

@Tony I modified the question to explain what I mean by wallet.

A traditional Bitcoin wallet contains a bunch of private keys and a bunch of accounts. Each private key has a corresponding Bitcoin address.
For each private key that the wallet has issued, some account is associated with that key. This serves only one purpose -- it determine which account gets its balance raised when funds are received at that address.
Intra-wallet transfers require no transaction and thus pay no fee. I can switch from 10 Bitcoins in account "Jeff" and 0 in account "Judy" to 0 in "Jeff" and 10 in "Judy" just by swapping a 0 and a 10 inside the wallet. The Bitcoins remain at the same address, they are just credited to a different account.

you mean to change the label of addresses, right ? Well, I was actually looking to aggregate large number of small transactions, is there a way ? so as to pay minimum fee.

@coding_idiot Not really. You're better off just waiting and letting the outputs get old.

To reply to your updated question: If you have two addresses in your wallet, and want to send coins from one of the addresses to the other, then you do have to pay fees. The network doesn't know whether two addresses in the same wallet. (Well, technically it could, if both addresses were used as inputs in the same transaction at some point, but I digress - in any case, the network doesn't care. Two different addresses are two different addresses.)

The network does not know if two addresses are in the same wallet or not.
Since the transaction fees are verified/received by the network, an inter-wallet transaction MUST be the same as an intra-wallet transaction, in every way.

This is not true. An intra-wallet transaction can consist of just moving a balance from one account to another without a transaction. Inside a wallet, it doesn't matter what addresses balances are associated with, just what account.

There are no fees say if you didn't want to actually move bitcoins in between address. As David Schwartz pointed out, you can have accounts which are basically bitcoin wallet organizers in which you can add or remove bitcoins from.
If say you need to send coins that are 'stored' in address A from address B which is in your wallet, you'd need to create a transaction to move coins from A to B, and then another transaction that moves coins from B to destination address. The Bitcoin-QT wallet doesn't yet let you do this through the GUI, maybe with a future coin control update.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Difference between regtest and testnet
I'm not sure what is the difference between regtest and testnet. They seem to be different things, but I can't find the difference between them.
Does one have more stuff than the other?
On the regtest, peers are seemingly to not to be found.

Regtest is a network designed solely to be private and for testing only.  You can create a private regtest network and do testing and experimentation with it.  It is not a public network and has no peers for you to connect to.
You can run multiple nodes on your regtest network, using the -connect startup option to specify the nodes to connect to.  Bitcoin Core has a built in miner specifically for regtest, using the generate command.  Check this developer site for details.


text:
What is the recommended OS for compiling a Windows binary for Litecoin?
I'd like to compile Litecoin for Windows, but have difficulties doing so. To be specific: I want to create the Windows executable (.exe) GUI (qt) Litecoin client (to be more specific I want to compile the executable for an alt-coin based on litecoin, but the code base is the same as litecoin).
Many tutorials are either focused on compiling for Linux, or they focus on Bitcoin compiling, which is a bit different nowadays than Litecoin. Rather than explaining my specific problems, which are likely unique to my situation, I was wondering what a recommended Operating System (and version) is to compile Litecoin? Who has specific (preferably smooth) experience building the GUI litecoin client for Windows?
A useful relevant topic is this one:
https://bitcointalk.org/index.php?topic=149479.0
But this is based on bitcoin, not litecoin. This referenced tutorial has a bit on Windows, but it's not too optimistic:

Windows -Qt
This is the trickiest one to crack of the GUI wallets. I am going to
  detail how I got this to work and offer you an easy way to get the
  dependencies in an attempt to make this work for you too. That said,
  it may not–and I’ve already said I won’t do tech support. So here’s
  the deal. I got this to work and then duplicated it on a second
  machine to ensure it wasn’t a fluke! Most of the information needed to
  compile the basic coind.exe or GUI wallet is in this thread:
  https://bitcointalk.org/index.php?topic=149479.0 Unfortunately nothing
  is as easy as it seems, and although the MinGW and QT installs went
  fine, I couldn’t compile it without a few tweaks to the .pro file.

In this tutorial, Windows is used to create the executable, but I understand one can also use Linux to create Windows executables, right? There are few experiences documenting successful or smooth compiling of Litecoin (forks) for Windows, so I was wondering if people do have good experiences doing so and what kind of OS or approach they used.

I'd go with Ubuntu as it's the distribution all guides are most likely going to focus on.

On windows this guide helped me: http://andarazoroflove.org/how-to-create-alt-coin-scrypt-clones-for-fun-and-profit-bitcoin-litecoin-secrets-exposed/

The title of your question is misleading. The best OS for compiling any bitcoin variant would clearly be a Linux dist. What you're asking is "What is the recommended OS for compiling a Windows binary for Litecoin."

I've tried using the following as a baseline,
https://bitcoinqtmsvc2012.codeplex.com/
however also needed some tweaking, but usefull resource for anyone looking to try.

The Litecoin source tree, in the doc/ folder, provides a file regarding building on Windows, using MinGW and MSYS. I can't personally vouch for those instructions, never having used them myself, so if you go that route, good luck. The way that the release binaries are built, if I'm not mistaken, is using the Gitian builder. If you're looking to do a Windows build, I would try that. If you want to actually use Gitian, you can clone the repo, and use the readme file to get Gitian set up, and then follow the build segment of the release process to do the actual build. On the other hand, if you're not necessarily looking for determinism or the ability to reproduce the exact build, you could try taking a look at the Gitian build scripts to see what packages are being used, what commands are run, et cetera, and try to reproduce that on a Linux system.

I have compiled litecoin-qt on Ubuntu 12.04 before. First install the necessary packages:
sudo apt-get install qt4-qmake libqt4-dev build-essential libboost1.48-all-dev libssl-dev libdb++-dev libminiupnpc-dev

Then, in the directory of the litecoin source code, run
qmake
make

And you're done!
However, the compiled file is a Linux binary, so you'll have to use it on Linux. As far as I know, trying to compile a Windows binary on Linux would be even harder than just compiling it on Windows.
Note: This answer was posted before the question was clarified.

Thanks but not really what I am looking for. I am specifically looking for compiling the Windows executable...

The question should be rephrased, as it's not clear what you're asking. Are you asking which is the best version of Windows to run while compiling litecoin?

@user1162541: I think it would be best if you explained your "unique situation" and why you can't just use the provided binary.

@John Thanks, I have rephrased the question to make it clearer.

@Tony I foresee the need to frequently recompile and would like to find the best configuration for this, so before trying to solve this problem I'd like to see if there are specific OS configurations that people recommend that make the process overall easier. I have added more detail in the post.

Just to clarify, are you making your own altcoin? If not, I don't see why you can't just download the Windows binary provided by the altcoin devs.

@Tony Yes I am making my own litecoin-based altcoin (see edited post). This is why compiling is necessary.


text:
Why does txID generation consider signatures?
I've been reading a lot about transaction malleability, and noticed that the only way to achieve it is by altering the transaction signature/s.
Why does the protocol even consider the signature in the payload for SHA256?
The protocol should only consider inputs and outputs as they are, ignore the signatures. txID should only be altered by altering inputs or outputs, thats it. If anybody would like to alter any input or output to change the txID, he'll need to change signatures as well.
In other words, if you don't consider signatures while generating the txID, transaction malleability would be impossible.
Please, let me know if I'm missing something here.

If the signature was not part of the transaction protected by the hash, then you could have two different transactions, one valid and one invalid, with the same transaction ID. The primary purpose of the transaction ID is to permit agreement on whether a transaction is valid or not. So excluding the signatures from the hash would make transaction IDs unsuitable for their primary purpose.
One proposal for dealing with malleability attacks it to have a second transaction ID just for the purpose of defeating such attacks. This transaction ID wouldn't cover the signatures. But it wouldn't be suitable for checking transaction validity, forcing a system in which there are two types of transaction identifiers.

You are right, you could have 2 transactions with the same ID, one valid and the other one invalid. So the client would discard the invalid, don't even relaying it to the network and keep the valid one on the pool. The client already works this way, so the only thing that must be changed is the way transactionID is generated, signatures shouldn't be included in the SHA256, because the signatures are already checked later.

@masize I think you're missing the point. If you changed the way the transaction ID is generated, the client would fail horribly. It might refuse to relay a transaction because it knows that transaction is invalid even though it's actually valid. Attacks on the network would be possible where you corrupt a signature, causing nodes to "know" that the transaction with that ID is invalid.

please let me know what I'm getting wrong.
The steps are as follows : Check transaction ID hash => Check unspent outputs => Check valid signatures => Relay tx to the network. If any of those fail, transaction would be discarded, txID is not tagged or cached as invalid forever, just this transaction as received is invalid. The current client does all this procedure. TransactionID generation give no assurance on valid signature, client must check those anyway.

@masize How could you ever know if you had a correct copy of a particular transaction? Currently, you can hash it and compare it to the transaction ID.

Thats the point, Doesn't matter if transactions are different as long as both do exactly the same thing.

@masize It does matter because some could have invalid signatures. A transaction with an invalid signature doesn't do anything.

Why would anyone have a transaction with invalid signatures?
The client ALWAYS checks the signatures, the txID is just a hash. The only thing that could happen is that you could relay the same transaction to the network with different signatures, but BOTH valids, and nodes would have the same transaction with different signatures, but ALWAYS valid, as the client ALWAYS checks this, but who cares, as long as they are valid and do the same thing, there's no problem at all.

@masize There's a huge problem for anyone who ever needs to store the transaction and index it by hash. Essentially, you're solving the transaction malleability problem by making transaction IDs useless as authoritative keys.

I'll have to read up on Txn malleability but I do know that without actually signing the txn there's no way to prove ownership of the public key being spent.
Signing the txn will always give a different signature (unless the k value is kept constant which was the source of the August 2013 Android bug). Ecdsa simply validates that the txn is signed with the correct key. A hash of the inputs and outputs is what's signed. So yes, that shouldn't change, but because the ECDSA signature does change the protocol is simply validating the signature.

Thanks for the response but Transaction Malleability has nothing to do with ownership proof.


text:
What happens if I fork an older block chain and rejoin the network later?
Let's call it a "time machine attack".
Suppose I have an isolated network of n clients with a significant hash rate, install my client software on that and start mining on an older blockchain.
Since the difficulty was lower back then, I get lots of blocks in very little time, but my timestamps could be forged, and the difficulty increases, but not so much as the real network difficulty (however I am fully in control of my network and can do anything on my blockchain, increasing and decreasing difficulty if I need to, faking timestamps, and so on).
As soon as my chain is longer than the official one, I rejoin the network.
What happens then? Does my longer chain overwrite the other one, effectively giving me all of the bitcoins ever generated after my split?
Update - explanation
The attacker doesn't need 51% of the computing power of the network. A single pool could start hashing from some point back in time when the difficulty was low, then increase or decrease it in order to grow the blockchain faster. The last "few" blocks could even be harder than the official network difficulty (and take days instead of minutes).
The attacker can do what he wants on his network, changing network difficulty and forging timestamps, and the length of the blockchain does not depend only on the current network difficulty. I think it's possible to generate a longer blockchain with "relatively low" hash rates (2% - 20% of the network), it could take months (instead of years) but it could be possible in theory.

You would not be able to fool any bitcoin client that had progressed further than your point of split.
However if you fed your blockchain to a client that was starting at the beginning or just starting from before you point of split you could feed them false blocks indefinitely as long as you were the only other peer they talked too.
I don't think it would be a practical attack. There is no practical way to prevent them from talking to another peer, and the moment another peer presents a blockchain of greater difficulty yours would be ignored.

How can it be longer if you don't have more computing power than the entire network? Remember that the network converges on the chain that was hardest to generate, not the one with more blocks.

I don't have more computing power than the entire network now, but a recent GPU alone has more than the entire network at the beginning. Say I have enough computing power multiplied by the number of blocks I can generate before rejoining. It would take some time, but it would still be less than 51% of the network (today).

For example: a big mining pool suddenly "disappears" from the official network, and starts mining alone on an older blockchain, getting the equivalent of 1 year (or more) in a few weeks/months, at a constant hash rate. The last few blocks could even be harder than the official network (it would take some hours instead of 10 minutes, but why not...). So, now I have the longest chain. Who is king? :-)

Longest chain means highest total difficulty NOT largest number of blocks.

You misunderstood concept of "longest chain". It must have higher total cumulative difficulty, not just more blocks.

You misunderstood my explanation: "higher total cumulative difficulty" can be forged, if you have enough hash power, (but still much less than 51%). I can go "fast forward and back" in time, then get some high diff blocks (at current network difficulty or higher) at the end, cumulative difficulty is the problem, not the solution.


text:
Understanding cryptocurrency exchange working
Exchanges are 2 types : centralised and decentralised. Now, if any new user is coming to the centralised platform then they need to deposit BTC (taking BTC as an example). For depositing, each time exchanges create a public key(give this to user to deposit) and private key(hold by exchange). Now, let user have 10 BTC in his wallet (Which he can check in bitcoin explorer). In a day, he traded his 2 BTC against ETH. This 2 BTC he traded with 10 people. So, now his actual balance at his public key would be 8 BTC. And all transaction which he did with 10 people will also get reflected at bitcoin explorer. Trading also works instantaneously. it means as soon as someone sell or bought BTC, his balances get chcngaed accordingly BUT bitcoin actual transaction take 10 minutes for getting first confirmation. So, how the actual system is working behind the scene?

When users deposit cryptocurrency into an exchange, the exchange takes custody. The cryptocurrency is now the exchange's and the exchange adds a corresponding balance for the depositor to their books. The user specific address is only used to identify whom to credit the deposit to. After the deposit, there is no reason to keep track of the funds in the context of the user.

So, now his actual balance at his public key would be 8 BTC.

Tracking all trades on the blockchain would be incredibly inefficient. Rather, the exchange updates the user's balance in their books to increase the ETH amount and decrease the BTC amount. Any balance changes of the user on the exchange are only tracked in the exchange's books until the user withdraws from the exchange. When they withdraw the exchange sends an on-chain transaction to the user and reduces the user's balance by a corresponding amount.

Building off Murch's response, centralized exchanges keep an internal balance sheet for accounts similar to how actual banks do. All trades that happen on the exchange happen between users who have an account and deposited money on them, so the exchange is able to adjust these balances instantaneously - it is just an entry in their database.
To get even more specific, most exchanges have a cold storage, multi-sig vault where users deposit funds to. When the user wants to deposit funds into the exchange, the exchange will generate a new address for the cold storage vault, associate that address with the user's account id, and then monitor that address for any incoming deposits. Once a deposit is received, that user's account is credited.
Until the user decides to withdraw their funds from the exchange, none of the activity is recorded on the blockchain, it is all just database manipulation within the exchange's database.
This is why efforts like Proof of Keys and Proof of Reserves are important, because in theory these exchanges could be adding entries into their database that say they have more Bitcoin in their vaults than they actually do, kind of like how fractional reserve banking works.


text:
Can we shutdown the Bitcoin blockchain?
I maybe wrong but what I think is the blockchain ledger exists as long as there are active nodes in the network. So can the Bitcoin blockchain be shutdown by shutting down power supply across the globe? What would happen in such a case?
I think the transactions that were not added to the ledger would definitely be lost and they would have to be re initiated. I think power shutdown would create inconsistency between different instances of the ledger at different nodes and even if the power is back we wouldn't be able to pick a safe state of the ledger.
What are the possible ways to shutdown the blockchain without having to go to each and every node and taking it down?

I'm voting to close this question as off-topic because it's about blockchains in general and only bitcoin questions are on-topic here.

@David Schwartz I know. But this is the most closest stack I found related to blockchains. Do you know a stack site for asking blockchain based questions in general? Is it better to post blockchain based questions on CS Stack Exchange?

@DeepBodra see the help centre: https://bitcoin.stackexchange.com/help/on-topic . On-topic subjects include questions about: "technologies or events that are applicable and relevant to the Bitcoin network and users of the Bitcoin currency in the past, present, or future". This question is arguably relevant to the Bitcoin network, and with a small edit it could easily be made more explicitly BTC-related --  but I still think it is reasonably fine as in.

@chytrik Unfortunately, what's happening is that it's getting bitcoin-specific answers that are not marked as bitcoin-specific and that's causing people to learn things that are actually wrong. This kind of question degrades the value of the site to future visitors. The alternatives seem to be changing the question to something the original poster didn't ask or closing it.

@DeepBodra Unfortunately, I don't have an answer for you. This site recently changed from being blockchain in general to being bitcoin specific and it's made the site much less educational and lead to it having a massive amount of content that's just wrong. (Because it's not clear whether it's bitcoin specific or not.)

@DavidSchwartz I have noticed this as well, I think on some level this could have been expected to happen for at least some amount of time as the new site rules become more well understood. In this specific case, I think editing the question slightly will be most beneficial, it is a basic question that may benefit others to read. Perhaps OP is looking for a generalized answer, but 'blockchain' has become such a generalized marketing term at this point, such that it is probably almost impossible to write a legitimately comprehensive answer to cover all cases of 'blockchain tech'.

@chytrik I don't think it would not have been that difficult to produce a high-quality answer that explains, fundamentally, why blockchains are difficult to shut down without going into the specifics any particular blockchain. But instead, we get one that blurs attributes of bitcoin with attributes of blockchains. Hopefully, the changes to the question will reduce the risk of confusion.

So can a blockchain can be shutdown by shutting down power supply across the globe? What would happen in such a case?

If the world's power supply was suddenly cut off, then everything electronic would stop working. The blockchain would be 'frozen', each node would retain its local copy, but be unable to communicate with other nodes, and miners would be unable to mine and add new blocks.

I think the transactions that were not added to the ledger would definitely be lost and they would have to be reinitiated.

This depends on what happens to the mempool of each node. If any nodes remembered any unconfirmed transactions from before the power loss, they could simply broadcast those transactions to the network once things went back online.

I think power shutdown would create inconsistency between different instances of the ledger at different nodes and even if the power is back we wouldn't be able to pick a safe state of the ledger.

Each individual node, once it came back online, would reconnect to some peers in the network, and listen for information about new blocks, just like normal. The decision about which chain to follow would also be made as per normal: which chain has the most work? Follow that one. If there are multiple choices for a most-work chain, then the decision about which one to follow will be made when the node hears about a new block, which extends one of those chains (the other chain will become 'stale', and will not be a part of the blockchain history).

What are the possible ways to shutdown a blockchain without having to go to each and every node and taking it down?

This is a strength of the bitcoin network: it is a decentralized network of computers, without any real hierarchy or centralized control. You would have to attack every node, otherwise the un-attacked nodes will be able to maintain the network. Given the highly distributed and anonymous nature of the network, this seems to be an extremely difficult task.

I maybe wrong but what I think is Blockchain ledger exists as long as there are active nodes in the network. So can a blockchain can be shutdown by shutting down power supply across the globe? What would happen in such a case?

If all the existing power supply suddenly and by magic shut down the ledger would be simply frozen but it would not disappear, it will stay on hard drive ready to be used and updated again as soon as the power back up and our bitcoin client finds nodes to talk to.

I think the transactions that were not added to the ledger would definitely be lost and they would have to be re initiated. I think power shutdown would create inconsistency between different instances of the ledger at different nodes and even if the power is back we wouldn't be able to pick a safe state of the ledger.

Transaction that were not treated before the power supply shutdown would stay in the mempool, so in the memory of different nodes just like there is transaction currently waiting and stacking up right now they would also wait to be updated and treated by the nodes

What are the possible ways to shutdown a blockchain without having to go to each and every node and taking it down?

They might exist different scenario but If I have to do it and I have superpower I would put a sleeping virus in every bitcoin client nodes that once activated prevent it from communicating with other nodes and then at a predetermined time activate the virus, all the nodes would be solely mining the blockchain just like Satoshi in 2008.

Welcome! I think one important thing to keep in mind is that even under normal circumstances the nodes are not necessarily all "synchronized." In fact overcoming this lack of instantaneous synchronization yet still ensuring eventual agreement on a single history is one of the amazing feats of bitcoin's protocol.
In other words, at any given time node A on one side of the world might have a different UTXO set (the set of confirmed transactions which is part of each nodes state) than node B on the other side of the world. This can happen because one of the nodes is still processing/verifying a block while the other node has already finished such processing. The important thing is that the vast majority of the time, the vast majority of the nodes also agree on the vast majority of the UTXO set.
Now, as to your question about possible ways to shut down a blockchain without having to go to each and every node and take it down, I can't think of any good ways. Additionally, even if someone could manage to do that, there is no guarantee that they also:
a) destroyed all copies of the block chain (e.g. what if someone printed the chain out on paper or in some other medium?),and
b) destroyed all copies of the source code (e.g. code that is used to interpret the chain and rebuild the state; such code could also be stored in some other medium)
I believe both of those things would also need to be addressed in order to truly shut down a blockchain.


text:
How do we know which soft-forks were implemented using User-Activated Soft Fork or Miner-Activated Soft Fork?
This seems to be the most common definition

When a majority of miners upgrade to enforce new rules, it is called a miner-activated soft fork (MASF). When full nodes coordinate to
enforce new rules, without the support from miners, it is called a
user-activated soft fork (UASF).

How do we know which soft-forks were implemented by full nodes upgrading clients or miners upgrading clients? Is there any way of knowing, or do we just guess?
https://github.com/bitcoin/bitcoin/blob/master/doc/bips.md

Most past soft forks have been deployed with miner activated soft forks, and we know this because the deployment mechanisms used only allowed for miner activated soft forks. For a user activated soft fork to happen, the deployment mechanism would have to have some form of activation that is not reliant on miners. Thus far, the activation methods used have all deployed based on miner behavior and not a flag day.
There are two exceptions to this where it could be argued that those soft forks were user activated and not miner activated. The first is BIP 16 which ostensibly activated via a flag day (i.e. all blocks after a certain time must support BIP 16). However the flag day was determined only after a certain threshold of miners signaled support for BIP 16 in their blocks. So while the code was a flag day, the process was still a miner activated soft fork, just done manually rather than through code.
The second exception is segwit. However segwit used BIP 9 as its activation mechanism and that is entirely dependent on miner signaling. So technically, it is strictly a miner activated soft fork. But you could argue that it was user activated because users were running software that would have activated on a flag day, but miners began signaling readiness before that day. So the exact mechanism was through miner signaling, but perhaps it was user activated because users threatened miners by running software that had a user activated mechanism.
For future soft forks, it is likely that a combination of BIP 9 (MASF) and BIP 8 (both MASF and UASF) will be used. You can then determine whether it was miner activated or user activated. For both, miners can signal in their blocks to activate the fork. For BIP 8 only, after a certain time, the fork will activate via a flag day, so if that occurs, then it is a user activated soft fork.
The only soft fork that should be classified as a UASF is BIP 30. This was activated by a flag day and miner signaling was not officially part of the activation.

Thanks Andrew. This was exactly what I was looking for. I was not entirely sure which type of soft-fork to classify BIP30 as, This is where I got stuck. I never really thought about BIP16 deployment as being different in code and practice. I will have to think about this a bit.

I think the SourceForge soft forks may also have been UASF, particularly 1MB Block Size.


text:
Who were the Bitcoin maintainers before 2014? When did maintainers gain/lose merge access?
The Bitcoin lead maintainer and co-maintainers have merge access (i.e. they can approve changes to Bitcoin source code). We can see who the maintainers were since 2014:
https://github.com/bitcoin/bitcoin/blob/master/contrib/verify-commits/trusted-keys

Is this a complete/accurate list of maintainers since 2014?
Is there a list of when maintainers had merge access before 2014?

Related: https://bitcoin.stackexchange.com/q/176/48884. Also a reddit post with tons more info on committers: https://www.reddit.com/r/Bitcoin/comments/5riy1j/list_of_people_who_have_had_commit_access_to/

Incredible work Andrew. Cheers :-)

@AndrewChow you should copy your reddit post as an answer here for archiving sake, it seems to be exactly what OP is looking for.

I've updated the post in the other question and will mark this as a duplicate.


text:
Change difficulty using bitcoin-cli/bitcoind in regtest mode
Is there an RPC command which would allow me to generate blocks under a given difficulty? For instance, I would like to do something like this:
bitcoind -daemon -regtest
bitcoin-cli -regtest setdifficulty 100
bitcoin-cli -regtest generatetoaddress 101 bcrt1qlckxrvk56kezy35xuw3tk5w5gkvnmjl0cahw3u
bitcoin-cli -regtest setdifficulty 1
bitcoin-cli -regtest generatetoaddress 101 bcrt1qlckxrvk56kezy35xuw3tk5w5gkvnmjl0cahw3u

I am aware that it would result in an invalid blockchain, since it would not match the expected target recalculation of the Bitcoin protocol. This "detail" aside, is there any way to use bitcoind/bitcoin-cli to perform something like this?

No there isn't. Regtest is not designed to test mining ASICs and the motivation for such a RPC seems limited given regtest is for local testing.
However, if you want to experiment with mining on a network with limited network hash rate you could either try mining on the default Signet administered by Kalle Alm and AJ Towns or you could set up your own custom Signet network and then use the Signet mining utility in this PR (19937) to mine blocks. AJ Towns lays out some examples in a PR comment.
e.g.

Generate blocks indefinitely, at 10 minute intervals, paying block
reward to the given address, using bitcoin-util to do multi-threaded
proof-of-work:
 ./contrib/signet/generate.py --cli='./bitcoin-cli' generate 10 --block-time=600 --address="tb1..." --grind-cmd='./bitcoin-util grind'


text:
Is that proved that breaking ECDSA as hard as solving ECDLP?
Is that proved that breaking ECDSA (Elliptic Curve Digital Signature Algorithm) as hard as solving ECDLP(Elliptic Curve Discrete Logarithm Problem)?

This is better suited for https://crypto.stackexchange.com/

Yes, correct. Thank you. I'll remove from here.

I think it is fine here. libsecp is Bitcoin https://github.com/bitcoin-core/secp256k1


text:
Are P2SH transactions with disabled opcodes relayed/mined?
Many OPCODES like OP_AND, OP_CAT, OP_SUBSTR are disabled as described in https://en.bitcoin.it/wiki/Script. Although, with Bitcoin Core 0.10.0 Gavin Andresen proposed "any Script" in P2SH transactions to be relayed/mined:

I'm still unsure if this applies disabled opcodes. I've come across a Mainnet coinbase transaction with disabled "OP_AND" opcode here, and a Testnet transaction here. I'm very curios if P2SH transactions with disabled opcodes in the redeem script relayed and mined on the BTC Mainnet.

On that wiki link you posted it states:

If any opcode marked as disabled is present in a script, it must abort and fail.

An attempted UTXO spend with a failed script will not be relayed between nodes running Bitcoin Core (and other major implementations) and certainly won't be included in a mined block.
The mainnet coinbase transaction you link to includes an OP_AND in an input scriptSig. But input(s) in a coinbase transaction are ignored as it is mining new Bitcoins and is not spending from any output from another transaction.
The testnet transaction you link to includes an OP_AND in an output scriptPubKey. It hasn't been spent and can't be spent because there is no scriptSig that can spend from it.
BIP-Tapscript (BIP 342) proposes to make disabled opcodes like OP_AND OP_SUCCESSx which would mean that the Tapscript would automatically succeed if the opcode is encountered. This is so that new opcode functionality could potentially be introduced in a future soft fork without breaking backwards compatibility. But Taproot/Tapscript is not yet activated on the network at the time of writing (October 2020).


text:
Taproot - Eliminating Key Path
I want to struct a transaction output the key-path of which I want to be eliminated, so enforcing only the script path. In order to eliminate key-path, in my opinion, I should set "Internal Public Key" to "0xFFFFFFFFFFF.." making the corresponding "Internal Private Key" unknown so the key-path signature is. By doing so anyone interested in unlocking this output, including me, must follow the script path by fulfilling its conditions. Correct me if I'm wrong. Thanks.

BIP 341 answers this. It is generally important to be able to prove that the internal public key that you choose is a point with an unknown private key. Otherwise a malicious party could hide a key path spend in the P2TR address, pretend not to have the private key for that internal public key but actually possess it.
BIP 341 also provides instructions on how to avoid leaking the fact that the P2TR address had no possible key path spend when it is spent from. If you don't care about this privacy leak you could just take the suggested point H.

Pick as internal key a point with unknown discrete logarithm. One
example of such a point is H =
lift_x(0x0250929b74c1a04954b78b4b6035e97a5e078a5a0f28ec96d547bfee9ace803ac0)
which is constructed by taking the hash of the standard uncompressed
encoding of the secp256k1 base point G as X coordinate.
In order to avoid leaking the information that key path spending is
not possible it is recommended to pick a fresh integer r in the range
0...n-1 uniformly at random and use H + rG as internal key. It is possible to prove that this internal key does not have a known
discrete logarithm with respect to G by revealing r to a verifier who
can then reconstruct how the internal key was created.

A closely related question is on how to grant access to a key path spend but not a script path spend.

Thanks! I publicly need to prove the fact that key-path spending is not possible at all. So either pick a point with an unknown discrete logarithm or pick a public key whose private key cannot be constructed.

The internal public key needs to be a point on the curve, different from infinity. So that implies that it has a discrete logarithm.

Hey, can you point out how to achieve this in JS, please? I'm having a hard time finding a library that does lift_X.


text:
Block hashing algorithm
The block header contains a number of things, among them the Merkle root hash. Wouldn't this mean that every time a new transaction is added to the block (before its mined), that all nonces previously attempted need to be tried yet again?
Wouldn't this discourage miners from adding transactions to the block if they have to hash everything with nonces again?
Also, is it possible that no nonces could produce a valid successful/winning hash?

You may want to consider breaking up this into multiple questions.  The third and fourth questions, in particular, don't directly relate to the overall question.

Sometimes miner's will mix up the transactions on purpose to get a new hash if they aren't able to find a valid nonce in a reasonable amount of time.

Hi @Jestin, as suggested, I've removed the forth question but I still think the 3rd question relates as per my other comment. Cheers.

@alexkb, the point isn't to have related questions grouped correctly.   The point is to have questions be independent posts so that people with the same question can find it easier.  Each of these could have been independent, and therefore simpler for someone to find in the future.

Wouldn't this mean that every time a new transaction is added to the block (before its mined), that all nonces previously attempted need to be tried yet again?

Yes, but "need" is a bit too strong of a word in this context.  Because you can't predict the outcome of a hash without performing the hash, each new hash attempt is just as likely to meet the target as the previous one.  As such, yes, it is possible that a nonce that has already been attempted will now succeed given that the merkle root has changed...but it is no more likely than the nonce you were about to attempt before adding the new transaction.  There's no real reason to re-try nonces.  The odds aren't any better.

Wouldn't this discourage miners from adding transactions to the block if they have to hash everything with nonces again?

No.  Because the odds do not change, there is no negative incentive to adding a new transaction.  However, transactions come with miner fees, so the positive incentive is still in place.

Also, is it possible that no nonces could produce a valid successful/winning hash?

Yes, it is possible, although not probable.  Remember that a successful hash is one that is less than the target.  These means there are quite a large number of nonces that will be successful, however, as a set they represent a very small proportion of the total possible nonces.  Even if hypothetically there is a block header that is "unmineable", a competing block would not be.  A block will still be found, and it will likely have a very similar set of transactions in it.

Great answer(s), thank you. I think the key take away is that just because a miner has performed a hash that was unsuccessful doesn't increase their chances of finding the winning hash with the next nonce - this is because there's no guarantee that any nounce with a Merkle root hash will be the winning hash. Have I got that right?

There are no transactions dynamically added to a block that miners try to solve.
Transactions are taken from mempool in the moment the last block was mined and the merkelroot has to be calculated only once for those picked transactions.
It´s every miners free choice to discard transactions from this list (e.g. tx with low fees) and the list is limited by the 1MB blocksize. So it´s approximately 2000 tx per block.


text:
Get latest Bitcoin transactions in real-time
How is it possible to get the latest transactions txid in real-time? I mean to implement something similar to what blockchain.info does on its homepage. How do they do it to get all transactions info that happens on the network? I'm talking about zero-confirmation transactions.
EDIT
I have found out the API getrawmempool. Does this answer the question? Does it contain all the real-time transactions in the network for the current live block?

Yes, getrawmempool will contain all unconfirmed transactions which have been verified and accepted as valid by your node but have not yet been included in a block. Due to transaction malleability (or moreso a misunderstanding of how to check transactions going through) and double-spends I would use this for notification purposes only, unless of course you value being able to credit a client immediately and are willing to deal with a few reversed payments in order to provide this convenience.

getrawmempool is the solution as mentioned in the edit of the question and validated by the comment in the question.

blockchain.info offers a web socket for real time information about blocks and tranasactions.
Have a look at https://blockchain.info/it/api/api_websocket
-- UPDATE
to avoid relying on third party services, I would now suggest to use a bitcoin node with zmq

"I mean to implement something similar to what blockchain.info does on its homepage. How do they do it to get all transactions info that happens on the network? I'm talking about zero-confirmation transactions."

-walletnotify=<cmd>    Execute command when a wallet transaction changes (%s in cmd is replaced by TxID), if you only care about wallet transaction.
If you want any TxID, I think that is not possible with the standard client. You could however program something yourself that listens for message passing between the Bitcoin software and the operating system, or you can fork the client of course.
In my opinion the layered approach is preferable, as it is more flexible and secure (although you have less possibilities). For the layered approach I see two possibilities:

The Wireshark option (i.e. hooking into the Windows networking implementation and filtering Bitcoin messages).
Create a SOCKS server and let the Bitcoin client connect to the network through it (using the -proxy option), and then log and process all passing messages.

A synced bitcoind daemon has a command showing txid of transactions passing thru it. It may be impossible to see the whole network at once from one spot, or know whether you can or not.


text:
Where I can download financial & technical data for graphs of bachelor thesis?
I'm writing bachelor thesis and I have to do some graphs. I think that I can take snapshot, but it isn't good idea. Can I somewhere download historical data about Bitcoin?
EDIT. I need technical data like there: https://blockchain.info/charts + financial data like USD/BTC etc. for create graphs. I can't use screenshots for bachelor thesis...

Bitcoinchain.com allows you to export market data. You can use that to create graphs for the historic exchange rate.
If you could be more specific which graphs you want to create, one would be able to point out sources for the respective data. I don't think anyone is interested in searching the source data for all of the charts on blockchain.info, while you were only going to use one or two.


text:
Can a publicly visible but privately mined block chain be trusted?
Outside of Bitcoin, if a company set up a blockchain like structure that can be publicly accessed but only the company itself mines blocks, does that mean the blockchain can't be trusted?
I have only a rough theory in mind, no hard examples. Say a card game was implemented using a block chain. Card collections and game stats are stored in a blockchain and accessed by all the players of the game. None of the players mine any of the blocks. As players play the game, trade cards, open booster packs, etc they submit varying types of transactions to the company and the company verifies and adds the data to the blockchain. But the company mines the blockchain to solidify those transactions themselves and not allowing clients to mine at all.
I can see how money (Bitcoin, fiat, etc) would change hands for rare cards or booster packs. Inherently there would be value for the cards not unlike Magic: The Gathering or Pokemon having rare cards that people want. Can a player trust the company's blockchain? Since the number of miners is controlled by the company, what would be an acceptable difficulty for the miners to have to reach? Wouldn't too low of a difficulty not be enough proof of work to be trusted?
Since it's in the company's best interest to promote the proper use of the chain it's not really a 51% attack (technically a 100% attack). The users would know they're not mining from the beginning. Would this be any different than the company choosing postgres over MySQL? It's just the company's choice for storage, right?

What's the point of mining or having a blockchain if only a single entity has write permissions?

@m1xolyd1an Clients will still submit "transactions" (game stats, trades, etc) to be verified and stored. Only the company will mine. I'm hoping that data transparency and data integrity can be preserved by having a publicly visible blockchain but since there's no reward to users for mining there's not much reason they will actually mine.

I don't think a reward could be set up because you can't split the commodity in this scenario. If players set up pools and the reward for solving a block is a booster pack or a full deck then how could the pool possibly divide the cards up? Without pools then individuals with large mining rigs take over and have a monopoly which I think would spoil the game.

A blockchain will make sense if you remove the company from your idea.  If it's a completely decentralized game, with no company running or creating it, then you should look at using a blockchain.

I'm hoping that data transparency and data integrity can be preserved by having a publicly visible blockchain but since there's no reward to users for mining there's not much reason they will actually mine.

There's really no point. There's much simpler ways to provide the same data transparency and integrity guarantees. The only reason to use a blockchain would be so you could say "it's cool because we use a blockchain".
You can use transactions without a blockchain. You can map ownership to cryptographic identities without a blockchain. The sole miner just sign that it's accepted a transaction rather than doing anything like mining.
A blockchain doesn't do anything for data transparency. Just because you have a blockchain doesn't compel you to make it publicly accessible. And you can make data publicly accessible regardless of its format, blockchain or not.
And there's lots of costs associated with blockchains, none of which give any benefit. The whole point of mining is to allow you to determine what is truth without a central authority to say what is true. If you have that anyway, what's the point?
This is such an easy to solve problem and blockchains have such complex tradeoffs, it would really only make sense if you were too lazy to build something more appropriate and just happened to have a blockchain solution you could easily drop in.


text:
Does an uneconomical output need to be kept in the UTXO set?
Even though it is consensus valid to spend a UTXO of an amount that is lower than the extra fees it would cost to include it in a transaction, most nodes wouldn't relay them because of the minRelayTxFee. Does it make sense to keep those outputs in the UTXO set, increasing its size?

Bitcoin Core removes provably unspendable outputs from the UTXO set, since by definition we know they will never be used to verify new transactions.
One could, in theory, do the same for UTXOs that are highly unlikely to ever be spent, such as those that are highly uneconomical as per your question. However, this introduces a very real risk of splitting yourself off the chain. If just a single one of those UTXOs is spent in a confirmed transaction, your node would reject that block (and any subsequent blocks building on it) since one of its transactions is spending an input that according to your database does not exist.
While it is reasonable to assume that most uneconomical outputs will not be spent (in the current fee environment), it is extremely risky to assume that none of them will be spent. They can be spent accidentally, as part of L2 protocols, or by an attacker that wants to exploit this vulnerability. As such, removing any UTXO that is not provably unspendable would be unwise.

Removing a UTXO from your UTXO set, if it still can be spent, however unlikely that is to actually happen, is a hard forking change. It's not something you should do without extremely widespread understanding that the entire ecosystem will do the same.

@PieterWuille: Only a hard forking change if you rejected a spend from that UTXO in a mined block right? If you removed it from your UTXO set then it was spent in a block you would request that transaction and the transaction creating that UTXO in the first place from your peers and update your UTXO set accordingly?

The Bitcoin protocol doesn't work like that. There is no way to synchronize individual transactions or UTXOs. If you delete something from your UTXO set, then by definition you'll reject any block which tries to spend that UTXO. Or you're talking about a wholly different model of what "UTXO set" means.

I imagine a wallet could move dust UTXOs into a separate dust UTXO set so that most transactions would verify with only reference to a smaller index for the non-dust UTXO set. Very few  transactions (perhaps almost none ever) would need the dust UTXO set to be examined. Whether this  significantly speeds up processing of incoming blocks might be doubtful.

@PieterWuille Thanks for this clarification I was thinking that it was possible to erase an Output from the UTXO set but still accepting blocks containing those outputs but yeah you cant verify them then.

@RedGrittyBrick I think moving dust UTXO's to an on-disk storage could be an improvement , but I don't have the skills to predict if it's worthwhile or the risks associated with it

@leslie Bitcoin Core keeps the UTXO database on disk, with an in-memory cache for recently-modified entries.

@PieterWuille Disallowing spending a spendable UTXO seems to tighten the rules - therefore it seems to be a soft-fork change, or am I missing something?

@ChrisChen You're right.

One example of an uneconomical (meeting the dust threshold) UTXO are haircomb commitments. While they could've been encoded as OP_RETURNs, it's preferable to bloat the bitcoin UTXO set. Without this, haircomb couldn't benefit from a fast Bitcoin sync technology that could be invented in the future.


text:
What states is Kraken doing business in
Is there a list of states that the kraken exchange is currently doing business in?

Kraken also known as Payward Ventures Inc is currently registered as a Money Service Business in every single state in the US.
As far as I can tell however they will still not do business for the 2nd 3rd or 4th tier verifications in any state. I have no idea why this is.


text:
Is the a library for generating signatures and transactions for scrypt coins?
I'm looking for a library similar to this bitcoin one but that works with scrypt based coins.

The use of scrypt vs SHA256 has an impact on proof of work but no impact on the signature scheme or transaction structures. The only thing that should impact you when using this library is the address prefix that differs with altcoins.
The standard address prefix for Bitcoin is 0. For Dogecoin for example it's 30. So you could reuse that same library and generate valid Dogecoin transactions just by passing that additional parameter to the pubtoaddr function like so:
pub = pubtoaddr(pub, 30)

authors: Евгений Вербицкий