text:
Bitcoin: How does Proof of Work and single transactions align?
I'm reading papers since hours and still don't get one thing:
A bitcoin transaction is an information like "Bob sends 2 BTC to Alice". This message is signed with Bobs private key and needs to be added to the blockchain that is distributed to every participant. I also got that you need to calculate a Proof of Work to be able to add something to the chain. Or in other words: The other participants won't accept your block without the proof of work.
How does Bob get this proof of work or how does Bobs message went to a person who has such I proof? I can't believe that every person making a transaction needs to calculate it's own proof. Is it really like that? If not: How does Bobs "message" is transmitted to a person who owns such a proof, how can the person make sure that Bobs message is a valid transaction (and not only garbage) and what the motivation for this person to process Bobs message at all?

How does Bob get this proof of work or how does Bobs message went to a person who has such I proof?

Bob offers a fee to whoever attaches his transaction to a block with the necessary proof of work. Now Bob wants to get his transaction to someone who can attach it and people who generate proof of work want Bob's transaction so they can get the fee.
When two people each want to communicate with each other, there is no difficulty in arranging that communication. We have the Internet.

I can't believe that every person making a transaction needs to calculate it's own proof. Is it really like that?

No. Those who want to execute transactions offer a fee to those who calculate proof of work. This is a mutually beneficial arrangement.

If not: How does Bobs "message" is transmitted to a person who owns such a proof,

Over the Internet.

how can the person make sure that Bobs message is a valid transaction (and not only garbage)

They just check to make sure whether it follows all the rules, which everyone knows. If so, it's valid. If not, it's invalid.

and what the motivation for this person to process Bobs message at all?

They want to make as much money as possible for the proof of work they provide. So they actively search for the transactions that offer the highest fees to them.

As a rough quick answer for any Blockchain not just Bitcoin..
-users authenticate their transactions with Cryptographic signatures, and submit a transaction Fee
-Transactions reside in the mining pool for miners to select usually thru an auction where fees r considered like bids, the auction format may differ from one Blockchain to another
-Miners verify the signatures of TXs upon selection, and collect the fees+block reward if the block was approved.
Here's a few diagrams from my early days in cryptocurrency, can't remember the source video

Ps

The word "Encrypting" in the images is not quite accurate, what is done is digitally signing using a public-private key Encryption Algorithm (elliptic curve in Bitcoin & many cryptocurrencies)
-Transactions remain publicly readable to the miners in the mining pool (unless something called "commit-then-reveal" scheme is used to deliberately hide the content of a TX from the miners thru encryption)
Checking the authenticity means verifying the correctness of the digital signature

The source of the figs
https://youtu.be/E2JRnFgrztM
( with only the clarifying red & blue text added by me after the comments)
It was like my first let's get an idea and see if it worth further study & research

This series of graphics is incorrect and misleading in several ways: 1. Bitcoin transactions are not encrypted. In fact the Bitcoin protocol does not use encryption anywhere at all. Transactions have cryptographic signatures, which authenticates their content as being issued by the private key holder, and protects them from being changed.

Miners don't "verify the authenticity of sender and receiver". All nodes (of which miners are a subset) verify that the transaction is well-formed and valid. The signatures in the inputs attest to the owner of the funds having signed off, the only check for the receiver is whether the output script is valid.

"The transaction would show that bitcoins are reduced from one wallet, and added to another." ⇒ Wallets don't exist as a construct in the Bitcoin protocol. A transaction consumes and creates Unspent Transaction Outputs.

Not sure what the last slide has to do with the topic. It seems to reference the oracle problem.

Anyway, the text in your answer is much better than the graphics which are not just oversimplified but wrong. I suggest removing the images.

Cryptographic signatures r performed thru elliptic curve cryptography which is a public key Encryption scheme. Authentication is defined as the process of verifying the identity, checking the correctness of the digital signature is one way to do that; ie just different terminologies for the same thing. You r right about the last blockdiagram, It's irrelevant & I removed it

No, while they're both applications of cryptography, signing and encrypting are two orthogonal concepts. The purpose of signing is to produce a zero-knowledge attestation that the signatory holds the private key and to protect against malleation of the message. The purpose of encryption is to prevent anyone but the intended recipient from reading a message. Bitcoin transactions are unencrypted and public information, but immutable due to the signatures.

Signing and encryption in RSA use the same algorithm with different inputs, which is the exception rather than the rule. Bitcoin does not use RSA, it uses ECDSA. ECDSA has nothing to do with encryption, it is only a signature scheme.

Elliptic Curve Cryptography follows the same "exception" as u call it RSA follows, all public-private key Encryption Algorithms have this feature of being used interchangeably for encryption & signature, beside I did added a remark to the main answer that the use of the word Encryption in the Figs is not quite accurate

That's not correct, @ShAr. The ECDSA algorithm cannot be used for encryption at all, it is purely a signing algorithm. There exist EC-based encryption algorithms, but they are unrelated to ECDSA. The fact that signing and encrypting are duals of eachother is unique to RSA (where signing is "encrypting with the private key and decrypting with the public", approximately).

ECC is a mathematical equation taken on its own, but ECDSA is the algorithm that is applied to ECC to make it appropriate for security encryption. https://www.ssl2buy.com/wiki/diffie-hellman-rsa-dsa-ecc-and-ecdsa-asymmetric-key-algorithms,  "Cryptography (ECC) is a key-based technique for encrypting data" https://avinetworks.com/glossary/elliptic-curve-cryptography/

Sorry, you're wrong. ECDSA does not do encryption. I am not going to argue further.

"The ECDSA algorithm can't be used for encryption. It's not that there's no accepted way to do it, it's that it's simply not possible to do so." —https://crypto.stackexchange.com/a/24520/8801

Both of u, I wonder why wasn't u that "enthusiased" to discuss the UTXOs lifespan Q that u voted to close & like chasing me( not a certain topic, like I pursue the frobiden UTXOs question) in every answer???

@ShAr: I'm not chasing you, a user flagged this post as "very low quality", so I took a look. After reading, I provided feedback about deficiencies in the content of your post to give you a chance to improve it. When my feedback wasn't convincing to you, I provided further evidence. My goals are two-fold: I am trying to improve the quality of posts on this site (as I have for eight years), and I'm trying to help people to contribute better in the future.--It is my impression that my approach is not working for you. How would you like me to provide guidance next time?


text:
Tor and i2p tradeoffs in Bitcoin Core
What are the different trade-offs involved in options mentioned below that users should consider while using Tor/i2p in Bitcoin Core based on privacy and security?

Only Tor or i2p
1.1 onlynet=i2p
1.2 onlynet=onion

Tor/i2p and clearnet peers
2.1 Tor + ipv4/ipv6
2.2 i2p + ipv4/ipv6

Other combinations
3.1 Tor + i2p + ipv4/ipv6
3.2 Tor + i2p (onlynet=onion and onlynet=i2p)

I am not sure about using 3.1 or 3.2 which was mentioned in a tweet thread. IMO one of them (2.1 or 2.2) should be used. Still doing research and looking for articles that explain trade-offs.
1.1 and 1.2 should be avoided considering the trade-offs mentioned in https://github.com/bitcoin/bitcoin/pull/21157

Not sure what this is asking. Is it asking are there downsides to using Tor, I2P, IPv4 and IPv6 all at the same time? Using all of them (or a subset) clearly offers redundancy and maximal flexibility for connecting to peers. With regards to downsides... is there a way to link IP addresses to onion addresses if they are used concurrently (privacy leak)? Other than that I can't think of any.

@MichaelFolkson Can we discuss this in chat? There are few things that I will be able to explain better

I think this question would be improved (it is a bit convoluted at the moment) if it was merely "What are the trade-offs between using Tor, using I2P or using both in Bitcoin Core?" You could do a separate question on whether using clearnet (IPv4, IPv6) in parallel impacts the usage of Tor, I2P but I suspect the answer is no (other than the obvious that when using clearnet you leak your IP address) The answer basically just answers that anyway.

Obsc location
Anon P2P
Bridge
Dissoc

Tor Only
Yes
Yes(Tor)
Yes(Tor)
Yes

i2p Only
Yes
Yes(I2P)
No
Yes

Tor+i2P
Yes
Yes
Yes(slow)
Yes

Tor+i2P+IPv4
No
Yes
Yes(fast)
No

https://www.reddit.com/r/TOR/comments/o5z4r1/tor_and_i2p_tradeoffs/h2sw202?utm_source=share&utm_medium=web2x&context=3
Basic differences between Tor and i2p: https://github.com/prayank23/Learning-Bitcoin-from-the-Command-Line/blob/i2p/15_0_Using_i2p.md

Only Tor or i2p

Tor/i2p and clearnet peers

The onlynet=onion configuration option can potentially ensure the node attempts to only
connect over Tor. It is more private when you combine it with no reachable IPv4/IPv6 address,
in particular if you want to broadcast transactions without them being correlatable with your IP.
Sybil Attacks
On the other hand, if you only make random Tor connections, you're much more vulnerable to Sybil attacks.
As Tor addresses may be created at no cost, an attacker can potentially flood the network with many Tor
nodes and receive all of the outbound Tor connections an onlynet=onion node makes.
This is significantly less a concern with if you make -addnode connections to trusted peers
(even if they're onion addresses). It's also alleviated with IPv4/IPv6 (especially when using the -asmap
configuration option) due to the cost of obtaining IPs in many networks.
If all of your connections are controlled by a Sybil attacker, they can easily prevent you from seeing confirmed
transactions  and, with more difficulty, even trick your node into falsely reporting a transaction as confirmed
on the blockchain with most cumulative “chainwork”.
Network Partitioning
If too many nodes use onlynet=onion, it could become difficult for onion nodes to communicate with clearnet nodes,
preventing the Tor network from seeing recent transactions and blocks. It is essential that some nodes access both
clearnet and Tor.
https://github.com/bitcoin/bitcoin/pull/22316/commits/2c18e7f6614d665f4eb9891f83391a5317e7d113

Tor + i2p + ipv4/ipv6

If looking for privacy, this may not be the best option however I don't see any security issues with this setup.

Tor + i2p

I would consider this as experimental for now and will update the answer if I find any issues.
For maximum privacy, it is preferable to disable accepting incoming connections.
Related Question: https://tor.stackexchange.com/questions/6930/tor-i2p-on-a-system-and-is-i2p-better-than-tor

Thanks for this. Would I2P + Tor not offer significant extra Sybil resistance, given the attacker can't easily figure out the I2P address that belongs to a certain Onion address (and the other way around)?
Also, can you explain what exactly the "Dissoc" column is about?

Another question, do you know if it's possible to use the unofficial I2P exits (e.g., false.i2p) to connect to clearnet nodes? EDIT: Just confirmed, just HTTP, doesn't help us.

Sybil attacks are possible in both Tor and I2P. Maybe i2p is more vulnerable considering less nodes. Don't think using both offers any resistance. Not sure though. 2. Don't understand how I2P address belong to onion. They are different services running on same machine for Bitcoin Core. 3. Dissoc: Disassociate anon and non-anon transactions

I'm suggesting that it's more difficult to Sybil-attack a node if that node is also getting blocks from a second network, and if it is not known which address the node has on the second network.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
CVE-2018-20586 and RPC AUTH
I was reading about CVE-2018-20586 which allowed injection of arbitrary data into the debug log via an RPC in https://medium.com/@lukedashjr/cve-2018-20586-disclosure-ff3e1ab9a21f
One thing that I couldn't understand:

Valid authentication (username/password/cookie) for the RPC service is NOT required to exploit this vulnerability, only the ability to connect to the RPC port (which is by default only exposed to the local machine).

Does this ability to connect to RPC port without authentication still exist because I had to use basic auth last time I tried sending POST requests locally?

When it says, "connect" it is referring to connecting in the TCP/IP sense. The vulnerability simply required you having the ability to open a TCP/IP connection to bitcoind's RPC port, which is before any concept of authentication even comes into play.
It isn't that it was possible to send JSON-RPC commands to bitcoind without authentication (this was never possible). It's that the attack occurs before authentication takes place.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Offline LN transaction
To broadcast on-chain Bitcoin transaction offline we need transaction hex which should be sent to a device which is online. Example: Government has shutdown internet in an area, text messaging still works so Alice can create a signed transaction, copy transaction hex and send it via text message to Bob who can broadcast this transaction (sendrawtransaction) because Internet works at his place.
How does this work for LN? I have not experimented enough with LN so not sure about everything involved. Tried doing it in Electrum 4.1.2 (testnet) but couldn't find necessary information and commands.

Bob creates an invoice for 0.001 BTC and sends it to Alice using text message

lntb1m1psdd0c3pp539vpce5yrw2zk0w23h0p4pnst58d9jfuugx2qmmmyw8zgz09q03ssp5a9ahkkj45k3e5a3q3ftnagqy27zrm0m7g9aknlrjpu5fce2k27nsdqqcqzynxqyjw5q9qxpq9qsqrzjqwfn3p9278ttzzpe0e00uhyxhned3j5d9acqak5emwfpflp8z2cng85awqqqq8gqqgqqqqlgqqqqqeqqjqtz9qwfn3p9278ttzzpe0e00uhyxhned3j5d9acqak5emwfpflp8z2cngqqqq05qqqqqvsqfqqxkhjszx8ftlnljk0uk6an5d6ksyhxrfgn4x7sgq7j48agj948n9gr2tn64qrxe84jczxwr56dtr9mw7al52awrl7alxn7pu68ndckgqhd7cz5

Alice pastes it in Electrum and click on "save"

Invoice details:

How do I use any of this information from the invoice which can be sent to Bob so that LN 0.001 BTC invoice is paid?
LN commands in Electrum: https://github.com/spesmilo/electrum/blob/3bc8ef6651ed9d9aff0531b3597f80eca4886301/electrum/commands.py#L1001

Lightning has an inherent liveliness requirement too it. To receive payments, Bob needs to be online to setup HTLCs on his commitment transaction and and then reveal the corresponding preimages to claim payments to atomically transfer the funds across the Lightning Network.
If Bob were to go offline due to a region wide shutdown of the internet, he would no longer be able to receive payments on the LN. Alice wouldn't be able to find a route to Bob, since he isn't online!
This is much different then sending bitcoin transactions onchain. Bob does NOT need to be online to receive bitcoin onchain. He can simply text the address to Alice, and then sign on 1 year later and verify Alice made the payment.
Of course, if Bob is a merchant sending goods/services to Alice, he should not send the goods and services until he can verify Alice sent the payment onchain.

Workaround:

Run LN node and wallet on a VPS with some funds
Use text messaging API provided by some third party
Listen for incoming messages
Send text mesaage with LN invoice which will be used to pay automatically
Send sucesss or error message as text message to number which sent the initial message


text:
What is the lowest number of blocks mined in a day?
There were only 67 blocks mined on the 2021-06-27 (UTC).
Is this the lowest number of blocks we have ever seen in a day? Or have we seen numbers counts in the past?

It was lowest since 2011 according to chart shared in a tweet thread:

But it was not lowest overall because there were fewer blocks mined in 24h a few times in 2009.


text:
I'm interested in knowing more about transfer codes withdrawing bitcoin
I purchased bitcoin via blockchain and transfer it to Cardonetradingoption.com via an individual I was corresponding with. I sent this person money three time because he advised me that I needed to do that to upgrade. I started with $800 then he said to upgrade and surpass $5k I have to send in more so I transfer another $1250. Later he wanted more so that I could upgrade pass 10k. He wanted $2400  but I was only able to send him $800.
This was where everything became fishy. I wanted to withdraw and is unable to do so for the past month. The balance only rose to just under 10K because he insisted that I should send the difference, which I didn't have.
I can see the account I open with cardone trading and I can see my balance but I cannot withdraw and when I tried it's asking for a transfer code, which I know nothing about. This individual is refusing to help me to withdraw.
CAN YOU HELP ME SORT THIS OUT. WAS THIS A SCAM WHICH I AM SUSPECTING THAT IT IS.

Yes, this most definitely sounds like a scam.

You got scammed. Your money is gone. Lol. How do people keep falling for these?


text:
Most transactions in a block
Block 689074 had 4,463 transactions.

Is this the highest number of transactions in a block until now?
What are the things that affect the number of transactions possible in a block?

Few observations based on comparison of two blocks: 666666(left) and 689074(right)
Size of transactions:

Type of transactions:

Outputs:

It is a bit confusing that you appear to give the labels for the examples in the reverse order as the images that follow, perhaps you could edit them to have the same order or label the images explicitly. Also FYI, the source of the images seems to have some bugs, since the virtualsize of blocks cannot be greater than 1MvB.

Thanks. Fixed the labels. Checking virtual size bug and reading more about 1MvB limit.

Well, more accurately, the blockweight limit is 4,000,000 weight units, which corresponds to 1,000,000 vbytes. I see three other explorers stating that the block had a weight of 3,993,107 WU (998,277 vB), which matches what my node says.

Is this the highest number of transactions in a block until now?

The highest number of transactions in a block is 12239 in block 00000000000000001080e6de32add416cd6cda29f35ec9bce694fea4b964c7be at height 367853.

What are the things that affect the number of transactions possible in a block?

Besides the block size/weight limit, the primary thing that affects how many transactions can fit in a block is the size of the transactions themselves.
When a transaction is added, it takes up space that other transactions could make use of. So the smaller a transaction is, the more transactions can be included in a block. If we look at the winning block, we see that the transactions in it tend to be very small, measuring 81 (v)bytes on average, compared to the 2-300 vbytes in the blocks that you mentioned.


text:
What is special about blocks 540574 and 564180?
I am running c-lightning with a pruned bitcoind backend. Every day at around the same time, c-lightning asks for a specific block which is already pruned, which looks like this in the logfile (block 540574 in this case):
2021-07-25T10:06:49.026Z UNUSUAL plugin-bcli: bitcoin-cli -rpcuser=... -rpcpassword=... getblock 00000000000000000010b0d763e1835bdb9921ae4b8adbb6d88d8cfcfbc392aa 0 exited with status 1

According to this comment from Christian, it is safe to ignore this warning. But because it appears every second in the logs and it feels spammy, I restart c-lighting and the error is gone until the other block (block 564180) is missing. Then I get the same message in the logs but for block 564180, also at around the same time every day.
I am pretty sure the two blocks have nothing to do with each other and also the blocks are probably specific to a channel (or two) I've opened with my node. So my question in the title is maybe a little bit misleading. That it happens with two blocks is probably also just a coincidence. But what my "real" questions are:

Why is the error gone after a restart? Does c-lightning "forget" it wanted to know about that specific block?
Why does the error appear at around the same time every day?

If this isn't the place to ask this, please tell me! I thought about opening an issue on github. Wasn't sure about this.

There is nothing special about those two blocks, aside from containing a funding transaction that c-lightning would like to verify (causing it to look the block up) and it being pruned in your bitcoind.
Restarting indeed results in the lookup, which is kept in memory until it succeeds or fails, is forgotten giving you the temporary result of no longer spamming the logs. But once a peer tells you about the channel your node will resume it's attempts to verify it. Nodes will send updates every once in a while to update the fees on channels, but i think what you're seeing is a keepalive: a regularly scheduled update that is used to tell the network the channel is not stale/zombie and can be used. This is because you mention it reappearing once a day.
As for a permanent solution I suggest either looking into a Bitcoin backend that supports pruned nodes (bcli shipped with c-lightning doesn't support pruned nodes yet), or use btc-rpc-proxy in front of bitcoind to fetch missing blocks on demand (I personally use this with my nodes wherever I can't have full nodes sitting next to the lightning nodes).


text:
Multisig and RBF after taproot

Are there any benefits of above setup after taproot is activated that improve privacy in any way?
What is the setup?

3of3 multisig address with 0.1 BTC unspent
Alice, Bob and Carol own the keys for it. Alice wants to send bitcoin to another 3of3 multisig address mentioned in red on the right.
Tx1 is RBF enabled transaction with 1 output of 0.1 BTC (Bob) and 1 input (Alice)
This is replaced by Tx2 with 1 output of 0.1 BTC (Carol) and 1 input (Bob)
Tx3 is replacement for Tx2 which has:

2 outputs: Carol (0.1 BTC) and 3of3 multisig destination (0.1 BTC)
2 inputs: Carol (0.1 BTC) and 3of3 multisig origin (blue)(0.1 BTC)

Why not do step 5 and skip everything else?
Difference: If someone is saving mempool logs and these transactions are present in it, Tx2 and Tx1 will be known and linked with Tx3 (all having different inputs and outputs)
How is it related to Taproot?
Using multisig in any privacy related project doesn't make sense if transactions look different from others.

I don't really see what taproot has to do with this.

I was assuming transactions involving multisig will look similar to other transactions after taproot. Is that incorrect?

Possibly, yes, if MuSig is used by the wallets in question. But that still doesn't have anything to do with RBF.

Yes its not related to RBF. I am trying to mix two things to see if this can together work to improve privacy.

You need to be clearer on exactly what you are seeking to keep private before you attempt to design something. There are various things you may want to keep private in an onchain or Lightning setting e.g. your total Bitcoin wealth, when you make a transaction, which address you are sending from (source), which Bitcoin address you are sending to (destination), how much you are sending, the encumbrance or script that was imposed on an output, the encumbrance or script that was satisfied to spend an output etc
If you use MuSig(2) (ignoring RBF) which is enabled by the Schnorr, Taproot soft fork you can hide the fact both when sending to and when sending from that multisignature was used because only one key and one signature goes onchain. It looks as if it was a normal single key spend and the transaction fee you need to include will be the same as for a single key spend.
RBF offers little in a privacy context (and nothing as far as I can tell in combination with a scheme like MuSig). Eventually a transaction is going onchain whether it be the original transaction or a RBF transaction. So the source and the destination are going onchain regardless and the amount sent is going onchain. The only thing that is potentially exposed or kept private is the transaction fee on a transaction that didn't make it into the blockchain. I'm not sure why you would care about keeping this information private. Perhaps including a high transaction fee leaks the level of urgency you have for a transaction being confirmed but if RBF is successful a high fee is going onchain regardless.

I have edited the question and added more information.


text:
I have received £46,21745 in my Bitcoin account. Should I confirm it?
I don't have a Bitcoin account (I never made one), but I got an email  that I have received £46,21745 in my account and my first payment is ready for my confirmation. I would like to know how Bitcoin account works before I confirm it. I need to confirm it within 48 hours or my account will be closed.

This is most definitely a scam. Just ignore it.

They can't even make up their mind which currency it's valued in: dollars, pounds or euros?

I don't have any Bitcoin account [...] but I got an email that I have received £46,21745 in my account

It is a lie. There's no money. It is a trick by criminals who will
later ask you to pay some fake fees that they invent.
After you realise you have been tricked, some other person (criminals also, maybe the same ones) will then contact you to offer fake recovery services. They too will trick you out of more money.
See

Is it normal to be asked for an advance fee on a withdrawal?
fraud
non-spendable
watch-only

And/or the "confirm" link will be malware.


text:
`addmultisigaddress` syntax for POST request
Copied the body part from https://bitcoincore.org/en/doc/0.21.0/rpc/wallet/addmultisigaddress/ and replaced addresses with 2 new addresses (regtest):
POST / HTTP/1.1
Host: localhost:18333
Authorization: Basic YmFzZTY0dXNlcjpiYXNlNjRwYXNz
Content-Type: text/plain
Content-Length: 186

{"jsonrpc": "1.0", "id": "curltest", "method": "addmultisigaddress", "params": [2, "[\"bcrt1qvtp07jfnk5wcslr6xjmntnkxchzt7j2rzswnqn\",\"bcrt1qzwpuwrykaacx7057tnafyrd4nnfecc602gptca\"]"]}

Error:
{
    "result": null,
    "error": {
        "code": -1,
        "message": "JSON value is not an array as expected"
    },
    "id": "curltest"
}

You don't need the backslashes. Those are to prevent the shell from interpreting the " characters, but you're not entering this in a shell.

If I remove those I get a different error:   "code": -32700, "message": "Parse error"

I suspect you're making yourself hard by trying to play shell yourself. JSON isn't very complicated: https://www.json.org/json-en.html

Oh, I see. You also have the remove the unquoted ".

That resolved the issue. Thanks :)

I was doing 2 things wrong:

backslash \ is used in shell
Extra quotes had to be removed

Thanks Pieter Wuille for the help.
Below request works fine:
POST / HTTP/1.1
Host: localhost:18333
Authorization: Basic YmFzZTY0dXNlcjpiYXNlNjRwYXNz
Content-Type: text/plain
Content-Length: 180

{"jsonrpc": "1.0", "id": "curltest", "method": "addmultisigaddress", "params": [2, ["bcrt1qvtp07jfnk5wcslr6xjmntnkxchzt7j2rzswnqn","bcrt1qzwpuwrykaacx7057tnafyrd4nnfecc602gptca"]]}

Response:
{
    "result": {
        "address": "bcrt1q5pza3z8vmt29xv2kyw9nvd30t0sgrfnzv55s6g9q22df4pkqp65qtuf0nn",
        "redeemScript": "5221024b72aac0b3a6dab576b344fa36384d39e95c0bc05f4e6ddf6567b581c50031712103e38d84ad58ddc696ee67e08ebec0df5931ce729fdd234acb50a2ed6785a9badc52ae",
        "descriptor": "wsh(multi(2,[7aabbdcc/0'/0'/16']024b72aac0b3a6dab576b344fa36384d39e95c0bc05f4e6ddf6567b581c5003171,[7aabbdcc/0'/0'/17']03e38d84ad58ddc696ee67e08ebec0df5931ce729fdd234acb50a2ed6785a9badc))#yg7pyr0r"
    },
    "error": null,
    "id": "curltest"
}


text:
Which projects and services are responsible for most of the Bitcoin transactions?
Which projects and services are responsible for most of the Bitcoin transactions?

Project
Transactions %

ABC
50

XYZ
20

I am interested to know which projects really contribute in paying more fees per day. Will be great if we could also get fee estimate paid by different types of projects and services. If not, more transactions can be assumed contributing more fees.
Reason: Analyze the downtrend in fees and its affect on Bitcoin based on below numbers.

In the Lightning Conference 2019, Jameson Lopp mentioned that Exchanges are responsible for most of the bitcoin transactions. It is discussed in a tweet thread.

Bitcoin is not used much in darknet markets now because people realized privacy isn't the priority for many users and Bitcoin developers. Still, we can look at the relevant charts and see that most of the source and destination associated with DNMs are exchanges or mixing services.

There is volume and few other charts for LN, Bisq, Coinjoin implementations etc on https://bitcoinkpis.com/ however its difficult to know percentage of transactions and fees from this data.


text:
Is it possible to change `their_to_self_delay` after a channel is open?
Rene said here that a c-lightning node is safe for around 24 hours to be offline (depending on the locktime). This is set with watchtime-blocks in the configuration file of c-ligthning. But this will only be used for new channels, as far as I understand.
Is it possible to change the their_to_self_delay value of a already existing channel?
The their_to_self_delay value is equal to watchtime-blocks value set in the general c-lightning config file.

I think as of now the protocol does not support updating those channel parameters as they would need to be renegotiated. I could not find any message in the Protocol that would support such a functionality
What you could do is to close the channel and reopen it But I guess you would want to avoid it and technically I do not see reasons why such an updated should be impossible.

You are right, i want to avoid closing channels to change those channel parameters. The process to get this implemented in the different node distributions would be to have a new  BOLT or is there another way to achieve this?

To be able for two nodes to do this, they would need a protocol. Thus an extension of the BOLTs would probably be necessary. At least I don't see any other way


text:
What fee is the starting point of the closing fee negotiation?
When I issue the command lightning-cli listpeers $peerid i get a feerate perkb of 1012 atm. What feerate is this exactly? Is this the feerate which will be proposed by the other peer for a mutual close? I don't think this is the case.
feerate: An object containing the latest feerate as both perkw and perkb.

I know that i can get the feerates of my node with lcli feerates perkb. With these results atm:
{
   "perkb": {
      "opening": 1012,
      "mutual_close": 1012,
      "unilateral_close": 1012,
      "delayed_to_us": 1012,
      "htlc_resolution": 1012,
      "penalty": 1012,
      "min_acceptable": 1012,
      "max_acceptable": 88300
   },
   "onchain_fee_estimates": {
      "opening_channel_satoshis": 177,
      "mutual_close_satoshis": 170,
      "unilateral_close_satoshis": 151,
      "htlc_timeout_satoshis": 168,
      "htlc_success_satoshis": 178
   }
}

That means my node will propose 1012 as a starting fee for a mutual close. But what is the starting fee of the other node?

You won't know what the other node proposes unless some one starts the process of closing the channel and then your node will see what they propose and start digging into the estimation.
In general one can access the man-page with lightning-cli help feerates (or in your case with lcli help feerates) from there we find further information with respect to negotiation of fees.
It says that the closing negotiation start with your node proposing mutual_close fess as you suggested. I quote:
mutual_close - the starting feerate used in mutual close negotiation.
Note that since mutual close is a negotiation, the actual feerate used in
mutual close will be somewhere between this and the corresponding mutual
close feerate of the peer.


text:
How to replace `peer_id` with `alias` in c-lightning?
There are commands like lightning-cli listpeers and lightning-cli listfunds.
How do I replace the peer_id in the output of the commands with the alias of the peer?
I know about the jq command line tool to parse JSON and that i can get the alias with lightning-cli listnodes $peer_id. But I am having a hard time achieving the desired output. I think it should be doable with the use of jq. But maybe there is another way I am not thinking about. Maybe Python? But I would prefer just the standard unix tools if possible.

unfortunately no for the moment


text:
Bitcoin alpha version with 15 minutes block time interval

In alpha software prior to Bitcoin's release, the target interblock interval was 15 minutes, and it was reduced to 10 minutes for release.

https://www.reddit.com/r/Bitcoin/comments/56nk94/on_chain_scaling_with_schnorr_signatures/d8lmwco
Where can I find this alpha version for research?

Cryddit on Bitcointalk posted some source code that Satoshi supposedly emailed to him. This code contains a target interval of 15 minutes rather than 10. It is in https://bitcointalk.org/index.php?topic=382374.msg4108739#msg4108739. Search for nTargetSpacing.

Interesting, I wasn't aware of this. Apparently it also had 30 day retargetting periods (2880 blocks).


text:
What is `ping` RPC used for?
ping doesn't return anything so what is it used for?
If a user wants to ping and check ping time for all peers, getpeerinfo can be used.

I think ping is a basic networking utility that would be expected to be included in all P2P software. getpeerinfo provides additional information that you may not be interested in if you are just looking at ping time.

It does not have any arguments so that user can mention address of a node to ping, it does not return any details, so I am not sure why would someone use it. I could not understand this comment by Greg Maxwell. If Bitcoin Core does not ping all peers automatically every few seconds then maybe there are use cases for this RPC else it can only be used for spamming (assuming there are no rate limits)

The help text is your friend:
Requests that a ping be sent to all other nodes, to measure ping time.
Results provided in getpeerinfo, pingtime and pingwait fields are decimal seconds.
Ping command is handled in queue with all other commands, so it measures processing backlog, not just network ping

After you use ping, you may see a new field, pingwait, in the getpeerinfo output. This is the time for waiting for the ping response. You will also see pingtime being updated once the peer responds to the ping. The purpose is to get an update on the ping times since the ping time reported may have been measured a long time ago.

getpeerinfo for one peer before ping: https://pastebin.com/s81cmpu7 getpeerinfo for same peer after ping: https://pastebin.com/6ZUgWTZ4 I don't see pingwait in it. pingtime changed from 0.077075 to 0.077956 but it changes regularly even without running ping command

I could see pingwait today in https://pastebin.com/vKi5sfK2


text:
Is it possible to send BTC directly to a hardware wallet (such as Trezor) simply by plugging the hardware wallet to a USB port?
This question is about how standalone a hardware wallet (such as Trezor) is. Or in other words, what other components, if any, does it depend on.
So this is about what other components are needed for the Bitcoins to end in Bobs hardware wallet when Alice wants to send Bitcoins to Bobs wallet address.
So for example, is a hardware wallet dependent on connecting to either

some software with the wallet address that runs on the PC and
connects to the internet, or
some web application, or maybe a part of the BTC ecosystem, that
hosts the wallet address

Or is it possible to send Bitcoin directly to a hardware wallet (such as Trezor) simply by plugging the hardware wallet to a USB port on your PC?

A hardware wallet such as Trezor is simply a clever storage of your private keys that can be made to do some things defined by its firmware (like signing a transaction). The instructions to do these are relayed to it over USB from any device running software capable of producing them. Usually this will be a PC or a phone running wallet software created either by the manufacturer or the open source community.
I will try to outline what exactly is needed for Alice to send some bitcoin to Bob when they are both using a hardware wallet:

Bob needs a bitcoin address to give to Alice. He uses his wallet software to instruct his hardware wallet to generate a new address from its extended public key. This step could be done without internet connection, however without connecting to the Bitcoin network the wallet software wouldn't know which addresses have already been used and might end up reusing an address. Bob looks at the generated address on the hardware wallet display (not on his PC because it might be infected to show an attacker's address instead) and gives it to Alice.

Alice uses her wallet software to create a transaction, entering Bob's address, amount to send and tx fee (potentionally more, like RBF, locktime...). She needs to connect to the Bitcoin network for this to know her UTXOs. Once created, she sends the unsigned transaction to her hardware wallet for signing. The hardware wallet will show on its display the transaction's details and asks her to confirm it. The signed transaction is then sent back to the wallet software and can be, now or at any point in the future, broadcasted to the Bitcoin network.

TL;DR: You need a PC or a phone running wallet software to use your hardware wallet.

Thank you @Vojtěch Strnad for the answer. Is it correct to say that the funds are transferred only in the sense of getting recorded in the blockchain as now belonging to Bob's address (i.e. the address provided by his HW wallet)? And does Bob's wallet have some functionality to allow Bob to release those funds and send them elsewhere?

@coderworks. 1) Correct, transference and recording of transactions with funds only happens once validated on the blockchain. 2) If Bob want to spend the funds then he has to provide a signature associated with his address that only his private key can produce.

what other components are needed for the Bitcoins to end in Bobs hardware wallet when Alice wants to send Bitcoins to Bobs wallet address.

None.
Because receiving wallets play no part whatsoever in the transaction process (other than some time in the past providing a receiving address that you communicate to Alice in the form of a QR code etc). A receiving wallet can be offline, unpowered and still
receive money because wallets don't contain money - money is just a record in the blockchain that everyone has a copy of (or access to a copy of).
Spending money, on the other hand, requires software that can work with the hardware wallet to create a transaction and make a few peers aware of it. As WillO pointed out, this software can either be software that you install on your personal computer or a web-based application accessed using an ordinary web-browser.
Related:

Is a Trezor hardware wallet dependent on wallet.trezor.io?

The OP might find it helpful to know that even for spending, you don't actually need to run the software yourself.  You can plug your Trezor into a USB port on your computer, then connect to Trezor's website at www.trezor.io, and that's all you need.

@WillO: Good point, answer updated accordingly, thanks.


text:
How can I decode the feature string of a Lightning node with BOLT #9?
Typing the following command (with a running lightningd):
 lightning-cli listpeers

I get peers which all have a features flag, for example 0252a1. In the manpage lightning-listpeers(7) it says, that it corresponds to the features mentioned in BOLT #9.
How do I get from 0252a1 to the features listed in the table seen in the BOLT #9 document?

A bit is a binary digit with the two possible expressions 0 and 1. Binary data is often represented as hexadecimal in writing as that is more compact and human-readable. Hexadecimal (base 16) encodes the (decimal) numbers 0 through 15 with the symbols 0-9 and A-F (where A-F represent 10-15).
A few examples (subscript indicates base):
110 = 116 = 00012
510 = 516 = 01012
1010 = A16 = 10102
1310 = D16 = 11012
The flag array that you quote, 0252a1 would translate to the binary array 100101001010100001
To use it as a flag array, we do not interpret it as a number but interpret a 1 as a feature being set and a 0 as a feature being unset. We refer to each bit by its position counting from the rightmost. The rightmost is the zeroth bit and even.
The Feature Flags (BOLT #9) in Lightning are always assigned in pairs. Even bits refer to features that our node will consider mandatory, odd bits refer to optional features our node supports. Other nodes will ignore odd bits that they don't understand ("It's okay to be odd."), but will disconnect if there are even bits set they don't support.
1001010010101000_01_
Bits 0/1 refer to the feature option_data_loss_protect. The flag is on the even bit (the 0th), thus the feature is mandatory.
10010100101010_00_01
Bits 2/3 refer to initial_routing_sync. Since neither bit is set, our node does not support that feature.
100101001010_10_0001
Bits 4/5 signal option_upfront_shutdown_script. The flag is on bit 5, and thus it's an optional feature.
1001010010_10_100001
Bits 6/7 signal gossip_queries. The flag is on bit 7, also optional.
10010100_10_10100001
Bits 8/9 signal var_onion_optin. As the flag is on the odd bit, it's an optional feature.
100101_00_1010100001
Bits 10/11 is for gossip_queries_ex. Our node does not support this feature.
1001_01_001010100001
Bits 12/13 indicate that option_static_remotekey is mandatory for this node's peers, as it is set on bit 12, the even bit.
10_01_01001010100001
Bits 14/15 refer to payment_secret which is also mandatory.
_10_0101001010100001
The flag on bit 17 indicates that basic_mpp is optional.
The absence of the feature flags on bits 18-27 indicates that the node does not recognize any of the corresponding features.


text:
P2TR outputs on testnet
Anthony Towns shared that Taproot is activated on Testnet.
I am not sure if we can create P2TR outputs easily on Testnet right now based on this Q&A by Michael Folkson:
How can I spend from a P2TR address on Signet with the Bitcoin Core wallet in 22.0?
I am interested in reviewing PR: https://github.com/bitcoin/bitcoin/pull/22421 and test utxoupdatepsbt on Testnet with PSBT spending Taproot outputs.

What is your question? Everything supported by its descriptors language should be easily doable with current Bitcoin Core master. You'll need to manually import the descriptor, and the scripts are fairly boring (only single-key, or 1-of-N, basically), but within those constraints it should work just fine as there are tests for it.

Question: How to create P2TR outputs on Testnet? For creating P2WPKH output, I just need to create new segwit address and send some bitcoin. Why is it different for Taproot? Importing descriptors is fine but import from where?

There is nothing specific to testnet about that; it's the same as on signet, or how it'll be on mainnet. You need to create the descriptor yourself because taproot descriptors aren't created by default yet (that'll only be added once it's active on mainnet). Import from where: write them yourself; the simplest would be tr(xprv/*) for some xprv. See https://github.com/bitcoin/bitcoin/blob/master/doc/descriptors.md for more advanced ones.

Yeah only difference between testnet and signet re Taproot is that Taproot only recently activated on testnet but Taproot has been active on signet for months. Of course you need to get testnet Bitcoin and use -testnet flag but that isn't Taproot related


text:
Are all blocks stored in one file or is every block stored in an individual file?
I couldn't find an answer to that. So, are blocks stored in one or multiple files and in which file format?

Are blocks stored in one file No Is every block stored in an individual file No So every blk*.dat file has lot of blocks. Pieter Wuille has explained few related things in this answer: https://bitcoin.stackexchange.com/a/11108/ and this is just for Bitcoin Core. Other implementations can do this in different ways.

@Prayank Thanks, that's what I needed.


text:
`getdescriptorinfo` for private key
Public key:
bitcoin-cli getdescriptorinfo "wpkh(027fda10f37e6c637a1cfeba2cbada86de36f28e63d4ded9e1314117739f6419a9)"
{
  "descriptor": "wpkh(027fda10f37e6c637a1cfeba2cbada86de36f28e63d4ded9e1314117739f6419a9)#99knjm9r",
  "checksum": "99knjm9r",
  "isrange": false,
  "issolvable": true,
  "hasprivatekeys": false
}

Private key:
bitcoin-cli getdescriptorinfo "wpkh(cRLX3mChef5FEFi6y46yH1YJ4Cy7K3WW4bJDBZ7LiygKp4CBsF6p)"
{
  "descriptor": "wpkh(027fda10f37e6c637a1cfeba2cbada86de36f28e63d4ded9e1314117739f6419a9)#99knjm9r",
  "checksum": "s5qjl8cq",
  "isrange": false,
  "issolvable": true,
  "hasprivatekeys": true
}

What is the correct way to use private keys in descriptors?
Why are two checksums returned in getdescriptorinfo for private key?
Why does the KEY inside parenthesis change when using private key with getdescriptorinfo?

Descriptors with a private key (or xprv) in it are equivalent to their version with the public key instead, apart from also conveying the private key information.

For what purpose?
The descriptor field only reports the descriptor itself (i.e., without private key material). The checksum field reports the checksum for your input. This is also explained in the help text for the RPC.
It's removing private key material.

I wanted to import a descriptor using private key but had no checksum. Used getdescriptorinfo to see checksum. 2. and 3. Yes it all makes sense now. I carefully looked at results again after reading your answer. Thanks.


text:
Test that mempool reattempts delivery of locally submitted transaction
Test: https://github.com/bitcoin/bitcoin/blob/master/test/functional/mempool_unbroadcast.py
I am trying this manually with below steps:

Run two nodes (regtest):

Node1:
bitcoind -port=18333 -rpcport=18332 -datadir="/home/prayank/node1" -conf="/home/prayank/node1/bitcoin.conf"

Node2:
bitcoind -port=18444 -rpcport=18443 -datadir="/home/prayank/node2" -conf="/home/prayank/node2/bitcoin.conf"

bitcoin.conf:

Node1:
regtest=1
server=1
listen=1

debug=mempool

regtest.rpcport=18332
rpcuser=user3
rpcpassword=password3

fallbackfee=0.0001

regtest.addnode=127.0.0.1:18444

Node2:
regtest=1
server=1
listen=1

debug=mempool

regtest.rpcport=18443
rpcuser=user3
rpcpassword=password3

fallbackfee=0.0001

Everything looks okay until Line 68

How would I do this without running this python code?
        # fast forward into the future & ensure that the second node has the txns
        node.mockscheduler(MAX_INITIAL_BROADCAST_DELAY)
        self.sync_mempools(timeout=30)

mockscheduler is a hidden RPC. This means that it will not be displayed when using the help command, but you can still use it by just calling it as usual. E.g. to do the same thing as this line in the test, you would do
bitcoin-cli -regtest mockscheduler 900

sync_mempools() is a function which just waits for the getrawmempool results of multiple nodes to match. You can just do that manually.

Interesting. This is like hidden gem.


text:
Unbroadcast count doesn't change as expected after 15 minutes
Related question: Test that mempool reattempts delivery of locally submitted transaction
I tried running hidden RPC mockscheduler, I think it worked that day after relaunching nodes few times and trying different things. Today again I am not able to do the relay similar to unbroadcast test.

Start Node 1
Start Node 2 with Node 1 in addnode
Use sendtoaddress to send some bitcoin to own address in Node 2
Check transaction is in getrawmempool and 1 unbroadcast in getmempoolinfo
Node 1 doesn't have these transactions in getrawmempool
Restart nodes
Run this multiple times because its not doing anything: bitcoin-cli -regtest mockscheduler 900
Nothing changes in getrawmempool or getmempoolinfo

After reading lot of things I think the reason is Node 1 not sending getdata request based on:

With these changes, the mempool tracks locally submitted transactions & periodically reattempts initial broadcast. Transactions submitted via the wallet or RPC are added to an "unbroadcast" set & are removed when a peer sends a getdata request, or the transaction is removed from the mempool. Every 10-15 minutes, the node reattempts an initial broadcast.

https://github.com/bitcoin/bitcoin/pull/18038/
I checked the packets in Wireshark to see what's going on because debug=mempool and debug=net spams with so many things that are confusing. I found that Node 2 had sent getdata packet with 13 hashes:

getrawmempool returns txid and we see wtxid in Wireshark. I reversed one from Wireshark to get wtxid
INV: 9f5e3c78e1e6a4f7f12b1e876ba6dadfcf3df6812a3ac214dda2455284e0e079

WTXID: 79e0e0845245a2dd14c23a2a81f63dcfdfdaa66b871e2bf1f7a4e6e1783c5e9f

However getrawtransaction doesn't accept wtxid so I am not sure how to check if this transaction belongs to Node 1 or Node 2. I am guessing its Node 1's transaction.
I have no clue what's going wrong and why simple relay for a transaction is not working between these 2 nodes.

On which node are you doing mockscheduler on? It has to be run on the node with the txs in the mempool. The wtxid is listed under hash in getrawtransaction. getrawmempool true and getmempoolentry also lists it as wtxid.

Both nodes have some transactions in mempool. Although I did mockscheduler on Node 2. Now restarted both. Did it on both nodes. Still Node 2 has 1 unbroadcastcount. Thanks for sharing this getrawmempool true which confirms the transaction I'd mentioned above was from Node 1. If it was sent to Node 2 it should have been in it's mempool though.

After restarting, are the nodes connected to each other?

Yes. getpeerinfo on Node 1: https://pastebin.com/raw/VPNYzBP2 and Node 2: https://pastebin.com/raw/VxBDaBZC

The steps that you have outlined don't make sense for what you are trying to achieve. You want to make sure that node 2 attempts to broadcast unbroadcasted transactions after 15 minutes. But node 2 and node 1 are connected when you do sendtoaddress, so the transaction that is not supposed to be broadcast is sent to node 1. That's not correct, and that's not what the test does.
To achieve the behavior that you want, you have to start node 2 without connecting it to node 1. They should be disconnected when you do the sendtoaddress. You should only be connecting the nodes after sendtoaddress completes. Then you should be able to observe the broadcast after 15 minutes behavior.

You are probably seeing node 1 doesn't have the transaction immediately after you do sendtoaddress because there is some delay between when the node processes the transaction and when it queues it to be broadcast to its peers. So if you are very fast on checking getrawmempool and getmempoolinfo`, then you will probably observe that the transaction is not broadcast yet. But this is a race condition, and it could be broadcast between your mempool checks and shutdown. This would then cause your later checks to fail.


text:
Fuzzing descriptor wallet RPCs
Is SQL injection a possibility while using JSON-RPC and Descriptor wallet RPCs?
Consider one parameter in some descriptor wallet RPC is vulnerable and returns some data with SQL queries that shouldn't be returned. Will it really affect anything if RPC is only accessible from localhost?
Any difference in security when using REST API because I have not tried -rest option yet?

You mean because descriptor wallets use sqlite as storage backend? None of that is exposed through RPC or otherwise.

Yes because of SQLite. Not sure if I understand second sentence. If a descriptor wallet RPC updates something in SQL db it should be assumed exposed.

I mean there is no way to create an RPC that will pass through SQL code to sqlite. All sql queries are constructed inside the wallet, so there isn't any way to test this.

The vast majority of database entries are generated by the wallet and do not contain anything that the user can enter. Things that users can enter are generally validated prior to writing to the database, e.g. descriptors imported with importdescriptors must be valid descriptors before they are written.
The sole exception are labels which, by definition, contain user generated data. As such, if the record writing were implemented incorrectly, a user could use setlabel to perform a SQL injection attack. This is because the label string provided to setlabel must be provided directly to the SQL query that actually adds it to the database.
However Bitcoin Core's use of sqlite prevents SQL injection attacks. Bitcoin Core uses prepared statements so it is not possible to perform a SQL injection, barring some implementation error on the side of sqlite. These statements are prepared at the time the wallet is loaded. They are fixed queries and cannot be modified by the user.

Ah, I had forgotten about labels and update statement of course. It's very limited, and using prepared statements it should categorically be impossible to do SQL injection, but that is indeed an example where RPC arguments are passed through.


text:
GetMinFee() in blockchain.cpp
  "mempoolminfee" : n,       (numeric) Minimum fee rate in BTC/kB for tx to be accepted. Is the maximum of minrelaytxfee and minimum mempool fee
  "minrelaytxfee" : n,       (numeric) Current minimum relay fee for transactions

https://bitcoincore.org/en/doc/0.21.0/rpc/blockchain/getmempoolinfo/
I am trying to understand how is mempoolminfee calculated. Based on above documentation it should be max of minrelaytxfee and minimum mempool fee. I understand minrelaytxfee is a config option which can be set to anything by user. What is the other fee: 'minimum mempool fee'?
Looking at the code for getmempoolinfo RPC I found this comment in one header file which explains the function that is used for calculation:
    ret.pushKV("mempoolminfee", ValueFromAmount(std::max(pool.GetMinFee(maxmempool), ::minRelayTxFee).GetFeePerK()));

https://github.com/bitcoin/bitcoin/blob/fdd80b0a53b4af0b29cb6e03118e2456d053a757/src/rpc/blockchain.cpp#L1655
    /** The minimum fee to get into the mempool, which may itself not be enough
      *  for larger-sized transactions.
      *  The incrementalRelayFee policy variable is used to bound the time it
      *  takes the fee rate to go back down all the way to 0. When the feerate
      *  would otherwise be half of this, it is set to 0 instead.
      */

https://github.com/bitcoin/bitcoin/blob/fdd80b0a53b4af0b29cb6e03118e2456d053a757/src/txmempool.h#L731
CFeeRate CTxMemPool::GetMinFee(size_t sizelimit) const {
    LOCK(cs);
    if (!blockSinceLastRollingFeeBump || rollingMinimumFeeRate == 0)
        return CFeeRate(llround(rollingMinimumFeeRate));

    int64_t time = GetTime();
    if (time > lastRollingFeeUpdate + 10) {
        double halflife = ROLLING_FEE_HALFLIFE;
        if (DynamicMemoryUsage() < sizelimit / 4)
            halflife /= 4;
        else if (DynamicMemoryUsage() < sizelimit / 2)
            halflife /= 2;

        rollingMinimumFeeRate = rollingMinimumFeeRate / pow(2.0, (time - lastRollingFeeUpdate) / halflife);
        lastRollingFeeUpdate = time;

        if (rollingMinimumFeeRate < (double)incrementalRelayFee.GetFeePerK() / 2) {
            rollingMinimumFeeRate = 0;
            return CFeeRate(0);
        }
    }
    return std::max(CFeeRate(llround(rollingMinimumFeeRate)), incrementalRelayFee);
}

https://github.com/bitcoin/bitcoin/blob/fdd80b0a53b4af0b29cb6e03118e2456d053a757/src/txmempool.cpp#L1080
I still don't understand what GetMinFee() does. Can someone explain this is in a different way or use some examples?
If I save minrelaytxfee=0.0002 and incrementalrelayfee=0.0003, mempoolminfee is 0.0002 which isn't max of both.

From the mempool's perspective, the rollingMinimumFeeRate is the actual minimum fee rate (the minrelayfee pertains to other parts of the codebase). It starts at 0, and is updated whenever a transaction package is evicted from the mempool due to size constraints. When the mempool reaches its configured maximum, it removes the lowest feerate package to free up space, and sets the rollingMinimumFeeRate to the feerate the removed package paid.
But then you might wonder how the rollingMinimumFeeRate goes down? What happens if the maximum mempool size is reached, but then transactions get confirmed and the mempool's size is reduced? The rollingMinimumFeeRate needs to decrease eventually, and that is what GetMinFee is doing.
GetMinFee is the second part of updating rollingMinimumFeeRate. It reduces it exponentially according to a schedule. If it has been more than 10 seconds since rollingMinimumFeeRate has changed (either up or down), then it will be divided by some power of 2.
If the mempool is more than half full, and no packages are removed to increase rollingMinimumFeeRate again, then after 12 hours, it will be half of what its original value was. That is to say, rollingMinimumFeeRate has a half life of 12 hours when the mempool is at least half full.
This half life decreases when the mempool is between a quarter full and half full. In that case, it is 6 hours. And when the mempool is less than a quarter full, the half life is 3 hours. This is what this section does:
    double halflife = ROLLING_FEE_HALFLIFE;
    if (DynamicMemoryUsage() < sizelimit / 4)
        halflife /= 4;
    else if (DynamicMemoryUsage() < sizelimit / 2)
        halflife /= 2;

To compute the reduced rollingMinimumFeeRate, we want to figure out how much time has passed since the last update and figure out how much lower it should be given the half life that we want. That is what this line does:
    rollingMinimumFeeRate = rollingMinimumFeeRate / pow(2.0, (time - lastRollingFeeUpdate) / halflife);

This is really just the half life formula: N(t) = N_0 * (1/2) ^ (t/t_1/2).
N_0 is the initial quantity (rollingMinimumFeeRate), t is the time that has passed (time - lastRollingFeeUpdate), and t_1/2 is the half life we want (halflife). If you move the exponent into the fraction, reduce the formula, and write it out in code, you get the aforementioned computation line.
The last piece to this update is to check whether the rollingMinimumFeeRate is less than half of the incrementalRelayFee. If it is, then rollingMinimumFeeRate is set to 0. This is done because exponential decay means that rollingMinimumFeeRate would technically never reach 0 (that's how the exponential decay function works). So instead we just bound it to be no less than half of the incrementalRelayFee. Once it reaches that, it just resets to 0 because at that point, it will be incredibly low anyways.
We use the incrementalRelayFee as the lower bound because that is the minimum feerate a transaction needs to pay after accounting for conflicts in order to be relayed. So a transaction must be paying at least incrementalRelayFee, so we can drop the rollingMinimumFeeRate after it is less than incrementalRelayFee, but to be safe, we do it at half of that value.
That last piece of GetMinFee is at the top of the function, which is simply that the current rollingMinimumFeeRate does not change until we find a block, and an optimization to return it when it is already 0. This first part is important because we don't want to be reducing the mempool's minimum fee when things aren't being removed from the mempool.

Awesome! Thanks for all the details. While I was waiting for someone to answer, tried reading the code and make sense of each line: https://i.imgur.com/t3OqqUF.png. Almost understood everything except few things like initial value of rollingMinimumFeeRate, how it updates, why name half life etc. After reading your answer, now I understand everything. TIL: https://en.wikipedia.org/wiki/Half-life#Formulas_for_half-life_in_exponential_decay


text:
How to protect my private key from an unexpected new owner of the address that already belongs to me?
I understand that the situation I am describing is unlikely (because we have 1.46 Quindecillion possible Bitcoin addresses), but let's imagine that someone generated the same address as mine.
Is there any protection against a duplication of private and public keys if someone's new address matches the existing address where bitcoins are already stored?

The mitigation is the absolutely massive address space. Other than that, there is no way to 'protect' or otherwise 'claim' a keypair/address as yours and yours alone. If someone (by some unfathomable coincidence) manages to generate the same address that your wallet did, then you will both be able to spend any funds locked to that address.
Thankfully, the chances of this happening are super duper incredibly small. You would expend far more energy (money) looking for an already-used address than you could ever be expected to make by taking the bitcoin you are hunting for.
Perhaps worth mentioning: the above holds true iff the private keys for the address in question are generated in a suitably random (high entropy) way. If you use a poor source of entropy, then it become much, much easier for someone to brute force search for your keys.

If you are worried of this extremely unlike event you could use a multisig address where multiple keys need to be known to spend the funds

Hi @MikeD, Does multisig guarantee that the second owner of address cannot spend bitcoins without my knowledge?

If for example it is a 2-of-2 multisig the second owner would need to find both of the keys to spend the coins

This does not help. Yes, it means that someone through the same script would indeed to find both keys, but that's overkill. All they need is to randomly stumble upon another script which hash matches your multisig, and the odds for that happening are either equal (for P2WSH or P2TR) or significantly larger (for P2SH) than randomly hitting the same keys! If you're worried about events which have such near-impossibly small probabilities, suddenly all security assumptions break down.

@PieterWuille the odds of finding another script with the same hash is the same but the odds of this random script being executable is pretty close to 0

@MikeD Well, no, because people only (deliberately) create scripts they themselves can spend from. If you're worried about people randomly creating keys that match your keys, you should be equally (if not more) be afraid of people randomly creating scripts that match your script - it's exactly the same.


text:
Do I have a chance to commit a transaction without any network fee?
I know how transaction fees are calculated. Also I understand that a transaction with a higher fee will be approved much faster than a transaction with the lowest one.
Does the bitcoin protocol allow committing a transaction without paying a fee (even if the transaction will be approved within 2 or 3 weeks)?

The protocol allows it but I believe many nodes will not relay a transaction whose fee is below a configured threshold.
See

What is the min relay min fee code -26?

The default minRelayTxFeeRate is 1,000 sat/kvB. Nodes using the default value will not relay transactions that pay a smaller feerate than that. I just did a search for confirmed transactions paying less than the minRelayTxFeeRate (feerates up to 0.00000249 BTC/kWU, excluding coinbase txs):
Lately, most days there are a couple and never more than a handful. Given that there are over 62 blocks worth of transactions tracked on mempool explorers offering feerates below the minRelayTxFeeRate, it's safe to assume that those few transactions were created by the corresponding miner themself and included in their own block, and there is no miner that includes below minRelayTxFeeRate transactions from the mempool without out-of-band encouragement (or there would be a lot more).
I would not expect any transaction paying less than minRelayTxFeeRate to get organically confirmed even when blocks are not full otherwise.

Yes you have, fee is optional. When the mempool clears some miner will likely mine your transaction

These days most nodes on the network, and miners, will not accept or forward a transaction with a fee below the minrelayfee (which for recent Bitcoin Core versions is 1 sat/vbyte by default). Perhaps if you send it to a miner directly, they may include it, but even that is unlikely without adding some incentive (in the form of a fee...).


text:
May bitcoin mining on laptop be possible again?
This is not a stupid question as it might seem at first glance.
I know how much the processing speed and power consumption on ASIC is superior to CPU and GPU. Recently, I've quite often heard suggestions that the Bitcoin network protocol contains the possibility of returning to mining on the CPU. That sounds nice.
Is such a scenario possible, since the network's Hash Rate is monstrously huge?

It is extremely unlikely that this will ever be possible.
Fundamentally, any computation can be done more efficiently on hardware that was specifically designed for that computation than on general-purpose hardware. The only question is when there is enough to gain from the (possibly enormous) costs to design and build custom hardware for the used proof-of-work function.
For Bitcoin, that happened somewhere in 2013, when the first ASICs became available for mining. Currently, Bitcoin mining ASICs are many orders of magnitude more efficient than CPUs. CPUs will continue to evolve, but as long as Bitcoin remains profitable, so will ASICs. There is no reason to assume that general-purpose CPUs will ever get close enough to be able to compete with ASICs.
Unless of course the Bitcoin proof of work function is changed. While that is theoretically possible, it is an extremely invasive change, which needs near unanymous support throughout the ecosystem. That is extremely unlikely to happen, unless the alternative was an almost certain threat to the system as a whole, and changing PoW was the solution. Even if that approach is taken, it would be a horrible precedent, hurting the premise that Bitcoin is immune to such changes. I don't see this happening.


text:
What types of smart contracts are available for bitcoin network in 2021?
We all know that the first smart contracts appeared on the Ethereum network. As far as I know, in the Bitcoin network, smart contracts work with a limited functionality – Bitcoin uses a non-Turing Complete language. At the moment, just the following types of smart contracts are possible on the bitcoin network:

Endowment contract
Escrow contract
Will of digital assets
Multisig contract
Safe for digital assets
etc...

Questions:

What other types of contracts can I implement in 2021?

What languages are mainly used to implement smart contracts in the network?

Can you give an example of an implemented smart contract in Apple Swift language?

The first smart contracts were not on the ethereum network.

We all know that the first smart contracts appeared on the Ethereum network. As far as I know, in the Bitcoin network, smart contracts work with a limited functionality.

These two sentences are contradictory. If Bitcoin has smart contracts with limited functionality, they cannot be the first on some other network. A smart contract with less functionality is still a smart contract.

Bitcoin uses a non-Turing Complete language

Answered in detail here: https://bitcoin.stackexchange.com/a/105931/

What other types of contracts can I implement in 2021?
What languages are mainly used to implement smart contracts in the network?

Old answer but still relevant: https://bitcoin.stackexchange.com/a/53843/
Other things:

Lightning Network is the biggest example of a smart contract live on Bitcoin

Taproot and future soft forks will further improve scripting capabilities but according to my understanding it will never be same as "World Computer" meme and do everything on-chain. Layer 2 works better for decentralized networks considering scaling, security etc.

Rootstock (sidechain) allows everything that is possible on Ethereum with better scaling, privacy etc.
https://bitcoin.stackexchange.com/a/103169/

Liquid (sidechain) works differently and allows devs, users, exchanges etc. to experiment with lot of projects.
https://blog.liquid.net/six-differences-between-liquid-and-lightning/
Parallel processing in a UTXO model: https://medium.com/bit-matrix/how-does-bitmatrix-achieve-parallel-processing-in-a-utxo-model-6ef70141bd59

If projects require oracle, discreet log contracts can be used: https://github.com/discreetlogcontracts/dlcspecs

Hi @Prayank, thanx for your answer. One notion, however, is unclear for me: "These two sentences are contradictory. If Bitcoin has smart contracts with limited functionality, they cannot be the first on some other network...." What do you mean in this sentence?

Smart contracts with limited functionality were used on Bitcoin even before Ethereum came in to existence.


text:
Is it true that a cryptocurrency exchange could freeze any account with suspicious bitcoins?
You are a law-abiding citizen and you have no idea how the bitcoins that you own now were used before. However, there is always a chance of getting "dirty" bitcoins that were used by criminals to buy drugs or weapons, or were used in money laundering.
What to do if the crypto exchange has frozen your account?
How many cases have there been that the exchange has frozen "illegal" bitcoins?

Is it true that a cryptocurrency exchange could freeze any account with suspicious bitcoins?

Yes, subject to terms & conditions and applicable law

What to do if the crypto exchange has frozen your account?

Contact the exchange.
Best to store Bitcoin in a standalone wallet

If an exchange decides to seize your Bitcoin, that's a civil matter between you and them. It happens not infrequently, as most exchanges use tools with questionable reliability to reduce their perceived risk. Generally speaking as a regulated business, it's their job to attempt to reduce any instance that may be money laundering or identity theft.


text:
Version in transaction
Original transaction hex:
02000000000101fff8d9d39105eeec162fb45bd2a3e796b255866469a2cc7de93e438f3722c1490000000000fdffffff02801a0600000000001600145b497cfd64324fb0c4f7317472ca591cd7eb3bbfe093040000000000160014b1348ec3acd422b7c84f97f8757a0a5c3db9e32d02473044022020c3a9d7d06429008de82f13de672aac7886c5d6557a7c330bcb8cbc20b8cf4f02203890042ed8bf331e92f07df5d2a7bf5d8ade12cbb216f14493a98d0a188c4918012103c65a6e4a66a4a6b4af981c5e81632bb2d7336bc75d7ccf83e055006a2693050f00000000

Changing first two characters from 02 to 03 (Version 3):
03000000000101fff8d9d39105eeec162fb45bd2a3e796b255866469a2cc7de93e438f3722c1490000000000fdffffff02801a0600000000001600145b497cfd64324fb0c4f7317472ca591cd7eb3bbfe093040000000000160014b1348ec3acd422b7c84f97f8757a0a5c3db9e32d02473044022020c3a9d7d06429008de82f13de672aac7886c5d6557a7c330bcb8cbc20b8cf4f02203890042ed8bf331e92f07df5d2a7bf5d8ade12cbb216f14493a98d0a188c4918012103c65a6e4a66a4a6b4af981c5e81632bb2d7336bc75d7ccf83e055006a2693050f00000000

Version in blocks is used by miners for signaling readiness in soft forks. Can there be use cases for different version numbers in transactions?
According to this answer: https://bitcoin.stackexchange.com/a/72742, if its not 1 or 2 then transaction is not standard. So I am assuming nodes will not relay such transactions as mentioned in another answer: https://bitcoin.stackexchange.com/a/41008/. This is weird because a field in transaction which could have been used for several things is just limited to 1 and 2. Are there any downsides if all versions are considered standard and let the projects use it however they want to?
What are the minimum, maximum and invalid values for it? I tried using ff for negative 1 but it returned 255 in version

Indeed, the only standard versions are still 1 and 2. The transaction version number can be used to indicate consensus feature support. So far there has only been one such feature - BIP 68. Because special rules for other versions have not yet been defined and accepted, the standardness rules limit the version just to the 2 known version numbers. But this does not mean that it can't be changed in the future. Much like how we have output scripts that are non-standard, but can become standard with a future update, same can the transaction version.

Are there any downsides if all versions are considered standard and let the projects use it however they want to?

Yes. Because the transaction version can control how validation is performed on transactions, letting people set the version to whatever they want could restrict what functionality could be added in the future or restrict what version numbers can be used. But it is also just a standardness rule, so you are welcome to use whatever version you want if a miner is willing to accept the transaction out of band.

What are the minimum, maximum and invalid values for it? I tried using ff for negative 1 but it returned 255 in version

The version number is 4 bytes, not 1 byte. It is actually a 32 bit little endian signed integer (note it is parsed and stored as signed, not unsigned, but cast to unsigned and used as unsigned in many but not all places). This makes it's minimum value is -2147483648 == 0xffffffff and it's maximum is 2147483647 == 0x7fffffff.

Note that the minimum value is 0xf000000..

Other transaction version numbers are for future consensus rules. If you create transactions using undefined versions numbers they may be invalidated by future consensus rules, or their outputs may be made unspendable by future consensus rules.
Using other version numbers today would have no legitimate purpose, but if applications weren't effectively blocked from doing it would mean that a future upgrade that needed to use version numbers would be effectively blocked or the funds of users who were pointlessly using the numbers might be irreparably destroyed.
This is why the use of these pointless encodings is inhibited-- to keep it safe to use them for extension, as they're intended and as a secondary purpose to keep faulty software from accidentally using them-- which has happened in the past, e.g. libbitcoin leaked uninitialized memory in version numbers-- and coming to harm when some rule is activated.


text:
When exactly are peers saved in peers.dat file?
I had tried few things when reviewed this PR: https://github.com/bitcoin/bitcoin/pull/22013
While reviewing PR https://github.com/bitcoin/bitcoin/pull/22651, I was expecting 1 peer to be saved in peers.dat using the below steps:

Run Node 1:

bitcoind -port=18333 -rpcport=18222 -datadir="/home/user/node1" -regtest=1 -listen=1 -server=1 -debug=net -rpcuser=user1 -rpcpassword=password1 -torcontrol='127.0.0.1:9051' -proxy='127.0.0.1:9050' -onlynet=onion

Run Node 2:

bitcoind -port=18777 -rpcport=18666 -datadir="/home/user/node2" -regtest=1 -listen=1 -server=1 -debug=net -rpcuser=user2 -rpcpassword=password2 -addnode='127.0.0.1:18333' -torcontrol='127.0.0.1:9051' -proxy='127.0.0.1:9050' -onlynet=onion

Stop Node 1 and 2:

bitcoin-cli -rpcport=18222 -rpcuser=user1 -rpcpassword=password1 stop
bitcoin-cli -rpcport=18666 -rpcuser=user2 -rpcpassword=password2 stop

Restart Node 2 with different config (onlynet=i2p and no proxy):

bitcoind -port=18777 -rpcport=18666 -datadir="/home/user/node2" -regtest=1 -listen=1 -server=1 -debug=net -rpcuser=user2 -rpcpassword=password2 -i2psam=127.0.0.1:7656 -onlynet=i2p

But nothing was saved in peers.dat according to logs. So I had to use addpeeraddress for the test which I guess does the same thing. So if node 1 is added as outbound connection in node 2, shouldn't this be saved in peers.dat? How does it work in case of testnet or mainnet?

Bitcoin Core has two mostly-separate mechanisms for creating outbound connections:

The manual mechanism, controlled by configuration settings addnode, connect, and the addnode RPC to change the addnode list at runtime. It can be used by the user to directly establish connections to peers of their choice, including ones on IP addresses that aren't publicly reachable (incl. localhost and your local network).
The automatic mechanism, where Bitcoin Core maintains a database of publicly reachable IP addresses of peers. This database is fed by seed nodes, by what its peers rumor, and by the addpeeraddress RPC. This database is large (can contain several 100000s of IP addresses), and used to establish a small (8-10) number of random connections. It is also used to answer queries by other peers (the getaddr protocol message). This database is saved to disk every 15 minutes, in the file peers.dat.

The connect configuration option (or -connect command line) enables the manual connection mechanism, and disables the automatic one. The addnode option enables the manual one, while leaving the automatic one intact (they both have independent maximum connection counts). The seednode option creates a manual connection to a certain node, asks it for peer IPs, feeds those into the automatic address database, and then disconnects.
So what is happening here is that you're using -addnode to establish a manual connection. That peer IP does not end up in peers.dat, because that file belongs to the automatic connection system. It also wouldn't work: the 127.0.0.1 address is not a public one that could even be rumored to other peers - they wouldn't be able to connect to it.


text:
send_and_ping() in bitcoin core tests
Is there any downside if send_and_ping() was a hidden RPC instead of just a function in python test framework?
    def send_and_ping(self, message, timeout=60):
        self.send_message(message)
        self.sync_with_ping(timeout=timeout)

    def send_message(self, message):
        """Send a P2P message over the socket.

        This method takes a P2P payload, builds the P2P header and adds
        the message to the send buffer to be sent over the socket."""
        tmsg = self.build_message(message)
        self._log_message("send", message)
        return self.send_raw_message(tmsg)

It sends a message via the P2P network; I don't see how it could be an RPC - it wouldn't be testing the same thing.

If an RPC for example sendmessage with message and peer as arguments would be implemented which sends messages to peer, we can remove this function and just use RPC in tests? This is my understanding which can be incorrect

I think you're confused about what this is testing. It's sending P2P messages directly to the node being tested, to see if it reacts correctly to them. You can't replace that with an RPC - the purpose is testing P2P behavior, not RPC behavior.

After thinking for sometime I understood the difference. It can't be a replacement but hidden RPC can be helpful maybe in some cases. Example: Sending message to other node using POST requests


text:
Is it possible to check which nodes banned my node?
I can ban peers with setban and check banned peers with listbanned in Bitcoin Core. Can I check which peers banned me?

Related: https://bitcoin.stackexchange.com/questions/35536/how-can-i-get-my-peers-ban-score-for-my-node

No, you cannot know this. But in general, since you're a normal, honest, node presumably, the most likely answer is "none".


text:
Why bitcoins storing on a smartphone are considered vulnerable?
Let's suppose, someone stores all their bitcoins in an Android or iOS wallet.

he/she never stores a seed on the phone
he/she uses different addresses for change
wallet and gadget are password protected
the smartphone is always kept only by the owner
there's no connection to iCloud, DropBox, etc
potentially dangerous apps are not installed on the phone

If you follow all of the above rules, then why is it generally accepted that storing cryptocurrency in a mobile wallet is considered unsafe?

If you exclude all reasons why typical smartphone usage is insecure, then yes, there are no problems.

Why bitcoins storing on a smartphone are considered vulnerable?

Its a perception based on their experience. Lot of things depend on user, type of smart phone, usage etc.
Two major things that are different on smartphone compared to desktop:

Its easy to run virtual machines on desktop. Few things exist for Android but I have not tried yet: https://www.xda-developers.com/vmos-lets-you-run-a-virtual-android-environment-on-your-phone/

Smartphone normally comes with lot of things you don't find in desktops. SIM & Network, GPS, Camera, Microphone etc. Neither good for privacy nor security. Although there are few people and companies who sell mobile phones without some of these: https://www.vice.com/en/article/wjwbmm/inside-the-phone-company-secretly-run-by-drug-traffickers

Some of the best practices mentioned in the question are not followed by users. Last point about apps is very important and if you can keep a separate smartphone with less apps then it can be considered safer. Other important things are processor, kernel, ROM etc.

s/safe/safer. Nothing in security is "safe" (100%) :)

Changed. Agree nothing is 100 % safe/secure.

If attackers hack into this smartphone and obtain the private key of your bitcoin wallet, your bitcoins will be stolen.
The higher risk than other devices of smartphones is:

Your smartphone may be lost or very easily stolen from your pocket, because it has a small size and always with you outside.
Your smartphone can be easily intentionally hacked remotely with its rich communication capabilities (LTE, WiFi, BT) and hypothetical backdoor or loaded from store under the guise of programs (games) trojans.

Hi awoland, what you said is generally true for all devices. Could you please elaborate how smartphones are generally at a higher risk than other devices?

Hi, @Murch!
The higher risk than other devices fo smartphones is:

Your smartphone may be lost or very easily stolen from your pocket, because it has a small size and always with you outside.
Your smartphone can be easily intentionally hacked remotely with its rich communication capabilities (LTE, WiFi, BT) and hypothetical backdoor or loaded from store under the guise of programs (games) trojans.


text:
As a developer, what steps do I need to take to generate private-public keys?
When developing (I'm using Apple Swift), what steps are needed in order to independently create my own pair of private-public keys (with the ability to restore a private key from a mnemonic phrase)?
There's no need to describe each step in detail, I would like to understand just the general idea.

I think this question is way too broad, but I'll point you to some useful resources you can look at: https://secg.org/   https://github.com/DavidEGrayson/ruby_ecdsa  https://en.bitcoin.it/wiki/Seed_phrase

@DavidGrayson, Thanx a lot.

I’ll omit some details to keep the answer compact but the general idea is this:
An elliptic curve point is a two dimensional point (x, y) where the x and y coordinates follow the curve equation y^2 = x^3 + 7
A private key is just an integer in the range 1.. 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEBAAEDCE6AF48A03BBFD25E8CD0364141
A public key is an EC point (two coordinates)
G is a special point with coordinates (0x79BE667EF9DCBBAC55A06295CE870B07029BFCDB2DCE28D959F2815B16F81798, 0x483ADA7726A3C4655DA4FBFC0E1108A8FD17B448A68554199C47D08FFB10D4B8)
Steps to create private/public keys

Implement Elliptic curve point addition (addition of two points)
Implement Elliptic curve point multiplication (multiplication between an EC point and an integer)
Generate a random integer in the range above, that’s your private key
Multiply your private key with the point G using step 2, that’s your public key

For generating keys from mnemonics there are two BIPs that describe the process.

BIP-39 describes how to turn a mnemonic phrase into a 512-bit seed
BIP-32 describes how to turn your seed and a derivation path into an EC key

This is written in swift, it's open source and you can check out the source code to get an idea of what's happening:
https://github.com/BlockchainCommons/GordianSeedTool-iOS
But as I'm coming to understand it myself, I'll attempt to summarize. You need to have a seed (a random number as a starting point) of sufficient entropy, or randomness, that's 2^256 bits long or 10^77, so pretty hard to repeat. Not every random number generator is actually random, for example some kinds of random are repeatable and are great for applications where you need to replicate results. But in this case you want to make it difficult to replicate. Every OS would have it's own set of random number generator syscalls that may or may not be random enough to be cryptographically secure, so you'd have to check the docs on whatever you use to ensure it's secure enough.
https://en.wikipedia.org/wiki/Cryptographically-secure_pseudorandom_number_generator
Generating mnemonic phrases is a little trickier. BIP-39 is the standard for implementing mnemonics in Bitcoin. This methods inserts bits of randomness, and then pulls some of the numbers out and selects the number pulled from an index of a vocabulary wordlist, hence generating the mnemonic phrase. It seems as if this is not quite as secure as a completely random key generated from sufficient entropy, as the possibilities is reduced from 2^256 to 2^132. Still pretty difficult to repeat though. See the following to understand how difficult this is to crack:
https://github.com/BlockchainCommons/SmartCustodyBook/blob/master/manuscript/00-randomness.md
So to generate a private key that can be restored via mnemonic phrase, it must be BIP-39 compatible.
https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki
Rather than trying to recreate all this functionality from scratch, which I wouldn't be able to explain succinctly, you could take advantage of libwally, or utilize the BC libraries or peruse them for your own learning/replication.
https://github.com/Sjors/libwally-swift
https://github.com/BlockchainCommons/seedtool-cli
https://github.com/BlockchainCommons/Learning-Bitcoin-from-the-Command-Line/blob/266d7e13a3ba1ce40d29c83a5d63a8d805c06621/src/16_2_genmnemonic.c
https://github.com/BlockchainCommons/Learning-Bitcoin-from-the-Command-Line/blob/45ccb82661b2f4086b9b3bcb062f0c738023aa4b/18_6_Accessing_Bitcoind_with_Swift.md

Hi Ian! Thank you for your answer. I know how to "calculate" a mnemonic phrase using BIP39's predefined 2048 words' list and math 2^128+4. But what's after that?


text:
Recovering an Hierarchical Deterministic wallet
Does the use of an HD wallet (with BIP 32 and BIP 39) guarantee the possibility of its recovery in an HD wallet of any manufacturer, whether it's software or hardware?
Or sometimes there are cases of incompatibility of HD wallets due to different principles of their creation?

You need to ensure they use the same derivation path. If there are a lot of unused addresses generated you may need to increase the gap limit.


text:
Why are segwit transactions cheaper than legacy transactions?
What's the reason to make fees for segwit addresses lower, while keeping legacy addresses with higher fees? --
I'm interested in both, why segwit transactions are cheaper than legacy addresses, and why legacy addresses still exist when they are more expensive.

Related: https://bitcoin.stackexchange.com/q/52196/5406, https://bitcoin.stackexchange.com/q/43596/5406, https://bitcoin.stackexchange.com/q/84004/5406

Why are segwit transactions cheaper?
Segregated witness transactions are cheaper because they are designed to be.
In legacy transactions, every byte has the same "cost"; they all count equally towards the 1000000 byte block limit. Segwit transactions on the contrary keep certain data (signatures etc) in a separate "witness" section, and that witness section is discounted: every witness byte only counts as 0.25 byte towards the block limit. Fees are determined by the market and not a system rule, but the fact that segwit transactions have some of their content discounted results in the market pricing them less.
There is a reason why this discount is justified: in legacy transactions, creating a transaction output is significantly cheaper than spending one. This encourages unspendable dust: outputs that were created at a time when fees were low may become uneconomical to spend (= cost more to spend than they're worth) when fees are high. This is a burden on the entire ecosystem, as full nodes (for now, at least) need to maintain fast access to the set of all unspent outputs.
Why are legacy transactions still in use?
Compatibility. There is little more to say about it: they're still permitted because it would be impossible to get a change that broke people's existing software without an extremely good reason. And people still use them, because they haven't upgraded their infrastructure to use segwit transactions - the engineering costs to perform this upgrade may be (rightfully or not) considered higher than what would be gained by upgrading.

Nit: The blocksize limit is dead, long live the blockweight limit.

@Murch Yes, though I wanted to avoid going into those details. Describing the witness as being discounted 0.25 towards the 1000000 limit is equivalent to the actual specification, I think?

So why is the witness section discounted? Is it somehow cheaper to store and transfer?

It is cheaper because the rules say it is. The justification for that rule is that indeed, they don't burden the network as much: outputs add to the UTXOs set, but inputs remove from it. Only inputs have a witness section, so a transaction with more inputs (i.e. one that will be discounted a lot) is less of a burden on the network.

There needs to be incentive for the use of a more efficient type of transaction which has lower concern to network scalability.


text:
What speaks against dropping the percentage fee in lightning?
If the goal of the lightning network is to route payments as cheap as possible (i know this is an assumption and maybe not everyone agrees), why wouldn't it be a good idea to just drop the percentage fee? A fee in general is necessary to prevent spamming, that is why no fee wouldn't work. Is that assumption correct?

Is that assumption correct?

Rene points out, that fees are not preventing spammers from spamming the network.
That leaves the question: Why not drop the percentage and keep the base fee?

With the current design of the lightning network fees do not prevent any spam at all. The price for setting up htlcs in remote channels is independent of the fee schedule only paid if the htlcs settle. An attacker who wants to spam will just not settle the HTLCs and thus will never pay any fee

thanks for clarifying this - will update my question

The lightning network is still young, so I don't think there is an absolute consensus amongst users on what the optimal fee policy is. That said, one consideration could be:
When deciding your lightning node's fee policy, one consideration is simply liquidity allocation. Generally speaking, channels that are well balanced will be more useful than channels that are unbalanced (ie, all of the funds sitting on one side). Routing a payment means pushing the channel balance around, so if someone pushes payments that unbalance your channel(s), you may want some compensation, to help cover any potential costs of rebalancing your channel.

thanks - this could answer the question why any fees at all. but i can get the needed funds by setting a base fee and not a percentage fee

I think the main reason that speaks agains dropping the fee rate and keeping the base fee instead is that finding optimal payment flows would become an NP-hard problem for nodes who wish to make a payment (if they try to find a cheap solution which I assume anyone would like to do)
If however we keep the fee rate and drop the base fee the problem to find the cheapest payment flow through the market of offered channels becomes a computationally much simpler problem.

why would using something variable be better than using something fixed?


text:
Why are OP_RETURN transactions discouraged? Does using version or locktime make any difference?
Why are OP_RETURN transactions discouraged? Are they paying less fees compared to other transactions?
Will it make any difference if same thing is done using version or locktime when data is either 1 or 2?
Example:
OP_RETURN
createrawtransaction '[]' '{"data":"32"}'

LOCKTIME
createrawtransaction '[]' 2

VERSION
Change source code and make this configurable although it's already 2 so don't need to change for this example: https://github.com/bitcoin/bitcoin/blob/7fcf53f7b4524572d1d0c9a5fdc388e87eb02416/src/primitives/transaction.h#L263

You can't achieve the same thing using the parts without consensus meaning of nSequence + nLockTime (like Lightning does for example). There is just not enough space: 48 "free" bits.
Changing the version is a bad idea, as it could be used to opt-in to a future soft fork which rules haven't been defined yet. Transactions with a version < 2 are also (and therefore) not standards.

Thanks. Just realised that there is a huge difference in size (4 bytes vs 80 bytes) if we compare OP_RETURN with other two. Also locktime is only useful if it's between 0 and last block, version is limited to 1 and 2 for standard transactions. Still not sure why are OP_RETURN transactions discouraged? A 40 byte OP_RETURN tx and a normal segwit tx of 40 byte should be considered same by full nodes? Both are paying fees for block space.

OP_RETURN scriptPubKeys should be prefered to carry data because they can be easily identified and pruned. The UTXO database space is a much scarcer resource than the block chain one.

Whether OP_RETURN transactions are discouraged or not is a matter of opinion.
Personally, I think transactions should not store any data that isn't needed for the world to verify them, which is by definition the case for the data stored in OP_RETURNs. Either you're using it to use the blockchain as data storage (which is a ridiculously inefficient technology for the purpose it serves, and under sufficient fee pressure, very expensive), or you're using it to communicate something to the receiver which could have been done out of band instead.
So it isn't a critique on OP_RETURN itself - it's a critique on what you're trying to do with it. It would equally apply to trying to use locktime or version or sequence numbers for the same purpose: in my view, this is all just building technically inferior solutions that should use other approaches (not using the blockchain) entirely.
There is no price difference; the bytes in the OP_RETURN count for just as much as any other (non-segwit) bytes. So why is this concerning? The issue is that every transaction, even ones with outputs that don't make it into the UTXO set, have a cost to the network. Every full node has to download and process it. Transactions pay a fee, but that fee goes to miners, not to full node operators. There are various reasons for running a node, but all of them in one way or another relate to it enabling participation in the Bitcoin economy. They don't do it for dealing with your personal and/private data.
I don't think this is an issue in the long term - fee pressure will push (and to a large extent has pushed) out pointless uses of the blockchain to other systems. However, the Bitcoin ecosystem is still relatively young, and to not stifle its growth, I find it wise to discourage people from burdening nodes with transaction data that does not help the ecosystem.
That said, if for whatever reason you're going to be creating a data storage solution based on the blockchain anyway, by all means, use OP_RETURN. At least that way you're burdening the blockchain, and not the UTXO set.

I can understand UTXO set argument however 'burdening the blockchain' is beyond my understanding. Bitcoin halving reduces rewards in every 210,000 blocks. So Bitcoin needs transactions and fees to survive in long term. If a project increases Bitcoin transactions and fees, it's net positive and should be encouraged. Full node has to verify everything so how does it matter what is being verified?

If people start using blockchain transactions for activity that doesn't contribute to the bitcoin economy (e.g. aren't related to economic activity that involves BTC moving), that's an opportunity cost for others who do. This is mostly a worry about systems like omni and counterparty which parasitically use the blockchain for economic activity of competing systems. Early on - and perhaps not anymore - they posed a very real threat IMO to bitcoin by pricing out real BTC activity.

@Prayank's argument seems like planning for failure to me. Yes Bitcoin needs fees to survive in the long term. But at least for me Bitcoin will have failed if the only way it can get those fees is through OP_RETURNs. High value transactions, Lightning channel opens and closes, whatever other L2 protocols need from the blockchain will be competing in the success case. And those use cases would price out the OP_RETURN use case. I don't think Bitcoin will be economically significant long term if OP_RETURNS are helping to prop up transaction fees.