text:
Segfault when clearing mempool. What am I doing wrong?
Background:
For a research project, I am trying to periodically clear the content of my Bitcoin node's mempool. For this purpose I have created a new thread in init.cpp using the boost threadgroup (I have experimented with this successfully with a different project in the past and I am sure this is not the cause of the problem) in the Bitcoin Core software. In the new thread, I periodically call a function func (defined below), the sole purpose of which is to clear the content of the node's mempool.
void func()
{
    LOCK2(cs_main, mempool.cs);
    AssertLockHeld(cs_main);
    AssertLockHeld(mempool.cs);
    mempool.clear();
}

I check for and lock critical sections that I think I should before I modify the mempool (the code above mimics what happens when a new transaction is added to the mempool; I basically looked at what happens in the NetMsgType::TX branch of ProcessMessage in net_processing.cpp and then AcceptToMemoryPoolWorker in validation.cpp and mimic that).
Issue:
When I run Bitcoin after successfully compiling my code, I sometimes immediately (i.e., as soon as the GUI window pops up) and sometimes much later in the execution face a segfault that causes the program to crash.
ubuntu@root:~$ bitcoin-qt -mpctimeout=1
Segmentation fault (core dumped)

In the code above, the function func is executed every 1 minute (represented by mpctimeout=1).
Question:
What could be causing the segfault? Could it possibly be race conditions when trying to get a lock on the mempool or execution of the main thread? It is my understanding that LOCK2 should keep trying to get a lock on the critical sections when they become available in case they are already locked by another thread/process. Since these locks only exist within the scope of the function func, they should ideally be released when returning from the function.
Edit # 1:
Some debugging shows that the segfault doesn't occur while func is being executed but while the thread is put in sleep for mpctimeout amount of time. This leaves me wondering whether clearing the mempool has some long-term effects? I have made sure that the lock is released by verifying that transactions are indeed accepted to the mempool while the new thread is asleep. The segfault still occurs even if the mpctimeout value is large (e.g., 1 hour).
FWIW, I use boost::this_thread::sleep_for to put the thread to sleep. I assume this works without fault and that the segfault is instead caused in the long run due to clearing the mempool. Any tips?
Edit # 2:
Below is the output of debugging the binary through gdb:
(gdb) run
Starting program: /usr/local/bin/bitcoin-qt
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe9900700 (LWP 25643)]
[New Thread 0x7fffdf37f700 (LWP 25644)]
[New Thread 0x7fffdeb7e700 (LWP 25645)]
[New Thread 0x7fffdd0e7700 (LWP 25646)]
[New Thread 0x7fffc7cd2700 (LWP 25648)]
[New Thread 0x7fffc74d1700 (LWP 25649)]
[New Thread 0x7fffc6cd0700 (LWP 25650)]
[New Thread 0x7fffc64cf700 (LWP 25651)]
[New Thread 0x7fffc5cce700 (LWP 25652)]
[New Thread 0x7fffc54cd700 (LWP 25653)]
[New Thread 0x7fffc4ccc700 (LWP 25654)]
[New Thread 0x7fffa7fff700 (LWP 25655)]
[New Thread 0x7fffa77fe700 (LWP 25656)]
[New Thread 0x7fffa6ffd700 (LWP 25657)]
[New Thread 0x7fffa67fc700 (LWP 25658)]
[New Thread 0x7fffa5ffb700 (LWP 25659)]
[New Thread 0x7fffa57fa700 (LWP 25660)]
[New Thread 0x7fffa4f63700 (LWP 25661)]
[New Thread 0x7fff8dffd700 (LWP 25662)]
[New Thread 0x7fff8d7fc700 (LWP 25663)]
[New Thread 0x7fff8cffb700 (LWP 25664)]
[New Thread 0x7fff7bfff700 (LWP 25665)]
[New Thread 0x7fff7b7fe700 (LWP 25666)]
[New Thread 0x7fff7affd700 (LWP 25667)]
[New Thread 0x7fff7a7fc700 (LWP 25668)]
[New Thread 0x7fff79ffb700 (LWP 25669)]
[New Thread 0x7fff797fa700 (LWP 25670)]
[New Thread 0x7fff78ff9700 (LWP 25671)]
[New Thread 0x7fff53fff700 (LWP 25672)]
[New Thread 0x7fff537fe700 (LWP 25673)]
[New Thread 0x7fff4a174700 (LWP 25677)]
[New Thread 0x7ffed1a7b700 (LWP 25678)]
[Thread 0x7ffed1a7b700 (LWP 25678) exited]
[New Thread 0x7ffed127a700 (LWP 25679)]
[New Thread 0x7ffed0a79700 (LWP 25680)]
[New Thread 0x7ffecbfff700 (LWP 25681)]
[New Thread 0x7ffecb7fe700 (LWP 25682)]
[New Thread 0x7ffecaffd700 (LWP 25683)]
[New Thread 0x7ffeca7fc700 (LWP 25684)]
[New Thread 0x7ffec9ffb700 (LWP 25685)]
[New Thread 0x7ffec97fa700 (LWP 25686)]
[New Thread 0x7ffec8ff9700 (LWP 25687)]
[Thread 0x7ffed127a700 (LWP 25679) exited]
[Thread 0x7ffecb7fe700 (LWP 25682) exited]

Thread 40 "bitcoin-msghand" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffec9ffb700 (LWP 25685)]
__gnu_cxx::__atomic_add_dispatch (__val=1, __mem=0x1a45fe36190ee438)
    at /usr/include/c++/7/ext/atomicity.h:96
96        __atomic_add(__mem, __val);
(gdb)

I suppose that the segfault is caused when trying to update an atomic variable.
Edit # 3:
gdb backtrace
#0  0x0000555555854178 in __gnu_cxx::__atomic_add_dispatch (__val=1, __mem=0xf) at /usr/include/c++/7/ext/atomicity.h:96
#1  0x0000555555854178 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_add_ref_copy() (this=0x7) at /usr/include/c++/7/bits/shared_ptr_base.h:138
#2  0x0000555555854178 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count(std::__shared_count<(__gnu_cxx::_Lock_policy)2> const&) (__r=..., this=0x7ffef5ff71b8)
    at /usr/include/c++/7/bits/shared_ptr_base.h:691
#3  0x0000555555854178 in std::__shared_ptr<CTransaction const, (__gnu_cxx::_Lock_policy)2>::__shared_ptr(std::__shared_ptr<CTransaction const, (__gnu_cxx::_Lock_policy)2> const&)
    (this=0x7ffef5ff71b0) at /usr/include/c++/7/bits/shared_ptr_base.h:1121
#4  0x0000555555854178 in std::shared_ptr<CTransaction const>::shared_ptr(std::shared_ptr<CTransaction const> const&) (this=0x7ffef5ff71b0)
    at /usr/include/c++/7/bits/shared_ptr.h:119
#5  0x0000555555854178 in CTxMemPoolEntry::GetSharedTx() const (this=0x7ffee12106a0) at ./txmempool.h:101
#6  0x0000555555854178 in PartiallyDownloadedBlock::InitData(CBlockHeaderAndShortTxIDs const&, std::vector<std::pair<uint256, std::shared_ptr<CTransaction const> >, std::allocator<std::pair<uint256, std::shared_ptr<CTransaction const> > > > const&) (this=this@entry=0x7ffef5ff9f30, cmpctblock=..., extra_txn=std::vector of length 100, capacity 100 = {...})
    at blockencodings.cpp:114
#7  0x00005555556a2d4a in ProcessMessage(CNode*, std::__cxx11::string const&, CDataStream&, int64_t, CChainParams const&, CConnman*, std::atomic<bool> const&, bool)
    (pfrom=pfrom@entry=0x7ffedc0010d0, strCommand="cmpctblock", vRecv=..., nTimeReceived=1593549860206352, chainparams=..., connman=0x7fffbc04ba20, interruptMsgProc=..., enable_bip61=false) at net_processing.cpp:3051
#8  0x00005555556a6f01 in PeerLogicValidation::ProcessMessages(CNode*, std::atomic<bool>&) (this=<optimized out>, pfrom=0x7ffedc0010d0, interruptMsgProc=...) at net_processing.cpp:3629
#9  0x000055555566b19e in CConnman::ThreadMessageHandler() (this=0x7fffbc04ba20) at net.cpp:1966

Did you know that you can run with no mempool using blocksonly? I fail to see any reason to manually clear it every minute.

@Anonymous I only clear the mempool every minute to make sure what I am trying to do works. In my experiment, this is going to be a larger value. My research project requires clearing the mempool so I cannot use the blocksonly mode.

Have you tried running this in GDB and just waiting for it to segfault? It should tell you where that happens. What is probably going on is that something has a reference to a mempool entry but when you clear the mempool in another thread, using that reference causes a segfault.

Why do you need to periodically clear the mempool? What in particular are you trying to research? I'm sure there's a way to do this without modifying the Bitcoin software. It should be possible to emulate the mempool state by having another software connected via ZMQ.

@AndrewChow I have added the output of the running the binary through gdb. It seems that the fault is caused when trying to update an atomic variable. Do you have any tips on which variables I should specifically debug (since there are so many in the software)?

@AndrewChow re: "What is probably going ... causes a segfault."
I assumed that the mempool.clear() function (implemented in txmempool.cpp) should have taken care of any references to the mempool whenever it is cleared. Could there be something missing in the function then?

@an4s In gdb, when you hit the segfault, use the bt command to get a backtrace. This will tell you exactly where the segfault occurred. Clearing a data structure will not clear the variables that point to data that was inside of it. There are probably variables in other functions in other threads that have pointers or references to now-cleared mempool data.

@AndrewChow thanks for the tip. The backtrace and some more debugging shows that the fault is caused when a call to GetSharedTx() is made in InitData() for a partially downloaded compact block in net_processing.cpp. I suppose then it makes sense to delay clearing the mempool if there are still blocks in flight. AFAIK, the mapBlocksInFlight data structure keeps track of all blocks currently in flight. However, it is defined in a namespace defined in net_processing.cpp and it may be tricky to access it in my new/custom thread. Is there another way to check for blocks in flight?

I'm still not quite sure why you want to do this. There's almost certainly an easier way to get the information you want. All this will cause is a huge amount of rejects on your node.

Could you say why you are trying to do this? What are you trying to observe. As nonted by Anonymous, this will have a lot of side effects on your node and potentially give you erroneous or biased results.

@AndrewChow We published a paper at ICBC'20 on the topic of orphan transactions in Bitcoin and are performing additional experiments to understand when such transactions are most likely to show up. Along that line, the purpose of clearing the mempool is to see how likely it may cause transactions to become orphans.

Instead of clearing the mempool like you are doing, could you turn off bitcoin core, delete the mempool.dat file and then start it again? This will clear the mempool...

@Pedro yes, I already started doing that when clearing the mempool from within the software started resulting in segfaults. I am still wondering if I can make this work (hence this question). Thanks!

@an4s That concept doesn't really make a whole lot of sense, as a question to be asking to begin with. Transactions arriving without pre-existing inputs in your UTXO or your mempool are pretty rare as they're not relayed around the network. By "clearing" your mempool you're just going to result in seeing huge numbers of them artificially.

@Anonymous the idea is to emulate churn.

@an4s Can't that be achieved by just setting your mempool limit arbitrarily low?

@an4s In any case, I'd suggest talking to some of the bitcoin domain experts about what you're doing and get some feedback. On the surface of it what you're describing sounds a little mislead and could probably do with some input to make it more relevant. You're more than welcome to ask me questions in the chat here if you want to.

@an4s I believe I've read your paper, there's some serious issues with the conclusions you've drawn from faulty bitnodes.io data. There is not significant churn of peers on the Bitcoin network, as evidenced by my node having hundreds of extremely long lived connections, which would not be the case if the claim that there was a significant amount of churn in operational nodes was correct. There's simply no property which would cause this to be the case.

The only place clear is used on the mempool is after all peers are down. Presumably there are references to it that you can't just leave dangling.
See CTxMemPool::Expire in txmempool.cpp for how expiration works.  It wouldn't be hard to modify that existing function to just remove everything at a particular point.
I assume you're the author of http://people.bu.edu/staro/Characterizing_churn.pdf ?
I saw that paper back on May 22nd and I found the claims of the paper to be false to the best of my ability to determine.  If the paper was correct nodes would see large numbers of getblocktxn messages-- patching in missing transactions that showed up in blocks, yet they see fairly few.
To give some concrete figures, yesterday my node received 210 getblocktxn requests from ~66 peers. There were 162 blocks. So that is about 0.0196 request per block*peer. I sent 3097 compact blocks during the same interval. My own rate of having missing transactions and needing a getblocktxn was 6.7%, which is not wildly inconsistent from what I see with other peers: it's higher because I'll only see 1/N-th my peers getblocktxn, since I'm not always the first to send them a block. But if I assume that the rate that I sent compact blocks (28.9%) reflects the rate that I'm first, I estimate my peers getblocktxn rate to be almost exactly the same as mine.
I did more analysis with multiple nodes back when I first saw the paper, but I'm not sure where I put the figures. :)
I believe the paper makes an error taking stats from "bitnodes":   There are enormous numbers of fake nodes on the networks which don't behave much all like real nodes-- e.g. for the purpose of spying on transactions to deanonymize bitcoin users or academic research, most of "nodes" displayed on bitnodes are of this sort. It's irrelevant to the network if fake 'monitoring' nodes churn a lot.
You might also be interested in the FIBRE protocol, which renders block propagation times close to to independent of the presence of a few missing transactions (Can someone please explain FIBRE to me like I'm 5 and why is it useful?) and Erlay (https://arxiv.org/abs/1905.10518) which uses reconciliation for transaction relay  to reduce relay overheads.

Thank you, Gregory, for pointing to CTxMemPool::Expire. I'll see if I can find what I am looking for. I also appreciate you sharing your analysis. I don't mean to point fingers or offend anybody in general but if what you say is true, then this surely raises many questions towards several past research work and blog posts from well-known websites in the Bitcoin space that use data from bitnodes.


text:
Do we still need SIGHASH_NOINPUT for Lightning Network?
In the original Lightning whitepaper, Joseph Poon advocates using SIGHASH_NOINPUT so that we can create the first commitment transaction which references the funding transaction txid without actually signing the funding transaction. However that paper was written before SegWit was implemented. Given that SegWit transactions do not have signature component in the txid, it removes the need to actually implement SIGHASH_NOINPUT.
However, today I came across a draft proposal of the sighash in BIP 118 which says it would be active with SegWit v1 and higher.
Am I missing anything else that this sighash flag brings to the table that would warrant it to be implemented in future SegWit versions?

For the most in-depth discussion, search the bitcoin-dev mailing list for SIGHASH_NOINPUT https://lists.ozlabs.org/pipermail/bitcoin-dev-moderation/

The lightning network as is is working without SIGHASH_NOINPUT. Thus in its current form it is not needed.
However Christian Decker came up with the eltoo payment channel construction. In a nutshell this would get rid of the penalty based construction we use right now and reduce a lot of overhead for maintaining channel state as the state becomes symmetrical for all participants.
In particular multiparty channels and channel factories would become much easier.
The eltoo paper however really depends on SIGHASH_NOINPUT as the channel state is encoded as a "linked" list of update transactions which in case of protocol breach are skipped from the breaching point to the current channel state which requires SIGHASH_NOINPUT.
It is my understanding that essentially all lightning network developers agree that lightning network channels should eventually be constructed via eltoo.
You can read a summary of eltoo on my blog or you can read the original publication to learn more.

Thanks for your answer. Unfortunately the formatting of your blog looks corrupted.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What software should I use to open the .dat flles
I've just downloaded the Bitcoin Core for Windows.
I can see several .dat file. What can I use to open them. i.e. Is there a way to view them in plaintext?
Also, which file is the blockchain file?
Thanks!

you can use console Windows parser from ragestack. This is a small utility for converting *.dat files to *.txt

@Denis, got an error when trying to run it.

what error? it’s work perfectly with dat files without any problem. you read readme file with instructions for how to use it?

The .dat files are opened by Bitcoin Core, they are not supposed to be opened in plaintext. The blockchain will be in the blocks/ subdirectory. You can see a list of the files bitcoin core uses here:
https://github.com/bitcoin/bitcoin/blob/master/doc/files.md

These files are in binary format, so not intended for a human reader. To have a look at the content, get a "hex reader" program, google for that.
Do you want to get the private keys to your Bitcoin adresses? If so, you will use the "debug window" in Bitcoin Core. See this answer for how to do that: https://bitcoin.stackexchange.com/a/5933/63522

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Security concerns when downloading the block chain from a 3rd party source?
It takes very long until bitcoin-core has downloaded the whole block chain. So I decided to download the block chain files via a bittorrent published by this website: http://www.getbitcoinblockchain.com
After the download was finished I noticed that the md5 checksums of the files from the torrent were different then of the files from bitcoin-core.
Why could that be?
Are there any security related concerns of using a block chain from a 3rd party source?

After the download was finished I noticed that the md5 checksums of the files from the torrent were different then of the files from bitcoin-core.
Why could that be?

The download they provide is a Bitcoin Core data directory. The data directories of two nodes are not likely to be the same.
This is because blocks are downloaded out of order. Since blocks are written to disk in the order that they are received, the blocks will be in a different order from node to node and thus the hash of all of the data will be different.
Furthermore, orphan blocks causes the data to be different because some nodes might have heard of an orphan block but another node did not.

Are there any security related concerns of using a block chain from a 3rd party source?

If it is more than just the blockchain but instead an entire data directory (includes folders like chainstate, blocks/index, etc.), then yes, there is. The concern lies with the chainstate database. This database contains the UTXO set and is used for validation. If the attacker provides a chainstate database that contains invalid entries, your node would consider those entries to be valid and thus accept invalid transactions that spend those entries.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What is better, pre-existing Bitcoin-core API bindings for python or custom wrapper functions
How advisable is using the following code to query the bitcoin core as compared to using some python RPC library.
import os
btc_prefix = 'bitcoin-cli '

def getblockcount(btc_prefix):
    print("in getblockcount")
    cmd = ' '.join([btc_prefix, 'getblockcount'])
    response = os.popen(cmd).read()
    return int(response)
getblockcount(btc_prefix)

It is kind of a wrapper function for the core API's. What are the drawbacks/advantages of using the above code. Also, does it affect security in anyway.

bitcoin-cli is just a wrapper for the RPC functions so you're essentially doing the same exact thing as an RPC call. Generally, using RPC is preferred since you don't have to rely on the bitcoin-cli executable being on the system and in your current path. This most likely won't work on windows, for example.
In general RPC is preferred because you can access it from other systems. With your program, you'll only be able to run it on the system that has bitcoin core installed whereas with RPC, you can run from other systems.

It also has far lower overhead, as you don't need to start a new process for each request

@PieterWuille What has a lower overhead?

@Jimmy There isn't a security flaw as such in my approach ?

Using bitcoin-cli is very slow. Use JSON-RPC from your program directly; it will be much more flexible and faster than using Bitcoin Core's wrapper binary.

The security flaw in getblockcount is that an attacker can execute anything on your system (e.g. pass in btc_prefix as rm -rf /). I would strongly advise you use JSON-RPC which will only execute what you want.

@JimmySong  But then as you mentioned that the approach I am using does not allow anyone accessing bitcoind remotely(which is required in my case). And it is an internal module not exposed to any API where you can pass btc_prefix. It is saved in a config file. Does it pose any similar security flaw given the conditions.

If your internal app is locked down from a security perspective, then of course it's going to be secure whatever code you run on it. Generally, when you ask for a security audit, you assume some external party has access.


text:
Best way to store private key from address
I make my own hot-wallet and wanted to know best way to store private key from address, because I need it for signing transactions.
I tried store private key in my bitcoin core node, but I realized that this is not best way. Maybe you know other way do it or how to get private key for transaction.

@Prayank Because, I thought that if someone gets access to bitcoin node(rpc server) then he can get private key from address.

@Prayank Yea, that's why I asked this question)
At your discretion which way is best for you?

If you can afford using several devices, you could store private keys in an air gapped cold storage device isolated from the server which exposes Bitcoin RPC. Would that work?

@Mercedes What do you mean about "isolate Bitcoin RPC server" ?
If you mean store private key in remote Bitcoin RPC wallet, I did it. I just want to know other ways to store private key

I mean have private keys in cold storage device, and to not have private keys in the node exposed through RPC. Cold storage is widely written of on the Internet.

@Mercedes I'll think about it, thanks, but I think that cold wallet it better for mobile device not for web wallet, but I can wrong)

@Prayank @Mercedes I've just looked that Bitcoin RPC have encryptwallet, and I'll be encrypt wallet password of user and then unlock wallet and get private key, I thinks it will be good.

A large enough Bitcoin stash on hot wallets, even password protected, is vulnerable to zero day vulnerabilities. Cold storage is a lot safer, specially if the safe room can be made free from electronic nano insects and other less futuristic intrusion menaces such as electromagnetic sniffing.

@Mercedes I know that cold wallet it's the best idea for store your keys, but for web wallet/exchange-swap it will be difficult(


text:
I've built a bitcoin miner and mined. Now what I do?
I've mined bitcoin to test and understand cryptocurrency better. Which generated this hash 00000431fb66ed96815b3829e975c2235a64b9a646c0bd1cc526dc1727c1f16b(nonce value of 300712) What I do with this now? Do I go to blockchain and submit it? How can I do that? Can someone explain me? I'm very new to cryptocurrency and I'd like to learn more about it.

That looks like a low difficulty. What chain did you mine?

Yeah it was to test. It's lvl 5 difficulty which is mined by python 3.9

To extend the blockchain, you have to mine a block at the current difficulty. The current difficulty is 20,082,460,130,830, so just a tad more than 5. You should check out https://bitcoin.stackexchange.com/q/148/5406, and https://bitcoin.stackexchange.com/q/41276/5406.

How do you view the current difficulty and how can you even get the reward when you mine that? I have some many questions

Block headers commit to the previous block hash and the difficulty, so you literally cannot begin mining without knowing this. Typically you'd get this information from a full node; for example using the getblocktemplate RPC call of Bitcoin Core.

generated this hash [...]. What I do with this now?

I'm pretty sure the things you should do include:

Check that the prior block referenced by your block template is still at the head of the chain.
Ensure your block will pass all the consensus validation rules.
Identify a handful of peer nodes using Bitcoin's peer discovery process/protocol
Send a Bitcoin message to those peers stating that you have a new block
Listen for incoming requests for that block and respond to them
Wait for another 100 blocks to be added to the chain that has your block in it.
Spend your block reward or continue to retain it

See also

What exactly is Mining?
Whats the process of Creating a Block on the BlockChain

How do you view the current difficulty

By viewing the difficulty value in a few recent blocks and if the difficulty interval is here, by applying the difficulty recalculation algorithm.

How is difficulty calculated?

and how can you even get the reward when you mine that?

The reward is in the coinbase transaction you added to your block template before hashing. You can spend it 100 blocks later (see above)

How do mining rewards get verified?


text:
How do I get my purchsed bitcoins from a old bitcoin wallet?
I had to refresh my computer do to updates and all my installed software such as Bitcoin wallet was gone after the process. I did back up my wallet almost everyday and also I did it on a external hard-drive, but when I re-install the Bitcoin wallet from bitcoin.org it give me a new receiving address and even if I have all back -up data and my previous receiving bit coin wallet where all my bit coins I have purchased were I do not know why is it not showing it in my new wallet. I have tried to recover and open my back up wallet files but it does not show my Bitcoins. Please help.
I did watch the recovery do to a failed hard drive but I can not access the files and I do not know how to get my Bitcoins from my old receiving wallet which I backed up on 2 places.

OK can you specify some more information, like which wallet software you are using, what EXACTLY did you back-up.

Go to your backup file of %appdata%/bitcoin/wallet.dat
Go to your local file of %appdata%/bitcoin/ and delete the wallet.dat
Replace the old Wallet.Dat

You should have all your bitcoins back now.


text:
Unknown new rules activated (versionbit 1) when running Core v22.0.0 on testnet
I have got "Unknown new rules activated (versionbit 1)".

I am running Bitcoin Core v22.0.0 on testnet.
This seems to be the last version as of today (April 17 2022).
I am running a taproot wallet created as per this guide.
I have checked signatures and sha256 sums before installing bitcoin core.
Any ideas what is this?

Does this answer your question? "New rules activated" warning after upgrading to 0.20.0 or possibly Impact: unknown rules (versionbit 2) have been activated

@RedGrittyBrick I have read that and not sure whether it apply. I have a fresh new install. Bitcoin core is up-to-date . Reloading do not help. Warning stays.


text:
Can multiple wallets send 1 transaction for the combined balance?
I have multiple wallets, I want to transfer all of the money to 1 wallet of some other person. But I want to do this as one transaction to avoid fees. Is this possible with bitcoind API?

For programmers interested in the underlying technical question here (unrelated to BitcoinD service), the Bitcoin protocol itself is capable of making a transaction from multiple inputs of different wallets according to this answer here: https://bitcoin.stackexchange.com/a/51687/4325 --- I have not verified this claim, am just sharing for reference.

If you are using a wallet which supports importing private keys, such an Bitcoin Core, you can just import the private key of your other wallet: https://bitcoin.stackexchange.com/a/5944/14325

authors: Александр Кузнецов


text:
Trickle Exchange System?
This idea came to me while I was trying to trade Frozencoins. For a miner like me that has to wait ages to get coins, its nearly impossible to trade small amounts of Frozencoins. People usually ask for huge amounts, from a hundred to 10,000. What if, people who made buy orders had the option of a "trickle" payment?
Basically, lets say that I only have 100 Frozencoin for X amount of Bitcoins but someone wants to buy 200. A trickle exchange would allow the person buying the Fronzencoins to receive multiple payments from different people with varying amounts until he reached the total he wanted. The people would get paid back ( in the cryptocurrency he/she was trading them for ) at the exact same rate ( Maybe 0.00002 BTC for every 1 FZ ). If he/she thought the wait time is too long, he/she can keep the coins and cut off the buy order. Is this possible? If so, what would it take to implement it?

Yea this is possible, there's nothing stopping you from making a new exchange that does things differently. Although to start making one you're going to need serious programming knowledge.
Consider the following questions

Is there an open source Bitcoin exchange?
What is needed to make a bitcoin exchange?


text:
Stale block verification at node level
It comes to my curiousity about how exactly the node A will mine block N+1 when he & node B generated block N (not in same time but) in a fairly close timeframe.
In many articles, it says that when 2 nodes (A&B) are generating the same height block at same time, it will cause a fork. This means some nodes will mine after A's new block and others mine B's. This is understood. However, it will take time when A's new block sync to B. Let's assume it takes 10 seconds to pass to B. In the middle of the sync (saying 5th or 6th second of total 10), B just generates the new block (with same height), and propagate it out before receiving A's block. What will happen when B receives A's block then?
I have 2 possible answers but not sure which one is correct, or neither.
(assuming all TX/UTXO in the block has no issue)
ANSWER-1: B will discard B's block, and accept A's, because B block's timestamp is older than A's (make sense?)
ANSWER-2: when B broadcasts his new block, B by default accepts its own generated block (N+1) and immediately start to mine block N+2 upon that (at the time when B has not received A's block yet.) Then B receives A's block (N+1), in this case, B should keep A's chain but still continue to mine at his N+2 block height.
Any advice?

ANSWER-2: when B broadcasts his new block, B by default accepts its own generated block (N+1) and immediately start to mine block N+2 upon that (at the time when B has not received A's block yet.) Then B receives A's block (N+1), in this case, B should keep A's chain but still continue to mine at his N+2 block height.

This is basically correct. B will receive A's block, validate it, and if it is valid, he will store the block. However he will not change his view of the blockchain to include A's block. He merely holds onto A's block just in case more blocks are found on top of it which would then require B to change his view of the blockchain.
B's view of the blockchain will remain with his N + 1 block and continue to mine his N + 2.
In general, when two blocks are found at the same height around the same time, nodes will update their view of the blockchain to include the block which they saw first. The view of the blockchain only changes if there exists a blockchain that has more cumulative work than the one that the node currently sees as the correct blockchain. If the two blockchains have the same cumulative work (i.e. orphan race as described in this question), then the node uses the one it saw first.

FWIW, Bitcoin Core won't fetch (much less validate) a block unless it would become the best block if validation passed.

Or is at least part of a chain whose tip would become the best block if validation passed.

Two miners may find a block at pretty much the same time, and so the network may split, with some miners working on top of each of the two blocks.
It is possible, but quite rare, for a miner on each side of the split to then find a block at pretty much the same time again, thus repeating the situation, and the split goes another block deep.
The situation eventually resolves when one side of the split mines a new block before the other side. The network then converges onto the longer (most work) chain, and the other chain is 'orphaned'.

i have 2 possible answers but not sure which one is correct, or neither. (assuming all TX/UTXO in the block has no issue)
ANSWER-1: B will discard B's block, and accept A's, because B block's timestamp is older than A's (make sense?)

No, timestamps are not used to decide which block is kept.

ANSWER-2: when B broadcasts his new block, B by default accepts its own generated block (N+1) and immediately start to mine block N+2 upon that (at the time when B has not received A's block yet.) Then B receives A's block (N+1), in this case, B should keep A's chain but still continue to mine at his N+2 block height.

B will not necessarily keep the chain, but B will continue to mine on B's N+1 block.


text:
`getaddressinfo` incorrect output for `ismine`
Running Core, trying to verify that I infact control a change address before I sign and broadcast a txn. But I can't find any way to list all future change addresses.
I searched and found listaddressgroupings but that only seems to show previously used change addresses.
Sipa on IRC suggested I try getaddressinfo, but I was alarmed when the output said ismine:false
I then tried using getaddressinfo again, but this time with one of the already used change addresses that already has coins. I used the first output from listaddressgroupings, but then again it returned ismine:false
So I believe there is a bug with the "ismine" output
Specter Desktop originally set up this wallet as a watch-only wallet in Core
Does anyone know how to view all future change addresses in Core? Specter wallet can show these to me. I believe Electrum can as well if I remember correctly.

edit
getwalletinfo output:

22:44:52
￼
getwalletinfo

22:44:52
￼
{
  "walletname": "specter3c64f804925fff38\\mymultisig",
  "walletversion": 169900,
  "format": "bdb",
  "balance": 0.00000000,
  "unconfirmed_balance": 0.00000000,
  "immature_balance": 0.00000000,
  "txcount": 64,
  "keypoololdest": 1635129892,
  "keypoolsize": 0,
  "keypoolsize_hd_internal": 0,
  "paytxfee": 0.00000000,
  "private_keys_enabled": false,
  "avoid_reuse": false,
  "scanning": false,
  "descriptors": false
}

and if those balances in the output for getwalletinfo are supposed to show my BTC, they are wrong... the Overview tab clearly shows my correct BTC amounts, and they are not 0

So I believe there is a bug with the "ismine" output

This is expected behavior. Your wallet is a watch only wallet (as it says "private_keys_enabled": false,) which means that the wallet is itself unable to spend coins associated with any addresses it is tracking. Furthermore, it is a legacy wallet (because it says "descriptors": false) which means that the meaning of ismine is somewhat complicated.
For legacy watch only wallets, ismine will always be false because the wallet itself lacks the private keys necessary to spend the Bitcoin.

Does anyone know how to view all future change addresses in Core? Specter wallet can show these to me. I believe Electrum can as well if I remember correctly.

Because Specter set up the wallet, you should be using Specter to view everything. Your type of wallet is not capable of generating future addresses, particularly the multisigs that you are using.
Additionally, it is not particularly useful for you to look for your addresses in Bitcoin Core. The private keys are not handled by Bitcoin Core; even if this wallet is watching for transactions to your addresses, you may not be able to spend them. You should be looking at the software or hardware which contains your private keys.


text:
How to identify Taproot transaction using big query?
From bitcoin magazine, I saw that

When Taproot/Schnorr goes live soon, they will use the new witness version one to distinguish them from older Segregated Witness transactions.

But when I check the type of inputs/outputs after Nov 14, I only see witness_v0_scripthash, witness_v0_keyhash, scripthash, multisig, nonstandard, witness_unknown and pubkeyhash.
Should the new taproot txns be labelled as witness_v1? or is there any other way to identify them on big-query?
Thanks

Where are you getting your data from? witness_v0_scripthash etc isn't blockchain data, it is manipulated data from the blockchain. Sounds like wherever you are getting your data from hasn't upgraded to recognize witness_v1 etc

BigQuery is just a data warehouse, I'm assuming it isn't manipulating and presenting  blockchain data to you. Please correct me if wrong!

Yup, this is processed data from Big Querys Crypto Bitcoin schema. I'm very new with this, and didn't find any way to extract bitcoin blockchain data in an sql style, so that's why I was using it. Thanks

You clarified in the comments that you are using BigQuery datasets and so you are right Taproot transactions should eventually be labelled as witness_v1_.... Taproot only activated a couple of days ago at the time of writing so presumably the authors haven't added Taproot support yet. You can either wait for them to do so (or request they do!) or find an alternative SQL data source that has added Taproot support. If it is the latter feel free to ask a separate question on this site, I'm not familiar with any alternative SQL blockchain data sources.


text:
How download whole blockchain and preprocess it for analysis?
Will be grateful for any information/starter points/tuts etc.
For ex. this question (answers don't contain any "real" answer, any url to "blockchain files", any suggestion where to find it and how to download).
Is it possible to download all blockchain raw data?
my question is how to download blockchain in format which is will be accesible for understanding, processing, information extraction, etc.
Is it possible?
p.s
If you push downvote button at least explain why, please

Does this answer your question? Is it possible to download all blockchain raw data?, there are also potentially useful answers to similar questions which point to various repositories

@Prayank Thank you for knots looks like ideal wallet for me :) but actually I looking something more simple to understand I want to analyse dynamic of transactions if I understand correctly blockchain contains all id's and all transactions so in theory you can use it info to understand "something" about transactions (ex. how often on id make transaction per day, etc)

@RedGrittyBrick Thank you for your comment. But your first url doesn't contain any url to blockchain. Ideally I want blockchain as document (not inside of a client) or something like that and simple way to open it and see blocks. I tried to find something similar in google and here but actually I didn't find any simple for my understanding how extract blockchain via some readable file :(

@tempdeltavalue: Don't think this should be downvoted but there is a replicate question and if the answer to that question isn't comprehensive enough the answer should be improved on that question or an additional answer should be added. Welcome to Bitcoin StackExchange though :)

I think this is the wrong place where to ask, but I know that also it is difficult to find a place where ask this question (maybe).
I would add an answer to this question because I made research in my bachelor's degree, where we analyze the bitcoin graph, but at the same time, I developed a way to make also a useful serialization of the blockchain with metadata for future research. The solution is called SpyCBlock and a demo that works on the JSON data is described in the readme (link).
However, you need to download the bitcoin blockchain with bitcoin core and make the parsing of it with SpyCBlock, and maybe make this serialization available for the world?
It is the only software that I know that produces a serialization in JSON of the blockchain, but maybe there is something new around?

Yeap, you are right (Thank you for your answer). I'm also found some python solution
https://github.com/citp/BlockSci

well it depends from what you are using, blocksci use a big machine to make the analysis, SpyCBlock use only a normal computer :) it ran also on Raspberry PI

yes, but for now I'm just downloading blockchain it takes some time :) Probably after that I'll try blocksci because I'm more familiar with python. But anyway will see, for now I'm not sure that exactly I want to :)

We just published a paper that deals exactly with this question, I think. It presents a method and scripts to download the whole data and add indexes that make it easy to use. It also provides the dataset obtained on March 2021, that contains all data available in the blockchain at that time. Please see http://bitcoin.complexnetworks.fr/

I'm windows user so I cannot really test your thing (I heard about VirtualBox). Also you paper it's really long... But in general I "don't want" to work with parsed dataset, also I'm not sure if you have access to id's of wallets... In other world can your tool answer on question "How much traders we have in blockchain (some date) and how much transactions they do?" , if yes can share some specific place in your doc? (sorry to bother you)


text:
Baffled by endianness: How do I compute the HASH160 of the public key of a base58 address?
Given an address as follows:
> spend_address=$(btc-cli getnewaddress '' legacy)
> btc-cli getaddressinfo $spend_address

spend_address info {
  "address": "moHaAMoTLRZ7dCyP4SunTBRvSfdFxUNrCe",
  "scriptPubKey": "76a91455391f93c7409df890be2804aa3cf85b22a903c188ac",
  "isscript": false,
  "iswitness": false,
  "pubkey": "03a6a9b25d8cd811af3162222d5e6ddaf876f915ae37ccb4b4b3f2ba56c39e26d2",
  "iscompressed": true,
  ...<stuff not needed for example elided>...
}

So we have pubkey = 03a6a9b25d8cd811af3162222d5e6ddaf876f915ae37ccb4b4b3f2ba56c39e26d2.
And if we decode the ScriptPubKey we find the HASH160 embedded in it: 55391f93c7409df890be2804aa3cf85b22a903c1
Problem is I can't get from pubkey to HASH160(pubkey).  Somewhere I'm baffled by endianness. (I determined that by looking at intermediate results in the wallet's getaddressinfo handler.)
I've tried:
(ripemd160 (sha256 spend_address_pubkey))
(ripemd160 (sha256 (reversebytes spend_address_pubkey)))
(ripemd160 (reversebytes (sha256 spend_address_pubkey)))
(reversebytes (ripemd160 (sha256 spend_address_pubkey)))
(ripemd160 (reversebytes (sha256 (reversebytes spend_address_pubkey))))

and none of those (or their byte reverses!) match the transactions HASH160(pubkey) in the vout[0].scriptPubKey.asm field. Help me find which end to work from please!
(I do know how to get the HASH160 straight from the address by stripping off the checksum.  But I want to know how to get it from the pubkey, given the standard ways in which keys and hashes are displayed w.r.t. endianness.)

@Michael Folkson I had reverted your #python tag earlier because while my answer uses Python, that's not what the question is about.

@PieterWuille: Ok fair enough, I'll remove it. I thought it would be nice to gather Python code answers under Python tag but you're right, the tag is for the question rather than the answer.

There should be no endianness issues here at all.
Bitcoin's handing of endianness is sometimes weird in its human interfaces (e.g. transaction and block hashes are printed in reverse order, because Bitcoin interprets them as little-endian numbers, which then get printed for human consumption in big-endian order). No such issues exist at the binary protocol level, or inside the script interpreter, though.
I compute the Hash160 of the public key you posted correctly, with this Python code (it needs the ripemd160 module, since recent version of Python do not ship with it built-in anymore):
import hashlib
import ripemd160

pubkey = bytes.fromhex("03a6a9b25d8cd811af3162222d5e6ddaf876f915ae37ccb4b4b3f2ba56c39e26d2")
pkhash = ripemd160.ripemd160(hashlib.sha256(pubkey).digest())
print("%s" % pkhash.hex())

Output:
$ python3 testhash.py
55391f93c7409df890be2804aa3cf85b22a903c1

I obviously can't tell you what you're doing wrong without seeing the actual code you're using, but it must not be exactly the same as what this code does.

Thanks Peter, i will look again.

Found it elsewhere in my script thanks for the confirmation I was doing something wrong (seriously, that's what I needed to know!)


text:
Why is OP_SUBSTR disabled? (and OP_LEFT, OP_RIGHT)
Why are the substring opcodes disabled?
I understand that OP_CAT is disabled because it can blow up the stack easily (in combination with OP_DUP).
But the substring opcodes reduce the stack size.
Context:

I'd like to use OP_LEFT to implement a proof of work requirement to
  spend a transaction. A substring operation is needed because OP_ADD
  (for salting) and OP_GREATERTHAN only work on 32bit values.

I'm afraid that my answer here can't be better than "Ask Satoshi".
There was a series of bugs found in Bitcoin's early life, and the response by its creator was to disable a number of scripting opcodes. Some of these may have been bad choices (or very wise ones...), but we don't know, and it does not matter. For all intents and purposes, these opcodes do not exist, as "reenabling" them would be just as hard as introducing them as new operations in the first place (i.e. a hard fork).

So do you agree with my analysis, or can you see potential issues with these substring operations?
Thanks for taking the time to answer my question!

I do not see any issues with those operations, no.

Looking at it from a software-development standpoint, string-manipulation operations (OP_CAT, OP_SUBSTR, OP_LEFT, and OP_RIGHT) are prone to implementation bugs with the potential for buffer overflows or other security issues.


text:
Why outputs' spendable and solvable are false?
I noticed a transaction whose spendable and solvable are both false, what does this mean and why?
 {
    "txid": "c940241a0ef626e865e2004bc120536fb7868d510115e923c76d5e658a97f6d5",
    "vout": 0,
    "address": "mnbFFLXUqTPBweLm6w1GA2ZAJQbrTndEGC",
    "account": "",
    "scriptPubKey": "76a9144d98b2350032c849ac6e6e7ad0eb88e00fd7fbff88ac",
    "amount": 1.10000000,
    "confirmations": 132,
    "spendable": false,
    "solvable": false
},

Is this means this utxo is invalid somehow so cannot spend it?
Also, this is on testnet.

Could you tell us how you produced this output? That doesn't look like a transaction to me, but rather like a transaction output with some added data.

sorry my bad word, i meant utxo, not entire transaction. I used bitcoin-cli.exe -conf="${pwd}\bitcoin-testnet.conf" -testnet listunspent to list outputs in my local testnet node.

i do not see anything strange with it. you can spend it with a private key of mnbFF...

The listunspent rpc call returns all unspent transaction outputs (UTXOs) tracked by this wallet. This includes the UTXOs that can be spent by this wallet, but also UTXOs that belong to watch-only addresses tracked by this wallet.
When spendable is false, this wallet does not have the private key to spend the UTXO. This is e.g. the case for watch-only addresses.
When solvable is false, the wallet does not know how to spend the UTXO, regardless of whether it has the corresponding private keys. It follows that any output that is solvable false will also be spendable false.

Spendable being false is implied by solvable being false. So perhaps he does have the private key, but not the necessary redeemscript?

Is there possible the case that spendable is true and solvable is false? Just out of curiosity.

"It is not possible for a script to be spendable but not solvable" - achow101 bitcoin PR review club March 9 2022 https://bitcoincore.reviews/19602

why wouldn't the wallet know how to spend an utxo?


text:
For running a lightning node and getting maximum exposure, should I forward port 8333 also (for bitcoin)? Or only 9735 (lightning)?
I've been wondering about this. From what I read, people only open 9735 to run a "successful" node.
To me it makes sense not to open bitcoins port, but I could also imagine that some scanners out there scan for bitcoin nodes and upon success check the same IP for an additional lightning node.

Maybe some lightning implementations scan for bitcoin nodes and then look for lightning capability on top?

I'm not aware of any Lightning implementations that do this. The Lightning P2P network is a separate network to the Bitcoin P2P network and although Lightning nodes are required to watch the Bitcoin blockchain for counterparty cheat attempts to enforce Lightning's security model this could be done through connecting to a block explorer (e.g. the Core Lightning plugin Sauron) rather than running a Bitcoin full node. It is not broadcast on the Lightning Network whether a Bitcoin full node is being run in parallel with that Lightning node or not and what the IP address of that Bitcoin full node is.

To me it makes sense not to open bitcoins port, but I could also imagine that some scanners out there scan for bitcoin nodes and upon success check the same IP for an additional lightning node.

If you are running a Bitcoin full node with your Lightning node (which would generally be recommended for trust minimization) then whether you open the Bitcoin port (8333) is a Bitcoin related question and not a Lightning one. Do you want to allow inbound connections on the Bitcoin P2P layer, potential peers to message your Bitcoin full node and to be discoverable on a full node indexing service like bitnodes.io?

No, I don't want to expose my bitcoin node to any other person to be honest. If it is to no advantage, I would keep my Bitcoin node in the onion network only and just open lightning for the clearnet. This answer is currently the best one, the other ones seem to be guessing. I will accept this answer and if no better one appears, you can have the bounty. Thanks!

I think this heavily depends on how you define exposure. However, I don't think the patterns you describe for scanning lightning nodes would make sense. Not only would it leave out nodes that are not exposing the 8333 port, but it also assumes that both nodes are being run on the same machine. It would be perfectly plausible to run them in different machines under the same network, or even as different containers under a dockerized setup.
So all in all I think exposing the lightning port should suffice

There's no need to open the bitcoin ports to run a "successful" lightning node (whatever successful means in that context). It arguably improves your security stance if you open up less ports to the outside world.
The key part is that you're accessible on port 9375 and some nodes expose on clearnet and onion to allow for more flexibility.


text:
Is it possible to check if an Address was generated from an HD Public Key.
Suppose I know a specific address and I know a full HD Public Key. Is it possible to determine if the address is a hash of one public keys generated from the HD public key in constant time, or in other words without checking every possibility?

No, it is impossible to know. You would have to check every possibility.

Thanks, that makes sense to me intuitively, but do you know of any way to prove it?

A strong hash function should have output that is indistinguishable from random data. Both SHA256 and Ripemd160 (tthe two hash functions involved here) are not known to be broken. There is no known way to find a relationship between the hash and the preimage. This means that you can't even figure out what the pubkey for an address is. Furthermore, deriving additional keys uses SHA512 (related to SHA256) which means that there is no way to find a relationship between the derived key and its parent except through brute force.

That makes sense. Would it be possible in Etherium given (at least as far as I understand it) that the address is just the truncated private key?

I don't use Ethereum or study it, so I don't know.


text:
Can I find the address of my Bitcoin in the debug.log file?
I cannot download the entire block chain (I’ve tried four times now and failed) so I can’t replace my encrypted old wallet.dat file into  the blockchain and do the rescan to check my coin balance. My question is, can I find my Bitcoin address in the debug.log file. Is there a way to use the information in this file to check my Bitcoin balance?

I don't think you'll find your address(es) in debug.log, but you can use the listaddressgroupings command without needing to download the blockchain. Open the debug window in bitcoin-qt and type "listaddressgroupings" (or use bitcoin-cli).
You can inspect the balance in each of the addresses using a standard blockchain explorer like https://mempool.space/.

Any addresses will be in the wallet.dat file and can be viewed or extracted without any need to download (synchronise with) the whole blockchain.
You can check the balance of an address by using several different public blockchain explorers and checking the results are consistent.
See also

dumpwallet output documentation & explanation?
What do the different .dat files contain?


text:
How can Bitcoin.org be hacked?
Suddenly:

The Bitcoin Foundation is giving back to the community!
We want to support our users who have helped us along the years.
Send Bitcoin to this address, and we will send double the amount in return!

https://bitcoin.org/
Obvious scam. The main Bitcoin website has been compromised.
I'm immediately removing every single instance of "bitcoin.org" in my code, in particular the scripts that check for new updates of Bitcoin Core and AUTO DOWNLOAD/INSTALL them after verifying the signatures which it also gets from the same domain...
Sorry, but this is like a cold shower. I never thought that this of all sites would get compromised. Now I cannot trust anything or anyone. If they can host such an obvious scam message, they can host modified EXEs and cryptographic signature files... Sigh.
How is this possible? What happened to allow for this?

You're verifying signatures, right?  Not just checking hashes like MD5 or SHA1?

If I tell you how bitcoin.org can be hacked, then you will be able to hack it so I'm not telling you ;p

That might be a joke?  Looks a little too obvious.  A more plausible scam attempt may've been something like a donation-request.

BTW, might be a bad idea to visit a site if it may've been hacked.

Having a script that would automatically download and install software from any website is obviously dangerous. A website is not magically hack-proof just because it's a .org.

It's very confusing that this question states that "bitcoin.org" is the "main bitcoin website".  It says right on the site:  Bitcoin.org is not Bitcoin's official website

FWIW, Stack Overflow were also hacked in 2019...

How is this possible? What happened to allow for this?

The same way that any website can be hacked. Bitcoin.org isn't hosted in some special way that makes it unhackable, it's a website, like every other website. As such, it can be hacked in all of those ways, including, but not limited to: web server compromise, DNS server/account compromise, registrar account compromise, etc.
It is probably that either the server hosting the website was compromised in some form (weak password, compromised SSH key, etc.) or the cloudflare account which is providing DDoS protection was compromised (weak password, support social engineered, etc.) and the settings changed to point to the attacker's server.

in particular the scripts that check for new updates of Bitcoin Core and AUTO DOWNLOAD/INSTALL

Bitcoin Core is no longer being uploaded to bitcoin.org anyways, at least not by the release maintainer. Bitcoin Core's official website is https://bitcoincore.org and you should be getting your binaries from there. AFAIK, the owner of bitcoin.org still mirrors the binaries on bitcoin.org, but it is no longer the official place for Bitcoin Core binaries.

them after verifying the signatures which it also gets from the same domain...

As long as the signatures verify and were created with keys that you trust (presumably you got those keys from somewhere else and did some verification of them), then the binaries downloaded should be fine. This is, after all, the point of having signatures. An attacker who has taken over the website will be unable to create a valid signature with the release key(s) unless they have also compromised those.
For 22.0+, you can also check that the binary's hashes match what the guix builders have built by checking the hashes and signatures in https://github.com/bitcoin-core/guix.sigs (with the new guix process, the signatures over the hashes actually comes from the guix builders too). For 0.21 and earlier, you can do the same check with the gitian signatures: https://github.com/bitcoin-core/gitian.sigs.

There are no websites that represent bitcoin, as bitcoin is decentralised.
bitcoin.org is owned by a person and is a site that gives information about bitcoin, as many other sites do, and there is no particular connection between the security of the bitcoin.org domain name and the security of bitcoin software clients.

Specifically to address in the OP: "The main Bitcoin website has been compromised." However https://bitcoincore.org/ apparently is the "reference implementation"?

bitcoincore.org is the website that gives information about the Bitcoin Core client. Bitcoin Core is the most used implementation. There are other bitcoin implementations. The design of the other implementations may be different, however the consensus rules they abide by are the same (to stay in consensus). Other implementations are not more or less important.  Personally, I think the term "reference implementation" that is suggesting that Bitcoin Core is the only "correct" client is an improper characterisation. All implementations that abide by the consensus rules are correct.

I'm immediately removing every single instance of "bitcoin.org" in my code, in particular the scripts that check for new updates of Bitcoin Core and AUTO DOWNLOAD/INSTALL them after verifying the signatures which it also gets from the same domain...

No version of Bitcoin Core has ever has any contact with bitcoin.org, no version checking, no manual or automatic updates. If you have made these tools yourself and decided to add it as a trusted source for anything in particular, that is your security risk to deal with and nobody else's.

If they can host such an obvious scam message, they can host modified EXEs and cryptographic signature files... Sigh.

Well no, the point of cryptographic signatures is expressly to prevent this situation. Someone replacing the binaries can not produce valid signatures for the keys used previously.

Sometimes it's as easy as calling the registrar or host and pretending to be the owner of the domain, almost like how attackers do SIM swaps. Hopefully most hosts and registrars are better at protecting against this type of social engineering attack these days as it has become more common.


text:
Why miners looking for zeros?
I am trying to understand how miners work.
Is it true that miners are looking for a number of zeros because if they needed to find the complete hash they will be able to see the content of the transaction?
Besides that, it is probably impossible to find a complete hash?
Please let me know if the question is unclear, thank you in advance :)

Why miners looking for zeros?

They aren't.

Is it true that miners are looking for a number of zeros ...

No.

... because if they needed to find the complete hash ...

It is nothing to do with partial or complete hashes. All hashes computed by miners are complete hashes of the hashed data.

... they will be able to see the content of the transaction?

No, miners don't need a hash in order to see the contents of the transactions.

Besides that, it is probably impossible to find a complete hash?

No. it is easy to compute a complete hash. A single modern hardware device can compute 100,000,000,000,000 complete hashes every second.
A hash is an unpredictable number between 0 and roughly 11579208923731619542357098500869123456789012345678901234567890123456789
Therefore most hashes will be very very big numbers and so it is difficult to find a set of data whose hash is in some small part of that range (e.g. within 0 to 21)

Miners are using a brute-force trial-and-error method of finding a set of data whose hash is numerically less than the current network target value.
For example, if the network target was 21, then a hash of 33 would be too big but a hash of 17 would be a winner. In Bitcoin the target numbers are currently much larger than 21 but the comparison is still numeric.
To find a set of data that hashes to a value smaller than the target they vary a part of the block data which is known as the nonce. For each variation they compute the hash and see if the hash is less than the target. There are other things the miners can change that also affect the value of the computed hash. Miners have to resort to changing these other things after they have tried every possible value of nonce.
The target is set by Bitcoin rules such that a hash less than the target will, on average, be found every ten minutes.
This is a lot of work. The arrangement of data whose hash is less than the target is proof of this work having been done. Hence the phrase proof-of-work.

A hash is just a number (often encoded in hex). If you take arbitrary input data and run it through a hash function, you'll get back the 'hash' of that data. In the case of SHA256, the hash will be a number somewhere in the range of 0 to 2^256. There is no way to guess what the output will be for a given input, other than just running the hash function to see what happens.
What bitcoin miners are doing is trying to create a block template that includes a block header that will hash to a number that is under some network-defined value (the 'target'). In practice, this means that the valid hashes they find will start with a bunch of zeros, since the goal is to find a hash output that is less than the target. Miners will already know the transactions that they included in the block are valid, finding a valid hash does not change their ability to see or validate transactions. A valid hash is simply the 'proof of work' to show that the miner spent some amount of energy to create the block.

Thank you both! Really appreciate it :) I don't know why but somehow I was assuming that based on the content (the transactions) in the blockchain there was already a hash and miners needed to guess this hash by increasing the nonce. If I understand it correctly it is the miner that creates the hash?

Transactions are hashed (hashes are used for various things in Bitcoin) but the miners don't change the transaction to try to get a particular hash like they do with the nonce in the block header. Transactions (and hence the transaction hash) don't get changed by the miner.

Miners are looking for hash with a number of leading zeros because if they do they can announce it and claim the coinbase (or mining reward). They can already see the content of every transaction that they have received through network gossip.

it is probably impossible to find a complete hash?

I'm not sure what you mean by "complete". If you mean a hash of all zeros that is answered here.


text:
Why does this not return TRUE?
scriptSig: <pubKey>
scriptPubKey: <pubKey> OP_EQUALVERIFY

The stack has <pubKey>, <pubKey> and then OP_EQUALVERIFY. So why, does it not return TRUE, even when the recent two elements in the stack are equal?

OP_EQUALVERIFY has two possible outcomes:

In case the last two items on the stack are different, it aborts script execution.
In case the last two items on the stack are equal, both are removed. Nothing is put back on the stack.

For a script execution to be treated as successful, it must end with a non-empty stack whose top element is not 0. In your case, the stack is empty after execution, and thus the script execution is treated as failed.
You probably want to use OP_EQUAL instead, which puts 0 or 1 on the stack (rather than aborting or doing nothing).

Andreas Antonopoulos discusses the VERIFY suffix here.

Anything that has VERIFY will only continue execution of the Script if the outcome of the conditional operator is TRUE in which case it will not push that TRUE back to the stack, it will simply continue execution.

This StackExchange post may also be of interest.
The code for OP_EQUALVERIFY in the Bitcoin Core codebase is here.

Thanks for the articles. It's now clear that VERIFY may end with TRUE, but never push it on the stack.


text:
Cannot import python-bitcoinlib into a simple script
Im writing a simple python script to connect to a local bitcoin node and list transactions.
I created a virtual environment with:
virtualenv bitcoinenv

And then activated:
source bitcoinenv/bin/activate

The I installed python-bitcoinlib with:
pip install python-bitcoinlib

I have a simple script:
import requests
import bitcoin.rpc

# Connect to the local Bitcoin node
connection = bitcoin.rpc.Proxy()

# Get a list of unspent transactions
unspent_transactions = connection.listunspent()

# Print the unspent transactions
print(unspent_transactions)

And I run this with:
python bitcoin.py

But I get the error:
import bitcoin.rpc
ModuleNotFoundError: No module named 'bitcoin.rpc'; 'bitcoin' is not a package

Heres more information about my environment. When I run:
which python

I get:
/Users/me/Bitcoin/exchange/bitcoinenv/bin/python

When I run:
echo $VIRTUAL_ENV

I get:
/Users/me/Bitcoin/exchange/bitcoinenv

When I go to /Users/me/Bitcoin/exchange/bitcoinenv/lib/python3.8/site-packages and list the contents I see:

When I run pip --version I get:
pip 22.3.1 from /Users/me/Bitcoin/exchange/bitcoinenv/lib/python3.8/site-packages/pip (python 3.8)

And when I run python --version I get:
Python 3.8.0

Has anyone any idea why I cant import the python-bitcoinlib library?


text:
How is the double-spend problem related to the Byzantine generals problem?
I’ve read that Bitcoin’s key innovation is solving the double-spend problem. I’ve also read that Bitcoin solves the Byzantine generals problem.
Are the double-spend and the Byzantine generals problem related somehow? If yes, how?

The Byzantine generals problem (BGP) refers to a situation where independent actors need to agree on a common strategy but can't know which actors or information they can trust.
In Bitcoin terms, participants need to verify that a transaction they receive is indeed valid and that the inputs of that transaction are not already spent by a different transaction. The relation between double-spend and BGP lies in the fact that there can be multiple valid parallel chains, and there is no central authority signing off on which is the valid chain. Without knowing which chain everyone else is using, it's impossible to agree on which outputs have already been spent and thus validate your transaction.
Bitcoin's approach to probabilistically solving this problem is to use Proof of Work, which makes it very expensive to produce multiple valid chains. With every additional block added to the chaintip, it becomes more expensive and thus less rewarding to double-spend a transaction.

You mention that PoW solves the BGP by helping us know which chain everyone is using. Then you add that without knowing such chain, it's impossible to agree on outputs spent (agreeing on this solves the double-spend problem). You lastly say that agreeing on outputs is necessary to validate a transaction.
But PoW is actually not needed to validate transactions. So I'm now confused on the order of events you outline.

@StatsScared how you describe it is not how I structured my answer, though. The order is important. Let me know what still does not make sense in the correct order and I'd be happy to revise where necessary. In short: in order to validate, we need to agree on a valid chain. With PoW it becomes very expensive to create multiple valid (longest) chains, so with each added block we get more confidence.

Are the double-spend and the Byzantines general problem related somehow? If yes, how?

They are the same problem. If you can solve one, you can solve the other.
Imagine you had a solution to the double spend problem. You could then solve the BGP as follows:

One general submits a first transaction to the system that can solve the double spend problem.
The general waits the time the system needs to solve the problem.
The general submits a second transaction to the system that can solve the double spend problem that is a double spend of the first transaction.
If the first transaction wins, the generals attack. If the second transaction wins, the generals do not attack.

If the system solves the double spend problem such that one and only one transaction confirms, then either both generals will attack or neither will attack, so the system can also solve the BGP.

Bitcoin does not solve the Byzantines General problem. Point.
Bitcoin purchases the temporary solution )))

This is a technically correctly but completely useless answer.


text:
What is a safe amount of characters in a passphrase?
The coldcard and trezor both has support for a passphrase. What amount of characters in this passphrase that would be considered unbreakable with current technology and within some years of bruteforcing?

Depends. Best to use a passphrase pulled from wordlists, like EFF Diceware's or BIP39, so then six words.
But do not go over 49 characters.
You can get away with a string under 20 characters if the characters are truly random generated, upper/lower.
Using a password manager is a great way to go if you don't want to use physical dice or coins.
Whatever you do, do NOT create one from your own brain, easily crackable in a short time no matter what, must come from high entropy, randomly generated.

Dont use a password manager, they tend to get hacked and leak information.
Dont use a random generator, if you loose it, you will never be able to guess it.
Usually 20+ character is a good and safe lenght, add names or words you could remember, adding numbers or dates.
Having helped hundreds of people with lost passwords, my experience is if you use a too complicated password, your will have a hard time restoring it if you loose the note.
If the passphrase would be just the password to your wallet, then disregard above, but since its used with a 24 word seed then its just an additional layer that does need as much security as a normal password would.


text:
How do platforms agree or unwillingly reach a common bicoin price?
My question is about the bitcoin price in exchange of FIAT, not taking into account the slight difference of fees between exchange platforms (Binance, Coinbase....). As far as I know, they all propose the same Bitcoin price, which is today (after I googled Bitcoin Price): 44926.29 euros.
Since the Bitcoin price in FIAT is based on offer and demand only (am I right?), how do those platforms find the same course since they have a different vision of people wanting to buy and sell Bitcoin?
For example, it is my understanding that: if I sell on Binance 100 Bitcoin, the price of Binance should go down, but in the same time if Alice buy 100 Bitcoin on Coinbase, the price there should incrase.
So how eventually do those platforms find a common price?
EDIT:
The suggested related question partially answers but there are still some points to add:

Who performs the arbitrage?
How fast is it?
Does it work for all crypto (of course traders work on Bitcoin but do enough people work on other subjects)?

@RedGrittyBrick Thank you for this link I did not find in previous research. It partially answers with the arbitrage strategy, but my question is still there: who performs arbitrage? Especially on "public platforms" which are not for traders but for common people?

Common people can perform arbitrage, it's just buying on one exchange and separately selling on another.

@RedGrittyBrick OK and do platforms adjust by themselves this ? I took the example of the bitcoin but for all the "small" cryptocurrencies that have a low price, does it work for same?

The exchanges themselves have separate order books, and thus will generally not have the same price. However, if there is significant difference between different exchanges, money can be made by performing arbitrary, and thus someone will do it. It doesn't matter who does - as soon as some parties perform arbitrage, the prices will tend to synchronize. This happens for every cryptocurrency, and even for every asset in general: when trades happen in multiple places, traders will arbitrage between them. How fast this happens depends on the efficiency of the market.

It's important to realize it is not the platforms agreeing on a price. The price you see quoted as the current exchange rate is simply the latest price any traders on that exchange agreed to perform a trade at. The fact that multiple platforms report a similar price is simply a sign of the fact that traders (i.e., the market) agrees that that is the current price, and traders wouldn't want to sell at anything but the highest price they can get, and buy at the lowest price they can pay, a value that is largely independent of the exchange used. The traders are the communication mechanism.


text:
Bitcoin marketplace payment provider (send BTC from 1 address to multiple addresses with one transaction)
I want to build a marketplace, so there is a buyer and a seller. So when a buyer buys something from the seller I take the fee. Thus part of buyer's money goes to my account like (10%) and the rest to the seller.
Is there any payment provider in the crypto world that allows that, either through a payment provider or to code it by myself? Or if there isn't, what's possible solution for my problem?


text:
0 Active Connection on my-new-coin
I have compiled my-newcoin-qt and my-newcoind on different platforms (windows, Linux and Mac), with my coin specification based on litecoin.
When I run my-newcoin-qt at the first time, I get no error but the message:
0 Active Connections

What should I do as the next step? Should I compile seeder software on VPS server?

I bought a droplet on DigitalOcean running Ubuntu 12.04. It is only $5/month. I opened the port, compiled my *coind program on it, created my .conf file and ran it. Now my client coin programs connect to it.

thank you, is that possible to run *coind and software seeder together on one server? (e.g VPS Server)

unfortunately I am not sure about that. I am still setting my coin nodes up too. I don't see why not, however. You should be able to just run *coind in addition to the seeder software. Also, I am assuming you have seen this: https://github.com/sipa/bitcoin-seeder


text:
Coinbase Security
Does anybody know how cold is the Coinbase cold storage?
I'm not really into Coinbase but when I read about their cold storage I become a little bit skeptical: for example, how do they process withdrawals in a cold storage system?
And what does this mean?

Up to 97% of customer funds are stored offline.

Do they store X BTC in Y wallets, and then store private keys in a cold storage (shouldn't this approach make several transaction fees pop out when withdrawing)?

Well for one it doesn't really matter because your bitcoins are intermingled with everyone eleses in cold storage. In other words, your account does not "own" private keys. Rather everyone's bitcoins are swept into Coinbases' addresses. They keep track of who owns what using internal accounting methods.
When you send money from your Coinbase account, Coinbase first has to send money to some wallet address 'owned' by you. So they send bitcoins from their hot wallet to your coinbase wallet, then a second transaction is issued from your wallet (spending the outputs just received) to the final destination.
As to how they manage and replenish their hot wallets, I'm not sure, but perhaps they don't have to do this too often if more bitcoins are entering Coinbase than leaving.

If BTCs are swept into CB's addresses and then records are kept in a db, doesn't this raise bigger security issues? If you can found & hack the db, you can steal from it and they could potentially never know about the breach until they run out of funds.

Right, that might be true. I don't know what protections they have against this. I didn't mean to imply what they were doing was safe. I wonder though, what stops someone from doing the same thing to Bank of America? Hacking the db to increase the funds associated with an account. How are those records maintained anyway?

Most exchanges and web services are based on this same "shared web wallet" principle. Basically you are trusting the service provider and you are taking the risk if they fiddle with accounts or not.


text:
How can I find out which wallet belongs to which cryptocoin?
I have multiple decrypted wallets in the same folder. How can I find out which wallet belongs to which cryptocoin? All wallets are named wallet_x.dat, so the name gives no hint.
UPDATE:
I have no access to the addresses within the wallet, because I can't import them, because I don't know which client I would need.

my answer is very similar to nate's: use https://gist.github.com/nlitsme/b3102191c2dc14c587e7,  and look for 'name' in the output

@WillemHengeveld I'm not familiar with reading python code. Can you explain the theory behind this code?

@NateEldredge I would like to have an answer, that explains me where exactly I can find the indicator, not just which tool to use to find it.

what the python code does: read the berkeley db records, and print the records in readable form. And btw, you don't need to be able to read python in order to run it, just type:  python dumpwallet

an even simpler method would be to load wallet.dat in any hexeditor or some other tool to view binaries, and look for the string 'name', directly followed by something which looks like an address.  I think even the unix commandline tool 'strings'  would be sufficient.

@WillemHengeveld The script is definitely doing what I wanted. Thanks a lot. But I prefer to understand how it works, so I can rewrite it in a language that I understand, because I want to make it fit my needs.

Take a look at the wallet address, if it starts with 1 it is a Bitcoin wallet, if it starts with an L it is a litecoin wallet and so on, google to know how to identify the starting of the rest of the crypto wallets (blockchains are the first places to go).

This doesn't solve my problem. To read addresses from the wallet I would have to import it first. But I can't import it, because I don't know which client I need. Therefore I need to identify the type of coin it belongs to, without having access to the addresses within the wallet.

Then if there is no special app to identify them (which I'm not aware of) you can do two things, either you install all clients and try to open the wallet with each one until you manage to identify your wallets OR you find and read the documentation describing wallet format and you read them and identify them yourself, I don't think wallet format differ so much from one another since almost all crypto currencies are copies of the bitcoin code modified and adapted to some special needs or somthing. https://en.bitcoin.it/wiki/Wallet


text:
bitcoind fails to start at boot
I am trying to set up bitcoind to start at boot as a specific user. I see the process start for a minute and then fail, following boot.
I can start bitcoin as the user just fine with a login and then just run bitcoind. But I cannot pull the same trick off in a boot script.
journlctl -b

Jun 05 16:43:09 mo.trader.com systemd[1]: bitcoind.service: Start operation timed out. Terminating.
Jun 05 16:43:09 mo.trader.com bitcoind[546]: 2020-06-05T15:43:09Z Shutdown requested. Exiting.
Jun 05 16:43:09 mo.trader.com bitcoind[546]: 2020-06-05T15:43:09Z Shutdown: In progress...
Jun 05 16:43:09 mo.trader.com bitcoind[546]: 2020-06-05T15:43:09Z scheduler thread exit
Jun 05 16:43:09 mo.trader.com bitcoind[546]: 2020-06-05T15:43:09Z Shutdown: done
Jun 05 16:43:11 mo.trader.com systemd[1]: bitcoind.service: Control process exited, code=exited, status=1/FAILURE
Jun 05 16:43:11 mo.trader.com systemd[1]: bitcoind.service: Failed with result 'timeout'.
Jun 05 16:43:11 mo.trader.com systemd[1]: Failed to start bitcoin.

.
cat /etc/systemd/system/bitcoind.service
[Unit]
Description=bitcoin
#After=network.target
After=multi-user.target
Conflicts=rescue.target
AllowIsolate=yes

[Service]
Type=forking
User=bitcoinusr
Group=bitcoinusr

Environment=BITCOIN_PID=/mnt/bitcoin/bitcoin/bitcoind.pid
Environment=BITCOIN_HOME=/home/bitcoinusr/.bitcoin

ExecStart=/usr/local/bin/bitcoind
ExecStop=/bin/kill -15 $MAINPID

[Install]
#WantedBy=multi-user.target
Alias=default.target

.
cat ../bitcoinusr/.bitcoin/bitcoin.conf
blocksonly=1
dbcache=50
maxorphantx=10
maxmempool=100
datadir=/mnt/bitcoin/bitcoin

How do I get bitcoind to start at boot as user?
UPDATE:
It was a total systemd failure. it actually failed with 'service bitcoind start'. It just took a while to fail out.

It looks like you have the bitcoind service listed as a forking service, but have not configured bitcoind as such. forking means that systemd expects the service to daemonize itself, i.e. the command it executes will exit once the service has started. However you have not configured bitcoind to behave in this way. So the bitcoind command that systemd runs will not exit as systemd is expecting it to. It is named forking because the process will call the fork() syscall and then the parent process will exit.
There are two solutions.

Configure bitcoind to daemonize itself. You can do this by adding either daemon=1 to your bitcoin.conf file or adding -daemon to the command in ExecStart.
Change your bitcoind.service file to say that bitcoind isn't expected to daemonize itself. To do this, change Type=forking to Type=simple. This instructs systemd to expect that the process configured with ExecStart will be the main process of the service.


text:
I own some bitcoin, but forgot how and where i stored them. How can I find my lost bitcoin?
I purchased one bitcoin back on 2/8/14 and now don't recall having any kind of account information.  I actually forgot about the purchase until recently.  My email address has not changed since I purchased the one bitcoin back in 2014.  Not sure how to access that one bitcoin in the event I would ever want to sell it or just know how to get to it.  Thanks for any guidance you can provide.

Without any additional information, it is tough to provide help. Bitcoin is a decentralized network, so there is no central authority that can do something like authenticate your identity and provide access to your coins again. Understand that if you lost your ability to access the coins, they are essentially unrecoverable.
Owning a bitcoin is equivalent to owning the private key that controls that bitcoin, (or in some cases, trusting a custodian to control that key for you). So you'll need to figure out what software or service you used to generate a receiving address (and thus, your private key). It is possible you used an online exchange and left the coins on your account, or that you used some web-wallet (eg blockchain.info), or that you stored the coins locally (eg. a paper wallet). I believe at the time of your purchase, there were very few options available for hardware or mobile wallets, but it may be worth considering these options as well.
I would try searching your hard drives, email inbox, financial history (bank / credit card records for that time), social media, browser history, notes, or anywhere else you may expect to find a trace of the transaction. Use keywords such as "bitcoin, address, blockchain, wallet, wallet.dat, exchange, btc, bit coin, etc", basically anything you can think of that may turn up a clue.
You can also search your physical space, it is possible to store bitcoin using a 'paper wallet' (a bitcoin keypair printed onto paper). You could also have a string of words written down (12, or 24 most likely), called a mnemonic seed phrase (this tech would have just been coming out at the time of your purchase iirc). If you find either of these things, DO NOT share the keypair / words with anyone! If you do, they'll have full access to your bitcoin, and can easily rob you.
Best of luck!

multibit hardware wallet was so popular at the time of purchase.


text:
Error installing on Mac - Build does not support Berkeley DB database format
Im following the guide here (https://github.com/bitcoin/bitcoin/blob/master/doc/build-osx.md) to install Bitcoin on my Mac. I want to be able to run bitcoind with and without the GUI.
The first time I went though this guide, I DIDNT install berkeley-db@4, and used the configuration:
./autogen.sh
./configure --without-bdb --with-gui=yes

However, when i run make and then ./src/bitcoind it fails and I get the message:
2022-11-27T16:20:32Z Bitcoin Core version v24.99.0-9c47eb450346 (release build)
2022-11-27T16:20:32Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
2022-11-27T16:20:32Z Using RdSeed as an additional entropy source
2022-11-27T16:20:32Z Using RdRand as an additional entropy source
2022-11-27T16:20:32Z Default data directory /Users/me/Library/Application Support/Bitcoin
2022-11-27T16:20:32Z Using data directory /Users/me/Library/Application Support/Bitcoin
2022-11-27T16:20:32Z Config file: /Users/me/Library/Application Support/Bitcoin/bitcoin.conf (not found, skipping)
2022-11-27T16:20:32Z Setting file arg: prune = "1907"
2022-11-27T16:20:32Z Setting file arg: wallet = ["My Bitcoin Core Wallet"]
2022-11-27T16:20:32Z Using at most 125 automatic connections (285 file descriptors available)
2022-11-27T16:20:32Z Using 16 MiB out of 16 MiB requested for signature cache, able to store 524288 elements
2022-11-27T16:20:32Z Using 16 MiB out of 16 MiB requested for script execution cache, able to store 524288 elements
2022-11-27T16:20:32Z Script verification uses 11 additional threads
2022-11-27T16:20:32Z scheduler thread start
2022-11-27T16:20:32Z Binding RPC on address ::1 port 8332
2022-11-27T16:20:32Z Binding RPC on address 127.0.0.1 port 8332
2022-11-27T16:20:32Z [http] creating work queue of depth 16
2022-11-27T16:20:32Z Using random cookie authentication.
2022-11-27T16:20:32Z Generated RPC authentication cookie /Users/me/Library/Application Support/Bitcoin/.cookie
2022-11-27T16:20:32Z [http] starting 4 worker threads
2022-11-27T16:20:32Z Using wallet directory /Users/me/Library/Application Support/Bitcoin/wallets
2022-11-27T16:20:32Z init message: Verifying wallet(s)…
2022-11-27T16:20:32Z Error: Failed to open database path '/Users/me/Library/Application Support/Bitcoin/wallets/My Bitcoin Core Wallet'. Build does not support Berkeley DB database format.
Error: Failed to open database path '/Users/me/Library/Application Support/Bitcoin/wallets/My Bitcoin Core Wallet'. Build does not support Berkeley DB database format.
2022-11-27T16:20:32Z Shutdown: In progress...
2022-11-27T16:20:32Z scheduler thread exit
2022-11-27T16:20:32Z Shutdown: done

The error being Error: Failed to open database path '/Users/me/Library/Application Support/Bitcoin/wallets/My Bitcoin Core Wallet'. Build does not support Berkeley DB database format
So I went back and installed berkley with brew install berkeley-db@4. Then I again ran:
./autogen.sh
./configure --without-bdb --with-gui=yes

and then:
make

But when I run ./src/bitcoind I STILL get the same error above. What am I doing wrong?

Did you set up "My Bitcoin Core Wallet" as a BerkeleyDB wallet a while ago and now your Bitcoin Core is trying to access that BerkeleyDB wallet but BerkeleyDB isn't configured? Try installing without the --without-bdb flag?

Turns out thats exactly what happened, leaving out  --without-bdb fixed it. Thanks!

This was resolved by installing Bitcoin Core without the --without-bdb flag. The Bitcoin Core wallet used BerkeleyDB in the past (if you set up a wallet today it will use SQLite by default) so if you set up a Bitcoin Core wallet in the past then Bitcoin Core will try to access that wallet but it can't if BerkeleyDB wasn't configured.
To avoid this problem in future you can migrate BerkeleyDB, non-descriptor Bitcoin Core wallets to SQLite, descriptor Bitcoin Core wallets using the migratewallet RPC.


text:
How is coinswap different from submarine swap?
What is the difference between a coinswap and a submarine swap?

Bitcoin Optech defines a coinswap as:

Coinswap is a protocol that allows two or more users to create a set
of transactions that look like independent payments but which actually
swap their coins with each other, optionally making a payment in the
process.

The most recent example of a coinswap implementation is Chris Belcher's Teleport. The important detail is it is swapping coins or UTXOs on the same blockchain to gain a privacy benefit.
Lightning Labs defines a submarine swap as:

A submarine swap is a trade between on-chain and off-chain digital assets (i.e.between Bitcoin held on-chain and Bitcoin on the Lightning network).

(I believe Alex Bosworth of Lightning Labs coined the term.)
Of course terminology is always a bit uncertain in emerging fields as formal definitions haven't been universally agreed upon. There is a wide overlap in terms of the actual technology (adaptor signatures, timelocks etc) used to trustlessly swap coins on the same blockchain, to swap between onchain and Lightning and to swap between different blockchains (generally referred to as an atomic swap).


text:
Why isn't the backed up sqlite3 exactly same as the lightningd.sqlite3?
In the backing up page of C-Lightning, it says:
The main and backup files will not be identical at every byte, but they will still contain the same data.

Is there any specific reason this happens? Wouldn't a backup work if I copied the exact same lightningd.sqlite3 elsewhere myself?

The replication using the --wallet=sqlite3:///file1.db:/file2.db works by applying the SQL statements against both databases in sequence. The main replica gets is executed first, followed by the identical statement being executed on the second database. So the first difference in the file is that any internal timestamps will differ slightly between the two copies.
Secondly, while we execute the exact same queries on both replicas, the secondary will not be read, i.e., when selecting rows from the database, only the first row is actually read from disk. This may cause on-disk caches and self-optimizing structures in the DB to differ, since the access patterns differ between the two. Despite this we can be sure that the two replicas would match since we implement an optimistic locking mechanism and versioning scheme using the data_version in the vars table. These will match as long as the DB has not been modified manually and both DBs have seen the same statements.
Furthermore sqlite3 may implement some FS-specific logic to ensure optimal performance, so if you have the main and secondary replicas on separate media (highly suggested as that is what gives you resilience), they might have different filesystems, causing different optimizations to be used.


text:
Can't get the bip32masterkey from mnemonic seed (Python)
I'm trying to understandand implement the functioning of Hierarchical deterministic wallets.
But I can't get the bip32masterkey
based on this tutorial : https://medium.com/mycrypto/the-journey-from-mnemonic-phrase-to-address-6c5e86e11e14
this mnemonic seed : 0x77cdf1d92225adc0e67b1b4f5a31820251d518b3af074df25a07b751947fd07ebd29a4d0e57b84ea9de03a9123e2a6ea1e3ed739d4c562efec21f1bb0a54a879
should give me this bip32masterkey : 0x300b155f751964276c0536230bd9b16fe7a86533c3cbaa7575e8d0431dbedf23f9945bb8b052bd0b0802c10c7c852e7765b69b61ce7233d9fe5a35ab108ca3b6
but instead I have (pbkdf2) : 0x11df5cd2d2095dde0a46a8de0c2aba91a565e16dd0e5877d21285e605659d587bd3b3c72cba372ecf3b26fdf27151d9dc5d9f341760784cb04540f95d193c383
and this (hmac-sha512):
0xc661464729610902524d0919cc3000af9bfbd4fa003e78979b6026bb24d150ae99c5f70573e3eaf1b04ab3c735adc7e410bb65557e1f35cdf29f87825f9699c5
here's the code i'm using :
def get_bip32masterkey(seed):
    master_key = pbkdf2_hmac("SHA512", bytes(binascii.a2b_hex(seed)), bytes("Bitcoin seed".encode()), 1).hex()
    return (master_key)

I also tried with hmac instead of pbkdf2 but I cant get the right bip32masterkey, does someone see the point i'm missing ?
that's the code I'm using with hmac :
def bip32masterkey(seed):
    seed = binascii.a2b_hex(seed)
    masterkey = hmac.new(key=bytes("Bitcoin seed".encode()), msg=seed, digestmod=sha512).hexdigest()
    return (masterkey)

(I'm not a python dev I'm just trying to understand the basis)
Thank you in advance to all for your answers
EDIT : It looks like the example I was using provide false information and my hmac implementation is working.

It looks like the example I was using provide false informations and my hmac implementation is working.
Edit so maybe it's a bit clearer :
I checked it by pasting my BIP39 seed here :
https://iancoleman.io/bip39/
and decoding the base58 with this site :
https://appdevtools.com/base58-encoder-decoder
and checked the decoded result with the return of my functions to see if it match
result of hmac matched

So provide the solution for others to learn from, and an account of where you had gone wring

The answer is given in the statement
I'll synthetize it for you :
I tought the BIP32 master_key for this BIP39 seed : 0x77cdf1d92225adc0e67b1b4f5a31820251d518b3af074df25a07b751947fd07ebd29a4d0e57b84ea9de03a9123e2a6ea1e3ed739d4c562efec21f1bb0a54a879
should be :
0x300b155f751964276c0536230bd9b16fe7a86533c3cbaa7575e8d0431dbedf23f9945bb8b052bd0b0802c10c7c852e7765b69b61ce7233d9fe5a35ab108ca3b6
but the BIP32 master_key for this example is :
0xc661464729610902524d0919cc3000af9bfbd4fa003e78979b6026bb24d150ae99c5f70573e3eaf1b04ab3c735adc7e410bb65557e1f35cdf29f87825f9699c5
hmac was correct

Look at the recommended FOSS implementations in the specs and tutorials. Reread the specs docs of course, parse closely.
Not just Python, even CLI software like BX and bash tools, scripts etc; get a clear understanding as a user before trying to code.

I've read a lot of documentation , the official bip32 repo, learnmeabitcoin, read books and so on...
I think I have some good understanding of the functionning of HD Wallets
I'm not asking how it works , I'm asking why this function doesnt work and even hmac doesnt. Is my use of binascii.a2b_hex or pbkdf2 flawed ?
You can still forward me some documentation if you think I missed something.
Thanks


text:
Bitcoin transactions/Full nodes/SPV nodes
This is a bit long - please bare with me, i want to understand exactly what a full node and SPV does in verifying transactions and some other things:

Transaction start, A wants to send B, 1 BTC of value.
Lets assume A's wallet controls 2 UTXOs which reference 1.1 of BTC of value in total (0.5 BTC & 0.6 BTC). At this point in time, pre any transaction from A to B, I assume full nodes will have recorded that these UTXOs are indeed unspent so far (?)
A goes ahead and creates this transaction:
the inputs to the transaction will refer to the transaction which contains the first 0.5 UTXO as an output with a reference number and a signature (to indeed unlock and spend this UTXO which has been locked to A's public key hash), same for the second UTXO we want to spend (?)
The outputs are just one to B (ignoring the 0.1 change back to A), which has the satoshi value being moved, 100m Sats in this case and also a script lock to B's public key hash (?)
The transaction is now sent on to the network.
Full nodes verify: the signature from A does indeed unlock the outputs he wants to spend. These outputs have not been double spent (i.e. they still exist in the set of UTXOs). Inputs >= Outputs (lets ignore tx fees) and some other cursory checks (i.e. size <=21m BTC etc) (?)
Once verified by a full Node (and accepted) they propagate the tx onwards to others they are connected to. They update their state of all UTXOs (removing those which are 'spent' and updating to the new ones created by this tx, which remain unspent).
The tx sits in a pool waiting to be picked up by a miner. Even if this tx in particular is not picked up by any miners for the next block it remains considered 'spent' (?)

Lets say it gets mined in block 1000, current block height is 1005.

Now moving to SPVs:

B wants to verify that the tx from A->B has indeed gone through.
B runs an SPV node and queries a full node to check this. The full node (for his address) returns him a merkle path and the blockheaders from 1000 to 1005 to verify this.
B, given the tx id, can indeed verify that given the Merkle Path & tx id he can recreate the Merkle Root (so the tx is in the block 1000) (?)
Are the remaining block headers just used for him to verify that given he has verified the Merkle Root in the block header of 1000, he can now hash that, to verify that the next block header (1001) does indeed contain the previous block hash (i.e. that of 1000) (?) to make sure that the block has been built upon and now has 5 confirmations?

In point 3 there will be 2 outputs: 1 BTC for B and 0.1 BTC for A (the change).

yep! sorry thats my typo.

i guess another question would be: before the tx is included in a block the full node probably doesn't drop it from its UTXO set, as it could be in the tx pool for some time before it is picked up. Does part of the full nodes verification of a transaction therefore rely on checking that the current UTXO set does include the UTXO being referenced and also has not already been consumed in the temp tx pool?