text:
What stops Ripple users from creating a pyramid scheme with IOU's?
What stops the Ripple users from acting like fractional reserve banks or creating a pyramid scheme of debt? If the practice of over leveraging debt is prevalent how could this effect the Ripple currency and possibly the network?

In most pyramid schemes, people are charged a "setup fee", then get rewarded every time they recruit somebody. How does this apply to Ripple?

I specifically meant people offering IOU's to one person who could then use that money to offer IOU's to others or Gateways massively over extending there IOU's.

Having an IOU from one person in no way changes your ability to offer IOUs to anyone else. So the basic premise doesn't make sense. When you decide to allow a particular account to hold a balance with you, you are electing to trust only that particular source. You can never be owed money by someone you didn't agree to allow you to owe money.
In the gateway model, you agree to allow a company to owe you money because you have an agreement with them that they will pay you any money owed whenever you ask them to. If they renege on that agreement, then they are violating their agreement with you and you have the normal legal recourse against them. But it doesn't affect anyone else unless you default on your own obligations.


text:
Creating An Altcoin: How to Lower or Increase Confirmation Times?
When Creating an Altcoin for fun, what variable/attribute do I change in main.cpp file to increase/lower the time it takes for confirmations to take place?
Also can the confirmation times be changed again after coin is re-compiled/re-built?
Thanks.

See the variable nTargetSpacing.  The difficulty adjustments will try to steer the average block time towards this value (in seconds).
You can't change this at runtime.  In fact, you can't easily change it at all once you've released your coin.  All clients on the network need to agree on this number, since it is used to calculate the required difficulty.  If you change it after release, then there will be a disagreement between old and new clients as to whether various blocks are valid, and your coin will have a hard fork.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
API or framework for handling transactions
I am developing a JSP web (based on Java) and I would like to let users send the altcoins to my servers wallet. I need something what would monitor if the coins (DOGE/BTC etc.) has arrived and just after that let the users do something.
Is there anything like this? I am new here and maybe it is broad but let me explain better if you need.
I mean something like this: http://bitcoinj.github.io/getting-started
but not only for BTC also for other coins.

Have you reviewed this document?
https://en.bitcoin.it/wiki/API_reference_%28JSON-RPC%29#Java
outlines how to communicate with bitcoin server directly, MOST altcoin use a replica of this interface to provide API access.  There are however some altcoins which use their own unique system.

for example you can subscribe to unconfirmed transaction on the blockchain websocket and monitor when your addresses appear in the flow.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
How to get senders address with bitcoinj (no duplicate)?
I am using bitcoinj (java implementation). I have read many answers here but none of them answers my question.
How to get senders address using bitcoinj?
I have transaction variable tx. I can get it's hash, I have used this as well: https://blockchain.info/tx/myTXhash and it showed me the transactions. The problem is not even that web is showing me the sender.
BUT: if I use that generated .wallet file and open it with multibit (bitcoin wallet) I can see there, under transactions, from who were the money sent. That means there is a way how to get senders address, but HOW?
EDIT: I was mistaken, multibit does not know from who the money came from, I was just confused because it showed me it was sent to the right address (but that was set up by me in args).
But is there any possibility to get the message somehow from client to know if it is him who sent the money?

Do you mean getting the Bitcoin address that sent the money or somehow getting a name?

Yes I mean Bitcoin address that sent the money. Or some other way how to check if the money came from that person. Consider it as a wallet which is receiving money constantly, but I need to check if it is coming from the ones who wants to access on a web.

For future reference:
In bitcoinJ (java version) the input/output addresses of transactions are located in TransactionInput and TransactionOutput objects.
For example: for Transaction tx:
List<TransactionInput> inputs = tx.getInputs();
List<TransactionOutput> outputs = tx.getOutputs();

for(TransactionOutput out : outputs){
    System.out.println(out.getAddressFromP2PKHScript(params));
    System.out.println(out.getAddressFromP2SH(params));
}

for(TransactionInput in : inputs){
    System.out.println(in.getFromAddress());
}

Keep in mind that some transactions might not have an input address.
Here is the github description of getFromAddress() method:

Convenience method that returns the from address of this input by parsing the scriptSig. The concept of a "from address" is not well defined in Bitcoin and you should not assume that senders of a transaction can actually receive coins on the same address they used to sign (e.g. this is not true for shared wallets).

You can see the addresses that sent and received money when looking at a transaction on most services. For example:
https://blockchain.info/tx/c6cb02e70442275ffb7b35cc8a9416323ff7e01b5378abc20ae7f5b6e7617884
The big green arrow shows the transfer from one address (1B898F2tcWHX1LZWWfD21FRzqWrtXLocXN) to the 2 others. Programmatically you could use an API that returns JSON with equivalent information:
https://api.blockcypher.com/v1/btc/main/txs/c6cb02e70442275ffb7b35cc8a9416323ff7e01b5378abc20ae7f5b6e7617884
However it isn't straightforward to extract this information when looking at a raw transaction as the addresses are embedded in the script.


text:
Possible To --enable-wallet After Syncronization With Blockahin?
I am running bitcoind from the command line (--without-gui) and my system apparently does not have the correct Berkley DB version (4.8+). Therefore, I chose to run the program without the wallet enabled (--disable-wallet) so that Berkley DB would not be required.
What I am now wondering however, is if I sync the entire blockchain (300GB+) and at some point want to --enable-wallet, will I be able to do so without any incompatibility issues?
Furthermore, how can I update to the correct version of Berkley DB (4.8+) so that I can --enable-wallet. I already ran the script $home/bitcoin/contrib/install_db4.sh which did not alleviate the error when running ./configure to test the system for compatibility. I still get the error that the current version is incompatible.

What I am now wondering however, is if I sync the entire blockchain (300GB+) and at some point want to --enable-wallet, will I be able to do so without any incompatibility issues?

Yes, there's no issue with this. You will need to rebuild with --enable-wallet (or use the release binaries which bundle BDB), but this will have no impact at all on the synchronization state. You can switch between having the wallet compiled in, and back with no concerns.

Furthermore, how can I update to the correct version of Berkley DB (4.8+) so that I can --enable-wallet. I already ran the script $home/bitcoin/contrib/install_db4.sh which did not alleviate the error when running ./configure to test the system for compatibility. I still get the error that the current version is incompatible.

Depending on your distribution, you almost certainly have a version which is too new (5+), and many distros don't bundle old versions of it anymore. Bitcoin Core uses a very old version for compatibility, as newer versions would produce a wallet that can no longer be opened in older versions. Eventually this will be removed as a dependency except for some importing tool, but for the moment we just have to deal with it, or use the static release binaries which are reproducably built.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why future Bitcoin Core release shouldn't be in python?
Python offers various advantages including simplified rules of coding and ease of readability. It offers OOP, cross platform compatibility and has numerous libraries that have been added over time. It can be understood why the original Bitcoin Core client was in C++ as python was not that popular as it is now.
Apart, from having to tear down the entire code, and re-write it in python and check for vulnerabilities, why aren't the core bitcoin developers thinking of migrating the entire reference client implementation language to python?

Apart, from having to tear down the entire code, and re-write it in python and check for vulnerabilities,

This is what is known as a self-answering question. The obstacles you list are not trivial.
There is a considerable cost, a great deal of effort that could instead be spent on functional improvements to existing code.
There is also the matter of losing core developers who are skilled in C++ but may not care to use Python
Also if you are going to switch languages, it would be sensible to evaluate a much larger range. You advocate Python, I like Go, others may push for Rust, Haskell, Java, C# or any of a huge number of other possibilities.
This isn't a can of worms anyone would open lightly.

True, the question was self-answering, but what I was looking for was more to what capabilities are being offered by c++ to Bitcoin Core that is difficult to replicate with other modern programming languages. Losing core developers who are skilled in C++ but may not care to use Python, and opening can of worms for other recommendations are a valid point.

C++ is a compiled language, so lives much "closer to the metal" than Python, which is an interpreted language. This means C++ can be more than 100x faster at doing the same job. As I'm sure you're aware, someone who can mine 100x faster has a significant advantage so even if Python had been the original language used, it would have changed to C++ (or a similar language) since.

"Not trivial" ... it would be impossible. The state of the art in engineering can't provide a reasonable level of guarantee that a piece of code in C++ and a piece of code in Python behave the same way in all circumstances (which is what is required to not introduce a fork when making this change).

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Does secp256k1 library provide API for generating uncompressed key?
Hi I am right now want to do a make a small tool to generate some test vector for ecdsa.
Openssl is quite good, for support secp256k1 bitcoin provides an excellent library.
However, it looks what I get from secp256k1 API only the compressed key.
I find some post about how to convert the compressed key (Qx) to uncompressed key (Qx,Qy). But I just wondering if I can directly get the uncompressed key?
Thanks!

Use secp256k1_ec_pubkey_parse and secp256k1_ec_pubkey_serialize:
secp256k1_context *ctx = secp256k1_context_create(SECP256K1_CONTEXT_NONE);
secp256k1_pubkey pubkey;

const unsigned char input[33] = {0x02,0x8c,0x28,0xa9,0x7b,0xf8,0x29,0x8b,0xc0,0xd2,0x3d,0x8c,0x74,0x94,0x52,0xa3,0x2e,0x69,0x4b,0x65,0xe3,0x0a,0x94,0x72,0xa3,0x95,0x4a,0xb3,0x0f,0xe5,0x32,0x4c,0xaa};

if (secp256k1_ec_pubkey_parse(ctx, &pubkey, input, 33) != 1) {
    puts("Public key not on curve");
    secp256k1_context_destroy(ctx);
    return 1;
}

unsigned char outpubkey[65];
size_t written_bytes = 65;
secp256k1_ec_pubkey_serialize(ctx, outpubkey, &written_bytes, &pubkey, SECP256K1_EC_UNCOMPRESSED);

printf("Uncompress public key: ");

for (int i = 0; i < 65; i++) {
    printf("%.2x", (unsigned int) outpubkey[i]);
}

puts("");

secp256k1_context_destroy(ctx);

thanks a lot for the suggestion. I find the similar way, firstly use the  secp256k1_ec_pubkey_create() to get the pub key, then using secp256k1_ec_pubkey_serialize to form the either compresss or uncompressed key.

by the way, for secp256k1 library, in linker stage, what is the -l flag to be used--I find someone metioned to use -lsecp256k1 but it does not work for me? I can compile the program by using some CMake but it is too heavy and what I need is just a small tool--so I want to make a small make file, then I meet warning as keygen.cpp:(.text+0x558): undefined reference to `secp256k1_ecdsa_signature_serialize_compact'

Did you run make install in the libsecp256k1 folder? @LeonMSH

oh yes, that solves my problem! thanks so much!!


text:
How does Bitcoin Core estimate the transaction fee?
I want to know how Bitcoin-Core estimates the transaction fee and what factors it takes into account to estimatesmartfee. What is the formula that it uses? Can you also refer me Bitcoin Core files where I can find it's code.

Your client [1] will make use of transactions it relays to build an estimation : by caching it (and the block height) when it first sees it. It will later be able to tell how much time (# of blocks) it took for it to be confirmed.
To build a reliable estimation, it will fill "buckets" (basically, a pack of transactions sharing a common characteristic) according to the feerate paid by each transaction it relays.
When comes the time to estimate the feerate for a transaction to be included in block before a specified number of blocks, your client will check the percentage of transactions confirmed in each bucket (so, with transactions paying that much on average) during this period of time. It will return the feerate of the first bucket with a success (confirmed transaction) rate higher to the threshold [2].
This probability threshold is set by the second argument to estimatesmartfee (CONSERVATIVE | ECONOMICAL).
To quote the documentation by Alex Morcos (who coded most of the fee estimator) :

/** \class CBlockPolicyEstimator
The BlockPolicyEstimator is used for estimating the feerate needed
for a transaction to be included in a block within a certain number of
blocks.
At a high level the algorithm works by grouping transactions into buckets
based on having similar feerates and then tracking how long it
takes transactions in the various buckets to be mined.  It operates under
the assumption that in general transactions of higher feerate will be
included in blocks before transactions of lower feerate.   So for
example if you wanted to know what feerate you should put on a transaction to
be included in a block within the next 5 blocks, you would start by looking
at the bucket with the highest feerate transactions and verifying that a
sufficiently high percentage of them were confirmed within 5 blocks and
then you would look at the next highest feerate bucket, and so on, stopping at
the last bucket to pass the test.
The average feerate of transactions in this
bucket will give you an indication of the lowest feerate you can put on a
transaction and still have a sufficiently high chance of being confirmed
within your desired 5 blocks.

[1]: Talking about each node, it goes without saying that all nodes on the network don't share the same estimation (as there is no consensus on the mempools).
[2]: this is simplified, the CONSERVATIVE and ECONOMICAL parameters have technically an influence across buckets (target / 2, target, target * 2) but it's conceptually the same logic.

Going further, this is the best (without using easily malleable data) we can do with the information we have at broadcast time but it's clearly not enough from a UX perspective as you'll never be able to predict the feerate.
For example if multiple megabytes of transactions are broadcast at the same time you broadcast yours, fee estimation won't help you.
This is why replace by fee.

Can you also refer me Bitcoin-Core files where I can find it's code.???

@HaseebSaeed edited with more information, links (including to the implementation), and some limits inherents to fee estimation.


text:
bip39 checksum python
Is there any Python package which will allow me to verify if the checksum of a bip39 mnemonic is valid?  For instance, I have a 24-word bip39 mnemonic, and I might have made a mistake for one of the words.  I want to see which words will result in a valid checksum.
I've seen Javascript versions, but nothing in Python

If this is being done with a significant amount of funds at stake (e.g. your life savings) be careful with what server you enter your mnemonic into. Ideally it should be a "airgapped computer that has no WiFi, Bluetooth, Ethernet or other networking capability"

Clone/download this code from Trezor (don't forget the word list).
Then check your word-list like this:
from mnemonic import Mnemonic

code = "word1 word2 [...] word23 word24"
mnemo = Mnemonic("english")
isValid = mnemo.check(code) # returns a boolean
print('Checksum is valid: ' + str(isValid))


text:
what is actually stored on a hardware wallet?
My understanding is that:

Your crypto is on the public ledger under a public key.
Your Hardware Wallet stores your private key (which is used to prove your ownership of the public key/wallet).
You create a password to protect your hardware wallet (something more easily remembered than a full private key hash).

What I don't understand, is that when creating a hardware wallet you're given a "recovery phrase" of 12 words you have to write down to recover your wallet if it gets lost.
How is this any different than just writing down your private key on the same piece of paper and foregoing the expensive wallet? If you have to write down what is essentially your private key anyway what exactly is the benefit/point of a hardware wallet?

The private key can be created from the recovery phrase.
The advantage of a recovery phrase is that people are less likely to mis-type familiar words than to mis-type a long meaningless mix of digits and letters.
None of this depends on the wallet being a hardware wallet. Some software wallets, which can be obtained at zero cost, also support the use of recovery phrases (AKA seed-phrases).
The advantage of a hardware wallet is that, properly used, it is immune to typical PC malware.

How is this any different than just writing down your private key on the same piece of paper and foregoing the expensive wallet?

With a hardware wallet you can sign transactions without the private key leaving the hardware wallet. To construct and broadcast transactions you need to create signatures using the private key.

If you have to write down what is essentially your private key anyway what exactly is the benefit/point of a hardware wallet?

You don't need to write it down. The hardware wallet is all you need to store a private key and sign transactions. However if you lose your hardware wallet or it gets corrupted you may be thankful you have a backup. There is no way to access or transfer your funds in the case that you lose your hardware wallet and you have no backup.

Your coins on the blockchain are not at a single address.
They are scattered around on dozens, over time on hundreds of addresses, seemingly randomly but in reality all derived from one Extended Master Private Key, aka the "BIP32 root"
which is in turn determined by the (numeric) Seed produced by BIP39
Which ultimately is derived from that mnemonic list + an optional passphrase of your choosing
Which together comprise your Seed Recovery Phrase.
With this SRP your wallet client derives ALL your key pairs and addresses, all Hierarchically Determined from that one Extended Master Private Key
you could have a dozen clients all set to the same master key, all will show the same info.
With the public master, they are Watch Only.
The private key is only required for spending.
All a hardware wallet (keychain) does is sign transactions, prevent others from seeing the private key.
You must keep the SRP safe, ideally engraved on stainless steel plates, the secure passphrase separate from the mnemonic
in multiple locations where no one will ever be able to see them.
All wallets can be lost or destroyed including hardware, and so long as you have the SRP - and no one else does - you can still access your assets stored on the blockchain.

If you want just one address, then writing down the private key for it works.
This was how Bitcoin worked in the beginning, wallets would generate 100 private keys for 100 addresses for you at a time.
But soon this caused problems, for example, your initial backup would not get the 101st address, as it was generated later.
The solution that stands til today is called deterministically hierarchical wallets. In plain English that means you have just one private key (Master private key) from which you can generate addresses endlessly, but in a standardized fashion. And not just Bitcoin keys, hardware wallets allow you to have keys/addresses on other chains too; while a single Master Key backs everything up.
That master key, you guessed it, is standardized to be displayed as 12 or 24 words, to make it easy on humans.


text:
Does my wallet.dat file contain all wallets?
I have backed up my wallet.dat file, but now I find myself wondering if bitcoin core stores all newly generated wallets in the same wallet.dat file. Of course this is after having accidentally deleted some files I shouldn’t have. Can I safely restart my computer and download bitcoin core again, and load the old wallet.dat file or should I have seperate wallet.dat files for every newly generated wallet?

Do you actually mean separate wallets, or just multiple addresses? A wallet.dat file contains one wallet, but all its addresses.

That is exactly what I am not so sure about - I never modified/replaced the wallet.dat file, but I clicked create new wallet in bitcoin core - is that wallet just a new address that is added to the same wallet.dat file?

If you click create a new wallet, it creates a new wallet, which will be in a different file.

Well Allright I hope I can recover them then, thanks for your answers :)

A wallet contains many addresses. An address is not the same thing as a wallet.
When you create a new address with Bitcoin Core, it is stored in the wallet that you are actively using at that time. This means it is stored in the wallet.dat file.


text:
I am trying to get past Binding RPC on address 1 port 8332 failed
My debug file shows:
2022-12-17T09:28:00Z Bitcoin Core version v24.0.1 (release build)
2022-12-17T09:28:00Z InitParameterInteraction: parameter interaction: -externalip set -> setting -discover=0
2022-12-17T09:28:00Z Using the 'x86_shani(1way,2way)' SHA256 implementation
2022-12-17T09:28:00Z Using RdSeed as an additional entropy source
2022-12-17T09:28:00Z Using RdRand as an additional entropy source
2022-12-17T09:28:00Z Default data directory /home/rick/.bitcoin
2022-12-17T09:28:00Z Using data directory /home/rick/.bitcoin
2022-12-17T09:28:00Z Config file: /home/rick/.bitcoin/bitcoin.conf
2022-12-17T09:28:00Z Config file arg: externalip="62676xoz5zt3jdgyjhombuluqf6ucwwoktfcpoc77g7my7yt6p4ua5qd.onion"
2022-12-17T09:28:00Z Config file arg: help-debug="1"
2022-12-17T09:28:00Z Config file arg: listenonion="1"
2022-12-17T09:28:00Z Config file arg: seednode="hhiv5pnxenvbf4am.onion"
2022-12-17T09:28:00Z Config file arg: torcontrol="127.0.01:9051"
2022-12-17T09:28:00Z Config file arg: torpassword=****
2022-12-17T09:28:00Z Using at most 125 automatic connections (1024 file descriptors available)
2022-12-17T09:28:00Z Using 16 MiB out of 16 MiB requested for signature cache, able to store 524288 elements
2022-12-17T09:28:00Z Using 16 MiB out of 16 MiB requested for script execution cache, able to store 524288 elements
2022-12-17T09:28:00Z Script verification uses 7 additional threads
2022-12-17T09:28:00Z scheduler thread start
2022-12-17T09:28:00Z Binding RPC on address ::1 port 8332 failed.
2022-12-17T09:28:00Z Binding RPC on address 127.0.0.1 port 8332 failed.
2022-12-17T09:28:00Z Unable to bind any endpoint for RPC server
2022-12-17T09:28:00Z Error: Unable to start HTTP server. See debug log for details.
Error: Unable to start HTTP server. See debug log for details.
2022-12-17T09:28:00Z Shutdown: In progress...
2022-12-17T09:28:00Z scheduler thread exit
2022-12-17T09:28:00Z Shutdown: done

My bitcoin.conf is:
 Specify your own public address
externalip=lotsofcharacters.onion
listenonion=1
seednode=hhiv5pnxenvbf4am.onion

torcontrol=127.0.01:9051

# Tor control port password (default: empty)
torpassword=16:LOTSOFCapitalletters

Hidden tor password & external ip for privacy.
(they are valid I have been using them on my lightning node)
PLease do you have any suggestions as to what I can do to get the BTC node back??

Can't tell if tor service is running
lightning_connectd: Connecting stream socket to Tor service: Connection refused
lightningd: connectd failed (exit status 1), exiting.

Whenever a service has failed to bind to a high numbered port I have always found it was because another application was already bound to that port. I have always discovered the culprit using netstat -anp  or similar.
N.b. High numbered is > 1023

lightning_connectd: Connecting stream socket to Tor service: Connection refused
lightningd: connectd failed (exit status 1), exiting.


text:
Details regarding bitcoin addr messages
What is the frequency of the addr messages in bitcoin?
And are the contents of the addr messages from the new or the tried table or both?

From the Bitcoin wiki:

The getaddr message sends a request to a node asking for information about known active peers to help with finding potential nodes in the network. The response to receiving this message is to transmit one or more addr messages with one or more peers from a database of known active peers. The typical presumption is that a node is likely to be active if it has been sending a message within the last three hours.

In the Bitcoin Core implementation AddrMan (Address Manager, .cpp file) manages a node's known addresses. As the correction below in the comments says the response to a getaddr request will return both tried and untried (new) addresses.

That's incorrect. Both tried and new addresses are sent (though as the other answer points out, this is implementation specific - the protocol has no notion of tried and new addresses).

I would have also thought superior logic would be to send tried addresses because your peer will disconnect from you if you send a bunch of untried addresses that turn out to be unresponsive. I guess it is a privacy, maintaining connections trade-off

No, we don't disconnect peers that give us unresponsible IP addresses.

What is the frequency of the addr messages in bitcoin?

I believe they are sent in response to a getaddr message. So the frequency at which your node receives addr messages depends on the frequency your node issues getaddr messages.

are the contents of the addr messages from the new or the tried table or both?

The network protocol specification doesn't specify the source in that way. It says
getaddr

The getaddr message sends a request to a node asking for information about known active peers to help with finding potential nodes in the network. The response to receiving this message is to transmit one or more addr messages with one or more peers from a database of known active peers. The typical presumption is that a node is likely to be active if it has been sending a message within the last three hours.

addr:

Provide information on known nodes of the network. Non-advertised nodes should be forgotten after typically 3 hours

To me this suggests each implementer chooses exactly how to source, categorise, store and maintain the data it uses in the response. If in doubt the implementer presumably consults the source code of whatever implementation they regard as the reference implementation.
I guess it would be prudent not to make any assumptions about the sending node's selection of entries in it's addr response. The sending node may be hostile or faulty.

I believe you mean getaddr? getdata based on my understanding is used for information regarding transactions

@Vinay: Thanks, I believe you are right. Answer updated accordingly.


text:
DNS Seeds for TORv3
Does querying the DNS Seeds via TOR return .onion addresses? If not, then if my node is a TOR only node, does it always get the .onion addresses to connect to, from the hard-coded list?
Additionally, are the DNS queries encrypted or in plain-text for IP nodes?

Does querying the DNS Seeds via TOR return .onion addresses?

Sometimes. Before the Bitcoin Core PR (#25678) you would expect to fall back to relying on the .onion addresses in the hardcoded seeds though you may get onion addresses if you are lucky to connect to a peer that happens to offer these.
As lightlike says in the comment:

For Tor Bitcoin Core currently first does an AddrFetch connection (a short-lived connection with the goal of getting addresses) via Tor exit nodes to an address each DNS seed resolves to, violating the -onlynet option. This may result in some onion addresses, but only if we are lucky to connect to a peer that happens to offer these. #25678 suggests to change that behavior to skip the DNS seed step and immediately resort to hardcoded peers

With regards to the second question:

Additionally, are the DNS queries encrypted or in plain-text for IP nodes?

Messages between nodes are currently not encrypted but there is ongoing work to change this in future.

See the OP of #25678, for Tor Bitcoin Core currently first does an AddrFetch connection (a short-lived connection with the goal of getting addresses) via Tor exit nodes to an address each DNS seed resolves to, violating the -onlynet option. This may result in some onion addresses, but only if we are lucky to connect to a peer that happens to offer these.
#25678 suggests to change that behavior to skip the DNS seed step and immediately resort to hardcoded peers.

@Lightlike: Thanks for the correction and the additional detail. Feel free to post your  own alternative answer in future in cases like this. Your answer is much better than mine :)

"Messages between nodes are currently not encrypted but there is ongoing work to change this in future."
I'm not very familiar with exactly how DNS is used in Core but afaik these queries are not between nodes, but rather between a node and a DNS server so I don't think BIP324 applies?


text:
Sending addr messages to nodes
I came across this resource that can be potentially used to send messages to nodes.
How would I go about sending addr messages to nodes?
Is there a built in utility that I could use(bitcoin-cli) or do I have to resort writing a script?

Not sure I understand the question. If you run a full node (e.g. Bitcoin Core) you will send addr messages to nodes by default and help them find other peers (part of the P2P protocol). What exactly do you want to do beyond this? You want to send them specific addr messages at the time of your choosing or you want to experiment with your own script?

I am trying to experiment with my own script to actually see how the buckets are populated with and without the use of an asmap.

Ah ok, cool. Afaik there isn't any built in utility. You'll have to write a script to send additional addr messages that Core wouldn't send.


text:
Reproducible Gitian Builds .. but not the same hash as bitcoincore.org
Bitcoincore.org and Bitcoin.org will let you download a tarball for bitcoin v0.20.1
When they do it
bitcoind:
4ec74161b2a90293926ae8e20a2efbe952bd23b53aeebf051e6a6285ace18271
bitcoin-0.20.1-x86_64-linux-gnu.tar.gz:
376194f06596ecfa40331167c39bc70c355f960280bd2a645fdbf18f66527397
They also say:

reproducing a binary for yourself will provide you with the highest
level of assurance currently available

So, it seems like the current method to do this is the "gitian build" maneuver.
When I do it
bitcoind:
c5fb850ed9e6afa0af5653e743084912c9bd71fb233acfe2d72738fc319f2181
bitcoin-0.20.1-x86_64-linux-gnu.tar.gz:
277599356bd2df760832c6636797fe5ea5a5c28d929d53635b685f5ac1e4689b
Went through the gitian process with Ubuntu 20.04 and Debian 10.
Both produced the same incorrect? tarball, 2 different setups/ 2 Operating Systems, same tarball.
Both my assert files are uploaded here for Debian and Ubuntu.
https://github.com/miketwenty1/images/tree/master/bad_asserts

As a pleb I can think 1 of 2 things..
Something is wrong/off with some of my packages / versions / env / OS / setup OR, All the verifiers are lying and bitcoin's been compromised. Help me narrow down the possibilities.
I would like to go through this exercise properly and reproduce the same binaries.. but unsure of the best way forward.. what are the next steps to verifying the build for v0.20.1? I can't seem to reproduce the tarball advertised on bitcoincore . org and bitcoin . org. I can however make a reproducible build which makes me question a lot of things.
Would love if someone could go through their process from scratch and see if they can produce the same tarballs seen in this repo by these people. https://github.com/bitcoin-core/gitian.sigs/blob/master/0.20.1-linux /*/ bitcoin-core-linux-0.20-build.assert's.
And share their complete process with me. After reproducing myself I would be happy to make a current walkthrough on the method and update the Docs, which (links below) are dated and broken for nearly all methods.
NOTE: If this is indeed the defacto way to currently do reproducible builds.. it would be nice to have documentation that is precise and current. I understand Ginx is something being worked on.. but would be nice to have a current doc in the interim for people who wants to verify builds. I would also be very satisfied if someone pointed something I was missing.
reference links:
https://gitian.org/
https://github.com/bitcoin-core/docs/blob/master/gitian-building.md
https://github.com/bitcoin-core/docs/tree/master/gitian-building
https://github.com/devrandom/gitian-builder/blob/master/README.md
https://github.com/bitcoin/bitcoin/blob/master/contrib/gitian-build.py
reference github issue
https://github.com/devrandom/gitian-builder/issues/235

FYI, reported the problem at https://github.com/bitcoin/bitcoin/issues/20389

I've done a rebuild of 0.20.1 and I get the same results that you do. This would indicate that a build dependency has updated to produce slightly different results than the version that was in use at the time of the release. The build dependency versions are pinned unlike the actual software dependencies. IIRC this is common for gitian builds and attempting to rebuild even older releases will result in similar mismatches.
There is ongoing work to move to a different reproducible builds system called guix. This build system will pin the exact dependency versions, along with the option to build the entire toolchain from scratch. This should enable fully reproducible builds that can be replicated at any point in time.

Here is a more in depth explanation of this particular mismatch, copied from my response in this GitHub issue.
If you look at the gitian build results, you will see a file named bitcoin-0.20.1-x86_64-linux-gnu-debug.tar.gz. If you untar this file, there will be *.dbg files, e.g. bitcoind.dbg and bitcoin-qt.dbg. These *.dbg files contain the debug data for their respective binaries, i.e. bitcoind.dbg contains debug data for bitcoind.
To ensure that you use the correct dbg file with the correct binary, gcc embeds a checksum of the dbg file within the binary itself. This means that bitcoind contains a checksum of bitcoind.dbg. This is to prevent attempting to debug bitcoind with another dbg file. For example, if you attempted to tell gdb that the bitcoin-qt.dbg file contained the debugging symbols for bitcoind, it would detect it does not and not attempt to load debugging symbols from bitcoin-qt.dbg.
The debugging symbols are initially compiled into the bitcoind binary, but later during the gitian build these are removed and placed into the bitcoind.dbg file. However because they are compiled into bitcoind initially, the debugging symbols have an effect on the build id that gcc embeds into the binary. The build ID is a hash of the compiled binary, including the debugging data.
So the end result is that the published binary contains two commitments to the debug symbols, but does not actually contain them itself.
You can read more about these separate debug files here: https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html
The crux of this issue is that the debug symbols from recent gitian builds differ from the debug symbols that were generated for the original release. This means that both the build ID and the debug symbol checksum that we find inside of bitcoind are different. This then results in the bitcoind hashes being different (as well as all of the other binaries). And of course that causes the tarfile hashes to be different.

Here is the diff of the binaries that diffoscope generates:
--- bitcoind
+++ /mnt/archive/bitcoin/bitcoin-binaries/0.20.1/bitcoin-0.20.1/bin/bitcoind
├── readelf --wide --notes {}
│ @@ -1,15 +1,15 @@
│
│  Displaying notes found in: .note.ABI-tag
│    Owner                Data size     Description
│    GNU                  0x00000010    NT_GNU_ABI_TAG (ABI version tag)        OS: Linux, ABI: 3.2.0
│
│  Displaying notes found in: .note.gnu.build-id
│    Owner                Data size     Description
│ -  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 6b464617f7f91fd270ac86f43ef4a58eeeedff19
│ +  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 3a439a31a5157ff7052ed310050df5643a02ea3f
│
│  Displaying notes found in: .note.stapsdt
│    Owner                Data size     Description
│    stapsdt              0x00000036    NT_STAPSDT (SystemTap probe descriptors)        Provider: libstdcxx
│      Name: throw
│      Location: 0x00000000006e550d, Base: 0x000000000086d140, Semaphore: 0x0000000000000000
│      Arguments: 8@%rdi 8@%rsi
├── readelf --wide --decompress --hex-dump=.gnu_debuglink {}
│ @@ -1,5 +1,5 @@
│
│  Hex dump of section '.gnu_debuglink':
│    0x00000000 62697463 6f696e64 2e646267 00000000 bitcoind.dbg....
│ -  0x00000010 b25ceebb                            .\..
│ +  0x00000010 114d519d                            .MQ.

As you can see, there are only two differences here, one in the build ID, and one in the .gnu_debuglink section. From the documentation I linked earlier, we can see that this .gnu_debuglink section has the first line is the debug filename followed by enough 0 bytes to pad to a 4 byte boundary. The second line is the 4 byte CRC checksum. And it is this CRC checksum that differs.
So why do the debug symbols differ here? Again, diffoscope can help us a bit.
--- bitcoind.dbg
+++ /mnt/archive/bitcoin/bitcoin-binaries/0.20.1/bitcoin-0.20.1/bin/bitcoind.dbg
├── readelf --wide --notes {}
│ @@ -1,15 +1,15 @@
│
│  Displaying notes found in: .note.ABI-tag
│    Owner                Data size     Description
│    GNU                  0x00000010    NT_GNU_ABI_TAG (ABI version tag)        OS: Linux, ABI: 3.2.0
│
│  Displaying notes found in: .note.gnu.build-id
│    Owner                Data size     Description
│ -  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 6b464617f7f91fd270ac86f43ef4a58eeeedff19
│ +  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 3a439a31a5157ff7052ed310050df5643a02ea3f
│
│  Displaying notes found in: .note.stapsdt
│    Owner                Data size     Description
│    stapsdt              0x00000036    NT_STAPSDT (SystemTap probe descriptors)        Provider: libstdcxx
│      Name: throw
│      Location: 0x00000000006e550d, Base: 0x000000000086d140, Semaphore: 0x0000000000000000
│      Arguments: 8@%rdi 8@%rsi
├── readelf --wide --debug-dump=info {}
│┄ error from `readelf --wide --debug-dump=info {}`:
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│ @@ -85052,36 +85052,36 @@
│      <29607>   DW_AT_decl_line   : 124
│      <29608>   DW_AT_decl_column : 16
│      <29609>   DW_AT_type        : <0x28761>
│      <2960d>   DW_AT_data_member_location: 12
│   <2><2960e>: Abbrev Number: 30 (DW_TAG_member)
│      <2960f>   DW_AT_name        : (indirect string, offset: 0x13aaf): __kind
│      <29613>   DW_AT_decl_file   : 108
│ -    <29614>   DW_AT_decl_line   : 148
│ +    <29614>   DW_AT_decl_line   : 128
│      <29615>   DW_AT_decl_column : 7
│      <29616>   DW_AT_type        : <0x287a6>
│      <2961a>   DW_AT_data_member_location: 16
│   <2><2961b>: Abbrev Number: 30 (DW_TAG_member)
│      <2961c>   DW_AT_name        : (indirect string, offset: 0x7a535): __spins
│      <29620>   DW_AT_decl_file   : 108
│ -    <29621>   DW_AT_decl_line   : 154
│ +    <29621>   DW_AT_decl_line   : 134
│      <29622>   DW_AT_decl_column : 3
│      <29623>   DW_AT_type        : <0x2879a>
│      <29627>   DW_AT_data_member_location: 20
...

There's a lot more output that I haven't included because it's pretty much all of the same.
Now this isn't terribly helpful, but we can see that for a bunch of functions, the line number for that function differs by 20 lines.
To get some more information, I used dwarfdump. This is what it says for the new build for the two functions I show in diffoscope (the functions are __kind and __spins).
0x0002960e:     DW_TAG_member
                  DW_AT_name    ("__kind")
                  DW_AT_decl_file       ("/usr/include/x86_64-linux-gnu/bits/thread-shared-types.h")
                  DW_AT_decl_line       (148)
                  DW_AT_decl_column     (0x07)
                  DW_AT_type    (0x000287a6 "int")
                  DW_AT_data_member_location    (0x10)

0x0002961b:     DW_TAG_member
                  DW_AT_name    ("__spins")
                  DW_AT_decl_file       ("/usr/include/x86_64-linux-gnu/bits/thread-shared-types.h")
                  DW_AT_decl_line       (154)
                  DW_AT_decl_column     (0x03)
                  DW_AT_type    (0x0002879a "short int")
                  DW_AT_data_member_location    (0x14)

As you can see by the given file name, these functions come from libraries installed to the system. These appear to be headers for gcc's implementation of the c++ stdlib.

So what's happened is that libstdc++ has updated in Ubuntu. Whatever updates hapened have moved some code in some header files that Bitcoin Core includes in its use of the c++ stdlib. In turn, compiling with those updated headers results in different debug symbols because function declarations have moved in those header files. This then results in gcc computing a different build ID and a different CRC checksum for the debug symbols. This lastly results in the final binaries being slightly different, which causes the hashes to mismatch.

I'm curious as to the choice of guix over Nix - do you have a link to any discussion on irc, github, etc. on why guix was chosen?

@RaghavSood From https://github.com/bitcoin/bitcoin/pull/15277: "Guix is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on bootstrappability and reproducibility which are attractive for security-sensitive projects like bitcoin." I don't think there has been much discussion about guix vs nix, just that guix is what someone has put the effort into doing, so it's what we get.

Thanks for the detailed info. If it's interesting I have cmp output here comparing both bitcoind files.
https://github.com/devrandom/gitian-builder/issues/235
Luke-jr let me know the files it looked like were in question were:
.note.gnu.build-id .gnu_debuglink
@AndrewChow you think this is a design flaw or working as intended?

It is a design flaw of the gitian system, but the gitian system is also working as intended.

As long as the underlying OS image is susceptible to changes, you will not get a reproducible build system that guarantees reproducibility over a long period of time. If the OS can update compiler toolchains, libc versions and whatever tools are used during a gitian build, changes in the future to the output are expected. Since this is the case since gitian is ubuntu based, and patches are still shipped to older ubuntu versions, this is to be expected after some time.


text:
Random generation of elliptic curve points: is there a preferred approach?
While most practical applications are concerned with the random generation of private keys, anyone who wishes to write unit-testing code on the secp256k1 elliptic curve group may wish to have the ability to randomly generate elliptic curve points.
I can immediately think to two approaches for doing so, one consisting in returning the elliptic point associated with a random secret, the other consisting in randomly generating X, computing X^3 + 7 until we obtain a quadratic residue mod p and compute a square root (which reduces to taking a power (p+1)/4 mod p since p = 3 mod 4). I am suspecting that the latter approach is computationally more efficient as it does not involve the scalar multiplication of the EC group, but only arithmetic operations and exponentiation mod p. Is there a known preferred approach to this question? Assuming my intuition is right, and generating X until X^3 + 7 is a quadratic residue is the way to go, another question naturally arises: what is the preferred approach to test whether a number mod p has a square root? Is it worth implementing the Legendre symbol, or is it better to simply check whether the prospective square root (raising to the power (p+1)/4) fits the bill?
Early optimization is the root of all evil, so maybe I shouldn't be asking this question, especially as the performance of unit testing code probably doesn't matter. But I figure that if some elements of best practice are known regarding bitcoin development, I may as well adopt them and do the right thing.


text:
Do you need to change 'rpcuser' and 'rpcpassword' for each bitcoind node you run?
When running several bitcoind nodes on different machines, can I simply keep the same bitcoin.conf file everywhere, or do I need to ensure that a different rpcuser is set up for each?

The rpc user/pass is only needed if you're planning to access the node via RPC. If you're running Armory on top of it, for example, that requires RPC and hence you would need it (Armory auto-creates the rpcuser/password for you, though).
If you're running a node and don't need the RPC functionality, no, you don't need those parameters set.

Thanks v much Jimmy. This means I can keep same config file.


text:
C secp256k1 : what is the purpose of secp256k1_context_randomize?
In the course of studying the secp256k1 library API, I came acrosssecp256k1_context_randomize which seems similar to providing a seed to a pseudo-random generator. At the same time, the file secp256k1.h seemingly does not offer any functionality for generating random private keys, and the function secp256k1_ecdsa_sign relies on a deterministic secret (rfc 6979) rather than anything randomly generated. Now I can see the API being extended in file secp256k1_ecdh.h with the function secp256k1_ecdh which returns an EC Diffie-Hellman secret. But this doesn't seem to rely on random generation either since the scalar is provided as argument. So I am wondering, why would a user care to randomize a context, or in other words, where is randomization used with the library?

I just merged a PR by Rusty Russell that aims to explain its purpose. From the text that was added:

While secp256k1 code is written to be constant-time no matter what secret values are, it's possible that a future compiler may output code which isn't, and also that the CPU may not emit the same radio frequencies or draw the same amount power for all values.
This function provides a seed which is combined into the blinding value: that blinding value is added before each multiplication (and removed afterwards) so that it does not affect function results, but shields against attacks which rely on any input-dependent behaviour.
You should call this after secp256k1_context_create or secp256k1_context_clone, and may call this repeatedly afterwards.


text:
What are the debug categories?
In the bitcoin configuration you can set debug logging for the following categories. I can't seem to find an explanation of what each of these categories does. Can someone explain them or point me in the right direction?

debug=category
Output debugging information (default: 0, supplying
 is optional). If  is not supplied or if
 = 1, output all debugging information.  can be:
net, tor, mempool, http, bench, zmq, db, rpc, estimatefee, addrman,
selectcoins, reindex, cmpctblock, rand, prune, proxy, mempoolrej,
libevent, coindb, qt, leveldb.

net: Networking. All messages related to communicating with other nodes on the network, including what P2P messages were sent and received and to whom and other information about the network messages.
tor: TOR. All messages related to using a TOR SOCKS5 proxy and TOR hidden service (used for receiving incoming connections over TOR). This includes messages about the creation and shutdown of the TOR hidden service and messages about the connection to the TOR proxy.
mempool: Memory pool. Messages related to actions done in the memory pool including the acceptance and removal of transactions.
http: HTTP Server. Message related to the HTTP server that is used to handle the RPC requests. These messages will typically be for the startup and shutdown of the server as well as received requests.
bench: Benchmarks. Messages about the performance of various parts of the software that can have performance issues.
zmq: ZeroMQ. Messages about the ZeroMQ notification system, including the startup and shutdown of the service as well as when notifications are issued and new clients connected.
db: Wallet Database. Messages about the status of the Berkeley Database engine used for the wallet database. This will include messages about the database flushes.
rpc: RPC Server. Messages about the RPC server including its startup and shutdown as well as when commands are issued.
estimatefee: Fee Estimation. Messages about the fee estimation algorithm including messages about when fee estimates are requested and information about the status of the fee estimator.
addrman: Address Manager. Messages about the status of the address manager and when addresses are added or removed from the address manager database.
selectcoins: Coin Selection. Messages about the UTXOs that are selected when sending money.
reindex: Reindexing. Messages about the reindexing process, in particular errors about out of order blocks and repeated blocks.
cmpctblock: Compact Blocks. Messages about the Compact Blocks relay protocol including when blocks are partially downloaded or reconstructed.
rand: Randomness. Messages for when randomness is needed by any function.
prune: Pruning. Messages about local blockchain pruning, including the result of a pruning operation.
proxy: Proxy. Messages about using a SOCKS5 proxy and its authentication.
mempoolrej: Memory pool Rejects. Messages about transactions that are rejected from the memory pool.
libevent: Libevent. Messages about the libevent library which is used for the HTTP server.
coindb: Coin Database. Messages about the coin database which contains the UTXO set. Includes messages about database flushes and writes.
qt: Qt. Messages about Qt, the GUI framework.
leveldb: LevelDB. Messages about LevelDB which is used for the block indexes and coin databse.

Note that this is not an exhaustive list of the types of messages you will see for each category. Also some categories have many possible messages whilst others have very few.

Awesome, thank you. This is very helpful. You might want to consider adding this to the bitcoin github or wiki page (assuming it's not already there).

where in the source might one find this list?

The log categories can be found in src/logging.h as the enum LogFlags.

Also see this doc from Jon Atack: https://github.com/jonatack/bitcoin-development/blob/master/logging.md#log-categories


text:
How does bitcoin functional test framework work?
Could someone explain how and where in the code the bitcoin test framework calls map to C++ bitcoin core code or RPC calls such that regression suite tests the bitcoin core functionality?
Background and more details
I am writing a blockchain application and was thinking of how I can do regression while I keep developing. I wanted to leverage the design used by bitcoin (or any other blockchain application). I would expect that framework would call either C++ classes in bitcoin core directly or use RPC from python scripts such that features are tested.
I spotted that bitcoin functional tests are written in python scripts whereby test_runner.py calls a bunch of scripts testing specific features But I cannot spot in the bitcoin framework as to where it eventually calls actual bitcoin core c++ code or RPC.
Appreciate your help in spotting the code in python scripts in the framework that does it.

The functional tests test the RPCs. The unit tests test the C++ code directly by calling the functions.
The functional test frameworks uses a version of python-bitcoinrpc which can be found here. This library allows the test framework to call RPC commands as if they were python functions; authproxy handles the conversion to HTTP POST requests for the RPC server. So when you see lines like self.nodes[0].getblockchaininfo(), it is actually calling the getblockchaininfo command on node0. You will see many lines like this throughout the functional tests.


text:
BIP39 Manual Phrase Calculations - How are Multiple checksums valid?
Need some help understanding the math regarding why multiple checksums work for mnemonic phrase generation (BIP39).
Let's assume a 12 word passphrase. If we divide the 2048 wordlist into groups of 16 ... exactly 1 word out of 16 word "block" will be a valid checksum for the 11 words selected.
With a 24 word passphrase, 1 word out of every 256 would be a correct checksum.
When generating a phrase by hand... I know that the ENT / 32 bits of the sha-256 hash are appended to the entropy to generate the checksum word.. but this generates one specific word.
So I guess my long-winded question is ... what is the math behind other checksum values being valid? I guess my real question is how is the ENT + CS validated as legitimate?
See this example:
Entropy (128 bits): 11010011 01100100 00000010 01011110 01010011 11101100 01010011 01101110 01101010 01111000 11010010 11011000 10111010 00100011 11101100 11110010
SHA-256 Hash of entropy = 14 c5 8b c9 05 11 5e 08 27 49 61 1e 48 d6 04 c0 2a 70 8c 39 ad 6c dc 0c 91 2f 70 62 c3 24 71 23
First 4 bits of SHA-256 Hash = 1 (hex) Binary = 0001
Generated Recovery Phrase: square cactus nurse pond share rescue prepare bottom suffer speed will tomorrow
another valid phrase (with same entropy but different checksum): square cactus nurse pond share rescue prepare bottom suffer speed will account
Account = 13 (on the BIPS39 wordlist) Subtract 1 since word index starts at 0 = 12 12 to binary = 1100 Hex = C
C (hex) != 1 (hex)
another valid phrase (with same entropy but different checksum): square cactus nurse pond share rescue prepare bottom suffer speed will acoustic
Acoustic = 17 (on the BIPS39 wordlist) Subtract 1 since word index starts at 0 = 16 16 to Binary: 10000 Hex = 10
10 (hex) != 1 (hex)
I guess I'm also confused how these checksums are valid when their values do not equal the first 4 bits of the SHA-256 Hash of ENT?  I'm guessing it has to do with how the checksum is validated (and pointing back to my original question)?

I'm not sure why I get a different digest using Python3. Any thoughts?
`b = b'11010011011001000000001001011110010100111110110001010011011011100110101001111000110100101101100010111010001000111110110011110010'
hashlib.sha256(b).hexdigest()`
I get 3f0d468d18072538bedc4d5d47eb789795f1e01782de20de795027ca4ff1790b

Update: the following code works. b= b"11010011011001000000001001011110010100111110110001010011011011100110101001111000110100101101100010111010001000111110110011110010" b = int(b,2) b = b.to_bytes(16, byteorder="big") hashlib.sha256(b).hexdigest()

You aren't actually encoding the same entropy with different checksums. The last word does not encode just the checksum, it also encodes some portion of the original entropy. For a 12 word mnemonic, the last word encodes 7 bits of the original entropy and 4 bits of the checksum.
In your example, the last 7 bits of the original entropy is 1110010. Combined with the 4 bit checksum of 0001, the corresponding word is tomorrow.
In your second phrase, the last bits of the encoded entropy is actually 0000000 which changes the checksum bits to 1100 so you get a different word, account.
In your third phrase, the last bits of the encoded entropy is actually 0000001 which changes the checksum bits to 0000 so you get another word, acoustic.
So in actuality, BIP 39 mnemonics have unique last word because it is a combination of the original entropy with the checksum which is unique. Changing the entropy bits changes the checksum, and changing the checksum invalidates it.

Original answer which is still correct but not applicable to this question.
A careful reading of BIP 39 shows that there are not multiple words which fit the checksum. Rather the BIP is so broad that invalid checksums are allowed and should only be warned against.
In your example, the second phrase is actually invalid. But BIP 39 says that such invalid phrases should be allowed, and so the wallet software will allow it. The checksum is effectively ignored and not checked (which kind of defeats the purpose of a checksum).

Thanks for this.  This is interesting.  Edits coming (hit enter too soon)

So really ledger's tool for BIP39 generation, https://www.ledgerwallet.com/support/bip39-standalone.html (looks like a copy of Ian's tool) has a technically incorrect implementation.  This is because BIP39 states that if an invalid checksum the software should say so.
The code at: https://github.com/bitcoinjs/bip39/blob/master/index.js#L78
Line 76-78 splits the ENT & CS from the binary of mnemonic.
Then Line 87 calls the function to generate the checksum from entropy, and compares it to the mnemonic phrase.  Line 88 should throw the invalid checksum error, but doesn't. Not sure why

I was actually using Ian's BIP39 tool, and not the code referenced above.  I'll need to test this other code out.  Thanks again!

@Andrew Chow Your answer is incorrect. A careful reading of BIP39 shows that his second phrase is totally valid for the first 4 bits of the sha256 of the first eleven words + 7 bits of "acoustic" are equal to the remaining 4 bits of "acoustic". What the BIP39 is referring to when talking about invalid checksums is, well, if the checksum is invalid (as when before mnemonic seeds were used and, say, plain random 256 bits were used).

@CedricMartin Indeed. This answer was written assuming that the mnemonics actually encoded the same entropy, but they actually do not. I will edit this answer.

I believe the checked answer is incorrect. However, so are some assumptions in the question. First off, since there are 2048 words, each word encodes eleven bits (not 8). And since 128/11 = 11 remainder 7, the entropy itself encodes the first 11 words and part of the 12th word. Conveniently the 4 bit checksum appended to the remaining 7 bits makes exactly 11 to encode the final word ("tomorrow").
In your example the other wordlists are also valid (and have valid checksums), but they do not have the 128 bits of entropy you wrote, and correspond to different entropy in the last 7 bits from above.

you're totally right. The upvoted and accepted answer is wrong:


text:
Setting up Bitcoin Core Wallet
Trying to set up Bitcoin Core Wallet.
I've started blockchain synchronization 4 times and each time it has reached a certain percentage and then failed. I've followed instructions on the website, tried running it in pruned mode, tried running it with reindex. Is anyone else facing the same problem?
Worse still, it restarts downloading every block after it fails. Any ideas? Is there a way to force a restart at a location. This is very frustrating. And have to say this is poor quality software. Bitcoin has no future if just setting up a full wallet is so difficult.
Debug.log

Could you please upload your debug.log file so we can help diagnose the issue?

Thanks. I've updated the question with a link to zipped log file

From your debug.log file, you can see these lines:
018-03-28 18:05:07 LevelDB read failure: Corruption: block checksum mismatch
2018-03-28 18:05:07 Corruption: block checksum mismatch

Typically this error is caused by faulty hardware. There have been a number of reports on github of the same issue, for example https://github.com/bitcoin/bitcoin/issues/10897
So nothing wrong with the software, you just have poor quality hardware :)

Thanks, that was helpful.
Now, I have never had a problem with that computer. Can't remember it blue screening a single time. So the question was, why was Bitcoin Core synchronization causing hardware failures?
From experience, I know unexpected hardware problems are usually a result of a system stretched to the limit, but neither CPU usage nor hard disk access was operating around 100%. So I downloaded Open Hardware Monitor (No plug here. I have no stake in this application) and started snooping around. Among other things, it provides current temperature information about the CPU and some other components. I noticed even though CPU wasn't running anywhere close to 100% capacity its operating temperature was hovering around its limit with synchronization running.
I realized I needed to figure out a way to keep the temperature under control. I am running Bitcoin Core on an Intel NUC, which is tiny and I figured the internal cooling wasn't up to snuff. So I blasted away a table fan on the computer and let it rip. Sure enough, operating temperature started staying reasonably below the limit. I also shut off the Bitcoin Core for 3-4 minutes every few hours to let the CPU cool down (I don't think that was necessary, but I wasn't taking any chances). In the end, it took some extra effort and monitoring, but block synchronization is now completed. My Bitcoin Core Wallet is up-to-date.
Thanks for your help again MeshCollider.


text:
Backing up my bitcoin wallet - is wallet.dat all I care about?
I know that there are many duplicates of this but I'm hoping that the way in which I word this explains and helps not just me, but others as well.
In any of the coin wallet  bitcoin and the alt coins, when I run a backup I see a wallet.dat file being created.

Does this wallet contain all the necessary information like private keys etc... in order to say encrypt this file and store it on a google drive , dropbox etc..  ?
If my hard drive takes a dive etc.. , CAN I simply take that wallet.dat file , and on another computer decrypt it and restore my information based on my coin currency?
If I do the encryption within the wallet, is the wallet.dat called something else?

I understand the exchanges , buying, trading , selling, mining...  I'm just not versed in the backup with my information... private keys etc...    thanks in advance.

Yes.
Yes.
No. Note that that the software's encryption encrypts only the private keys, not transaction and address book information, so to protect your privacy you may want to add encrypting the file itself.

Ok, so say I have multiple addresses, I should shutdown the wallet and encrypt and backup the wallet.dat , then If I had a hard drive failure, would I be able to recover all my crypto coins ?    If I add another address after i did a backup,  I would still have all the coins that i can restore onto another machine based on the wallet.dat file right?  I just would not have the transaction information or would that include a coin transfer incoming as well that i would lose ?

I hope that is clear above.  So all addresses are in the wallet.dat correct?  If I create a new address it won't be in the backup i gather, but would i lose coins sent to that address in the case of a needed restoration of the wallet.dat file ?    thx

@TomStickel: The wallet file has a keypool with 100 addresses (the default in bitcoin-qt, can vary by alt or configuration). So if you have a backup of wallet.dat it will contain your next 100 addresses. But if you've generated more than 100 addresses since the backup you can lose your coins. So if you use your wallet frequently, you should also update your backup frequently.

1 Yes if you use that wallet only to keep coins but you don't use it to pay.
2 Same as answer nº 1
3 No.
Trying to keep transactions difficult to trace, whenever you transfer bitcoins a second transaction is placed. That transaction moves a small amount to a new address created in your wallet. I don't remember how many "wallets" within your wallet are kept (I think 20 but not sure). After that, new wallets are created as old ones are emptied. So if you have a wallet with quite a few transactions and you check your balance in blockchain.info you will see that a few coins are lost. Well they are not really lost they are in those wallets.
If you use your wallet a lot, you better do backups often, or you will end with missing coins.
I hope i've been able to make myself clear, but if you have any question just ask.

Yes, thank you.  Hasn't blockchain.info been hacked before?

Blockchain.info is a web page, it only displays the information gathered from the Bitcoin's blockchain, and no, that blockchain can't be hacked (except the 51% attack)

51% is really impossible.  The people in a pool wanting to destroy their bitcoin and others is such not likely to happen.

Impossible whenever the hash power is distributed, but when the time comes that all the coins have been mined and having as a reward only the transactions fees, how many miners will continue? I guess we'll just have to wait and see.

I'm betting that is will never happen,  too many people watching this, involved.   I have been listening to this God of Bitcoin.  Andreas rules them all.   http://letstalkbitcoin.com/author/andreas/#.Ut2fVBDn-Co


text:
Trying to import a private key into bitcoind
I am in a special situation where I need to import my private key credentials and basically restore a wallet to bitcoind. bitcoind is running and I do have an rpc user/pass and server=1 in the bitcoind.conf, but no matter what I do, when I try and import the key I get the following error.
There is no password nor encryption on the wallet.
root@coined:/home#  bitcoind importprivkey "5JxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxHjdCia9dt7" "root"
error: {"code":-4,"message":"Error adding key to wallet"}

Are there any useful messages in debug.log?  Also, FYI, if you run this on the command line, your private key is probably now logged in your shell's history file (perhaps ~root/.bash_history).

If you typed it up manually, you might have a typo in there or are missing a character? I once had a uppercase "S" instead of lowercase "s" in something and it took me forever to find it.

@Murch copy/paste

Without anything in the log, it's difficult to day. But here are a few suggestions what could try to find out the cause of the error:

Do you have write permission on the wallet.dat file in your .bitcoin directory?
Is the wallet locked with a passphrase? Then you need to unlock it (would probably give another error message but worth asking).
Did you set the disablewallet option either in command line or configuration file?
Is your wallet possibly locked due to a -reindex operation?

leave away the "" around the key (and the label)?

Same issue with or without the quotes.

Not a trace of the error in  /root/.bitcoin/debug.log  :(

I have tried with a new user as well.


text:
What does "Safe mode: Warning: The network does not appear to fully agree! Some miners appear to be experiencing issues." mean?
My bitcoind client that runs on testnet has started returning a lot of errors today. It has been doing so for a few of hours (as of 2014-08-29 9:50 UTC).
When I call it via RPC, I get the following response on ALL requests
{
  "result":null,
  "error": {
    "code":-2,
    "message":"Safe mode: Warning: The network does not appear to fully agree! Some miners appear to be experiencing issues."},
    "id":"1"
  }
}

It doesn't seem to be downloading any new blocks. It seems that I'm not the only one who's affected. As of now, http://tbtc.blockr.io/ is also stuck on blocks that are almost 5 hours old.
The version of bitcoind I use is 0.9.2.1, and the debug log says:
2014-08-29 05:11:59 ProcessBlock: ORPHAN BLOCK 0, prev=0000000000000e819b253066739275290fb4274b580bf62fbd279c518e6e2ebd
2014-08-29 05:11:59 ProcessBlock: ORPHAN BLOCK 1, prev=0000000000000e70d898fa12f328b9ee2d97912e9a495fc80eaeeb781096cadf
2014-08-29 05:11:59 ProcessBlock: ORPHAN BLOCK 2, prev=000000000000256194246bdabb62d3b50e2c309217258b17c543d13ec839367c
2014-08-29 05:12:00 CheckForkWarningConditions: Warning: Large valid fork found
  forking the chain at height 277766 (000000009cf297bc2a9610af823b49fc1d98e001239e99204c3c410e1ad3fe54)
  lasting to height 277767 (0000000000000e819b253066739275290fb4274b580bf62fbd279c518e6e2ebd).
Chain state database corruption likely.
2014-08-29 05:12:00 UpdateTip: new best=00000000735de11454d7de10a71a14e9695135f59b70a4eb906b1107df842270  height=277770  log2_work=61.959615  tx=2398741  date=2014-08-29 05:20:26 progress=1.000001
2014-08-29 05:12:00 UpdateTip: new best=00000000e9cc8d1bea3894d11cd4bacf4099dac3b644cb4f39b4ba3ad89ed2f8  height=277769  log2_work=61.959615  tx=2398732  date=2014-08-29 05:00:09 progress=0.999999
2014-08-29 05:12:00 UpdateTip: new best=00000000d553cd36cc43b3d1255ec395c05e1d4ad730178cd987e960931399ae  height=277768  log2_work=61.959615  tx=2398721  date=2014-08-29 04:39:40 progress=0.999997
2014-08-29 05:12:00 UpdateTip: new best=0000000000000f46520ca51be43443905945d2db5b513f35c2283c2d95a41077  height=277767  log2_work=61.959615  tx=2398683  date=2014-08-29 04:19:31 progress=0.999995
2014-08-29 05:12:00 UpdateTip: new best=000000009cf297bc2a9610af823b49fc1d98e001239e99204c3c410e1ad3fe54  height=277766  log2_work=61.959004  tx=2398667  date=2014-08-29 04:16:35 progress=0.999995
2014-08-29 05:12:00 UpdateTip: new best=0000000000000e819b253066739275290fb4274b580bf62fbd279c518e6e2ebd  height=277767  log2_work=61.959615  tx=2398670  date=2014-08-29 04:15:08 progress=0.999995
2014-08-29 05:12:00 UpdateTip: new best=0000000000000e70d898fa12f328b9ee2d97912e9a495fc80eaeeb781096cadf  height=277768  log2_work=61.960226  tx=2398674  date=2014-08-29 04:24:20 progress=0.999996
2014-08-29 05:12:00 CheckForkWarningConditions: Warning: Large valid fork found
  forking the chain at height 277766 (000000009cf297bc2a9610af823b49fc1d98e001239e99204c3c410e1ad3fe54)
  lasting to height 277767 (0000000000000e819b253066739275290fb4274b580bf62fbd279c518e6e2ebd).
Chain state database corruption likely.

What does this message mean? How is a fork of a single block a "large valid fork"? Why is this that this seems to completely freeze bitcoind? How can I fix it?


text:
Launch bitcoind at startup on Ubuntu
I have bitcoind installed on my VPS runing Ubuntu 18.04, and I want to configure it to launch bitcoind on startup.
What the best way to do so with my configuration ?

There’s a suitable systemd configuration for bitcoind in the contrib directory.

Wanted to close this question but finally posted an answer as i could not fine a decent motivation (aside from "there is a major troll as the only answer" :p). Will also edit for clarity.

To start bitcoind at startup on a system using systemd, use the service configuration provided in contrib/init/ from the root of the bitcoin-core repository.
You can then manage bitcoind as any other systemd service (and possibly have some other rely on it, such as a Lightning Network implementation's service).
systemctl start bitcoind
systemctl stop bitcoind
# This enables launching the service at startup
systemctl enable bitcoind

You could add a cron program that runs on startup, creates a temp file to say it has already ran, and then execute. This makes it so that every time cron tries to run the program, it turns out empty.
Example:
touch /tmp/btc.true
[ ! -f /tmp/btc.true ] && sudo -u $USER sh -c 'HOME=~$USER; bitcoind -daemon'
exit 0

That’s awful. There’s a @reboot flag for cron if you want to be hacky.

Ok, I'm noobie in this theme, I really dont understand so well... My main question is why not to use systemctl enable bitcoind?


text:
what if the timelock is in the past?
I just started learning about bitcoin scripting and wonder:

what if the timelock in CLTV is in the past?
what if the timelock in nLocktime is equal to 500.000?

If the nLockTime is less than or equal to the next block height, it can be included in the next block.
To explicitly answer your questions:

The CLTV op restricts the nLockTime field of the spending transaction, therefore if it is set in the past the condition is already fulfilled.
If the nLockTime field of a transaction is set to 500 000 it is valid to be included in any block which height is superior or equal to 500 000.
This basically means any block right now if there is no block chain reorganization deeper than 160 000 blocks :).

This mechanism was cleverly used to discourage reorgs. First introduced in the bitcoin-core wallet, but now most wallets set the nLockTime field of their transactions to the next block.
This prevents, in a future where miners would mainly be compensated for their work by transaction fees, a pernicious incentive to rollback the last block if it included a high fee-paying transaction.

As darosior says you can only impose a timelock such that Bitcoin can be spent after a certain block height. You can't impose a timelock such that Bitcoin can only be spent before a certain block height.
A related question is this one on the possibility of a not() condition in Miniscript. As Pieter Wuille stated in that answer:

The nSequence and nLocktime fields themselves do observe time, and can only express larger-than constraints. Bitcoin transaction validity should be monotonic: once valid, absent double-spend, they remain valid.

The alternative would be a logistical nightmare. All mempool transactions would need to be reevaluated every block. Reasoning about validity of unconfirmed transactions would be a pain (e.g. what if you're paid using a tx spending the output of a soon-to-be expired tx?)


text:
bitcoin-cli failed with error: incorrect rpcuser or rpcpassword
When I run this command
bitcoind -testnet -printtoconsole -rpcuser=123456 -rpcpassword=123456

After bitcoind is up. I run this on the same Ubuntu 17.10
bitcoin-cli -testnet -rpcuser=123456 -rpcpassword=123456 getbalance

I get this result:
error: incorrect rpcuser or rpcpassword (authorization failed)
The second day I tried exactly the same way. I get the correct result. It is really weird.
I try to put these two lines in /home/user/.bitcoin/bitcoin.conf
rpcuser=123456
rpcpassword=123456

And I run this command
bitcoind -testnet -printtoconsole -conf=/home/gogogo237/.bitcoin/bitcoin.conf

And this
bitcoin-cli -testnet -rpcuser=123456 -rpcpassword=1234564 getbalance

notice that I intentionally changed -rpcpassword to a wrong password. But I still can get a correct result.
This is really unpredictable and obviously a mistake. Anyone can explain this error? Thanks!

interesting, but for sure not the default behavious. I am normally on OpenSuSE. I have self compiled and downloaded full clients. Please provide some more info: did you use bitcoin core from website? Self compiled? apt-get? Did you check download files (hash)?
I think I have a fresh Ubuntu virtual machine, where I could try to install the one you where using...

Self compiled from the newest source code.

similiar discussion here, as bitcoin15 is not yet released: https://bitcointalk.org/index.php?topic=2172765.0. You downloaded from https://github.com/bitcoin/bitcoin?

just downloaded from github, and couldn't compile, cause autoconf throws already some errors. And for sure ./configure as well. Damn, wanting to help, and now need help myself ...

To avoid others doing same error (RTFM correctly, not in a rush): autogen.sh, not autoconf !! Now it's compiling. Update to follow.

re-compiled on an Ubuntu system, fresh from github.com:
$ uname -a
Linux ubuntu-ae 4.10.0-19-generic

$ bitcoind -version
Bitcoin Core Daemon version v0.15.99.0-g96ac26e5662

$ bitcoind -testnet -printtoconsole -rpcuser=123456 -rpcpassword=123456

$ bitcoin-cli -testnet -rpcuser=123456 -rpcpassword=123456 getaccountaddress "myaccount"
n2r1aQiA37LLgy4qWjuLHXqA9ZhtcX4KG6

$ bitcoin-cli -testnet -rpcuser=123456 -rpcpassword=123456 getbalance
0.00000000

and with wrong password:
$ bitcoin-cli -testnet -rpcuser=123456 -rpcpassword=1234564 getbalance
error: incorrect rpcuser or rpcpassword (authorization failed)

so s.th. is wrong with the sources? Or with Ubuntu setup. What kernel do you have (uname -a) ?

just compiled the newest version. Here is my command
$ bitcoind -testnet -printtoconsole -rpcuser=123456 -rpcpassword=123456
bitcoin-cli -testnet getbalance
It just work! Wierd.

In case you changed your /etc/systemd/system/bitcoind.service make sure you run
 sudo systemctl stop bitcoind
 sudo systemctl daemon-reload
 sudo systemctl start bitcoind

to apply the changes.
The error from the question could (and did in my case) arise because I did not use daemon-reload to load the new config.

While this advice may be useful, the question doesn't ask anything about systemd or with changing systemd service files.

The error from the question could (and did in my case) arise because I did not use daemon-reload to load the new config. Since this solved my problem - which was the same as OPs -  I thought it would be a good thing to know.


text:
How do I identify my IP address on bitnodes?
I have supposedly successfully setup a node using the Bitcoin core, but when I go to bitnodes to remotely see if the node is communicating with the network, bitnodes says i am unreachable. Bitcoin core disagrees. How can I find my node's unique address? Using cmd to run a netstat query shows me an ipv4 address that is also unsearchable with a fluid port range (49800-49950).
Something tells me I know just barely enough about networking to be dangerous.

Are you behind NAT or a firewall?

using windows 8.1 pro and bitcoin core is listed as an exception

Are you behind Nat?

ah, perfect evidence of my lack of knowledge of networking. am i behind a NAT, yes because my computer is connecting to the internet through a router that identifies my computer with a 10.x.x.x IPv4 address?

Yes that's a private address range. Try setting up port forwarding on your router.


text:
Application security of Bitcoin Core
Is there any information about how Bitcoin Core has applied secure development or application security in the past and currently? For example, the inclusion of third-party dependencies must be checked and tracked so that a dependency doesn't contain a vulnerability or an exploit, and the same reasoning about the build system.
Is there any policy and mechanism in practice for secure development and/or testing and verification of the security, including the dependencies and the build system?
I read somewhere that early development of Bitcoin Core was done in a "jail" but I couldn't find any other source than the mentioning of it in that specific article:

To address uncontrolled build inputs, it is tempting to “jail” builds
into sanitized environments that always present a canonical
interface to the underlying build system. Indeed, this was the
approach taken by early projects such as Bitcoin and Tor
(rbm.torproject.org). However, jails result in slower build times and
impose technical and social restrictions on developers who may be
accustomed to choosing their tooling. Most jails cannot address
non-determinism issues either

Is there any more information about it? Was it a real BSD jail or just an isolated environment to make sure that only the allowed dependencies were included?

The authors of the article most likely referred to a system named Gitian that's been used to check the integrity of Bitcoin builds https://bitcoinmagazine.com/technical/what-is-gitian-building-how-bitcoin-s-security-processes-became-a-model-for-the-open-source-community-1461862937

Note that this system has meanwhile been replaced by Guix builds.


text:
Port 9735 for both MAINNET and TESTNET
The default port for the Lightning Network is 9735. This is the case not only for nodes on MAINNET, but also for nodes on TESTNET.
Why is that? What's the purpose of this? Wouldn't it make more sense to use different ports for the different networks?
Particularly if you run a MAINNET node and a TESTNET node on the same machine, how would the host know which incoming traffic is intended for which node?

Which implementation are you using? c-lightning has merged different ports for mainnet/testnet but it hasn't made it into a release yet. (It also hasn't made it into the BOLTs yet) https://github.com/ElementsProject/lightning/pull/4900

I don't know what the design decisions were that led to this, or their rationale, but:

Particularly if you run a MAINNET node and a TESTNET node on the same machine, how would the host know which incoming traffic is intended for which node?

It's not possible to run two services on the same IP with the same port. If you want to run both a mainnet and a testnet node, at least one of them will need to run on a non-default port. Apart from needing configuration to change the port on one of them, this shouldn't be a big deal.

I don't know why was made in this manner but if you wanna run both nodes (mainnet & testnet) in this particular port you could use tor hidden services.


text:
Bitcoin "accounts" feature - Is it practical or scalable?
On the bitcoin wiki's "Accounts explained" page:
https://en.bitcoin.it/wiki/Accounts_explained#Account_Weaknesses
It clearly states:

The accounts code does not scale up to thousands of accounts with tens of thousands of transactions, because by-account (and by-account-by-time) indices are not implemented. So many operations (like computing an account balance) require accessing every wallet transaction.

So my question is that should the standard web developer creating a bitcoin web app using the RPC interface use (or ever use) accounts?
Should we always use the default "" account since we technically always have to use accounts when accessing the RPC for things like getting the entire servers balance?
Should we just handle our users accounts on our own databases and ONLY use bitcoind for things like listsinceblock() to store transactions and user balances in our databases?

As mentioned in the page you linked, it is a bit awkward to use Bitcoin-qt's accounts system. Don't bother with accounts as they are intended to be a way to manage your money within the program. When you're having thousands of accounts you are already doing it wrong.
So how do you structure a service to do this?
Handle accounts completely on your database. When you generate an address, store it as a field under a user in whatever database table/collection you have for users. This will allow you to quickly and easily scale your address counts in the event you grow large, Bitcoind does not scale well.
Handling receives
This is awkward to do with the current state of Bitcoind, the program never really intended to support large services. Bitcoind can efficiently detect receives to addresses in wallet. When you use the RPC interface and use the "getnewaddress" command, Bitcoind will store it in it's wallet. I've tested out Bitcoind with a million addresses before, it takes a solid half hour to start up but receives are snappy with no noticeable slowness.
Use walletnotify and/or blocknotify to get new transactions and blocks. Those two are arguments you can put in the bitcoin.conf file that specify a program to run when one transaction or block happens. Walletnotify is called per new transaction to or from an address in wallet.dat and gives a notification when it receives the transaction (unconfirmed) and then again when it's in a block (1 confirm). Blocknotify works per block. See the suggestions on how to use walletnotify here.
Just load the hashes walletnotify gives you in a database and then on block check if each one has enough confirmations, be thoughtful of the fact that Bitcoind usually has 2 walletnotify events per txn (one for unconfirmed and one for when a txn makes it into a block). And make sure the program you call is light weight and quick.
The pro method
This ways a bit advanced, rarely used and you will need some C++/Java/Python/Go proficiency. The community have made many Bitcoin implementations available as open-source libraries you can link to. Many larger services simply need more complexity with receiving transaction and better scalability, which Bitcoind struggles to provide.
It's a lot more work but you can make your own "node" with features you need easily with the libraries available for a wide range of languages. For Java there's BitcoinJ, widely depended on and reliable. For C++ there is Libbitcoin, which has pretty good examples and looks simple. Python has a bunch of node implementations and modules. Even Golang has something pretty popular.
If you need examples, Github has a lot you can find on search.

Gavin Andresen also mentioned during his Bitcoin 2014 keynote that the accounts feature may be removed from bitcoind in the next year or so.

The accounts feature has now been removed

Hey, this is exactly what I am trying to do, store deposit transactions for each unique user address from the output received from listsinceblock. I would like to ask, when computing the sum of deposits for each unique user address, does one consider transactions only of category receive and generate and ignore all other transactions categories like orphan, immature and send?


text:
would universal use of arbitrage bot stabilize the market?
If each bitcoin exchange ran a bitcoin-arbitrage bot (let's say the same one)themselves, would that quickly eliminate the spread, and therefore the opportunity for arbitrage?

No it would not. Due to fees involved with transferring assets between exchanges, and exchange commissions, inter-exchange spreads would still exist. Everyone running bots would at best lower spreads, not eliminate them.

There are already bots running on every market you can imagine. Because exchange operators are highly technical, most exchanges offer API's from the beginning.

authors: Башкир Хитер Уфимский


text:
Searching through the blockchain
When a miner includes a transaction in a block, according to the integrity rules the miner should search for a previous transaction which output the coins being used in the current transaction. Now, the previous transaction is a record stored somewhere deep inside the blockchain.
What is the time taken to search and validate the previous txn output in the blockchain data structure?

If the block is released by mining pool or miner, each peer running bitcoin wallet such as bitcoin-qt or bitcoind will verify the transactions in the block before relaying the block.
The steps are following:

Verify SHA256 block hash which is structured from version, pref_block,merkle_root,timestamp, bits, nonce and standard SHA256 padding. When the block hash is invalid or minimum difficulty level it is rejected.

The pools or miners are supposed to release blocks according to these protocol rules
One of them:

For each input, look in the main branch and the transaction pool to find the referenced output transaction. If the output transaction is missing for any input, this will be an orphan transaction. Add to the orphan transactions, if a matching transaction is not in there already.

So there are multiple rules that must be followed for a block to get accepted and then relayed.

For the highligted part, how much time does it take to find the referenced output transaction? My question is really about the time taken to do the search for the referenced output transaction

@Smagula We really cannot say that it takes x ms it depends on the proccessing power that is verifying the transactions outputs and inputs. I cannot help in this way.


text:
Bitcoin core wallet backup corrupted by an external tool
Do you know if my wallet.dat which has become wallet.axx due to incorrect handling of an external program
(I do not know the name of the program, but it is used to encrypt Windows files)
has the possibility of reverting to initial format ".dat"
by myself ?
And above all without damaging the backup "Core" that it is ?

According to a random website coughed up by Google:

A file with the AXX file extension is an AxCrypt Encrypted file. AxCrypt is a file encryption program that scrambles (encrypts) a file to the point that it's unusable without first being decrypted with a specific password/passphrase.
When an AXX file is created, it's automatically assigned the exact same name as the unencrypted file but with the .AXX file extension appended to the end. For example, encrypting vacation.jpg results in a file called vacation.jpg.axx.
You can double-click an AXX file to open it with the AxCrypt software. However, note that if you're signed in to your AxCrypt account, double-clicking the AXX file will open the true file and not actually decrypt the AXX file.
Use the program's File > Open Secured menu to open the AXX file but not actually decrypt it. To decrypt the AXX file requires that you either right-click it and choose AxCrypt > Decrypt or use the File > Stop Securing option

You would first need to decrypt the file using whatever program you used to encrypt it in the first place. Once decrypted, the file will return to its original state. I'm marking this answer as off-topic since it is not necessarily related to bitcoin, but to file encryption software.


text:
What is the correct way to extract all the private keys and transaction data from many wallet.dat files?
I have a ton of old backed up wallet.dat files. More than 150 ones. It's a big mess. I don't think that any of them have any coins sent to them, but there is a non-zero chance that one of them does have a small donation. I'm also trying to grab their transaction history, like you can manually do with "Export as CSV" in Bitcoin Core.
I have of course tried using my Bitcoin Core, but this is absolutely hopeless. It does a full "rescan" every time which in practice makes it just sit there forever. It doesn't matter if I load the wallet.dat files in via the config file or with a command in its console window; it just "loads" forever for no reason. This method is completely ruled out as it would take many months of nonstop full system utilization, and meanwhile I can't even use my current wallet.dat since Bitcoin Core is busy doing this pointless busywork which has nothing to do with its theoretical ability to extract data from the wallet files that it created.
I have also spent countless hours installing Python and the many weird dependencies that pywallet.py demands to run at all. No matter what, it just gives error messages about how it doesn't understand the wallet format when I load in my wallet.dats. Seems like completely broken and useless software. There is nothing weird about the wallet format, and they aren't corrupted. All of them were created by Bitcoin Core between 2013 and 2017 with no fancy/special options.
What am I supposed to do? I just want the old private keys transferred to my current wallet and their transaction history for bookkeeping purposes.
No existing Stack Exchange question or online discussion/webpage has a working answer. Please see the comments below...

What kind of hardware are you using? On a modern desktop, synchronizing Bitcoin Core shouldn't take more than a day or so, if it takes months, there is likely something else wrong.

@PieterWuille Very powerful PC. "A day or so"? So you want me to sit here for 150 straight days with this which should take seconds?

Oh, I see the problem. It's rescanning all wallets in serial, not in parallel. A hacky workaround would be to add them all to your bitcoin.conf, and then start with -reindex. This will make it redo the block validation work, but while all wallets are already loaded, so they'll all be synced at the same time.

You can also use the separate bitcoin-wallet tool, which comes with Bitcoin Core. It's a standalone tool that doesn't try to synchronize with the network or anything. It has a dump command, but I'm not sure how useful/readable the dump format will be for you.

@PieterWuille Thanks for making me aware of the bitcoin-wallet tool which is indeed bundled with Bitcoin Core and which I was unaware of until you mentioned it. After a bit of fiddling about, I was able to make a dumpfile, and it was very quick. Unfortunately, I'm not at all sure what to make of the contents of that dumpfile, and I see no obvious way to import it back into another wallet, certainly not with any option in that tool? Also, does it include all data from the wallet.dat, including not only the private keys but also made transactions?

It includes everything, including transactions, keys, addresses, ...

@PieterWuille Do I use the importwallet console/RPC command from Bitcoin Core when running my current wallet to "merge" the old one (as the dumpfile) into my current one?

No, you can use the wallet-tool createfromdump command to create a new wallet, but I don't think a feature to merge them is really supported. You could try to concatenate all dumps (removing the first 2 lines from all but the first one), and then recreate (no promises if this will work, but it might). If you quickly want to check if the wallets have any transactions at all; transaction lines will start with "027478" (serialization of "tx" in hex).

@PieterWuille That's... frustrating. The idea was to grab all the old ones's unique data into my current one. Your idea might work, but I bet it's going to bark when it sees duplicate keys/entries and it'll be a hell to clean that up...

@PieterWuille I'm sad to report that what you suggested doesn't seem to work. I tried by first creating a new wallet manually with just one receive address labeled "TEST", then using bitcoin-wallet.exe to dump my current wallet as well as the new manually created one into a dumpfile each. Then I made a new file called combo.txt where I copied the dumpfile from my current wallet and then pasted all but the two first lines of the second dumpfile afterwards, then used createfromdump to create a new wallet. When starting Bitcoin Core, it only sees the data for my current

wallet -- no "TEST" receive address from the test wallet. So it doesn't work like that. :(

(And yes, naturally I have selected the right wallet in the GUI.)

Sigh. I can't find any solution. What is the way?

Can't "shrink" the number of wallets? Is any wallet a strict subset of another?

@Mercedes Many dupes, but not identical files still. But there's still a bunch even after removing near-similar ones...

I don't understand why Bitcoin Core has this tool which dumps wallets in an undocumented manner and then has no way to import/restore/merge them back in.

I'd use wallet metadata (you should have elsewhere, out of the wallet) to restore only the last wallet (or a few wallets) per user. If you can't do that, you could save the wallets, wait till the node program is separated from the wallet program (several versions from now), then in the future run multiple instances of the wallet program on top of the single node program.

@Mercedes I don't understand what you're saying at all...

If those wallets aren't from users, that could mean all of the wallets are yours. How on earth could you have more than 150 wallets for a single person? Some must certainly be strict subsets of others. About process separation, read this.

@Mercedes Why not stay on topic of how to solve the problem instead of speculating about these were used for?

@Ubaid bitcoin-wallet -help will give you documentation on the supported features, which includes restoring from a dump. Merging wallets has never been a supported operation. My suggestion at this point is what I said earlier: load all wallets, and start with -reindex, so all wallets are rescanned in parallel.

@PieterWuille I'm well aware of its help section. How do you think I used the tool at all? But it has no way to import the dump -- only to create a new wallet from one of them. Which is pointless. Might as well use that wallet.dat then.

Sigh... If that's the only way...