text:
So I mined a block, but why would other nodes accept my mined block?
When a miner finds a hash and is thus able to confirm a block of transactions, that miner will then broadcast the confirmation (block?) to the other Bitcoin nodes so that it can be added to the blockchain. My question is, what is the incentive for other nodes to accept this mined block, even if it is correct and checks out against older blocks? I ask because if the reward for mining blocks decreases over time, other nodes may selfishly not want to accept this miner's effort and prefer to figure it out on their own for a greater reward now than later.

The incentive to mine on the currently longest chain is that there is a risk to the dishonest miner that honest, non-mining nodes may have already propagated the first block and hence reject and not propagate the second block found at an equal block height.
As Proof of Work is not reusable this leads the dishonest miner to waste resources.

Thanks but I'm not asking about the incentive to mine I'm asking about why the incentive for other non-mining nodes to accept a newly mined block on teh current longest chain.  I mean could they theoretically all be a-holes and say, "Hey we don't like you so we're going to reject your block and let someone else mine hte current longest chain."

There is no direct incentive. Nodes could arbitrarily reject blocks, but it's not possible to know their point of origin with any certainty. A node couldn't reliable keep rejecting a single miners blocks.

There is a lot of risk to rejecting the longest chain. If you decide to reject the most recent block and try to mine your own version of it, there is no guarantee that the network will choose your block over the first one. It is more likely that the first one will be chosen because nodes will only accept the first block they see at a given block height. So there's risk there that you are just wasting energy on a block that won't be accepted and thus you don't get paid.

@AndrewChow, when you say, "nodes will only accept the first block they see at a given block height" ... why?  Is this part of the Bitcoin software?  What guarantees that this is the case?

@Dave Bitcoin primarily works on a first come first serve basis. Of course there are exceptions to that, but generally the first thing that a node receives is what it accepts as true. In the case of blocks, this is overridden by the most work chain rule, but until that happens, it sticks with the first block it sees. There is no guarantee that that becomes part of the most work chain, and if it does not, then it will switch to the competing block at that height that is part of the most work chain. It is part of the software and has been since Bitcoin's creation.

Ok, thanks for that clarification, that makes sense.

I'm curious if there's any game theoretic explanation for this. E.g. would we reach a nash equilibrium if everybody accepts proper blocks mined by other miners.

Interestingly enough, since answering the question I was introduced to this paper which shows that my answer only applies when there is a constant block reward. When the income of miners is dependent on transaction fees with a potentially high variance, then the incentives to behave honestly breakdown and miners are likely to engage in Undercutting strategies.

Funny how Andrew Chow’s first comment is the closest thing to actually addressing the question.

The thing to remember here is that a Bitcoin is valid if people think it is valid (i.e. is willing to accept it as payment).  When you mine a block, you get coins, no matter where the block is in the blockchain (even if you fork it into a block-tree).  However, these coins you just minted are only valuable to people who think they are valuable.  Bitcoin has a general rule that one should consider the longest blockchain (or, more precisely, the highest difficulty blockchain) to be the best.
If you tried to fork the blockchain by mining your own block out of the middle of the chain, rather than the end, you would mine coins, but find it hard to actually spend them.  Few people (if any) would be willing to accept your coins as payment because there is a much longer blockchain out there (the main block chain), so it is highly unlikely that your coins will be recognized as useful tender.
Added to this, the proof of work for bitcoin does not benefit in any way from partial work.  If you are 99.9999999999% of the way to mining a block and someone announces a block before you get there, it is still in your best interest to start mining off of their block immediately, because there's 0 advantage to continuing to mine your old block, and plenty of disadvantages.  In fact, you don't even know how close you are to mining a block, due to the randomness of the process.
(At least, this is true assuming all blocks are worth the same.  If, for some reason, one individual transaction had some massive transaction fee on it (say, 1000BTC), and someone else just nabbed it, it might be worth trying to finish mining the block yourself and fight them for that block.  You would, however, be fighting an uphill battle since they announced it first.  But those are edge cases)

"If you are 99.9999999999% of the way to mining a block" That's confusing phrasing. Do you mean "You have a hash with 40 bits matching the target"? Given that partial work doesn't count, after each hash you're either done or no closer than you started (well, technically you've eliminated one possibility out of 2^whatever). So there's no such thing as "99.9999999999%" done.

@Acccumulation You're right.  I'm not sure if there's a clean way of saying what I wanted to say there.  The idea I was going towards was if you had an oracle that could tell the future, and they said you would have to do 40,000,000,000,000 hashes to mine this block (i.e. your 40 trillionth one was the lucky one), and you were at 39,999,999,999,998 when someone else posted their block, it's still more effective to abandon the block you thought you were mining.

@CortAmmon Err, why would it still be more effective to abandon if you know you're only 2 hashes away from posting your own block? What if this Oracle also told you, you would be one of the first few nodes to receive your competitors block?

@Ash True.  The issue is that, when you receive a block, that's a good indicator that others have received it and are working on the next block in that chain as well.  If you had a way to get others to work on your block (such as knowing that your competetor's block was not widely distributed, and you can distribute yours "better"), then there would be a balance between continuing and abandoning, and my extreme numbers would probably be less than ideal.

it would have to be a coordinated effort between over 50% of miners, known colloquially as a 51% attack. then they could reject any blocks not their own. otherwise Matthew Stannard's answer applies.

Correct me if I'm wrong, but the 51% attack means that one entity controls 51% of the mining power and could therefore reject an honest mined block until they come up with it on their own.  But I'm asking If someone controlled 25% of the mining power and someone else controlled 26%, could those two people get together and decide to reject an honest-mined block on teh current longest chain?

since 25+26=51, yes.

@Dave: It's math.  It doesn't care about the difference between people, organizations, and temporary alliances to screw over one miner.  They're all mathematically equivalent cases.

The (partly social, partly technical) rule is that the longest branch of the blockchain is the one that counts as the Bitcoin blockchain.
If your newly mined block is the highest-numbered valid block in existence, other miners will have an interest in mining on top of that, because that gives them a chance to reach an even higher number, and thus minimizes the risk that the next block they mine will end up being and abandoned branch.
If you and someone else mine a block with the same number before either of you hears of the other's discovery, there's no guarantee about which of them people will try to mine ahead from. (This is your incentive to keep yourself appraised of new blocks being discovered). But as soon as someone finds a block the builds upon either your block or the other guy's, that will now be the longest existing branch and everyone will flock towards that for their further work.
Most of the time mining block n+1 carries exactly the same reward as block n, so there's no reason not to be working from the best known chain head. Just around the time when the reward decreases, there may be some shuffling to get into the last higher-reward block, but as soon as there are a few of the last-higher-reward block number floating around, it becomes a better bet to mine for a higher number (and therefore better chance of staying in the chain).

The rule non-miners follow is that the longest chain wins. Miners need their blocks to be accepted by non-miners because it's non-miners (like exchanges and wallet providers) that decide which bitcoins have actual value.
A miner who mines after your block will have the longest chain if he finds a block. A miner who ignores your block has to mine a block just to tie the longest chain and then mine another block to have the longest chain. The odds of him being able to mine two blocks before anyone else finds a block after your block are very, very low unless he controls (or conspires with) a very high proportion of mining power.
So if he wants to get paid if he mines a block, he has a very strong incentive to mine after yours.

Hi, you say, "The odds of him being able to mine two blocks before anyone else finds a block after your block are very, very low" but in the comments to Cort's answer, the point is made you don't know how close you are to mining a block, so couldn't someone luck out and just happen to find those nonce's that result in two consecutive blocks being mined?  What formula dictatse these odds you speak of?

@Dave Someone could luck out in that way, but they have no way to know that they will. So it would make no sense for them to take the risk that they won't.

People work on the longest chain because they expect other people to work on the longer chain. If there was a general sentiment of "We should ignore Dave's blocks", then miners would indeed avoid a blockchain with your blocks in it, because they expect others to not validate any blocks they mine on that chain. Otherwise, they would be mining a block that they have no reason to expect would be accepted. And even if there were some anti-Dave conspiracy, this would be counterproductive because the value of bitcoin lies in its perception of security. Gaming the system would decrease the legitimacy of bitcoin and therefore make the bitcoins they just mined less valuable. If a consortium of 51% of the mining power were to take over bitcoin, why would anyone not in that consortium want to buy bitcoin?
As for the decreasing reward, that really only applies to one block per halving, and halving are rare; there's only been two so far.


text:
Does Bitcoin Core write blockchain and chainstate data atomically?
In particular, when a new block arrives:

Is the block written to disk in an atomic fashion, so a new block either exists, or it doesn't?   (reader can never see a partially written state)

Are the summary databases (eg chainstate, block-index) written to, and flushed to disk in an atomic fashion along with the new block, such that the blockchain filestore will never be out-of-sync with the levelDB filestore(s)?

More generally, can the bitcoin datastores be said to exhibit ACID properties?

Are there any plans to change the datastore behavior in this regard?

(I believe the answers to 1-3 are: no, no, and no.  But I'd like to hear more details, and also rationale for the status quo).

Yes, yes, yes(*), and no.
The blockchain-related on-disk data Bitcoin Core keeps generally fall into these three categories:

The block data (flat blocks/blk?????.dat files with concatenated blocks).
The block index (LevelDB database blocks/index/*).
The UTXO set (LevelDB database chainstate/*).

In this list, the later entries contain references to the previous ones. The chainstate contains the hash of the block that is currently considered "active" (i.e., the UTXO corresponds to it). This hash must exist in the block index at all times. Further, the block index stores in which block file and position within that file each block is stored.
To maintain consistency:

Whenever block index writes to disk happen, the currently open block files are flushed. It is possible that bitcoind crashes during the block file write, which may leave a partially written block on disk, but since this block will not be written to the block index, after restart, it will be as if this block wasn't received at all; it will be redownloaded, and written to disk again (possibly in the same file byte range that the earlier attempt was written to as well).
Block index writes are done using a LevelDB batch(*), which either makes it entirely to disk, or not at all.
Whenever chainstate writes to disk happen, the block index is flushed.
When the chainstate is written to disk, and there is a large number of modified entries, the write is split up into multiple LevelDB batches, to reduce peak memory usage during writing. Before this happens, a "flush in progress" entry is written to the database, which is removed after the flush completes. If bitcoind crashes in the middle, the chainstate will be in an inconsistent state, but on restart the "in progress" record will be detected, and the relevant range of blocks will be processed and applied again to the UTXO set.

(*) There have been reports of LevelDB itself causing corruption of its on-disk files if interrupted during a write, on some platforms.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Is the Regtest genesis hash always the same or not?
I would like to know if we have two nodes that are each running the regtest network will the genesis block digest be the same for the nodes or not?  I ask because I notice in chainparams.cpp that regtest takes params whereas mainnet and testnet do not.
If not, how should apps like block explorers deal with this?    It seems they could not know the regtest genesis hash in advance, and might have to query it from a full node based on block height or something.   Is that correct?

Yes, regtest always uses the same genesis block. It is hardecoded into the software. The parameters that regtest can take are for soft fork deployment so that they can be tested.

You might be looking at the wrong place. The line that defines the regtest genesis block is here.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Are Tor .onion-addresses IP address independent?
Is it possible to move the node to a different IP address and keep the existing (=preciously open and funded) channels functional?

Can you double check your grammar? I think there is a to too much and that confuses me. Other that that changing your internet Adress works and keeps channels open

Yes. The channels are announced with the node ID. If the ip Address changes we get a new node announcement message so it takes some time until the information is through the gossip protocol. But moving a node to a new server is no problem


text:
Is it advisable to operate a release candidate full node on mainnet?
Bitcoin Core 26.0rc2 was released a few days ago.
I've observed that several node operators have already installed this test release candidate version and are running it on the mainnet. Data from Bitnodes.io indicate that there are at least 12 such individuals (0.07% of all nodes).
Is it advisable to operate a full node using the Bitcoin Core test release candidate version on the mainnet?
Are there any benefits for the network, or could this practice potentially jeopardize the network in any way?

Running release candidates poses little to no danger to the network. The node software is designed to be highly self-reliant, at worst a node would ignore incorrect information and disconnect another node that misbehaves. A release candidate node might be less useful to its operator, in case it’s unstable or has issues making connections, but that would not significantly impact other users.
So, it depends on what you are using your node for. If you e.g. run a node as a datasource for the backend of your business processes, for your personal wallet, or as the source of blockchain information for an active Lightning Node, you should wait for the final version before updating.
If you run a node that is not directly involved in managing funds or business processes and you could stomach it being unavailable for a few hours, then I’d encourage you to update and play around with the release candidate.
If you do rely on a node for your business, you might want to spin up another instance of your testing setup to try that the new version works out for you and e.g. you don’t rely on any API calls that may have changed.
If you do have any unexpected problems, please file an issue in the Bitcoin Core issue tracker.

There is basically no risk to the network, as it's designed to resist even malicious actors, much less honest users just running untested software.
There is some risk to you as the user if you use the node as a wallet or to verify transactions, as any change to the code can introduce bugs, and you're one of the first people to run this new code. However, every change merged into Bitcoin Core's master branch needs to pass a lot of tests and (often many rounds of) code review, which should keep the bugs and their impact to a minimum.
I don't recall a bug that could result in a consensus split or loss of funds ever being caught during the release candidate phase, but there have been e.g. bugs that can cause a crash of the software in certain situations (I've discovered one such myself). So, only use a release candidate in such a way that new unexpected bugs wouldn't cause significant harm.

"There is basically no risk to the network, as it's designed to resist even malicious actors, much less honest users just running untested software." This is an important point that I have had to argue with people before. "If you're worried we might accidentally do X, remember that an attacker can freely do X on purpose at any time."

As its name suggests, it is not recommended to run a release candidate in production if you are not sure what you are doing. However Bitcoin Core release candidates aim to meet high quality standards. Running a release candidate on mainnet for testing purpose should not cause any issue. And if it does, please do report it!


text:
What are the last common blocks of Bitcoin and its hard forks (Bitcoin Cash, Bitcoin SV etc.)?
What are the last common blocks of Bitcoin and its most popular hard forks like Bitcoin Cash, Bitcoin SV, Bitcoin Gold, Bitcoin Diamond etc.?

Here listed only the top 4 forks by market cap (on 2023-10-12).

Name
Symbol
Market Cap
Fork Date / Forked From
Last Common Block with Bitcoin

Bitcoin
BTC
$523.0bn
N/A
N/A

Bitcoin Cash
BCH
$4.2bn
2017-08-01 / BTC
478,558

Bitcoin SV
BSV
$0.6bn
2018-11-15 / BCH
478,558

eCash
XEC
$0.5bn
2020-11-15 / BCH
478,558

Bitcoin Gold
BTG
$0.2bn
2017-10-24 / BTC
491,406

Here's an infographic showing their ancestry.

(infographic credits to Ludovic Lars)


text:
Is there a well defined last block?
It's fairly well known when the last bitcoin will be mined, but is there a well defined last block?
By that I mean, is there a block height H after which Bitcoin Core nodes will refuse to accept any further blocks due to technical limitations either within Bitcoin Core, or within the Bitcoin consensus rules as currently defined? If so, what are those limitations, what is the value of H, and how do the limitations cause the final block height to be exactly H?
This answer supposes that block production can continue to infinity, but surely there must be some technical limitation within Bitcoin Core which bounds the maximum block height, even if that block height is extremely far in the future.

is there a well defined last block?

I don't know of a specific limit to an exact block height like the well-defined and well-known upper limit to the money supply (<21M).
However, with the current consensus rules, the last block will be produced around Feb 7th 2106, at 06:28:15 UTC.
A hard fork sometime in the next 80 years could extend that.

As you say there are inevitably implementation specific limits - maybe someone used an int16 for block-height somewhere. But I guess these would be relatively easy to increase in a later release of a particular software implementation.
In the network protocol, there are some limits, for example some timestamps such as the block time-stamp and lock-time are uint32. I guess it may not be possible to use lock-time after the 500 millionth block? There may be other constraints for which new messages will have to be introduced to increase the size of some fields or perhaps software changes will be needed to reinterpret them or handle them correctly.

Related

UNIX END TIME January 19, 2038 - Will it affect Timelock nLOCKTIME smart contracts past that date?
January 19th, 2038: RIP Unix Timestamps

The current consensus rules do not permit a new block to be created once the unix timestamp of blocks reaches its limit. A hardfork will be necessary to address that.

Thanks, answer amended accordingly.

Generally, there is no defined endpoint of the blockchain. Eventually the subsidy will go to zero, but blocks would continue to collect transaction fees. In fact, since the subsidy shrinks exponentially, transaction fees may well start exceeding the subsidy in the next two decades. There are a few issues such as the space for the timestamp running out, but the time horizons for these issues are very long, and I would expect a fix for such glaring issues to be uncontroversial.
There is a well-known tongue-in-cheek counterpart to the Genesis Block, though, the Eschaton Block:

There is much talk of the Genesis block, however rarely mentioned is the Eschaton block. It is the Omega to the Genesis block's Alpha. Its hash has 256 leading zeroes, with the time required for 2^256 operations its pretext proves that the universe has [successfully] completed


text:
Generate Public Keys from Extended Public Key in python
I want to get a normal Public Key from an extended Public Key in Python. Actually I want to get the same result as here in the Extended Public Key Children section. I know I need this structure:
child_public_key = parent_public_key + point(lefthand_hash_output)

I used this to get the lefthand hash:
         0x00 not sure if I need this                                         Index
         --                                                                  --------
pub_key="000343b337dec65a47b3362c9620a6e6ff39a1ddfa908abab1666c8a30a3f8a7cccc00000000"
chain_code="1d7d2a4c940be028b945302ad79dd2ce2afe5ed55e1a2937a5af57f8401e73dd"

h = hmac.new( chain_code.encode("utf-8"),pub_key.encode("utf-8"), hashlib.sha512 ).digest()
lefthand_hash_output=slice(0,len(h)//2)

But now I don't know how to make this point(lefthand_hash_output).
I also tried this, but got an error
seed=b'...' # replaced with the seed
root_key = bip32utils.BIP32Key.fromEntropy(seed)
root_public_hex = root_key.PublicKey().hex()
key=bip32utils.BIP32Key.CKDpub((root_public_hex) ,0)
print(key)

A sample source code would be very helpful. Thanks in advance!

The conversion from extended public key to child public key is specified in BIP32:

The function CKDpub((Kpar, cpar), i) →
(Ki, ci) computes a child extended public key
from the parent extended public key. It is only defined for
non-hardened child keys.

Check whether i ≥ 231 (whether the child is a hardened key).

If so (hardened child): return failure
If not (normal
child): let I = HMAC-SHA512(Key = cpar, Data =
serP(Kpar) || ser32(i)).

Split I into two 32-byte sequences, IL and IR.
The returned child key Ki is point(parse256(IL)) + Kpar.
The returned chain code ci is IR.
In case parse256(IL) ≥ n or Ki is the point at infinity, the resulting key is invalid, and one should
proceed with the next value for i.

The other used functions are defined so:

point(p): returns the coordinate pair resulting from EC point multiplication (repeated application of the EC group operation) of the
secp256k1 base point with the integer p.
ser32(i): serialize a 32-bit unsigned integer i as a 4-byte sequence, most significant byte first.
ser256(p): serializes the integer p as a 32-byte sequence, most significant byte first.
serP(P): serializes the coordinate pair P = (x,y) as a byte sequence using SEC1's compressed form: (0x02 or 0x03) ||
ser256(x), where the header byte depends on the parity of
the omitted y coordinate.
parse256(p): interprets a 32-byte sequence as a 256-bit number, most significant byte first.

The first problem I see in your code is that your public key is 34 bytes long. According to the specification it should be 33 bytes long and start with either 0x02 or 0x03.
Next, chain_code.encode("utf-8") doesn't convert a hexadecimal string to bytes. Use something like bytes.fromhex(chain_code) instead.
Finally, lefthand_hash_output is created wrong, should be h[:32].
For sample source code you should check out the implementation in python-bip32. It uses coincurve for secp256k1 curve operations.
def _derive_public_child(pubkey, chaincode, index):
    """A.k.a CKDpub, in bip-0032.

    :param pubkey: The parent's (compressed) public key, as bytes
    :param chaincode: The parent's chaincode, as bytes
    :param index: The index of the node to derive, as int

    :return: (child_pubkey, child_chaincode)
    """
    assert isinstance(pubkey, bytes) and isinstance(chaincode, bytes)
    assert not index & HARDENED_INDEX
    # payload is the I from the BIP. Index is 32 bits unsigned int, BE.
    payload = hmac.new(
        chaincode, pubkey + index.to_bytes(4, "big"), hashlib.sha512
    ).digest()
    try:
        tmp_pub = coincurve.PublicKey.from_secret(payload[:32])
    except ValueError:
        raise BIP32DerivationError(
            "Invalid private key at index {}, try the " "next one!".format(index)
        )
    parent_pub = coincurve.PublicKey(pubkey)
    try:
        child_pub = coincurve.PublicKey.combine_keys([tmp_pub, parent_pub])
    except ValueError:
        raise BIP32DerivationError(
            "Invalid public key at index {}, try the " "next one!".format(index)
        )
    return child_pub.format(), payload[32:]


text:
Satoshi's coding quality
I have little knowledge about C++, so I want to ask two questions from "expert" (C++) programmers here:

Why Satoshi wrote Bitcoin in C++ and not in (for example) C language? any technical reasons? OOP? just a personal choice?

How good is the quality of original Satoshi Bitcoin source code? I saw some opposite viewpoints about this topic on the internet; some say that it is like an academic code with low quality, some say that it is a code written by old pre-2000 C++ standard, some say it is a very secure code with high quality, and...

As far as I know, Satoshi never gave a definitive answer on why C++ was chosen, so we can only speculate. There are existing questions on this site about that question, e.g. Why is bitcoin written in C++?

Quality comes in many forms. I think it's fair to say that the original Bitcoin codebase, as published by Satoshi, was remarkably good, in the sense that it (even in retrospect) had very few bugs. However, the code was also not modern, in that it didn't have clear abstractions between the components, had no tests (that were published), and was written in a somewhat outdated style.

I'm not sure anybody could answer this except Satoshi. My guess would be that they were comfortable with it. There's no technical reason why C++ would be chosen over any other language, there are Bitcoin full node implementations in other languages. C++ is fast, C++ can lower development time compared to C and has some more functionality.

I can't speak to this, but I can say that much of the code has changed since then, if you are using this as a proxy for the security Bitcoin provides. A better proxy would be the size of the Bitcoin network, the length of time it has been online without any major security issues, the number of eyes on that code (a lot!), and the strength of the encryption underlying it. There are many classes of security problems the Bitcoin core client could without endangering people's funds.


text:
How to calculate the hashes in Lightning
I was wondering how I calculate the hashes regarding BOLT03 Specification:
The Revocation Pubkey is calculated like this:
revocationpubkey = revocation_basepoint * SHA256(revocation_basepoint || per_commitment_point) + per_commitment_point * SHA256(per_commitment_point || revocation_basepoint)

How do I calculate this hash:
Is it like all in hex:
SHA256(revocation_basepoint || per_commitment_point) = SHA256(compressed revocation_basepoint with prefix  || compressed per_commitment_point with prefix)

or is it like all in hex:
SHA256(revocation_basepoint || per_commitment_point) = SHA256(revocation_basepoint_x|| revocation_basepoint_y || per_commitment_point_x || per_commitment_point_y)


text:
What is the relationship between the Ripple secret key and the wallet name/passphrase?
The Ripple client has both a wallet name/passphrase and a secret key, and to the newbie it's not clear why there are two secrets. The client advises the user to write down the secret key and keep it safe so the funds can be recovered if the wallet is lost. If the funds can be recovered using just the secret key, then why does it have a wallet name/passphrase as well?
Is the wallet name/passphrase just for convenience (easy to remember)?
Why can't the wallet name/passphrase be changed?
Why not generate the secret key from the wallet name/passphrase so that the user has only one secret to keep safe?

Is the wallet name/passphrase just for convenience (easy to remember)?

Yes. Because anyone trying to steal your funds would first have to get hold of your wallet file before he could use the passphrase, it doesn't have to be quite as secure as the secret.

Why can't the wallet name/passphrase be changed?

Because then you'd have two passphrases you'd have to keep secure forever to protect your account. The decrypted wallet yields the secret and the secret owns the account. There's generally no way to be sure you've destroyed every copy of your wallet.

Why not generate the secret key from the wallet name/passphrase so that the user has only one secret to keep safe?

That would defeat the point of the passphrase because it would no longer be able to be any less secure than the secret. Anyone could try common name / passphrase combinations at a very high rate to see if any of them matched existing accounts.

If I know your wallet name/passphrase I can obtain your secret key by logging in to the Ripple client. Is it not? So it has to be as secure as the secret key. Or am I missing something?

@Manish: If you know my wallet name/passphrase yes. But if you had it on a list of a hundred billion possible name/passphrases, no. If you had the secret on a list of a hundred billion possible secrets, my account would be toast.

So you're saying that it would be easier to brute-force the secret key if it were generated from the wallet name/passphrase? Because then you could do something like a dictionary attack? And that's why it's better to keep them separate?

Yes, that's it. You can't brute force the wallet unless you have the wallet. There's nothing to stop you from trying to brute force the secret -- so it must be very secure. 38 bits is probably just barely enough for the passphrase (assuming someone doesn't have a copy of your wallet). 70 bits is probably just barely enough for the secret.

I've tried to summarize in a separate answer; please correct if wrong.

Is it possible for a different Ripple client to implement another kind of wallet (like a Bitcoin wallet with multiple secret keys), or is that something incompatible with the way Ripple works?

That's definitely possible.

The wallet name/passphrase is the key to the wallet file. The wallet file contains the secret key. The secret key is the key to the funds.
If you lose the wallet file, you can still access your funds with the secret key.
If you lose the secret key, it's there in the wallet file.
If you lose both the wallet file and the secret key, then you've lost access to your funds.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
What are the special Ripple accounts ACCOUNT_ZERO, ACCOUNT_ONE, and root account?
What are the special accounts ACCOUNT_ZERO, ACCOUNT_ONE, and root account? It's not clear from the Ripple wiki what they are meant for. A little explanation would be nice.
What makes these accounts "special?"

The root account is the account that owns the XRP in the initial ledger.
ACCOUNT_ZERO is an account whose 160-bit identifier is zero. When dealing with a currency/issuer pair, ACCOUNT_ZERO is used as both the currency and the issuer to indicate XRP.
ACCOUNT_ONE is an account whose 160-bit identifier is one. ACCOUNT_ONE is used in cases where no account is applicable but the currency is not XRP. For example, in a ripple balance node between two accounts, the "balance" belongs equally to both accounts and is really a property of the two of them. It has an issuer of ACCOUNT_ONE.
ACCOUNT_ONE is also sometimes used as a flag in internal arithmetic operations to indicate that non-XRP semantics are desired. XRP amounts are stored as integer drops (millionths of an XRP) while other currency's amounts are stored in exponential notation. Sometimes an amount isn't in any particular currency (for example, when outputs and inputs are divided to compute an offer's quality). To indicate non-XRP semantics without having a valid currency, ACCOUNT_ONE is used. It essentially means "not XRP".

So essentially all the XRP starts out as belonging to the root account and is distributed from there. Does this mean that the root account still owns most of the XRP (as of this date)? And looks like ACCOUNT_ZERO and ACCOUNT_ONE are just special values, they're not actual accounts with any balance in them.

The root account has long since been emptied. And yes, they are just special values, not actual accounts. You could never create them because you'd need a key that hashed to 0 or 1. Good luck finding one.

Does this mean that, if someone, by an incredible amount of luck, happens to generate a key for one of these accounts, he will be a rich man?

@StevenRoose Probably not, but if he happens to generate a key for an account that holds a lot of funds, then he will be.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Why does Ripple's consensus process have a 80% threshold?
A transaction is confirmed if 80% of the nodes in the UNL accept it. Where does this 80% number come from? Could it just as well have been 79% or 81%? What about 51%?
Can different nodes have different thresholds (for faster transactions, say), or must they all follow the same rule?

The explanation video says it is "the threshold for mathematical certainty"...

@StevenRoose Yeah, but is it? I mean, why not 90% or 75%?

If validators waited until other validators declared consensus to declare consensus themselves, no validator would ever declare consensus. So you must declare consensus and hope you turn out to be right. Most of the time you will be. If not, you will later not be mathematically certain of the results and will not pass them on to clients.
If you use a 75% threshold for mathematical certainty, many times you wouldn't be certain and would have to repeat the process. If you picked a 90% threshold for mathematical certainty, you'd be certain almost all of the time, but you'd be wasting time.
The video is a bit misleading. In fact, any threshold could be used and you would still have mathematical certainty if you actually did reach consensus. However, if you set the threshold too low, you would only rarely have certainty. 80% is the threshold we've chosen for mathematical certainty because it's a compromise that results in certainty being reached often and quickly.
Maybe that's not clear enough. Let me try it another way. When we report a transaction as fully validated, we want to be mathematically certain of that. We will not make such a report until we are. Choosing 80% as the consensus threshold allows us to reach consensus quickly, requires at least 20% of validators to be completely malicious to delay consensus, and allows us to (in a later step) be mathematically certain of the result almost all of the time.
Note that almost all of the time is not always. Sometimes, we reach 80% and later are not certain of the results. In that case, we wait until the next consensus round completes and hope we fare better in that one.
Say for some stupid reason you set your threshold to 10%. You would think you had consensus all the time. But equally obviously, a lot of the time there would be no actual consensus. If there were, you could be certain that it had happened (because you would see it). But it often wouldn't happen, and then validators would have to resynchronize and clients would be waiting patiently for the network to fix itself so their transactions can be confirmed.
Ledgers that pass the consensus process are still held up by the "validation gate". They are not reported to clients as "fully validated" until they pass this gate.
80% worked very, very well in simulation. Validators are free to change this threshold if they want. Perhaps in a future network topology, a different threshold will give better performance characteristics.
It's hard to write clearly at 3AM. Maybe I'll rewrite this when I'm more awake.

"Sometimes, we reach 80% and later are not certain of the results." Why would you be "not certain" when 80% is the very definition of certainty? Do you mean you wouldn't be certain "in a later step" (which you allude to)? "... requires at least 20% of validators to be completely malicious to delay consensus." That's 20% + 1, right? And what is the definition of "fully validated?" (By the way, your detailed answers are much appreciated. I'm sure this will be useful to anyone taking an interest in Ripple.)

Once you reach the 80% threshold, you move on to mathematical certainty. If you didn't actually have a consensus, the certainty step would fail. A transaction is "Fully validated" if a majority of validators you trust have signed off on a ledger that includes it. If there's no actual consensus, then the ledger won't fully validate. (Which is fine, it just wastes time.)

So there are two steps: (1) consensus (80%); (2) validation (majority of validators). But why two steps? And how much is the "majority" in the second step?

There need to be two steps because someone has to build a ledger first or nobody ever will. Think of a crowd of people walking towards two doors and they want to walk through the same door. Once 80% are closer to one door, you walk through it. Then you look back to see if everyone follows you (if so, you are mathematically certain you walked through the right door). If not, you failed and try again. The majority in the second step is currently 50+%.

Why ripple needs 80% of threshold in consensus? What is the mathematical prove behind that? Why not 66% threshold? Because i think 66% threshold in consensus is enough to validate a transaction in a distributed environment. Because (2m+1)/(3m+1) equation gives a valid threshold in a Byzantine fault, where 2m+1 represents the genuine validation nodes and 3m+1 represents the entire validation nodes.

With a lower percentage, the required overlap in trust lists is greater, making it harder to avoid catastrophic failure (double spends).

@tdumidu To make this a little clearer. Ripple needs 80% to reach the 66% required for the algorithm to work because people are taking 80% of different things and we need them to be taking 66% of the same thing. Say you and I have lists of names with lots of overlap. If we each require 80% of the people on our list to agree, that gives us a good chance that we will have 66% overlapping agreement. That's what we need to still make safe forward progress with the highest number of malicious nodes.

@DavidSchwartz Where can we read more about how the final step works that ensures "mathematical certainty"? Sure, if you demand 80% local "pre-commit" certainty, there will be a higher chance that the final commit succeeds. But starting from this, how do you actually guarantee that > 2/3 or > 1/2 of the total participants agree on the same state and thus pass the "validation gate" you mention? It seems that to achieve that, knowing the commitment of UNL servers is not enough, as you need to know if n% of ALL nodes have committed.

@nh2 You don't need to be certain to commit. You just need to be certain that if you do commit, no other honest node will commit to a different ledger.

@DavidSchwartz But how do you achieve that certainty? In your example of going through the door, what you just said would be analogous to saying "you don't need to be certain to go through the door. You just need to be certain that if you go through the door, no other person will go through another door". But in your example you highlighted that if not everybody followes you, you failed and try again (you go back through the door). So that sounds like no single node can be certain that everybody will follow if they go through the door. [continuing in next comment ...]

[... continued] In other Byzantine consensus algorithms, they obtain hard guarantees on the agreed value by collecting a supermajority (typically > 1/2 or > 2/3) of all nodes (not only some subset of the network graph). As soon as more than this many yes-votes have been received, guarenteed finality is achieved (the node receiving this many votes is 100% certain that the network will not accept a different value instead, and any form of rollback is impossible). Does this concept exist like this in Ripple?

80% is a supermajority of all nodes that you consider to be participating in the consensus process. There has to be sufficient agreement on who is participating to ensure that no two participants can see an 80% supermajority for a different ledger. You can think of sufficient agreement on participants as a network rule, akin to block size in bitcoin. Agree on it with others or you risk/create a fork.

Or, to put it another way, instead of being forced to pay three companies we didn't choose millions of dollars a day to decide which transactions go in a block, let's just agree on those three companies subject to them following deterministic, fair rule for transaction inclusion and retain the right to gradually change the companies if they don't behave as we wish them to.


text:
How does Ripple decide on the exchange rate between different currencies?
When A pays in USD but B receives EUR, how exactly does Ripple decide on the EUR/USD exchange rate?

Since Ripple has an integrated exchange, I guess it simply places market orders in its order book, trying to get the best overall price.
You find the order book in Advanced/Trade, if you want to try manually.

During a payment, it's actually a bit more complicated than that because the system will consider multiple payment paths. For example, it may consider EUR->BTC->USD, or it may consider EUR->USD on more than one exchange. It's not unusual to find a better rate on a payment than you saw in the order book. (In fact, you can make a payment to yourself to exchange currencies, and it's usually better!)


text:
Block inventory messages
I've written some code for connecting to the network, receiving inventories, and requesting data. What I am noticing is that my local node never sends me inventory messages with blocks in them unless I specifically request blocks.
So if I do a getblocks call with a zero stop hash I will get all of the blocks up to the tip, but never any inventory messages after that. Is there some kind of signalling necessary to tell the node i am connected to that I want inventory block messages to stream to me when new blocks are found?
I am connecting with protocol version 70002, relaybool is 1, nodeservices is 1.
I don't see other mentions of this issue so I'm wondering if there is a negotiation issue that isn't being caught or some other problem.
There is no visible ban score, connection is from localhost, I get a verack and normal tx inventory messages are streamed. This happens regardless of whether -whitelist is used. I just never get block inventory updates or unsolicited block messages unless I request blocks via getblocks. After I've retreived blocks up until tip streaming does not commence.
Update: The code has no problem syncing with a bitcoin-core instance on another computer, but the local bitcoin core instance will never send block inventories or respond to getheaders requests. Problem only occurs on localhost to localhost initated connections.

Bitcoind's block directory was corrupted. (0.11.0)
After exhaustive testing, it turns out that bitcoind's block directory was in some way corrupted, and so even though everything indicated it was at the correct chain tip it was not serving blocks to peers without an explicit getblocks request. It would ignore getheaders requests, and not relay block inventories. A complete resync solved the problem.
The key was realizing that the same code could connect to foreign nodes without issue, and that during wireshark dumps the localnode never relayed  blocks onto the network at all.

You need to issue a getdata on the last entry in the returned invs. This is used as trigger that you have processed the response, and the remainer will be sent automatically.
Note that since version 0.10, Bitcoin Core does no longer use the getdata/inv/getdata/block cycle to synchronize, but getheaders/headers (for synchronizing headers). Actual blocks are then fetched asynchronously from different peers after the headers are discovered.

In the case where my nodes height is equal to the local peer, if I ask for blocks none are returned (which makes sense), and no inv messages are sent that I can issue a getdata for. I understand that I can switch to the headers first mode, but for compatibility I would like to support blocks signalled via inv messages.

I appreciate the answer though, I feel like in the base case I should be getting those inv messages after a getblocks where I am tip of chain. That isn't happening which leads me to believe there is some other condition that is preventing it from happening. Is there something else I could be missing?


text:
Duplex micropayment channels: criticisms of Lightning and implementation
A paper from 2015 by Decker and Wattenhofer proposes an interesting design for a payment channel network, which differs from the Lightning network. The authors claim in the Related work section that Lightning:

"requires exchanging keying material for each update in the channels, which results in either massive storage or computational requirements", and
"cannot be decomposed into smaller units that can be analyzed in isolation, making the security analysis difficult and resulting in complex implementations".

I have two questions:

To what extent are those criticisms of Lightning valid as of November 2017?
Is there an implementation of Duplex micropayment channels or at least plans to create one?

As of today, (November 2018):

To what extent are those criticisms of Lightning valid as of November 2017?

This is still a case for Lightning. Eltoo was the first to suggest a simpler way of coping with such storage requirement. They do not require storing keys or older transactions. Their disadvantage is that there is no punishment in the event of fraud, leading to selfish users always publishing the transaction that maximises their balance. In current Lightning Implementations, it is important to keep track of Breach Remedy Transactions, and of course never lose new Commitment Transactions (as this can incur in a punishment to oneself after publishing an old balance unknowingly).

Is there an implementation of Duplex micropayment channels or at least plans to create one?

Current implementations are focusing in Lightning Channels for two main reasons:
1. Eltoo channels are too recent (perhaps somebody is working publicly in an implementation, I am not aware)
2. DMC channels have a trade-off between the utility of the channel and the worst-case collateral cost of the channel, that discourages its use. This is even more important when considering n parties (such as factories). For more information about this trade-off, see the results in this document
DMCs are however better when one knows of a period of time that one will be offline, so I believe it will be implemented in the future. But there is no current implementation in the main ligthning development teams.


text:
Question on the integer size
Is there any kind of boolean division with bitcoin?  There has to be something before the decimal number.  Kind of say it seems like end to end and it always should be the number that is largest.

What do you mean by boolean division?

Question on the integer size

Bitcoin uses many sizes of integer from 8-bit integers up to at least 256-bit integers. It also has variable-size integers. It uses a mix of signed and unsigned integers.

Is there any kind of boolean division with bitcoin?

There is no kind of division at all in Bitcoin scripting language since OP_DIV and OP_2DIV were disabled.
I imagine fairly normal division is involved in calculating network difficulty and in calculating network time for checking block timestamps.

There has to be something before the decimal number.

The Bitcoin network probably doesn't have significant use for decimal numbers.
Units of money are integer Satoshi. There are 100,000,000 Satoshi per Bitcoin. The user interface of software sometimes converts money in integer Satoshis to some decimal unit such as Bitcoin or millibitcoin for user-interface purposes only. Internally it is all integers.
Hashes and signatures are mainly working with large integers so far as I know.
The compressed "bits" representation of network difficulty bears some superficial resemblance to floating point number representation but nothing else decimal-like occurs to me.


text:
How to get transaction fee info using Python from an Electrum server?
I can get a raw transaction and parse it using electrum and bitcoinlib fairly easily:
import bitcoinlib

from electrum.daemon import Daemon
from electrum.simple_config import SimpleConfig

class TxInfoClient:
    """Transaction info client"""

    def __init__(self):
        common_config = parse_config()
        electrum_config = SimpleConfig(
            {"testnet": False, "server": common_config["electrum"]["server"]}
        )
        self.daemon = Daemon(electrum_config, listen_jsonrpc=False)

    def get_tx(self, tx_id):
        """Returns a bitcoinlib.transactions.Transaction:
        https://bitcoinlib.readthedocs.io/en/latest/source/bitcoinlib.transactions.html#bitcoinlib.transactions.Transaction
        """
        raw_tx = self.daemon.network.run_from_another_thread(
            self.daemon.network.get_transaction(tx_id)
        )
        return bitcoinlib.transactions.Transaction.parse_hex(raw_tx)

However, the resulting Transaction object won't contain fee information because it isn't possible to calculate Bitcoin Tx fees from a single transaction.
I could go through and query input transactions manually, as suggested in that link. However, I am looking for a simple solution that uses well established and maintained libraries to do it.
Is there a way to make this happen using bitcoinlib and electrum, or other mainstream libraries?


text:
Are there any plans to make Bitcoin quantum-resistant?
There is a lot of discussion about quantum computing and how it will be able to crack wallet addresses. This could happen as early as 2025 says one source. Microsoft and other entities are working to secure their networks against quantum computers. I am curious to know if Bitcoin is addressing this issue?

See also: What would a bitcoin quantum resistant address look like?

First of all, Bitcoin is a decentralized protocol with no one in control. It doesn't and can't have any plans or roadmap, and any changes to it require consensus of its users.
With that out of the way, I don't think quantum computing is seen as an imminent threat. From what I could find, even optimistic predictions place practical quantum computing at least 10 years from now, and it's not a given it'll even ever be achieved.
On the other hand, quantum-resistant signature schemes use much larger public keys and/or signatures, which would seriously limit transaction throughput. According to a recent blog post from Ledger, the state of the art PQC scheme uses 1563-byte signatures, more than 20 times larger than the signatures currently used in Bitcoin. Since this is an area of active research, I expect people will want to wait for more improvements before committing to adding a specific scheme to Bitcoin.

Satoshi Nakamoto was questioned on this in 2010 and had the following reply:
Bitcoin Forum: Re: Major Meltdown July 10, 2010, 01:36:17 PM
Quote from user "llama":

However, if something happened and the signatures were compromised (perhaps integer factorization is solved, quantum
computers?), then even agreeing upon the last valid block would be
worthless.

Response by Satoshi Nakamoto

True, if it happened suddenly.  If it happens gradually, we can still
transition to something stronger.  When you run the upgraded software
for the first time, it would re-sign all your money with the new
stronger signature algorithm.  (by creating a transaction sending the
money to yourself with the stronger sig)


text:
How can the Bitcoin blockchain be forked?
I understand in the past the blockchain has been forked (I.e BCH, etc). Can this be done by a single miner? How exactly does this happen?

Anyone can fork Bitcoin, they don't even need to be a miner. The general steps are:

First and foremost, you need to change some consensus rules. Without this step, you aren't actually creating a new cryptocurrency, because Bitcoin is the consensus rules.

Assuming you don't expect your fork to have more hashrate than Bitcoin when it launches, you need to make sure future Bitcoin blocks won't be valid on your fork, otherwise your fork's blocks will just get overwritten (i.e. a soft fork – see Does a hard fork need to include a soft fork as well?). This also applies to other forks that you might accidentally be compatible with.
To allow people to transact on one chain without their transactions being replayed on the other, you need to implement some form of replay protection, for example by changing the sighash algorithm.
A commonly implemented consensus rule change in Bitcoin forks is changing to a more responsive difficulty adjustment algorithm and/or a one-time difficulty decrease at the fork height, otherwise you might have to wait a long time before even a single block is mined. If you make the difficulty low enough you can even mine on a CPU before it goes back up (you could also change the proof-of-work algorithm to an ASIC-resistant one).
Pick a specific block (preferably in the near future) where your fork will start.

To launch a new peer-to-peer network you should also change the network magic number, or make any other invasive change to the peer-to-peer protocol that makes it incompatible with the Bitcoin network. You also want new DNS seeds.
Watch as your fork falls into irrelevance, assuming it ever had any relevance to begin with.


text:
Why doesn't bitcoin script number follow two's complement?
Bitcoin script number encoding follows sign-magnitude according to BIP62, while most of programming language uses two's complement for signed integer(i.e. Rust format hex representation of signed integer following two's complement).
As it's just encoding and the number of hex digit is not fixed, it might not have an advantage of two's complement for arithmetic operation, but wonder if there's an advantage of adopting sign-magnitude over two's complement.

I'm not aware of any rationale explicitly given by Satoshi for using sign-magnitude over another representation. It has the advantage of being able to more compactly represent some common numbers.

Thanks for your answer! However, as far as I know, isn't it a very little bit more compact to use two's complement as it doesn't have negative zero(so can represent -128 in a single byte unlike sign-magnitude)?

@Hyunhum That's true, though the difference is absolutely minimal (and only relevant for negative numbers, which are rarely used anyway). Ultimately the question is guessing Satoshi's motivations, which isn't possible in a conclusive way.


text:
Is there an economic incentive to use P2WSH over P2TR?
As I understand, p2pkh and p2sh are nothing better than p2wpkh and p2wsh, other than compatibilty.
After taproot, p2wpkh is also nothing better than p2tr, other than compatibilty(no economic incentive according to this).
However, when it comes to comparison between p2tr script spend and p2wsh, there would be an "economic incentive" to use p2wsh over p2tr.
If under following assumptions,

Tapscript only contains "single" script(tap leaf is single).
Script itself cannot be divided into multiple leaves(i.e. OP_IF A OP_ELSE B can be divided into 2 leaves).
Script requires less than 5 signatures, as schnorr signature is more compact(about 6~8 bytes) compared to ecdsa DER format in size.
No motivation to spend in p2tr key path.

All of the witness items to provide would be same for p2wsh and p2tr if locking script to spend is same,
except that p2tr must provide tap control block of 33 bytes, even if it's only for single script spend.
However, I found the answer regarding comparison of p2wsh and p2tr here, which states,

As far as I can tell, P2TR is better than P2WSH in almost every way. I cannot think of a case in which it would be a significant advantage to use P2WSH over P2TR, except that P2WSH is already established.

Am I missing something? or is there other significant benefit which can offset this?

Yes, having only a single leaf and all else being equal, a P2TR input costs 34 witness bytes (8.5 vbytes) more than a P2WSH input with the same script.
However, almost all current usage of P2WSH (which consists mainly of multisigs and Lightning channels) can benefit from the features of P2TR to reduce costs. Multisigs are cheaper with key aggregation schemes such as MuSig and FROST and can often use the key path to reduce costs even further. Lightning benefits from splitting conditions into multiple leaves, using the key path for simple key conditions and the aforementioned key aggregation.


text:
What prevents Dash miners from refusing to pay to masternodes?
I struggle to understand the repationship between Dash miners and masternodes. The proposition is that masternodes provide additional services to the network and automatically get paid a part (45%) of the block reward. What I don't understand is how this share is enforced.
From the Dash whitepaper:

When mining on the network, pool software (websites that merge the
  efforts of individual miners) use the RPC API interface to get
  information about how to make a block. To pay the Masternodes, this
  interface must be extended by adding a secondary payee to
  GetBlockTemplate. Pools then propagate their successfully mined
  blocks, with a split payment between themselves and a Masternode.

What prevents miners from simply ignoring the masternodes and assigning 100% of block reward to themselves?
Edit: from Dash documentation:
If a miner tried to take the entire block reward for themselves or tried to run an old version of the Dash software, the masternode network would orphan that block, and it would not be added to the blockchain.
But the question remains: what does it mean for a masternode to orphan a block?

I would think the masternodes would reject blocks from miners that had not assigned part payment out and taken 100% themselves, or the blocks would be invalid blocks. is dash open source now?

@Fuzzybear Sure it is open-source. For a miner, to reject a block means to refuse to mine on top of it. But what does it mean for a masternode to "reject a block"?

I believe the block reward is split at the time of the block generation and this means that in order for the block to be valid it must send 45% to a masternode.  As all masternodes are aware of each other on the network (need to be online 24/7 for over a week) and their receiving address (need 1000 Dash in an address at the node online) so they can each verify if the correct distribution has been made and thus is a valid block.  Further info on Masternodes
If the miners try to take 100% of the reward the masternodes would not accept this as a verified block and thus would not allow the chain to be broardcast through them to other nodes.
This is the same way as how Devcoin works where % goes to the miners and % goes to the devtome distribution for work done.  The miners in devcoin have no say over the distribution of the shares only that it is part of the block that they can mine and read the distribution from a receiver file uploaded each month for the next round of blocks distribution.
As devcoin's model was tried and tested as was one of the earlier coins I think dash works along the same lines from looking at the code and blockchain.
https://explorer.dash.org/tx/b8f8024bb84340a0473faffa1f7d41e8b182f763ae5e876f0448d8799cb5d919 is the transaction for the generation of 911503 block
Here the two outputs are set and the distribution is made to each receiver of the % share.  The miner does not have the ability to control these outputs from the original generation of the block as far as I am aware. Would be like in bitcoin someone trying to verify a block that has generated 75 bitcoins in the block. It is just not possible to get this block accepted by other nodes as they would all reject it. As would the master nodes and other miners in the Dash situation.
Hope this helps and if anyone knows differently please feel free to explain where I am wrong, always keen to learn more :)

"As all masternodes are aware of each other on the network" -- you mean, they know each others' Dash addresses? A private coin indeed :) If not, could a miner create a block with 45/45/10 reward split, but with both 45% parts sending to their own fresh addresses? Also, "would not allow the chain to be broadcast through them" -- does it imply that network broadcast is monopolized by masternodes? Or can non-master full nodes just broadcast txs and blocks as in Bitcoin?

"As all masternodes are aware of each other on the network" they are connected to each other and sharing the same blockchain is what i mean, they verify each others activity and validity.  "would not allow the chain to be broadcast through them" - no i mean they would reject the block as invalid

Ok so anyone can become a masternode on the dash network but does have 7 day connection requirement and need 1000+ dash in an address online. This gives the knowledge to other masternodes at of the address of a new masternode to distribute to. thus again it is out of the miners hands to chose where to distribute funds. It is part of what makes up a valid block.  run the node through a proxy for regaining some privacy as agreed ip address and dash address are then known on network.

Ok, but the question on whether blocks and txs can only propagate though masternodes remains open. What if the majority of miners just said "screw those masternodes, we'll just pretend we are on Bitcoin"? 45/45/10 structure of block reward in consensus rules -- no problem, just make all three parts point to our addresses.

What is the mechanism to ensure all masternodes are compensated evenly? What’s to stop a miner running their own masternode and then always just paying out to themself (reward + masternode reward both to themself)?


text:
Exit bitcoind gracefully in a bash script
I'm writing a bash script for some Bitcoin / Lightning experiments. I want to start bitcoind in the background (and also LN instances, but they are not the focus of this question). I want the script to shutdown bitcoind gracefully and exit on Ctrl+C. This is my attempt to implement this:
#!/bin/bash

echo "Starting bitcoind in the background..."
bitcoind &
sleep 5
echo "bitcoind started."
BITCOIND_PID=$!
echo $BITCOIND_PID

cleanup() {
    echo 'cleaning up'
    pkill $BITCOIND_PID
    exit
}

trap "cleanup" INT

# Run the script until we stop it
wait

This script produces the following output:
<some bitcoind output>
bitcoind started.
18574
2019-09-17T14:12:20Z Adding fixed seed nodes as DNS doesn't seem to be available.
^C2019-09-17T14:14:07Z tor: Thread interrupt
cleaning up
2019-09-17T14:14:07Z opencon thread exit
2019-09-17T14:14:07Z torcontrol thread exit
2019-09-17T14:14:07Z addcon thread exit
2019-09-17T14:14:07Z Shutdown: In progress...
2019-09-17T14:14:07Z net thread exit
sergei:~$ 2019-09-17T14:14:07Z msghand thread exit
2019-09-17T14:14:07Z scheduler thread interrupt
2019-09-17T14:14:07Z Dumped mempool: 2e-06s to copy, 0.004053s to dump
2019-09-17T14:14:07Z [default wallet] Releasing wallet
2019-09-17T14:14:07Z Shutdown: done

Then I have to press Ctrl+C once again to exit the script.
How do I make the script shutdown bitcoind first and then exit automatically?
Related: How to use Ctrl+C to kill all background processes started in a Bash script?, Best way to kill processes created by bash script?.

Is bitcoin-cli available? If so, bitcoin-cli stop instead of pkill $BITCOIND_PID.
My own preference is to add sleep 2 following a stop, as I've experienced issues immediately restarting the stopped daemon where it behaves like a child thread hasn't finished cleaning up before the parent exits. That may not be what's happening - I've not investigated that behaviour.

I replaced 'pkill $BITCOIND_PIDwithbitcoin-cli stop; sleep 2. Now I have: ```^C2019-09-18T09:23:50Z tor: Thread interrupt 2019-09-18T09:23:50Z Shutdown: In progress... cleaning up <...> error: Could not connect to the server 127.0.0.1:18443 <...> 2019-09-18T09:23:50Z Shutdown: done``` which probably means the stop signal is sent to bitcoind two times: "directly" and through my trap. So when bitcoin-cliis called,bitcoindis already stopped. How can I preventbitcoindfrom exiting before I callbitcoin-cli stop`?

I see the same behaviour here, with differences I attribute to testing on 0.13.2. It might not be the stop signal being sent twice - it could be an attempt to discern if the initial shutdown was successful.


text:
What is the rationale for the Lightning network's path length limit (20 hops)?
A sender in the Lightning network can choose a path with up to 20 intermediaries (BOLT4).
Tor is another network that uses onion routing for anonymity. However, in Tor all paths have 3 intermediaries, and longer paths are explicitly advised against (source 1, source 2). The sources make a point that attacks mostly rely on compromising the first and the last nodes, and increasing the number of nodes in between doesn't improve anonymity, but increases latency.
Do similar arguments apply to Lightning (maybe for path of slightly higher lengths but still under 20 hops)? Where does the number 20 come from? Would it be a good idea to decrease it?

The onion routing in Lightning is based on the Sphinx Mix Format described in this paper. The situation is different from Tor

We don't have entry and exit nodes in Lightning as all nodes are inside the Network.
In Tor a new connection along the path is as easy as creating a TCP socket between the tor routing nodes. In Lightning a payment channel with enough liquidity would have to be available and can't be spawn instantaneously.

The latter argument already calls for longer paths as nodes cannot just ask routing nodes to make certain hops. (That might change though if the trampoline network is coming to Lightning with which your Tor argument might hold.) Let us additionally assume I would always have paths of length 3 in Lightning. A routing node would basically see who paid whom as there are the incoming and outgoing HTLCs.
The main reasons for the Sphinx packet to be 20 hops are - as far as I understand - the following:

The per hop data is 65 Byte and with 20 hops it will always fit into a small TCP/IP package giving no clues for traffic analysis and timing attacks.
We assume that the Lightning Network will still somehow be a small world network with a rather small diameter, so 20 hops should be sufficient.

Has anyone done any calculations on whether a smaller number of hops would also be sufficient?

if you look at 41m twitter users with more than a billion edges http://konect.uni-koblenz.de/networks/twitter you will see that the diameter is 23. in that case not everyone would be able to pay everyone. However looking at https://en.wikipedia.org/wiki/Six_degrees_of_separation which has been validated even by facebook https://research.fb.com/blog/2016/02/three-and-a-half-degrees-of-separation/ also in a lightning network it seems safe to assume that you should be able to reach about anyone with 6 hops. https://bitcoinvisuals.com/ln-eccentricity shows  the diameter was always below 10

It is worth to point out that depending on the implementations of the nodes in the route, the path lenght limit could be lower than 20. This is due to the cltv_expiry_delta reduction at each hop that could become negative before the 20 hops.


text:
Where did Satoshi Nakamoto state that bitcoin is gold or digital gold?
I'm confronted by someone who states that Satoshi Nakamoto never compared Bitcoin to gold in the monetary sense (not about mining).
However, I'm sure that I recall a quote from Satoshi stating Bitcoin is digital gold which was used several years ago in the debate of Bitcoin vs Bitcoin Cash as an argument that Bitcoin is a store of value and not a means of payment.
I'm failing to find it again. The only thing I found is about Bitcoin mining, but this isn't in the meaning I need to prove.

There is an allegedly complete collection of Satoshi's public writings at https://satoshi.nakamotoinstitute.org/.  The github repo has everything in json format which you could search for the word "gold".  I got 66 hits which you should be able to go through fairly fast.

This?

@Nat I said in monetary terms so not about mining.

This would seem to be the next closest thing.  Barring that, potentially a misquote?

In pondering what the usefulness of Bitcoin might be, there are many interesting and pertinent considerations, but "What Satoshi intended" isn't really one of them. Perhaps his motivation and intent is of historical interest, but it does not change what Bitcoin is today. And anyways, arguments about 'store of value vs payments' are naïvely mutually exclusive.

@chytrik the problem when you have someone citing is own Bitcoin proponent that Bitcoin was never meant to be scarce (not on purpose). Then tour best bet is to refer Satoshi.

Thats not a problem, people are free to have opinions about what Satoshi intended, but such opinions have no affect on what bitcoin is. Someone arguing that it "was never meant to be scarce" can simply be directed to the lines of code that defined the block subsidy in the original codebase. We don't need to read between the lines of Satoshi's writings on that point, scarcity was hard-coded in from day one. Whether that means Satoshi saw BTC as a gold equivalent (or not) doesn't change the reality of it being such a thing (or not).

@chytrik in my case it s used as an argument that the fiat credit system was meant to cohoxist with Bitcoin and thus I shouldn t search borrowing in Bitcoin. In contrast, if Bitcoin scarcity is on purpose then it s a step at demonstrating that Bitcoin is meant to replace fiat as seen by the ecb and many of it s proponents and that having a proper credit system on Bitcoin is a milestone or not.

@chytrik The argument that Bitcoin is not digital gold has to be defeated for me to receive answers in my search or founding of a service for borrowing in Bitcoin. As I m paid in cryptocurrency, this is the only thing I can borrow safely against my income which is why I care about the answer.

@chytrik so it s about wiewing scarcity as a requirement for the payment system to work or as an on purpose design decision.

@user2284570 I think that the questions you posed in your comment replies are great, but I mean to point out that the answer to them doesn't depend on what Satoshi intended, it instead depends on what the reality of Bitcoin is. And in the debate about what Bitcoin is, opinions about Satoshi's intent are rather inconsequential.

Where did Staoshi Nakamoto stated that bitcoin is gold or digital gold?

He didn't.
Not in any of his public writings.

an argument that Bitcoin is a store of value and not a meant of payment.

In his original whitepaper, Satoshi Nakamoto was very clear that they saw Bitcoin as a means of payment.

What is needed is an electronic payment system

The whitepaper does not mention the phrase "store of value". Since they were undoubtedly aware of this property of currencies, we must infer it was not of primary importance in the motivation for the design of Bitcoin.
So it would be surprising if there is any evidence available that would conclusively prove that Nakomoto completely changed their mind about this.

See also Breakdown of all Satoshi's Writings Proves Bitcoin not Built Primarily as Store of Value

Did Satoshi build Bitcoin to serve primarily as a store of value, or did he build it for making payments?
To answer this question, I went straight to the source. Thanks to the Satoshi Nakamoto Institute’s archives I was able to read every single thing Satoshi ever posted publicly. Comments from 260 forum threads, 63 emails, and his original source code.
After reviewing all of Satoshi’s writings, I can confidently state that Bitcoin was not purpose-built to first be a store of value. It was built for payments.

Satoshi compared Bitcoin mining to gold in the digital realm in his original design paper (white paper): https://bitcoin.org/bitcoin.pdf
Satoshi wrote:

"The steady addition of a constant of amount of new coins is analogous to gold miners expending resources to add gold to circulation. In our case, it is CPU time and electricity that is expended."

This is literally Satoshi drawing a parallel between Bitcoin and gold in terms of their creation process and is akin to saying Bitcoin is like digital gold. Both Bitcoin and gold involve a resource-intensive extraction process.

The question is titled "Where did Staoshi Nakamoto state that bitcoin is gold or digital gold?". Saying a characteristic of the creation of A is analogous to a characteristic of the creation of  B is not saying A is B. It is not saying that A and B have any other characteristics in common.

Hi Adelle - welcome to bitcoin.se, and thanks for contributing.  Please note that we try like to keep content helpful, direct, and on-point, and consider everyone to be asking questions in good faith.  Please take a look at the [tour] and read the expected behavior section in the [help].


text:
Is it as easy as buying 1BTC and leaving it in a wallet? is there anything else to "Bitcoin investment"?
Just as title says, I always hear about BTC lately and want to get in on it, or is it too late?

You've asked different questions in title and body.

Yes, holding them in your wallet is pretty much the thing.
Also, remember to backup and secure your wallet.
It may be hard to win the game, but it's very easy to lose it with a hard drive failure.

There is NO higher authority to turn to to help you recover misplaced bitcoins.

Would you really consider purchasing 1 BTC a valid investment? That's like purchasing 1 share of everything on the NYSE. In the end you won't earn as much as you would want to. Furthermore, you have to be a bit more interested in the success of the BTC to justify investment.
To answer the question, BTC investment is as easy as leaving 1 BTC in a wallet. The BTC will probably reach a higher value but that speculative because the entire economy is an experiment that has never been carried out in practice. This is why it is a volatile currency.
Mining is not a viable way to invest since the difficulty of successfully mining BTC has gone way up due to tech advances in hashing and BTCs increased popularity.

Considering there are going to be 7 billion people having to divide 21 million bitcoin... I'd say 1BTC is a pretty good investment... and fairly low risk in absolute terms. (saying this 1 hour after it dropped from 900 to 600 USD...)

@Jannes: This assumes, of course, that anywhere near 7 billion people are actually going to want any of those 21 million bitcoins.  That's by no means a given.

Yes, investing in Bitcoin basically means buying bitcoins and holding them in your Bitcoin wallet.
You can still invest, it's not too late. But nobody knows if you'll lose everything or make a profit.


text:
Why is wtxid called "hash" in Bitcoin Core's getrawtransaction RPC?
In the industry txid is known as "transaction hash". Why wasn't the "hash" field called "wtxid"?

The idea is that it is returning the hash of the entire transaction data. Currently, that is always identical to the wtxid.
But in a hypothetical future, where more data may get added to transaction serializations, in addition to the base data (covered by txid) and witness data, hash would change to always cover everything.


text:
Is the "witness commitment" introduced in bip-0141 optional?
In BIP-0141, it states:

a new block rule is added which requires a commitment to the wtxid.
the coinbase's input's witness must consist of a single 32-byte array for the witness reserved value.

However, I spotted two recent blocks (732085 and 767471) that broke the mentioned rules and didn't include a scriptPubKey starting with 0x6a24aa21a9ed
And, their txinwitness field is 5b5032506f6f6c5d5b5032506f6f6c5d5b5032506f6f6c5d5b5032506f6f6c5d, which decodes to the text '[P2Pool][P2Pool][P2Pool][P2Pool]'
Apparently, both the above rules stipulated in BIP-0141 were ignored.
So, unconforming to BIP-0141 (or other BIPs) by miners doesn't render the blocks invalid?

The commitment is placed on an output's spk. In 732085, is the first output.
6a24aa21a9ed69e5a4d80fbc380c2fef1fbb91f0184ffd74e4612f6c6d7ca9d08c7e995c583c

This satisfies the first rule. The second one, since this value has no consensus meaning yet, any 32-byte array can be placed there. They pushed a silly string, but it's 32-byte, so rule 2 isn't violated either.
Segwit commitment is only optional if there is no segwit tx in a block.

Thank you. I just realized the my calculation of witness commitment was wrong. I assumed the reserved value to be 32 bytes of zeros.

The second block also has a witness commitment.


text:
Trying to figure out non-canonical DER signature error for a specific tx
I have two transactions, they produce the same hash. But the first one throws Non-canonical DER signature error and the second one doesn't. This is a standard 2/3 multi-sig transaction.
Tx 1:
020000000001018b4025d76e67ac4b05010832917b3407ee11c17f3ac1f8537ad92dbda905d2760000000000ffffffff020355ea4000000000160014104d11991b81471fce9439dfb22ff144fb5ecf0cabdca700000000001600145809ea020e7a3ff5c09f57b0a52c98ae7e7aee7a040047304402201b4b4642b9bea7274ea74a39d7657867308efbd5866ccedf62f5e2cd0b5f2c9902200996fb466241a1a715d9a93128fe5efd8bde0a3453c109ea0b47a1b0dd4d38150147304402201db836dacb6290d4014f87ecd95ec5fe1c618973437d0e86f2a6266b8861e20c02207415a722f64070f12d2d2b18031b6c9f1829a88dfb8c0f871571a57aaf34930b016952210325da1b6830e251da8acb91d09d7fc162568e9c68dd12bd0f94643fed936dc87c21035497f2acb88278b2323d88f1bc353c0daa3c42abc289fab6ccbd8edb718ae8cd2103cbe6dec1b7944f5e4d4385acaf2fb6980eaa54169b6d422e24db2de61e01a5f553ae00000000

Tx 1 decoded:
{
    "addresses": [
        "ltc1qunwwcdm45r94n0vun35pvhh8d7duxzfcxe4dc9vlwlhe9ych7ufqvp9nn3",
        "ltc1qzpx3rxgms9r3ln55880mytl3gna4ancvuw4pyz",
        "ltc1qtqy75qsw0glltsyl27c22tyc4el84mn65mnej2"
    ],
    "block_height": -1,
    "block_index": -1,
    "confirmations": 0,
    "double_spend": false,
    "fees": 300,
    "hash": "b86b27a622ce8b5336debda81e0dc33e8ab3eb07354789e4bc708e38caa2dce4",
    "inputs": [
        {
            "addresses": [
                "ltc1qunwwcdm45r94n0vun35pvhh8d7duxzfcxe4dc9vlwlhe9ych7ufqvp9nn3"
            ],
            "age": 2718775,
            "output_index": 0,
            "output_value": 1100100314,
            "prev_hash": "76d205a9bd2dd97a53f8c13a7fc111ee07347b91320801054bac676ed725408b",
            "script_type": "pay-to-witness-script-hash",
            "sequence": 4294967295,
            "witness": [
                "",
                "304402201b4b4642b9bea7274ea74a39d7657867308efbd5866ccedf62f5e2cd0b5f2c9902200996fb466241a1a715d9a93128fe5efd8bde0a3453c109ea0b47a1b0dd4d381501",
                "304402201db836dacb6290d4014f87ecd95ec5fe1c618973437d0e86f2a6266b8861e20c02207415a722f64070f12d2d2b18031b6c9f1829a88dfb8c0f871571a57aaf34930b01",
                "52210325da1b6830e251da8acb91d09d7fc162568e9c68dd12bd0f94643fed936dc87c21035497f2acb88278b2323d88f1bc353c0daa3c42abc289fab6ccbd8edb718ae8cd2103cbe6dec1b7944f5e4d4385acaf2fb6980eaa54169b6d422e24db2de61e01a5f553ae"
            ]
        }
    ],
    "outputs": [
        {
            "addresses": [
                "ltc1qzpx3rxgms9r3ln55880mytl3gna4ancvuw4pyz"
            ],
            "script": "0014104d11991b81471fce9439dfb22ff144fb5ecf0c",
            "script_type": "pay-to-witness-pubkey-hash",
            "value": 1089099011
        },
        {
            "addresses": [
                "ltc1qtqy75qsw0glltsyl27c22tyc4el84mn65mnej2"
            ],
            "script": "00145809ea020e7a3ff5c09f57b0a52c98ae7e7aee7a",
            "script_type": "pay-to-witness-pubkey-hash",
            "value": 11001003
        }
    ],
    "preference": "low",
    "received": "2024-07-12T17:33:01.857762451Z",
    "relayed_by": "3.239.118.85",
    "size": 367,
    "total": 1100100014,
    "ver": 2,
    "vin_sz": 1,
    "vout_sz": 2,
    "vsize": 177
}

Tx 2:
020000000001018b4025d76e67ac4b05010832917b3407ee11c17f3ac1f8537ad92dbda905d2760000000000ffffffff020355ea4000000000160014104d11991b81471fce9439dfb22ff144fb5ecf0cabdca700000000001600145809ea020e7a3ff5c09f57b0a52c98ae7e7aee7a040047304402201db836dacb6290d4014f87ecd95ec5fe1c618973437d0e86f2a6266b8861e20c02207415a722f64070f12d2d2b18031b6c9f1829a88dfb8c0f871571a57aaf34930b014730440220780eedf386e77f5acaceaf948251fa3bbf51e7fcc401bf7973a08f65b99eadc702200019a281127211a29c5b827b80967154d05b9224f6bf7b77baff13e63d576545016952210325da1b6830e251da8acb91d09d7fc162568e9c68dd12bd0f94643fed936dc87c21035497f2acb88278b2323d88f1bc353c0daa3c42abc289fab6ccbd8edb718ae8cd2103cbe6dec1b7944f5e4d4385acaf2fb6980eaa54169b6d422e24db2de61e01a5f553ae00000000

Tx 2 decoded:
{
    "addresses": [
        "ltc1qunwwcdm45r94n0vun35pvhh8d7duxzfcxe4dc9vlwlhe9ych7ufqvp9nn3",
        "ltc1qzpx3rxgms9r3ln55880mytl3gna4ancvuw4pyz",
        "ltc1qtqy75qsw0glltsyl27c22tyc4el84mn65mnej2"
    ],
    "block_height": -1,
    "block_index": -1,
    "confirmations": 0,
    "double_spend": false,
    "fees": 300,
    "hash": "b86b27a622ce8b5336debda81e0dc33e8ab3eb07354789e4bc708e38caa2dce4",
    "inputs": [
        {
            "addresses": [
                "ltc1qunwwcdm45r94n0vun35pvhh8d7duxzfcxe4dc9vlwlhe9ych7ufqvp9nn3"
            ],
            "age": 2718775,
            "output_index": 0,
            "output_value": 1100100314,
            "prev_hash": "76d205a9bd2dd97a53f8c13a7fc111ee07347b91320801054bac676ed725408b",
            "script_type": "pay-to-witness-script-hash",
            "sequence": 4294967295,
            "witness": [
                "",
                "304402201db836dacb6290d4014f87ecd95ec5fe1c618973437d0e86f2a6266b8861e20c02207415a722f64070f12d2d2b18031b6c9f1829a88dfb8c0f871571a57aaf34930b01",
                "30440220780eedf386e77f5acaceaf948251fa3bbf51e7fcc401bf7973a08f65b99eadc702200019a281127211a29c5b827b80967154d05b9224f6bf7b77baff13e63d57654501",
                "52210325da1b6830e251da8acb91d09d7fc162568e9c68dd12bd0f94643fed936dc87c21035497f2acb88278b2323d88f1bc353c0daa3c42abc289fab6ccbd8edb718ae8cd2103cbe6dec1b7944f5e4d4385acaf2fb6980eaa54169b6d422e24db2de61e01a5f553ae"
            ]
        }
    ],
    "outputs": [
        {
            "addresses": [
                "ltc1qzpx3rxgms9r3ln55880mytl3gna4ancvuw4pyz"
            ],
            "script": "0014104d11991b81471fce9439dfb22ff144fb5ecf0c",
            "script_type": "pay-to-witness-pubkey-hash",
            "value": 1089099011
        },
        {
            "addresses": [
                "ltc1qtqy75qsw0glltsyl27c22tyc4el84mn65mnej2"
            ],
            "script": "00145809ea020e7a3ff5c09f57b0a52c98ae7e7aee7a",
            "script_type": "pay-to-witness-pubkey-hash",
            "value": 11001003
        }
    ],
    "preference": "low",
    "received": "2024-07-12T17:39:55.39046822Z",
    "relayed_by": "3.227.22.189",
    "size": 367,
    "total": 1100100014,
    "ver": 2,
    "vin_sz": 1,
    "vout_sz": 2,
    "vsize": 177
}

Why is the first one failing? What am I doing wrong?
I'm using this python lib: litecoin-utils

Is it tx1 or tx2 that is failing? Your question says tx1, although analyzing these txs, I believe it should be tx2.

Although the field is named "hash", it is not actually a hash of the entire transaction. That is actually the txid. Since the inputs are segwit, the signatures are excluded from the txid calculation. As the only difference between these two transactions are the signatures, they have the same txid.
Looking at the signatures of both transactions, I believe it is actually tx2 that is failing. The error message "Non-canonical DER signature" gives us a hint of what to look for.
ECDSA signatures using the DER encoding, and since the activation of BIP 66, must adhere to a strict set of rules. Analyzing the signatures following the rules specified in the BIP, I find that the signature 30440220780eedf386e77f5acaceaf948251fa3bbf51e7fcc401bf7973a08f65b99eadc702200019a281127211a29c5b827b80967154d05b9224f6bf7b77baff13e63d57654501 in tx2 fails the rule that "Null bytes at the start of S are not allowed, unless S would otherwise be interpreted as a negative number." The S value in the signature is 0019a281127211a29c5b827b80967154d05b9224f6bf7b77baff13e63d576545 and we can immediately see that it begins with a null byte. The byte following it is not greater than or equal to 0x80, so this signature fails this check and is therefore a non-canonical signature.
The correct encoding of this signature would simply exclude that null byte and adjusting the lengths accordingly. The correct signature should be 30430220780eedf386e77f5acaceaf948251fa3bbf51e7fcc401bf7973a08f65b99eadc7021f19a281127211a29c5b827b80967154d05b9224f6bf7b77baff13e63d57654501

I just checked again and you're correct. It is Tx 2 that is failing. Thank you for pinpointing the null byte in R. Is there a way to quickly analyze these values using any tool so that I could've found the issue myself?
I have a few more txns failing and apparently, the error rate is 1/300.
I'll try and fix the bull byte issue and see if that reduces the error rate. If it does, that litecoin-utils lib definitely needs a PR as fix.

Nit: R is the first integer in a signature; the second integer (with the error here) is S. But this doesn't make a difference because DER rules are the same for all integers -- as defined in ASN.1; the coordinate(s) in a public key for a curve on an F_p field (including secp256k1) are semantically integers, but the X9 representation makes them an octet string in ASN.1, so the ASN.1 integer rules don't apply there.

@dave_thompson_085 Oops, good catch, fixed.


text:
How to remove a tx from local node mempool that's been removed from all other node
I'm running a fullnode with electrumx on top for indexing. I've noticed that sometimes, due to low fee, a tx gets dropped from mempool after some hours/days. However, it stays for longer periods in my node. Here's bitcoind config.
daemon=1
server=1
txindex=1
listen=1
maxmempool=350
mempoolexpiry=48
mempoolfullrbf=1
maxconnections=42
minrelaytxfee=0.00001

Here mempoolexpiry is set as 48. Is this what's preventing that txn from getting dropped for two days even though it's already dropped from other mempools? What would be an ideal number for this parameter?

What mempool are you comparing to? Do you mean mempool.space?

@VojtěchStrnad I checked in 3 explorers. mempool.space, blockstream, blockchair. They all say tx not found.

Okay, so purging txn below certain feerate is what seems to be causing this problem. Other nodes are probably purging aggressively. While my node keeps that txn for 2 days before dropping them.

As far as I know, mempool.space never purges transactions and has its mempool capped at 2 GB, so if it doesn't have your transaction it's more likely that it never propagated to it through the network.

I see. Would you know what it means on their home page Purging < 9.85 sat/vB

That refers to the default mempool of 300 MB. Transactions below that fee rate don't propagate through most nodes.

Default value for mempoolexpiry is 336. That is two weeks. Your value is 48 so that means that a transaction in your mempool for more than 2 days will be dropped. I consider that value to be inappropriate (too low) given the current mempool size (+300 blocks). Is there any reason for that value in your node?
Also, any node do not (re)transmit txs that are already on the mempoool. That is a anti-spam measurement. It's possible that you received the transaction when you were starting your node (poorly connected) by another starting node also poorly connected, so that transaction got stuck in both mempools. Also the agent (maybe yourself) that send the transaction in first place has not re-transmitted the transaction since then.
There is no way to remove that tx from your mempool but you can rebroadcast it again by other means (i.e. https://bitaccelerate.com/pushtx/) and wait to be mined.
I've also found that my node (which feeds my webpage https://mempoolexplorer.com) sometimes has transactions that any other service have. Maybe for the same reason.

2 days works for our specific usecase. If the txn is indeed mined after a week, it will be synced anyway as part of new block data. So this hasn't caused any problems.
In fact, I'm looking at a way to drop (or purge) txns even before 2 days when they get purged by other nodes.


text:
How to parse out the recipient address of "change" from transactions in tx?
I just read in this page that a transaction typically has two addresses, one for the recipient of a transaction and another for the "change" associated with the transaction. My question is why don't blockchain explorer websites show this information? Is it capable of being parsed?

The whole point of having change addresses in Bitcoin's design (as opposed to an account based model where every participant has a key with a balance) is hiding who the payment is for.
Change addresses don't exist at the protocol level at all. They're just outputs, to addresses, just like the payments are. Whether something is considered a payment address or a change address is information that only exists in the participants' wallets.
So, no, you can't parse this off the block chain. If that were possible it'd defeat the purpose. Of course, in some cases there may be additional information that allows you to make an informed guess about what is change. E.g. if you see an output with a round amount (say, 0.01000000 BTC) and another with a non-round amount (say, 0.00715618 BTC), you may (correctly or not) conclude that the round amount is the payment. There are also transactions that just don't have change.


text:
Does adding OP_1 to scriptSig create a fork between non-segWit and segWit nodes?
According to this, when the scriptPubKey equals to:
OP_n (with n between 0 and 16, inclusive) followed by a direct push of exactly 2 to 40 bytes inclusive
... it denotes the start of SegWit validating process. SegWit validation process include the checking whether the scriptSig is an empty field. If it is, old nodes (non-segWit) will automatically accept this as a valid script since after the "concatenation" (I know it's not a real concatenation) the top-most element in the stack will be non-zero, while the segWit nodes will go into further checking.
So, my question is: what will happen if someone just add OP_1 in scriptSig? I mean, for non-segWit nodes it would still be accepted as
a valid script since top-most element in the stack is non-zero, and for segWit nodes it will considered as invalid since scriptSig is not an empty field. Will this make some kind of fork? How is this resolved?
Additional question is what will happen if non-segWit nodes accept some "segWit" transaction as valid since top-most item is non-zero, and segWit on the other hand do not accept that transaction since something in its witness data is not okay? What is happening in this case, how is this resolved?

For any soft fork, it's possible for a block to contain a transaction violating the new consensus rules. Old nodes would accept this block and new nodes would refuse it.
However as long as more than half of the network hashrate updated to the new rules the fork would only be temporary. This is because updated miners would not mine on top of an invalid block. If the majority of the hashrate does so they would eventually create a chain with more work  than the one with the invalid block, which old nodes would therefore drop.
The lower the percentage of updated hashrate on the network, the higher the probability of a long chain containing an invalid block being created. This is why historically care has been taken to wait for miners to upgrade their nodes before full nodes started enforcing the new rules. You want to minimize the probability of an unsuspecting user receiving a payment on their non-updated node with a decent number of confirmations which eventually gets wiped as the invalid chain gets reorg'd out!
To go further:

"On the security of soft forks" a post from Pieter Wuille which examines the failure modes of a soft fork activation.
A tiny Python script by Jonas Nick which computes the probability of an old node seeing a chain longer than N blocks given the % of the network hashrate which updated to the new rules.

How will the old nodes know when to switch to the new chain? Also, won't the fork happen again when the old nodes switch to a "valid" chain because they will again have a similar problem with validation? Is the switch to a "valid" fork done only when they are updated?

Because it has more work. Bitcoin nodes follow the (valid) chain with the most work. By definition of a soft fork, the chain valid according to the new nodes is also valid according to the old nodes. If the valid chain gets longer (loosely speaking), old nodes will start following it.

Yes, but will the old node run into the problem again when validating those blocks? Is the switch to right chain only done after they are "updated"?

No, the valid blocks are valid in their view. They switch even without being updated, that's the beauty of a soft fork (combined with a majority hashrate). You should checkout the mailing list post i link above.

Oh sorry, I got confused. If I understand, the point of a soft fork is that the old nodes always accept "something" , while updated nodes have some additional rules. So, it may happen that some transaction is accepted by the old nodes but not by the new one (due to new rules), so a fork is created. The old nodes will at some point move to the right fork and accept all these block and their transactions because they are valid for them (the problem was that some of transactions were not valid for updated nodes, but they are not even part of the block). Is this correct?

The problem would occur if some transaction was accepted by the updated nodes while the old nodes reject it. In this case it would be imposible for the old and updated nodes to "synchronize" (to be on a same fork) since the old nodes will never accept these transaction as valid. In this case the fork is permanent, until the old nodes decide to move on a new rules. This is called hard fork. Is this correct?

Yes, exactly. One way of looking at it is that a soft fork tightens the validation rules whereas a hardfork loosens them. It doesn't make a soft fork necessarily safe or necessarily "compatible" though, as you could do many things by simply tightening the validation rules, but i guess it's a discussion for a different question. :)


text:
Is there any online service/API to get the complete transaction in raw form?
Is there any online service or API that allows you to get the transaction in raw form, i.e. in "byte" format? I know it can be done through bitcoin core, however, I need some online service/API. Also, I know that the mempool.space site shows certain parts of the transaction in raw format, however, I need a view of the entire transaction in raw form.
Is there such an online service/API?

I know of several public APIs that can return serialized transactions:

mempool.space: https://mempool.space/api/tx/$txid/hex
blockstream.info: https://blockstream.info/api/tx/$txid/hex
blockchain.info: https://blockchain.info/rawtx/$txid?format=hex
blockchair.com: https://api.blockchair.com/bitcoin/raw/transaction/$txid (returns a JSON where one of the fields is the serialized transaction)


text:
Why are OP_MUL and OP_DIV disabled in Bitcoin?
What is the reason why OP_MUL and OP_DIV operators were removed from Bitcoin? Is this because someone could maliciously create a script that multiplies large numbers and thus blows the stack, or is there another reason (some network attack has already happened using these operators so they were thrown out)?

OP_MUL and OP_DIV were removed by Satoshi Nakamoto in commit 4bd188c4383d6e614e18f79dc337fbabe8464c82. As with most changes in early days of the development, little rationale was explicitly given by Satoshi Nakamoto for this change.
In light of the recent shortcomings with the Script interpreter (the 1 RETURN bug was fixed only 10 commits (and 15 days) before this change for instance and there is more), it is often assumed Satoshi Nakamoto disabled some of the opcodes out of precaution. OP_MUL and OP_DIV were some of them.
I'm not aware of any exploitable bugs using OP_MUL and OP_DIV at the time, but the implementation of OP_LSHIFT (disabled in the same batch) for instance could be remotely exploited to crash a node.